Fixed the memory consumption for secondary sorted columns