ENH #6416: performance improvements on write 

- tradoff higher memory use for faster writes. This replaces the earlier PR where the history was a mess!