Redo the previous fix a little more carefully

Reuse data allocation where possible and properly define dimensions
for the slow case.