Fix issue with OScalars on the device.

On the device we use the same data layout as on the host.
But the data layout changer was still in place for them.