Revert change that break too many test in floatX=float32

This was introduced recently, but this make too many tests fail. So I revert it.