Reduce the number of allocation/reuse allocation of ndarray #1943

#1943 
@nouiz 

Hey, Fred, this is my draft idea, is the direction right?

If the direction is right, what I wanna do in the future is fill the details, and optimize it to reduce the "if" inside the loop.