Reduce the number of allocation/reuse allocation of ndarray #1943

#1943 
@nouiz 

Hey, Fred, this is my draft idea, is the direction right?

If the direction is right, what I wanna do in the future is fill the details, and optimize it to reduce the "if" inside the loop.
I think it is not a good idea to try to find right now which node to put it. Otherwise, you can change the same variable multiple time if multiple previous variable is able to reset it.

I think you should base yourself on the count_running_memory methods in profiling.py. There, when we reduce the current memory usage, it mean we can reuse this variable. Not before. (I didn't told this in the issues).

Here, you do not check if the variable you try to reallocate have a other variable in the graph that will reuse it. Maybe ins will be reused by other node in the graph. Doing this isn't trivial, as there can be view and destroyed version of this variable. count_running_memory already handle those cases.

So base yourself on count_running_memory. When you "free" a variable (that mean to -= on that variable), put it in a list of node available for reuse.

Then when there is node available for reuse, you can reuse them for output variable.