Fix bugs when memory shuffle is turned off
Add TODO