Free memory usage as soon as we can to prevent #1385