CellProfiler/CellProfiler — Issue #208: JIRA [IMG-569] ConserveMemory: make memory conservation automatic
Author: thouis
Created: 2011-11-23 20:50:38+00:00
State: open

Currently, the ConserveMemory module allows the user to select which images to remove from memory, in order to conserve memory. We would like the pipeline to examine itself and remove images from memory that are not going to be needed downstream in the pipeline. Lee says this most likely can be made to work although there are probably cases that might be tough, like modules that require aggregation across cycles for example.

Original reporter: anne
Jira link: https://jira.broadinstitute.org/browse/IMG-569
Assignee: leek

**Comment from mbray on Wed, 21 Apr 2010 11:04:44 -0400:**
From CP TODO:
# (Requested by Anne 12-15-05)- MEMORY ISSUES: Should we somehow refresh memory after every cycle of CP? It gets slower and slower with each cycle which can be fixed by using Restart module, so something must be up with the memory handling. See matlab's pack function, perhaps. Another idea: after every cycle should we check to see what images are used downstream (checking all the imagegroup members) and strip the rest out? This would save a lot of memory. Also, maybe try using MATLAB's function memmapfile to use memory mapping when loading really large images to see if it helps relieve'out of memory' issues. We have super-large 5x scans of entire slides to use in this testing. If this works well, we should have"run in memory mapping mode for large images?" as a preference in Set Preferences, and then we would adjust the CPretrieveimage subfunction (or whatever it's called) to check whether we are in memory mapping mode or not in order to decide how to retrieve images. It's unclear to me whether this would involve a ridiculous amount of change - for example, would we have to change every module that stores an image in the handles to store it in a particular special way?