Repository: spacetelescope/jwql

Issue:
Failing monitors - probably due to memory issues with large files
It looks like all instrument monitors except the bias monitor are failing. In most cases, the logs from the monitor runs are not helpful. They show that the cal pipeline or some other calculation was starting, and that's the end of the log, implying that they are failing silently. 

For the bad pixel monitor, there is a line in the log saying that there was not enough memory to create an array for the data in the file being worked on. (It needed 1.5 GB). I looked in the logs for the other monitors and in all cases, the monitors were working on especially large files at the time that they failed. e.g. 4 integrations x 50 groups, 20 integrations x 10 groups. 

We may need to adjust the instrument monitors to be able to handle these cases. One option would be for the monitors to skip these files entirely. Or perhaps we can create a module that checks the number of groups and integrations and if a file is too large, read in and work on only a portion of the data? Unfortunately for the dark monitor, the failure came while the ramp-fitting step of the pipeline was running. I'm not sure how to get around that other than skipping the file entirely. We should check and see how large the official commissioning and cycle 1 cal dark data will be, in order to know whether this is going to continue to be a problem.

@bsunnquist 


Discussion:
NIRCam's commissioning darks, at least the initial 3 visits for each module for the full frame apertures, are larger than the files mentioned above, so I would expect those to fail. The other 7 visits for each module are debatable. The files are smaller, but they are still quite large (1 int x 130 groups). 
Yeah the commisioning data is huge. I'll check out the Readnoise Monitor - the Bias Monitor probably isn't failing because I have it extracting the 0th group only before running the pipeline.

The readnoise calculation is pretty memory intensive, but it's easy enough for me to do it in chunks if that's the issue. If it's the pipeline, I could probably just use the mast dark.fits files instead.
I tried spreading out the monitor runs last night in order to reduce overlap. Looks like things still crashed though. The readnoise monitor failed because it couldn't allocate 2.5 GB for an array. The final entry in the dark monitor log is at the same time as well, so it looks like that failed silently. 

We're going to ask IT for more memory, but I think anything we can do to reduce memory usage would help. I'll have to look at the dark monitor and see what I can do. I think the biggest memory hog there is just running the ramp-fitting step of the pipeline.
Ah ok, I'll switch to doing the readnoise calculation in chunks. Now I'm wondering if memory pressure might be one of the reasons for the overall website slowness issue.
Hopefully not. The web app doesn't need to fool with fits files when rendering pages. But I guess if someone tries to open the web app while e.g. the dark monitor is running....who knows? 

#867 will hopefully help some with app slowness.

Resolving patch:
diff --git a/jwql/instrument_monitors/common_monitors/dark_monitor.py b/jwql/instrument_monitors/common_monitors/dark_monitor.py
--- a/jwql/instrument_monitors/common_monitors/dark_monitor.py
+++ b/jwql/instrument_monitors/common_monitors/dark_monitor.py
@@ -522,6 +522,9 @@ def process(self, file_list):
         mean_slope_file = self.save_mean_slope_image(slope_image, stdev_image, slope_files)
         logging.info('\tSigma-clipped mean of the slope images saved to: {}'.format(mean_slope_file))
 
+        # Free up memory
+        del slope_image_stack
+
         # ----- Search for new hot/dead/noisy pixels -----
         # Read in baseline mean slope image and stdev image
         # The baseline image is used to look for hot/dead/noisy pixels,


Test patch: