acquaman/acquaman — Issue #243: Looking past AMInMemortyDataStore
Author: dretrex
Created: 2012-03-01 03:07:09+00:00
State: open
Labels: Impact - High, Module - Data Management, Reason - Missing from API

Unfortunately, I think I'm coming up to this wall before anybody else, but I need to start thinking of another kind of data store that doesn't hold all of the data in memory. Current examples are the ones I've depicted in #235. These are alright for now, but already beginning to pose problems because long long scans will fill up a ton of memory (ie: 17 scans brought up AM from 600 MB to 1800 MB). With scans of that type I feel that the ability to "turn on" or "turn off" the big data sources will become important. And by turn on I don't just mean hide from users. If they are in a hidden from users state they should not be in memory.

This becomes more important with 2D map scans. Although the maps shouldn't be that bad (say 16 maps of 100x100 is still going to be less than the super huge scan I mentioned in issue 235), however, if I'm using the four element vortex detector then it will _also_ have in memory four elements and a corrected sum spectra for _every_ point. That's 100x100x2048x5 = 102.4 million extra points or over 800 MB if they were stored as a double (whereas in fact they are stored as an AMNumber which is larger).

So that's my dilemma. I would like to know what you guys think on a possible solution for this problem. For the time being, I'm likely going to have to come up with an imaginative way of dealing with those spectra data sources where I don't store them in memory or something.

Thoughts?