markmfredrickson/RItools — Issue #10: Memory Issues with Multicore and expand.grid
Author: jwbowers
Created: 2011-10-17 14:32:02+00:00
State: open

I love that we can easily use multicore when we have multiple cores. One issue, however, that basically locked up my computer is when the multiple R sessions are spawned, each carries with it the entire relevant workspace. So, I was doing some attributable effects with `theparams<-list(t1=-10:10,t2=-10:10,t3=-10:10,t4=-10:10)` (we have four pairs and each pair could receive between -10 and 10 pct points of turnout from the experiment). Although the enumerated distribution is small and everything else is fast, each R process took up about 3Gb of RAM thereby maxing out my 10Gb RAM computer (and causing everything to lock up --- renicing was irrelevant since the issue was RAM use not CPU use).

Now, `params` is not sparse, and converting from data.frame to matrix doesn't seem to help. I am wondering about using arrays.

For example,

```
theparams<-list(t1=-10:10,t2=-10:10,t3=-10:10,t4=-10:10)
theas<-expand.grid(theparams)
object.size(theas) ## roughly 3Mb
library(abind)
junk<-abind(theparams,along=-1)
str(junk)
int [1:4, 1:21] -10 -10 -10 -10 -9 -9 -9 -9 -8 -8 ...
- attr(*, "dimnames")=List of 2
..$ : chr [1:4] "t1" "t2" "t3" "t4"
..$ : NULL
object.size(junk) ## roughly 1k
```

I am not sure is this would be a good idea or not. Just an idea that I wanted to record.