ctn-waterloo/nef-py — pull request #21 by jaberg (2013-03-22 21:32:25+00:00)
Title: WIP: workspace idea (do not merge!)

So what I'm calling "workspaces" have been on my radar for a long time for Theano as a way of simplifying the use of shared variables and eliminating hundreds (thousands??) of lines of crap in Theano.

The idea is to not have shared variables, instead have a workspace:
```python
x = tensor.vector('x')
y = tensor.vector('y')

ws = Workspace()
ws[x] = [1, 2]
ws[y] = [3, 4]
ws.compile_update('f', [
(x, 2 * x),
(y, x + y)])

assert np.allclose([ws[x], ws[y]],[[1, 2], [3, 4]])

# compute stuff
ws.run_update('f')
assert np.allclose([ws[x], ws[y]],[[2, 4], [4, 6]])

# trigger graph optimizations
# and potentially rearrange the layout of variables in physical memory
# copy over the values of the variables if they have moved
ws.optimize_memory_layout(device='gpu')

# now run a freshly-compiled function that is running on the GPU
ws.run_update('f')

# bring the value back from wherever it is (like shared.get_value())
print ws[x]

```

The reason to use this mechanism is so that
* an easily-debugged version of math can be built and run in the default workspace
* this basic workspace is something that we can pickle properly
* optimized workspaces can be converted back to simple ones for saving
* being able to "jointly" optimize functions based on runtime statistics is a big win, because Theano's shape inference is not strong
* __crucial for nef-py__ the optimize_memory_layout function can merge together the ensemble state information so that large numbers of ensembles can be evaluated without large numbers of graph nodes

This last point follows up on recent discussion of how to conglomerate the state information across ensembles (and create ensemble views etc.) for the purpose of running models fast. It wouldn't be the same thing as making ensembles all views, but it would allow us to get fast implementations of the *current* type of ensemble, namely, the kind that appears to manage its own memory.