brian-team/brian2 — pull request #135 by mstimberg (2013-10-08 17:34:29+00:00)
Title: Index refactoring

This is not quite ready to merge yet (the usual: testing and documentation is still too sparse), but after some major wrestling it is now working (i.e. the test suite passes and cpp_standalone works) and I'd like to get some feedback whether you see anything wrong with the approach that I missed so far.

I wrote, deleted and rewrote several approaches to the indexing system that had all their advantages and drawbacks, from fully automated nested indexing that is completely handled by codegen to almost no support from codegen where everything (synapse indices, subgroups) is handled in the templates. I think I finally found some middle ground that is reasonably understandable and simple and should work in complicated scenarios (accessing post-/presynaptic state variables in synapses via subgroups, etc.).

The major changes/assumptions/approaches are the following:
* Each `NeuronGroup` now stores an actual array containing all its indices (which is quite boring, just an arange from 0 to N). This is a bit of a waste of memory but not too bad since we know the number of neurons and can use the appropriate data type. And we don't store indices twice, the indices `i` and `j` in a `Synapses` class refer to the respective indices of their source/target, indexed appropriately.
* To make the above work with standalone, devices have to implement an additional `arange` method that works like `array` but initializes the array with integers from `start` to `stop` instead of with zeros. I added this method for the C++ standalone.
* The `ItemMapping` classes disappeared. There is now only a `calc_indices` method that returns a mapping from arbitrary (tuple/slice/array) indices to flat 1d indices appropriate for the group. For a `NeuronGroup` this does basically nothing but for synapses it converts from the `[i, j, k]` index to the synapses index. This function no longer takes care of string indices.
* There is one special index `_idx` that is defined in the templates and loops up to `N` (formerly `_num_idx`). If the index of an `ArrayVariable` is not `'_idx'`, it has to be the name of another `ArrayVariable` in the `variables` dictionary. There is no longer a special `indices` dictionary or `Index` objects. Note that this does currently not allow for arbitrary nesting: The index for the `ArrayVariable` used as an index has to be `_idx`, it can't be another `ArrayVariable`, therefore we do not have to do a sorting of the dependencies or something like that. Currently, this is not checked, though, it will just fail in the generated code.
* Subgroups are implemented via `Subexpression` objects that subtract the offset from the index array of the referred group (i.e. subgroups do not store another array). The `Synapse` object stores the indices of the *real* targets (this allows direct access to target variables) but the list-of-synapses-per-neuron structure uses relative subgroup-specific indices. All this greatly reduces the number of places that have to be "subgroup-aware": everything that is going through code generation automatically takes subgroups into account, the only remaining spots in the code (apart from `Subgroup` itself) are the `push_spikes` method of `Synapses`, the `calc_indices` method of `Synapses` (used when indexing `i` and `j` with numbers, not with code), the creation of synapses, and the `SpikeMonitor` template.
* This indexing system also means that `_presynaptic_idx` and `_postsynaptic_idx` are automatically taken care of, we no longer need special synaptic templates for stateupdate or variable setting.
* Finally, variable setting now works in many cases for standalone, for convenience `G.v = -60*mV` is automatically interpreted as `G.v = '-60*mV'`.
* Because the `i` index of a `NeuronGroup` is now a standard (integer) array stored in the group I needed to get the data types right. I introduced `AuxiliaryVariable` objects that one can use for specifying the type of auxiliary variables (very similar to the previous `OutputVariable` objects). This is not used everywhere yet, though. This fixes #63.
* There's a tricky issue surrounding the use of post- or presynaptic subexpressions in synaptic expressions: if the expression is `I = x + y` and is referred to as `I_post` in the synapse code, this will simply use the code `x + y` -- but these two variables could potentially be defined as synaptic variables and therefore evaluated incorrectly. I changed the way subexpressions are handled so that this is now checked and if there is a clash, `x_post + y_post` would be used instead.

In general, with all these changes it should now be possible to use `i`, `j`, references to neuronal state variables, etc. everywhere which is pretty nice I think :)

Some known issues:
* Using small data types to store the indices for neurons can lead to nasty problems: if a group with 100 neurons stores its indices as int8, then an expression like `i + j` in a synapse will lead to a wrap around... I currently reduce the number of situations where this occurs by treating all literal numbers as floats (i.e. `1` becomes `1.0`) but still... Probably we should simply not care about saving some kilobytes of memory and always use int32 to be safe?
* One currently can't use `k` in synaptic string expressions.
* Related to the `Subexpression` issue I mentioned above, there's still some possibility for mistakes if the external namespace is manually defined for a `NeuronGroup`. If the `Subexpression` is evaluated in the context of a `Synapse`, it will use the synapses's namespace, not the one from the `NeuronGroup`.