ctn-waterloo/nengo_spinnaker — pull request #103 by mundya (2014-11-03 18:31:57+00:00)
Title: Migration to new Manchester toolchain, upstream changes.

This is a horribly huge commit, sadly everything kind of needed to happen at once; I will rewrite history before merging. :japanese_ogre:

New Manchester toolchain
=====================

The new Manchester toolchain aims to be a selection of useful tools rather than a one size fits all approach to running code on SpiNNaker. This entails that Nengo/SpiNNaker take on a greater responsibility for preparing simulations for SpiNNaker. Code that performs just this purpose now lives within the `spinnaker` directory within nengo_spinnaker.

At present this code is mostly the kind required for writing the data for SpiNNaker executables to file and recording where files need to be loaded into memory. Data is represented in regions (such as those found in `spinnaker/regions.py`). Each region is responsible for reporting its requirements (DTCM and SDRAM memory usage), and for writing out values stored in subregions (partitioned versions of itself).

The "output generator" accepts a list of partitions and placements and generates files and commands for writing all region data to a SpiNNaker machine.

Todo:
- [ ] Modify the build process to call the new PACMAN place/partition and route tools
- [ ] Write a "key allocator" to associate each routing edge with an appropriate key and routing mask. This is just a case of reading the correct value from the keyspace attached to each Nengo edge.
- [ ] Modify the simulator to use the new transceiver to load all data and executables to the SpiNNaker machine (mostly just iterate and call), and control the simulation.

New Vertex model
--

A new vertex model is built for this purpose, the following objects require new vertices constructing:
- [ ] LIF Ensembles (*needs testing*)
- [ ] Filter vertices (`filter`)
- [ ] Function of time nodes (`value_source`)
- [ ] Probes (`value_sink`)
- [ ] Ethernet TX/RX elements (`sdp_tx` and `sdp_rx`)

The Connection "Tree"
==================

A big part of this rewrite is some changes toward formalising the on-host model of connections between Nengo objects. The intent is to reduce the transmission of "duplicated" packets on the SpiNNaker machine as this incurs higher than necessary memory and network utilisation. This reduction is achieved through combining compatible connections. Compatible connections originate from the same object, live in the same keyspace and transmit the same data (that is, have the same transform and function*).

For example, the code below will result in the "tree":

```
(a) ----- (conn1 & 2 combined) -----> (c)
\----> (d)

(b) ----- (conn3) ------------------> (c)
|
+------- (conn4) ------------------> (d)
|
\------- (conn5 & 6) --------------> (c)
\-------------> (d)
```

The code in `connections/connection_tree.py` represents the high-level class for building up trees of connections. Correct aliasing of connection types is based on breaking each `Connection` object into a `ReducedOutgoingConnection` and a `ReducedIncomingConnection`; each type stores the minimum of information required when transmitting or receiving packets respectively.

The connection "tree" is generated by taking each connection in turn, breaking it into an outgoing and an incoming form, and indexing these such that: `originating object -> [outgoing connection -> [incoming connections]]`. Aliasing connections relies on correct definition of the `__hash__` and `__eq__` methods.

*Function equivalence is tricky. For connections from ensembles we can check that functions are equivalent by checking they have the same `id`, or by evaluating them at all evaluation points and checking for the equivalence of the outputs. For connections from other objects we can only check `id` (or the bytecode :skull:). Some interesting results follow:

```python
with nengo.Network(label='Connection Sharing') as model:
a = nengo.Ensemble(100, 1)
b = nengo.Node(lambda t: t**2, size_in=0, size_out=1)
c = nengo.Ensemble(100, 1)
d = nengo.Ensemble(100, 1)

# These two connections will be considered equivalent (shared)
conn1 = nengo.Connection(a, c, function=lambda x: x**2)
conn2 = nengo.Connection(a, d, function=lambda x: x**2)

# Whereas these will not
conn3 = nengo.Connection(b, c, function=lambda x: x**2)
conn4 = nengo.Connection(b, d, function=lambda x: x**2)

# But these will
squared = lambda x: x**2
conn5 = nengo.Connection(b, c, function=squared)
conn6 = nengo.Connection(b, d, function=squared)
```

Upstream changes
===============

I've been trying to keep up with changes in Nengo, thorough testing will be necessary... I'm a little concerned about learning rules.

Other TODO
==========
- [ ] Write proper tests for the IO builders, probably refactor
- [ ] Write better tests for the simulator

For a different PR
==
- Correct behaviour of filters in C. Each filter routing entry should allow a single packet to be included in the input to multiple filters (or even the same one multiple times).
- Provide some mapping between the dimension number from the key of a packet to the dimension it should be included with; this will allow greater sharing of connections at the cost of some on-chip compute time and memory.
- Add "functional" tests that run through a model and compare to expected results.