That sounds right, both tensor and length-1-tuple should not be allowed.
It makes me realize I'm not sure at which points we should enforce that sources and spaces (and corresponding variables and values) have to be flat tuples. I think that's what I did for `SumOfCosts`, but not other `Costs`, and at each point either `make_theano_batch` or `Dataset.iterator` is called.
In fact, in both `SGD` and `BGD`, I always call `cost.expr` with a flat tuple of Theano tensors, so that's a place where I make this assumption. This is useful, because it avoids duplication of Theano variables corresponding to the same expression, and makes it easier to interact with `function` and the resulting function object.
I'm not sure where we should put these flattening/renesting operations:
- in each of the `get_data_specs` of the sub-classes of `Cost`?
- in the base `Cost` class, say in a `get_flat_data_specs`, that would be close to the current version of `SumOfCosts.get_data_specs()`?
- in each `TrainingAlgorithm`?
I think the most reasonable is the second one, but we would also need access to the mapping, so the TrainingAlgorithm can map to and from this mapping. We would also have a special flattening mode where everything returned is a tuple (tuple of component spaces, tuple of strings, even if their length is 0 or 1).