Brick inputs and outputs are tagged with an application call

The application call can be used to e.g. include auxiliary variables, such as monitors, extra cost terms etc.

Also: fixes to graph parsing.
On 17.12.2014 15:06, Bart van Merriënboer wrote:
>
> In blocks/bricks/__init__.py 
> <https://github.com/bartvm/blocks/pull/68#discussion-diff-21972887>:
>
> > +
> > +    The application call object is created before the call to the application method and can be accessed by
> > +    specifying an application_call argument.
> > +    """
> > +    def __init__(self, brick, application):
> > +        self.brick = brick
> > +        self.application = application
> > +        self.auxiliary_variables = []
> > +
> > +    def add_auxiliary_variable(self, expression, role):
> > +        #the copy destorys the name. I believe adding a role tag is pretty harmless, so don't copy
> > +        #expression = expression.copy()
> > +        expression.tag.role=role
> > +        self.auxiliary_variables.append(expression)
> > +
> > +    def add_monitor(self, expression):
>
> Just wondering if you have given any more thought to the semantics of 
> adding monitoring channels: The variables need a bit more information 
> to be able to be interpreted as a monitoring channel e.g. do they need 
> data, and if so, what operator should be used for the reduction (i.e. 
> do we want the mean, max, etc. over batches). I guess we should add 
> that information as tags, and then search the |ComputationGraph| later 
> in order to convert them into |MonitorChannel| instances (see my 
> monitor_and_train branch), because we don't want the |Brick| class to 
> rely directly on the |MonitorChannel| interface.
>
Actually we did have a lengthy discussion with Jan about that. We 
converged at the following: in addition to inputs and outputs the user 
can construct "auxiliary variables" as results of an application call.
He can do it by using the `ApplicationCall` object which is passed as an 
`application_call` parameter to the application method. The user can 
specify how to average this variable over multiple batches by chaging 
the `.tag`:

```python
penalty = one - another
penalty.tag.averaging_method = ArithmeticMean()
```

Seems like it is pretty much the same what you mean, except for data 
specification. In fact the way it is done in PyLearn2 is quite weird, 
and Jan has some nice ideas about complete and compromise-free 
separation of monitoring and validation from the main loop class.

We did not agree though on how exactly the variables that are subject to 
monitoring should be marked. `variable.tag.role == MONITOR`? 
`variable.tag.is_monitor == True`? `hasattr(variable, "averaging_method"`?
>
> I guess an interface like this would be nice:
>
> def  apply(inputs):
>      ...
>      return  outputs
>
> @apply.monitor_channels
> def  apply_monitor_channels(self,inputs,outputs):
>      rvals=  {}
>      rvals['mean_weights']=  create_monitor_variable(self.W.mean(),needs_data=False)
>      rvals['shannon_entropy']=  create_monitor_variable(-tensor.sum(outputs*  tensor.log(outputs)),needs_data=True,reduce=np.mean)
>      return  rvals
>
> But it raises the question as to how we add monitor channels that rely 
> on targets, but maybe those shouldn't be given by bricks in the first 
> place?
>
The interface you propose does not allow to monitor intermediate results 
of the computation, does it?

Targets must be given to bricks: it is only feedforward networks when 
you can first do fprop and then compute the cost. In the case of 
structured (e.g. sequential) outputs it is often not possible, see 
`cost` method of the `BaseSequenceGenerator`.
>
> —
> Reply to this email directly or view it on GitHub 
> <https://github.com/bartvm/blocks/pull/68/files#r21972887>.
>