Monitors

This is first attempt at automatic validation routine creation. I sidestepped the issue of selecting proper variables for monitoring - since the tagged theano variables carry the information on how to aggregate them, the minimal validator needs just the list of variables. Selection should be handled elsewhere.

My idea is that such a validator would be run in the training loop via a hook. Another function would save somewhere or print the computed values.

**Update:**
I added a helper class for monitoring during training. It takes a list of variables, allocates storage for them, creates a list of updates to compute and store the monitored values during a call of the training function and provides a helper function which reads out the storage as a dict.

I changed the names to be more descriptive, but I am still not too fond of them.

What is important is that the *Monitor classes are agnostic to the model (both read a list/dict of theano variables) and to the way the training/main loop runs as long as:
1. updates can be added to the training function - this is a common requirement
2. The trainer and main loop have provisions to run hooks after each batch/epoch - this again is quite easy to meet.

Storage of computed values should be done elsewhere. It will also be easy since the *Monitor classes return dicts from names (or other user-specified keys) to values.
On 23.12.2014 17:57, Bart van Merriënboer wrote:
>
> In blocks/extensions/monitors.py 
> <https://github.com/bartvm/blocks/pull/85#discussion-diff-22223292>:
>
> > +        initialization_updates = [(numerator_acc, 0.0),
> > +                                  (denominator_acc, 0.0)]
> > +        accumulation_updates = [(numerator_acc,
> > +                                 numerator_acc + self.numerator),
> > +                                (denominator_acc,
> > +                                 denominator_acc + self.denominator)]
> > +        aggregator = Aggregator(aggregation_scheme=self,
> > +                         initialization_updates=initialization_updates,
> > +                         accumulation_updates=accumulation_updates,
> > +                         readout_expression=numerator_acc / denominator_acc)
> > +        aggregator._numerator_acc = numerator_acc
> > +        aggregator._denominator_acc = denominator_acc
> > +        return aggregator
> > +
> > +
> > +def frac(numerator, denominator, name=None):
>
> What's wrong with |monitored_mean|? I personally really don't like the 
> |ratio| or |fraction| thing. We're not measuring the ratio between 
> examples and cost, or the fraction of activations per batch. We're 
> calculating a mean here. The only other two things that I can think of 
> that will be measured are min and max, and the set "mean, min, max" 
> makes much more sense to me than "ratio, min, max".
>
If we make it mean, it is gonna look like that (adapting the code from 
Jan's test):

monitored_variable = monitored_mean(input_.sum(), input_.shape.prod(), 
'mean_input')

How do you explain semantics of the second argument than? Note, the 
second argument is necessary in order to accumulate values from 
different batches in the right way.

With the fraction terminology used, it is denominator.
>
> —
> Reply to this email directly or view it on GitHub 
> <https://github.com/bartvm/blocks/pull/85/files#r22223292>.
>