boulanni/theano-hf — pull request #1 by gwtaylor (2012-04-23 16:06:45+00:00)
Title: Small fixes to make examples work out-of-the-box

Hi Nicolas,
Thank you for making the code available. It's really nice.

There were a couple of small fixes I needed to make to get the examples work out of the box.

First, by default, self.function_Gv was throwing an exception because it is getting passed some extra inputs (on which the output is not dependent). This behaviour in Theano may have changed since you released this code. Anyways, the quick fix is to add on_unused_input='ignore' to the function arguments.

The other issue is with:
```python
if isinstance(validation, SequenceDataset)
```

If the validation object is created outside of this file, isinstance() will not work. For example, if I just do:

```
>>> from hf_examples import *
>>> example_NN(hf=True)
```

The above isinstance() line will evaluate to false, because validation.__class__ is hf.SequenceDataset, not SequenceDataset. I've fixed this by avoiding isinstance() and relying on ```__class__.__name__```.

Finally, there's one more issue I'm experiencing that is not addressed by this pull request. I find that each time I run the simple NN experiment with HF (i.e. issuing the two lines above), after the cost drops to the 10^(-9) range, phi and cost suddenly go to NaN during the CG iterations, and it does not recover from this point onward. I've tried it with different random seeds and it continues to happen.
Here's some sample output which demonstrates the problem:

```
In [55]: example_NN(hf=True)
update 1/100, cost= [ 0.8742049 0.499912 ] lambda=1.00000, [CG iter 20, phi=-0.19082, cost=0.67980] backtracked 8/20 validation= [ 0.67970671 0.500568 ] *NEW BEST
update 2/100, cost= [ 0.67947162 0.49913 ] lambda=0.66667, [CG iter 17, phi=-0.00896, cost=0.66289] backtracked 11/17 validation= [ 0.66305449 0.500548 ] *NEW BEST
update 3/100, cost= [ 0.66283585 0.499496 ] lambda=0.44444, [CG iter 13, phi=-0.01077, cost=0.64215] backtracked 9/13 validation= [ 0.64230651 0.250172 ] *NEW BEST
update 4/100, cost= [ 0.6422807 0.250248 ] lambda=0.29630, [CG iter 14, phi=-0.01410, cost=0.61453] backtracked 14/14 validation= [ 0.61452358 0.250744 ] *NEW BEST
update 5/100, cost= [ 0.61449523 0.25117 ] lambda=0.19753, [CG iter 15, phi=-0.02072, cost=0.57149] backtracked 14/15 validation= [ 0.57122654 0. ] *NEW BEST
update 6/100, cost= [ 0.57141034 0. ] lambda=0.13169, [CG iter 16, phi=-0.03522, cost=0.49668] backtracked 14/16 validation= [ 0.49660948 0. ] *NEW BEST
update 7/100, cost= [ 0.49647298 0. ] lambda=0.08779, [CG iter 17, phi=-0.06120, cost=0.36942] backtracked 14/17 validation= [ 0.36962343 0. ] *NEW BEST
update 8/100, cost= [ 0.36967558 0. ] lambda=0.05853, [CG iter 17, phi=-0.08492, cost=0.21679] backtracked 14/17 validation= [ 0.21664556 0. ] *NEW BEST
update 9/100, cost= [ 0.21692665 0. ] lambda=0.03902, [CG iter 17, phi=-0.07354, cost=0.13903] backtracked 3/17 validation= [ 0.11308892 0. ] *NEW BEST
update 10/100, cost= [ 0.11306463 0. ] lambda=0.02601, [CG iter 18, phi=-0.04514, cost=0.08512] backtracked 3/18 validation= [ 0.04768618 0. ] *NEW BEST
update 11/100, cost= [ 0.04770465 0. ] lambda=0.01734, [CG iter 16, phi=-0.01645, cost=0.02605] backtracked 14/16 validation= [ 0.02604803 0. ] *NEW BEST
update 12/100, cost= [ 0.02605669 0. ] lambda=0.01156, [CG iter 15, phi=-0.00911, cost=0.01485] backtracked 3/15 validation= [ 0.01214502 0. ] *NEW BEST
update 13/100, cost= [ 0.01215126 0. ] lambda=0.00771, [CG iter 13, phi=-0.00379, cost=0.00673] backtracked 8/13 validation= [ 0.00673069 0. ] *NEW BEST
update 14/100, cost= [ 0.0067307 0. ] lambda=0.00514, [CG iter 14, phi=-0.00217, cost=0.00417] backtracked 2/14 validation= [ 0.00347982 0. ] *NEW BEST
update 15/100, cost= [ 0.00347986 0. ] lambda=0.00343, [CG iter 14, phi=-0.00100, cost=0.00200] backtracked 11/14 validation= [ 0.00200425 0. ] *NEW BEST
update 16/100, cost= [ 0.00200428 0. ] lambda=0.00228, [CG iter 14, phi=-0.00058, cost=0.00121] backtracked 14/14 validation= [ 0.00121148 0. ] *NEW BEST
update 17/100, cost= [ 0.00121162 0. ] lambda=0.00152, [CG iter 14, phi=-0.00036, cost=0.00076] backtracked 14/14 validation= [ 0.00076248 0. ] *NEW BEST
update 18/100, cost= [ 0.00076274 0. ] lambda=0.00101, [CG iter 14, phi=-0.00024, cost=0.00049] backtracked 14/14 validation= [ 0.00048572 0. ] *NEW BEST
update 19/100, cost= [ 0.00048582 0. ] lambda=0.00068, [CG iter 14, phi=-0.00015, cost=0.00031] backtracked 14/14 validation= [ 0.00030584 0. ] *NEW BEST
update 20/100, cost= [ 0.00030594 0. ] lambda=0.00045, [CG iter 14, phi=-0.00009, cost=0.00019] backtracked 14/14 validation= [ 0.00019072 0. ] *NEW BEST
update 21/100, cost= [ 0.00019074 0. ] lambda=0.00030, [CG iter 14, phi=-0.00006, cost=0.00012] backtracked 14/14 validation= [ 0.00011746 0. ] *NEW BEST
update 22/100, cost= [ 0.00011744 0. ] lambda=0.00020, [CG iter 14, phi=-0.00003, cost=0.00007] backtracked 8/14 validation= [ 7.17567731e-05 0.00000000e+00] *NEW BEST
update 23/100, cost= [ 7.18246055e-05 0.00000000e+00] lambda=0.00013, [CG iter 14, phi=-0.00002, cost=0.00004] backtracked 14/14 validation= [ 4.36872329e-05 0.00000000e+00] *NEW BEST
update 24/100, cost= [ 4.36837772e-05 0.00000000e+00] lambda=0.00009, [CG iter 14, phi=-0.00001, cost=0.00003] backtracked 14/14 validation= [ 2.65775183e-05 0.00000000e+00] *NEW BEST
update 25/100, cost= [ 2.66045366e-05 0.00000000e+00] lambda=0.00006, [CG iter 13, phi=-0.00001, cost=0.00002] backtracked 9/13 validation= [ 1.62784217e-05 0.00000000e+00] *NEW BEST
update 26/100, cost= [ 1.62797180e-05 0.00000000e+00] lambda=0.00004, [CG iter 12, phi=-0.00000, cost=0.00001] backtracked 9/12 validation= [ 1.00432709e-05 0.00000000e+00] *NEW BEST
update 27/100, cost= [ 1.00459375e-05 0.00000000e+00] lambda=0.00003, [CG iter 12, phi=-0.00000, cost=0.00001] backtracked 9/12 validation= [ 6.26165187e-06 0.00000000e+00] *NEW BEST
update 28/100, cost= [ 6.26432712e-06 0.00000000e+00] lambda=0.00002, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 3.94180473e-06 0.00000000e+00] *NEW BEST
update 29/100, cost= [ 3.94192270e-06 0.00000000e+00] lambda=0.00001, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 2.49794640e-06 0.00000000e+00] *NEW BEST
update 30/100, cost= [ 2.49911835e-06 0.00000000e+00] lambda=0.00001, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 1.59410554e-06 0.00000000e+00] *NEW BEST
update 31/100, cost= [ 1.59506473e-06 0.00000000e+00] lambda=0.00001, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 1.02278341e-06 0.00000000e+00] *NEW BEST
update 32/100, cost= [ 1.02372308e-06 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 6.58956706e-07 0.00000000e+00] *NEW BEST
update 33/100, cost= [ 6.59182416e-07 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 4.26100415e-07 0.00000000e+00] *NEW BEST
update 34/100, cost= [ 4.26238240e-07 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 2.76162077e-07 0.00000000e+00] *NEW BEST
update 35/100, cost= [ 2.76106773e-07 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 1.79193795e-07 0.00000000e+00] *NEW BEST
update 36/100, cost= [ 1.79367617e-07 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 1.16637466e-07 0.00000000e+00] *NEW BEST
update 37/100, cost= [ 1.16699151e-07 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 7.59488081e-08 0.00000000e+00] *NEW BEST
update 38/100, cost= [ 7.59721471e-08 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 4.94909063e-08 0.00000000e+00] *NEW BEST
update 39/100, cost= [ 4.95348778e-08 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 3.23212435e-08 0.00000000e+00] *NEW BEST
update 40/100, cost= [ 3.23186435e-08 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 2.10819624e-08 0.00000000e+00] *NEW BEST
update 41/100, cost= [ 2.11080324e-08 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 1.37741227e-08 0.00000000e+00] *NEW BEST
update 42/100, cost= [ 1.37812744e-08 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 9.00776538e-09 0.00000000e+00] *NEW BEST
update 43/100, cost= [ 9.01200628e-09 0.00000000e+00] lambda=0.00000, [CG iter 11, phi=-0.00000, cost=0.00000] backtracked 11/11 validation= [ 5.89278625e-09 0.00000000e+00] *NEW BEST
update 44/100, cost= [ 5.88988813e-09 0.00000000e+00] lambda=0.00000, [CG iter 214, phi=+nan, cost=nan]
[CG iter 250, phi=+nan, cost=nan] backtracked 248/250 validation= [ nan 1.]
update 45/100, cost= [ nan 1.] lambda=0.00000, [CG iter 250, phi=+nan, cost=nan] backtracked 248/250 validation= [ nan 1.]
update 46/100, cost= [ nan 1.] lambda=0.00000, [CG iter 250, phi=+nan, cost=nan] backtracked 248/250 validation= [ nan 1.]
update 47/100, cost= [ nan 1.] lambda=0.00000, [CG iter 250, phi=+nan, cost=nan] backtracked 248/250 validation= [ nan 1.]
update 48/100, cost= [ nan 1.] lambda=0.00000, [CG iter 250, phi=+nan, cost=nan] backtracked 248/250 validation= [ nan 1.]
update 49/100, cost= [ nan 1.] lambda=0.00000, [CG iter 250, phi=+nan, cost=nan] backtracked 248/250 validation= [ nan 1.]
update 50/100, cost= [ nan 1.] lambda=0.00000, [CG iter 250, phi=+nan, cost=nan] backtracked 248/250 validation= [ nan 1.]
update 51/100, cost= [ nan 1.] lambda=0.00000, [CG iter 250, phi=+nan, cost=nan] backtracked 248/250 validation= [ nan 1.]
update 52/100, cost= [ nan 1.] lambda=0.00000, [CG iter 246, phi=+nan, cost=nan]update 49/100, cost= [ nan 1.] lambda=0.00000, [CG iter 250, phi=+nan, cost=nan] backtracked 248/250 validation= [ nan 1.] ^C ^CInterrupted by user.
```
I guess this last problem can be avoided with early stopping, but it would be nice if it didn't go to NaN.

Thanks again for releasing the code.