Started theanoizing latent-LR.
Merge branch 'master' of github.com:alexisVallet/dpm-identification
Still debugging the theanoized code.
Theanoization pretty much done. SSM still heavily diverging sometimes. More tweaking might be necessary.
Added beginning of implementation for the pegasos algorithm.
Now changed matching so it works more correctly. Also implemented pegasos, with some changes.
Started implementing a simple logistic regression to test out gradient descent methods.
Now have a "satisfying" SGD. It is clear now that it **will** require grid search + CV to find out the best learning rate, and possibly batch size for a fixed number of iterations (which should be chosen to keep the running time acceptable).
Some very slight tweaks to LLR. Should really move on to OVA multi class classification.
Started implementing logistic calibration of binary classifier output scores.
Linear-logistic calibration implemented and successfully tested.
One vs all DPM classifier now functional, although prediction deserves some more testing.
One vs all tested also against warping classifier. Checked that calibration is crucial to one vs all performance - could possibly be improved?
More testing of one vs all classification. Comparing with scikit learn's LR implementation, mine certainly has a bug for multi-class - either in prediction or in calibration.
Finally found the bug. It turns out that the batch size was key - I perform best with just raw gradient descent here (i.e. full batches). Seems SGD is only interesting when the size of the data is really, really large (which is not so much the case here). Also brings forward more opportunities for GPU acceleration once I get to that.
Cleaned up some stuff. Now debugging multinomial latent logistic regression.
Latent MLR successfully tested in the non-latent case. Will now endeavour to make multi-class DPM classification run on it, hopefully improving multi-class performance.
MLR DPM classification now running, although there is a bug where the cost
function goes crazy (negative values, infs and nans). When this bug doesn't
occur, the results look pretty good, so that's encouraging.
The bug was due to lack of feature scaling: the squared deformation displacements are of a scale much larger than the histogram bins (which are in the [0;1] range), so GD was messing up. Now scaling displacements to the [-1;1] range. This also makes deformation cost much less important to the cost function. There might be a tradeoff to find.
Ran some profiling on multi class DPM classification with multinomial logistic regression, convolution does indeed account for roughly 70% of the training time. Really need some GPU help here...