yamins81/devthor — pull request #18 by hahong (2012-09-08 12:51:36+00:00)
Title: "Standard" consistency

This includes — well, long-awaited — enhancements for computing the "standard" consistency metric used in the HvM paper. In particular:
* "Standard" task specs are defined in ``HvMWithDiscfade``. Examples: ``task_basic_v0, task_basic_v3, ..., task_subord_face_v3``
* ``get_standard_consistency()``, the all-in-one "standard" consistency calculator, is added to ``v4toit``. This function collects all necessary CMs with the task specs in ``HvMWithDiscfade``, computes d's, and calls the low-level consistency function. The implementation is not that elegant, but it seems working okay for now. Also, I've put some TODO lists and considerations as "TODO/Thoughts" in the docstring. It would be great if you can give me some feedbacks.
* I've also put unit-tests for theses things. New functions passed those basic tests.

One thing I've noticed is that the consistency of IT is somewhat low (~0.84 for leave-2-out with 50 splits) compared to what I usually get (~0.9). I'll look into that. Meanwhile, V4 is okay and significantly lower (~0.42) than IT.