tricycle/predictionbook — pull request #55 by foxbenjaminfox (2014-05-22 13:58:10+00:00)
Title: Scoring

I implemented Gwern's scoring rule, as described in issue #3. (Also requested in issue #44.)

I am not sure that this is the best scoring rule to use, but I wouldn't argue with Gwern over the matter. I find it slightly strange to have a score that revolves around 1. (1 is what a random guesser gets. A score from 0 to 1 is worse than a random guesser, and a score greater than 1 is better than a random guesser.) But after all, it is a *logarithmic* rule, so this behavior makes perfect sense, even though it's not perfectly intuitive.

I stuck pretty closely to what was described in issue #3 with respect to the display/layout. My only change was that I added some CSS to make the score green if the score is above the baseline, and to make it red if below baseline. This still isn't particularly fancy, and I may in the future go back and spiff it up. (Add a leaderboard, that kind of thing.)

Writing the scoring code was relatively easy, it was the unit tests that really sucked up my time. The previous implementation of statistics_spec.rb was quite brittle, and wouldn't work with the code I added. So I built a FactoryGirl factory (WagersFactory) to generate the test wagers in a way that was properly extensible, and that worked with my scoring code.

This also allowed me to vastly simplify the setup code for the Interval unit tests, in the same module. I replaced the long complicated setup code with a call to WagersFactory, which not only made the code quite a bit simpler, but also revealed a very subtle bug in the old testing code.

The previous code didn't take into account the fact that in a wager less than 50%, whether or not the wager is "correct" is the *reverse* of whether the prediction's result was marked as right. (See the implementation for the correct? method of the Response class in response.rb. This is what correct? usually returns. In the old tests, correct? was treated as if it was supposed to return prediction.right?, regardless of agree?. This lead to a test that expected the opposite of reality, with setup code that gave it that false reality.)

In my fixed setup code, I set correct? for my test responses properly, revealing the fact that the test was testing for the wrong thing this whole while. I changed the offending line (@i100.accuracy should actually be 0, not 100) and now the correct thing is being tested for, and passing the test.

This was a very subtle bug, but luckily it was a bug in the test code, not affecting the production code in any way. In development and production, the real correct? method was and is used, bypassing this bug entirely. The test wagers are not linked to predictions, however, and so need their correct? methods overwritten. However, in the old code, those correct? methods were being overwritten wrongly, giving us the bug.

Anyway, that's not important. That was an interesting bug, but it never affected the users, and my refactoring easily discovered it. The important part is the log scoring rule, allowing for comparison both across users (letting the users compete between themselves), and across time (letting one track one's improvement in calibration over the months.)