Khan/khan-exercises — pull request #22496 by divad12 (2012-05-24 02:26:41+00:00)
Title: first pass at solving problem of duplicate problems

I went through a few iterations before settling on this solution:

Attempted solutions
===================

1. Have an RNG that does not repeat itself.
- cons: requires "flag day" where viewing old problems are invalidated because
the RNG is different. Also, if we wish to have only 200 possible problems as
we do now, we could only ensure that the first call to the RNG in a problem
is a non-duplicate, which would require exercise developers to specify the
first non-trivial call to the RNG. This is a leaky abstraction. :(

2. Pre-process (eg. during deploy-time) the exercises to find seeds that
generate duplicate problems, and discard (all but 1 of) those seeds.

I spent a day attempting to hack around jsdom's deficiencies (eg. no XHRs)
and bugs, but doing all this required lots of changes to khan-exercise.js,
may be brittle, and ultimately I decided to try out phantomjs before getting
this to work.

I then tried out phantomjs, which is headless webkit using QtWebkit. This
took only 10 mins to get working with our exercises, but QtWebkit is very
slow and after a day of optimizations removing unnecessary steps, it still
took over half an hour to pre-process less than half the exercises. Due to
other cons below, I dropped this solution as well.

- pros: does not invalidate viewing of old problems done. Allows
- cons: pre-processing slows down deploy time by hours. This is because we
need to run makeProblem 200 times on every problem type in each exercise, and
QtWebkit is slow. We could cache these results, but every time there is
a change to khan-exercise.js or any javascript file, would need to re-run the
pre-processing to be safe. I also didn't think it'd be wise to spend another
day doing optimizations.

3. At run-time, check if the current problem is similar to any of the last
5 problems done. If so, skip it.

This is the final solution that was implemented in this commit. It uses
localStorage for state persistence. It does not screw up problem numbers
(they still only increase 1 by 1) and refreshing the page would not change
the problem.

- pros: is not brittle. Does not require very long pre-process time during
deploy.
- cons: a tiny bit slower due to having to re-run half of makeProblem if we
need to skip.

I need to go for dinner now. If I think of other things to add, I'll add it to
the pull request message.

@beneater @kamens @spicyj (almost a bengineering)