Khan/khan-exercises — pull request #161347 by pjmattingly (2014-04-21 19:23:27+00:00)
Title: Added end-to-end testing of exercises with page_load

As promised, here:

https://github.com/Khan/khan-exercises/pull/161343

"page_load" https://github.com/pjmattingly/page_load.

In short:

Fork a process to sandbox Phantomjs

Completely load the page with PhantomJS
Tracking each resource to verify it either loaded completely or failed
(Some fonts in local mode fail to load for whatever reason, this doesn't appear to effect the testing or rendering of the page)

Once we've accounted for each resource we use a callback to do a test against the page

When the callback returns or if the callback raises and exception, PhantomJS returns a JSON payload to the caller detailing the results (a simple "true" or the exception object via JSON.stringify(), etc)

The caller kills the child (to prevent weird behavior from phantom.exit())

Repeat until all exercises have been inspected in this way.

With a simple test runner / phantomjs manager (node_page_load_handler.js) this provides end-to-end testing across all exercises. I have added code to travis.yml to run these tests and have verified they all pass.

See: https://travis-ci.org/pjmattingly/khan-exercises/builds/23450461

In the example callback included with the pull request (moduleDependencies-test.js), I show that the changes I made previously (see above pull request, e.g. remove moduleDependencies) to khan-exercises produce identical results to the original code (as compared to a fresh pull from upstream and all changes to khan-exercises since the submission of pull #161343 as above).

This provides a framework to run tests against the exercise pages themselves. In theory, any callback (compatible with the PhantomJS runtime) should be able to run against the exercise pages in this manner, to build tests to verify them in whatever manner you see fit.

page_load is in an unfinished state (as per the numerous TODO tags and the occasional HACK tag) and should be used with caution. The program needs to be modularized, black / white listing of files to inspect, as well as modified to use a reporter (the program was modified to work with Travis CI on the khan-exercises repo, which broke the reporting used for the tests), etc. But I wanted to provide this functionality as is, to be able to show the testing process I used in the above pull request instead of a simple "trust me".