code-dot-org/blockly — pull request #380 by Bjvanminnen (2014-01-30 00:41:15+00:00)
Title: Better support for creating tests for levels.

The existing unit tests have the following drawbacks:
(1) Currently only exist for maze levels
(2) Only check one solution per level
(3) Only check that result is true. In the case of the bug I was investigating, the result is true, but we still give the user an error because we claim that they aren't using all the required blocks.

This replaces our xml files with json files that allow you to specify multiple tests per level, as well as validate any field on the report that we wish (for example testResult). We can also include failure tests where we assert that a certain block of xml fails a level.

All of the xml in the tests here was taken directly from the existing tests, with a couple exceptions:
2_2: The solution had three go forward blocks instead of two, which resulted in test success, but a testResult of TOO_MANY_BLOCKS_FAIL. I got rid of one of the blocks.
2_13: The solution worked, but we again got TOO_MANY_BLOCKS_FAIL. This was because we were counting 3 blocks that start in the workspace. I gave this blocks deletable=false.
karel_2_10: These are new. The first test is currently asserting that we get MISSING_BLOCK_FINISHED. The bug that inspired me to do all this work is that this should instead be giving ALL_PASS. When I submit the fix, I will also submit a change to this test, but I didn't want to check in a failing test.