mina/Markus — pull request #3 by mmargel (2012-11-12 00:27:38+00:00)
Title: Preliminary copy of test runner

I'm doing this as a pull request instead of a direct commit because I'd like to have this reviewed now, rather than when we submit a pull request for the master repo.

This contains the test runner (testrunner.rb), a readme (explaining how to use the test runner), and a handful of sample files (3 testing scripts, a sample student-submitted file, and sample input & output files)

The basic idea behind the test runner is this:
For each (filename flag) pair in the input, fork a child that will run filename.
When executed, filename will print some data to stdout (number of marks the question is worth, test input, expected output, and actual output).
The testrunner will use the child's stdout as input.

Then testrunner will compare the output and expected output, and determine how many marks should be awarded, and sets the status to Pass/Fail/Error appropriately.
Then the results are parsed into XML and added to the output for the test runner.
If a test with the flag set to true fails, then the testing ends.
When testing ends (either because a critical test failed, or because all tests were run), the output is printed to stdout.

At the moment, the main issues (that are apparent to me) are that only ruby is supported, there is no way to stop infinite loops, and stderr from a student submission that crashes isn't returned to the test runner.
There's also a huge security hole, where the code that is executed isn't checked at all, and I'll need to look into what security options ruby has available. I'm really not sure how to address it.