kytrinyx/exercism — pull request #79 by ecmendenhall (2013-08-25 21:38:00+00:00)
Title: Run tests suites before submission

Here's a first hack at fixing issue #54.

### What's new:
- A `TestSuite` class that runs a submission's tests and captures stdout, stderr, and the exit code. This uses Open3 from the standard library. Initialize a `TestSuite` with an assignment's filename, then call `TestSuite.run` or `TestSuite.passes?` to run tests. `TestSuite` runs the test commands specified in the Exercism.io setup instructions for each language, and should print a friendly help message when the environment is misconfigured. `TestSuite` expects passing suites to exit 0 and considers everything else a failure.
- A new "test" command from the CLI. Run `exercism test ` to run a submission's test suite.

### What's changed:
- I added passing and failing test fixtures for each current curriculum language. (Let me know if you think of a better way to test this). These require your environment to be set up according to the Exercism.io setup instructions for each language. Since they'll fail the Travis build, I extracted them to a separate Rake task. Run `rake test` to run default tests (including most of the `TestSuite` tests) and `rake test env_dependent` to add in the non-Ruby fixtures.
- The CLI runs tests as part of `exercism submit` and bounces failing submissions.

### What needs to change:
- The Haskell and Clojure tests from Exercism.io don't always exit correctly. I've added logic to the .hs and .clj test fixtures that handle exiting correctly on failure. Clojure tests are a simple fix—just wrap `(run-tests)` in a conditional:

```
(when (not (successful? (run-tests)))
(System/exit 1))
```

Haskell tests are more complicated. I came up with something, but I'm a Haskell noob:

```
import Test.HUnit (Assertion, (@=?), runTestTT, Test(..), Counts (failures, errors))
import Control.Monad (void)
import System.Exit (ExitCode (ExitSuccess, ExitFailure), exitWith)
import FailingSubmission ( thisIsFalse )

[ ... ]

testsPass :: Counts -> Bool
testsPass testcounts = and [failures testcounts == 0,
errors testcounts == 0]

exitStatus :: Counts -> ExitCode
exitStatus testcounts
| testsPass testcounts = ExitSuccess
| otherwise = ExitFailure 1

main :: IO ()
main = do
result <- runTestTT (TestList failingSubmissionTests)
exitWith (exitStatus result)
```

If the Haskellers around here have a better solution, please feel free to add it. Point is, existing Haskell and Clojure assignments need to include these checks before updating the gem.
- Likewise, test suites for future languages and examples (Go, Rust, etc.) need to exit correctly.
- Since `TestSuite` runs the exact commands from the Exercism.io setup pages, it should be clear that these are hard requirements. For Clojure, this means Leiningen and the lein-exec plugin are required.