privat/nit — pull request #807 by xymus (2014-10-03 18:44:25+00:00)
Title: Tester over MPI

On turing, running
`time mpirun -np 120 ../nitester --engine nitg-s,nitg-g,nitg-sg,nitg-e,niti ../src/nit*.nit ../examples/*.nit *.nit`
from the `tests` directory takes 2m 46s on 120 processors to evaluate 9003 programs/engine combinations.

# Results (for the execution specified above)

* 9003 total test programs
* 5728 oks & 0ks
* 1245 fails (a lot of libraries and SDKs are missing)
* 175 no savs
* 191 fixmes
* 1494 sosos
* 125 skips
* 45 unknowns (unsupported results: skip soso, changed, etc.)

# Features and todo

- [x] Uses OpenMPI
- [x] Supports all engines through tests.sh
- [x] Collects and classify results
- [x] Works on the Turing cluster
- [x] Optimized to avoid overloading NFS
- [ ] Correctly detect the all the results (no more unknowns)
- [ ] Use with Jenkins (see what we can do with the generated XML files)
- [ ] Gather and keep failed results temporary files
- [ ] Be more customizable to run on different clusters
- [ ] Integrate some features from tests.sh