qiime/qiime — pull request #1320 by jrrideout (2014-01-08 03:48:38+00:00)
Title: Refactoring of script usage testing framework to better handle errors

Refactored script usage testing framework to handle errors in addition to script failures. The test harness collates and reports the following types of errors:

- missing scripts (the user provided an non-existing script name, or there is test data that doesn't match a script)
- import error (trying to import the script results in an ```ImportError```)
- usage example error (trying to grab the usage examples from the script fails)
- some other error (anything that doesn't fall into the above categories, such as a Python syntax error in the script file)

The test harness reports each of the error types separately, and does not stop execution if they occur. It also reports the original error message (if there is one), usually taken from the original exception that was thrown.

This refactoring also allows the same test harness to be used on multiple input directories, allowing the user to generate a single report at the end of all test runs. This may be useful as we transition to having both old-style and pyqi-based scripts (these will be in separate directories).

Fixes #1304.

For an example of the output, I broke a few scripts to illustrate the different error types that are handled:

```
$ python tests/all_tests.py --suppress_unit_tests --script_usage_tests add_alpha_to_mapping_file,filter_fasta,foobarbaz,make_distance_boxplots,blahblahblah,load_remote_mapping_file
Scripts to test:
add_alpha_to_mapping_file filter_fasta foobarbaz make_distance_boxplots blahblahblah load_remote_mapping_file

Could not load usage examples in script add_alpha_to_mapping_file.py. Original error message:

Traceback (most recent call last):
File "/home/jrideout/.virtualenvs/qiime/qiime/qiime/test.py", line 902, in _load_usage_examples
usage_examples = script.script_info['script_usage']
KeyError: 'script_usage'

Could not load the script filter_fasta.py. Original error message:

Traceback (most recent call last):
File "/home/jrideout/.virtualenvs/qiime/qiime/qiime/test.py", line 887, in _load_usage_examples
script = __import__(script_name)
File "/home/jrideout/.virtualenvs/qiime/qiime/scripts/filter_fasta.py", line 24
dlksllksd sld.sdf
^
SyntaxError: invalid syntax

Script foobarbaz.py does not exist.

Testing 4 usage examples from: /home/jrideout/.virtualenvs/qiime/qiime/scripts/make_distance_boxplots.py
Running tests in: /tmp/script_usage_tests/make_distance_boxplots
Tests:
/home/jrideout/.virtualenvs/qiime/qiime/scripts/make_distance_boxplots.py -d unweighted_unifrac_dm.txt -m Fasting_Map.txt -f "Treatment" -o out1 : Pass
/home/jrideout/.virtualenvs/qiime/qiime/scripts/make_distance_boxplots.py -d unweighted_unifrac_dm.txt -m Fasting_Map.txt -f "Treatment" -o out2 -g png --suppress_all_within --suppress_all_between : Pass
/home/jrideout/.virtualenvs/qiime/qiime/scripts/make_distance_boxplots.py -d unweighted_unifrac_dm.txt -m Fasting_Map.txt -f "Treatment" -o out3 -g svg --save_raw_data : Failed
Stdout:

Stderr:
Traceback (most recent call last):
File "/home/jrideout/.virtualenvs/qiime/qiime/scripts/make_distance_boxplots.py", line 256, in
main()
File "/home/jrideout/.virtualenvs/qiime/qiime/scripts/make_distance_boxplots.py", line 252, in main
meh
NameError: global name 'meh' is not defined

/home/jrideout/.virtualenvs/qiime/qiime/scripts/make_distance_boxplots.py -d unweighted_unifrac_dm.txt -m Fasting_Map.txt -f "Treatment" -o out4 --suppress_significance_tests : Pass

Script blahblahblah.py does not exist.

Could not import the script load_remote_mapping_file.py. Original error message:

Traceback (most recent call last):
File "/home/jrideout/.virtualenvs/qiime/qiime/qiime/test.py", line 887, in _load_usage_examples
script = __import__(script_name)
File "/home/jrideout/.virtualenvs/qiime/qiime/scripts/load_remote_mapping_file.py", line 15, in
import bleh
ImportError: No module named bleh

==============
Result summary
==============

Script usage test result summary
------------------------------------

Ran 4 commands to test 6 scripts. 1 of these commands failed.
Failed scripts were: make_distance_boxplots.py
1 scripts could not have their usage examples loaded. Scripts were: add_alpha_to_mapping_file.py
1 scripts could not be imported. Scripts were: load_remote_mapping_file.py
1 scripts could not be loaded. Scripts were: filter_fasta.py
2 scripts could not be loaded because they do not exist. Scripts were: foobarbaz.py blahblahblah.py
Failures and errors are summarized in /home/jrideout/.virtualenvs/qiime/qiime/log.txt
```

The log file that's created (contains only failures/errors):

```
Could not load usage examples in script add_alpha_to_mapping_file.py. Original error message:

Traceback (most recent call last):
File "/home/jrideout/.virtualenvs/qiime/qiime/qiime/test.py", line 902, in _load_usage_examples
usage_examples = script.script_info['script_usage']
KeyError: 'script_usage'

Could not load the script filter_fasta.py. Original error message:

Traceback (most recent call last):
File "/home/jrideout/.virtualenvs/qiime/qiime/qiime/test.py", line 887, in _load_usage_examples
script = __import__(script_name)
File "/home/jrideout/.virtualenvs/qiime/qiime/scripts/filter_fasta.py", line 24
dlksllksd sld.sdf
^
SyntaxError: invalid syntax

Script foobarbaz.py does not exist.

/home/jrideout/.virtualenvs/qiime/qiime/scripts/make_distance_boxplots.py -d unweighted_unifrac_dm.txt -m Fasting_Map.txt -f "Treatment" -o out3 -g svg --save_raw_data: Failed
Stdout:

Stderr:
Traceback (most recent call last):
File "/home/jrideout/.virtualenvs/qiime/qiime/scripts/make_distance_boxplots.py", line 256, in
main()
File "/home/jrideout/.virtualenvs/qiime/qiime/scripts/make_distance_boxplots.py", line 252, in main
meh
NameError: global name 'meh' is not defined

Script blahblahblah.py does not exist.

Could not import the script load_remote_mapping_file.py. Original error message:

Traceback (most recent call last):
File "/home/jrideout/.virtualenvs/qiime/qiime/qiime/test.py", line 887, in _load_usage_examples
script = __import__(script_name)
File "/home/jrideout/.virtualenvs/qiime/qiime/scripts/load_remote_mapping_file.py", line 15, in
import bleh
ImportError: No module named bleh
```