ansible/ansible — pull request #5938 by sivel (2014-02-10 19:08:37+00:00)
Title: Add functionality to unit test cloud modules

@jctanner approached me about thinking of ways to unit test our cloud modules a day or so ago, and since then he and I have been chatting about ideas to make unit testing modules like rax and ec2 possible. Currently they are not easily unit testable, and require integration testing.

Not everyone is going to have access to cloud environments that they will want to use for testing, and this could still help us catch issues with updates and such. James pointed out that #5285 as a prime example of needing this functionality.

Thinking over the process, I couldn't really see a way in, other than taking the approach of how test-module works. For those that don't know, test-module basically uses ModuleReplacer to build out the module as it would appear during ansible execution, writes it to a file, and then subprocess.Popen()s it, and evaluates stdout.

I took this a little further, and after ModuleReplacer does it's thing, I strip out the `main()` call, and replace it with Mock, monkey patching, and test cases, from a file that would live in `test/inject_module_unit_test`. The files in there would be named something like `MODULE_NAME_tests.py`, such as `test/inject_module_unit_test/rax_facts_tests.py`.

Then we run subprocess.Popen on the file, optionally running it via coverage.

I've only created tests for rax_facts right now, as it was simple and required minimal test cases an Mock/Monkey patching for testing.

A sample run looks like:

```
$ ./unit-test-module -c -v -m ../library/cloud/rax_facts
* including generated source, if any, saving to: /Users/matt/.ansible_unit_test_rax_facts
* this may offset any line numbers in tracebacks/debuggers!

Name Stmts Miss Cover Missing
------------------------------------------------------------------------
/Users/matt/.ansible_unit_test_rax_facts 886 546 38% 86-88, 115-119, 128-129, 138, 237-245, 250, 258-259, 263-264, 268, 290, 294-308, 316-332, 355-357, 392-417, 431-436, 439-449, 453-456, 467-469, 473-485, 488-503, 506-510, 513-516, 520-545, 548-567, 570-589, 592-630, 634-646, 649-661, 672-702, 714, 718, 723, 730-733, 738, 743, 745, 756, 764, 768, 772, 777-778, 786, 788, 798-803, 808-835, 852-895, 898, 920-921, 942, 953, 956-965, 978-996, 1000-1009, 1015, 1021, 1035, 1041-1053, 1057, 1061-1063, 1068-1075, 1078-1082, 1088-1130, 1146-1195, 1198-1211, 1237-1291, 1345-1350, 1355-1358, 1363-1371

test_correct_id (__main__.RaxFactsTest) ... ok
test_correct_ip (__main__.RaxFactsTest) ... ok
test_correct_name (__main__.RaxFactsTest) ... skipped 'The FakeClient does not currently support filtered queries'
test_incorrect_id (__main__.RaxFactsTest) ... ok
test_incorrect_ip (__main__.RaxFactsTest) ... ok
test_incorrect_name (__main__.RaxFactsTest) ... skipped 'The FakeClient does not currently support filtered queries'
test_multi_name_match (__main__.RaxFactsTest) ... skipped 'The FakeClient does not currently support filtered queries'
test_mutually_exclusive (__main__.RaxFactsTest) ... ok

----------------------------------------------------------------------
Ran 8 tests in 0.007s

OK (skipped=3)
```

For info, the actual module code extends to line 169, so a majority of that coverage is not pertinent, but can still be useful.

cc @willthames @claco @angstwad @j2sol @erjohnso

For everyone CC'ed here, just trying to raise visibility. I think @jctanner and I are ready to see this go in.