flux-framework/flux-core — pull request #45 by grondo (2014-10-09 23:44:20+00:00)
Title: testsuite: use automake's TAP driver for TAP tests

This pull request is just an *RFC*. It is not meant to be merged yet, but I open the pull request to get comments on the changes.

This set of changes is an experiment in using newer automake versions
built-in TAP driver for tests. This has the benefit of correcting the `make check`
summary for tests, however the drawback is that tests are more verbose.
Maybe a good, maybe a bad thing.

What do people think?
On older automake versions, this will issue a warning

configure.ac:9: required file `config/tap-driver.sh' not found

However, this doesn't seem to cause any ill effects.

I was also kind of hoping a summary for all `make check` results across all SUBDIRS
would be available at the end, but I'm not sure how to do that yet.

example output from the `src/common/libutil` directory:
```
make[5]: Entering directory `/home/grondo/flux-core/src/common/libutil'
PASS: test_nodeset.t 1
PASS: test_nodeset.t 2 - consecutive adds become range
PASS: test_nodeset.t 3
PASS: test_nodeset.t 4 - singleton prepended to range
PASS: test_nodeset.t 5
PASS: test_nodeset.t 6 - singleton appended to range
PASS: test_nodeset.t 7
PASS: test_nodeset.t 8 - singleton embedded in range
PASS: test_nodeset.t 9
PASS: test_nodeset.t 10 - singleton embedded in range 2
PASS: test_nodeset.t 11
PASS: test_nodeset.t 12 - overlapping range
PASS: test_nodeset.t 13
PASS: test_nodeset.t 14 - overlapping range 2
PASS: test_nodeset.t 15
PASS: test_nodeset.t 16 - overlapping range 3
PASS: test_nodeset.t 17
PASS: test_nodeset.t 18 - add range that contains existing
PASS: test_nodeset.t 19
PASS: test_nodeset.t 20 - add range contained by existing
PASS: test_nodeset.t 21
PASS: test_nodeset.t 22
PASS: test_nodeset.t 23 - edge case 1 merges with 0
PASS: test_nodeset.t 24
PASS: test_nodeset.t 25
PASS: test_nodeset.t 26
PASS: test_nodeset.t 27 - reverse merge works
PASS: test_nodeset.t 28
PASS: test_nodeset.t 29
PASS: test_nodeset.t 30 - mundane range string works
PASS: test_nodeset.t 31
PASS: test_nodeset.t 32
PASS: test_nodeset.t 33 - numerically reversed range handled
PASS: test_nodeset.t 34
PASS: test_nodeset.t 35
PASS: test_nodeset.t 36
PASS: test_nodeset.t 37 - empty string produces empty range
PASS: test_nodeset.t 38 - comma by itself produces error
PASS: test_nodeset.t 39 - range missing start produces error
PASS: test_nodeset.t 40 - range missing end produces error
PASS: test_nodeset.t 41 - alpha with numerical suffix produces error
PASS: test_nodeset.t 42
PASS: test_nodeset.t 43 - bracketed range works
PASS: test_nodeset.t 44
PASS: test_nodeset.t 45 - alpha by itself produces error
PASS: test_nodeset.t 46
PASS: test_nodeset.t 47
PASS: test_nodeset.t 48 - nodeset_test_range works
PASS: test_nodeset.t 49 - nodeset_del_rank works
PASS: test_nodeset.t 50 - nodeset_test_rank works
PASS: test_nodeset.t 51
PASS: test_nodeset.t 52
PASS: test_nodeset.t 53
PASS: test_nodeset.t 54
PASS: test_nodeset.t 55
PASS: test_nodeset.t 56 - cached string is invalidated
PASS: test_nodeset.t 57
PASS: test_nodeset.t 58
PASS: test_nodeset.t 59
PASS: test_nodeset.t 60
PASS: test_nodeset.t 61
PASS: test_nodeset.t 62 - iterator_next works on first element
PASS: test_nodeset.t 63 - iterator_next works on next element
PASS: test_nodeset.t 64 - iterator_next works on last element
PASS: test_nodeset.t 65 - iterator_next returns EOF
PASS: test_nodeset.t 66 - iterator rewind works
PASS: test_nodeset.t 67
PASS: test_nodeset.t 68
PASS: test_nodeset.t 69 - nodeset_dup says it worked
PASS: test_nodeset.t 70 - nodeset_dup returned identical nodeset
PASS: test_nodeset.t 71 - orig unaffected by changes in dup
PASS: test_nodeset.t 72 - dup unaffected by changes in orig
PASS: test_nodeset.t 73
PASS: test_nodeset.t 74 - results not zero padded by default
PASS: test_nodeset.t 75 - padding 3 on all all works
PASS: test_nodeset.t 76 - padding 2 on subset works
PASS: test_nodeset.t 77 - padding 4 on all works
PASS: test_nodeset.t 78
PASS: test_nodeset.t 79 - explicitly resize to 1000000
PASS: test_nodeset.t 80 - added 1000000 consecutive ranks [0.16s 129 Mbytes]
PASS: test_nodeset.t 81 - string conversion 0-999999 [0.00s 129 Mbytes]
PASS: test_nodeset.t 82 - large nodeset count is sane
PASS: test_nodeset.t 83
PASS: test_nodeset.t 84 - explicitly resize to 1000000
PASS: test_nodeset.t 85 - added 500000 non-consecutive ranks [0.08s 129 Mbytes]
PASS: test_nodeset.t 86 - string conversion [0.00s 129 Mbytes]
PASS: test_nodeset.t 87 - large nodeset count is sane
PASS: test_nodeset.t 88 - veb size is the minimum 1024
PASS: test_nodeset.t 89 - adding max+1 4294967295 rank fails [0.00s 0 Mbytes]
PASS: test_nodeset.t 90 - veb size is the minimum 1024
SKIP: test_nodeset.t 91 # SKIP too slow
SKIP: test_nodeset.t 92 # SKIP too slow
SKIP: test_nodeset.t 93 # SKIP too slow
SKIP: test_nodeset.t 94 # SKIP too slow
SKIP: test_nodeset.t 95 # SKIP too slow
SKIP: test_nodeset.t 96 # SKIP too slow
SKIP: test_nodeset.t 97 # SKIP too slow
SKIP: test_nodeset.t 98 # SKIP too slow
SKIP: test_nodeset.t 99 # SKIP too slow
SKIP: test_nodeset.t 100 # SKIP too slow
SKIP: test_nodeset.t 101 # SKIP too slow
SKIP: test_nodeset.t 102 # SKIP too slow
SKIP: test_nodeset.t 103 # SKIP too slow
SKIP: test_nodeset.t 104 # SKIP too slow
SKIP: test_nodeset.t 105 # SKIP too slow
SKIP: test_nodeset.t 106 # SKIP too slow
PASS: test_nodeset.t 107 - nodeset_del max works
PASS: test_nodeset.t 108
PASS: test_nodeset.t 109
PASS: test_nodeset.t 110 - resize to zero returns success
PASS: test_nodeset.t 111 - nodeset size is the minimum 1024
make[6]: Entering directory `/home/grondo/flux-core/src/common/libutil'
make[6]: Nothing to be done for `all'.
make[6]: Leaving directory `/home/grondo/flux-core/src/common/libutil'
============================================================================
Testsuite summary for flux-core 0.1.0
============================================================================
# TOTAL: 111
# PASS: 95
# SKIP: 16
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
```