t/TEST: comment some of the <$results> loop vars

Add comments to var declarations, as 1st step to rewrite for clarity.
t/TEST: $failure doesnt need to use defined

$failure handling in <$results> processing loop does not need extra
subtlety of defined-ness checks - just give it an initial (false)
value, and check for truth.
t/TEST: add -perfstat option to run tests under linux perf stat

Add a simple way to run each test-script under linux perf stat, via a
-perfstat(=\d+)? cmdline option.  To make the option available for
make test and make test_harness, add TEST_OPTS envar, prepend it to
@ARGV, and move early return 1 if $::do_nothing (for subroutine reuse
via require) below cmdline option processing.

With this:

    export TEST_OPTS=-perfstat
    make test

writes a single file (t/perf.out.$$) containing perf reports for each
test (2345 currently), like this:

 Performance counter stats for './perl base/cond.t':

          3.155154 task-clock                #    0.807 CPUs utilized
                 2 context-switches          #    0.634 K/sec
                 1 CPU-migrations            #    0.317 K/sec
               464 page-faults               #    0.147 M/sec
         4,558,961 cycles                    #    1.445 GHz
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
         3,904,114 instructions              #    0.86  insns per cycle         [ 9.00%]
     <not counted> branches
     <not counted> branch-misses

       0.003909833 seconds time elapsed

The perf-reports have lots of info, and it takes very little
extra CPU to run:
				export TEST_OPTS=-perfstat
	time make test		time make test
	--------------		--------------
	real	25m41.122s	real	26m30.719s
	user	18m28.773s	user	18m43.286s
	sys	1m12.705s	sys	1m33.398s

When -perfstat=N is passed, "perf stat --repeat N -- perl test-script"
runs the script N times, and computes standard deviation of each
measurement.  This provides a noise-floor when evaluating the
performance impacts of a change, and demonstrates the limited
suitability of each test-script for use in performance assessment.

  export TEST_OPTS="-perfstat=reps=4:rptdir=~/perl/perf:out=newname:tool=perf stat"

This envar value uses all the flexibility of the command wrapper, to
set repeat count, report dir and file-name, and tool (and tool
options) to run under.  This aims to support running under valgrind,
cachegrind etc, but needs more work; especially for the postprocessing
done on valgrind outputs.

NOTES/TODO:

The single report (t/perf.out.$$) differs from the 1 report per
testfile style done by test.valgrind (which can also do perf stat
testing).  This is convenient for casual perusal, and much easier to
manage as data (simple cp vs find t -name \*.perf-stat ....).

- save reports to $ENV{TEST_RPT_DIR}, not just $PWD
- make this wrapperizer work for test.valgrind cases
- replace some/most of existing test.valgrind support

TBD:

The <$results> processing loop needs changes to work with -perfstat=N,
this is a bug in the <$results> loop tweaks, TBD..
t/TEST: make test loop work for perf stat --repeat 3

Make <$results> processing loop accept repeated output from
test-script if run as "perf stat --repeat X -- test-script".

This boils down to tweaking leader (1..N) handling, by disarming
if ($seen_ok) ... check, allowing "ok N" in X passes after 1st.

TBD - further tweaks needed, get 2 dozen failures now on make test
t/TEST: simplify <$results> processing

This is a wholesale =head/=cut and reimplement.  It works, insofar as
tests run with reps=2.  Cleanup needed; drop =head/=cut block,
consolidate $failure vs @fails, move error checks up above =cut block,
move more err checks above valgrind handling (no reason for
separation).  Defer for now, run some tests overnite.
t/TEST: finish <$results> processing simplification

new loop now works, with:
 TEST_OPTS=-perfstat=reps=4:rptdir=/home/jimc/projects/perl-core/perf
 HARNESS_TIMER=/home/jimc/projects/perl-core/timings

t/re/pat_advanced ............................................. ok     2578 ms  1690 ms   180 ms	# ET: u:440 s:40 cu:0 cs:0 pid:7509  # ET: u:400 s:20 cu:0 cs:0 pid:7515  # ET: u:400 s:20 cu:0 cs:0 pid:7521  # ET: u:380 s:40 cu:0 cs:0 pid:7532

Change details include (tb merged into previous):
- drop $failure, use @fails (accumulate fails)
- move all err checks together (above Valgrind handling)
- some corrections to err tests
- comment out now unused vars

Tested code includes next patch, which fixes uninit warnings enabled
by -w added previously, but untested for -d HARNESS_TIMER case.
t/TEST: tolerate bare ok, or extra spaces between ok and N

- tolerate bare ok
- allow extra spaces after ok, not just single space
drop test_ET from this branch
seen-test check
fixup: rename output file perfrpt.$tstamp, trim perf stat cmdline
drop =cutted code
comment tweaks
fixup: TEST_OPTS split on ;

splitting on /\s+/ broke tool=... overrides that had spaces, so
overriding with the default value would break:

  export TEST_OPTS="-perfstat=reps=4:tool=perf stat:name=foo"

Change split to /;/, which is kinda weird, but almost never used
(never so far).  Note that multiple k=v pairs are separated by ':', as
in example above.

Also change filename key from out=<fname> to name=<fnameroot>, and
append $tstamp to it automatically.  This makes name=<..> more useful
for conveying branch or config-names w/o overwriting previous reports.
fix err msg, --repeat is clearer
drop unused vars
fix VMS RESULTS