Output sizing parameters from bench script

This makes it easy to compare results using different sizing parameters.
Add option to cuda_correlator, bench not to dump

Both cuda_correlator and the bench script accept a "-n" option to
inhibit dumping.  This allows benchmarking the X engine's input and
correlation without any influence from dumping.  This may be useful for
applications that will integrate on the GPU for more than one call to
xgpuCudaXengine().
Add "-c COUNT" option to cuda_correlator

This allows looping over the call to xgpuCudaXengine COUNT times.  COUNT
defaults to 1.  This can be used to reduce noise on timing measurements.
Print xgpuCudaXengine timing from cuda_correlator

The cuda_correlator program now prints the total time spent in
xgpuCudaXengine() and the average time per call.  Use -c option to
cuda_correlator to call xgpuCudaXengine() multiple times.
Add RUNTIME_STATS compile time option

Compiling with RUNTIME_STATS defined will cause cuda_correlator to print
a line of runtime statistics to stderr for every call to
xgpuCudaXengine().  The line currently consists on relative time from
just before the first call to xgpuCudaXengine() and time spent in
xgpuCudaXengine().  All times are in milliseconds.  For example:

$ ./cuda_correlator -c 2
Correlating 64 stations with 64 channels and integration length 512
Sending fixed point data to GPU.
Calling CPU X-Engine
Calling GPU X-Engine
   0.000119     3.874946 ms      <--- These two lines
   3.900374     3.641811 ms      <--- go to stderr
   7.549456 ms total,    3.774728 ms/call average
Checking result (tolerance == 1e-05)...
Outer product summation failed with 133 deviations (max error 0.00159975)
Change cuda_correlator -n option

This changes cuda_correlator's -n option to dump on the last call to
xgpuCudaXengine() rather than not dump at all.  This actually exposes
a problem in xgpuCudaXengine() when called back-to-back with doDump=0.
If the calls are too close, it is possible for the second call to start
transferring data into the buffer still in use by asynchronous kernel
calls launched by the first call.
Fix back-to-back xgpuCudaXengine(doDump=0) calls

When calling xgpuCudaXengine() back-to-back too quickly with doDump=0,
it was possible that the second call would start a host-to-device
transfer before asynchronous kernel calls launched in the first call was
done with the data being overwritten.  To see this, run:

cuda_correlator -c 1
cuda_correlator -c 2 -n

The first invocation shows the expected (small) number of discrepancies
(also small) between CPU and GPU.  The second command should show the
same number of small discrepancies.  If that test is done using xGPU
versions prior to this one, then the results for the second command will
show an excessive (and inconsistent!) number of errors.

This bug was due to a lack of stream synchronization between calls to
xgpuCudaXengine() when doDump=0.  When doDump=1, the synchronous
cudaMemcpy call performs an explicit synchronization of the streams, so
overwriting a still-active buffer will not be possible.

The fix for this involves moving the "completion events" into the
internal context, waiting for the "kernel completion #0" event
before the initial host-to-device copy, and recording the "kernel
completion #1" event after the final kernel launch.
Create completion events with flags (no timing)

Create completion events with cudaEventCreateWithFlags and pass the
cudaEventDisableTiming option to disable event timing.  NVIDIA claims
that disabling event timing improves performance. I'm not sure how
significant it is for us, but every little bit helps.
Fix Makefile anamoly with local includes

The Makefile specified "-I." so that some local include files could be
found.  This causes problems if local files are created with the same
name as a system header file (e.g. "new") because it PREpends "." to the
search path for system header files.  The solution is to fix the
includes of local files by using '#include "local.h"' rather than
'#include <local.h>'.  Using quotes instead of angle brackets
automatically searches first in the same directory as the source file,
so "-I." can be removed from Makefile.
Make array_len,matrix_len use consistent units

When using host allocated buffers, array_len and matrix_len are in units
of sizeof(ComplexInput) and sizeof(Complex), respectively.  When using
CUDA allocated buffers, these fields were populated in units of bytes,
but now they are populated in the same units as when using host
allocated buffers.
Add -r option to cuda_correlator

The -r option causes cuda_correlator to use host allocated buffers that
is registered (-r for registered) with CUDA rather than the default of
using CUDA allocated buffers.
Enhancements for streaming operation

In addtion to dumping or doing nothing, the caller of xgpuCudaXengine
can now specify two other actions to perform after sending all the
asynchronous tasks to CUDA.  The doDump parameter has ben renamed and
repurposed to "syncOp" to specify this operation.  The four possible
values are:

SYNCOP_NONE - No further action is taken.

SYNCOP_DUMP - Waits for all transfers and computations to
              complete, then dumps to output buffer at
              "context->matrix_h + context->output_offset".

SYNCOP_SYNC_TRANSFER - Waits for all transfers to complete, but not
                       necessrily all computations.  This ensures that
                       all data from the input buffer has been
                       transferred to the device before xgpuCudaXengine
                       returns.

SYNCOP_SYNC_COMPUTE - Waits for all computations (and transfers) to
                      complete, but does not dump.

These new options, especially SYNCOP_SYNC_TRANSFER, make the code better
suited to streaming applications that don't dump on every call.
Change output format of cuda_correlator

Cleans up formatting of the line showing the total and average elapsed
time.  If compiled with RUNTIME_STATS defined, prints an extra " final"
at the end of the line for the final call to xgpuCudaXengine since that
call always uses SYNCOP_DUMP and the earlier calls (if any) use the
user-specified syncOp (if any).  This facilitates special treatment of
that line by any post analysis software.
Change bench to use -o option of cuda_correlator
Show max bandwidth at end of cuda_correlator run

In addition to the total elapsed time and per call elapsed time,
cuda_correlator now displays the theoretical maximum bandwidth that
could be sustained.  The theoretical maximum bandwidth is calculated as:

  time_per_channel = time_per_call / (NTIME * NFREQUENCY)

  max_bw = 1 / time_per_channel
         = (NTIME * NFREQUENCY) / time_per_call

The reported bandwidth is for a single X engine only.  How the X engine
bandwidth relates to the overall bandwidth depends on what fraction of
the overall band is processed by a single X engine.
Change cuda_correlator -n option to -f

The -n option (shorthand for "-o 0") has been replaced by a
"-f FINAL_SYNC_OP" option.  This allows the user to specify which sync
operation to perform on the final call.  Used mainly for debugging and
timing analysis.  The final sync operation still defaults to
SYNCOP_DUMP.
Make cuda_correlator's help message more useful