Change index; include atime; pack time as xstat timespec; use ns in memory.

Update the index format header to 'BUPI\0\0\0\4' (version 4).

Change the index to include atimes, and write all times to bupindex as
xstat timespecs.  This is in preparation for indexing all metadata.
After moving all of the times to the index, there should be far fewer
unique instances of the remaining metadata in many/most cases.

Change the index and index-related code to handle all time
values as integer nanoseconds since the epoch, excepting the packed
format, which (as mentioned above) is now an xstat timespec.

After these changes the index-related in-memory and on-disk time value
handling should match that of the metadata code (i.e. metadata.py,
.bupm, etc.).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: change 10e8 to 10**9 for ns per second.]
Document accommodations for filesystems with low-resolution timestamps.

...and some effects thereof.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Store metadata in the index, in bupindex.meta; only store unique values.

See DESIGN for more information.

Update the index format header to 'BUPI\0\0\0\5' (version 5).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Add the filename/fd and invalid value to stat_struc_to_py error messages.

Thanks to Nathan Bird <ecthellion@gmail.com> for the initial report.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Progress check in before rebase.
Merge remote-tracking branch 'upstream/master' into relativistic-indexing

Conflicts:
	Documentation/bup-index.md
	cmd/index-cmd.py
	cmd/save-cmd.py
	lib/bup/_helpers.c
	lib/bup/index.py
	lib/bup/t/tindex.py
	t/test-meta.sh
Normalize negative nanosecond values in stat_struct_to_py()

 * Adds normalize_stat_time() to handle normalization

Turn sane negative nanosecond values (where negative nanoseconds simply
subtract from the seconds value to give the correct time) into the
equivalent positive nanoseconds time values.

This allows systems which use them (such as Cygwin) to report correct
timestamps to bup. Systems which return garbage or values > 1e+9 will
still (correctly) be identified and fail testing.

Signed-off-by: Will Rouesnel <w.rouesnel@gmail.com>
Set BUP_MAIN_EXE and BUP_DIR in testpacks()

testpacks did not set the BUP_MAIN_EXE and BUP_DIR environment variables
before invoking git.PackWriter(), which caused the wvtest to be run
looking for the wrong files.

This is currently Cygwin-specific problem, but the all other tests in this
series set these environment variables before running any other logic,
but for a currently unknown reason the environment does not appear to be
preserved properly under cygwin. With this change, this test executes
successfully while still being a correct test.

Signed-off-by: Will Rouesnel <w.rouesnel@gmail.com>
Procedurally generate t/sampledata

  * Remove existing sampledata from t/
  * Initial check-in of generate-test-data.sh script.
  * Add make testdata dependency to the make runtests target.
  * Remove testdata when running make clean.
  * Add t/sampledata and t/testfile1, t/testfile2 to .gitignore.

  * Fixes symlink tests in tindex.py not working on Cygwin.

Storing the sample data for tests in the Git repository causes problems
with cross-platform use due to the differing handling of symlinks by
Git, and the very different behavior that can arise when using symlinks
to files and symlinks to directories. This is particularly a problem on
Cygwin.

This commit resolves this problem by replacing the fixed sample data
with a shell script (generate-test-data.sh) which dynamically creates
suitable test data before tests are run.

Signed-off-by: Will Rouesnel <w.rouesnel@gmail.com>
Add -e,--sshcmd support to remote commands.

Added support for the rsync-style -e command line option to explicitly
specify the command line bup should use for the remote shell. This
allows the use of advanced SSH options by specifying them on the command
line ala:

	bup init -r user@remotehost -e 'ssh -oCompression=yes'

or the use of alternate shells which accept addressing and command
invocation in the SSH style.

Signed-off-by: Will Rouesnel <w.rouesnel@gmail.com>
Revised documentation to show the new -e,--sshcmd arguments.

Updates the documentation for init, join, save and split to be inline
with the -e,--sshcmd argument and their usage in relation to
-r,--remote.

Updates the documentation for -r,--remote for init, join, save, split
and on to indicate that user@host notation can be used (no code changes
required it was already possible).

Signed-off-by: Will Rouesnel <w.rouesnel@gmail.com>
Merge branch 'improved-ssh-operation' into relativistic-indexing

Updates relativistic-indexing with the rebased commits for
improved-ssh-operation (now atop upstream/master).