Overhaul restore destination handling, and stripping/grafting behavior.

Change restore to respond to source paths like this (assume outdir
corresponds to "." if there no -C argument or to -C outdir):

  /foo/what/ever - extract ever to outdir/ever
  /foo/what/ever/ - extract ever/* to outdir/*
  /foo/what/ever/. - extract ever/. to outdir/. (i.e. outdir == ever).

Also fix handling of top-level commit symlinks.  Previously bup would
just restore /foo/latest as a dummy symlink like this:

   latest -> ../.commit/SHA

Instead, dereference latest and restore the target instead.

Tighten up stripping/grafting with additional argument checks, and
handle any root collisions by creating a fake root dir (see comments
in save-cmd.py).  Bup still doesn't handle other path collisions yet,
i.e. if both /foo/bar and /bar are remapped to /bar.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Cache all password (pwd) and group (grp) database lookups.

Thanks to Jann Horn <jannhorn@googlemail.com> for determining that
even just caching user names and group names could provide a notable
performance improvement in some cases.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Handle nonexistent groups in Metadata._apply_common_rec().

Thanks to Oei, Yung-Chin <yungchin@yungchin.nl> for tracking this
down.

Reported-by: Yung-Chin <yungchin@yungchin.nl>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Change index; include atime; pack time as xstat timespec; use ns in memory.

Update the index format header to 'BUPI\0\0\0\4' (version 4).

Change the index to include atimes, and write all times to bupindex as
xstat timespecs.  This is in preparation for indexing all metadata.
After moving all of the times to the index, there should be far fewer
unique instances of the remaining metadata in many/most cases.

Change the index and index-related code to handle all time
values as integer nanoseconds since the epoch, excepting the packed
format, which (as mentioned above) is now an xstat timespec.

After these changes the index-related in-memory and on-disk time value
handling should match that of the metadata code (i.e. metadata.py,
.bupm, etc.).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: change 10e8 to 10**9 for ns per second.]
Document accommodations for filesystems with low-resolution timestamps.

...and some effects thereof.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Store metadata in the index, in bupindex.meta; only store unique values.

See DESIGN for more information.

Update the index format header to 'BUPI\0\0\0\5' (version 5).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Add the filename/fd and invalid value to stat_struc_to_py error messages.

Thanks to Nathan Bird <ecthellion@gmail.com> for the initial report.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Add "bup index --clear" to clear the index.

Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
bloom-cmd: don't explicitly create the bloom file (allow bloom.create()).

Don't explicitly create the bloom file since bloom.create() will
handle it, and will actually use the correct (binary) mode instead of
"w+".

The binary mode has no effect under Linux or Cygwin, but is important
under Windows.

Signed-off-by: Patrick Rouleau <prouleau72@gmail.com>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
clear_index(): remove unused "cleared" variable.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add 'x' to the getopts call in t/compare-trees (i.e. allow -x).

Signed-off-by: Anton Eliasson <devel@antoneliasson.se>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Attempt to fix HACKING link in README.md.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Update HACKING to reflect the glorious death of tmp/pending/meta.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Officially drop support for Python 2.4.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Alexander Barton <alex@barton.de>
Mention what's "in-scope" for 0.25 in HACKING.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
wvtestrun: invoke perl via /usr/bin/env for portability.

This also matches format-subst.pl.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
bup-save.md: fix --graft documentation bug; use "chroot" consistently.

Change mixture of "chroot" and "chroots" in examples to "chroot".

Thanks to "jon.seymour" <jon.seymour@gmail.com> for the report.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Update bup-restore.md to match reality; document "/." handling.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Move parse_excludes to helpers and rework it a bit.

Don't use f before it's initialized.

Narrow the catch to open(), and only catch IOException.

Call Options.fatal() when there's an error (as does
helpers.parse_date_or_fatal()).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Fix use of Python assert() with respect to optimization.

Fix a number of places where bup's assertions had material
side-effects, or where other code expected to see the AssertionError,
neither of which happen when optimization is enabled.

Reported-by: Jon Dowland <jmtd@debian.org>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>