Overhaul restore destination handling, and stripping/grafting behavior.

Change restore to respond to source paths like this (assume outdir
corresponds to "." if there no -C argument or to -C outdir):

  /foo/what/ever - extract ever to outdir/ever
  /foo/what/ever/ - extract ever/* to outdir/*
  /foo/what/ever/. - extract ever/. to outdir/. (i.e. outdir == ever).

Also fix handling of top-level commit symlinks.  Previously bup would
just restore /foo/latest as a dummy symlink like this:

   latest -> ../.commit/SHA

Instead, dereference latest and restore the target instead.

Tighten up stripping/grafting with additional argument checks, and
handle any root collisions by creating a fake root dir (see comments
in save-cmd.py).  Bup still doesn't handle other path collisions yet,
i.e. if both /foo/bar and /bar are remapped to /bar.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Cache all password (pwd) and group (grp) database lookups.

Thanks to Jann Horn <jannhorn@googlemail.com> for determining that
even just caching user names and group names could provide a notable
performance improvement in some cases.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Handle nonexistent groups in Metadata._apply_common_rec().

Thanks to Oei, Yung-Chin <yungchin@yungchin.nl> for tracking this
down.

Reported-by: Yung-Chin <yungchin@yungchin.nl>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Change index; include atime; pack time as xstat timespec; use ns in memory.

Update the index format header to 'BUPI\0\0\0\4' (version 4).

Change the index to include atimes, and write all times to bupindex as
xstat timespecs.  This is in preparation for indexing all metadata.
After moving all of the times to the index, there should be far fewer
unique instances of the remaining metadata in many/most cases.

Change the index and index-related code to handle all time
values as integer nanoseconds since the epoch, excepting the packed
format, which (as mentioned above) is now an xstat timespec.

After these changes the index-related in-memory and on-disk time value
handling should match that of the metadata code (i.e. metadata.py,
.bupm, etc.).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Remove tmax; should be unnecessary with high resolution timestamps.

This effectively reverses 7956f4eb2646d6ff9d56e62ef9e09e8d4763bb46.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Store metadata in the index, in bupindex.meta; only store unique values.

FIXME: add the relevant information to DESIGN (and any suitable code
comments) before adding this to master.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Move bup index's tick to the correct place; document related risks.

Put bup index's tick *after* update_index(), where it belongs, and
document the risks involved in modifying the filesytem during an index
run, or within too close proximity.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add the filename/fd and invalid value to stat_struc_to_py error messages.

Thanks to Nathan Bird <ecthellion@gmail.com> for the initial report.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add support for "bup index --exclude-rx <pattern> ...".

When --exclude-rx <pattern> is provided to bup index, exclude any path
matching <pattern>, which must be a Python regular expression
(http://docs.python.org/library/re.html).  The pattern will be
compared against the full path, without anchoring, so "x/y" will match
"ox/yard" or "box/yards".  To exclude the contents of /tmp, but not
the directory itself, use "^/tmp/.".

You may check the behavior at runtime by setting BUP_DEBUG=2 in the
environment.

Thanks to Zoran Zaric <zz@zoranzaric.de> for reporting a bug in an
earlier version of this patch.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add support for "bup restore --exclude-rx <pattern> ...".

When --exclude-rx <pattern> is provided to bup restore, don't restore
any path matching <pattern>, which must be a Python regular expression
(http://docs.python.org/library/re.html).  The pattern will be
compared against the full path, without anchoring, so "x/y" will match
"ox/yard" or "box/yards".  To exclude the contents of /tmp, but not
the directory itself, use "^/tmp/.".

You may check the behavior at runtime by setting BUP_DEBUG=2 in the
environment.

Thanks to Zoran Zaric <zz@zoranzaric.de> for reporting a bug in an
earlier version of this patch.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Implement git.PackIdxV{1,2}.hashes_sorted_by_ofs()
Implement BitArray
Add repack command
Implement repository wide locking

Add git.lock(), git.unlock() and git.is_locked(), which handle locking
of the repository.
Add locking to repack command
Make save and repack respect a lock
unlock repository on error "No reachable objects found"

running bup repack on an empty repository left the repository locked.

Signed-off-by: Tim Riemenschneider <debian@tim-riemenschneider.de>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
also delete pack-*.vol000+200.par2 files after repack

otherwise these would be left behind

Signed-off-by: Tim Riemenschneider <debian@tim-riemenschneider.de>
[zz@zoranzaric.de: Remove the test that counts packfiles]
Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
Bugfix: repack did not work for "larger" repositories

repack only worked, if the git.PackIdxList still contained all the
*.idx files, one per *.pack file. If they got replaced by a *.midx file,
not only it traversed objects multiple times, these objects are not
considered for repack. (Since I think that the other pack-files are
untouched, no data-loss should occur)

This fix tells PackIdxList to ignore the midx-files.

Signed-off-by: Tim Riemenschneider <git@tim-riemenschneider.de>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Fix: Use env-var BUP_MAIN_EXE

Instead of running 'bup fsck -g' (and thus require bup being in the PATH)
use the environment variable BUP_MAIN_EXE
(otherwise the testcase with par2 fails, too)

Signed-off-by: Tim Riemenschneider <git@tim-riemenschneider.de>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>