Don't crash; defer an error when a path changes while indexing it.

Mirror the error handling already used by save.

With this change the result for a path that changes or is deleted
between the drecurse traversal and the from_path() call should be the
same as the result for a path that changes or is deleted between index
and save.

Thanks to Yann Autissier <yann.autissier@gmail.com> for reporting the
problem and Gabriel Filion <gabster@lelutin.ca> for helping figure out
the solution.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Makefile: simplify runtests-python since metadata tests aren't conditional.

Since the metadata tests are no longer conditional, revert
runtests-python to the simpler implementation.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Manually set TMPDIR to $(CURDIR)/tmp for all tests for now.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
metadata.py: fix variable access in Writer.add(); tmax -> self.tmax.

Thanks to Patrick Rouleau <prouleau72@gmail.com> for reporting the
problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
metadata.py: fix incorrectly indented line in Writer._add().

Thanks to Patrick Rouleau <prouleau72@gmail.com> for reporting the
problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Completely disable ACL support on NetBSD for now.

Signed-off-by: Thomas Klausner <tk@giga.or.at>
[rlb@defaultvalue.org: adjust commit message; wrap code.]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
DESIGN: fix typo.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Remove redundant *bits assignment in bupsplit_find_ofs().

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Enforce MAX_PER_TREE by always _squish()ing in spit_to_shalist().

Previously bup would ignore MAX_PER_TREE whenever it hit a long run of
data that didn't produce a non-zero hashplit "level" (see the
discussion of fanout in DESIGN).  That can happen, for example, when
traversing a file containing large zero regions (sparse or not).

As a result, bup could produce an arbitrarily large number of blobs at
level 0 of the hashsplit tree, causing it to consume increasing memory
during split/save, and to behave badly during join/restore.

To fix that, don't try to outsmart _squish() -- just call it every
time, and let it enforce MAX_PER_TREE when appropriate.

Thanks to trebor <robert.rebstock@tempelhof-projekt.de> for reporting
the problem a long while back, Yung-Chin Oei <yungchin@yungchin.nl>
for tracking down the cause and proposing a slightly different fix,
and Aidan Hobson Sayers <aidanphs@gmail.com> for suggesting this
particular approach.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Fix off-by-one in MAX_PER_TREE guard in _squish().

Previously bup would allow MAX_PER_TREE + 1 items in a tree.  Change
the guard to ensure that trees really are limited to MAX_PER_TREE.

Thanks to Patrick Rouleau <prouleau72@gmail.com> for reporting the
problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Don't check ACLs in t/compare-trees on FreeBSD.

Treat FreeBSD like Cygwin and Darwin (OS X) in t/compare-trees.  This
should finish the process of disabling POSIX ACL support on FreeBSD
that was started here: 349ff15c7db09883abc20bdf3e8a4df6bff12cd3.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Revert "Don't check ACLs in t/compare-trees on FreeBSD."

This reverts commit 985b17f45707fe3759ca13f005625934f5f63b4c.

That should have been *NetBSD*.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Don't check ACLs in t/compare-trees on NetBSD.

Treat NetBSD like Cygwin and Darwin (OS X) in t/compare-trees.  This
should finish the process of disabling POSIX ACL support on NetBSD
that was started here: 349ff15c7db09883abc20bdf3e8a4df6bff12cd3.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Use the correct types when calling FS_IOC_GETFLAGS/FS_IOC_SETFLAGS.

Despite the definitions in <linux/fs.h> these ioctls take int
arguments, not longs. This is important for 64-bit big endian
machines.

See http://marc.info/?l=linux-fsdevel&m=138552482917220&w=2

Signed-off-by: Aurelien Jarno <aurelien@aurel32.net>
[rlb@defaultvalue.org: adjust commit message; use "I" conversion
 to/from Python instead of "k".]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Don't compute local script_home in lib.sh realpath (depended on pwd).

Compute bup_t_lib_script_home once at load time so that it won't be
affected by changes to pwd, and use it in realpath.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Create t/force-delete from t/lib.sh force-delete and use it everywhere.

Move the t/lib.sh force-delete code to t/force-delete as a standalone
command, and use it everywhere -- particularly to delete t/tmp during
"make clean", given the recent move to run all tests inside t/ via
TMPDIR (18f838c9a66d521f9fa042eead3c5771ca99b03a).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Remove t/ from the test-meta.sh test tree now that it contains t/tmp/.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add helpers.fdatasync(); fix interleaved mmap/read-write in midx.

Have helpers.fdatasync() fall back to os.fsync() whenever
os.fdatasync() isn't available.

Thanks to Yann Autissier <yann.autissier@gmail.com> for reporting the
problem that prompted this work.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
tmetadata.py: chmod tree before deletion to fix NetBSD test failure.

Apparently NetBSD won't let you recursively delete a parent directory
containing a 000 subdirectory, so "chmod -R u+rwX" the tree first.

Signed-off-by: Thomas Klausner <tk@giga.or.at>
[rlb@defaultvalue.org: adjust commit message.]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
t/test-meta.sh: handle systems that have sgid directories.

Explicitly chown the test directory to a group we're in so that the
tests will work on systems that have sgid directories by default
(i.e. NetBSD, etc.).

Thanks to Thomas Klausner <tk@giga.or.at> for reporting the problem
and helping track down the solution.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>