Omit htonll when already #defined

OS X 10.10 Yosemite defines it as a macro.

Signed-off-by: R. Andrew Ohana <andrew.ohana@gmail.com>
[rlb@defaultvalue.org: adjust comments; adjust commit message]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
test-fuse.sh: make ls output locale independent

Call "ls" with LC_ALL set to "C".

(Otherwise it reports "total" for English, "insgesamt" for German;
other locales would be similar.)

Signed-off-by: Tim Riemenschneider <git@tim-riemenschneider.de>
[rlb@defaultvalue.org: adjust commit message]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
Add atomically_replaced_file for safer output

Use `tempfile.mkstemp` to write files safely and then rename them
atomically to the target filename at the end. This guarantees that two
concurrent processes trying to write the same target file will result
in one process "winning" with a consistent version.

Use this to protect:
 * midx writing
 * .idx cache writing

Previously both of these processes `open(destination + '.tmp', 'w')`
which would lead to concurrent processes writing the same tmp file at
the same time.

See also https://groups.google.com/d/msg/bup-list/qdPUaXGO1cI/g8K2aFaaXz0J

Signed-off-by: Nathan Bird <ecthellion@gmail.com>
[rlb@defaultvalue.org: adjust commit message]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
Adjust atomically_replaced_file documentation.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
thelpers.py: use t/tmp as the tmpdir parent

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Merge work on atomic writes
atomically_replaced_file: ensure error leaves original

Make sure an exception inside the body leaves the original file
content.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
Only silence ENOENT in helpers.unlink

This is what it is documented as doing and what makes sense in this
case.

Signed-off-by: Nathan Bird <ecthellion@gmail.com>
[rlb@defaultvalue.org: adjust commit message]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
Handle kernel/FUSE disagreement over Linux attrs

Use long for the Linux attr type, so that it should work on all
little-endian systems, across both normal and FUSE-backed filesystems.

Disable Linux attr support for now on big-endian systems where
sizeof(long) > sizeof(int).  See the changes to bup-index.md and
_helpers.c for more information.

Eventually, we may want to add an argument to allow re-enabling
support on affected systems, when the user can guarantee a homogeneous
filesystem type -- or better yet, the kernel and FUSE will finally
sort out their issues.

Thanks to Tilo Schwarz <mail@tilo-schwarz.de> and daryl5@arcor.de for
reporting and pursuing the problems that lead to this patch.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Assume FS_IOC_GETFLAGS may trash output on error

Don't assert that the attr value is reasonable (<= UINT_MAX) when
FS_IOC_GETFLAGS returns an error, because it may change the the value
to something invalid in that situation.

This has been observed to occur when, for example, the underlying
filesystem does not support the operation,

Signed-off-by: Mark J Hewitt <m.hewitt@computer.org>
[rlb@defaultvalue.org: adjust commit message]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
Fix typo in atomically_replaced_file docstring

Thanks to Gabriel Filion <gabster@lelutin.ca> for pointing out the
problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
save: catch and log SSH errors

Currently, if the ssh connection to the remote server fails, we get the
server errors, then an ugly traceback that ends up saying "look at the
errors above". It makes it look like the traceback is part of the errors
that we need to look at, when in reality it's just an exception that's
not handled correctly by bup-save.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
[rlb@defaultvalue.org: adjust commit summary]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
Add support for "restore --sparse"

Thanks to Patrick Rouleau <prouleau72@gmail.com> for spotting a
serious potential short-write problem in an earlier revision of this
work, and suggesting other improvements.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
test-cat-file.sh: handle hashsplit bupm files

A bit back we started allowing .bupm files to be hashsplit:

  ee777400cf624b4f7a24b15614c2b6c745560dcb

So add t/git-cat-tree and use it in test-cat-file.sh to prevent the
test from crashing whenever the bupm is a tree.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
test-meta.sh: finish renaming variables

Apparently the changes started here weren't completely finished;
finish them:

  b836113e503ab3f8e3c30c3283eb59f936fb0023

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
test-meta.sh: escape user/group expressions

Escape external user/group strings before passing them to 'grep -E',
otherwise calls like this will fail when the string includes regular
expression metacharacters:

Thanks to Pierre-Emmanuel Novac <piernov@gmail.com> for reporting the
problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
Change incorrect unknown-owners references

The test program is actually called unknown-owner (t/unknown-owner).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
test-meta.sh: add missing "$() || exit $?" tests

Their omission was masking bugs fixed in several of the previous
commits.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
Allow per-instance specification of the repo_dir for CatPipe

This allows the creation of CatPipes that refer to repositories other
than the default, which in turn allows copying between repositories,
etc.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
Allow the specification of a repo_dir to some VFS and git operations

Previously, these VFS and git operations would only operate on the
default repository (git.repo()).

Have vfs.cp() handle more than one repository (via an internal cache).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>