Use the correct C types when converting between C and Python in _helpers.c.

Previously, bup might fail when handling large repositories.  For
example:

    Traceback (most recent call last):
     File "/usr/lib/bup/cmd/bup-midx", line 258, in <module>
       do_midx_dir(path)
     File "/usr/lib/bup/cmd/bup-midx", line 199, in do_midx_dir
       all = list(do_midx_group(path, part1)) + part2
     File "/usr/lib/bup/cmd/bup-midx", line 216, in do_midx_group
       rv = _do_midx(path, None, sublist, gprefix)
     File "/usr/lib/bup/cmd/bup-midx", line 123, in _do_midx
       count = merge_into(fmap, bits, total, inp)
  OverflowError: size does not fit in an int
  ['/usr/bin/bup', 'midx', '--auto', '--dir',
  '/backup/bup/repository/objects/pack']: returned 1

Signed-off-by: Mark J Hewitt <m.hewitt@computer.org>
[rlb@defaultvalue.org: Remove vestigal header includes.
 Remove Py_ssize_t definition for Python < 2.5.]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Take additional precautions with the Python/C conversions in _helpers.c.

bitmatch(): Use Py_ssize_t for byte, to match the values it's compared
against.  Make sure the result will fit in the Python destination.

bloom_add(): Use "n" for the len conversion since len is Py_ssize_t.

bloom_contains(): Make sure the Python -> C converted values' ranges
are appropriate before using them.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Mark J Hewitt <mjh@idnet.com>
Don't even test for Linux xattrs if platform.system() doesn't include Linux.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
metadata.py: always set xattr to None by default.

Reported-by: Will Rouesnel <w.rouesnel@gmail.com>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
tmetadata.py: skip test_apply_to_path_restricted_access() on Cygwin.

Reported-by: Will Rouesnel <w.rouesnel@gmail.com>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
metadata.py: ignore posix1e if platform.system() includes CYGWIN.

Reported-by: Will Rouesnel <w.rouesnel@gmail.com>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
bup drecurse/index: allow non-directory --exclude and --exclude-from paths.

Previously bup only allowed directory exclusions.

Signed-off-by: Jakob Matthes <jakob.matthes@gmail.com>
Reviewed-by: Gabriel Filion <lelutin@gmail.com>
[rlb@defaultvalue.org: change test file from g to j to avoid conflict
 with subsequent "exclude-from" test; add WVPASS to mkdir so problem
 is evident; edit commit message.]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add "set -eo pipefail" to "exclude" and "exclude-from" tests.

Don't allow non-tests to silently fail (i.e. mkdir, touch, echo, rm,
etc.).  Eventually all test code should have similar behavior.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Create symlinks in t/configure-sampledata; remove them from git.

Since git doesn't always handle symlinks the way bup expects (Cygwin),
don't keep them in git, create them via "t/configure-sampledata
--setup".

This also provides a place to handle anything else that falls outside
git's purview that we might want to add.

Integrate configure-sampledata into the makefile clean and all
targets.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
configure-sampledata: test -h before -e when appropriate in --clean.

Use "test -h" before "test -e" for items that might be dangling
symlinks since "test -e" returns false in that case.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Ignore artefacts from sampledata

9b5ae5f53e introduced the creation of some symlinks to avoid issues on
Cygwin. "make" now leaves the repository in a dirty state since we
forgot to exclude those symlinks.

The symlinks in question are products of the build/test process so we
need to avoid commiting them.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Add a --no-check-device option to bup index (cf. tar).

Ignore stat.st_dev when comparing files against existing index entries, like
git does without #define USE_STDEV:

  http://www.kernel.org/pub/software/scm/git/docs/v1.7.10.1/technical/racy-git.txt

See also: "info tar".

Signed-off-by: Zak Wilcox <iwilcox@iwilcox.me.uk>
[rlb@defaultvalue.org: change original --ignore-dev to --no-check-device to match tar.
 Adjust code to work with current master.
 Remove tests -- will be reintroduced shortly.]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add index --no-check-device to the index manpage.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add tests for index --no-check-device, and support for t/mnt.

Add support for t/mnt; anything mounted there will be cleaned up
during "make clean".

Thanks to Zak Wilcox <iwilcox@iwilcox.me.uk> for an earlier version of
these tests.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
restore-cmd.py: make sure restore paths have at least two components.

Reported-by: Tim Riemenschneider <t.riemenschneider@detco.de>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
wvtest.py: add a fallback definition of relpath() for Python < 2.6.

Since bup still supports Python 2.5 and relpath was added to Python in
2.6, add an os.path.relpath replacement, adapted from CPython source
(tag v2.6, file Lib/posixpath.py, hg-commit 95fff5a6a276).

Signed-off-by: Thomas Haller <thom311@gmail.com>
[rlb@defaultvalue.org: update LICENSE and add comment to code about
 updating LICENSE.]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
LICENSE: mention the lib/tornado license in lib/tornado/README.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
t/test.sh: work around NetBSD setting -A for 'ls' when root (again).

Apply the same workaround for the "compression9" tests that was
applied in commit 51bd5832b391f4c128cc42a3a5970733775f1ba7.

Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Add support for "bup index --exclude-rx <pattern> ...".

When --exclude-rx <pattern> is provided to bup index, exclude any path
matching <pattern>, which must be a Python regular expression
(http://docs.python.org/library/re.html).  The pattern will be
compared against the full path, without anchoring, so "x/y" will match
"ox/yard" or "box/yards".  To exclude the contents of /tmp, but not
the directory itself, use "^/tmp/.".

You may check the behavior at runtime by setting BUP_DEBUG=2 in the
environment.

Thanks to Zoran Zaric <zz@zoranzaric.de> for reporting a bug in an
earlier version of this patch.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Add support for "bup restore --exclude-rx <pattern> ...".

When --exclude-rx <pattern> is provided to bup restore, don't restore
any path matching <pattern>, which must be a Python regular expression
(http://docs.python.org/library/re.html).  The pattern will be
compared against the full path, without anchoring, so "x/y" will match
"ox/yard" or "box/yards".  To exclude the contents of /tmp, but not
the directory itself, use "^/tmp/.".

You may check the behavior at runtime by setting BUP_DEBUG=2 in the
environment.

Thanks to Zoran Zaric <zz@zoranzaric.de> for reporting a bug in an
earlier version of this patch.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Zoran Zaric <zz@zoranzaric.de>