Drop helpers.next() and just use Python's built-in.

The python built-in 'next' function has an optional second parameter
that specifies what to return instead of excepting with StopIteration.
There is therefore no need for the function in helpers.py.

Signed-off-by: Aidan Hobson Sayers <aidanhs@cantab.net>
[rlb@defaultvalue.org: adjust commit message.]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Only define helpers.next() if Python's isn't new enough.

Since helpers.next() now matches the Python built-in's semantics
(since Python 2.6), just rely on the built-in when it's new enough.
Otherwise, set up a matching replacement.

Borrow the fallback next documentation string from Python.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Merge work to make helpers.next() match the newer Python next().
Write the system ARG_MAX to an autogenerated config.py at configure time.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add a batchpipe() command to helpers that behaves somewhat like xargs(1).

Add batchpipe(), which will yield the output produced by calling a
given external command with a given list of arguments.

The resulting output may be provided in chunks, from multiple
invocations of the command, if the limits imposed by ARG_MAX make that
necessary.

See http://www.in-ulm.de/~mascheck/various/argmax/ for details, but
note that batchpipe() takes the additional precaution of adding room
for the argv pointers in addition to the envp pointers.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Retrieve the dates for all branches with one bulk git call in the VFS.

Instead of calling "git rev-list" once per branch to get the branch
tip dates for the VFS branch list, make a single call to "git show"
using a suitable pretty format, and pass all the branch refs as
arguments.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Revert "Write the system ARG_MAX to an autogenerated config.py..."

Prepare to try a more direct approach.

This reverts commit b72365ad597edbc528da5b93528fc3a67a18d58e.
Read ARG_MAX directly via os.sysconf('SC_ARG_MAX').

For now, this code also adds an optional arg_max parameter to
batchpipe(), exclusively for testing.

There appears to be an issue with wvtest at the moment that causes
assignments to something like helpers.arg_max to have no effect from
the importers perspective.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Find SC_ARG_MAX via C sysconf().

Apparently os.sysconf('SC_ARG_MAX') returns -1, even in some common
cases.

Thanks to Alexander Barton <alex@barton.de> for reporting that at
least when run under make 3.81, os.sysconf('SC_ARG_MAX') was just
returning -1.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Always line-buffer bup restore stdout

Flush stdout after every line, even when stdio is not a tty, to
provide more regular progress information.  Useful for progress
monitoring by a parent process or when watching a logfile with "tail".

Signed-off-by: Simon Persson <simonpersson1@gmail.com>
[rlb@defaultvalue.org: adjust commit message]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Return integers, not strings from get_commit_dates()

Returning strings broke bup fuse (surprising that it worked
elsewhere), and since we have no fuse tests, it wasn't immediately
apparent.

So fix the bug and add some initial (trivial) fuse tests so that at
least this particular problem doesn't happen again.

To cleanup after the fuse tests, detect fuse mounts in
t/cleanup-mounts-under, and try to unmount them.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Move unshared version code from helpers to version-cmd.py

While this is relatively "harmless", the import * directive brings in a
whole lot of unused code into the version subcommand.

At the same time, the three version_* functions are only used in the
version subcommand, so their place inside helpers.py is unjustified. The
same goes for the _version module.

Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
[rlb@defaultvalue.org: adjust commit message]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Re-allow backup set names containing "/"

Revert the prohibition because all releases up to now have allowed
"/", and so that's nothing new.  Disabling "/" in 0.26 would be a
regression, and one we're not ready to commit to -- in fact, we may
eventually do the opposite, and add comprehensive support for "/".

This reverts commit 52a98179490c6ccb990e221bbdeb511f1de59de4.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Initialize MetaStoreWriter self._file to None

Without this, if something goes wrong in the constructor self._file
may not be initialized, which causes close(), invoked via __del__(),
to crash.

Thanks to Fabian Bakkum <fabianbakkum@hotmail.com> for reporting the
problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Fix git_commit_dates() to handle (obvious) duplicates

Since git show will suppress duplicate results, change
git_commit_dates() to detect that situation and repair the damage.

Though currently, it can only handle cases where the duplicates are
obvious, i.e. have the same exact string in the argument list.  It
won't be able to handle cases where two arguments differ, but resolve
to the same underlying hash.  For our current purposes, that should be
fine since we only pass in hashes.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Move VFS cp() to git.py; handle repodir changes

Move the VFS's cp() to git.py and reset the cp() CatPipe whenever the
global repodir changes.

Previously, if two different lib/bup/t/ tests (for example) needed to
use two different repositories, and both happened to indirectly call
cp(), the second test would end up with a CatPipe() connected to the
wrong repository.

In the longer run, we may want to consider further cleanup here, but
this should fix the immediate problem without too much risk.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Use CatPipe, not show, in git_commit_dates()

Import the commit parser from the experimental bup-get branch and use
it along with CatPipe to produce the git commit dates in
git_commit_dates().

This appears to resolve the performance problem with real archives
that was introduced by the use of "git show -s --pretty=format:%ct
..." (cf. 00ba9fb811e71bb6182b9379461bc6b493e3b7a4), and is still much
faster than 0.25 for at least a 1000 branch synthetic repository here.

Thanks to Gabriel Filion <gabster@lelutin.ca> for reporting the
problem, and to him and Patrick Rouleau <prouleau72@gmail.com> for
helping test the solution.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Don't call _mksubs() in VFS Node nlink()

The VFS Node nlink() code calls _mksubs(), but then just returns 1.
It looks like that may be unnecessary work, so stop doing it.

Thanks to William Stein <wstein@gmail.com> for pointing it out.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
midx: close the mmap before erasing an midx file

Linux allows erasing a file even if it is still memory mapped, but
Windows does not. However, Cygwin tries to emulate Linux and, under
certain conditions, it silently moves the file to $RECYCLED.BIN/, but
not in a way where Windows takes care of it when we use the "empty the
recycled bin" command. The erased midx ends up comsuming more and more
disk space.

To solve this, we have to close the midx's mmap before erasing
the midx.

Cygwin users: you can use this command to clean up your
$RECYCLE.BIN/ directories:
find /cygdrive/?/\$RECYCLE.BIN/ -name "\.???[[:xdigit:]]*" -print -delete

Signed-off-by: Patrick Rouleau <prouleau72@gmail.com>
[rlb@defaultvalue.org: adjust commit message]
Keep PackMidx pointers until finished with mmap

Now that PackMidx objects close their mmap, don't drop the pointers to
them in midx-cmd.py until finished with the mmaps.

The isinstance() call may need revisiting later.  i.e. should all
indexes support close()?

Signed-off-by: Rob Browning <rlb@defaultvalue.org>