index-cmd: prevent a division by zero while computing paths_per_second.

Signed-off-by: Patryck Rouleau <prouleau72@gmail.com>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
index.py: fix typo in Writer._add.

Thanks to bk <b.karran@gmail.com> for reporting the problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add initial support for ls(1) style "-d" argument to "bup ls".

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
vfs.py: don't redundantly _populate_metadata for Dir()s.

This fix should significantly improve performance.  Without it, every
call to node.metadata() caused the VFS to read the entire .bupm file
in order to (re)initialize the metadata for every entry in the
directory.  Don't do that.

I thought I'd already implemented this, but apparently only in my
head.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Make bup-split commits appear as files to the vfs layer.

When viewing branches that were generated by bup-split through bup-fuse
(or any other frontend relying on vfs.py), these are presented as trees
of the hashsplitted blobs. This means that bup-split output is only
usefully accessible through bup-join.

This change makes bup-split store named commits such that they appear as
files, named with the last component of their branch name(*). That is,
from the vfs layer, they now appear like so:
    branch_name/latest/branch_basename

(*) While bup doesn't currently handle slashes in branch names, patches
    to this end are on the mailing list, so this patch should handle
    them, in anticipation of their general support in bup.

To address potential concerns: the storage format is changed in subtle
ways, in that the top level tree now contains a "normally" named object,
rather than byte-offset names.  However, bup-join doesn't care about
this, and as bup-join was previously the only way to use these commits,
the user experience is not affected.

We also add a test for the new functionality. (The test uses an empty
string as input data, because this is the second way in which this patch
changes the behaviour of bup-split: previously, passing empty strings to
bup-split would make it generate an empty git tree, whereas now it
relies on hashsplit.split_to_blob_or_tree() to make a blob for the empty
string. This is meaningful because vfs.py chokes on empty git trees.)

Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl>
[rlb@defaultvalue.org: rebase to current master; adjust code indentation.]
Always name the top-level split file "data", instead of the branch basename.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Merge work to have "bup split" produce a top-level VFS file.
save-cmd.py: hashsplit .bupm metadata files.

Previously the metdata for each directory's content was being written
as a single blob, with no deduplication -- probably not what we want.
Fix it.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add VFS Node release() and release nodes during restore, after traversal.

Add a release() method to Node that will drop resources that can (and
will) be automatically restored when required -- though restoring the
resources may have a non-trivial cost.  For now, drop the node's
metadata and its children.

Call node.release() in restore-cmd.py after restoring a node.  This
substantially decreases the memory required by a restore because the
whole tree is no longer retained in RAM.

Thanks to Patrick Rouleau <prouleau72@gmail.com> for helping track
down the problem, and proposing a slightly different initial patch to
fix it.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Patrick Rouleau <prouleau72@gmail.com>
test-ls.sh: handle Cygwin's coupling of timestamp/permission modifications.

In Cygwin, the file's modification date is updated when its
permissions are modified. To make the tests compatible with Cygwin,
create the files first, set their permissions, then set the date.

Signed-off-by: Patrick Rouleau <prouleau72@gmail.com>
[rlb@defaultvalue.org: adjust commit message.]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
helpers.py: check subprocess exit status in readpipe().

Checking the exit status seems like a good idea.  Note that we can't
just use check_output() because it wasn't added until Python 2.7.

Thanks to Alexander Barton <alex@barton.de> for reporting the version
incompatibility with respect to an earlier version of this patch.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Use the author date, not committer date for the bup save date.

Since it's the author date that indicates when the save was made, use
that instead of the committer date to represent saves.

Doing so means that any operations on branches (like rebase,
cherry-pick, etc.) won't change the "save date" as far as bup is
concerned.

This should have no effect with respect to backward compatibility
because until now, bup always used the same date for the author and
committer.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
helpers.py: use returncode to get the subprocess exit code in readpipe().

Hello?  It's returncode, not retcode -- and how about a test?

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Drop helpers.next() and just use Python's built-in.

The python built-in 'next' function has an optional second parameter
that specifies what to return instead of excepting with StopIteration.
There is therefore no need for the function in helpers.py.

Signed-off-by: Aidan Hobson Sayers <aidanhs@cantab.net>
[rlb@defaultvalue.org: adjust commit message.]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Only define helpers.next() if Python's isn't new enough.

Since helpers.next() now matches the Python built-in's semantics
(since Python 2.6), just rely on the built-in when it's new enough.
Otherwise, set up a matching replacement.

Borrow the fallback next documentation string from Python.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Merge work to make helpers.next() match the newer Python next().
Write the system ARG_MAX to an autogenerated config.py at configure time.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Add a batchpipe() command to helpers that behaves somewhat like xargs(1).

Add batchpipe(), which will yield the output produced by calling a
given external command with a given list of arguments.

The resulting output may be provided in chunks, from multiple
invocations of the command, if the limits imposed by ARG_MAX make that
necessary.

See http://www.in-ulm.de/~mascheck/various/argmax/ for details, but
note that batchpipe() takes the additional precaution of adding room
for the argv pointers in addition to the envp pointers.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Retrieve the dates for all branches with one bulk git call in the VFS.

Instead of calling "git rev-list" once per branch to get the branch
tip dates for the VFS branch list, make a single call to "git show"
using a suitable pretty format, and pass all the branch refs as
arguments.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Revert "Write the system ARG_MAX to an autogenerated config.py..."

Prepare to try a more direct approach.

This reverts commit b72365ad597edbc528da5b93528fc3a67a18d58e.