git-svn: only look at the new parts of svn:mergeinfo

In a Subversion repository where many feature branches are merged into a
trunk, the svn:mergeinfo property can grow very large. This severely
slows down git-svn's make_log_entry() because it is checking all
mergeinfo entries every time the property changes.

In most cases, the additions to svn:mergeinfo since the last commit are
pretty small, and there is nothing to gain by checking merges that were
already checked for the last commit in the branch.

Add a mergeinfo_changes() function which computes the set of interesting
changes to svn:mergeinfo since the last commit. Filter out merged
branches whose ranges haven't changed, and remove a common prefix of
ranges from other merged branches.

This speeds up "git svn fetch" by several orders of magnitude on a large
repository where thousands of feature branches have been merged.

Signed-off-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: only look at the root path for svn:mergeinfo

Subversion can put mergeinfo on any sub-directory to track cherry-picks.
Since cherry-picks are not represented explicitly in git, git-svn should
just ignore it.

Signed-off-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: reduce check_cherry_pick cache overhead

We do not need to store entire lists of commits, only the
number of incomplete and the first commit for reference.
This reduces the amount of data we need to store in memory
and on disk stores.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: cache only mergeinfo revisions

This should reduce excessive memory usage from the new mergeinfo
caches without hurting performance too much, assuming reasonable
latency to the SVN server.

Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: remove mergeinfo rev caching

This should further reduce memory usage from the new mergeinfo
speedups without hurting performance too much, assuming
reasonable latency to the SVN server.

Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn.txt: advertise pushurl with dcommit

Advertise that the svn-remote.<name>.pushurl config key allows specifying
the commit URL for the entire SVN repository in the documentation of the
git svn dcommit command.

Signed-off-by: Sveinung Kvilhaugsvik <sveinung84@users.sourceforge.net>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: reload RA every log-window-size

Despite attempting to use local memory pools everywhere we can,
(including our call to SVN::Ra::do_update and all subsequent reporter
calls), there does not appear to be a way to force the Git::SVN::Fetcher
callbacks to use a pool other than the per-SVN::Ra pool.
Git::SVN::Fetcher ends up using the main RA pool which grows
monotonically in size for the lifetime of the RA object.

Thus the only way to free that memory appears to be to destroy and
recreate the RA connection for at every --log-window-size interval.

This reduces memory usage over the course of fetching 10K revisions
using a test repository created with the script at the end of this
commit message.

As reported by time(1) on my x86-64 system:

	before: 54024k
	 after: 28680k

Unfortunately, there remains some yet-to-be-tracked-down slow memory
growth which would be evident as the `nr' parameter increases in
the repository generation script:
-----------------------------8<------------------------------
set -e
tmp=$(mktemp -d svntestrepo-XXXXXXXX)
svnadmin create "$tmp"
repo=file://"$(cd $tmp && pwd)"
svn co "$repo" "$tmp/wd"
cd "$tmp/wd"
if ! test -f a
then
	> a
	svn add a
	svn commit -m 'A'
fi

nr=10000
while test $nr -gt 0
do
	echo $nr > a
	svn commit -q -m A
	nr=$((nr - 1))
done
echo "repository created in $repo"
-----------------------------8<------------------------------

Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: remove unnecessary DESTROY override

This override was probably never necessary, but most likely a no-op
as it does not appear to do anything in SVN::Ra itself.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: save a little memory as fetch progresses

There is no reason to keep entries in the %revs hash after we're
done processing a revision, so allow entries become freed as
processing continues.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: disable _rev_list memoization

This memoization appears unneeded as the check_cherry_pick2 cache is
in front of it does enough.

With this change applied, importing from local svn+ssh and http copies
of the R repo[1] takes only 2:00 (2 hours) on my system and the git-svn
process never uses more than 60MB RSS on my x86-64 GNU/Linux system[2].
This 60M measurement is only for the git-svn Perl process itself and
does not include memory used by git subprocesses accessing large packs
(subprocess memory usage _is_ measured by my time(1) tool).

Before this change, an import took longer (2:20) on svn+ssh:// but
git-svn used around 240MB during the imports.  Worse yet, git-svn
ballooned to over 400M when writing out the cache to the filesystem.

I also tried removing memoization for `has_no_changes', too, but a
local copy of the R repository(*) was not close to finishing within
10 hours on my system.

[1] http://svn.r-project.org/R
[2] file:// repos causes libsvn to use more memory internally

Signed-off-by: Eric Wong <normalperson@yhbt.net>
Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
use child_process_init() to initialize struct child_process variables

Call child_process_init() instead of zeroing the memory of variables of
type struct child_process by hand before use because the former is both
clearer and shorter.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git.pm: add specified name to tempfile template

This should help me track down errors in git-svn more easily:

	write .git/Git_XXXXXX: Bad file descriptor
	 at /usr/lib/perl5/SVN/Ra.pm line 623

Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: prepare SVN::Ra config pieces once

Memoizing these initialization functions saves some memory for
long fetches which require scanning many unwanted revisions
before any wanted revisions happen.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
gitk: Do not depend on Cygwin's "kill" command on Windows

Windows does not necessarily mean Cygwin, it could also be MSYS. The
latter ships with a version of "kill" that does not understand "-f".
In msysgit this was addressed by shipping Cygwin's version of kill.

Properly fix this by using the stock Windows "taskkill" command instead,
which is available since Windows XP Professional.

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
gitk: Show detached HEAD if --all is specified

If HEAD is detached, 'gitk --all' does not show it. This is inconvenient
for frontend program, and for example git log does show the detached HEAD.

gitk uses git rev-parse to find a list of branches to show.
Apparently, the command does not include detached HEAD to output if
--all argument is specified. This has been discussed in [1] and stated
as expected behavior. So rev-parse's parameters should be tuned in gitk.

[1] http://thread.gmane.org/gmane.comp.version-control.git/255996

Signed-off-by: Max Kirillov <max@max630.net>
Signed-off-by: Paul Mackerras <paulus@samba.org>
gitk: Remove boilerplate for configuration variables

Signed-off-by: Max Kirillov <max@max630.net>
Signed-off-by: Paul Mackerras <paulus@samba.org>
git-svn: (cleanup) remove editor param passing

Neither find_extra_svk_parents or find_extra_svn_parents ever
used the `$ed' parameter.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: add space after "W:" prefix in warning

And minor reformatting while we're in the area.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
Merge git://ozlabs.org/~paulus/gitk

* git://ozlabs.org/~paulus/gitk:
  gitk: Remove boilerplate for configuration variables
  gitk: Show detached HEAD if --all is specified
  gitk: Do not depend on Cygwin's "kill" command on Windows
cache-tree: avoid infinite loop on zero-entry tree

The loop in cache-tree's update_one iterates over all the
entries in the index. For each one, we find the cache-tree
subtree which represents our path (creating it if
necessary), and then recurse into update_one again. The
return value we get is the number of index entries that
belonged in that subtree. So for example, with entries:

    a/one
    a/two
    b/one

We start by processing the first entry, "a/one".  We would
find the subtree for "a" and recurse into update_one. That
would then handle "a/one" and "a/two", and return the value
2. The parent function then skips past the 2 handled
entries, and we continue by processing "b/one".

If the recursed-into update_one ever returns 0, then we make
no forward progress in our loop. We would process "a/one"
over and over, infinitely.

This should not happen normally. Any subtree we create must
have at least one path in it (the one that we are
processing!). However, we may also reuse a cache-tree entry
we found in the on-disk index. For the same reason, this
should also never have zero entries. However, certain buggy
versions of libgit2 could produce such bogus cache-tree
records. The libgit2 bug has since been fixed, but it does
not hurt to protect ourselves against bogus input coming
from the on-disk data structures.

Note that this is not a die("BUG") or assert, because it is
not an internal bug, but rather a corrupted on-disk
structure. It's possible that we could even recover from it
(by throwing out the bogus cache-tree entry), but it is not
worth the effort; the important thing is that we report an
error instead of looping infinitely.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>