xread, xwrite: limit size of IO to 8MB

Checking out 2GB or more through an external filter (see test) fails
on Mac OS X 10.8.4 (12E55) for a 64-bit executable with:

    error: read from external filter cat failed
    error: cannot feed the input to external filter cat
    error: cat died of signal 13
    error: external filter cat failed 141
    error: external filter cat failed

The reason is that read() immediately returns with EINVAL when asked
to read more than 2GB.  According to POSIX [1], if the value of
nbyte passed to read() is greater than SSIZE_MAX, the result is
implementation-defined.  The write function has the same restriction
[2].  Since OS X still supports running 32-bit executables, the
32-bit limit (SSIZE_MAX = INT_MAX = 2GB - 1) seems to be also
imposed on 64-bit executables under certain conditions.  For write,
the problem has been addressed earlier [6c642a].

Address the problem for read() and write() differently, by limiting
size of IO chunks unconditionally on all platforms in xread() and
xwrite().  Large chunks only cause problems, like causing latencies
when killing the process, even if OS X was not buggy.  Doing IO in
reasonably sized smaller chunks should have no negative impact on
performance.

The compat wrapper clipped_write() introduced earlier [6c642a] is
not needed anymore.  It will be reverted in a separate commit.  The
new test catches read and write problems.

Note that 'git add' exits with 0 even if it prints filtering errors
to stderr.  The test, therefore, checks stderr.  'git add' should
probably be changed (sometime in another commit) to exit with
nonzero if filtering fails.  The test could then be changed to use
test_must_fail.

Thanks to the following people for suggestions and testing:

    Johannes Sixt <j6t@kdbg.org>
    John Keeping <john@keeping.me.uk>
    Jonathan Nieder <jrnieder@gmail.com>
    Kyle J. McKay <mackyle@gmail.com>
    Linus Torvalds <torvalds@linux-foundation.org>
    Torsten Bögershausen <tboegi@web.de>

[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/read.html
[2] http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html

[6c642a] commit 6c642a878688adf46b226903858b53e2d31ac5c3
    compate/clipped-write.c: large write(2) fails on Mac OS X/XNU

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Revert "compat/clipped-write.c: large write(2) fails on Mac OS X/XNU"

This reverts commit 6c642a878688adf46b226903858b53e2d31ac5c3.

The previous commit introduced a size limit on IO chunks on all
platforms.  The compat clipped_write() is not needed anymore.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
submodule: fix confusing variable name

cmd_summary reads the output of git diff, but reads in the submodule path into a
variable called name.  Since this variable does not contain the name of the
submodule, but the path, rename it to be clearer what data it actually holds.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Acked-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
submodule: don't print status output with ignore=all

git status prints information for submodules, but it should ignore the status of
those which have submodule.<name>.ignore set to all.  Fix it so that it does
properly ignore those which have that setting either in .git/config or in
.gitmodules.

Not ignored are submodules that are added, deleted, or moved (which is
essentially a combination of the first two) because it is not easily possible to
determine the old path once a move has occurred, nor is it easily possible to
detect which adds and deletions are moves and which are not.  This also
preserves the previous behavior of always listing modules which are to be
deleted.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Acked-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
config: properly range-check integer values

When we look at a config value as an integer using the
git_config_int function, we carefully range-check the value
we get and complain if it is out of our range. But the range
we compare to is that of a "long", which we then cast to an
"int" in the function's return value. This means that on
systems where "int" and "long" have different sizes (e.g.,
LP64 systems), we may pass the range check, but then return
nonsense by truncating the value as we cast it to an int.

We can solve this by converting git_parse_long into
git_parse_int, and range-checking the "int" range. Nobody
actually cared that we used a "long" internally, since the
result was truncated anyway, and there were no other callers
of git_parse_long.

Of the existing callers, all of them expect small-ish values
that are fine for an "int", and should not be impacted
except when nonsense is fed to them. The one arguable
exception is http.lowSpeedLimit, which is given in bytes per
second, and is fed to curl as a long. However, nobody is
likely to consider 2GB/sec as "low speed", and even if they
did, the new behavior is much better (it barfs and complains
rather than considering the value negative and silently
ignoring it).

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
teach git-config to output large integers

Internally we use "unsigned long" to represent large config
values like packed window memory sizes, packfile size
limits, etc.  On 32-bit systems, this limits these config
options to 4G (and we detect and complain about overflow).
On 64-bit systems, they get the full 64-bit range.

However, there is no way for script callers of git-config to
see the numeric values (they cannot treat them as pure
strings, since they would want git-config to process "3g"
into "3221225472").

This patch teaches git-config a "--ulong" type that can
output these unsigned long values, allowing scripts to
examine the values as the internal code would.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-remote-mediawiki: reset private ref after non-dumb push

Git-mediawiki's "dumb push" sends the local revisions to the remote wiki,
but does not update the local metadata to reflect the push (hence, the
next pull will have to re-import the exported revisions).

The previous implementation was simply omitting the update to the private
ref after a dumb push. This was broken by 664059fb62 (Felipe Contreras,
Apr 17 2013, transport-helper: update remote helper namespace), which
does an automatic update of the private ref (not just the
remote-tracking) on push.

This patch fixes git-remote-mediawiki to reset the private ref after the
push is completed, cancelling the automatic update triggered by
664059fb62.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-remote-mediawiki: add test and check Makefile targets

There are a few level 4 and 2 perlcritic issues in the current code.
We make level 5 fatal, and keep level 2 as warnings so that "make
check" pass.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Document the HTTP transport protocols

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Revised-by: Tay Ray Chuan <rctay89@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
rebase --preserve-merges: ignore "merge.log" config

When "merge.log" config is set, "rebase --preserve-merges" will add
the log lines to the message of the rebased merge commit.  A rebase
should not modify a commit message automatically.

Teach "git-rebase" to ignore that configuration by passing
"--no-log" to the git-merge call.

Signed-off-by: Ralf Thielow <ralf.thielow@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
sha1-lookup: handle duplicate keys with GIT_USE_LOOKUP

The sha1_entry_pos function tries to be smart about
selecting the middle of a range for its binary search by
looking at the value differences between the "lo" and "hi"
constraints. However, it is unable to cope with entries with
duplicate keys in the sorted list.

We may hit a point in the search where both our "lo" and
"hi" point to the same key. In this case, the range of
values between our endpoints is 0, and trying to scale the
difference between our key and the endpoints over that range
is undefined (i.e., divide by zero). The current code
catches this with an "assert(lov < hiv)".

Moreover, after seeing that the first 20 byte of the key are
the same, we will try to establish a value from the 21st
byte. Which is nonsensical.

Instead, we can detect the case that we are in a run of
duplicates, and simply do a final comparison against any one
of them (since they are all the same, it does not matter
which). If the keys match, we have found our entry (or one
of them, anyway).  If not, then we know that we do not need
to look further, as we must be in a run of the duplicate
key.

Furthermore, we know that one of our endpoints must be
the edge of the run of duplicates. For example, given this
sequence:

 idx 0 1 2 3 4 5
 key A C C C C D

If we are searching for "B", we might hit the duplicate run
at lo=1, hi=3 (e.g., by first mi=3, then mi=0). But we can
never have lo > 1, because B < C. That is, if our key is
less than the run, we know that "lo" is the edge, but we can
say nothing of "hi". Similarly, if our key is greater than
the run, we know that "hi" is the edge, but we can say
nothing of "lo". But that is enough for us to return not
only "not found", but show the position at which we would
insert the new item.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
index-pack: optionally reject packs with duplicate objects

Git should never generate packs with duplicate objects.
However, we may see such packs due to bugs in Git or other
implementations (e.g., JGit had such a bug a few years ago).

In theory, such packs should not be a problem for us (we
will simply find one of the instances of the object when
looking in the pack). However, the JGit bug report mentioned
possible infinite loops during repacking due to cycles in
the delta chain.  Though this problem hasn't specifically
been reproduced on modern git, there is no reason not to be
careful with incoming packs, given that only buggy
implementations should be producing such packs, anyway.

This patch introduces the pack.indexDuplicates option to
allow or reject such packs from index-pack. The default
remains to allow it.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
reject duplicates when indexing a packfile we created

The pack index-writing code is capable of detecting and
rejecting duplicate entries. This can be used with
index-pack to prevent buggy foreign packs from entering the
repository.  However, we can also be careful about the packs
we generate, by double-checking during the index write that
we do not have any duplicate objects. This should never
happen, but it serves as an additional check that we are not
generating such packfiles.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t3404: preserve test_tick state across short SHA-1 collision test

The short SHA-1 collision test requires carefully crafted commits in
order to ensure a collision at rebase time. This involves managing state
which impacts the resulting SHA-1, including commit time. To accomplish
this, test_tick is set to a known state for the short SHA-1 collision
test.

Unfortunately, doing so throws away the existing state of test_tick,
which may be problematic for subsequently added tests. Fix this by
preserving the existing state of test_tick across the short SHA-1
collision test.

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t3404: make tests more self-contained

As its very first action, t3404 installs (via set_fake_editor) a
specialized $EDITOR which simplifies automated 'rebase -i' testing. Many
tests rely upon this setting, thus tests which need a different editor
must take extra care upon completion to restore $EDITOR in order to
avoid breaking following tests. This places extra burden upon such tests
and requires that they undesirably have extra knowledge about
surrounding tests. Ease this burden by having each test install the
$EDITOR it requires, rather than relying upon a global setting.

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t3404: simplify short SHA-1 collision test

The short SHA-1 collision test employs two "magic" values in commit
messages to trigger an "ambiguous SHA-1 error" during 'rebase -i' -- one
for 'collide3' during setup, and one for 'collide2' at rebase time --
even though the collision can just as easily be triggered by a single
magic value at rebase time only. The setup-time 'collide3' magic value
serves no purpose [1], but can easily mislead readers into thinking that
it is somehow significant even though it is not. Drop this unneeded bit
of magic.

As a side-effect, this also simplifies the setup-time "test_commit
collide3" invocation, making it easier to read.

[1]: The magic value on 'collide3' gave it a short SHA-1 prefix of
"badbeef" which could be spotted easily in verbose output during
development of the test, but is otherwise not meaningful or helpful.

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-completion.bash: use correct Bash/Zsh array length syntax

The syntax for retrieving the number of elements in an array is:

   ${#name[@]}

Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t9902-completion.sh: old Bash still does not support array+=('') notation

Old Bash (3.0) which is distributed with RHEL 4.X and other ancient
platforms that are still in wide use, does not understand the
array+=() notation.  Let's use an explicit assignment to the new array
element which works everywhere, like:

   array[${#array[@]}+1]=''

The right-hand side '' is not strictly necessary, but in this case
I think it is more clear.

Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Revert "bash prompt: avoid command substitution when finalizing gitstring"

This reverts commit 69a8141a5d81925b7e08cb228535e9ea4a7a02e3.

Old Bash (3.0) which is distributed with RHEL 4.X and other ancient
platforms that are still in wide use, does not have a printf that
supports -v.  Let's revert this patch and go back to using printf
in the traditional way.

Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-send-email: add optional 'choices' parameter to the ask sub

Make it possible for callers of ask() to provide a list of
choices. Entering an appropriate integer chooses from that list,
otherwise the input is treated as usual.

Each choice can either be a single string, which is used both for the
prompt and for the return value, or a two-element array ref, where the
zeroth element is the choice and the first is used for the prompt.

Signed-off-by: Rasmus Villemoes <rv@rasmusvillemoes.dk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>