log --graph --stat: three-dash separator should come after graph lines

Output from "git log --graph --stat -p" emits the three-dash separator
line before the graph that shows ancestry lines.  The separator should
come after the ancestry lines just like all the other output.

Signed-off-by: Lucian Poston <lucian.poston@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
log --graph: fix break in graph lines

Output from "git log --graph --stat -p" broke the ancestry graph lines
with a single empty line between the diffstat and the patch.

Signed-off-by: Lucian Poston <lucian.poston@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t4202: add test for "log --graph --stat -p" separator lines

Add tests to make sure that the three-dash separator lines appear
after the graph ancestry lines, and also the graph ancestry lines
are not broken between the diffstat and the patch.

Signed-off-by: Lucian Poston <lucian.poston@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
fetch-pack: new --stdin option to read refs from stdin

If a remote repo has too many tags (or branches), cloning it over the
smart HTTP transport can fail because remote-curl.c puts all the refs
from the remote repo on the fetch-pack command line. This can make the
command line longer than the global OS command line limit, causing
fetch-pack to fail.

This is especially a problem on Windows where the command line limit is
orders of magnitude shorter than Linux. There are already real repos out
there that msysGit cannot clone over smart HTTP due to this problem.

Here is an easy way to trigger this problem:

	git init too-many-refs
	cd too-many-refs
	echo bla > bla.txt
	git add .
	git commit -m test
	sha=$(git rev-parse HEAD)
	tag=$(perl -e 'print "bla" x 30')
	for i in `seq 50000`; do
		echo $sha refs/tags/$tag-$i >> .git/packed-refs
	done

Then share this repo over the smart HTTP protocol and try cloning it:

	$ git clone http://localhost/.../too-many-refs/.git
	Cloning into 'too-many-refs'...
	fatal: cannot exec 'fetch-pack': Argument list too long

50k tags is obviously an absurd number, but it is required to
demonstrate the problem on Linux because it has a much more generous
command line limit. On Windows the clone fails with as little as 500
tags in the above loop, which is getting uncomfortably close to the
number of tags you might see in real long lived repos.

This is not just theoretical, msysGit is already failing to clone our
company repo due to this. It's a large repo converted from CVS, nearly
10 years of history.

Four possible solutions were discussed on the Git mailing list (in no
particular order):

1) Call fetch-pack multiple times with smaller batches of refs.

This was dismissed as inefficient and inelegant.

2) Add option --refs-fd=$n to pass a an fd from where to read the refs.

This was rejected because inheriting descriptors other than
stdin/stdout/stderr through exec() is apparently problematic on Windows,
plus it would require changes to the run-command API to open extra
pipes.

3) Add option --refs-from=$tmpfile to pass the refs using a temp file.

This was not favored because of the temp file requirement.

4) Add option --stdin to pass the refs on stdin, one per line.

In the end this option was chosen as the most efficient and most
desirable from scripting perspective.

There was however a small complication when using stdin to pass refs to
fetch-pack. The --stateless-rpc option to fetch-pack also uses stdin for
communication with the remote server.

If we are going to sneak refs on stdin line by line, it would have to be
done very carefully in the presence of --stateless-rpc, because when
reading refs line by line we might read ahead too much data into our
buffer and eat some of the remote protocol data which is also coming on
stdin.

One way to solve this would be to refactor get_remote_heads() in
fetch-pack.c to accept a residual buffer from our stdin line parsing
above, but this function is used in several places so other callers
would be burdened by this residual buffer interface even when most of
them don't need it.

In the end we settled on the following solution:

If --stdin is specified without --stateless-rpc, fetch-pack would read
the refs from stdin one per line, in a script friendly format.

However if --stdin is specified together with --stateless-rpc,
fetch-pack would read the refs from stdin in packetized format
(pkt-line) with a flush packet terminating the list of refs. This way we
can read the exact number of bytes that we need from stdin, and then
get_remote_heads() can continue reading from the same fd without losing
a single byte of remote protocol data.

This way the --stdin option only loses generality and scriptability when
used together with --stateless-rpc, which is not easily scriptable
anyway because it also uses pkt-line when talking to the remote server.

Signed-off-by: Ivan Todoroski <grnch@gmx.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
rebase -i continue: don't skip commits that only change submodules

When git-rebase--interactive stops due to a conflict and the only change
to be committed is in a submodule, the test for whether there is
anything to be committed ignores the staged submodule change.  This
leads rebase to skip creating the commit for the change.

While unstaged submodule changes should be ignored to avoid needing to
update submodules during a rebase, it is safe to remove the
--ignore-submodules option to diff-index because --cached ensures that
it is only checking the index.  This was discussed in [1] and a test is
included to ensure that unstaged changes are still ignored correctly.

[1] http://thread.gmane.org/gmane.comp.version-control.git/188713

Signed-off-by: John Keeping <john@keeping.me.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
fast-import: tighten parsing of datarefs

The syntax for the use of mark references in fast-import
demands either a SP (space) or LF (end-of-line) after
a mark reference.  Fast-import does not complain when garbage
appears after a mark reference in some cases.

Factor out parsing of mark references and complain if
errant characters are found.  Also be a little more careful
when parsing "inline" and SHA1s, complaining if extra
characters appear or if the form of the dataref is unrecognized.

Buggy input can cause fast-import to produce the wrong output,
silently, without error.  This makes it difficult to track
down buggy generators of fast-import streams.  An example is
seen in the last line of this commit command:

    commit refs/heads/S2
    committer Name <name@example.com> 1112912893 -0400
    data <<COMMIT
    commit message
    COMMIT
    from :1M 100644 :103 hello.c

It is missing a newline and should be:

    [...]
    from :1
    M 100644 :103 hello.c

What fast-import does is to produce a commit with the same
contents for hello.c as in refs/heads/S2^.  What the buggy
program was expecting was the contents of blob :103.  While
the resulting commit graph looked correct, the contents in
some commits were wrong.

Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
remote-curl: send the refs to fetch-pack on stdin

Now that we can throw an arbitrary number of refs at fetch-pack using
its --stdin option, we use it in the remote-curl helper to bypass the
OS command line length limit.

Signed-off-by: Ivan Todoroski <grnch@gmx.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
fetch-pack: test cases for the new --stdin option

These test cases focus only on testing the parsing of refs on stdin,
without bothering with the rest of the fetch-pack machinery. We pass in
the refs using different combinations of command line and stdin and then
we watch fetch-pack's stdout to see whether it prints all the refs we
specified (but we ignore their order).

Signed-off-by: Ivan Todoroski <grnch@gmx.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
remote-curl: main test case for the OS command line overflow

This is main test case for the original problem that triggered this
patch series. We create a repo with 50k tags and then test whether
git-clone over the smart HTTP protocol succeeds.

Note that we construct the repo in a slightly different way than the
original script used to reproduce the problem. This is because the
original script just created 50k tags all pointing to the same commit,
so if there was a bug where remote-curl.c was not passing all the refs
to fetch-pack we wouldn't know. The clone would succeed even if only one
tag was passed, because all the other tags were pointing at the same SHA
and would be considered present.

Instead we create a repo with 50k independent (dangling) commits and
then tag each of those commits with a unique tag. This way if one of the
tags is not given to fetch-pack, later stages of the clone would
complain about it.

This allows us to test both that the command line overflow was fixed, as
well as that it was fixed in a way that doesn't leave out any of the
refs.

Signed-off-by: Ivan Todoroski <grnch@gmx.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
blame: accept --need-minimal

Between v1.7.1 and v1.7.2, 582aa00bdffb switched the default "diff"
invocation not to use XDF_NEED_MINIMAL, but this breaks "git blame"
rather badly.

Allow the command line option to ask for an extra careful matching.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
submodules: recursive fetch also checks new tags for submodule commits

Since 88a21979c (fetch/pull: recurse into submodules when necessary) all
fetched commits are examined if they contain submodule changes (unless
configuration or command line options inhibit that). If a newly recorded
submodule commit is not present in the submodule, a fetch is run inside
it to download that commit.

Checking new refs was done in an else branch where it wasn't executed for
tags. This normally isn't a problem because tags are only fetched with
the branches they live on, then checking the new commits in the fetched
branches for submodule commits will also process all tags. But when a
specific tag is fetched (or the refspec contains refs/tags/) commits only
reachable by tags won't be searched for submodule commits, which is a bug.

Fix that by moving the code outside the if/else construct to handle new
tags just like any other ref. The performance impact of adding tags that
most of the time lie on a branch which is checked anyway for new submodule
commit should be minimal, as since 6859de4 (fetch: avoid quadratic loop
checking for updated submodules) all ref-tips are collected first and then
fed to a single rev-list.

Spotted-by: Jeff King <peff@peff.net>
Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'maint' of https://github.com/git-l10n/git-po

By Byrial Jensen (2) and others
via Jiang Xin (1) and Ralf Thielow (1)
* 'maint' of https://github.com/git-l10n/git-po:
  l10n: Initial German translation
  l10n: Update Simplified Chinese translation
  l10n: Update git.pot (2 new messages)
  l10n: Add the German translation team and initialize de.po
  l10n: Add Danish team (da) to list of teams
  l10n: New da.po file with 0 translations
  l10n: Updated pt_PT language
Merge branch 'jk/rebase-i-submodule-conflict-only' into maint

Giving "--continue" to a conflicted "rebase -i" session skipped a
commit that only results in changes to submodules.

By John Keeping
* jk/rebase-i-submodule-conflict-only:
  rebase -i continue: don't skip commits that only change submodules
Merge branch 'lp/maint-diff-three-dash-with-graph' into maint

"log -p --graph" used with "--stat" had a few formatting error.

By Lucian Poston
* lp/maint-diff-three-dash-with-graph:
  t4202: add test for "log --graph --stat -p" separator lines
  log --graph: fix break in graph lines
  log --graph --stat: three-dash separator should come after graph lines
Merge branch 'jc/maint-blame-minimal' into maint

"git blame" started missing quite a few changes from the origin since we
stopped using the diff minimalization by default in v1.7.2 era.

Teach "--minimal" option to "git blame" to work around this regression.

* jc/maint-blame-minimal:
  blame: accept --need-minimal
Merge branch 'jl/maint-submodule-recurse-fetch' into maint

"git fetch" that recurses into submodules on demand did not check if it
needs to go into submodules when non branches (most notably, tags) are
fetched.

By Jens Lehmann
* jl/maint-submodule-recurse-fetch:
  submodules: recursive fetch also checks new tags for submodule commits
Merge branch 'it/fetch-pack-many-refs' into maint

When "git fetch" encounters repositories with too many references, the
command line of "fetch-pack" that is run by a helper e.g. remote-curl, may
fail to hold all of them. Now such an internal invocation can feed the
references through the standard input of "fetch-pack".

By Ivan Todoroski
* it/fetch-pack-many-refs:
  remote-curl: main test case for the OS command line overflow
  fetch-pack: test cases for the new --stdin option
  remote-curl: send the refs to fetch-pack on stdin
  fetch-pack: new --stdin option to read refs from stdin

Conflicts:
	t/t5500-fetch-pack.sh
Merge branch 'pw/fast-import-dataref-parsing' into maint

The parser in "fast-import" did not diagnose ":9" style references that is
not followed by required SP/LF as an error.

By Pete Wyckoff
* pw/fast-import-dataref-parsing:
  fast-import: tighten parsing of datarefs
Git 1.7.10.1

Signed-off-by: Junio C Hamano <gitster@pobox.com>