diff: do not use null sha1 as a sentinel value

The diff code represents paths using the diff_filespec
struct. This struct has a sha1 to represent the sha1 of the
content at that path, as well as a sha1_valid member which
indicates whether its sha1 field is actually useful. If
sha1_valid is not true, then the filespec represents a
working tree file (e.g., for the no-index case, or for when
the index is not up-to-date).

The diff_filespec is only used internally, though. At the
interfaces to the diff subsystem, callers feed the sha1
directly, and we create a diff_filespec from it. It's at
that point that we look at the sha1 and decide whether it is
valid or not; callers may pass the null sha1 as a sentinel
value to indicate that it is not.

We should not typically see the null sha1 coming from any
other source (e.g., in the index itself, or from a tree).
However, a corrupt tree might have a null sha1, which would
cause "diff --patch" to accidentally diff the working tree
version of a file instead of treating it as a blob.

This patch extends the edges of the diff interface to accept
a "sha1_valid" flag whenever we accept a sha1, and to use
that flag when creating a filespec. In some cases, this
means passing the flag through several layers, making the
code change larger than would be desirable.

One alternative would be to simply die() upon seeing
corrupted trees with null sha1s. However, this fix more
directly addresses the problem (while bogus sha1s in a tree
are probably a bad thing, it is really the sentinel
confusion sending us down the wrong code path that is what
makes it devastating). And it means that git is more capable
of examining and debugging these corrupted trees. For
example, you can still "diff --raw" such a tree to find out
when the bogus entry was introduced; you just cannot do a
"--patch" diff (just as you could not with any other
corrupted tree, as we do not have any content to diff).

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
do not write null sha1s to on-disk index

We should never need to write the null sha1 into an index
entry (short of the 1 in 2^160 chance that somebody actually
has content that hashes to it). If we attempt to do so, it
is much more likely that it is a bug, since we use the null
sha1 as a sentinel value to mean "not valid".

The presence of null sha1s in the index (which can come
from, among other things, "update-index --cacheinfo", or by
reading a corrupted tree) can cause problems for later
readers, because they cannot distinguish the literal null
sha1 from its use a sentinel value.  For example, "git
diff-files" on such an entry would make it appear as if it
is stat-dirty, and until recently, the diff code assumed
such an entry meant that we should be diffing a working tree
file rather than a blob.

Ideally, we would stop such entries from entering even our
in-core index. However, we do sometimes legitimately add
entries with null sha1s in order to represent these sentinel
situations; simply forbidding them in add_index_entry breaks
a lot of the existing code. However, we can at least make
sure that our in-core sentinel representation never makes it
to disk.

To be thorough, we will test an attempt to add both a blob
and a submodule entry. In the former case, we might run into
problems anyway because we will be missing the blob object.
But in the latter case, we do not enforce connectivity
across gitlink entries, making this our only point of
enforcement. The current implementation does not care which
type of entry we are seeing, but testing both cases helps
future-proof the test suite in case that changes.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
fsck: detect null sha1 in tree entries

Short of somebody happening to beat the 1 in 2^160 odds of
actually generating content that hashes to the null sha1, we
should never see this value in a tree entry. So let's have
fsck warn if it it seen.

As in the previous commit, we test both blob and submodule
entries to future-proof the test suite against the
implementation depending on connectivity to notice the
error.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
send-email: improve RFC2047 quote parsing

The RFC2047 unquoting, used to parse email addresses in From and Cc
headers, is broken in several ways:

* It erroneously substitutes ' ' for '_' in *the whole* header, even
  outside the quoted field. [Noticed by Christoph.]

* It is too liberal in its matching, and happily matches the start
  of one quoted chunk against the end of another, or even just
  something that looks like such an end. [Noticed by Junio.]

* It fundamentally cannot cope with encodings that are not a
  superset of ASCII, nor several (incompatible) encodings in the
  same header.

This patch fixes the first two by doing a more careful decoding of
the outer quoting (e.g. "=AB" to represent an octet whose value is
0xAB).  Fixing the fundamental issues is left for a future, more
intrusive, patch.

Noticed-by: Christoph Miebach <christoph.miebach@web.de>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
setup: clarify error messages for file/revisions ambiguity

The previous "Use '--' to separate filenames from revisions" may sound
obvious for an old-time Unix user, but does not make it clear how to use
this '--'. In addition to mentionning this '--', give an idea of what the
new command should look like.

Ideally, we could provide cut-and-paste ready commands based on the
command that just failed, but we have no easy access to argv[] in this
place of the code.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t/t5400: demonstrate breakage caused by informational message from prune

When receive-pack triggers 'git gc --auto' and 'git prune' is called to
remove a stale temporary object, 'git prune' prints an informational
message to stdout about the file that it will remove.  Since this message
is written to stdout, it is sent back over the transport channel to the git
client which tries to interpret it as part of the pack protocol and then
promptly terminates with a complaint about a protocol error.

Introduce a test which exercises the auto-gc functionality of receive-pack
and demonstrates this breakage.

Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
receive-pack: do not leak output from auto-gc to standard output

The standard output channel of receive-pack is a structured protocol
channel, and subprocesses must never be allowed to leak anything
into it by writing to their standard output.

Use RUN_COMMAND_STDOUT_TO_STDERR option to run_command_v_opt() just
like we do when running hooks to prevent output from "gc" leaking to
the standard output.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
prune.c: only print informational message in show_only or verbose mode

"git prune" reports removal of loose object files that are no longer
necessary only under the "-v" option, but unconditionally reports
removal of temporary files that are no longer needed.

The original thinking was that the presence of a leftover temporary
file should be an unusual occurrence that may indicate an earlier
failure of some sort, and the user may want to be reminded of it.
Removing an unnecessary loose object file, on the other hand, is
just part of the normal operation.  That is why the former is always
printed out and the latter only when -v is used.

But neither report is particularly useful.  Hide both of these
behind the "-v" option for consistency.

Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
sh-setup: protect from exported IFS

Many scripted Porcelains rely on being able to split words at the
default $IFS characters, i.e. SP, HT and LF.  If the user exports a
non-default IFS to the environment, what they read from plumbing
commands such as ls-files that use HT to delimit fields may not be
split in the way we expect.

Protect outselves by resetting it, just like we do so against CDPATH
exported to the environment.

Noticed by Andrew Dranse <adranse@oanda.com>.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
rebase -i: use full onto sha1 in reflog

'git rebase' uses the full onto sha1 for the reflog message whereas 'git
rebase -i' uses the short sha1. This is not only inconsistent, but can
lead to problems when the reflog is inspected at a later time at which
that abbreviation may have become ambiguous.

Make 'rebase -i' use the full onto sha1, as well.

Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
gitweb: URL-decode $my_url/$my_uri when stripping PATH_INFO

When gitweb is used as a DirectoryIndex, it attempts to strip
PATH_INFO on its own, as $cgi->url() fails to do so.

However, it fails to account for the fact that PATH_INFO has
already been URL-decoded by the web server, but the value
returned by $cgi->url() has not been. This causes the stripping
to fail whenever the URL contains encoded characters.

To see this in action, setup gitweb as a DirectoryIndex and
then use it on a repository with a directory containing a
space in the name. Navigate to tree view, examine the gitweb
generated html and you'll see a link such as:

  <a href="/test.git/tree/HEAD:/directory with spaces">directory with spaces</a>

When clicked on, the browser will URL-encode this link, giving
a $cgi->url() of the form:

   /test.git/tree/HEAD:/directory%20with%20spaces

While PATH_INFO is:

   /test.git/tree/HEAD:/directory with spaces

Fix this by calling unescape() on both $my_url and $my_uri before
stripping PATH_INFO from them.

Signed-off-by: Jay Soffian <jaysoffian@gmail.com>
Acked-by: Jakub Narebski <jnareb@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'js/gitweb-path-info-unquote' into maint-1.7.11

"gitweb" when used with PATH_INFO failed to notice directories with
SP (and other characters that need URL-style quoting) in them.

* js/gitweb-path-info-unquote:
  gitweb: URL-decode $my_url/$my_uri when stripping PATH_INFO
Merge branch 'mm/die-with-dashdash-help' into maint-1.7.11

When the user gives an argument that can be taken as both a
revision name and a pathname without disambiguating with "--", we
used to give a help message "Use '--' to separate".  The message
has been clarified to show where that '--' goes on the command
line.

* mm/die-with-dashdash-help:
  setup: clarify error messages for file/revisions ambiguity
Merge branch 'tr/maint-send-email-2047' into maint-1.7.11

"git send-email" did not unquote encoded words that appear on the
header correctly, and lost "_" from strings.

* tr/maint-send-email-2047:
  send-email: improve RFC2047 quote parsing
Merge branch 'jk/maint-null-in-trees' into maint-1.7.11

"git diff" had a confusion between taking data from a path in the
working tree and taking data from an object that happens to have
name 0{40} recorded in a tree.

* jk/maint-null-in-trees:
  fsck: detect null sha1 in tree entries
  do not write null sha1s to on-disk index
  diff: do not use null sha1 as a sentinel value
Merge branch 'bc/receive-pack-stdout-protection' into maint-1.7.11

When "git push" triggered the automatic gc on the receiving end, a
message from "git prune" that said it was removing cruft leaked to
the standard output, breaking the communication protocol.

* bc/receive-pack-stdout-protection:
  receive-pack: do not leak output from auto-gc to standard output
  t/t5400: demonstrate breakage caused by informational message from prune
Merge branch 'jc/maint-protect-sh-from-ifs' into maint-1.7.11

When the user exports a non-default IFS without HT, scripts that
rely on being able to parse "ls-files -s | while read a b c..."
start to fail.  Protect them from such a misconfiguration.

* jc/maint-protect-sh-from-ifs:
  sh-setup: protect from exported IFS
Merge branch 'mg/rebase-i-onto-reflog-in-full' into maint-1.7.11

The reflog entries left by "git rebase" and "git rebase -i" were
inconsistent (the interactive one gave an abbreviated object name).

* mg/rebase-i-onto-reflog-in-full:
  rebase -i: use full onto sha1 in reflog
Almost 1.7.11.6

Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'maint-1.7.11' into maint

* maint-1.7.11:
  Almost 1.7.11.6
  gitweb: URL-decode $my_url/$my_uri when stripping PATH_INFO
  rebase -i: use full onto sha1 in reflog
  sh-setup: protect from exported IFS
  receive-pack: do not leak output from auto-gc to standard output
  t/t5400: demonstrate breakage caused by informational message from prune
  setup: clarify error messages for file/revisions ambiguity
  send-email: improve RFC2047 quote parsing
  fsck: detect null sha1 in tree entries
  do not write null sha1s to on-disk index
  diff: do not use null sha1 as a sentinel value