stream_blob_to_fd: detect errors reading from stream

We call read_istream, but never check its return value for
errors. This can lead to us looping infinitely, as we just
keep trying to write "-1" bytes (and we do not notice the
error, as we simply check that write_in_full reports the
same number of bytes we fed it, which of course is also -1).

Signed-off-by: Jeff King <peff@peff.net>
check_sha1_signature: check return value from read_istream

It's possible for read_istream to return an error, in which
case we just end up in an infinite loop (aside from EOF, we
do not even look at the result, but just feed it straight
into our running hash).

Signed-off-by: Jeff King <peff@peff.net>
read_istream_filtered: propagate read error from upstream

The filter istream pulls data from an "upstream" stream,
running it through a filter function. However, we did not
properly notice when the upstream filter yielded an error,
and just returned what we had read. Instead, we should
propagate the error.

Signed-off-by: Jeff King <peff@peff.net>
avoid infinite loop in read_istream_loose

The read_istream_loose function loops on inflating a chunk of data
from an mmap'd loose object. We end the loop when we run out
of space in our output buffer, or if we see a zlib error.

We need to treat Z_BUF_ERROR specially, though, as it is not
fatal; it is just zlib's way of telling us that we need to
either feed it more input or give it more output space. It
is perfectly normal for us to hit this when we are at the
end of our buffer.

However, we may also get Z_BUF_ERROR because we have run out
of input. In a well-formed object, this should not happen,
because we have fed the whole mmap'd contents to zlib. But
if the object is truncated or corrupt, we will loop forever,
never giving zlib any more data, but continuing to ask it to
inflate.

We can fix this by considering it an error when zlib returns
Z_BUF_ERROR but we still have output space left (which means
it must want more input, which we know is a truncation
error). It would not be sufficient to just check whether
zlib had consumed all the input at the start of the loop, as
it might still want to generate output from what is in its
internal state.

Signed-off-by: Jeff King <peff@peff.net>
add test for streaming corrupt blobs

We do not have many tests for handling corrupt objects. This
new test at least checks that we detect a byte error in a
corrupt blob object while streaming it out with cat-file.

Signed-off-by: Jeff King <peff@peff.net>
streaming_write_entry: propagate streaming errors

On Mon, Mar 25, 2013 at 02:39:34PM -0700, Jonathan Nieder wrote:

> > --- a/entry.c
> > +++ b/entry.c
> > @@ -126,8 +126,10 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,
> >  	fd = open_output_fd(path, ce, to_tempfile);
> >  	if (0 <= fd) {
> >  		result = stream_blob_to_fd(fd, ce->sha1, filter, 1);
> > -		*fstat_done = fstat_output(fd, state, statbuf);
> > -		result = close(fd);
> > +		if (!result) {
> > +			*fstat_done = fstat_output(fd, state, statbuf);
> > +			result = close(fd);
> > +		}
>
> Should this do something like
> [...]
> to avoid leaking the file descriptor?

Yes, Eric Sunshine noticed this, too. Re-rolled patch is below, which I
think is even a little cleaner.

> > +test_expect_success 'read-tree -u detects bit-errors in blobs' '
> > +	(
> > +		cd bit-error &&
> > +		rm content.t &&
> > +		test_must_fail git read-tree --reset -u FETCH_HEAD
> > +	)
>
> Makes sense.  Might make sense to use "rm -f" instead of "rm" to avoid
> failures if content.t is removed already.

Yeah, good point. My original test looked like:

  git init bit-error &&
  git fetch .. &&
  corrupt ...
  test_must_fail ...

but I ended up refactoring it to re-use the corrupted directories, and
added the "rm" after the fact. The use of FETCH_HEAD is also bogus
(read-tree is failing, but because we are giving it a bogus ref, not
because of the corruption, so we are not actually testing anything
anymore, even though it still passes).

Both fixed in my re-roll.

-- >8 --
Subject: [PATCH] streaming_write_entry: propagate streaming errors

When we are streaming an index blob to disk, we store the
error from stream_blob_to_fd in the "result" variable, and
then immediately overwrite that with the return value of
"close". That means we catch errors on close (e.g., problems
committing the file to disk), but miss anything which
happened before then.

We can fix this by using bitwise-OR to accumulate errors in
our result variable.

While we're here, we can also simplify the error handling
with an early return, which makes it easier to see under
which circumstances we need to clean up.

Signed-off-by: Jeff King <peff@peff.net>
add tests for cloning corrupted repositories

We try not to let corruption pass unnoticed over fetches and
clones. For the most part, this works, but there are some
broken corner cases, including:

  1. We do not detect missing objects over git-aware
     transports. This is a little hard to test, because the
     sending side will actually complain about the missing
     object.

     To fool it, we corrupt a repository such that we have a
     "misnamed" object: it claims to be sha1 X, but is
     really Y. This lets the sender blindly transmit it, but
     it is the receiver's responsibility to verify that what
     it got is sane (and it does not).

  2. We do not detect missing or misnamed blobs during the
     checkout phase of clone.

Signed-off-by: Jeff King <peff@peff.net>
clone: die on errors from unpack_trees

When clone is populating the working tree, it ignores the
return status from unpack_trees; this means we may report a
successful clone, even when the checkout fails.

When checkout fails, we may want to leave the $GIT_DIR in
place, as it might be possible to recover the data through
further use of "git checkout" (e.g., if the checkout failed
due to a transient error, disk full, etc). However, we
already die on a number of other checkout-related errors, so
this patch follows that pattern.

In addition to marking a now-passing test, we need to adjust
t5710, which blindly assumed it could make bogus clones of
very deep alternates hierarchies. By using "--bare", we can
avoid it actually touching any objects.

Signed-off-by: Jeff King <peff@peff.net>
clone: run check_everything_connected

When we fetch from a remote, we do a revision walk to make
sure that what we received is connected to our existing
history. We do not do the same check for clone, which should
be able to check that we received an intact history graph.

The upside of this patch is that it will make clone more
resilient against propagating repository corruption. The
downside is that we will now traverse "rev-list --objects
--all" down to the roots, which may take some time (it is
especially noticeable for a "--local --bare" clone).

Note that we need to adjust t5710, which tries to make such
a bogus clone. Rather than checking after the fact that our
clone is bogus, we can simplify it to just make sure "git
clone" reports failure.

Signed-off-by: Jeff King <peff@peff.net>
sha1_file: remove recursion in packed_object_info

packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.

This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.

The task of the two functions is not all that hard to describe without
any recursion, however.  It proceeds in three steps:

- determine the representation type and size, based on the outermost
  object (delta or not)

- follow through the delta chain, if any

- determine the object type from what is found at the end of the delta
  chain

The only complication stems from the error recovery.  If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere.  If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object.  (This is conveniently checked by t5303.)

To achieve that within the pack, we keep track of the entire delta
chain in a stack.  When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1.  To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array.  The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().

It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that.  If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.

Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Refactor parts of in_delta_base_cache/cache_or_unpack_entry

The delta base cache lookup and test were shared.  Refactor them;
we'll need both parts again.  Also, we'll use the clearing routine
later.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
sha1_file: remove recursion in unpack_entry

Similar to the recursion in packed_object_info(), this leads to
problems on stack-space-constrained systems in the presence of long
delta chains.

We proceed in three phases:

1. Dig through the delta chain, saving each delta object's offsets and
   size on an ad-hoc stack.

2. Unpack the base object at the bottom.

3. Apply the deltas from the stack.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'tr/packed-object-info-wo-recursion' into jch

Attempts to reduce the stack footprint of sha1_object_info()
codepath.

* tr/packed-object-info-wo-recursion:
  sha1_file: remove recursion in unpack_entry
  Refactor parts of in_delta_base_cache/cache_or_unpack_entry
  sha1_file: remove recursion in packed_object_info
Merge branch 'sr/am-show-final-message-in-applying-indicator' into jch

* sr/am-show-final-message-in-applying-indicator:
  git-am: show the final log message on "Applying:" indicator
Merge branch 'yd/doc-merge-annotated-tag' into jch

* yd/doc-merge-annotated-tag:
  Documentation: merging a tag is a special case
Merge branch 'yd/doc-is-in-asciidoc' into jch

* yd/doc-is-in-asciidoc:
  CodingGuidelines: our documents are in AsciiDoc
Merge branch 'jc/apply-ws-fix-tab-in-indent' into jch

* jc/apply-ws-fix-tab-in-indent:
  apply --whitespace=fix: avoid running over the postimage buffer
Merge branch 'jk/no-more-self-assignment' into jch

* jk/no-more-self-assignment:
  match-trees: simplify score_trees() using tree_entry()
  submodule: clarify logic in show_submodule_summary
Merge branch 'kk/revwalk-slop-too-many-commit-within-a-second' into jch

* kk/revwalk-slop-too-many-commit-within-a-second:
  Fix revision walk for commits with the same dates
Merge branch 'yd/use-test-config-unconfig' into jch

* yd/use-test-config-unconfig:
  t7600: use test_config to set/unset git config variables
  t7502: remove clear_config
  t7502: use test_config to set/unset git config variables
  t9500: use test_config to set/unset git config variables
  t7508: use test_config to set/unset git config variables
  t7500: use test_config to set/unset git config variables
  t5541: use test_config to set/unset git config variables
  t5520: use test_config to set/unset git config variables
  t4202: use test_config/test_unconfig to set/unset git config variables
  t4034: use test_config/test_unconfig to set/unset git config variables
  t4304: use test_config to set/unset git config variables
  t3400: use test_config to set/unset git config variables