packfile: use varint functions

The OFS_DELTA codepaths can use encode/decode_varint() to read and write
variable-length integers in the pack stream.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
new representation types in the packstream

In addition to the four basic types (commit, tree, blob and tag), the pack
stream can encode a few other "representation" types, such as REF_DELTA
and OFS_DELTA.  As we allocate 3 bits in the first byte for this purpose,
we do not have much room to add new representation types in place, but we
do have one value reserved for future expansion.

When encoding a new representation type, the early part of the in-pack
object header is encoded as if its type is OBJ_EXT (= 5).  The lower 4-bit
of the first byte is used for the lowest 4-bit of the size information,
the next 3-bit has the type information, and the MSB says if the
subsequent bytes need to be read to decode higher bits for the size
information.

An in-pack object header that records OBJ_EXT as its type is followed by
an integer in the same variable-length encoding as OFS_DELTA offset is
encoded in.  This value is the real type of the representation minus 8 (as
we do not need to use OBJ_EXT to encode types smaller than 8).  Because we
do not foresee very many representation types, in practice we would have a
single byte with its MSB clear, to represent types 8-135.

The code does not add nor use type=8 and upwards for anything yet.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
bulk-checkin: allow the same data to be multiply hashed

This updates stream_to_pack() machinery to feed the data it is writing out
to multiple hash contexts at the same time. Right now we only use a single
git_SHA_CTX, so there is no change in functionality.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
bulk-checkin: support chunked-object encoding

Instead of recording a huge object as a single entry in the packfile,
record it as a concatenation of smaller blob objects. This is primarily to
make it possible to repack repositories with huge objects and transfer
them out of such repositories.

This is the first step of a long and painful journey. We would still need
to teach many codepaths about the new encoding:

 * The streaming checkout codepath must learn how to read these from the
   object store and write a concatenation of component blob contents to
   the working tree. Note that after a repack, a component blob object
   could be represented as a delta to another blob object.

 * The in-core object reading codepath must learn to notice and at least
   reject reading such objects entirely in-core. It is expected that
   nobody is interested in producing a patch out of these huge objects, at
   least initially.

 * The object connectivity machinery must learn that component blob
   objects are reachable from an object that uses them, so that "gc"
   will not lose them, and "fsck" will not complain about them.

 * The pack-object machinery must learn to copy an object that is encoded
   in chunked-object encoding as-is to the destination, while perhaps
   validating the object name, without having to first come up with a huge
   concatenated blob and then split it again into smaller pieces. It also
   must learn that the component blobs need to be packed when it does so.

 * The index-pack and verify-pack machineries need to be told about the
   objects in the chunked encoding.

The split-chunk logic used here is kept deliberately useless in order to
avoid distracting the reviewers.  We will want to replace it with a better
heuristics---the rest of the system is designed not to care, so that the
heuristics can be improved without any compatibility issues.

One possibility is to use the heuristics "bup" uses, which is based on a
self-synchronizing rolling checksum logic; if it works as advertised, we
should be able to obtain a good reuse of component blobs when only small
parts of a large blob in chunked encoding are updated.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
chunked-object: fallback checkout codepaths

This prepares the default codepaths based on the traditional "slurping
everything in-core" model around read_sha1_file() API for objects that use
chunked encoding. Needless to say, these codepaths are unsuitable for the
kind of objects that use chunked encoding and are intended to only serve
as the fallback where specialized "large object API" support is still
lacking.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
chunked-object: streaming checkout

Define a new streaming decoder to handle blobs stored in chunked encoding,
so that we do not have to slurp the data for the whole object to check it
out to the working tree.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
fetch/push: allow refs/*:refs/*

There are a handful of places where we call check_refname_format() on a
substring after "refs/" of a refname we are going to use, and filter out a
valid match with "refs/stash" with such a pathspec.  Not sending a stash
may arguably be a feature (as stash is inherently a local workflow
element), but the code in the transport layer is oblivious to this
filtering performed by the lower layer of the code, and complains that the
other side did not send all the objects that needs to complete refs/stash
at the end, even though the code will not write refs/stash out anyway, and
making the whole command fail.

This is an attempt to "fix" it by using check_refname_format() on the
whole "refs/....." string and allowing refs/stash to be also copied.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
get_fetch_map(): tighten checks on dest refs

The code to check the refname we store the fetched result locally did not
bother checking the first 5 bytes of it, presumably assuming that it
always begin with "refs/".  For a fetch refspec (or the result of applying
wildcard on one), we always want the RHS to map to something inside
"refs/" hierarchy, so let's spell that rule out in a more explicit way.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
vcs-svn: fix clang-analyzer error

vcs-svn/svndiff.c:298:3: warning: Assigned value is garbage or undefined
                off_t pre_off = pre_off; /* stupid GCC... */
                ^               ~~~~~~~

Signed-off-by: David Barr <davidbarr@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
vcs-svn: simplify cleanup in apply_one_window()

As a side-effect, fix clang-analyzer warning:

vcs-svn/svndiff.c:278:3: warning: expression result unused [-Wunused-value]
                error("invalid delta: incorrect postimage length");
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This warning caused by an insanely concise error() upstream.

Signed-off-by: David Barr <davidbarr@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
vcs-svn: prefer constcmp to prefixcmp

Comparisons in svndump.c are always guarded by length.
As a bonus, elimate dependency on prefixcmp for upstream.

Signed-off-by: David Barr <davidbarr@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
vcs-svn: prefer strstr over memmem

The common pattern is to use strstr to match a fixed needle.
As a bonus, elimate dependency on memmem for upstream.

Signed-off-by: David Barr <davidbarr@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
vcs-svn: fix signedness warnings

Stop mixing off_t and size_t.

Signed-off-by: David Barr <davidbarr@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
vcs-svn: drop no-op reset methods

Since v1.7.5~42^2~6 (vcs-svn: remove buffer_read_string)
buffer_reset() does nothing thus fast_export_reset() also.

Signed-off-by: David Barr <davidbarr@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
pager: drop "wait for output to run less" hack

Commit 35ce862 (pager: Work around window resizing bug in
'less', 2007-01-24) causes git's pager sub-process to wait
to receive input after forking but before exec-ing the
pager. To handle this, run-command had to grow a "pre-exec
callback" feature. Unfortunately, this feature does not work
at all on Windows (where we do not fork), and interacts
poorly with run-command's parent notification system. Its
use should be discouraged.

The bug in less was fixed in version 406, which was released
in June 2007. It is probably safe at this point to remove
our workaround. That lets us rip out the preexec_cb feature
entirely.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t: Replace 'perl' by $PERL_PATH

GIT-BUILD-OPTIONS defines PERL_PATH to be used in the test suite. Only a
few tests already actually use this variable when perl is needed. The
other test just call 'perl' and it might happen that the wrong perl
interpreter is used.

This becomes problematic on Windows, when the perl interpreter that is
compiled and installed on the Windows system is used, because this perl
interpreter might introduce some unexpected LF->CRLF conversions.

This patch makes sure that $PERL_PATH is used everywhere in the test suite
and that the correct perl interpreter is used.

Signed-off-by: Vincent van Ravesteijn <vfr@lyx.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
apply: fix an incomplete comment in check_patch()

This check is not only about type-change (for which it would be
sufficient to check only was_deleted()) but is also about a swap
rename.  Otherwise to_be_deleted() check is not justified.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
apply: a bit more comments on PATH_TO_BE_DELETED

The code is littered with to_be_deleted() whose purpose is not so clear.
Describe where it matters.  Also remove an extra space before "#define"
that snuck in by mistake at 7fac0ee (builtin-apply: keep information about
files to be deleted, 2009-04-11).

Signed-off-by: Junio C Hamano <gitster@pobox.com>
apply: clear_image() clears things a bit more

The clear_image() function did not clear the line table in the image
structure; this does not matter for the current callers, as the function
is only called from the codepaths that deal with binary patches where the
line table is never populated, and the codepaths that do populate the line
table free it themselves.

But it will start to matter when we introduce a codepath to retry a failed
patch, so make sure it clears and frees everything.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
apply: refactor read_file_or_gitlink()

Reading a blob out of the object store does not have to require that the
caller has a cache entry for it.

Create a read_blob_object() helper function that takes the object name and
mode, and use it to reimplement the original function as a thin wrapper to
it.

Signed-off-by: Junio C Hamano <gitster@pobox.com>