varint-in-pack: refactor varint encoding/decoding

Refactor encode/decode_in_pack_varint() functions from OFS_DELTA codepaths
to read and write variable-length integers in the pack stream.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
new representation types in the packstream

In addition to the four basic types (commit, tree, blob and tag), the pack
stream can encode a few other "representation" types, such as REF_DELTA
and OFS_DELTA.  As we allocate 3 bits in the first byte for this purpose,
we do not have much room to add new representation types in place, but we
do have one value reserved for future expansion.

When encoding a new representation type, the early part of the in-pack
object header is encoded as if its type is OBJ_EXT (= 5).  The lower 4-bit
of the first byte is used for the lowest 4-bit of the size information,
the next 3-bit has the type information, and the MSB says if the
subsequent bytes need to be read to decode higher bits for the size
information.

An in-pack object header that records OBJ_EXT as its type is followed by
an integer in the same variable-length encoding as OFS_DELTA offset is
encoded in.  This value is the real type of the representation minus 8 (as
we do not need to use OBJ_EXT to encode types smaller than 8).  Because we
do not foresee very many representation types, in practice we would have a
single byte with its MSB clear, to represent types 8-135.

The code does not add nor use type=8 and upwards for anything yet.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
bulk-checkin: allow the same data to be multiply hashed

This updates stream_to_pack() machinery to feed the data it is writing out
to multiple hash contexts at the same time. Right now we only use a single
git_SHA_CTX, so there is no change in functionality.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
bulk-checkin: support chunked-object encoding

Instead of recording a huge object as a single entry in the packfile,
record it as a concatenation of smaller blob objects. This is primarily to
make it possible to repack repositories with huge objects and transfer
them out of such repositories.

This is the first step of a long and painful journey. We would still need
to teach many codepaths about the new encoding:

 * The streaming checkout codepath must learn how to read these from the
   object store and write a concatenation of component blob contents to
   the working tree. Note that after a repack, a component blob object
   could be represented as a delta to another blob object.

 * The in-core object reading codepath must learn to notice and at least
   reject reading such objects entirely in-core. It is expected that
   nobody is interested in producing a patch out of these huge objects, at
   least initially.

 * The object connectivity machinery must learn that component blob
   objects are reachable from an object that uses them, so that "gc"
   will not lose them, and "fsck" will not complain about them.

 * The pack-object machinery must learn to copy an object that is encoded
   in chunked-object encoding as-is to the destination, while perhaps
   validating the object name, without having to first come up with a huge
   concatenated blob and then split it again into smaller pieces. It also
   must learn that the component blobs need to be packed when it does so.

 * The index-pack and verify-pack machineries need to be told about the
   objects in the chunked encoding.

The split-chunk logic used here is kept deliberately useless in order to
avoid distracting the reviewers.  We will want to replace it with a better
heuristics---the rest of the system is designed not to care, so that the
heuristics can be improved without any compatibility issues.

One possibility is to use the heuristics "bup" uses, which is based on a
self-synchronizing rolling checksum logic; if it works as advertised, we
should be able to obtain a good reuse of component blobs when only small
parts of a large blob in chunked encoding are updated.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
chunked-object: fallback checkout codepaths

This prepares the default codepaths based on the traditional "slurping
everything in-core" model around read_sha1_file() API for objects that use
chunked encoding. Needless to say, these codepaths are unsuitable for the
kind of objects that use chunked encoding and are intended to only serve
as the fallback where specialized "large object API" support is still
lacking.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
chunked-object: streaming checkout

Define a new streaming decoder to handle blobs stored in chunked encoding,
so that we do not have to slurp the data for the whole object to check it
out to the working tree.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'jc/fmt-merge-msg-people' into pu

The "fmt-merge-msg" command learns to list the primary contributors
involved in the side topic you are merging.

Will defer til 1.7.10.

* jc/fmt-merge-msg-people:
  fmt-merge-msg: show those involved in a merged series
Merge branch 'dg/test-from-elsewhere' into pu

Better support for out-of-tree test scripts, but it appears that the
approach needs to be rethought.  By repointing TEST_DIRECTORY to a
directory other than $(pwd)/.., an out of place test script can reach
test helpers and freshly built Git relative to it (GIT_BUILD_DIR is
a mere short-hand for $TEST_DIRECTORY/..).

* dg/test-from-elsewhere:
  Support out-of-tree Valgrind tests
  Allow overriding GIT_BUILD_DIR
Merge branch 'nl/http-proxy-more' into pu

The code to talk to http proxies learn to use the same credential
API used to talk to the final http destinations.

Will defer til 1.7.10.

* nl/http-proxy-more:
  http: rename HTTP_REAUTH to HTTP_AUTH_RETRY
  http: Avoid limit of retrying request only twice
  http: handle proxy authentication failure (error 407)
  http: handle proxy proactive authentication
  http: try http_proxy env var when http.proxy config option is not set
Merge branch 'ss/git-svn-prompt-sans-terminal' into pu

The bottom one has been replaced with a rewrite based on comments
from Ævar. The second one needs more work, both in perl/Git.pm and
prompt.c, to give precedence to tty over SSH_ASKPASS when terminal
is available.

* ss/git-svn-prompt-sans-terminal:
  fixup! 15eaaf4
  git-svn, perl/Git.pm: extend Git::prompt helper for querying users
  perl/Git.pm: "prompt" helper to honor GIT_ASKPASS and SSH_ASKPASS
Merge branch 'hv/submodule-recurse-push' into pu

The bottom one was not clearly explained and needs a reroll.

* hv/submodule-recurse-push:
  push: teach --recurse-submodules the on-demand option
  Refactor submodule push check to use string list instead of integer
  Teach revision walking machinery to walk multiple times sequencially

Conflicts:
	transport.h
Merge branch 'nd/columns' into pu

Rerolled again.  Modulo minor nits, looked nicer than the previous round.

* nd/columns:
  column: support grouping entries
  column: support "denser" mode
  ls-files: support --column
  tag: add --column
  column: support piping stdout to external git-column process
  status: add --column
  branch: add --column
  help: reuse print_columns() for help -a
  column: add dense layout support
  column: add columnar layout
  Stop starting pager recursively
  Add column layout skeleton and git-column
Merge branch 'jc/split-blob' into pu

Not ready.

I finished the streaming checkout codepath, but as explained in
127b177 (bulk-checkin: support chunked-object encoding, 2011-11-30),
these are still early steps of a long and painful journey. At least
pack-objects and fsck need to learn the new encoding for the series
to be usable locally, and then index-pack/unpack-objects needs to
learn it to be used remotely.

Given that I heard a lot of noise that people want large files, and
that I was asked by somebody at GitTogether'11 privately for an
advice on how to pay developers (not me) to help adding necessary
support, I am somewhat dissapointed that the original patch series
that was sent almost two months ago still remains here without much
comments and updates from the developer community. I even made the
interface to the logic that decides where to split chunks easily
replaceable, and I deliberately made the logic in the original patch
extremely stupid to entice others, especially the "bup" fanboys, to
come up with a better logic, thinking that giving people an easy
target to shoot for, they may be encouraged to help out. The plan is
not working :-(.

* jc/split-blob:
  chunked-object: streaming checkout
  chunked-object: fallback checkout codepaths
  bulk-checkin: support chunked-object encoding
  bulk-checkin: allow the same data to be multiply hashed
  new representation types in the packstream
  varint-in-pack: refactor varint encoding/decoding

Conflicts:
	t/t1050-large.sh
Merge branch 'jh/apply-free-patch' into pu

Will defer til 1.7.10.

* jh/apply-free-patch:
  apply: do not leak patches and fragments
Merge branch 'mm/push-default-switch-warning' into pu

This resurrects an ancient patch I wrote during a discussion we had
in the 1.6.3-1.6.4 era.  It also conflicts with the long-stalled
jc/advise-push-default topic and for a good reason---they address a
similar issue.  The latter is a lower impact patch that uses a more
focused approach to limit the scope to the target audience that
actually needs help.

* mm/push-default-switch-warning:
  push: start warning upcoming default change for push.default
Merge branch 'jc/run-hook-env-1' into pu

Updates run_hook() API to be much less specific to "commit".
May not even be needed.

* jc/run-hook-env-1:
  run_hook(): enhance the interface to pass arbitrary environment
Merge branch 'nd/threaded-index-pack' into pu

Another reroll after a bugreport on pthread usage discovered by
Ramsey.

* nd/threaded-index-pack:
  index-pack: support multithreaded delta resolving
  index-pack: split second pass obj handling into own function
Merge branch 'jn/diffstat-tests' into pu

Some tests checked the "diff --stat" output when they do not have to,
which unnecessarily made things harder to verify under GETTEXT_POISON.

Not urgent.

* jn/diffstat-tests:
  diffstat summary line varies by locale: miscellany
  test: use numstat instead of diffstat in binary-diff test
  test: use --numstat instead of --stat in "git stash show" tests
  test: test cherry-pick functionality and output separately
  test: modernize funny-names test style
  test: use numstat instead of diffstat in funny-names test
  test: use test_i18ncmp when checking --stat output
Merge branch 'tr/maint-word-diff-regex-sticky' into pu

The regexp configured with wordregex was incorrectly reused across
files.
Not urgent.

* tr/maint-word-diff-regex-sticky:
  diff: tweak a _copy_ of diff_options with word-diff
  diff: refactor the word-diff setup from builtin_diff_cmd
  t4034: diff.*.wordregex should not be "sticky" in --word-diff
Merge branch 'zj/test-cred-helper-nicer-prove' into pu

Minor improvement to t0303.
Not urgent.

* zj/test-cred-helper-nicer-prove:
  t0303: resurrect commit message as test documentation
  t0303: immediately bail out w/o GIT_TEST_CREDENTIAL_HELPER