varint-in-pack: refactor varint encoding/decoding

Refactor encode/decode_in_pack_varint() functions from OFS_DELTA codepaths
to read and write variable-length integers in the pack stream.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
new representation types in the packstream

In addition to the four basic types (commit, tree, blob and tag), the pack
stream can encode a few other "representation" types, such as REF_DELTA
and OFS_DELTA.  As we allocate 3 bits in the first byte for this purpose,
we do not have much room to add new representation types in place, but we
do have one value reserved for future expansion.

When encoding a new representation type, the early part of the in-pack
object header is encoded as if its type is OBJ_EXT (= 5).  The lower 4-bit
of the first byte is used for the lowest 4-bit of the size information,
the next 3-bit has the type information, and the MSB says if the
subsequent bytes need to be read to decode higher bits for the size
information.

An in-pack object header that records OBJ_EXT as its type is followed by
an integer in the same variable-length encoding as OFS_DELTA offset is
encoded in.  This value is the real type of the representation minus 8 (as
we do not need to use OBJ_EXT to encode types smaller than 8).  Because we
do not foresee very many representation types, in practice we would have a
single byte with its MSB clear, to represent types 8-135.

The code does not add nor use type=8 and upwards for anything yet.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
bulk-checkin: allow the same data to be multiply hashed

This updates stream_to_pack() machinery to feed the data it is writing out
to multiple hash contexts at the same time. Right now we only use a single
git_SHA_CTX, so there is no change in functionality.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
bulk-checkin: support chunked-object encoding

Instead of recording a huge object as a single entry in the packfile,
record it as a concatenation of smaller blob objects. This is primarily to
make it possible to repack repositories with huge objects and transfer
them out of such repositories.

This is the first step of a long and painful journey. We would still need
to teach many codepaths about the new encoding:

 * The streaming checkout codepath must learn how to read these from the
   object store and write a concatenation of component blob contents to
   the working tree. Note that after a repack, a component blob object
   could be represented as a delta to another blob object.

 * The in-core object reading codepath must learn to notice and at least
   reject reading such objects entirely in-core. It is expected that
   nobody is interested in producing a patch out of these huge objects, at
   least initially.

 * The object connectivity machinery must learn that component blob
   objects are reachable from an object that uses them, so that "gc"
   will not lose them, and "fsck" will not complain about them.

 * The pack-object machinery must learn to copy an object that is encoded
   in chunked-object encoding as-is to the destination, while perhaps
   validating the object name, without having to first come up with a huge
   concatenated blob and then split it again into smaller pieces. It also
   must learn that the component blobs need to be packed when it does so.

 * The index-pack and verify-pack machineries need to be told about the
   objects in the chunked encoding.

The split-chunk logic used here is kept deliberately useless in order to
avoid distracting the reviewers.  We will want to replace it with a better
heuristics---the rest of the system is designed not to care, so that the
heuristics can be improved without any compatibility issues.

One possibility is to use the heuristics "bup" uses, which is based on a
self-synchronizing rolling checksum logic; if it works as advertised, we
should be able to obtain a good reuse of component blobs when only small
parts of a large blob in chunked encoding are updated.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
chunked-object: fallback checkout codepaths

This prepares the default codepaths based on the traditional "slurping
everything in-core" model around read_sha1_file() API for objects that use
chunked encoding. Needless to say, these codepaths are unsuitable for the
kind of objects that use chunked encoding and are intended to only serve
as the fallback where specialized "large object API" support is still
lacking.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
chunked-object: streaming checkout

Define a new streaming decoder to handle blobs stored in chunked encoding,
so that we do not have to slurp the data for the whole object to check it
out to the working tree.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
push: Provide situational hints for non-fast-forward errors

Pushing a non-fast-forward update to a remote repository will result in
an error, but the hint text doesn't provide the correct resolution in
every case. Give better resolution advice in three push scenarios:

1) If you push your current branch and it triggers a non-fast-forward
error, you should merge remote changes with 'git pull' before pushing
again.

2) If you push to a shared repository others push to, and your local
tracking branches are not kept up to date, the 'matching refs' default
will generate non-fast-forward errors on outdated branches. If this is
your workflow, the 'matching refs' default is not for you. Consider
setting the 'push.default' configuration variable to 'current' or
'upstream' to ensure only your current branch is pushed.

3) If you explicitly specify a ref that is not your current branch or
push matching branches with ':', you will generate a non-fast-forward
error if any pushed branch tip is out of date. You should checkout the
offending branch and merge remote changes before pushing again.

Teach transport.c to recognize these scenarios and configure push.c
to hint for them. If 'git push's default behavior changes or we
discover more scenarios, extension is easy. Standardize on the
advice API and add three new advice variables, 'pushNonFFCurrent',
'pushNonFFDefault', and 'pushNonFFMatching'. Setting any of these
to 'false' will disable their affiliated advice. Setting
'pushNonFastForward' to false will disable all three, thus preserving the
config option for users who already set it, but guaranteeing new
users won't disable push advice accidentally.

Based-on-patch-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Christopher Tiwald <christiwald@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Pull gitweb If-Modified-Since handling out into its own function and use for snapshots.

The current gitweb only generates Last-Modified and handles
If-Modified-Since headers for the git_feed action.  This patch breaks
the Last-Modified and If-Modified-Since handling code out from
git_feed into a new function modified_since.  This makes the code easy
to reuse for other actions where it is appropriate and adds the code
to do that to git_snapshot.

Signed-off-by: W Trevor King <wking@drexel.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
contrib/completion: "local var=()" is misinterpreted as function decl by zsh

Certain versions (or option combinations) of zsh appear to treat
things like "local some_var=()" as a function declaration.  When
using it in combination with the GIT_PS1_SHOWUPSTREAM option, this
seems to show errors.

Signed-off-by: Alex Merry <dev@randomguy3.me.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
log --graph --stat: three-dash separator should come after graph lines

Output from "git log --graph --stat -p" emits the three-dash separator
line before the graph that shows ancestry lines.  The separator should
come after the ancestry lines just like all the other output.

Signed-off-by: Lucian Poston <lucian.poston@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
log --graph: fix break in graph lines

Output from "git log --graph --stat -p" broke the ancestry graph lines
with a single empty line between the diffstat and the patch.

Signed-off-by: Lucian Poston <lucian.poston@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t4202: add test for "log --graph --stat -p" separator lines

Add tests to make sure that the three-dash separator lines appear
after the graph ancestry lines, and also the graph ancestry lines
are not broken between the diffstat and the patch.

Signed-off-by: Lucian Poston <lucian.poston@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
apply: free patch->result

This is by far the largest piece of data, much larger than the
patch and fragment structures the previous change tried to free.

I did not audit all the codepaths so this change needs to be
eyeballed carefully. We may be pointing an unfreeable piece of
memory or a piece of memory that belong to other structures that
are going to be freed otherwise.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
merge: backport GIT_MERGE_AUTOEDIT support

Even though 1.7.9.x series does not open the editor by default
when merging in general, it does do so in one occassion: when
merging an annotated tag. And worse yet, there is no good way
for older scripts to decline this.

Backport the support for GIT_MERGE_AUTOEDIT environment variable
from 1.7.10 track to help those stuck on 1.7.9.x maintenance
track.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'nd/stream-more' into jch

Use API to read blob data in smaller chunks in more places to
reduce the memory footprint.  In general, looked fairly good.

Will defer til 1.7.10.

* nd/stream-more:
  update-server-info: respect core.bigfilethreshold
  fsck: use streaming API for writing lost-found blobs
  show: use streaming API for showing blobs
  parse_object: avoid putting whole blob in core
  cat-file: use streaming API to print blobs
  Add more large blob test cases
  streaming: make streaming-write-entry to be more reusable
Merge branch 'rs/unpack-trees-leakfix' into jch

Will defer til 1.7.10.

* rs/unpack-trees-leakfix:
  unpack-trees: plug minor memory leak
Merge branch 'jc/commit-hook-authorship' into jch

"git commit --author=$name" did not tell the name that was being
recorded in the resulting commit to hooks, even though it does do so
when the end user overrode the authorship via the "GIT_AUTHOR_NAME"
environment variable.  This is a simpler of the two approaches.

Will defer til 1.7.10.

* jc/commit-hook-authorship:
  commit: pass author/committer info to hooks
  t7503: does pre-commit-hook learn authorship?
  ident.c: add split_ident_line() to parse formatted ident line
Merge branch 'jc/diff-algo-cleanup' into jch

Resurrects the preparatory clean-up patches from another topic that was
discarded, as this would give a saner foundation to build on diff.algo
configuration option series.

Not urgent.

* jc/diff-algo-cleanup:
  xdiff: PATIENCE/HISTOGRAM are not independent option bits
  xdiff: remove XDL_PATCH_* macros
Merge branch 'da/difftool-test' into jch

Makes sure "difftool" options can be given in any order.

* da/difftool-test:
  t7800: Test difftool passing arguments to diff
Merge branch 'jn/diffstat-tests' into jch

Some tests checked the "diff --stat" output when they do not have to,
which unnecessarily made things harder to verify under GETTEXT_POISON.

Not urgent.

* jn/diffstat-tests:
  diffstat summary line varies by locale: miscellany
  test: use numstat instead of diffstat in binary-diff test
  test: use --numstat instead of --stat in "git stash show" tests
  test: test cherry-pick functionality and output separately
  test: modernize funny-names test style
  test: use numstat instead of diffstat in funny-names test
  test: use test_i18ncmp when checking --stat output