pack-objects: use streaming interface for reading large loose blobs

git usually streams large blobs directly to packs. But there are cases
where git can create large loose blobs (unpack-objects or hash-object
over pipe). Or they can come from other git implementations.
core.bigfilethreshold can also be lowered down and introduce a new
wave of large loose blobs.

Use streaming interface to read/compress/write these blobs in one
go. Fall back to normal way if somehow streaming interface cannot be
used.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Unindent excluded_from_list()

Return early if el->nr == 0. Unindent one more level for FNM_PATHNAME
code block as this block is getting complex and may need more
indentation.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
exclude: do strcmp as much as possible before fnmatch

this also avoids calling fnmatch() if the non-wildcard prefix is
longer than basename

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Reduce cost of deletion in levenstein distance (4 -> 3)

Before this patch, a character deletion has the same cost as 2 swaps, or
4 additions, so Git prefers suggesting a completely scrambled command
name to removing a character. For example, "git tags" suggests "stage",
but not "tag".

By setting the deletion cost to 3, we keep it higher than swaps or
additions, but prefer 1 deletion to 2 swaps. "git tags" now suggests
"tag" in addition to staged.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Do not autosquash in case of an implied interactive rebase

The option to autosquash is only used in case of an interactive rebase.
When merges are preserved, rebase uses an interactive rebase internally,
but in this case autosquash should still be disabled.

Signed-off-by: Vincent van Ravesteijn <vfr@lyx.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t7403-*.sh: Avoid use of the nonportable '==' operator

Some shells, including dash, do not support using the '==' string
equality operator. This results in the failure of tests 7-12 with
'test' complaining of an "unexpected operator".

In order to suppress the errors, we replace the '==' operator with
'=', which must be supported by any POSIX shell.

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'nl/http-proxy-more' into pu

I queued only the later two patches from this series, even though they do
not make much sense without the first one that seems to need a bit more
work, so that we won't forget.

By Jeff King (1) and Nelson Benitez Leon (1)
* nl/http-proxy-more:
  http: rename HTTP_REAUTH to HTTP_AUTH_RETRY
  http: Avoid limit of retrying request only twice
Merge branch 'mm/push-default-switch-warning' into pu

Will squash the two, but this has to wait for a few release cycles.

By Junio C Hamano (1) and Matthieu Moy (1)
* mm/push-default-switch-warning:
  t5541: warning message is given even with --quiet
  push: start warning upcoming default change for push.default
Merge branch 'jc/split-blob' into pu

Not ready.

I finished the streaming checkout codepath, but as explained in
127b177 (bulk-checkin: support chunked-object encoding, 2011-11-30),
these are still early steps of a long and painful journey. At least
pack-objects and fsck need to learn the new encoding for the series
to be usable locally, and then index-pack/unpack-objects needs to
learn it to be used remotely.

Given that I heard a lot of noise that people want large files, and
that I was asked by somebody at GitTogether'11 privately for an
advice on how to pay developers (not me) to help adding necessary
support, I am somewhat dissapointed that the original patch series
that was sent almost two months ago still remains here without much
comments and updates from the developer community. I even made the
interface to the logic that decides where to split chunks easily
replaceable, and I deliberately made the logic in the original patch
extremely stupid to entice others, especially the "bup" fanboys, to
come up with a better logic, thinking that giving people an easy
target to shoot for, they may be encouraged to help out. The plan is
not working :-(.

* jc/split-blob:
  chunked-object: streaming checkout
  chunked-object: fallback checkout codepaths
  bulk-checkin: support chunked-object encoding
  bulk-checkin: allow the same data to be multiply hashed
  new representation types in the packstream
  packfile: use varint functions

Conflicts:
	builtin/pack-objects.c
	streaming.c
	t/t1050-large.sh
Merge branch 'jc/run-hook-env-1' into pu

Not urgent.

Updates run_hook() API to be much less specific to "commit".  It would
only be useful if people start doing more interesting things with hooks.

* jc/run-hook-env-1:
  run_hook(): enhance the interface to pass arbitrary environment
Merge branch 'jc/maint-push-refs-all' into pu

Allows pushing and fetching refs/stash.
Not ready.
There still seem to be other bugs hiding (e.g. try pushing twice).

* jc/maint-push-refs-all:
  get_fetch_map(): tighten checks on dest refs
  fetch/push: allow refs/*:refs/*
Merge branch 'jc/apply-3way' into pu

"git apply" learns to wiggle the base version and perform three-way merge
when a patch does not exactly apply to the version you have.

It turns out that it is somewhat unpleasant to handle add/add conflicts in
this code, but it seems necessary if we want to use "apply -3" to replace
the use of "apply --build-fake-ancestor" followed by the slow "merge" in
"am -3".

* jc/apply-3way:
  WIP: the message is bogus in apply:::check_patch()
  apply: refactor "previous patch" logic
  apply: a bit more comments on PATH_TO_BE_DELETED
  apply: document --3way option
  apply: allow rerere() upon --3way results
  apply: register conflicted stages to the index
  apply: plug the three-way merge logic in
  apply: fall back on three-way merge
  apply: accept -3/--3way command line option
  apply: split load_preimage() helper function out
  apply: refactor read_file_or_gitlink()
  apply: clear_image() clears things a bit more

Conflicts:
	builtin/apply.c
Merge branch 'fc/git-prompt-script' into pu

The last remaining sticking point is what to do with the duplicated shell
function.

By Felipe Contreras (2) and SZEDER Gábor (2)
* fc/git-prompt-script:
  completion: split __git_ps1 into a separate script
  completion: remove executable mode
  tests: add tests for the bash prompt functions in the completion script
  tests: move code to run tests under bash into a helper library
Merge branch 'js/submodule-relative' into pu

Expecting a reroll.

By Jon Seymour (2) and Ramsay Jones (1)
* js/submodule-relative:
  t7403-*.sh: Avoid use of the nonportable '==' operator
  submodule: fix handling of relative superproject origin URLs
  submodule: document handling of relative superproject origin URLs
Merge branch 'cr/persistent-https' into pu

A remote helper that acts as a proxy that caches ssl session for the
https:// transport is added to the contrib/ area.

By Colby Ranger
* cr/persistent-https:
  Add persistent-https to contrib
Merge branch 'nd/stream-index-pack' into pu

Use streaming API to read from the object store to avoid having to hold
a large blob object in-core while running index-pack.

By Nguyễn Thái Ngọc Duy
* nd/stream-index-pack:
  index-pack: use streaming interface for collision test on large blobs
  index-pack: factor out unpack core from get_data_from_pack
  index-pack: use streaming interface on large blobs (most of the time)
  index-pack: hash non-delta objects while reading from stream
Merge branch 'sg/bash-prompt' into pu

This is only the "correction" bits taken from the beginning of a larger
series that is to be rerolled.  The first two are tangled with Felipe's
topic so a reroll, if comes, should build on top of them.

By SZEDER Gábor
* sg/bash-prompt:
  completion: respect $GIT_DIR
  completion: use __gitdir() in _git_log()
Merge branch 'jc/revert-nh-empty-rebase' into pu

The earlier series was found to regress the more important "cherry-pick";
I'd prefer to see it fixed, but in case if it does not happen soon, this
is a fallback measure that can be merged before 1.7.11 final.

* jc/revert-nh-empty-rebase:
  Revert nh/empty-rebase topic
submodules: print "registered for path" message only once

Since 2cd9de3e (submodule add: always initialize .git/config entry) the
message "Submodule '\$name' (\$url) registered for path '\$sm_path'" is
printed every time cmd_init() is called, e.g. each time "git submodule
update" is used with the --init option.

This was not intended and leads to bogus output which can confuse users
and build systems. Apart from that the $url variable was not set after the
first run which did the actual initialization and only "()" was printed
in subsequent runs where "($url)" was meant to inform the user about the
upstream repo.

Fix that by moving the say command in question into the if block where the
url is initialized, restoring the behavior that was in place before the
2cd9de3e commit. While at it also remove the comment which still describes
the logic used before 2cd9de3e and add a comment about how things work now.

Reported-by: Nicolas Viennot and Sid Nair <nicolas@viennot.com>
Reported-by: Heiko Voigt <hvoigt@hvoigt.net>
Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'jl/submodule-report-new-path-once' into pu

By Jens Lehmann
* jl/submodule-report-new-path-once:
  submodules: print "registered for path" message only once