pack-objects: use streaming interface for reading large loose blobs

git usually streams large blobs directly to packs. But there are cases
where git can create large loose blobs (unpack-objects or hash-object
over pipe). Or they can come from other git implementations.
core.bigfilethreshold can also be lowered down and introduce a new
wave of large loose blobs.

Use streaming interface to read/compress/write these blobs in one
go. Fall back to normal way if somehow streaming interface cannot be
used.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Unindent excluded_from_list()

Return early if el->nr == 0. Unindent one more level for FNM_PATHNAME
code block as this block is getting complex and may need more
indentation.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
exclude: do strcmp as much as possible before fnmatch

this also avoids calling fnmatch() if the non-wildcard prefix is
longer than basename

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
docs: drop asciidoc7compatible flag

When we made the switch to supporting asciidoc 8 in 4c7100a
(Documentation: adjust to AsciiDoc 8, 2007-06-14), we were
able to leave most of the documentation intact by defining
asciidoc7compatible.

Since commit 6cf378f (docs: stop using asciidoc no-inline-literal,
2012-04-26), we don't support versions of asciidoc older
than 8.4.1, which is when inline literals were introduced.
Therefore there is not much point in keeping our
documentation compatible with asciidoc 7.

So we are now free to drop the asciidoc7compatible flag and
update the documentation itself to assume asciidoc8.
Fortunately, doing the latter is very easy; we weren't using
any of the constructs impacted by asciidoc7compatible, so
there are no changes to make.

The reason is somewhat subtle. The asciidoc7compatible
affects only super/sub-scripts ("^" and "~") and index
terms. We don't use the latter at all. Nor we do we use the
former, but we did have to protect them from accidental
expansion in constructs like "rev^1". However, all of our
uses of "~" and "^" are either in code blocks (which are
rendered literally), or inside backticks. Prior to 6cf378f,
backticks were not inline literals, and needed proper
quoting. But post-6cf378f, we don't have to worry whether we
are using the old or new rules, as those characters are not
interpreted at all in either case.

I verified that the result of "make install-html
install-man" is identical before and after this patch on
asciidoc 8.6.7.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
docs: drop antique comment from Makefile

This comment warns about a bug in asciidoc 6, and points to
a patch from 2005. Since we don't even support versions of
asciidoc that old, we can safely get rid of the warning.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
docs/clone: mention that --local may be ignored

The --local flag is not "treat this like a local
repository", but rather "if we are local, turn on
optimizations". Therefore it does nothing in the case of:

  git clone --local file:///path/to/repo

Let's make that more clear in the documentation.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
clone: allow --no-local to turn off local optimizations

This is basically the same as using "file://", but is a
little less subtle for the end user. It also allows relative
paths to be specified.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
improve no-op push output

When a push is a no-op because all refs are up-to-date, we print
"Everything up-to-date". That is reasonable when push.default is
"matching" (or when a wildcard refspec is given), because "Everything"
pretty obviously means "everything you asked git to push".

But when one of the single-ref push.default modes is used, the
"Everything" is slightly misleading; we only tried to push one thing,
and we should not give the user the impression that the remote is
completely in sync with what is in their local repo.

Instead, let's detect the case that we attempted to push a single ref,
and if so, just show the verbose status table (which includes the
up-to-date ref). We don't want to show it if we tried to push many refs,
because it could be quite long (e.g., in the case of "matching").

Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'jl/submodule-report-new-path-once' into jch

Will merge to master by 1.7.11 final.

By Jens Lehmann
* jl/submodule-report-new-path-once:
  submodules: print "registered for path" message only once
Merge branch 'mm/levenstein-penalize-deletion-less' into jch

Will merge to master by 1.7.11 final.

By Matthieu Moy
* mm/levenstein-penalize-deletion-less:
  Reduce cost of deletion in levenstein distance (4 -> 3)
Merge branch 'vr/rebase-autosquash-does-not-imply-i' into jch

Will merge to master by 1.7.11 final.

By Vincent van Ravesteijn
* vr/rebase-autosquash-does-not-imply-i:
  Do not autosquash in case of an implied interactive rebase
Merge branch 'nh/empty-rebase' into jch

Will merge to master by 1.7.11 final.

* nh/empty-rebase:
  cherry-pick: regression fix for empty commits
Merge branch 'ef/maint-rebase-error-message' into jch

Will merge to master by 1.7.11 final.

By Erik Faye-Lund
* ef/maint-rebase-error-message:
  rebase: report invalid commit correctly
Add persistent-https to contrib

Git over HTTPS has a high request startup latency, since the SSL
negotiation can take up to a second. In order to reduce this latency,
connections should be left open to the Git server across requests
(or invocations of the git commandline).

Reduce SSL startup latency by running a daemon job that keeps
connections open to a Git server. The daemon job
(git-remote-persistent-https--proxy) is started on the first request
through the client binary (git-remote-persistent-https) and remains
running for 24 hours after the last request, or until a new daemon
binary is placed in the PATH. The client determines the daemon's
HTTP address by communicating over a UNIX socket with the daemon.
From there, the rest of the Git protocol work is delegated to the
"git-remote-http" binary, with the environment's http_proxy set to
the daemon.

Accessing /pub/scm/linux/kernel/git/torvalds/linux repository hosted
at kernel.googlesource.com with "git ls-remote" over https:// and
persistent-https:// 5 times shows that the first request takes about
the same time (0.193s vs 0.208s---there is a slight set-up cost for
the local proxy); as expected, the other four requests are much
faster (~0.18s vs ~0.08s).

Incidentally, this also has the benefit of HTTP keep-alive working
across Git command invocations. Its common for servers to use a 5
minute keep-alive on an HTTP 1.1 connection. Git-over-HTTP commonly
uses Transfer-Encoding: chunked on replies, so keep-alive will
generally just work, even though a pack stream's length isn't known
in advance. Because the helper is an external process holding that
connection open, we also benefit from being able to reuse an
existing TCP connection to the server.  The same "git ls-remote"
test against http:// vs persistent-https:// URL shows that the
former takes ~0.09s while the first request for the latter is about
0.134s with set-up cost, and subsequent requests are ~0.065s,
shaving around one RTT to the server.

Signed-off-by: Colby Ranger <cranger@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'nl/http-proxy-more' into pu

I queued only the later two patches from this series, even though they do
not make much sense without the first one that seems to need a bit more
work, so that we won't forget.

By Jeff King (1) and Nelson Benitez Leon (1)
* nl/http-proxy-more:
  http: rename HTTP_REAUTH to HTTP_AUTH_RETRY
  http: Avoid limit of retrying request only twice
Merge branch 'mm/push-default-switch-warning' into pu

Will squash the two, but this has to wait for a few release cycles.

By Junio C Hamano (1) and Matthieu Moy (1)
* mm/push-default-switch-warning:
  t5541: warning message is given even with --quiet
  push: start warning upcoming default change for push.default
Merge branch 'jc/split-blob' into pu

Not ready.

I finished the streaming checkout codepath, but as explained in
127b177 (bulk-checkin: support chunked-object encoding, 2011-11-30),
these are still early steps of a long and painful journey. At least
pack-objects and fsck need to learn the new encoding for the series
to be usable locally, and then index-pack/unpack-objects needs to
learn it to be used remotely.

Given that I heard a lot of noise that people want large files, and
that I was asked by somebody at GitTogether'11 privately for an
advice on how to pay developers (not me) to help adding necessary
support, I am somewhat dissapointed that the original patch series
that was sent almost two months ago still remains here without much
comments and updates from the developer community. I even made the
interface to the logic that decides where to split chunks easily
replaceable, and I deliberately made the logic in the original patch
extremely stupid to entice others, especially the "bup" fanboys, to
come up with a better logic, thinking that giving people an easy
target to shoot for, they may be encouraged to help out. The plan is
not working :-(.

* jc/split-blob:
  chunked-object: streaming checkout
  chunked-object: fallback checkout codepaths
  bulk-checkin: support chunked-object encoding
  bulk-checkin: allow the same data to be multiply hashed
  new representation types in the packstream
  packfile: use varint functions

Conflicts:
	builtin/pack-objects.c
	streaming.c
	t/t1050-large.sh
Merge branch 'jc/run-hook-env-1' into pu

Not urgent.

Updates run_hook() API to be much less specific to "commit".  It would
only be useful if people start doing more interesting things with hooks.

* jc/run-hook-env-1:
  run_hook(): enhance the interface to pass arbitrary environment
Merge branch 'jc/maint-push-refs-all' into pu

Allows pushing and fetching refs/stash.
Not ready.
There still seem to be other bugs hiding (e.g. try pushing twice).

* jc/maint-push-refs-all:
  get_fetch_map(): tighten checks on dest refs
  fetch/push: allow refs/*:refs/*
Merge branch 'jc/apply-3way' into pu

"git apply" learns to wiggle the base version and perform three-way merge
when a patch does not exactly apply to the version you have.

It turns out that it is somewhat unpleasant to handle add/add conflicts in
this code, but it seems necessary if we want to use "apply -3" to replace
the use of "apply --build-fake-ancestor" followed by the slow "merge" in
"am -3".

* jc/apply-3way:
  WIP: the message is bogus in apply:::check_patch()
  apply: refactor "previous patch" logic
  apply: a bit more comments on PATH_TO_BE_DELETED
  apply: document --3way option
  apply: allow rerere() upon --3way results
  apply: register conflicted stages to the index
  apply: plug the three-way merge logic in
  apply: fall back on three-way merge
  apply: accept -3/--3way command line option
  apply: split load_preimage() helper function out
  apply: refactor read_file_or_gitlink()
  apply: clear_image() clears things a bit more

Conflicts:
	builtin/apply.c