pack-objects: use streaming interface for reading large loose blobs

git usually streams large blobs directly to packs. But there are cases
where git can create large loose blobs (unpack-objects or hash-object
over pipe). Or they can come from other git implementations.
core.bigfilethreshold can also be lowered down and introduce a new
wave of large loose blobs.

Use streaming interface to read/compress/write these blobs in one
go. Fall back to normal way if somehow streaming interface cannot be
used.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Unindent excluded_from_list()

Return early if el->nr == 0. Unindent one more level for FNM_PATHNAME
code block as this block is getting complex and may need more
indentation.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
exclude: do strcmp as much as possible before fnmatch

this also avoids calling fnmatch() if the non-wildcard prefix is
longer than basename

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t7403-*.sh: Avoid use of the nonportable '==' operator

Some shells, including dash, do not support using the '==' string
equality operator. This results in the failure of tests 7-12 with
'test' complaining of an "unexpected operator".

In order to suppress the errors, we replace the '==' operator with
'=', which must be supported by any POSIX shell.

Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
docs: drop asciidoc7compatible flag

When we made the switch to supporting asciidoc 8 in 4c7100a
(Documentation: adjust to AsciiDoc 8, 2007-06-14), we were
able to leave most of the documentation intact by defining
asciidoc7compatible.

Since commit 6cf378f (docs: stop using asciidoc no-inline-literal,
2012-04-26), we don't support versions of asciidoc older
than 8.4.1, which is when inline literals were introduced.
Therefore there is not much point in keeping our
documentation compatible with asciidoc 7.

So we are now free to drop the asciidoc7compatible flag and
update the documentation itself to assume asciidoc8.
Fortunately, doing the latter is very easy; we weren't using
any of the constructs impacted by asciidoc7compatible, so
there are no changes to make.

The reason is somewhat subtle. The asciidoc7compatible
affects only super/sub-scripts ("^" and "~") and index
terms. We don't use the latter at all. Nor we do we use the
former, but we did have to protect them from accidental
expansion in constructs like "rev^1". However, all of our
uses of "~" and "^" are either in code blocks (which are
rendered literally), or inside backticks. Prior to 6cf378f,
backticks were not inline literals, and needed proper
quoting. But post-6cf378f, we don't have to worry whether we
are using the old or new rules, as those characters are not
interpreted at all in either case.

I verified that the result of "make install-html
install-man" is identical before and after this patch on
asciidoc 8.6.7.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
docs: drop antique comment from Makefile

This comment warns about a bug in asciidoc 6, and points to
a patch from 2005. Since we don't even support versions of
asciidoc that old, we can safely get rid of the warning.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
docs/clone: mention that --local may be ignored

The --local flag is not "treat this like a local
repository", but rather "if we are local, turn on
optimizations". Therefore it does nothing in the case of:

  git clone --local file:///path/to/repo

Let's make that more clear in the documentation.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
clone: allow --no-local to turn off local optimizations

This is basically the same as using "file://", but is a
little less subtle for the end user. It also allows relative
paths to be specified.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
improve no-op push output

When a push is a no-op because all refs are up-to-date, we print
"Everything up-to-date". That is reasonable when push.default is
"matching" (or when a wildcard refspec is given), because "Everything"
pretty obviously means "everything you asked git to push".

But when one of the single-ref push.default modes is used, the
"Everything" is slightly misleading; we only tried to push one thing,
and we should not give the user the impression that the remote is
completely in sync with what is in their local repo.

Instead, let's detect the case that we attempted to push a single ref,
and if so, just show the verbose status table (which includes the
up-to-date ref). We don't want to show it if we tried to push many refs,
because it could be quite long (e.g., in the case of "matching").

Signed-off-by: Junio C Hamano <gitster@pobox.com>
ls-files -i: pay attention to exclusion of leading paths

"git ls-files --exclude=t/ -i" does not show paths in directory t/
that have been added to the index, but it should.

The excluded() API was designed for callers who walk the tree from
the top, checking each level of the directory hierarchy as it
descends if it is excluded, and not even bothering to recurse into
an excluded directory.  This would allow us optimize for a common
case by not having to check if the exclude pattern "foo/" matches
when looking at "foo/bar", because the caller should have noticed
that "foo" is excluded and did not even bother to read "foo/bar"
out of opendir()/readdir() to call it.

The code for "ls-files -i" however walks the index linearly, feeding
paths without checking if the leading directory is already excluded.

Introduce a helper function path_excluded() to let this caller
properly call excluded() check for higher hierarchies as necessary.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
ls-files -i: micro-optimize path_excluded()

As we know a caller that does not recurse is calling us in the index
order, we can remember the last directory we found to be excluded
and see if the path we are looking at is still inside it, in which
case we can just answer that it is excluded.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
request-pull: really favor a matching tag

After tagging the tip of "dev" branch with a "for-linus" tag and
pushing both out, running

	$ git request-pull $url $last_release dev

would produce an output asking the 'dev' branch of $url to be
pulled, because that is what the user asked the message to say.

We already detect this situation locally and include the contents of
the tag in the output; if the $url has that tag, favor that tag
(i.e. "for-linus") in the generated message over the branch name the
user gave us (i.e. "dev") from the command line, to make the output
look more consistent.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'cr/persistent-https' into jch

A remote helper that acts as a proxy that caches ssl session for the
https:// transport is added to the contrib/ area.

By Colby Ranger
* cr/persistent-https:
  Add persistent-https to contrib
Merge branch 'nl/http-proxy-more' into pu

I queued only the later two patches from this series, even though they do
not make much sense without the first one that seems to need a bit more
work, so that we won't forget.

By Jeff King (1) and Nelson Benitez Leon (1)
* nl/http-proxy-more:
  http: rename HTTP_REAUTH to HTTP_AUTH_RETRY
  http: Avoid limit of retrying request only twice
Merge branch 'mm/push-default-switch-warning' into pu

Will squash the two, but this has to wait for a few release cycles.

By Junio C Hamano (1) and Matthieu Moy (1)
* mm/push-default-switch-warning:
  t5541: warning message is given even with --quiet
  push: start warning upcoming default change for push.default
Merge branch 'jc/split-blob' into pu

Not ready.

I finished the streaming checkout codepath, but as explained in
127b177 (bulk-checkin: support chunked-object encoding, 2011-11-30),
these are still early steps of a long and painful journey. At least
pack-objects and fsck need to learn the new encoding for the series
to be usable locally, and then index-pack/unpack-objects needs to
learn it to be used remotely.

Given that I heard a lot of noise that people want large files, and
that I was asked by somebody at GitTogether'11 privately for an
advice on how to pay developers (not me) to help adding necessary
support, I am somewhat dissapointed that the original patch series
that was sent almost two months ago still remains here without much
comments and updates from the developer community. I even made the
interface to the logic that decides where to split chunks easily
replaceable, and I deliberately made the logic in the original patch
extremely stupid to entice others, especially the "bup" fanboys, to
come up with a better logic, thinking that giving people an easy
target to shoot for, they may be encouraged to help out. The plan is
not working :-(.

* jc/split-blob:
  chunked-object: streaming checkout
  chunked-object: fallback checkout codepaths
  bulk-checkin: support chunked-object encoding
  bulk-checkin: allow the same data to be multiply hashed
  new representation types in the packstream
  packfile: use varint functions

Conflicts:
	builtin/pack-objects.c
	streaming.c
	t/t1050-large.sh
Merge branch 'jc/run-hook-env-1' into pu

Not urgent.

Updates run_hook() API to be much less specific to "commit".  It would
only be useful if people start doing more interesting things with hooks.

* jc/run-hook-env-1:
  run_hook(): enhance the interface to pass arbitrary environment
Merge branch 'jc/maint-push-refs-all' into pu

Allows pushing and fetching refs/stash.
Not ready.
There still seem to be other bugs hiding (e.g. try pushing twice).

* jc/maint-push-refs-all:
  get_fetch_map(): tighten checks on dest refs
  fetch/push: allow refs/*:refs/*
Merge branch 'jc/apply-3way' into pu

"git apply" learns to wiggle the base version and perform three-way merge
when a patch does not exactly apply to the version you have.

It turns out that it is somewhat unpleasant to handle add/add conflicts in
this code, but it seems necessary if we want to use "apply -3" to replace
the use of "apply --build-fake-ancestor" followed by the slow "merge" in
"am -3".

* jc/apply-3way:
  WIP: the message is bogus in apply:::check_patch()
  apply: refactor "previous patch" logic
  apply: a bit more comments on PATH_TO_BE_DELETED
  apply: document --3way option
  apply: allow rerere() upon --3way results
  apply: register conflicted stages to the index
  apply: plug the three-way merge logic in
  apply: fall back on three-way merge
  apply: accept -3/--3way command line option
  apply: split load_preimage() helper function out
  apply: refactor read_file_or_gitlink()
  apply: clear_image() clears things a bit more

Conflicts:
	builtin/apply.c
Merge branch 'fc/git-prompt-script' into pu

The last remaining sticking point is what to do with the duplicated shell
function.

By Felipe Contreras (2) and SZEDER Gábor (2)
* fc/git-prompt-script:
  completion: split __git_ps1 into a separate script
  completion: remove executable mode
  tests: add tests for the bash prompt functions in the completion script
  tests: move code to run tests under bash into a helper library