archive-zip: fix compressed size for stored export-subst files

Currently ZIP archive entries of files with export-subst attribute are
broken if they are stored uncompressed.

We get the size of a file from sha1_object_info(), but this number is
likely wrong for files whose contents are changed due to export-subst
placeholder expansion.  We use sha1_file_to_archive() to get the
expanded file contents and size in that case.  We proceed to use that
size for the uncompressed size field (good), but the compressed size
field is set based on the size from sha1_object_info() (bad).

This matters only for uncompressed files because for deflated files
we use the correct value after compression is done.  And for files
without export-subst expansion the sizes from sha1_object_info() and
sha1_file_to_archive() are the same, so they are unaffected as well.

This patch fixes the issue by setting the compressed size based on the
uncompressed size only after we actually know the latter.

Also make use of the test file substfile1 to check for the breakage;
it was only stored verbatim so far.  For that purpose, set the
attribute export-subst and replace its contents with the expected
expansion after committing.

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
name-hash.c: fix endless loop with core.ignorecase=true

With core.ignorecase=true, name-hash.c builds a case insensitive index of
all tracked directories. Currently, the existing cache entry structures are
added multiple times to the same hashtable (with different name lengths and
hash codes). However, there's only one dir_next pointer, which gets
completely messed up in case of hash collisions. In the worst case, this
causes an endless loop if ce == ce->dir_next:

    # "V/", "V/XQANY/" and "WURZAUP/" all have the same hash_name
    mkdir V
    mkdir V/XQANY
    mkdir WURZAUP
    touch V/XQANY/test
    git init
    git config core.ignorecase true
    git add .
    git status

Use a separate hashtable and separate structures for the directory index
so that each directory entry has its own next pointer. Use reference
counting to track which directory entry contains files.

There are only slight changes to the name-hash.c API:
- new free_name_hash() used by read_cache.c::discard_index()
- remove_name_hash() takes an additional index_state parameter
- index_name_exists() for a directory (trailing '/') may return a cache
  entry that has been removed (CE_UNHASHED). This is not a problem as the
  return value is only used to check if the directory exists (dir.c) or to
  normalize casing of directory names (read-cache.c).

Getting rid of cache_entry.dir_next reduces memory consumption, especially
with core.ignorecase=false (which doesn't use that member at all).

With core.ignorecase=true, building the directory index is slightly faster
as we add / check the parent directory first (instead of going through all
directory levels for each file in the index). E.g. with WebKit (~200k
files, ~7k dirs), time taken in lazy_init_name_hash is reduced from 176ms
to 130ms.

Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'nd/count-garbage' into jch

Will cook in 'next'.

* nd/count-garbage:
  count-objects: report how much disk space taken by garbage files
  count-objects: report garbage files in pack directory too
  sha1_file: reorder code in prepare_packed_git_one()
  git-count-objects.txt: describe each line in -v output
Merge branch 'jc/fetch-raw-sha1' into jch

Allows requests to fetch objects at any tip of refs (including
hidden ones).  It seems that there may be use cases even outside
Gerrit (e.g. $gmane/215701).

Will cook in 'next'.

* jc/fetch-raw-sha1:
  fetch: fetch objects by their exact SHA-1 object names
  upload-pack: optionally allow fetching from the tips of hidden refs
  fetch: use struct ref to represent refs to be fetched
  parse_fetch_refspec(): clarify the codeflow a bit

Conflicts:
	cache.h
Merge branch 'jc/remove-treesame-parent-in-simplify-merges' into jch

The --simplify-merges logic did not cull irrelevant parents from a
merge that is otherwise not interesting with respect to the paths
we are following.

This touches a fairly core part of the revision traversal
infrastructure; even though I think this change is correct, please
report immediately if you find any unintended side effect.

Will cook in 'next'.

* jc/remove-treesame-parent-in-simplify-merges:
  simplify-merges: drop merge from irrelevant side branch

Conflicts:
	revision.c
Merge branch 'jc/remove-export-from-config-mak-in' into jch

config.mak.in template had an "export" line to cause a few
common makefile variables to be exported; if they need to be
expoted for autoconf/configure users, they should also be exported
for people who write config.mak the same way.  Move the "export" to
the main Makefile.  Also, stop exporting mandir that used to be
exported (only) when config.mak.autogen was used.  It would have
broken installation of manpages (but not other documentation
formats).

Will cook in 'next'.

* jc/remove-export-from-config-mak-in:
  Makefile: do not export mandir/htmldir/infodir
  config.mak.in: remove unused definitions

Conflicts:
	config.mak.in
Merge branch 'nd/read-directory-recursive-optim' into jch

"git status" has been optimized by taking advantage of the fact
that paths that are already known to the index do not have to be
checked against the .gitignore mechanism under some conditions.

Will cook in 'next'.

* nd/read-directory-recursive-optim:
  read_directory: avoid invoking exclude machinery on tracked files
Merge branch 'jc/push-2.0-default-to-simple' into jch

Will cook in 'next' until Git 2.0.

* jc/push-2.0-default-to-simple:
  t5570: do not assume the "matching" push is the default
  t5551: do not assume the "matching" push is the default
  t5550: do not assume the "matching" push is the default
  doc: push.default is no longer "matching"
  push: switch default from "matching" to "simple"
  t9401: do not assume the "matching" push is the default
  t9400: do not assume the "matching" push is the default
  t7406: do not assume the "matching" push is the default
  t5531: do not assume the "matching" push is the default
  t5519: do not assume the "matching" push is the default
  t5517: do not assume the "matching" push is the default
  t5516: do not assume the "matching" push is the default
  t5505: do not assume the "matching" push is the default
  t5404: do not assume the "matching" push is the default

Conflicts:
	Documentation/config.txt
	builtin/push.c
Merge branch 'bc/commit-complete-lines-given-via-m-option' into jch

'git commit -m "$str"' when $str was already terminated with a LF
now avoids adding an extra LF to the message.

Will cook in 'next'.

* bc/commit-complete-lines-given-via-m-option:
  Documentation/git-commit.txt: rework the --cleanup section
  git-commit: only append a newline to -m mesg if necessary
  t7502: demonstrate breakage with a commit message with trailing newlines
  t/t7502: compare entire commit message with what was expected
Merge branch 'ob/imap-send-ssl-verify' into jch

Correctly connect to SSL/TLS sites that serve multiple hostnames on
a single IP by including Server Name Indication in the client-hello.

Will cook in 'next'.

* ob/imap-send-ssl-verify:
  imap-send: support Server Name Indication (RFC4366)
Merge branch 'jk/pkt-line-cleanup' into jch

Cleans up pkt-line API, implementation and its callers to make
them more robust.

Will cook in 'next'.

* jk/pkt-line-cleanup:
  remote-curl: always parse incoming refs
  remote-curl: move ref-parsing code up in file
  remote-curl: pass buffer straight to get_remote_heads
  teach get_remote_heads to read from a memory buffer
  pkt-line: share buffer/descriptor reading implementation
  pkt-line: provide a LARGE_PACKET_MAX static buffer
  pkt-line: move LARGE_PACKET_MAX definition from sideband
  pkt-line: teach packet_read_line to chomp newlines
  pkt-line: provide a generic reading function with options
  pkt-line: drop safe_write function
  pkt-line: move a misplaced comment
  write_or_die: raise SIGPIPE when we get EPIPE
  upload-archive: use argv_array to store client arguments
  upload-archive: do not copy repo name
  send-pack: prefer prefixcmp over memcmp in receive_status
  fetch-pack: fix out-of-bounds buffer offset in get_ack
  upload-pack: remove packet debugging harness
  upload-pack: do not add duplicate objects to shallow list
  upload-pack: use get_sha1_hex to parse "shallow" lines

Conflicts:
	builtin/fetch-pack.c
Merge branch 'da/difftool-fixes' into jch

Minor maintenance updates to difftool, and updates to its tests.

Will cook in 'next'.

* da/difftool-fixes:
  t7800: "defaults" is no longer a builtin tool name
  t7800: modernize tests
  t7800: update copyright notice
  difftool: silence uninitialized variable warning
Merge branch 'rt/commit-cleanup-config' into jch

Fix tests that contaminated their environments and affected new
tests introduced later in the sequence by containing their effects
in their own subshells.

Will cook in 'next'.

* rt/commit-cleanup-config:
  t7502: perform commits using alternate editor in a subshell
Merge branch 'bc/append-signed-off-by' into jch

Will cook in 'next'.

* bc/append-signed-off-by:
  git-commit: populate the edit buffer with 2 blank lines before s-o-b
  Unify appending signoff in format-patch, commit and sequencer
  format-patch: update append_signoff prototype
  t4014: more tests about appending s-o-b lines
  sequencer.c: teach append_signoff to avoid adding a duplicate newline
  sequencer.c: teach append_signoff how to detect duplicate s-o-b
  sequencer.c: always separate "(cherry picked from" from commit body
  sequencer.c: require a conforming footer to be preceded by a blank line
  sequencer.c: recognize "(cherry picked from ..." as part of s-o-b footer
  t/t3511: add some tests of 'cherry-pick -s' functionality
  t/test-lib-functions.sh: allow to specify the tag name to test_commit
  commit, cherry-pick -s: remove broken support for multiline rfc2822 fields
  sequencer.c: rework search for start of footer to improve clarity

Conflicts:
	builtin/log.c
	t/t4014-format-patch.sh
Merge branch 'jc/perl-cat-blob' into jch

perl/Git.pm::cat_blob slurped everything in core only to write it
out to a file descriptor, which was not a very smart thing to do.

Will cook in 'next'.

* jc/perl-cat-blob:
  Git.pm: fix cat_blob crashes on large files
Merge branch 'nd/branch-error-cases' into jch

"git branch" had more cases where it did not bother to check
nonsense command line parameters.

Will cook in 'next'.

* nd/branch-error-cases:
  branch: segfault fixes and validation
Merge branch 'jc/color-diff-doc' into jch

Will cook in 'next'.

* jc/color-diff-doc:
  diff-options: unconfuse description of --color
Merge branch 'ap/maint-diff-rename-avoid-overlap' into jch

The logic used by "git diff -M --stat" to shorten the names of
files before and after a rename did not work correctly when the
common prefix and suffix between the two filenames overlapped.

Will cook in 'next'.

* ap/maint-diff-rename-avoid-overlap:
  diff: prevent pprint_rename from underrunning input
  diff: Fix rename pretty-print when suffix and prefix overlap
Merge branch 'ap/maint-update-index-h-is-for-help' into jch

Will cook in 'next'.

* ap/maint-update-index-h-is-for-help:
  update-index: allow "-h" to also display options
Merge branch 'wk/doc-pre-rebase' into jch

Will cook in 'next'.

* wk/doc-pre-rebase:
  Documentation/githooks: Explain pre-rebase parameters