name-hash.c: fix endless loop with core.ignorecase=true

With core.ignorecase=true, name-hash.c builds a case insensitive index of
all tracked directories. Currently, the existing cache entry structures are
added multiple times to the same hashtable (with different name lengths and
hash codes). However, there's only one dir_next pointer, which gets
completely messed up in case of hash collisions. In the worst case, this
causes an endless loop if ce == ce->dir_next (see t7062).

Use a separate hashtable and separate structures for the directory index
so that each directory entry has its own next pointer. Use reference
counting to track which directory entry contains files.

There are only slight changes to the name-hash.c API:
- new free_name_hash() used by read_cache.c::discard_index()
- remove_name_hash() takes an additional index_state parameter
- index_name_exists() for a directory (trailing '/') may return a cache
  entry that has been removed (CE_UNHASHED). This is not a problem as the
  return value is only used to check if the directory exists (dir.c) or to
  normalize casing of directory names (read-cache.c).

Getting rid of cache_entry.dir_next reduces memory consumption, especially
with core.ignorecase=false (which doesn't use that member at all).

With core.ignorecase=true, building the directory index is slightly faster
as we add / check the parent directory first (instead of going through all
directory levels for each file in the index). E.g. with WebKit (~200k
files, ~7k dirs), time spent in lazy_init_name_hash is reduced from 176ms
to 130ms.

Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
config: factor out config file stack management

Because a config callback may start parsing a new file, the
global context regarding the current config file is stored
as a stack. Currently we only need to manage that stack from
git_config_from_file. Let's factor it out to allow new
sources of config data.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
config: drop file pointer validity check in get_next_char()

The only location where cf is set in this file is in do_config_from().
This function has only one callsite which is config_from_file(). In
config_from_file() its ensured that the f member is set to non-zero.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
config: make parsing stack struct independent from actual data source

To simplify adding other sources we extract all functions needed for
parsing into a list of callbacks. We implement those callbacks for the
current file parsing. A new source can implement its own set of callbacks.

Instead of storing the concrete FILE pointer for parsing we store a void
pointer. A new source can use this to store its custom data.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
teach config parsing to read from strbuf

This can be used to read configuration values directly from gits
database.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Refactor parse_loc

We want to use the same style of -L n,m argument for 'git log -L' as
for git-blame.  Refactor the argument parsing of the range arguments
from builtin/blame.c to the (new) file that will hold the 'git log -L'
logic.

To accommodate different data structures in blame and log -L, the file
contents are abstracted away; parse_range_arg takes a callback that it
uses to get the contents of a line of the (notional) file.

The new test is for a case that made me pause during debugging: the
'blame -L with invalid end' test was the only one that noticed an
outright failure to parse the end *at all*.  So make a more explicit
test for that.

Signed-off-by: Bo Yang <struggleyb.nku@gmail.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
blame: introduce $ as "end of file" in -L syntax

To save the user a lookup of the last line number, introduce $ as a
shorthand for the last line.  This is mostly useful to spell "until
the end of the file" as '-L<begin>,$'.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Export rewrite_parents() for 'log -L'

The function rewrite_one is used to rewrite a single
parent of the current commit, and is used by rewrite_parents
to rewrite all the parents.

Decouple the dependence between them by making rewrite_one
a callback function that is passed to rewrite_parents. Then
export rewrite_parents for reuse by the line history browser.

We will use this function in line.c.

Signed-off-by: Bo Yang <struggleyb.nku@gmail.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Implement line-history search (git log -L)

This is a rewrite of much of Bo's work, mainly in an effort to split
it into smaller, easier to understand routines.

The algorithm is built around the struct range_set, which encodes a
series of line ranges as intervals [a,b).  This is used in two
contexts:

* A set of lines we are tracking (which will change as we dig through
  history).
* To encode diffs, as pairs of ranges.

The main routine is range_set_map_across_diff().  It processes the
diff between a commit C and some parent P.  It determines which diff
hunks are relevant to the ranges tracked in C, and computes the new
ranges for P.

The algorithm is then simply to process history in topological order
from newest to oldest, computing ranges and (partial) diffs.  At
branch points, we need to merge the ranges we are watching.  We will
find that many commits do not affect the chosen ranges, and mark them
TREESAME (in addition to those already filtered by pathspec limiting).
Another pass of history simplification then gets rid of such commits.

This is wired as an extra filtering pass in the log machinery.  This
currently only reduces code duplication, but should allow for other
simplifications and options to be used.

Finally, we hook a diff printer into the output chain.  Ideally we
would wire directly into the diff logic, to optionally use features
like word diff.  However, that will require some major reworking of
the diff chain, so we completely replace the output with our own diff
for now.

Signed-off-by: Bo Yang <struggleyb.nku@gmail.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
log -L: :pattern:file syntax to find by funcname

This new syntax finds a funcname matching /pattern/, and then takes from there
up to (but not including) the next funcname.  So you can say

  git log -L:main:main.c

and it will dig up the main() function and show its line-log, provided
there are no other funcnames matching 'main'.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
contrib/subtree: allow addition of remote branch with name not locally present

cmd_add() attempts to check for the validity of the given commit
before adding the repository as a subtree. It tries to do so before
contacting the repository. If the given ref happens to exist locally
(say 'master') then the test passes and the repo is fetched. But that
local 'master' does not have anything to do with the 'master' we are
going to fetch from the remote repository in the first place.

Remove this pointless test, and replace it with a new check to
prevent pulling in multiple branches or creating a remote tracking
branch for the subtree.

A typo made on the command line will still be caught in the "git
fetch" that is run immediately after this part of the code, that
will die with:

  fatal: Couldn't find remote ref <refspec>

which is an accurate diagnosis.

Signed-off-by: Paul Campbell <pcampbell@carnegiecollege.ac.uk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'nd/count-garbage' into jch

Will cook in 'next'.

* nd/count-garbage:
  count-objects: report how much disk space taken by garbage files
  count-objects: report garbage files in pack directory too
  sha1_file: reorder code in prepare_packed_git_one()
  git-count-objects.txt: describe each line in -v output
Merge branch 'jc/fetch-raw-sha1' into jch

Allows requests to fetch objects at any tip of refs (including
hidden ones).  It seems that there may be use cases even outside
Gerrit (e.g. $gmane/215701).

Will cook in 'next'.

* jc/fetch-raw-sha1:
  fetch: fetch objects by their exact SHA-1 object names
  upload-pack: optionally allow fetching from the tips of hidden refs
  fetch: use struct ref to represent refs to be fetched
  parse_fetch_refspec(): clarify the codeflow a bit

Conflicts:
	cache.h
Merge branch 'jc/remove-treesame-parent-in-simplify-merges' into jch

The --simplify-merges logic did not cull irrelevant parents from a
merge that is otherwise not interesting with respect to the paths
we are following.

This touches a fairly core part of the revision traversal
infrastructure; even though I think this change is correct, please
report immediately if you find any unintended side effect.

Will cook in 'next'.

* jc/remove-treesame-parent-in-simplify-merges:
  simplify-merges: drop merge from irrelevant side branch

Conflicts:
	revision.c
Merge branch 'jc/remove-export-from-config-mak-in' into jch

config.mak.in template had an "export" line to cause a few
common makefile variables to be exported; if they need to be
expoted for autoconf/configure users, they should also be exported
for people who write config.mak the same way.  Move the "export" to
the main Makefile.  Also, stop exporting mandir that used to be
exported (only) when config.mak.autogen was used.  It would have
broken installation of manpages (but not other documentation
formats).

Will cook in 'next'.

* jc/remove-export-from-config-mak-in:
  Makefile: do not export mandir/htmldir/infodir
  config.mak.in: remove unused definitions

Conflicts:
	config.mak.in
Merge branch 'nd/read-directory-recursive-optim' into jch

"git status" has been optimized by taking advantage of the fact
that paths that are already known to the index do not have to be
checked against the .gitignore mechanism under some conditions.

Will cook in 'next'.

* nd/read-directory-recursive-optim:
  read_directory: avoid invoking exclude machinery on tracked files
Merge branch 'jc/push-2.0-default-to-simple' into jch

Will cook in 'next' until Git 2.0.

* jc/push-2.0-default-to-simple:
  t5570: do not assume the "matching" push is the default
  t5551: do not assume the "matching" push is the default
  t5550: do not assume the "matching" push is the default
  doc: push.default is no longer "matching"
  push: switch default from "matching" to "simple"
  t9401: do not assume the "matching" push is the default
  t9400: do not assume the "matching" push is the default
  t7406: do not assume the "matching" push is the default
  t5531: do not assume the "matching" push is the default
  t5519: do not assume the "matching" push is the default
  t5517: do not assume the "matching" push is the default
  t5516: do not assume the "matching" push is the default
  t5505: do not assume the "matching" push is the default
  t5404: do not assume the "matching" push is the default

Conflicts:
	Documentation/config.txt
	builtin/push.c
Merge branch 'bc/commit-complete-lines-given-via-m-option' into jch

'git commit -m "$str"' when $str was already terminated with a LF
now avoids adding an extra LF to the message.

Will cook in 'next'.

* bc/commit-complete-lines-given-via-m-option:
  Documentation/git-commit.txt: rework the --cleanup section
  git-commit: only append a newline to -m mesg if necessary
  t7502: demonstrate breakage with a commit message with trailing newlines
  t/t7502: compare entire commit message with what was expected
Merge branch 'ob/imap-send-ssl-verify' into jch

Correctly connect to SSL/TLS sites that serve multiple hostnames on
a single IP by including Server Name Indication in the client-hello.

Will cook in 'next'.

* ob/imap-send-ssl-verify:
  imap-send: support Server Name Indication (RFC4366)
Merge branch 'jk/pkt-line-cleanup' into jch

Cleans up pkt-line API, implementation and its callers to make
them more robust.

Will cook in 'next'.

* jk/pkt-line-cleanup:
  remote-curl: always parse incoming refs
  remote-curl: move ref-parsing code up in file
  remote-curl: pass buffer straight to get_remote_heads
  teach get_remote_heads to read from a memory buffer
  pkt-line: share buffer/descriptor reading implementation
  pkt-line: provide a LARGE_PACKET_MAX static buffer
  pkt-line: move LARGE_PACKET_MAX definition from sideband
  pkt-line: teach packet_read_line to chomp newlines
  pkt-line: provide a generic reading function with options
  pkt-line: drop safe_write function
  pkt-line: move a misplaced comment
  write_or_die: raise SIGPIPE when we get EPIPE
  upload-archive: use argv_array to store client arguments
  upload-archive: do not copy repo name
  send-pack: prefer prefixcmp over memcmp in receive_status
  fetch-pack: fix out-of-bounds buffer offset in get_ack
  upload-pack: remove packet debugging harness
  upload-pack: do not add duplicate objects to shallow list
  upload-pack: use get_sha1_hex to parse "shallow" lines

Conflicts:
	builtin/fetch-pack.c