safe_create_leading_directories: fix race that could give a false negative

If two processes are racing to create the same directory tree, they will
both see that the directory doesn't exist, both try to mkdir(), and one
of them will fail.  This is okay, as we only care that the directory
gets created.  So, we add a check for EEXIST from mkdir, and continue if
the directory now exists.

Signed-off-by: Steven Walter <stevenrwalter@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
SQUASH???
avoid segfaults on parse_object failure

Many call-sites of parse_object assume that they will get a
non-NULL return value; this is not the case if we encounter
an error while parsing the object.

This patch adds a wrapper function around parse_object that
handles dying automatically, and uses it anywhere we
immediately try to access the return value as a non-NULL
pointer (i.e., anywhere that we would currently segfault).

This wrapper may also be useful in other places. The most
obvious one is code like:

  o = parse_object(sha1);
  if (!o)
	  die(...);

However, these should not be mechanically converted to
parse_object_or_die, as the die message is sometimes
customized. Later patches can address these sites on a
case-by-case basis.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
use parse_object_or_die instead of die("bad object")

Some call-sites do:

  o = parse_object(sha1);
  if (!o)
	  die("bad object %s", some_name);

We can now handle that as a one-liner, and get more
consistent output.

In the third case of this patch, it looks like we are losing
information, as the existing message also outputs the sha1
hex; however, parse_object will already have written a more
specific complaint about the sha1, so there is no point in
repeating it here.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
pack-refs: write peeled entry for non-tags

When we pack an annotated tag ref, we write not only the
sha1 of the tag object along with the ref, but also the sha1
obtained by peeling the tag. This lets readers of the
pack-refs file know the peeled value without having to
actually load the object, speeding up upload-pack's ref
advertisement.

The writer marks a packed-refs file with peeled refs using
the "peeled" trait at the top of the file. When the reader
sees this trait, it knows that each ref is either followed
by its peeled value, or it is not an annotated tag.

However, there is a mismatch between the assumptions of the
reader and writer. The writer will only peel refs under
refs/tags, but the reader does not know this; it will assume
a ref without a peeled value must not be a tag object. Thus
an annotated tag object placed outside of the refs/tags
hierarchy will not have its peeled value printed by
upload-pack.

The simplest way to fix this is to start writing peel values
for all refs. This matches what the reader expects for both
new and old versions of git.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
pack-refs: add fully-peeled trait

Older versions of pack-refs did not write peel lines for
refs outside of refs/tags. This meant that on reading the
pack-refs file, we might set the REF_KNOWS_PEELED flag for
such a ref, even though we do not know anything about its
peeled value.

The previous commit updated the writer to always peel, no
matter what the ref is. That means that packed-refs files
written by newer versions of git are fine to be read by both
old and new versions of git. However, we still have the
problem of reading packed-refs files written by older
versions of git, or by other implementations which have not
yet learned the same trick.

The simplest fix would be to always unset the
REF_KNOWS_PEELED flag for refs outside of refs/tags that do
not have a peel line (if it has a peel line, we know it is
valid, but we cannot assume a missing peel line means
anything). But that loses an important optimization, as
upload-pack should not need to load the object pointed to by
refs/heads/foo to determine that it is not a tag.

Instead, we add a "fully-peeled" trait to the packed-refs
file. If it is set, we know that we can trust a missing peel
line to mean that a ref cannot be peeled. Otherwise, we fall
back to assuming nothing.

[commit message and tests by Jeff King <peff@peff.net>]

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
SQUASH???
fast-export: rename handle_object function

The handle_object function is rather vaguely named; it only
operates on blobs, and its purpose is to export the blob to
the output stream. Let's call it "export_blob" to make it
more clear what it does.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
fast-export: do not load blob objects twice

When fast-export wants to export a blob object, it first
calls parse_object to get a "struct object" and check
whether we have already shown the object.  If we haven't
shown it, we then use read_sha1_file to pull it from disk
and write it out.

That means we load each blob from disk twice: once for
parse_object to find its type and check its sha1, and a
second time when we actually output it. We can drop this to
a single load by using lookup_object to check the SHOWN
flag, and then checking the signature on and outputting a
single buffer.

This provides modest speedups on git.git (best-of-five, "git
fast-export HEAD >/dev/null"):

  [before]                [after]
  real    0m14.347s       real    0m13.780s
  user    0m14.084s       user    0m13.620s
  sys     0m0.208s        sys     0m0.100s

and somewhat more on more blob-heavy repos (this is a
repository full of media files):

  [before]                [after]
  real    0m52.236s       real    0m44.451s
  user    0m50.568s       user    0m43.000s
  sys     0m1.536s        sys     0m1.284s

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'nd/count-garbage' into jch

Will cook in 'next'.

* nd/count-garbage:
  count-objects: report how much disk space taken by garbage files
  count-objects: report garbage files in pack directory too
  sha1_file: reorder code in prepare_packed_git_one()
  git-count-objects.txt: describe each line in -v output
Merge branch 'jc/fetch-raw-sha1' into jch

Allows requests to fetch objects at any tip of refs (including
hidden ones).  It seems that there may be use cases even outside
Gerrit (e.g. $gmane/215701).

Will cook in 'next'.

* jc/fetch-raw-sha1:
  fetch: fetch objects by their exact SHA-1 object names
  upload-pack: optionally allow fetching from the tips of hidden refs
  fetch: use struct ref to represent refs to be fetched
  parse_fetch_refspec(): clarify the codeflow a bit

Conflicts:
	cache.h
Merge branch 'jc/remove-treesame-parent-in-simplify-merges' into jch

The --simplify-merges logic did not cull irrelevant parents from a
merge that is otherwise not interesting with respect to the paths
we are following.

This touches a fairly core part of the revision traversal
infrastructure; even though I think this change is correct, please
report immediately if you find any unintended side effect.

Will cook in 'next'.

* jc/remove-treesame-parent-in-simplify-merges:
  simplify-merges: drop merge from irrelevant side branch

Conflicts:
	revision.c
Merge branch 'jc/remove-export-from-config-mak-in' into jch

config.mak.in template had an "export" line to cause a few
common makefile variables to be exported; if they need to be
expoted for autoconf/configure users, they should also be exported
for people who write config.mak the same way.  Move the "export" to
the main Makefile.  Also, stop exporting mandir that used to be
exported (only) when config.mak.autogen was used.  It would have
broken installation of manpages (but not other documentation
formats).

Will cook in 'next'.

* jc/remove-export-from-config-mak-in:
  Fix `make install` when configured with autoconf
  Makefile: do not export mandir/htmldir/infodir
  config.mak.in: remove unused definitions

Conflicts:
	config.mak.in
Merge branch 'nd/read-directory-recursive-optim' into jch

"git status" has been optimized by taking advantage of the fact
that paths that are already known to the index do not have to be
checked against the .gitignore mechanism under some conditions.

Will cook in 'next'.

* nd/read-directory-recursive-optim:
  read_directory: avoid invoking exclude machinery on tracked files
Merge branch 'jc/push-2.0-default-to-simple' into jch

Will cook in 'next' until Git 2.0.

* jc/push-2.0-default-to-simple:
  t5570: do not assume the "matching" push is the default
  t5551: do not assume the "matching" push is the default
  t5550: do not assume the "matching" push is the default
  doc: push.default is no longer "matching"
  push: switch default from "matching" to "simple"
  t9401: do not assume the "matching" push is the default
  t9400: do not assume the "matching" push is the default
  t7406: do not assume the "matching" push is the default
  t5531: do not assume the "matching" push is the default
  t5519: do not assume the "matching" push is the default
  t5517: do not assume the "matching" push is the default
  t5516: do not assume the "matching" push is the default
  t5505: do not assume the "matching" push is the default
  t5404: do not assume the "matching" push is the default

Conflicts:
	Documentation/config.txt
	builtin/push.c
Merge branch 'bc/commit-complete-lines-given-via-m-option' into jch

'git commit -m "$str"' when $str was already terminated with a LF
now avoids adding an extra LF to the message.

Will cook in 'next'.

* bc/commit-complete-lines-given-via-m-option:
  Documentation/git-commit.txt: rework the --cleanup section
  git-commit: only append a newline to -m mesg if necessary
  t7502: demonstrate breakage with a commit message with trailing newlines
  t/t7502: compare entire commit message with what was expected
Merge branch 'ob/imap-send-ssl-verify' into jch

Correctly connect to SSL/TLS sites that serve multiple hostnames on
a single IP by including Server Name Indication in the client-hello.

Will cook in 'next'.

* ob/imap-send-ssl-verify:
  imap-send: support Server Name Indication (RFC4366)
Merge branch 'jk/pkt-line-cleanup' into jch

Cleans up pkt-line API, implementation and its callers to make
them more robust.

Will cook in 'next'.

* jk/pkt-line-cleanup:
  remote-curl: always parse incoming refs
  remote-curl: move ref-parsing code up in file
  remote-curl: pass buffer straight to get_remote_heads
  teach get_remote_heads to read from a memory buffer
  pkt-line: share buffer/descriptor reading implementation
  pkt-line: provide a LARGE_PACKET_MAX static buffer
  pkt-line: move LARGE_PACKET_MAX definition from sideband
  pkt-line: teach packet_read_line to chomp newlines
  pkt-line: provide a generic reading function with options
  pkt-line: drop safe_write function
  pkt-line: move a misplaced comment
  write_or_die: raise SIGPIPE when we get EPIPE
  upload-archive: use argv_array to store client arguments
  upload-archive: do not copy repo name
  send-pack: prefer prefixcmp over memcmp in receive_status
  fetch-pack: fix out-of-bounds buffer offset in get_ack
  upload-pack: remove packet debugging harness
  upload-pack: do not add duplicate objects to shallow list
  upload-pack: use get_sha1_hex to parse "shallow" lines

Conflicts:
	builtin/fetch-pack.c
Merge branch 'da/difftool-fixes' into jch

Minor maintenance updates to difftool, and updates to its tests.

Will cook in 'next'.

* da/difftool-fixes:
  t7800: "defaults" is no longer a builtin tool name
  t7800: modernize tests
  t7800: update copyright notice
  difftool: silence uninitialized variable warning
Merge branch 'rt/commit-cleanup-config' into jch

Fix tests that contaminated their environments and affected new
tests introduced later in the sequence by containing their effects
in their own subshells.

Will cook in 'next'.

* rt/commit-cleanup-config:
  t7502: perform commits using alternate editor in a subshell