index-pack: fix race condition with duplicate bases

When we are resolving deltas in an indexed pack, we do it by
first selecting a potential base (either one stored in full
in the pack, or one created by resolving another delta), and
then resolving any deltas that use that base.  When we
resolve a particular delta, we flip its "real_type" field
from OBJ_{REF,OFS}_DELTA to whatever the real type is.

We assume that traversing the objects this way will visit
each delta only once. This is correct for most packs; we
visit the delta only when we process its base, and each
object (and thus each base) appears only once. However, if a
base object appears multiple times in the pack, we will try
to resolve any deltas based on it once for each instance.

We can detect this case by noting that a delta we are about
to resolve has already had its real_type field flipped, and
we already do so with an assert().  However, if multiple
threads are in use, we may race with another thread on
comparing and flipping the field. We need to synchronize the
access.

The right mechanism for doing this is a compare-and-swap (we
atomically "claim" the delta for our own and find out
whether our claim was successful). We can implement this
in C by using a pthread mutex to protect the operation. This
is not the fastest way of doing a compare-and-swap; many
processors provide instructions for this, and gcc and other
compilers provide builtins to access them. However, some
experiments showed that lock contention does not cause a
significant slowdown here. Adding c-a-s support for many
compilers would increase the maintenance burden (and we
would still end up including the pthread version as a
fallback).

Note that we only need to touch the OBJ_REF_DELTA codepath
here. An OBJ_OFS_DELTA object points to its base using an
offset, and therefore has only one base, even if another
copy of that base object appears in the pack (we do still
touch it briefly because the setting of real_type is
factored out of resolve_data).

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
index-pack: handle duplicate base objects gracefully

If a pack contains duplicates of an object, and if that
object has any deltas pointing at it with REF_DELTA, we will
try to resolve the deltas twice. While unusual, this is not
strictly an error, but we currently die() with an unhelpful
message.  We should instead silently ignore the delta and
move on. The first resolution already filled in any data we
needed (like the sha1).

The duplicate base object is not our concern during the
resolution phase, and the .idx-writing phase will decide
whether to complain or allow it (based on whether --strict
is set).

Based-on-work-by: René Scharfe <l.s.r@web.de>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'ta/config-set' into jch

Add in-core caching layer to let us avoid reading the same
configuration files number of times.

* ta/config-set:
  test-config: add tests for the config_set API
  add `config_set` API for caching config-like files
Merge branch 'jk/pretty-empty-format' into jch

"git log --pretty/format=" with an empty format string did not mean
the more obvious "No output whatsoever" but "Use default format",
which was counterintuitive.

* jk/pretty-empty-format:
  pretty: make empty userformats truly empty
  pretty: treat "--format=" as an empty userformat
  revision: drop useless string offset when parsing "--pretty"
Merge branch 'pr/remotes-in-hashmap' into jch

Optimize remotes configuration look-up in a repository with very
many remotes defined.

* pr/remotes-in-hashmap:
  use a hashmap to make remotes faster
Merge branch 'bc/archive-pax-header-mode' into jch

Implementations of "tar" that do not understand an extended pax
header would extract the contents of it in a regular file; make
sure the permission bits of this file follows the same tar.umask
configuration setting.

* bc/archive-pax-header-mode:
  archive: honor tar.umask even for pax headers
Merge branch 'ta/pretty-parse-config' into jch

* ta/pretty-parse-config:
  pretty.c: make git_pretty_formats_config return -1 on git_config_string failure
Merge branch 'jc/reopen-lock-file' into jch

Needed in the "commit -p" code path to update the cache tree in the
index (e.g. for dt/cache-tree-repair topic).

* jc/reopen-lock-file:
  lockfile: allow reopening a closed but still locked file
Merge branch 'dt/cache-tree-repair' into jch

Add a few more places in "commit" and "checkout" that make sure
that the cache-tree is fully populated in the index.

* dt/cache-tree-repair:
  cache-tree: Write updated cache-tree after commit
  cache-tree: subdirectory tests
  test-dump-cache-tree: invalid trees are not errors
  cache-tree: create/update cache-tree on checkout

# Conflicts:
# 	builtin/checkout.c
# 	builtin/commit.c
Merge branch 'mm/config-edit-global' into jch

Start "git config --edit --global" from a skeletal per-user
configuration file contents, instead of a total blank, when the
user does not already have any.  This immediately reduces the need
for a later "Have you forgotten setting core.user?" and we can add
more to the template as we gain more experience.

* mm/config-edit-global:
  commit: advertise config --global --edit on guessed identity
  home_config_paths(): let the caller ignore xdg path
  config --global --edit: create a template file if needed
Merge branch 'rs/init-no-duplicate-real-path' into jch

* rs/init-no-duplicate-real-path:
  init: avoid superfluous real_path() calls
Merge branch 'rs/strbuf-getcwd' into jch

Reduce the use of fixed sized buffer passed to getcwd() calls
by introducing xgetcwd() helper.

* rs/strbuf-getcwd:
  use strbuf_add_absolute_path() to add absolute paths
  abspath: convert absolute_path() to strbuf
  use xgetcwd() to set $GIT_DIR
  use xgetcwd() to get the current directory or die
  wrapper: add xgetcwd()
  abspath: convert real_path_internal() to strbuf
  abspath: use strbuf_getcwd() to remember original working directory
  setup: convert setup_git_directory_gently_1 et al. to strbuf
  unix-sockets: use strbuf_getcwd()
  strbuf: add strbuf_getcwd()
Merge branch 'bc/imap-send-doc' into jch

* bc/imap-send-doc:
  imap-send doc: omit confusing "to use imap-send" modifier
Merge branch 'jk/command-line-config-empty-string' into jch

"git -c section.var command" and "git -c section.var= command"
should pass the configuration differently (the former should be
a boolean true, the latter should be an empty string).

* jk/command-line-config-empty-string:
  config: teach "git -c" to recognize an empty string

# Conflicts:
# 	config.c
Merge branch 'jc/apply-ws-prefix' into jch

Applying a patch not generated by Git in a subdirectory used to
check the whitespace breakage using the attributes for incorrect
paths. Also whitespace checks were performed even for paths
excluded via "git apply --exclude=<path>" mechanism.

* jc/apply-ws-prefix:
  apply: omit ws check for excluded paths
  apply: hoist use_patch() helper for path exclusion up
  apply: use the right attribute for paths in non-Git patches

# Conflicts:
# 	builtin/apply.c
Merge branch 'lf/bundle-exclusion' into jch

* lf/bundle-exclusion:
  bundle: fix exclusion of annotated tags
Merge branch 'mm/log-branch-desc-plug-leak' into jch

* mm/log-branch-desc-plug-leak:
  builtin/log.c: fix minor memory leak
Merge branch 'jk/stash-list-p' into jch

Teach "git stash list -p" to show the difference between the base
commit version and the working tree version, which is in line with
what "git show" gives.

* jk/stash-list-p:
  stash: default listing to working-tree diff
Merge branch 'la/init-doc' into jch

* la/init-doc:
  Documentation: git-init: flesh out example
  Documentation: git-init: template directory: reword and cross-reference
  Documentation: git-init: reword parenthetical statements
  Documentation: git-init: --separate-git-dir: clarify
  Documentation: git-init: template directory: reword
  Documentation: git-init: list items facelift
  Documentation: git-init: typographical fixes
Merge branch 'tf/imap-send-create' into jch

* tf/imap-send-create:
  imap-send: create target mailbox if it is missing
  imap-send: clarify CRAM-MD5 vs LOGIN documentation