has_sha1_file: re-check pack directory before giving up

When we read a sha1 file, we first look for a packed
version, then a loose version, and then re-check the pack
directory again before concluding that we cannot find it.
This lets us handle a process that is writing to the
repository simultaneously (e.g., receive-pack writing a new
pack followed by a ref update, or git-repack packing
existing loose objects into a new pack).

However, we do not do the same trick with has_sha1_file; we
only check the packed objects once, followed by loose
objects. This means that we might incorrectly report that we
do not have an object, even though we could find it if we
simply re-checked the pack directory.

By itself, this is usually not a big deal. The other process
is running simultaneously, so we may run has_sha1_file
before it writes, anyway. It is a race whether we see the
object or not.  However, we may also see other things
the writing process has done (like updating refs); and in
that case, we must be able to also see the new objects.

For example, imagine we are doing a for_each_ref iteration,
and somebody simultaneously pushes. Receive-pack may write
the pack and update a ref after we have examined the
objects/pack directory, but before the iteration gets to the
updated ref. When we do finally see the updated ref,
for_each_ref will call has_sha1_file to check whether the
ref is broken. If has_sha1_file returns the wrong answer, we
erroneously will think that the ref is broken.

In the case of git-fsck, which uses the
DO_FOR_EACH_INCLUDE_BROKEN flag, this will cause us to
erroneously complain that the ref points to an invalid
object. But for git-repack, which does not use that flag, we
will skip the ref completely! So not only will we fail to
process the new objects that the ref points to (which is
acceptabale, since the processes are running simultaneously,
and we might well do our whole repack before the other
process updates the ref), but we will not see the ref at
all. Its old objects may be omitted from the pack (and even
lost, if --unpack-unreachable is used with an expiration
time).

There's no test included here, because the success case is
two processes running simultaneously forever. But you can
replicate the issue with:

  # base.sh
  # run this in one terminal; it creates and pushes
  # repeatedly to a repository
  git init parent &&
  (cd parent &&

    # create a base commit that will trigger us looking at
    # the objects/pack directory before we hit the updated ref
    echo content >file &&
    git add file &&
    git commit -m base &&

    # set the unpack limit abnormally low, which
    # lets us simulate full-size pushes using tiny ones
    git config receive.unpackLimit 1
  ) &&
  git clone parent child &&
  cd child &&
  n=0 &&
  while true; do
    echo $n >file && git add file && git commit -m $n &&
    git push origin HEAD:refs/remotes/child/master &&
    n=$(($n + 1))
  done

  # fsck.sh
  # now run this simultaneously in another terminal; it
  # repeatedly fscks, looking for us to consider the
  # newly-pushed ref broken.
  cd parent &&
  while true; do
    broken=`git fsck 2>&1 | grep remotes/child`
    if test -n "$broken"; then
      echo $broken
      exit 1
    fi
  done

Without this patch, the fsck loop fails within a few seconds
(and almost instantly if the test repository actually has a
large number of refs). With it, the two can run
indefinitely.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
repack: rewrite the shell script in C

The motivation of this patch is to get closer to a goal of being
able to have a core subset of git functionality built in to git.
That would mean

 * people on Windows could get a copy of at least the core parts
   of Git without having to install a Unix-style shell

 * people using git in on servers with chrooted environments
   do not need to worry about standard tools lacking for shell
   scripts.

This patch is meant to be mostly a literal translation of the
git-repack script; the intent is that later patches would start using
more library facilities, but this patch is meant to be as close to a
no-op as possible so it doesn't do that kind of thing.

Signed-off-by: Stefan Beller <stefanbeller@googlemail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
repack: retain the return value of pack-objects

During the review process of the previous commit (repack: rewrite the
shell script in C), Johannes Sixt proposed to retain any exit codes from
the sub-process, which makes it probably more obvious in case of failure.

As the commit before should behave as close to the original shell
script, the proposed change is put in this extra commit.
The infrastructure however was already setup in the previous commit.
(Having a local 'ret' variable)

Signed-off-by: Stefan Beller <stefanbeller@googlemail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
remote-hg: use notes to keep track of Hg revisions

Keep track of Mercurial revisions as Git notes under the 'refs/notes/hg'
ref.  This way, the user can easily see which Mercurial revision
corresponds to certain Git commit.

Unfortunately, there's no way to efficiently update the notes after
doing an export (push), so they'll have to be updated when importing
(fetching).

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
contrib/remote-helpers: style updates for test scripts

During the review of the main series it was noticed that these test
scripts can use updates to conform to our coding style better, but
fixing the style should be done in a patch separate from the main
series.

This updates the test-*.sh scripts only for styles:

 * We do not leave SP between a redirection operator and the
   filename;

 * We change line before "then", "do", etc. rather than terminating
   the condition for "if"/"while" and list for "for" with a
   semicolon;

 * When HERE document does not use any expansion, we quote the end
   marker (e.g. "cat <<\EOF" not "cat <<EOF") to signal the readers
   that there is no funny substitution to worry about when reading
   the code.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
contrib/remote-helpers: quote variable references in redirection targets

Even though it is not required by POSIX to double-quote the
redirection target in a variable, our code does so because some
versions of bash issue a warning without the quotes.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'nd/sq-quote-buf' into jch

Code simplification as a preparatory step to something larger.

* nd/sq-quote-buf:
  quote: remove sq_quote_print()
  tar-tree: remove dependency on sq_quote_print()
  for-each-ref, quote: convert *_quote_print -> *_quote_buf
Merge branch 'mm/no-shell-escape-in-die-message' into jch

Fixes a minor bug in "git rebase -i" (there could be others, as the
root cause is pretty generic) where the code feeds a random, data
dependeant string to 'echo' and expects it to come out literally.

* mm/no-shell-escape-in-die-message:
  die_with_status: use "printf '%s\n'", not "echo"
Merge branch 'jc/parseopt-command-modes' into jch

Many commands use --dashed-option as a operation mode selector
(e.g. "git tag --delete") that the user can use at most one
(e.g. "git tag --delete --verify" is a nonsense) and you cannot
negate (e.g. "git tag --no-delete" is a nonsense).  Make it easier
for users of parse_options() to enforce these restrictions.

* jc/parseopt-command-modes:
  tag: use OPT_CMDMODE
  parse-options: add OPT_CMDMODE()
Merge branch 'sb/parseopt-boolean-removal' into jch

Convert most uses of OPT_BOOLEAN/OPTION_BOOLEAN that can use
OPT_BOOL/OPTION_BOOLEAN which have much saner semantics, and turn
remaining ones into OPT_SET_INT, OPT_COUNTUP, etc. as necessary.

* sb/parseopt-boolean-removal:
  revert: use the OPT_CMDMODE for parsing, reducing code
  checkout-index: fix negations of even numbers of -n
  config parsing options: allow one flag multiple times
  hash-object: replace stdin parsing OPT_BOOLEAN by OPT_COUNTUP
  branch, commit, name-rev: ease up boolean conditions
  checkout: remove superfluous local variable
  log, format-patch: parsing uses OPT__QUIET
  Replace deprecated OPT_BOOLEAN by OPT_BOOL
  Remove deprecated OPTION_BOOLEAN for parsing arguments
Merge branch 'ap/remote-hg-tilde-is-home-directory' into jch

* ap/remote-hg-tilde-is-home-directory:
  remote-hg: fix path when cloning with tilde expansion
Merge branch 'sb/misc-cleanup' into jch

* sb/misc-cleanup:
  rm: remove unneeded null pointer check
  diff: fix a possible null pointer dereference
  diff: remove ternary operator evaluating always to true
Merge branch 'sb/diff-delta-remove-needless-comparison' into jch

* sb/diff-delta-remove-needless-comparison:
  create_delta_index: simplify condition always evaluating to true
Merge branch 'sg/bash-prompt-lf-in-cwd-test' into jch

* sg/bash-prompt-lf-in-cwd-test:
  bash prompt: test the prompt with newline in repository path
Merge branch 'jl/some-submodule-config-are-not-boolean' into jch

* jl/some-submodule-config-are-not-boolean:
  avoid segfault on submodule.*.path set to an empty "true"
Merge branch 'km/svn-1.8-serf-only' into jch

Subversion 1.8.0 that was recently released breaks older subversion
clients coming over http/https in various ways.

* km/svn-1.8-serf-only:
  Git.pm: revert _temp_cache use of temp_is_locked
  git-svn: allow git-svn fetching to work using serf
  Git.pm: add new temp_is_locked function
Merge branch 'jc/check-x-z' into jch

"git check-ignore -z" applied the NUL termination to both its input
(with --stdin) and its output, but "git check-attr -z" ignored the
option on the output side.

This is potentially a backward incompatible fix.  Let's see if
anybody screams before deciding if we want to do anything to help
existing users (there may be none).

* jc/check-x-z:
  check-attr -z: a single -z should apply to both input and output
  check-ignore -z: a single -z should apply to both input and output
  check-attr: the name of the character is NUL, not NULL
  check-ignore: the name of the character is NUL, not NULL

# Conflicts:
# 	builtin/check-attr.c
# 	builtin/check-ignore.c
Merge branch 'rr/feed-real-path-to-editor' into jch

* rr/feed-real-path-to-editor:
  editor: use canonicalized absolute path
Merge branch 'jk/fast-import-empty-ls' into jch

* jk/fast-import-empty-ls:
  fast-import: allow moving the root tree
  fast-import: allow ls or filecopy of the root tree
  fast-import: set valid mode on root tree in "ls"
  t9300: document fast-import empty path issues
Merge branch 'da/darwin' into jch

* da/darwin:
  OS X: Fix redeclaration of die warning
  Makefile: Fix APPLE_COMMON_CRYPTO with BLK_SHA1
  imap-send: use Apple's Security framework for base64 encoding