fast-export: do not parse non-commit objects while reading marks file

We read from the marks file and keep only marked commits, but in
order to find the type of object, we are parsing the whole thing,
which is slow, specially in big repositories with lots of big files.

There's no need for that, we can query the object information with
sha1_object_info().

Before this, loading the objects of a fresh emacs import, with 260598
blobs took 14 minutes, after this patch, it takes 3 seconds.

This is the way fast-import does it. Also die if the object is not
found (like fast-import).

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
fast-export: don't parse commits while reading marks file

We don't need the parsed objects at this point, merely the
information that they have marks.

Seems to be three times faster in my setup with lots of objects.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
clone: allow cloning local paths with colons in them

Usually "foo:bar" is interpreted as an ssh url. This patch allows to
clone from such paths by putting at least one slash before the colon
(i.e. /path/to/foo:bar or just ./foo:bar).

file://foo:bar should also work, but local optimizations are off in
that case, which may be unwanted. While at there, warn the users about
--local being ignored in this case.

Reported-by: William Giokas <1007380@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
get_sha1: improve ambiguity warning regarding SHA-1 and ref names

When we get 40 hex digits, we immediately assume it's an SHA-1. Warn
about ambiguity if there's also refs/heads/$sha1 (or similar) on system.

When we successfully resolve a ref like "1234abc" and "1234abc"
happens to be valid abbreviated SHA-1 on system, warn also.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t1514: Add tests of shortening refnames in strict/loose mode

These tests verify the correct behavior of "git rev-parse --abbrev-ref"
in both "strict" and "loose" modes. Really, it tests the correct behavior
of refs.c:shorten_unambiguous_ref() with its 'strict' argument set to
either true of false.

Signed-off-by: Johan Herland <johan@herland.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t1514: Demonstrate failure to correctly shorten "refs/remotes/origin/HEAD"

There is currently a bug in refs.c:shorten_unambiguous_ref() that causes
"refs/remotes/origin/HEAD" to be shortened to "origin/HEAD" instead of
"origin" (which is expected from matching against the "refs/remotes/%.*s"
pattern from refs.c:ref_rev_parse_rules).

Signed-off-by: Johan Herland <johan@herland.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
shorten_unambiguous_ref(): Fix shortening refs/remotes/origin/HEAD to origin

When expanding shorthand refs to full ref names (e.g. in dwim_ref()),
we use the ref_rev_parse_rules list of expansion patterns. This list
allows "origin" to be expanded into "refs/remotes/origin/HEAD", by
using the "refs/remotes/%.*s/HEAD" pattern from that list.

shorten_unambiguous_ref() exists to provide the reverse operation:
turning a full ref name into a shorter (but still unambiguous) name.
It does so by matching the given refname against each pattern from
the ref_rev_parse_rules list (in reverse), and extracting the short-
hand name from the matching rule.

However, when given "refs/remotes/origin/HEAD" it fails to shorten it
into "origin", because we misuse the sscanf() function when matching
"refs/remotes/origin/HEAD" against "refs/remotes/%.*s/HEAD": We end
up calling sscanf like this:

  sscanf("refs/remotes/origin/HEAD", "refs/remotes/%s/HEAD", short_name)

In this case, sscanf() will match the initial "refs/remotes/" part, and
then match the remainder of the refname against the "%s", and place it
("origin/HEAD") into short_name. The part of the pattern following the
"%s" format is never verified, because sscanf() apparently does not
need to do that (it has performed the one expected format extraction,
and will return 1 correspondingly; see [1] for more details).

This patch replaces the misuse of sscanf() with a fairly simple function
that manually matches the refname against patterns, and extracts the
shorthand name.

[1]: If we assume that sscanf() does not do a verification pass prior
to format extraction, there is AFAICS _no_ way for sscanf() - having
already done one or more format extractions - to indicate to its caller
that the input fails to match the trailing part of the format string.
In other words, AFAICS, the scanf() family of function will only verify
matching input up to and including the last format specifier in the
format string. Any data following the last format specifier will not be
verified. Yet another reason to consider the scanf functions harmful...

Cc: Bert Wesarg <bert.wesarg@googlemail.com>
Improved-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Johan Herland <johan@herland.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'fc/transport-helper-error-reporting' into jch

Update transport helper to report errors and maintain ref hierarchy
used to keep track of remote helper state better.

* fc/transport-helper-error-reporting:
  t5801: "VAR=VAL shell_func args" is forbidden
  transport-helper: update remote helper namespace
  transport-helper: trivial code shuffle
  transport-helper: warn when refspec is not used
  transport-helper: clarify pushing without refspecs
  transport-helper: update refspec documentation
  transport-helper: clarify *:* refspec
  transport-helper: improve push messages
  transport-helper: mention helper name when it dies
  transport-helper: report errors properly

# Conflicts:
# 	t/t5801-remote-helpers.sh
Merge branch 'js/transport-helper-error-reporting-fix' into jch

Finishing touches to fc/transport-helper-error-reporting topic.

* js/transport-helper-error-reporting-fix:
  git-remote-testgit: build it to run under $SHELL_PATH
  git-remote-testgit: further remove some bashisms
  git-remote-testgit: avoid process substitution
Merge branch 'mh/fetch-into-shallow' into jch

* mh/fetch-into-shallow:
  t5500: add test for fetching with an unknown 'shallow'
  upload-pack: ignore 'shallow' lines with unknown obj-ids
Merge branch 'tr/line-log' into jch

* tr/line-log:
  git-log(1): remove --full-line-diff description
  line-log: fix documentation formatting
  log -L: improve comments in process_all_files()
  log -L: store the path instead of a diff_filespec
  log -L: test merge of parallel modify/rename
  t4211: pass -M to 'git log -M -L...' test
  log -L: fix overlapping input ranges
  log -L: check range set invariants when we look it up
  Speed up log -L... -M
  log -L: :pattern:file syntax to find by funcname
  Implement line-history search (git log -L)
  Export rewrite_parents() for 'log -L'
  Refactor parse_loc

# Conflicts:
# 	Documentation/git-log.txt
# 	log-tree.c
Merge branch 'jc/prune-all' into jch

We used the approxidate() parser for "--expire=<timestamp>" options
of various commands, but it is better to treat --expire=all and
--expire=now a bit more specially than using the current timestamp.
Update "git gc" and "git reflog" with a new parsing function for
expiry dates.

* jc/prune-all:
  prune: introduce OPT_EXPIRY_DATE() and use it
  api-parse-options.txt: document "no-" for non-boolean options
  git-gc.txt, git-reflog.txt: document new expiry options
  date.c: add parse_expiry_date()
Merge branch 'jl/submodule-mv' into jch

"git mv A B" when moving a submodule A does "the right thing",
inclusing relocating its working tree and adjusting the paths in
the .gitmodules file.

* jl/submodule-mv:
  submodule.c: duplicate real_path's return value
  rm: delete .gitmodules entry of submodules removed from the work tree
  Teach mv to update the path entry in .gitmodules for moved submodules
  Teach mv to move submodules using a gitfile
  Teach mv to move submodules together with their work trees
Merge branch 'jk/submodule-subdirectory-ok' into jch

Allow various subcommands of "git submodule" to be run not from the
top of the working tree of the superproject.

* jk/submodule-subdirectory-ok:
  submodule: fix quoting in relative_path()
  submodule: drop the top-level requirement
  rev-parse: add --prefix option
Merge branch 'jh/checkout-auto-tracking' into jch

Updates "git checkout foo" that DWIMs the intended "upstream" and
turns it into "git checkout -t -b foo remotes/origin/foo" to
correctly take existing remote definitions into account.  The
remote "origin" may be what uniquely map its own branch to
remotes/some/where/foo but that some/where may not be "origin".

* jh/checkout-auto-tracking:
  glossary: Update and rephrase the definition of a remote-tracking branch
  branch.c: Validate tracking branches with refspecs instead of refs/remotes/*
  t9114.2: Don't use --track option against "svn-remote"-tracking branches
  t7201.24: Add refspec to keep --track working
  t3200.39: tracking setup should fail if there is no matching refspec.
  checkout: Use remote refspecs when DWIMming tracking branches
  t2024: Show failure to use refspec when DWIMming remote branch names
  t2024: Add tests verifying current DWIM behavior of 'git checkout <branch>'
Merge branch 'as/check-ignore' into jch

Enhance "check-ignore" (1.8.2 update) to work more like "check-attr"
over bidi-pipes.

* as/check-ignore:
  t0008: use named pipe (FIFO) to test check-ignore streaming
  Documentation: add caveats about I/O buffering for check-{attr,ignore}
  check-ignore: allow incremental streaming of queries via --stdin
  check-ignore: move setup into cmd_check_ignore()
  check-ignore: add -n / --non-matching option
  t0008: remove duplicated test fixture data

# Conflicts:
# 	builtin/check-ignore.c
Merge branch 'mh/packed-refs-various' into jch

Updates reading and updating packed-refs file, correcting corner
case bugs.

* mh/packed-refs-various: (33 commits)
  refs: handle the main ref_cache specially
  refs: change do_for_each_*() functions to take ref_cache arguments
  pack_one_ref(): do some cheap tests before a more expensive one
  pack_one_ref(): use write_packed_entry() to do the writing
  pack_one_ref(): use function peel_entry()
  refs: inline function do_not_prune()
  pack_refs(): change to use do_for_each_entry()
  refs: use same lock_file object for both ref-packing functions
  pack_one_ref(): rename "path" parameter to "refname"
  pack-refs: merge code from pack-refs.{c,h} into refs.{c,h}
  pack-refs: rename handle_one_ref() to pack_one_ref()
  refs: extract a function write_packed_entry()
  repack_without_ref(): write peeled refs in the rewritten file
  t3211: demonstrate loss of peeled refs if a packed ref is deleted
  refs: change how packed refs are deleted
  search_ref_dir(): return an index rather than a pointer
  repack_without_ref(): silence errors for dangling packed refs
  t3210: test for spurious error messages for dangling packed refs
  refs: change the internal reference-iteration API
  refs: extract a function peel_entry()
  ...
Merge branch 'rj/sparse' into jch

* rj/sparse:
  sparse: Fix mingw_main() argument number/type errors
  compat/mingw.c: Fix some sparse warnings
  compat/win32mmap.c: Fix some sparse warnings
  compat/poll/poll.c: Fix a sparse warning
  compat/win32/pthread.c: Fix a sparse warning
  compat/unsetenv.c: Fix a sparse warning
  compat/nedmalloc: Fix compiler warnings on linux
  compat/nedmalloc: Fix some sparse warnings
  compat/fnmatch/fnmatch.c: Fix a sparse error
  compat/regex/regexec.c: Fix some sparse warnings
Merge branch 'jk/test-output' into jch

When TEST_OUTPUT_DIRECTORY setting is used, it was handled somewhat
inconsistently between the test framework and t/Makefile, and logic
to summarize the results looked at a wrong place.

* jk/test-output:
  t/Makefile: don't define TEST_RESULTS_DIRECTORY recursively
  test output: respect $TEST_OUTPUT_DIRECTORY
  t/Makefile: fix result handling with TEST_OUTPUT_DIRECTORY
Merge branch 'jk/subtree-do-not-push-if-split-fails' into jch

"git subtree" (in contrib/) had one codepath with loose error
checks to lose data at the remote side.

* jk/subtree-do-not-push-if-split-fails:
  contrib/subtree: don't delete remote branches if split fails