fetch-pack: avoid quadratic list insertion in mark_complete

We insert the commit pointed to by each ref one-by-one into
the "complete" commit_list using insert_by_date. Because
each insertion is O(n), we end up with O(n^2) behavior.

This typically doesn't matter, because the number of refs is
reasonably small. And even if there are a lot of refs, they
often point to a smaller set of objects (in which case the
optimization in commit ea5f220 keeps our "n" small).

However, in pathological repositories (hundreds of thousands
of refs, each pointing to a unique commit), this quadratic
behavior can make a difference. Since we do not care about
the list order until we have finished building it, we can
simply keep it unsorted during the insertion phase, then
sort it afterwards.

On a repository like the one described above, this dropped
the time to do a no-op fetch from 2.0s to 1.7s. On normal
repositories, it probably does not matter at all, but it
does not hurt to protect ourselves from pathological cases.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
commit.c: make compare_commits_by_commit_date global

This helper function was introduced as a prio_queue
comparator to help topological sorting. However, other users
of prio_queue who want to replace commit_list_insert_by_date
will want to use it, too. So let's make it public.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
fetch-pack: avoid quadratic behavior in rev_list_push

When we call find_common to start finding common ancestors
with the remote side of a fetch, the first thing we do is
insert the tip of each ref into our rev_list linked list. We
keep the list sorted the whole time with
commit_list_insert_by_date, which means our insertion ends
up doing O(n^2) timestamp comparisons.

We could teach rev_list_push to use an unsorted list, and
then sort it once after we have added each ref. However, in
get_rev, we process the list by popping commits off the
front and adding parents back in timestamp-sorted order. So
that procedure would still operate on the large list.

Instead, we can replace the linked list with a heap-based
priority queue, which can do O(log n) insertion, making the
whole insertion procedure O(n log n).

As a result of switching to the prio_queue struct, we fix
two minor bugs:

  1. When we "pop" a commit in get_rev, and when we clear
     the rev_list in find_common, we do not take care to
     free the "struct commit_list", and just leak its
     memory. With the prio_queue implementation, the memory
     management is handled for us.

  2. In get_rev, we look at the head commit of the list,
     possibly push its parents onto the list, and then "pop"
     the front of the list off, assuming it is the same
     element that we just peeked at. This is typically going
     to be the case, but would not be in the face of clock
     skew: the parents are inserted by date, and could
     potentially be inserted at the head of the list if they
     have a timestamp newer than their descendent. In this
     case, we would accidentally pop the parent, and never
     process it at all.

     The new implementation pulls the commit off of the
     queue as we examine it, and so does not suffer from
     this problem.

With this patch, a fetch of a single commit into a
repository with 50,000 refs went from:

  real    0m7.984s
  user    0m7.852s
  sys     0m0.120s

to:

  real    0m2.017s
  user    0m1.884s
  sys     0m0.124s

Before this patch, a larger case with 370K refs still had
not completed after tens of minutes; with this patch, it
completes in about 12 seconds.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
merge: handle --ff/--no-ff/--ff-only as a tri-state option

These three options mean "favor fast-forwarding when possible,
without creating an unnecessary merge", "never fast-forward and
always create a merge commit even when the commit being merged is a
strict descendant", and "we do not want to create any merge commit;
update only when the merged commit is a strict descendant".

They are "pick one out of these three possibilities" options, and
correspond to "merge.ff" configuration that is tri-state (yes, no
and only).

However, the implementation did not follow the usual convention for
the command line options (later one wins, and command line overrides
what is in the configuration).

Fix this by consolidating two variables (fast_forward_only and
allow_fast_forward) used in the implementation into one enum that
can take one of the three possible values.

Signed-off-by: Miklos Vajna <vmiklos@suse.cz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
submodule update: allow custom command to update submodule working tree

Users can set submodule.$name.update to '!command' which will cause
'command' to be run instead of checkout/merge/rebase. This allows
the user finer-grained control over how the update is done.

The primary motivation for this was interoperability with stgit;
however being able to intercept the submodule update process may
prove useful for integrating with or extending other tools.

Signed-off-by: Chris Packham <judge.packham@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add --depth to submodule update/add

Add the --depth option to the add and update commands of "git submodule",
which is then passed on to the clone command. This is useful when the
submodule(s) are huge and you're not really interested in anything but
the latest commit.

Tests are added and some indention adjustments were made to conform to the
rest of the testfile on "submodule update can handle symbolic links in pwd".

Signed-off-by: Fredrik Gustafsson <iveqy@iveqy.com>
Acked-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
pretty.c: drop const-ness from pretty_print_context

In the current code, callers are expected to fill in the
pretty_print_context, and then the pretty.c functions simply
read from it. This leaves no room for the pretty.c functions
to communicate with each other by manipulating the context
(e.g., data seen while printing the header may impact how we
print the body).

Rather than introduce a new struct to hold modifiable data,
let's just drop the const-ness of the existing context
struct.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
teach format-patch to place other authors into in-body "From"

Format-patch generates emails with the "From" address set to the
author of each patch. If you are going to send the emails, however,
you would want to replace the author identity with yours (if they
are not the same), and bump the author identity to an in-body
header.

Normally this is handled by git-send-email, which does the
transformation before sending out the emails. However, some
workflows may not use send-email (e.g., imap-send, or a custom
script which feeds the mbox to a non-git MUA). They could each
implement this feature themselves, but getting it right is
non-trivial (one must canonicalize the identities by reversing any
RFC2047 encoding or RFC822 quoting of the headers, which has caused
many bugs in send-email over the years).

This patch takes a different approach: it teaches format-patch a
"--from" option which handles the ident check and in-body header
while it is writing out the email.  It's much simpler to do at this
level (because we haven't done any quoting yet), and any workflow
based on format-patch can easily turn it on.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
send-email: provide port separately from hostname

If the SMTP port is provided as part of the hostname to Net::SMTP, it passes
the combined string to the SASL provider; this causes GSSAPI authentication to
fail since Kerberos does not want the port information.  Instead, pass the port
as a separate argument as is done for SSL connections.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'jk/fetch-pack-many-refs'

Fetching between repositories with many refs employed O(n^2)
algorithm to match up the common objects, which has been corrected.

* jk/fetch-pack-many-refs:
  fetch-pack: avoid quadratic behavior in rev_list_push
  commit.c: make compare_commits_by_commit_date global
  fetch-pack: avoid quadratic list insertion in mark_complete
Merge branch 'mv/merge-ff-tristate'

The configuration variable "merge.ff" was cleary a tri-state to
choose one from "favor fast-forward when possible", "always create
a merge even when the history could fast-forward" and "do not
create any merge, only update when the history fast-forwards", but
the command line parser did not implement the usual convention of
"last one wins, and command line overrides the configuration"
correctly.

* mv/merge-ff-tristate:
  merge: handle --ff/--no-ff/--ff-only as a tri-state option
Merge branch 'jk/format-patch-from'

"git format-patch" learned "--from[=whom]" option, which sets the
"From: " header to the specified person (or the person who runs the
command, if "=whom" part is missing) and move the original author
information to an in-body From: header as necessary.

* jk/format-patch-from:
  teach format-patch to place other authors into in-body "From"
  pretty.c: drop const-ness from pretty_print_context
Merge branch 'cp/submodule-custom-update'

In addition to the choice from "rebase, merge, or checkout-detach",
allow a custom command to be used in "submodule update" to update
the working tree of submodules.

* cp/submodule-custom-update:
  submodule update: allow custom command to update submodule working tree
Merge branch 'fg/submodule-clone-depth'

Allow shallow-cloning of submodules with "git submodule update".

* fg/submodule-clone-depth:
  Add --depth to submodule update/add
Merge branch 'bc/send-email-use-port-as-separate-param'

Pass port number as a separate argument when send-email initializes
Net::SMTP, instead of as a part of the hostname, i.e. host:port.
This allows GSSAPI codepath to match with the hostname given.

* bc/send-email-use-port-as-separate-param:
  send-email: provide port separately from hostname
Merge branch 'mt/send-email-cc-match-fix' into maint

Logic used by git-send-email to suppress cc mishandled names like "A
U. Thor" <author@example.xz>, where the human readable part needs to
be quoted (the user input may not have the double quotes around the
name, and comparison was done between quoted and unquoted strings).
It also mishandled names that need RFC2047 quoting.

* mt/send-email-cc-match-fix:
  send-email: sanitize author when writing From line
  send-email: add test for duplicate utf8 name
  test-send-email: test for pre-sanitized self name
  t/send-email: test suppress-cc=self with non-ascii
  t/send-email: add test with quoted sender
  send-email: make --suppress-cc=self sanitize input
  t/send-email: test suppress-cc=self on cccmd
  send-email: fix suppress-cc=self on cccmd
  t/send-email.sh: add test for suppress-cc=self
Merge branch 'fc/sequencer-plug-leak' into maint

"cherry-pick" had a small leak in its error codepath.

* fc/sequencer-plug-leak:
  sequencer: avoid leaking message buffer when refusing to create an empty commit
  sequencer: remove useless indentation
Merge branch 'fg/submodule-non-ascii-path' into maint

Many "git submodule" operations did not work on a submodule at a
path whose name is not in ASCII.

* fg/submodule-non-ascii-path:
  t7400: test of UTF-8 submodule names pass under Mac OS
  handle multibyte characters in name
Merge branch 'jk/pull-into-dirty-unborn' into maint

"git pull" into nothing trashed "local changes" that were in the
index.

* jk/pull-into-dirty-unborn:
  pull: merge into unborn by fast-forwarding from empty tree
  pull: update unborn branch tip after index
Merge branch 'bc/http-keep-memory-given-to-curl' into maint

Older cURL wanted piece of memory we call it with to be stable, but
we updated the auth material after handing it to a call.

* bc/http-keep-memory-given-to-curl:
  http.c: don't rewrite the user:passwd string multiple times