Teach "git submodule add" the --name option

"git submodule add" initializes the name of a submodule to its path. This
was ok as long as the .git directory lived inside the submodule's work
tree, but since 1.7.8 it is stored in the .git/modules/<name> directory of
the superproject, making the submodule name survive the removal of the
submodule's work tree. This leads to problems when the user tries to add a
different submodule at the same path - and thus the same name - later, as
that will happily try to restore the submodule from the old repository
instead of the one the user specified and will lead to a checkout of the
wrong repository.

Add the new "--name" option to let the user provide a name for the
submodule. This enables the user to solve this conflict without having to
remove .git/modules/<name> by hand (which is no viable solution as it
makes it impossible to checkout a commit that records the old submodule
and populate it, as that will still check out the new submodule for the
same reason).

To achieve that the submodule's name is added to the parameter list of
the module_clone() helper function. This makes it possible to remove the
call of module_name() there because both callers of module_clone() already
know the name and can provide it as argument number two.

Reported-by: Jonathan Johnson <me@jondavidjohn.com>
Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
submodule add: Fail when .git/modules/<name> already exists unless forced

When adding a new submodule it can happen that .git/modules/<name> already
contains a submodule repo, e.g. when a submodule is removed from the work
tree and another submodule is added at the same path. But then the work
tree of the submodule will be populated using the existing repository and
not the one the user provided, which results in an incorrect work tree. On
the other hand the user might reactivate a submodule removed earlier, then
reusing that .git directory is the Right Thing to do.

As git can't decide what is the case, error out and tell the user she
should use either use a different name for the submodule with the "--name"
option or can reuse the .git directory for the newly added submodule by
providing the --force option (which only makes sense when the upstream
matches, so the error message lists all remotes of .git/modules/<name>).

In one test in t7406 the --force option had to be added to "git submodule
add", as that test re-adds a formerly removed submodule.

Reported-by: Jonathan Johnson <me@jondavidjohn.com>
Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Remove the hard coded length limit on variable names in config files

Previously while reading the variable names in config files, there
was a 256 character limit with at most 128 of those characters being
used by the section header portion of the variable name.  This
limitation was only enforced while reading the config files.  It was
possible to write a config file that was not subsequently readable.

Instead of enforcing this limitation for both reading and writing,
remove it entirely by changing the var member of the config_file
struct to a strbuf instead of a fixed length buffer.  Update all of
the parsing functions in config.c to use the strbuf instead of the
static buffer.

The parsing functions that returned the base length of the variable
name now return simply 0 for success and -1 for failure.  The base
length information is obtained through the strbuf's len member.

We now send the buf member of the strbuf to external callback
functions to preserve the external api.  None of the external
callers rely on the old size limitation for sizing their own buffers
so removing the limit should have no externally visible effect.

Signed-off-by: Ben Walton <bdwalton@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
l10n: de.po: fix a few minor typos

Signed-off-by: Simon Ruderich <simon@ruderich.org>
Signed-off-by: Ralf Thielow <ralf.thielow@gmail.com>
gitcli: parse-options lets you omit tail of long options

Describe the behaviour, but do warn people against taking it too
literally and expect an abbreviation valid today will stay valid
forever.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
peel_ref: use faster deref_tag_noverify

When we are asked to peel a ref to a sha1, we internally call
deref_tag, which will recursively parse each tagged object
until we reach a non-tag. This has the benefit that we will
verify our ability to load and parse the pointed-to object.

However, there is a performance downside: we may not need to
load that object at all (e.g., if we are listing peeled
simply listing peeled refs), or it may be a large object
that should follow a streaming code path (e.g., an annotated
tag of a large blob).

It makes more sense for peel_ref to choose the fast thing
rather than performing the extra check, for two reasons:

  1. We will already sometimes short-circuit the tag parsing
     in favor of a peeled entry from a packed-refs file. So
     we are already favoring speed in some cases, and it is
     not wise for a caller to rely on peel_ref to detect
     corruption.

  2. We already silently ignore much larger corruptions,
     like a ref that points to a non-existent object, or a
     tag object that exists but is corrupted.

  2. peel_ref is not the right place to check for such a
     database corruption. It is returning only the sha1
     anyway, not the actual object. Any callers which use
     that sha1 to load an object will soon discover the
     corruption anyway, so we are really just pushing back
     the discovery to later in the program.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
peel_ref: do not return a null sha1

The idea of the peel_ref function is to dereference tag
objects recursively until we hit a non-tag, and return the
sha1. Conceptually, it should return 0 if it is successful
(and fill in the sha1), or -1 if there was nothing to peel.

However, the current behavior is much more confusing. For a
regular loose ref, the behavior is as described above. But
there is an optimization to reuse the peeled-ref value for a
ref that came from a packed-refs file. If we have such a
ref, we return its peeled value, even if that peeled value
is null (indicating that we know the ref definitely does
_not_ peel).

It might seem like such information is useful to the caller,
who would then know not to bother loading and trying to peel
the object. Except that they should not bother loading and
trying to peel the object _anyway_, because that fallback is
already handled by peel_ref. In other words, the whole point
of calling this function is that it handles those details
internally, and you either get a sha1, or you know that it
is not peel-able.

This patch catches the null sha1 case internally and
converts it into a -1 return value (i.e., there is nothing
to peel). This simplifies callers, which do not need to
bother checking themselves.

Two callers are worth noting:

  - in pack-objects, a comment indicates that there is a
    difference between non-peelable tags and unannotated
    tags. But that is not the case (before or after this
    patch). Whether you get a null sha1 has to do with
    internal details of how peel_ref operated.

  - in show-ref, if peel_ref returns a failure, the caller
    tries to decide whether to try peeling manually based on
    whether the REF_ISPACKED flag is set. But this doesn't
    make any sense. If the flag is set, that does not
    necessarily mean the ref came from a packed-refs file
    with the "peeled" extension. But it doesn't matter,
    because even if it didn't, there's no point in trying to
    peel it ourselves, as peel_ref would already have done
    so. In other words, the fallback peeling is guaranteed
    to fail.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
peel_ref: check object type before loading

The point of peel_ref is to dereference tags; if the base
object is not a tag, then we can return early without even
loading the object into memory.

This patch accomplishes that by checking sha1_object_info
for the type. For a packed object, we can get away with just
looking in the pack index. For a loose object, we only need
to inflate the first couple of header bytes.

This is a bit of a gamble; if we do find a tag object, then
we will end up loading the content anyway, and the extra
lookup will have been wasteful. However, if it is not a tag
object, then we save loading the object entirely. Depending
on the ratio of non-tags to tags in the input, this can be a
minor win or minor loss.

However, it does give us one potential major win: if a ref
points to a large blob (e.g., via an unannotated tag), then
we can avoid looking at it entirely.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
upload-pack: use peel_ref for ref advertisements

When upload-pack advertises refs, we attempt to peel tags
and advertise the peeled version. We currently hand-roll the
tag dereferencing, and use as many optimizations as we can
to avoid loading non-tag objects into memory.

Not only has peel_ref recently learned these optimizations,
too, but it also contains an even more important one: it
has access to the "peeled" data from the pack-refs file.
That means we can avoid not only loading annotated tags
entirely, but also avoid doing any kind of object lookup at
all.

This cut the CPU time to advertise refs by 50% in the
linux-2.6 repo, as measured by:

  echo 0000 | git-upload-pack . >/dev/null

best-of-five, warm cache, objects and refs fully packed:

  [before]             [after]
  real    0m0.026s     real    0m0.013s
  user    0m0.024s     user    0m0.008s
  sys     0m0.000s     sys     0m0.000s

Those numbers are irrelevantly small compared to an actual
fetch. Here's a larger repo (400K refs, of which 12K are
unique, and of which only 107 are unique annotated tags):

  [before]             [after]
  real    0m0.704s     real    0m0.596s
  user    0m0.600s     user    0m0.496s
  sys     0m0.096s     sys     0m0.092s

This shows only a 15% speedup (mostly because it has fewer
actual tags to parse), but a larger absolute value (100ms,
which isn't a lot compared to a real fetch, but this
advertisement happens on every fetch, even if the client is
just finding out they are completely up to date).

In truly pathological cases, where you have a large number
of unique annotated tags, it can make an even bigger
difference. Here are the numbers for a linux-2.6 repository
that has had every seventh commit tagged (so about 50K
tags):

  [before]             [after]
  real    0m0.443s     real    0m0.097s
  user    0m0.416s     user    0m0.080s
  sys     0m0.024s     sys     0m0.012s

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
attr: avoid strlen() on every match

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
attr: avoid searching for basename on every match

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
gitignore.txt: suggestions how to get literal # or ! at the beginning

We support backslash escape, but we hide the details behind the phrase
"a shell glob suitable for consumption by fnmatch(3)". So it may not
be obvious how one can get literal # or ! at the beginning of pattern.
Add a few lines on how to work around the magic characters.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git url doc: mark ftp/ftps as read-only and deprecate them

It is not even worth mentioning their removal; just discourage
people from using them.

Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
tests: "cp -a" is a GNUism

These tests just want a bit-for-bit identical copy; they do not need
even -H (there is no symbolic link involved) nor -p (there is no
funny permission or ownership issues involved).

Just use "cp -R" instead.

Signed-off-by: Ben Walton <bdwalton@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'jl/submodule-add-by-name' into next

If you remove a submodule, in order to keep the repository so that
"git checkout" to an older commit in the superproject history can
resurrect the submodule, the real repository will stay in $GIT_DIR
of the superproject.  A later "git submodule add $path" to add a
different submodule at the same path will fail.  Diagnose this case
a bit better, and if the user really wants to add an unrelated
submodule at the same path, give the "--name" option to give it a
place in $GIT_DIR of the superproject that does not conflict with
the original submodule.

* jl/submodule-add-by-name:
  submodule add: Fail when .git/modules/<name> already exists unless forced
  Teach "git submodule add" the --name option
Merge branch 'bw/config-lift-variable-name-length-limit' into next

The configuration parser had an unnecessary hardcoded limit on
variable names that was not checked consistently. Lift the limit.

* bw/config-lift-variable-name-length-limit:
  Remove the hard coded length limit on variable names in config files
Merge branch 'jk/peel-ref' into next

Speeds up "git upload-pack" (what is invoked by "git fetch" on the
other side of the connection) by reducing the cost to advertise the
branches and tags that are available in the repository.

* jk/peel-ref:
  upload-pack: use peel_ref for ref advertisements
  peel_ref: check object type before loading
  peel_ref: do not return a null sha1
  peel_ref: use faster deref_tag_noverify
Merge branch 'jc/doc-long-options' into next

The end-user facing documentation claimed to be an "exhaustive"
list of features of parse-options command line parser, but it
didn't mention anything about abbreviating long option names to
their unique prefixes. Mention it, warn against relying too much on
it (as uniqueness may disappear in future versions of Git), and
drop the claim to be exhaustive.

* jc/doc-long-options:
  gitcli: parse-options lets you omit tail of long options
Merge branch 'nd/attr-match-optim' into next

* nd/attr-match-optim:
  attr: avoid searching for basename on every match
  attr: avoid strlen() on every match
Merge branch 'nd/doc-ignore' into next

* nd/doc-ignore:
  gitignore.txt: suggestions how to get literal # or ! at the beginning