smart http: use the same connectivity check on cloning

This is an extension of c6807a4 (clone: open a shortcut for
connectivity check - 2013-05-26) to reduce the cost of connectivity
check at clone time, this time with smart http protocol.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
editor: use canonicalized absolute path

By improving the relative_path() algorithm, e02ca72 (path.c:
refactor relative_path(), not only strip prefix, 2013-06-25)
uncovered a latent bug in Emacs.  While most editor applications
like cat and vim handle non-canonicalized relative paths fine, emacs
does not.  This is due to a long-standing bug in emacs, where it
refuses to resolve symlinks in the supplied path:

  #!/bin/sh
  cd /tmp
  mkdir z z/a z/b
  echo moodle >z/a/file
  ln -s z/b
  cd b
  emacs ../a/file # fail: attempts to open /tmp/a/file

Even if emacs were to be patched to fix this bug, it may be nicer to
help users running older versions.

Note that this can potentially regress for users of all editors,
when they ask "what file am I editing?" to the editor, as it is
likely to answer with an unsightly long full path.

Co-authored-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
for-each-ref, quote: convert *_quote_print -> *_quote_buf

The print_value() function in for-each-ref.c prints values to stdout
immediately using {sq|perl|python|tcl}_quote_print().  Change these
lower-level quote functions to instead leave their results in strbuf
so that we can later add post-processing to the results of them.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
tar-tree: remove dependency on sq_quote_print()

By rewriting the loop that formats the argv[] in cmd_tar_tree()
function using sq_quote_argv() for code simplicity, the last use of
sq_quote_print() goes away.

Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
quote: remove sq_quote_print()

Remove sq_quote_print() since it has no callers.

Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
imap-send: use Apple's Security framework for base64 encoding

Use Apple's supported functions for base64 encoding instead
of the deprecated OpenSSL functions.

Signed-off-by: Jeremy Huddleston <jeremyhu@apple.com>
Signed-off-by: David Aguilar <davvid@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
http: add http.savecookies option to write out HTTP cookies

HTTP servers may send Set-Cookie headers in a response and expect them
to be set on subsequent requests. By default, libcurl behavior is to
store such cookies in memory and reuse them across requests within a
single session. However, it may also make sense, depending on the
server and the cookies, to store them across sessions. Provide users
an option to enable this behavior, writing cookies out to the same
file specified in http.cookiefile.

Signed-off-by: Dave Borowitz <dborowitz@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
parse-options: add OPT_CMDMODE()

This can be used to define a set of mutually exclusive "command
mode" options, and automatically catch use of more than one from
that set as an error.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
tag: use OPT_CMDMODE

This is just a demonstration of how the code would look like; I do
not think it is particularly easier to read than before myself.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
mv: move submodules together with their work trees

Currently the attempt to use "git mv" on a submodule errors out with:

  fatal: source directory is empty, source=<src>, destination=<dest>

The reason is that mv searches for the submodule with a trailing slash in
the index, which it doesn't find (because it is stored without a trailing
slash). As it doesn't find any index entries inside the submodule it
claims the directory would be empty even though it isn't.

Fix that by searching for the name without a trailing slash and continue
if it is a submodule. Then rename() will move the submodule work tree just
like it moves a file.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
mv: move submodules using a gitfile

When moving a submodule which uses a gitfile to point to the git directory
stored in .git/modules/<name> of the superproject two changes must be made
to make the submodule work: the .git file and the core.worktree setting
must be adjusted to point from work tree to git directory and back.

Achieve that by remembering which submodule uses a gitfile by storing the
result of read_gitfile() of each submodule. If that is not NULL the new
function connect_work_tree_and_git_dir() is called after renaming the
submodule's work tree which updates the two settings to the new values.

Extend the man page to inform the user about that feature (and while at it
change the description to not talk about a script anymore, as mv is a
builtin for quite some time now).

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
submodule.c: add .gitmodules staging helper functions

Add the new is_staging_gitmodules_ok() and stage_updated_gitmodules()
functions to submodule.c. The first makes it possible for call sites to
see if the .gitmodules file did contain any unstaged modifications they
would accidentally stage in addition to those they intend to stage
themselves. The second function stages all modifications to the
.gitmodules file, both will be used by subsequent patches for the mv
and rm commands.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
http.c: fix parsing of http.sslCertPasswordProtected variable

The existing code triggers only when the configuration variable is
set to true.  Once the variable is set to true in a more generic
configuration file (e.g. ~/.gitconfig), it cannot be overriden to
false in the repository specific one (e.g. .git/config).

Signed-off-by: Junio C Hamano <gitster@pobox.com>
config: add helper to normalize and match URLs

Some http.* configuration variables need to take values customized
for the URL we are talking to.  We may want to set http.sslVerify to
true in general but to false only for a certain site, for example,
with a configuration file like this:

	[http]
		sslVerify = true
	[http "https://weak.example.com"]
		sslVerify = false

and let the configuration machinery pick up the latter only when
talking to "https://weak.example.com".  The latter needs to kick in
not only when the URL is exactly "https://weak.example.com", but
also is anything that "match" it, e.g.

	https://weak.example.com/test
	https://me@weak.example.com/test

The <url> in the configuration key consists of the following parts,
and is considered a match to the URL we are attempting to access
under certain conditions:

  . Scheme (e.g., `https` in `https://example.com/`). This field
    must match exactly between the config key and the URL.

  . Host/domain name (e.g., `example.com` in `https://example.com/`).
    This field must match exactly between the config key and the URL.

  . Port number (e.g., `8080` in `http://example.com:8080/`).  This
    field must match exactly between the config key and the URL.
    Omitted port numbers are automatically converted to the correct
    default for the scheme before matching.

  . Path (e.g., `repo.git` in `https://example.com/repo.git`). The
    path field of the config key must match the path field of the
    URL either exactly or as a prefix of slash-delimited path
    elements.  A config key with path `foo/` matches URL path
    `foo/bar`.  A prefix can only match on a slash (`/`) boundary.
    Longer matches take precedence (so a config key with path
    `foo/bar` is a better match to URL path `foo/bar` than a config
    key with just path `foo/`).

  . User name (e.g., `me` in `https://me@example.com/repo.git`). If
    the config key has a user name, it must match the user name in
    the URL exactly. If the config key does not have a user name,
    that config key will match a URL with any user name (including
    none), but at a lower precedence than a config key with a user
    name.

Longer matches take precedence over shorter matches.

This step adds two helper functions `url_normalize()` and
`match_urls()` to help implement the above semantics. The
normalization rules are based on RFC 3986 and should result in any
two equivalent urls being a match.

Signed-off-by: Kyle J. McKay <mackyle@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
config: add generic callback wrapper to parse section.<url>.key

Existing configuration parsing functions (e.g. http_options() in
http.c) know how to parse two-level configuration variable names.
We would like to exploit them and parse something like this:

	[http]
		sslVerify = true
	[http "https://weak.example.com"]
		sslVerify = false

and pretend as if http.sslVerify were set to false when talking to
"https://weak.example.com/path".

Introduce `urlmatch_config_entry()` wrapper that:

 - is called with the target URL (e.g. "https://weak.example.com/path"),
   and the two-level variable parser (e.g. `http_options`);

 - uses `url_normalize()` and `match_urls()` to see if configuration
   data matches the target URL; and

 - calls the traditional two-level configuration variable parser
   only for the configuration data whose <url> part matches the
   target URL (and if there are multiple matches, only do so if the
   current match is a better match than the ones previously seen).

Signed-off-by: Junio C Hamano <gitster@pobox.com>
log: use true parents for diff even when rewriting

When using pathspec filtering in combination with diff-based log
output, parent simplification happens before the diff is computed.
The diff is therefore against the *simplified* parents.

This works okay, arguably by accident, in the normal case:
simplification reduces to one parent as long as the commit is TREESAME
to it.  So the simplified parent of any given commit must have the
same tree contents on the filtered paths as its true (unfiltered)
parent.

However, --full-diff breaks this guarantee, and indeed gives pretty
spectacular results when comparing the output of

  git log --graph --stat ...
  git log --graph --full-diff --stat ...

(--graph internally kicks in parent simplification, much like
--parents).

To fix it, store a copy of the parent list before simplification (in a
slab) whenever --full-diff is in effect.  Then use the stored parents
instead of the simplified ones in the commit display code paths.  The
latter do not actually check for --full-diff to avoid duplicated code;
they just grab the original parents if save_parents() has not been
called for this revision walk.

For ordinary commits it should be obvious that this is the right thing
to do.

Merge commits are a bit subtle.  Observe that with default
simplification, merge simplification is an all-or-nothing decision:
either the merge is TREESAME to one parent and disappears, or it is
different from all parents and the parent list remains intact.
Redundant parents are not pruned, so the existing code also shows them
as a merge.

So if we do show a merge commit, the parent list just consists of the
rewrite result on each parent.  Running, e.g., --cc on this in
--full-diff mode is not very useful: if any commits were skipped, some
hunks will disagree with all sides of the merge (with one side,
because commits were skipped; with the others, because they didn't
have those changes in the first place).  This triggers --cc showing
these hunks spuriously.

Therefore I believe that even for merge commits it is better to show
the diffs wrt. the original parents.

Reported-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Helped-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Thomas Rast <trast@inf.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t5540/5541: smart-http does not support "--force-with-lease"

The push() method in remote-curl.c is not told and does not pass the
necessary information to underlying send-pack, so this extension
does not yet work.  Leave a note in the test suite.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
Provide some linguistic guidance for the documentation.

This will hopefully avoid questions over which spelling and grammar should
be used.  Translators are of course free to create localizations for
specific English dialects.

Signed-off-by: Marc Branchaud <marcnarc@xiplink.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
sha1_file: introduce close_one_pack() to close packs on fd pressure

When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window.  If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed.  When this happens, git may exceed the number of file
descriptors permitted by the system.

This latter situation can occur in show-ref or receive-pack during ref
advertisement.  During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository.  If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated.  When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs.  Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:

   error: cannot create standard input pipe for rev-list: Too many open files
   error: Could not run 'git rev-list'

This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit.  If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object.  But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.

When we have file descriptor pressure, we just need to find an open
pack to close.  We can leave the existing mmap windows open.  If
additional windows need to be mapped into the pack file, it will be
reopened when necessary.  If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.

Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):

   * pack with oldest mtime and no allocated mmap windows
   * pack with the least-recently-used windows, i.e. the pack
     with the oldest most-recently-used window, where none of
     the windows are in use
   * pack with the least-recently-used windows

Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Don't close pack fd when free'ing pack windows

Now that close_one_pack() has been introduced to handle file
descriptor pressure, it is not strictly necessary to close the
pack file descriptor in unuse_one_window() when we're under memory
pressure.

Jeff King provided a justification for leaving the pack file open:

   If you close packfile descriptors, you can run into racy situations
   where somebody else is repacking and deleting packs, and they go away
   while you are trying to access them. If you keep a descriptor open,
   you're fine; they last to the end of the process. If you don't, then
   they disappear from under you.

   For normal object access, this isn't that big a deal; we just rescan
   the packs and retry. But if you are packing yourself (e.g., because
   you are a pack-objects started by upload-pack for a clone or fetch),
   it's much harder to recover (and we print some warnings).

Let's do so (or uh, not do so).

Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>