convert: drop arguments other than 'path' from would_convert_to_git()

It is only the path that matters in the decision whether to filter
or not.  Clarify this by making path the only argument of
would_convert_to_git().

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
config.c: add git_env_ulong() to parse environment variable

The new function parses an integeral value that fits in unsigned
long in human readable form, i.e. possibly with unit suffix, e.g.
10k = 10240, etc., from an environment variable.  Parsing of
GIT_MMAP_LIMIT and GIT_ALLOC_LIMIT will use it in later patches.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
memory_limit: use git_env_ulong() to parse GIT_ALLOC_LIMIT

GIT_ALLOC_LIMIT limits xmalloc()'s size, which is of type size_t.
Better use git_env_ulong() to parse the environment variable, so
that the postfixes 'k', 'm', and 'g' can be used; and use size_t to
store the limit for consistency.  The change to size_t has no direct
practical impact, because the environment variable is only meant to
be used for our own tests, and we use it to test small sizes.

The cast of size in the call to die() is changed to uintmax_t to
match the format string PRIuMAX.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
mmap_limit: introduce GIT_MMAP_LIMIT to allow testing expected mmap size

In order to test expectations about mmap in a way similar to testing
expectations about malloc with GIT_ALLOC_LIMIT introduced by
d41489a6 (Add more large blob test cases, 2012-03-07), introduce a
new environment variable GIT_MMAP_LIMIT to limit the largest allowed
mmap length.

xmmap() is modified to check the size of the requested region and
fail it if it is beyond the limit.  Together with GIT_ALLOC_LIMIT
tests can now confirm expectations about memory consumption.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
copy_fd(): do not close the input file descriptor

The caller, not this function, opened the file descriptor; it is
selfish for the callee to close it when it is done reading from it.
The caller may want an option to rewind and re-read the contents
after it returns.

Simplify the loop to copy the input in full to the output; its
body essentially is what a call to write_in_full() helper does.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
convert: stream from fd to required clean filter to reduce used address space

The data is streamed to the filter process anyway.  Better avoid mapping
the file if possible.  This is especially useful if a clean filter
reduces the size, for example if it computes a sha1 for binary data,
like git media.  The file size that the previous implementation could
handle was limited by the available address space; large files for
example could not be handled with (32-bit) msysgit.  The new
implementation can filter files of any size as long as the filter output
is small enough.

The new code path is only taken if the filter is required.  The filter
consumes data directly from the fd.  If it fails, the original data is
not immediately available.  The condition can easily be handled as
a fatal error, which is expected for a required filter anyway.

If the filter was not required, the condition would need to be handled
in a different way, like seeking to 0 and reading the data.  But this
would require more restructuring of the code and is probably not worth
it.  The obvious approach of falling back to reading all data would not
help achieving the main purpose of this patch, which is to handle large
files with limited address space.  If reading all data is an option, we
can simply take the old code path right away and mmap the entire file.

The environment variable GIT_MMAP_LIMIT, which has been introduced in
a previous commit is used to test that the expected code path is taken.
A related test that exercises required filters is modified to verify
that the data actually has been modified on its way from the file system
to the object store.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
sha1_file: don't convert off_t to size_t too early to avoid potential die()

xsize_t() checks if an off_t argument can be safely converted to
a size_t return value.  If the check is executed too early, it could
fail for large files on 32-bit architectures even if the size_t code
path is not taken.  Other paths might be able to handle the large file.
Specifically, index_stream_convert_blob() is able to handle a large file
if a filter is configured that returns a small result.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t/lib-credential: use write_script

Use write_script to create the helper "askpass" script, instead of
hand-creating it with hardcoded "#!/bin/sh" to make sure we use the
shell the user told us to use.

Signed-off-by: Ben Walton <bdwalton@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'hj/pretty-naked-decoration'

The pretty-format specifier "%d", which expanded to " (tagname)"
for a tagged commit, gained a cousin "%D" that just gives the
"tagname" without frills.

* hj/pretty-naked-decoration:
  pretty: add %D format specifier
Merge branch 'da/rev-parse-verify-quiet'

"rev-parse --verify --quiet $name" is meant to quietly exit with a
non-zero status when $name is not a valid object name, but still
gave error messages in some cases.

* da/rev-parse-verify-quiet:
  stash: prefer --quiet over shell redirection of the standard error stream
  refs: make rev-parse --quiet actually quiet
  t1503: use test_must_be_empty
  Documentation: a note about stdout for git rev-parse --verify --quiet
Merge branch 'rs/graph-simplify'

* rs/graph-simplify:
  graph: simplify graph_padding_line()
Merge branch 'rs/remote-simplify'

* rs/remote-simplify:
  remote: simplify match_name_with_pattern() using strbuf
Merge branch 'sb/t9300-typofix'

* sb/t9300-typofix:
  t9300-fast-import: fix typo in test description
Merge branch 'sb/t6031-typofix'

* sb/t6031-typofix:
  t6031-test-merge-recursive: do not forget to add file to be committed
Merge branch 'jk/mbox-from-line'

Some MUAs mangled a line in a message that begins with "From " to
">From " when writing to a mailbox file and feeding such an input
to "git am" used to lose such a line.

* jk/mbox-from-line:
  mailinfo: work around -Wstring-plus-int warning
  mailinfo: make ">From" in-body header check more robust
Update draft release notes to 2.2
Merge branch 'bw/use-write-script-in-tests' into next

* bw/use-write-script-in-tests:
  t/lib-credential: use write_script
Merge branch 'master' into next

* master:
  Update draft release notes to 2.2
Merge branch 'so/rebase-doc' into maint

* so/rebase-doc:
  Documentation/git-rebase.txt: <upstream> must be given to specify <branch>
  Documentation/git-rebase.txt: -f forces a rebase that would otherwise be a no-op
Merge branch 'jk/send-pack-many-refspecs' into maint

"git push" over HTTP transport had an artificial limit on number of
refs that can be pushed imposed by the command line length.

* jk/send-pack-many-refspecs:
  send-pack: take refspecs over stdin