config.c: add git_env_ulong() to parse environment variable

The new function parses an integeral value that fits in unsigned
long in human readable form, i.e. possibly with unit suffix, e.g.
10k = 10240, etc., from an environment variable.  Parsing of
GIT_MMAP_LIMIT and GIT_ALLOC_LIMIT will use it in later patches.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
memory_limit: use git_env_ulong() to parse GIT_ALLOC_LIMIT

GIT_ALLOC_LIMIT limits xmalloc()'s size, which is of type size_t.
Better use git_env_ulong() to parse the environment variable, so
that the postfixes 'k', 'm', and 'g' can be used; and use size_t to
store the limit for consistency.  The change to size_t has no direct
practical impact, because the environment variable is only meant to
be used for our own tests, and we use it to test small sizes.

The cast of size in the call to die() is changed to uintmax_t to
match the format string PRIuMAX.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
mmap_limit: introduce GIT_MMAP_LIMIT to allow testing expected mmap size

In order to test expectations about mmap in a way similar to testing
expectations about malloc with GIT_ALLOC_LIMIT introduced by
d41489a6 (Add more large blob test cases, 2012-03-07), introduce a
new environment variable GIT_MMAP_LIMIT to limit the largest allowed
mmap length.

xmmap() is modified to check the size of the requested region and
fail it if it is beyond the limit.  Together with GIT_ALLOC_LIMIT
tests can now confirm expectations about memory consumption.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
copy_fd(): do not close the input file descriptor

The caller, not this function, opened the file descriptor; it is
selfish for the callee to close it when it is done reading from it.
The caller may want an option to rewind and re-read the contents
after it returns.

Simplify the loop to copy the input in full to the output; its
body essentially is what a call to write_in_full() helper does.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
convert: stream from fd to required clean filter to reduce used address space

The data is streamed to the filter process anyway.  Better avoid mapping
the file if possible.  This is especially useful if a clean filter
reduces the size, for example if it computes a sha1 for binary data,
like git media.  The file size that the previous implementation could
handle was limited by the available address space; large files for
example could not be handled with (32-bit) msysgit.  The new
implementation can filter files of any size as long as the filter output
is small enough.

The new code path is only taken if the filter is required.  The filter
consumes data directly from the fd.  If it fails, the original data is
not immediately available.  The condition can easily be handled as
a fatal error, which is expected for a required filter anyway.

If the filter was not required, the condition would need to be handled
in a different way, like seeking to 0 and reading the data.  But this
would require more restructuring of the code and is probably not worth
it.  The obvious approach of falling back to reading all data would not
help achieving the main purpose of this patch, which is to handle large
files with limited address space.  If reading all data is an option, we
can simply take the old code path right away and mmap the entire file.

The environment variable GIT_MMAP_LIMIT, which has been introduced in
a previous commit is used to test that the expected code path is taken.
A related test that exercises required filters is modified to verify
that the data actually has been modified on its way from the file system
to the object store.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
docs/fast-export: explain --anonymize more completely

The original commit made mention of this option, but not why
one might want it or how they might use it. Let's try to be
a little more thorough, and also explain how to confirm that
the output really is anonymous.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'jk/fast-export-anonymize' into jch

* jk/fast-export-anonymize:
  docs/fast-export: explain --anonymize more completely
  teach fast-export an --anonymize option
Merge branch 'jk/commit-author-parsing' into jch

* jk/commit-author-parsing:
  determine_author_info(): copy getenv output
  determine_author_info(): reuse parsing functions
  date: use strbufs in date-formatting functions
  record_author_date(): use find_commit_header()
  record_author_date(): fix memory leak on malformed commit
  commit: provide a function to find a header in a buffer
Merge branch 'sp/stream-clean-filter' into jch

Reroll posted but haven't had time to look at it yet.

* sp/stream-clean-filter:
  convert: stream from fd to required clean filter to reduce used address space
  copy_fd(): do not close the input file descriptor
  mmap_limit: introduce GIT_MMAP_LIMIT to allow testing expected mmap size
  memory_limit: use git_env_ulong() to parse GIT_ALLOC_LIMIT
  config.c: add git_env_ulong() to parse environment variable
  convert: drop arguments other than 'path' from would_convert_to_git()

# Conflicts:
# 	wrapper.c
Merge branch 'jk/send-pack-many-refspecs' into jch

* jk/send-pack-many-refspecs:
  send-pack: take refspecs over stdin
Merge branch 'rs/ref-transaction-1' into jch

The second batch of the transactional ref update series.

No more nitpicks and better design suggestions?

* rs/ref-transaction-1:
  refs.c: make delete_ref use a transaction
  refs.c: make prune_ref use a transaction to delete the ref
  refs.c: remove lock_ref_sha1
  refs.c: remove the update_ref_write function
  refs.c: remove the update_ref_lock function
  refs.c: make lock_ref_sha1 static
  walker.c: use ref transaction for ref updates
  fast-import.c: use a ref transaction when dumping tags
  receive-pack.c: use a reference transaction for updating the refs
  refs.c: change update_ref to use a transaction
  branch.c: use ref transaction for all ref updates
  fast-import.c: change update_branch to use ref transactions
  sequencer.c: use ref transactions for all ref updates
  commit.c: use ref transactions for updates
  replace.c: use the ref transaction functions for updates
  tag.c: use ref transactions when doing updates
  refs.c: add transaction.status and track OPEN/CLOSED
  refs.c: make ref_transaction_begin take an err argument
  refs.c: update ref_transaction_delete to check for error and return status
  refs.c: change ref_transaction_create to do error checking and return status

# Conflicts:
# 	fast-import.c
# 	refs.c
# 	sequencer.c
# 	walker.c
Merge branch 'rs/ref-transaction' into jch

* rs/ref-transaction:
  ### rs/ref-transaction
  refs.c: fix handling of badly named refs
  refs.c: make write_ref_sha1 static
  fetch.c: change s_update_ref to use a ref transaction
  refs.c: propagate any errno==ENOTDIR from _commit back to the callers
  refs.c: pass a skip list to name_conflict_fn
  refs.c: call lock_ref_sha1_basic directly from commit
  refs.c: move the check for valid refname to lock_ref_sha1_basic
  refs.c: pass NULL as *flags to read_ref_full
  refs.c: pass the ref log message to _create/delete/update instead of _commit
  refs.c: add an err argument to delete_ref_loose
  wrapper.c: add a new function unlink_or_msg
  wrapper.c: simplify warn_if_unremovable
  ### rs/ref-transaction-1

# Conflicts:
# 	builtin/log.c
# 	git-compat-util.h
Merge branch 'rs/ref-transaction-reflog' into jch

Cover updates to reflog with the same transaction mechanism as used
for reflog manipulations.

No more nitpicks and better design suggestions?

* rs/ref-transaction-reflog:
  ### rs/ref-transaction-reflog
  refs.c: allow deleting refs with a broken sha1
  refs.c: remove lock_any_ref_for_update
  refs.c: make unlock_ref/close_ref/commit_ref static
  refs.c: rename log_ref_setup to create_reflog
  reflog.c: use a reflog transaction when writing during expire
  refs.c: allow multiple reflog updates during a single transaction
  refs.c: only write reflog update if msg is non-NULL
  refs.c: add a flag to allow reflog updates to truncate the log
  refs.c: add a transaction function to append a reflog entry
  lockfile.c: make hold_lock_file_for_append preserve meaningful errno
  refs.c: add a function to append a reflog entry to a fd
  refs.c: add a new update_type field to ref_update
  refs.c: rename the transaction functions
  refs.c: make ref_transaction_delete a wrapper for ref_transaction_update
  refs.c: make ref_transaction_create a wrapper to ref_transaction_update

# Conflicts:
# 	copy.c
Merge branch 'rs/ref-transaction-rename' into jch

Follow-up on rs/ref-transaction series to make renaming a ref
transactional (i.e. "delete old and create new" should not leave
an in-between state behind when it fails).

No more nitpicks and better design suggestions?

* rs/ref-transaction-rename:
  ### rs/ref-transaction-rename
  refs.c: rollback the lockfile before we die() in repack_without_refs
  refs.c: update rename_ref to use a transaction
  refs.c: use packed refs when deleting refs during a transaction
  refs.c: return error instead of dying when locking fails during transaction
  refs.c: allow passing raw git_committer_info as email to _update_reflog
Merge branch 'rs/ref-transaction-multi' into jch

Follow-up on rs/ref-transaction series to use the packed-ref to
achieve atomicity when multiple refs are involved.

No more nitpicks and better design suggestions?

* rs/ref-transaction-multi:
  ### rs/ref-transaction-multi
  refs.c: make the *_packed_refs functions static
  refs.c: make repack_without_refs static
  remote.c: use a transaction for deleting refs
  refs.c: write updates to packed refs when a transaction has more than one ref
  refs.c: move reflog updates into its own function
Document LF appearing in shallow command during send-pack/receive-pack

The implementation sends an LF, but the protocol documentation was
missing this detail.

Signed-off-by: Shawn Pearce <spearce@spearce.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'sp/pack-protocol-doc-on-shallow' into jch

* sp/pack-protocol-doc-on-shallow:
  Document LF appearing in shallow command during send-pack/receive-pack