xread, xwrite: limit size of IO to 8MB

Checking out 2GB or more through an external filter (see test) fails
on Mac OS X 10.8.4 (12E55) for a 64-bit executable with:

    error: read from external filter cat failed
    error: cannot feed the input to external filter cat
    error: cat died of signal 13
    error: external filter cat failed 141
    error: external filter cat failed

The reason is that read() immediately returns with EINVAL when asked
to read more than 2GB.  According to POSIX [1], if the value of
nbyte passed to read() is greater than SSIZE_MAX, the result is
implementation-defined.  The write function has the same restriction
[2].  Since OS X still supports running 32-bit executables, the
32-bit limit (SSIZE_MAX = INT_MAX = 2GB - 1) seems to be also
imposed on 64-bit executables under certain conditions.  For write,
the problem has been addressed earlier [6c642a].

Address the problem for read() and write() differently, by limiting
size of IO chunks unconditionally on all platforms in xread() and
xwrite().  Large chunks only cause problems, like causing latencies
when killing the process, even if OS X was not buggy.  Doing IO in
reasonably sized smaller chunks should have no negative impact on
performance.

The compat wrapper clipped_write() introduced earlier [6c642a] is
not needed anymore.  It will be reverted in a separate commit.  The
new test catches read and write problems.

Note that 'git add' exits with 0 even if it prints filtering errors
to stderr.  The test, therefore, checks stderr.  'git add' should
probably be changed (sometime in another commit) to exit with
nonzero if filtering fails.  The test could then be changed to use
test_must_fail.

Thanks to the following people for suggestions and testing:

    Johannes Sixt <j6t@kdbg.org>
    John Keeping <john@keeping.me.uk>
    Jonathan Nieder <jrnieder@gmail.com>
    Kyle J. McKay <mackyle@gmail.com>
    Linus Torvalds <torvalds@linux-foundation.org>
    Torsten Bögershausen <tboegi@web.de>

[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/read.html
[2] http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html

[6c642a] commit 6c642a878688adf46b226903858b53e2d31ac5c3
    compate/clipped-write.c: large write(2) fails on Mac OS X/XNU

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Revert "compat/clipped-write.c: large write(2) fails on Mac OS X/XNU"

This reverts commit 6c642a878688adf46b226903858b53e2d31ac5c3.

The previous commit introduced a size limit on IO chunks on all
platforms.  The compat clipped_write() is not needed anymore.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
read-cache: use fixed width integer types

Use the fixed width integer types uint16_t and uint32_t for on-disk
structures; unsigned short and unsigned int do not have a guaranteed
size.

Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'nd/fetch-pack-shallow-fix' into next

Recent "short-cut clone connectivity check" topic broke a shallow
repository when a fetch operation tries to auto-follow tags.

* nd/fetch-pack-shallow-fix:
  fetch-pack: do not remove .git/shallow file when --depth is not specified
Merge branch 'hv/config-from-blob' into next

Portability fix.

* hv/config-from-blob:
  config: do not use C function names as struct members
Merge branch 'rj/doc-rev-parse' into next

* rj/doc-rev-parse:
  rev-parse(1): logically group options
  rev-parse: remove restrictions on some options
Merge branch 'mb/docs-favor-en-us' into next

Declare that the official grammar & spelling of the source of this
project is en_US, but strongly discourage patches only to "fix"
existing en_UK strings to avoid unnecessary churns.

* mb/docs-favor-en-us:
  Provide some linguistic guidance for the documentation.
Merge branch 'nd/sq-quote-buf' into next

Code simplification as a preparatory step to something larger.

* nd/sq-quote-buf:
  quote: remove sq_quote_print()
  tar-tree: remove dependency on sq_quote_print()
  for-each-ref, quote: convert *_quote_print -> *_quote_buf
Merge branch 'mm/no-shell-escape-in-die-message' into next

Fixes a minor bug in "git rebase -i" (there could be others, as the
root cause is pretty generic) where the code feeds a random, data
dependeant string to 'echo' and expects it to come out literally.

* mm/no-shell-escape-in-die-message:
  die_with_status: use "printf '%s\n'", not "echo"
Merge branch 'jc/parseopt-command-modes' into next

Many commands use --dashed-option as a operation mode selector
(e.g. "git tag --delete") that the user can use at most one
(e.g. "git tag --delete --verify" is a nonsense) and you cannot
negate (e.g. "git tag --no-delete" is a nonsense).  Make it easier
for users of parse_options() to enforce these restrictions.

* jc/parseopt-command-modes:
  tag: use OPT_CMDMODE
  parse-options: add OPT_CMDMODE()
Merge branch 'sb/parseopt-boolean-removal' into next

Convert most uses of OPT_BOOLEAN/OPTION_BOOLEAN that can use
OPT_BOOL/OPTION_BOOLEAN which have much saner semantics, and turn
remaining ones into OPT_SET_INT, OPT_COUNTUP, etc. as necessary.

* sb/parseopt-boolean-removal:
  revert: use the OPT_CMDMODE for parsing, reducing code
  checkout-index: fix negations of even numbers of -n
  config parsing options: allow one flag multiple times
  hash-object: replace stdin parsing OPT_BOOLEAN by OPT_COUNTUP
  branch, commit, name-rev: ease up boolean conditions
  checkout: remove superfluous local variable
  log, format-patch: parsing uses OPT__QUIET
  Replace deprecated OPT_BOOLEAN by OPT_BOOL
  Remove deprecated OPTION_BOOLEAN for parsing arguments
Merge branch 'ap/remote-hg-tilde-is-home-directory' into next

* ap/remote-hg-tilde-is-home-directory:
  remote-hg: fix path when cloning with tilde expansion
Merge branch 'rt/doc-merge-file-diff3' into next

* rt/doc-merge-file-diff3:
  Documentation/git-merge-file: document option "--diff3"
Merge branch 'sb/misc-cleanup' into next

* sb/misc-cleanup:
  rm: remove unneeded null pointer check
  diff: fix a possible null pointer dereference
  diff: remove ternary operator evaluating always to true
Merge branch 'mm/war-on-whatchanged' into next

* mm/war-on-whatchanged:
  whatchanged: document its historical nature
  core-tutorial: trim the section on Inspecting Changes
Merge branch 'sb/diff-delta-remove-needless-comparison' into next

* sb/diff-delta-remove-needless-comparison:
  create_delta_index: simplify condition always evaluating to true
Merge branch 'sg/bash-prompt-lf-in-cwd-test' into next

* sg/bash-prompt-lf-in-cwd-test:
  bash prompt: test the prompt with newline in repository path
Merge branch 'jl/some-submodule-config-are-not-boolean' into next

* jl/some-submodule-config-are-not-boolean:
  avoid segfault on submodule.*.path set to an empty "true"
Merge branch 'sp/doc-smart-http' into next

* sp/doc-smart-http:
  Document the HTTP transport protocols
Merge branch 'bc/completion-for-bash-3.0' into next

Some people still use rather old versions of bash, which cannot
grok some constructs like 'printf -v varname' the prompt and
completion code started to use recently.

* bc/completion-for-bash-3.0:
  contrib/git-prompt.sh: handle missing 'printf -v' more gracefully
  t9902-completion.sh: old Bash still does not support array+=('') notation
  git-completion.bash: use correct Bash/Zsh array length syntax