DirCache: Buffer TREE extension to $GIT_DIR

Increase the in-memory buffer for the TREE extension to 5 MiB, and
overflow to $GIT_DIR instead of /tmp.  Using a larger buffer reduces
the chances a repository will overflow and need to spool the extension
to disk.  Using $GIT_DIR allows the TREE extension contents to have
the same file system protections as the final $GIT_DIR/index.

Wrap the entire thing in a try/finally to ensure the temp file is
deleted from disk after the block has finished using it. To avoid
dangling NFS files, LocalFile.destroy() does close the local file
before deleting it.

Change-Id: I8f871181a4689e3ebf0cdd4fd1769333cf7546c3
AmazonS3: Buffer pushed pack content under $GIT_DIR

This applies the same filesystem permissions as the source objects.
Users may override in properties files using the tmpdir value.

Change-Id: I3ec332cf41f12eae246cfaee9fd792c52cb2908b
Switch FileHeader.extractFileLines to TemporaryBuffer.Heap

File contents are processed into a single byte[] for character
conversion. The data must fit entirely in memory, so avoid any
file IO.

Change-Id: I3fe8be2e5f37d5ae953596dda1ed3fe6d4f6aebc
Deprecate TemporaryBuffer.LocalFile without parent directory

Encourage callers to explicitly name a directory to hold any
overflow data. Call sites have more information about what is
going into the buffer and how it should be protected at the
filesystem level than just throwing content to the system wide
temporary directory.

Callers that still really don't care (or need to care) can pass
null for the File argument to have the system directory used.

Change-Id: I89009bbee49d3850d42cd82c2c462e51043acda0
Detect buffering failures while writing rebase todo file

By routing writes through SafeBufferedOutputStream the caller can be
alerted to any flush at close failures while writing or appending to
the rebase todo script.

Switch the character encoding to be done at the line granularity, as
this is sufficiently long enough that encoding overheads will not be a
bottleneck, but short enough that the amount of temporary data will
not cause memory problems for the JVM.

Change-Id: Ice5ec10a7cbadc58486d481b92940056f9ffc43a
Fix two nits about DirCacheEntry constructors

Explicitly pass STAGE_0 when creating a DirCacheEntry from String.
This matches the immediate next constructor that accepts the int
stage argument better, making the code easier to read.

Fix a weird line break where the comma was orphaned by itself.

Change-Id: Icf0970dd02a63877f9e41b51b982b0265e8b8887
Apache HttpClientConnection: replace calls to deprecated LocalFile()

Change-Id: I79f422e004f386b3f2875de6997e5a0949fff566
Merge changes Icf0970dd,Ice5ec10a into stable-3.4

* changes:
  Fix two nits about DirCacheEntry constructors
  Detect buffering failures while writing rebase todo file
Replace "a." with "a-" in unit tests

Windows does not like naming files "a.".  The trailing "." may be
dropped by the filesystem, which is confusing.  Even though these
tests currently do not write to disk, future tests like them might.

Replace "." with "-", which has the same sorting properties that
were desirable about ".", but does not have the same limitations.

Change-Id: Ie5b7594bf5e79828d1341883c73ddb70123d5055
DirCache: Replace isValidPath with DirCacheCheckout.checkValidPath

isValidPath is an older simple form of the validation performed by
checkValidPath. Use the latter as it more consistently matches
git-core's validation rules.

By running the same validation as fsck, callers creating an entry
for the DirCache are more likely to learn early they are trying
to build trees that will fail fsck.

Change-Id: Ibf5ac116097156aa05c18e231bc65c0854932eb1
DirCache: Refuse to read files with invalid paths

If the DirCache contains a path that is known to be invalid, refuse to
read the DirCache into memory.  This avoids confusing errors later if
an invalid path read from the DirCache were to be passed into a new
DirCacheEntry constructor.

Change-Id: Ic033d81e23a5fbd554cc4dff80a232504562ffa8
Always ignore case when forbidding .git in ObjectChecker

The component name ".GIT" inside a tree entry could confuse a
case insensitive filesystem into looking at a submodule and
not a directory entry.

Disallow any case permutations of ".git" to prevent this
confusion from entering a repository and showing up at a
later date on a case insensitive system.

Change-Id: Iaa3f768931d0d5764bf07ac5f6f3ff2b1fdda01b
ObjectChecker: Disallow ".git." and ".git<space>"

Windows treats "foo." and "foo " as "foo". The ".git" directory is
special, as it contains metadata for a local Git repository. Disallow
variations that Windows considers to be the same.

Change-Id: I28eb48859a95a89111b4987c91de97557e3bb539
ObjectChecker: Disallow Windows shortname "GIT~1"

Windows creates shortnames for all non-8.3 files (see [1]). Hence we
need to disallow all names which could potentially be a shortname for
".git". Example: in an empty directory create a folder "GIT~1". Now you
can't create another folder ".git".

The path "GIT~1" may map to ".git" on Windows. A potential victim to
such an attack first has to initialize a git repository in order to
receive any git commits. Hence the .git folder created by init will get
the shortname "GIT~1". ".git" will only get a different shortname if the
user has created a file "GIT~1" before initialization of the git
repository.

[1] http://en.wikipedia.org/wiki/8.3_filename

Change-Id: I9978ab8f2d2951c46c1b9bbde57986d64d26b9b2
Signed-off-by: Christian Halstrick <christian.halstrick@sap.com>
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
ObjectChecker: Disallow names potentially mapping to ".git" on HFS+

Mac's HFS+ folds concatentations of ".git" and ignorable Unicode
characters [1] to ".git" [2]. Hence we need to disallow all names which
could potentially be a shortname for ".git". Example: in an empty
directory create a folder ".g\U+200Cit". Now you can't create another
folder ".git".

The following characters are ignorable Unicode which are ignored on
HFS+:

unicode   hex         name
-------------------------------------------------
U+200C    0xe2808c    ZERO WIDTH NON-JOINER
U+200D    0xe2808d    ZERO WIDTH JOINER
U+200E    0xe2808e    LEFT-TO-RIGHT MARK
U+200F    0xe2808f    RIGHT-TO-LEFT MARK
U+202A    0xe280aa    LEFT-TO-RIGHT EMBEDDING
U+202B    0xe280ab    RIGHT-TO-LEFT EMBEDDING
U+202C    0xe280ac    POP DIRECTIONAL FORMATTING
U+202D    0xe280ad    LEFT-TO-RIGHT OVERRIDE
U+202E    0xe280ae    RIGHT-TO-LEFT OVERRIDE
U+206A    0xe281aa    INHIBIT SYMMETRIC SWAPPING
U+206B    0xe281ab    ACTIVATE SYMMETRIC SWAPPING
U+206C    0xe281ac    INHIBIT ARABIC FORM SHAPING
U+206D    0xe281ad    ACTIVATE ARABIC FORM SHAPING
U+206E    0xe281ae    NATIONAL DIGIT SHAPES
U+206F    0xe281af    NOMINAL DIGIT SHAPES
U+FEFF    0xefbbbf    ZERO WIDTH NO-BREAK SPACE

[1] http://www.unicode.org/versions/Unicode7.0.0/ch05.pdf#G40025
    http://www.unicode.org/reports/tr31/#Layout_and_Format_Control_Characters
[2] http://dubeiko.com/development/FileSystems/HFSPLUS/tn1150.html#UnicodeSubtleties

Change-Id: Ib6a1dd090b2649bdd8ec16387c994ed29de2860d
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
JGit v3.4.2.201412180340-r

Change-Id: Ie088cf129b04ec64738edbc8c3ce25aa43b557ca
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Merge branch 'stable-3.4' into stable-3.5

* stable-3.4:
  JGit v3.4.2.201412180340-r
  ObjectChecker: Disallow names potentially mapping to ".git" on HFS+
  ObjectChecker: Disallow Windows shortname "GIT~1"
  ObjectChecker: Disallow ".git." and ".git<space>"
  Always ignore case when forbidding .git in ObjectChecker
  DirCache: Refuse to read files with invalid paths
  DirCache: Replace isValidPath with DirCacheCheckout.checkValidPath
  Replace "a." with "a-" in unit tests
  Apache HttpClientConnection: replace calls to deprecated LocalFile()
  Fix two nits about DirCacheEntry constructors
  Detect buffering failures while writing rebase todo file
  Deprecate TemporaryBuffer.LocalFile without parent directory
  Switch FileHeader.extractFileLines to TemporaryBuffer.Heap
  AmazonS3: Buffer pushed pack content under $GIT_DIR
  DirCache: Buffer TREE extension to $GIT_DIR

Change-Id: I398cf40b006a05a6537788fc6eb1f84df1ed8814
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
JGit v3.5.3.201412180710-r

Change-Id: Iadbd460da494c04fba71f98c200d6b65ce1709c6
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
Merge branch 'stable-3.5' into stable-3.6

* stable-3.5:
  JGit v3.5.3.201412180710-r
  JGit v3.4.2.201412180340-r
  ObjectChecker: Disallow names potentially mapping to ".git" on HFS+
  ObjectChecker: Disallow Windows shortname "GIT~1"
  ObjectChecker: Disallow ".git." and ".git<space>"
  Always ignore case when forbidding .git in ObjectChecker
  DirCache: Refuse to read files with invalid paths
  DirCache: Replace isValidPath with DirCacheCheckout.checkValidPath
  Replace "a." with "a-" in unit tests
  Apache HttpClientConnection: replace calls to deprecated LocalFile()
  Fix two nits about DirCacheEntry constructors
  Detect buffering failures while writing rebase todo file
  Deprecate TemporaryBuffer.LocalFile without parent directory
  Switch FileHeader.extractFileLines to TemporaryBuffer.Heap
  AmazonS3: Buffer pushed pack content under $GIT_DIR
  DirCache: Buffer TREE extension to $GIT_DIR

Change-Id: Iee8acbaa9d4d9047b550641db1b8845d64530785
Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>