Avoid double close when file has already gone away in abort().
Initial work on a fastexport module.
Correctly avoid parsing ShaFiles with fixed SHAs when calling sha().

This required some reworking of the _needs_* and _sha ivars. Improved
check() to force computing the SHA and verifying that it matches the
previously-set value. Added a test for this check.

Change-Id: I6782693d7d7708bc7c28f357c27419d51409b884
Add some basic tests for fastexport.
Fix fastexport commit exporter, add test.
Add basic test for write_commit_patch.
Cope with diffstat not being present in test_patch.
fix bug #557585 GitFile breaks dulwich on Windows
fix is to add os.O_BINARY flag as os.open doesn't open files as binary
on this platform by default
Prevent server stack trace when the client request is a no-op.

The exception was not killing the server, just the connection, and was
technically correct since the server is supposed to close the connection
anyway when the client sends no wants. As a result, the compat tests
weren't failing.

A better long-term solution would be to modify the server used during
testing to stop serving if one request generates an exception.

Change-Id: I065463803f38175a50f364b11588929094a28aab
Strip excess whitespace from capabilities lines.

C git separates client capabilities during send-pack with both \0 and a
space, so we need to handle that as well. This change is significantly
more lenient, but the cgit behavior is not well-documented, so there may
be more corner cases. It's hard to imagine that the extra strip() will
actually result in capabilities lists being parsed incorrectly.

Change-Id: I74b734e146add683b4c1e4ea6b575f458ec25de7
Add ReceivableProtocol that supports recv() as well as read().

There are two different ways of dealing with reads beyond the end of a
TCP stream buffer. The usual rfile.read method blocks until exactly the
number of bytes (or EOF) are read. This causes deadlocks when we try to
do buffered reads from the stream. When reading a packfile. the cgit
client leaves the connection open, so EOF is never read.

The other method is to use the socket.recv method, which blocks until
at least one byte is read, then returns whatever is available (up to
bufsize).

ReceivableProtocol supports both ways of dealing with reads past the
end of a stream, which should be used in different contexts. Note that
currently the ReceivePackHandler code still deadlocks, since it doesn't
know when to stop calling recv.

Change-Id: I77bfea72bb09326d9946f64426637296d0389714
Fix read_zlib_chunks to avoid appending junk data.

Adding junk data to the end of data fed to the zlib.decompressobj
creates the potential for zlib errors if a read doesn't fill the entire
buffer. This implementation is cleaner, and only requires the assumption
that there is some data in the read source after the zlib stream. Since
compressed streams in packfiles are always followed by either a new
packed object or the SHA-1 trailer, this assumption should be safe.

Added more robust tests for zlib reading with various buffer sizes.

Change-Id: I813d30997edf3e0c924eceac05675e0eb121b96c
Make the server decode a pack as it streams.

This, in combination with using recv() instead of read(), makes it so we
never do blocking reads past the end of the pack stream, even when the
client doesn't close the connection.

This is done via a PackStreamVerifier class that reads from a Protocol,
unpacks and counts objects, writes through to a file, and computes the
SHA-1 checksum on the fly. It is necessary because the only way we know
when the pack is supposed to end is by parsing the header and reading
the correct number of objects; otherwise, any further reads from the
client would hang.

Changed the Handler constructors to take a Protocol instead of taking
read and write callbacks separately. Modified some interfaces to utility
functions in pack.py so they can be used by the server-side code.

Change-Id: Id4d11e34658d1f00ad06e45330d0d128b367d8e5
Clean up pack.py.

-Removed trailing whitespace.
-Fixed long lines.
-Various docstring cleanup.

Change-Id: I2ccde03bc56d0f69ba2e3be7e11b3effce2d30ed
Merge various pack improvements from Dave.
Attempt to close index files properly.
Remove unused imports.
Fix ordering of imports.
Split generic pack reading code out of PackStreamVerifier.
Move PackStreamReader to dulwich.pack.