Initial implementation of similarity scoring algo

This adds a new `git_buf_text_hashsig` type and functions to
generate these hash signatures and compare them to give a
similarity score.  This can be plugged into diff similarity
scoring.
Some similarity metric adjustments

This makes the text similarity metric treat \r as equivalent
to \n and makes it skip whitespace immediately following a line
terminator, so line indentation will have less effect on the
difference measurement (and so \r\n will be treated as just a
single line terminator).

This also separates the text and binary hash calculators into
two separate functions instead of have more if statements inside
the loop. This should make it easier to have more differentiated
heuristics in the future if we so wish.
wip: adding metric to diff
Change similarity metric to sampled hashes

This moves the similarity metric code out of buf_text and into a
new file.  Also, this implements a different approach to similarity
measurement based on a Rabin-Karp rolling hash where we only keep
the top 100 and bottom 100 hashes.  In theory, that should be
sufficient samples to given a fairly accurate measurement while
limiting the amount of data we keep for file signatures no matter
how large the file is.
More tests of file signatures with whitespace opts

Seems to be working pretty well...
Pluggable similarity metric API
Refine pluggable similarity API

This plugs in the three basic similarity strategies for handling
whitespace via internal use of the pluggable API.  In so doing, I
realized that the use of git_buf in the hashsig API was not needed
and actually just made it harder to use, so I tweaked that API as
well.

Note that the similarity metric is still not hooked up in the
find_similarity code - this is just setting out the function that
will be used.
Replace diff delta binary with flags

Previously the git_diff_delta recorded if the delta was binary.
This replaces that (with no net change in structure size) with
a full set of flags.  The flag values that were already in use
for individual git_diff_file objects are reused for the delta
flags, too (along with renaming those flags to make it clear that
they are used more generally).

This (a) makes things somewhat more consistent (because I was
using a -1 value in the "boolean" binary field to indicate unset,
whereas now I can just use the flags that are easier to understand),
and (b) will make it easier for me to add some additional flags to
the delta object in the future, such as marking the results of a
copy/rename detection or other deltas that might want a special
indicator.

While making this change, I officially moved some of the flags that
were internal only into the private diff header.

This also allowed me to remove a gross hack in rename/copy detect
code where I was overwriting the status field with an internal
value.
Initial integration of similarity metric to diff

This is the initial integration of the similarity metric into
the `git_diff_find_similar()` code path.  The existing tests all
pass, but the new functionality isn't currently well tested.  The
integration does go through the pluggable metric interface, so it
should be possible to drop in an alternative to the internal
metric that libgit2 implements.

This comes along with a behavior change for an existing interface;
namely, passing two NULLs to git_diff_blobs (or passing NULLs to
git_diff_blob_to_buffer) will now call the file_cb parameter zero
times instead of one time.  I know it's strange that that change
is paired with this other change, but it emerged from some
initialization changes that I ended up making.
Add diff rename tests with partial similarity

This adds some new tests that actually exercise the similarity
metric between files to detect renames, copies, and split modified
files that are too heavily modified.

There is still more testing to do - these tests are just partially
covering the cases.

There is also one bug fix in this where a change set with only
MODIFY being broken into ADD/DELETE (due to low self-similarity)
without any additional RENAMED entries would end up not processing
the split requests (because the num_rewrites counter got reset).
Fix tests for find_similar and related

This fixes both a test that I broke in diff::patch where I was
relying on the current state of the working directory for the
renames test data and fixes an unstable test in diff::rename
where the environment setting for the "diff.renames" config was
being allowed to influence the test results.
Replace static data with configured metric

Instead of creating three git_diff_similarity_metric statically
for the various config options, just create the metric structure
on demand and populate it, using the payload to specific the
extra flags that should be passed to the hashsig.  This removes
a level of obfuscation from the code, I think.
Minor improvements to find_similar code

This moves a couple of checks outside of the inner loop of the
find_similar rename/copy detection phase that are only dependent
on the "from" side of a detection.

Also, this replaces the inefficient initialization of the
options structure when a value is not provided explicitly by the
user.
More rename detection tests

This includes tests for crlf changes, whitespace changes with the
default comparison and with the ignore whitespace comparison, and
more sensitivity checking for the comparison code.
don't dereference at the end of the workdir iterator
Merge pull request #1372 from ethomson/checkout_workdir_end

don't dereference at the end of the workdir iterator
Merge pull request #1233 from arrbee/file-similarity-metric

Add file similarity scoring to diff rename/copy detection