libgit2/libgit2 — pull request #1588 by arrbee (2013-05-15 22:14:53+00:00)
Title: Bug fixes for checkout and diff

Some users of [GitHub for Mac](https://mac.github.com) were observing some odd crashes in the diff code, particularly in the new `diff_scan_inside_untracked_dir()` routine. I dredged up an old test that I had that took large repositories, rev-walked the history, and picked random OIDs to checkout. I updated it the test and added doing random diffs as well as random checkouts, and I was able to flush out a number of bugs. This PR has four commits that fix four separate bugs:

1. The actual crash that users were seeing is fixed in 79ef3be449c9d81dd0b37a30999563aa92e4679e where if the alphabetically last item in a diff (against the working directory) was an untracked directory that only contained ignored items (e.g. "zobjects/" only contains "*.o" files which are ignored), then we would run the iterator off the end and dereference NULL. Once I figured this out, it was easy to write a test that reproduced the error and put in a fix.
2. When updating an existing blob in checkout, the code was trying to do an "update in place" - i.e. it would open the existing file and just overwrite the data with the new content. That doesn't work for symlinks, so when we are doing a safe update to a symlink, I made it so we'll remove the old symlink and create a new one.
3. If a user forgets to commit the `.gitmodules` file after adding a new submodule, it is possible to create a tree with a submodule (i.e. commit) entry that does not have a corresponding entry in `.gitmodules` - in the checkout code, the call to `git_submodule_lookup` would fail when we try to checkout that submodule and we would abort the entire checkout midway through. Core git just creates an empty directory in these circumstances and proceeds with the rest of the checkout. I made our checkout code do the same thing.
4. Inside `git_diff_find_similar` depending on the flags that you pass to the original diff call, you may end up with entries that cannot actually be opened for purposes of rename detection. I fixed things so that if a blob or file on disk cannot be opened, instead of generating an error, we just exclude that from the list if valid rename candidates.

I'm putting all of these fixes in one PR because I think they should probably all be merged. If there are any that you're nervous about or want changes to, let me know and I'll either drop the bad ones to hurry the good ones in, or fix the problems.

Thanks!