libgit2/libgit2 — pull request #1805 by arrbee (2013-08-21 23:40:48+00:00)
Title: Thread safety for the refdb_fs

This is an attempt to deal with thread safety for the refdb_fs.

It contains the first commit from #1798 because my tests need that commit. If that PR gets merged, I'll rebase this work.

This contains:

1. A new container for libgit2 that I've named a `git_sortedcache`. This is a very specialized container dedicated to keeping a sorted list of items each of which ends with a `GIT_FLEX_ARRAY` name that they will be sorted by. It uses both a vector and a hash table to provide both sorted indexed access and direct name lookups. It also provides a mutex for locking the cache during access and modification. And lastly, it has a facility so that the cache can be backed by a file on disk and makes it easy to reload the file when it has been changed (wrapping tracking the timestamp and file size, etc.).
2. Tests for the `git_sortedcache`
3. Reimplementation of `refdb_fs_backend` to use a `git_sortedcache` for storing `packed-refs` instead of the simple hashtable that it used previously. This allowed me to throw out and/or simplify a lot of code while adding thread safety to the in-memory refdb updates.
4. New tests in `tests-clar/threads/refdb.c` that access and update the refdb in various ways across threads. These tests were failing before the rewritten refdb backend and now pass consistently for me.

I first wrote this only holding locks around routines that modified the cached packed-refs data, but of course, if you reload the packed-refs which another thread is holding one a `struct packref` from the old cache, then it causes problems. I suppose an alternative approach would have been to add refcounts to the `struct packref` so that they could live across threads, but that didn't actually occur to me and the implementation I have will do fewer discrete memory allocations due to it's heavy usage of memory pools.

P.S. I secretly plan to eventually use some future iteration of the `git_sortedcache` for storing the contents of the index which is why I thought it was worthwhile adding a dedicated new container. Otherwise, it would be seem like overkill just to store the list of packed-refs.

P.P.S. `git_sortedcache` is a pretty poor name, I think. I'm open to suggestions!