libgit2/libgit2 — pull request #851 by scunz (2012-08-02 18:12:12+00:00)
Title: RFC: UTF-8 only on Windows

I've taken up @vmg's `utf8-win` branch and mostly integrated the new utf8 => utf16 converter into libgit2.

I limited the maximum path length to MAX_PATH, which is 260 code points. According to [Microsoft](http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx#maxpath), Windows supports longer path names only prefixed with `\\?\`. Those prefixed path names cannot be relative and no forward-to-backward-slash conversion is done. We currently don't account for that, so this is actually no back-step in my book.

Though, there is no vision of the anticipated speed improvements. I also tried with a static (not yet per thread) buffer, but the result was the same. The maximum gain I could get out of it, was about 0.2 seconds running the tests (msvc8, release build, with or without customized optimization settings, tests take approximately 70 seconds). And those 0.2 seconds are probably due to one failing test.

After some digging, i found that even `_waccess` takes about 10 times as much time as Microsoft's MultiByte-Converter. So the result is not suprising at all. Unless vmg can point me to some hidden voodoo, I think doing this for speed reasons is a hoax. It might still be interesting, since we can - in all but the `p_rename` case - get away with one static thread local buffer of 520 bytes size. (No need to run through mutex'ed malloc code then).

Along the way, i stumbled over 2 tests. The first one was easy to fix: If we don't accept anything else than UTF-8, we can't feed libgit2 with an ANSI encoded string. (This is changed in the PR).

The other one is `core_env__0`. I actually don't get what we're really trying to test here. I _think_ the test's purpose is to figure out, whether we can find the user profile if the user's name is made up of some unicode (or rather some leet-speak) codepoints. So I guess testing with one plain ASCII string and one UTF-8 string should be enough.

Still left to do:
- `git__utf8_to_utf16` still doesn't do much error testing. It differs from the Microsoft implementation (0x80..0xBF are copied plainly, depending on input parameters, Microsoft either drops them or errors out) and it doesn't error out if the string doesn't end with a complete code point.
- Should find a way to get rid of the `strlen(path)` parameter to `git__utf8_to_utf16`; think i didn't see a single place where the length was already known.
- Should do better zero-length and `NULL` pointer validation of input to `git__utf8_to_utf16`
- `fileops.c`'s `win32_find_file`
- then probably removal of `gitwin_append_utf16`
- an analog utf16 => utf8 conversion

What do you think about the change in general and the approach taken here?