Commit message: delta: fix sign-extension of big left-shift

Our delta code was originally adapted from JGit, which itself adapted it
from git itself. Due to this heritage, we inherited a bug from git.git
in how we compute the delta offset, which was fixed upstream in
48fb7deb5 (Fix big left-shifts of unsigned char, 2009-06-17). As
explained by Linus:

    Shifting 'unsigned char' or 'unsigned short' left can result in sign
    extension errors, since the C integer promotion rules means that the
    unsigned char/short will get implicitly promoted to a signed 'int' due to
    the shift (or due to other operations).

    This normally doesn't matter, but if you shift things up sufficiently, it
    will now set the sign bit in 'int', and a subsequent cast to a bigger type
    (eg 'long' or 'unsigned long') will now sign-extend the value despite the
    original expression being unsigned.

    One example of this would be something like

            unsigned long size;
            unsigned char c;

            size += c << 24;

    where despite all the variables being unsigned, 'c << 24' ends up being a
    signed entity, and will get sign-extended when then doing the addition in
    an 'unsigned long' type.

    Since git uses 'unsigned char' pointers extensively, we actually have this
    bug in a couple of places.

In our delta code, we inherited such a bogus shift when computing the
offset at which the delta base is to be found. Due to the sign extension
we can end up with an offset where all the bits are set. This can allow
an arbitrary memory read, as the addition in `base_len < off + len` can
now overflow if `off` has all its bits set.

Fix the issue by casting the result of `*delta++ << 24UL` to an unsigned
integer again. Add a test with a crafted delta that would actually
succeed with an out-of-bounds read in case where the cast wouldn't
exist.

Reported-by: Riccardo Schirone <rschiron@redhat.com>
Test-provided-by: Riccardo Schirone <rschiron@redhat.com>
CVE: CVE-2018-10887
CWE: CWE-125
Vulnerability description: A flaw was found in libgit2 before version 0.27.3. It has been discovered that an unexpected sign extension in git_delta_apply function in delta.c file may lead to an integer overflow which in turn leads to an out of bound read, allowing to read before the base object. An attacker may use this flaw to leak memory addresses or cause a Denial of Service.

Project: libgit2

static int binary_cb(
	const git_diff_delta *delta,
	const git_diff_binary *binary,
	void *payload)
{
	struct diff_data *diff_data = payload;

	GIT_UNUSED(delta);

	git_buf_encode_base85(&diff_data->old_binary_base85,
		binary->old_file.data, binary->old_file.datalen);
	diff_data->old_binary_inflatedlen = binary->old_file.inflatedlen;
	diff_data->old_binary_type = binary->old_file.type;

	git_buf_encode_base85(&diff_data->new_binary_base85,
		binary->new_file.data, binary->new_file.datalen);
	diff_data->new_binary_inflatedlen = binary->new_file.inflatedlen;
	diff_data->new_binary_type = binary->new_file.type;

	return 0;
}

Project: libgit2

static int file_cb(
	const git_diff_delta *delta,
	float progress,
	void *payload)
{
	struct diff_data *diff_data = payload;

	GIT_UNUSED(progress);

	if (delta->old_file.path)
		diff_data->old_path = git__strdup(delta->old_file.path);

	if (delta->new_file.path)
		diff_data->new_path = git__strdup(delta->new_file.path);

	git_oid_cpy(&diff_data->old_id, &delta->old_file.id);
	git_oid_cpy(&diff_data->new_id, &delta->new_file.id);

	return 0;
}

Project: libgit2

static int hunk_cb(
	const git_diff_delta *delta,
	const git_diff_hunk *hunk,
	void *payload)
{
	GIT_UNUSED(delta);
	GIT_UNUSED(hunk);
	GIT_UNUSED(payload);

	cl_fail("did not expect hunk callback");
	return 0;
}

Project: libgit2

static int print_cb(
	const git_diff_delta *delta,
	const git_diff_hunk *hunk,
	const git_diff_line *line,
	void *payload)
{
	git_buf *buf = (git_buf *)payload;

	GIT_UNUSED(delta);

	if (hunk)
		git_buf_put(buf, hunk->header, hunk->header_len);

	if (line)
		git_buf_put(buf, line->content, line->content_len);

	return git_buf_oom(buf) ? -1 : 0;
}