Avoid casting

ffi.cast() takes up a surprising amount of time. We can define the
functions to take in git_object* instead.
Don't keep a copy of the object pointer

That memory area is just for writing the pointer to, the Object does not
need to keep a reference to that bit of memory, just the pointer that
was returned.