Fix cython compilation; refactor a bit.

It's not necessary to use DeferredStatus, because the only value that
was being pulled out of it was the returned_at. So, just store
returned_at instead.