Destroy things properly, fixes memory leak

I noticed some pretty bad memory leaks when doing many VCDiff-encodes, this seems to solve it.