Clean up garbage collection related stuff

Don't implement IDisposable anywhere since nothing here actually handles
unmanaged resources.