Remove the cache object before we release the prisoner.