Keep this regardless of cache_directory if cache_directory is set and instead make sure `self.should_cleanup` is false if cache_directory is specified.