Bugfix: Correctly delete old cache files

* Don't store the absolute path so that state files are portable
* Delete more than one cache file if necessary