Clears memoized test files on additions and removals

Like discussed in #80, this makes sure added files are picked up and removed files won't be tested in subsequent runs any longer (thus preventing the file not found errors).