Strip out the caching. We could use an LRU module with weak refs, but without that this memory use will grow unbounded. Doing any sort of caching is unnecessary at the moment.