Store previous requests on disk rather than in memory

cc @dwoos . 

Merging.