Be smart w cleaning up in-flight Store requests

Fix #575