BUG: Removing a nonexistent store raises a KeyError

Re: Issue #3446