Remove from temp store when removed from collection