Actually remove things from the cancellation store.