Move all the defer cancelling to a method.

Also add X_defer = None for the defers after being cancelled.