Better handling of removing an action during the action.

And by better I mean fixing my previous fubar fix. #463