Fix a race condition if queued actions have been handled already

This should fix #1354.

Cc @Bananeweizen @akarnokd