Do not remove the application widget twice when it changes state

Hopefully resolves: https://bugzilla.gnome.org/show_bug.cgi?id=709355