Publish still active after widget removed

If a widget is removed from the page and then it's published, the
Publish button is still active until published again.