Only remove empty desktops