Don't try to remove the button span on destroy, crashes in IE11.