check that menu is empty before adding refresh

Had some more discussions with @dottertrotter and we'll leave the more generic case alone for now.