Don't show various interface elements on the map when you can't interact

Part of bug #16350 by eleazar.