Hat etiquettes suggest that a man should remove his hat when entering a building. That being said, it is considered rude to sit in a restaurant, office, or home with a hat on. However, this does not apply to more casual and modern wear like baseball caps.