Historically speaking, it’s considered rude for men to wear hats indoors, and taking them off upon entry is a sign of respect. However, times are changing. In general, it is rude to wear a hat (including beanies) as a guest at someone’s house, at the dining table, at places of worship (unless specifically required), at a restaurant, at the movies, or when a national anthem is being played. Beyond this, perception changes between people and cultures. If you’re unsure, ask your host/boss/friend/parent/acquaintance, or just read the room.