No, using "hen" in conversation is not considered impolite. It is a regular part of the Japanese language and is used appropriately in various situations.