It’s unlikely that wearing a hat is causing your hair loss unless your hat constantly pulls on your hair roots and damages them. This is known as traction alopecia, and may make your male pattern baldness appear more advanced.