In many cases, it is difficult or impossible for men to reduce their breast size with diet and exercise. The exact cause of excess breast tissue isn’t known; however, it affects a significant number of men and can be linked to excess weight, hormonal imbalances, and the use of steroids, marijuana, and certain other substances