Male pattern baldness can start at any age. Factors like genetics, stress, and illness can play a part in how soon you start losing hair. If hair loss runs in your family, you may notice the early signs of hair loss before you reach the age of 21. This happens to around a quarter of men that have hereditary hair loss. Some males notice a receding hairline as early as 15 years old, though this is very rare.