There are many causes of small breast size. Some common causes include genetics, hormonal imbalance, and weight loss or gain. Breast size can also be affected by age, as breasts tend to shrink and become less fatty as a woman gets older.