If your gynecomastia is caused by genetic and hormone abnormalities, you may also experience fertility issues. However, gynecomastia itself will not directly impact fertility.