Consuming a single serving of almonds (about ¼ cup or approximately 23 nuts) a day is enough for you to reap the benefits. While this post focuses primarily on women, this is also true for men who are trying to boost their fertility. In one study, researchers found men who consumed about two handfuls of mixed nuts (almonds, hazelnuts, and walnuts) a day for 14 weeks experienced a higher sperm count with more viable swimmers.