Fewer matings mean fewer offspring. However, selection is evidently favoring this behavior, so there must be some way in which the net effect is to increase the male's reproductive success. One likely explanation is that an infant is significantly more likely to survive to reproductive age if a male carries it. This might be true if there were a lot of predators around, and being carried by the male made the infant significantly less vulnerable to them. In such a case, the increase in the number of surviving offspring that a male produces thanks to this behavior might be greater than the decrease in the number of offspring that he fathers due to mating less.
Natural selection favors traits in males that increase their success at mating.
Since in many kinds of social groups, the males have to compete with each other for access to females, selection favors traits that help males succeed in this male-male competition. Such competition often involves fights and threats of fights. Larger males tend to be more successful in these confrontations, so they are more successful at mating than smaller males. Thus larger males leave more offspring, and as time passes, the size of males increases. Among females, selection favors traits that maximize the survival and health of their offspring.