Testosterone is the hormone most directly correlated with libido in both sexes, being responsible for the growth and maintenance of reproductive tissues and stimulating sexual desire.