It's just that he can't respond in some circumstances. I don't know if a doctor can help and it would depend on who is frustrated by this. If this is just a curious question and neither one of you is really concerned by it, then I would just let it go. If this is a big issue with both of you, then you might want to see a counselor to work it out. Things that would affect that would be religion, culture and your own personal history.
Men are much better with separating sex and love than women are.
There is an old joke about how women need a relationship in order to have sex where as men only need a location. There is a lot of overlap in gender stereotypes. It could be that you are such a good lover that everyone falls in love with you. The alcohol from perfume and cologne evaporates quickly. Body oils are good.