Also since you and your husband are fairly new to the church, he may be going out of his way to make you feel welcome. Then too, as Mona said, he could be hot for your bod. I had a pastor admit to me once that he was attracted to me. He never acted out on it though and to this day he is in my mind the epitome of what a Christian should be. The man would literally have given anyone who needed it the shirt off his back.
Pastors are capable of falling into sin as much as anyone else.
Here where I live, just about three blocks from my house, there is a church whose pastor began having an affair with his secretary and eventually killed his wife and her husband to be with her. Of course both murders were made to look like accidents. But when rumors of their suspected affair came out the police put two and two together. This was a big shock to our little town. A movie was even made about it.