A passive aggressive man may reject you sexually after having avoided a conflict with you or agreeing to do something he didn’t like. For example, he may go to a movie with you although he was never in the mood. Later, you may find him punishing you by rejecting you sexually. His passivity did not allow him to accept that he did not want to go with you. Out of his fear of conflict, however, he would have to punish you in covert ways for making him do something he did not want.