We may not be responsible for our desires but it doesn't follow that we aren't responsible for controlling them. Pinker sometimes writes of natural selection as if it were a conscious process of a Grand Designer, not an impersonal process with no particular goals in mind. He doesn't call this a metaphor and writes as if it is to be taken literally. At other times he is very specific in not only writing of natural selection as impersonal and indifferent but also in stating that not all behavior is adaptive. In any case, it is also true that men who can't get dates normally don't rape, either.
Some rapists are married with children.
What seems most unscientific about the speculations of evolutionary psychologists like Thornhill and Palmer is that they seem to assume before they begin their investigation that there can only be one explanatory principle for the behavior they are investigating. They then proceed to fit all the evidence, even the apparently contrary evidence, to the hypothesis. The number of causal explanations for behavior should be discovered not stipulated. But this should be discovered not assumed. We should accept or reject a theory because we think it explains the facts better than any competing theory, not because we find the explanation attractive or repulsive.