You have the right to decide who can touch your body. YOU have the right to set sexual limits and defend yourself If they are not respected. You have the right to change your mind about having sex at any time. Trust your feelings. If you feel you are being pressured into unwanted sex, or if you aren't feeling comfortable, trust those feelings and know you are reading the situation correctly.
Men often interpret passivity as permission.
If you can tell he assumes you'll have sex with him and you ignore it, it will strengthen his assumption. State clearly and directly that you will not accept his pressuring you. If you try to be too nice as a way of sparing his feelings, he may interpret your niceness as consent. Beware of intoxication with alcohol or other drugs. If you are going to be alone with someone you don't know well, be aware that you may not be able to think clearly, judge accurately and act quickly when under the influence of such substances.