If no one is attempting to put you down or to exploit you, you need not be assertive. Defensiveness is not assertiveness. It is not to fight back, but to exercise your rights and interests. Assertiveness focuses on how you handle a situation and how to protect your rights. It is one thing to be assertive, it is another to be able to cope with assertive people.
Most people fail in coping with the latter.
When you assert yourself, stay on a simple, straight-forward theme. Do not get sidetracked. Keep repeating our assertiveness until the hearer complies with your rights. An assertive person maintains control of the situation. Stall for time and think on your feet.