Not necessarily. Fighting well is more about technique than raw strength. For example, a small person can fairly easily knock a larger opponent down if they are able to throw them off balance. Consider taking martial arts or self-defense classes to learn techniques that can help you if a larger or stronger person attacks you.