It depends on the martial art. In Jiu Jitsu, you do not need to be fit. Specifically, Gracie Jiu Jitsu system of teaching was designed to empower the less physically gifted through the use of leverage based techniques.