When you get down to the logistics of this question, it can actually be pretty difficult to answer. If you look at it from one lens, you can surf without having an elite fitness level. However, through another lens, it also may be easier for you to surf if you are somewhat of an athlete. To make an argument for the latter, you may find the transition to surfing much easier if you currently enjoy skateboarding or snowboarding (as the same general principle applies). Yet, at the same time, this is going to take practice anyway and the more you practice it, the better you will be at it (much like anything else).