Swim shorts are specifically designed for swimming, with quick-drying materials and a mesh lining. Gym shorts, on the other hand, are more versatile for various types of workouts, usually made of breathable fabrics with pockets for convenience.