There will be more than enough XP to progress without completing the side quests, but they can make certain difficult bosses like Medusa much easier. They are worth completing to understand your role in the game entirely. It's essential to enjoy a few of them at least, and this one is worth the quick rewards because it doesn't drag on forever.