If you fail a Maven's Invitation, you'll need to re-witness the required bosses and obtain a new invitation. Your progress is not saved, so it's crucial to be well-prepared before attempting the encounter.