No, you don’t have to defeat all of the guardians. It’s impossible to do so.