No, both parties do not have to agree to an annulment of marriage in California. A person can file a lawsuit without their current spouse’s consent. Getting an annulment is easier in some situations, e.g., when the marriage is between close relatives or one party is already married.
However, the parties may have more difficulty proving other grounds for the judge to declare the marriage invalid.