Verbal agreements could potentially be legally binding. The question is whether an offer was made, it was accepted and both parties exchanged something of value. In a car accident settlement, the insurance company pays compensation to the victim in exchange for the victim giving up the right to pursue more compensation from the insurance company.
You may question whether the insurance company can prove you gave verbal acceptance of the offer. However, they may have been recording your phone conversation, or at the very least taking notes. This could make it harder to back out of the settlement agreement.