Unless your annuity has a bailout provision, you likely won't be able to cancel your annuity contract without a hefty surrender charge, even if the rate on your annuity declines.