Some Auto-Renewals in CustomersDot are throwing errors and not renewing

### Problem

**Based on the data found in Zuora for Subscriptions with `Enter a number of active users greater than or equal to your current` ERROR recorded in the Subscription Notes field, the first occurrence was on February 19, 2021.**

**Summary:**
There seems to be an issue with the CustomersDot Auto-Renewal flow that is sometimes causing only the upgrade portion (UpdateProduct amendment) of a customer's renewal to be recorded in Zuora and then terminate with out an actual renewal. In some cases no UpdateProduct amendment is done and an error is recorded instead. There is also no copy of this, even the upgrade part, being sent to SFDC.

The only other explanation I can think of is that a customer made it to the True-up flow and then decided to walk away and not move forward, but CustomersDot still sent Zuora the upgrade portion.

**Possible Impacts and Effects:**
1. I am not sure of the experience these customers receive, if the flow seems to work or they receive an error.
1. Licenses might not be being sent out to the customer.
1. Invoices might not be being generated for these transactions.
1. Despite the customer taking action in the web store (by the presence of the seat increase) we are effectively letting their subscription run out on them.
1. These upgrade/renewals are not being sent to Salesforce, lowering our Net ARR attainment as it looks like there were going to upgrade.
1. In fact we are churning many of these renewals in Salesforce because of the lack of renewal. We have taken **~125k of Churn (-$125k Net ARR)** related to these subs so far this year that might have to be reversed.

**Findings**:
The Sales team raised some good examples of strange Net ARR data on some of march's churned opportunities and we dug into their Zuora Subscription data and found examples of subscriptions ending with an additional seats purchase but then no extension of their term (renewal). 

Both examples below have this in their Notes field, which looks like there was an issue with the the True up behavior in the renewal flow: `RENEWAL_ERROR: ${:code=>“RENEWAL_ERROR”, :message=>“Enter a number of active users greater than or equal to your current number of active users (10)“}$` which originates from: https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/staging/app/services/amend_subscription_service.rb#L243

- Example 1: https://www.zuora.com/apps/Subscription.do?method=view&id=2c92a00d77fc14ea017809fc0a27731e
- Example 2: https://www.zuora.com/apps/Subscription.do?method=view&id=2c92a00777f8d0860177fa8912370fce (Note: Billing actually finds this one and is confused about the lack of Invoice or Web Direct, and then Ammends the customer back to their old seat count: https://gitlab.my.salesforce.com/5004M00000j6rRg)

**Further Analysis:**
- Using the `RENEWAL_ERROR` in the Notes field as a marker, I was able to find 264 Subscriptions with an error. All but 1 of the subscriptions was created since 2/19/2021.
- Doing a cross reference using our Customer Subscription system in Salesforce, I was able to find 56 Renewals that have been Closed Lost since 2/19/2021 totaling ~-$150k Net ARR. These were closed lost as part of standard Sales Ops clean up of expired subscriptions. If these churns were in error that loss of Net ARR should be reversed and any additional users would be positive Net ARR. I have attached a scrubbed csv for anyone else who wants to review.

cc @james_harrison @lschoenfeld @tav_scott this is the end findings of the Opps that you all raised today. Please watch this issue for any developments as Fulfillment investigates.

### Proposal

Look into the use cases mentioned and why renewals amendments aren't being created. Specifically:

1. Subscriptions renewing for less quantity than the previous term #2837 
   1. Currently in this case, only the error gets recorded and no Amendments made.
   1. Example: https://www.zuora.com/apps/Subscription.do?method=view&id=2c92a00d77b9842f0177b99ba8c403c9
1. Subscriptions renewing for more quantity than the previous term
   1. Example: This subscription had UpdateProduct amendment, but then an Error and no auto-renewal. https://www.zuora.com/apps/Subscription.do?method=view&id=2c92a00c77bf60da0177c1e313380c06


### Result

Auto-renewals will happen for all subscriptions that are either increasing or decreasing Users.

### Next steps (if any)

### How will we measure success?

[SubscriptionsWithRenewalError.csv](/uploads/73c0d56120163fdd7be036ca072b11f8/SubscriptionsWithRenewalError.csv)

Story points: 3.0