OH Testing: UC-12

## Summary

This is a specific testing issue for use case 12 on https://gitlab.com/groups/gitlab-org/-/epics/6606

### Testing DRIs

|Tester ID | Tester Name | Status |
|:---:|---|---|
| 1 |  @a_luna | UC-12a :white_check_mark: Completed & Passed <br> UC-12b :white_check_mark: Completed & Passed <br> UC-12c :white_check_mark: Completed & Passed |
| 2 |  |  |

### UC-12: Quarterly reconciliation adds seats to a subscription

<details>
<summary>UC-12.a: Self-managed subscription</summary>

| Step ID | Test Step   | Description | Expected Outcome | Testable? | Pass/Fail |
|---------|--------|----|----------|:---:|:---:|
| T-1     | Purchase Premium for 5 users (12 months)                                             | New Subscription Purchase. SM Premium link: https://customers.staging.gitlab.com/subscriptions/new?plan_id=2c92a01176f0d50a0176f3043c4d4a53                                                                                                                   | (1) Purchase flow is successful. <br> <br> (2) Completes with legacy license and activation code email.                                                                                                                                                                                                                                                                                                                                                                                                                                                           |Y | |
| T-2     | Activate license                                                                     | Receive the code via email and follow the activation steps in the email. Must be on version xx.x+                                            | (1) The instance successfully activates and provisions paid features. <br> <br> (2) `Turn On Cloud Licensing` flag in Zuora is set to `Yes` <br> <br> (3) `Turn On Seat Reconciliation` flag in Zuora in Zuora is set to `Yes`|Y | |
| T-3     | View license/subscription details                                                    | Navigate to the Subscription page in the SM instance to verify the license details are accurate                                              | Validate Subscription `start date`, `end date`, `Users in License` <br> <br> Billable Users count should be `1`, Max Users count should be `1`, Users in License count should be `5`, Users over License count should be `0`                                                                                                                                                                                                                                                                                                                                      |Y | |
| T-4     | Create 6 users in your instance. Assign them to a group with the role of `developer` | https://docs.gitlab.com/ee/user/profile/account/create_accounts.html <br> <br> https://docs.gitlab.com/ee/user/group/#add-users-to-a-group             | This is not a test, just a required step for the test. Expected outcome is to have 7 billable users on the instance.                                                                                                                                                                                                                                                                                                                                                                                                                                    | N/A| N/A|
| T-5     | View license/subscription details                                                    | Navigate to the Subscription page in the SM instance to verify the instance has exceed its licensed user count                               | This is not a test to verify a new feature, just a step to validate your test steps until now are valid.                                                                                                                                                                                                                                                                                                                                                                                                                                                |N/A |N/A |
|T-6| Purchase 2 users in CustomersDot |Select `Add seats`, purchase `2` additional seats|Expect that the purchase will prorate the charge for the remainder of the subscription period. This is not a test, just a required step for the test. |N/A|
|T-7| Receive purchased users | (1) One tester can use the `Sync` button on the instance to immediately provision the 2 purchased users, (2) One test can wait 24 hours to verify that the 2 users are provisioned after the next sync job|Y| |
| T-8     | View license/subscription details| Navigate to the Subscription page in the SM instance to verify `Users in License` = `7` <br> <br> Billable Users count should be `7`, Max Users count should be `7`, Users in License count should be `7`, Users over License count should be `0`  |Y |N/A |
| T-9     | Q1 reconciliation occurs| This reconciliation step will need to be simulated/forced by development.| This subscription is eligible for quarterly reconciliation because they are sending us data, but there is no overage, so no add-on is processed.        |Y | |
| T-10     | Block 2 users in the instance| https://docs.gitlab.com/ee/api/users.html#block-user                                                                                         | This is not a test, just a required step for the test.|N/A |N/A |
| T-11     | View license/subscription details| Navigate to the Subscription page in the SM instance to verify user counts.| Billable Users count should be `5`, Max Users count should be `7`, Users in License count should be `7`, Users over License count should be `0` <br> <br> This is not a test to verify a new feature, just a step to validate your test steps until now are valid.                                                                                                                                                                                                                                                                                                |N/A |N/A |
| T-12    | Q2 reconciliation occurs| This reconciliation step will need to be simulated/forced by development.                                                                    | This subscription is elgible for quarterly reconciliation because they are sending us data, but there is no overage, so no add-on is processed.|Y| |
| T-13    | Add 4 new users and assign them to projects with role "maintainer"                   | https://docs.gitlab.com/ee/user/profile/account/create_accounts.html https://docs.gitlab.com/ee/user/project/members/#add-users-to-a-project | This is not a test, just a required step for the test.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |N/A |N/A |
| T-14    | View license/subscription details                                                    | Navigate to the Subscription page in the SM instance to verify user counts.                                                                  | Billable Users count should be `9`, Max Users count should be `9`, Users in License count should be `7`, Users over License count should be `2` <br> <br> This is not a test to verify a new feature, just a step to validate your test steps until now are valid.                                                                                                                                                                                                                                                                                                |N/A |N/A |
| T-15    | Q3 reconciliation occurs                                                             | This reconciliation step will need to be simulated/forced by development.                                                                    | (1) the customer is charged for an overage of `2` Premium user for the remaining 1 quarter of the agreement. <br> <br>(2) email sent to customer <br> <br>(3) The system records that reconciliation was performed <br> <br>(4) Data sent to SFDC - [opp created? details?] | Y| |
| T-16    | View license/subscription details                                                    | Navigate to the Subscription page in the SM instance to verify user counts.                                                                  | Billable Users count should be `9`, Max Users count should be `9`, Users in License count should be `9`, Users over License count should be `0` <br> <br> This is not a test to verify a new feature, just a step to validate your test steps until now are valid.                                                                                                                                                                                                                                                                                               |N/A |N/A|
| T-17    | Add 4 new users and assign them to projects with role "guest"                        | https://docs.gitlab.com/ee/user/profile/account/create_accounts.html <br> <br> https://docs.gitlab.com/ee/user/project/members/#add-users-to-a-project | This is not a test, just a required step for the test.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |N/A |N/A|
| T-18    | View license/subscription details                                                    | Navigate to the Subscription page in the SM instance to verify user counts.                                                                  | Billable Users count should be `13`, Max Users count should be `13`, Users in License count should be `9`, Users over License count should be `4` <br> <br> This is not a test to verify a new feature, just a step to validate your test steps until now are valid.                                                                                                                                                                                                                                                                                              |N/A |N/A|
| T-19   | No Q4 reconciliation occurs.                                                         |                                                                                                                                              | We cannot test this real-world step since we are forcing reconciliations.                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | N/A|N/A|

</details>

<details>
<summary>UC-12.b: SaaS subscription</summary>

| Step ID | Test Step   | Description| Expected Outcome |Testable? | Pass/Fail|
|---------|--------|----|----------|:---:|:---:|
|T-1     | Create a group in staging.gitlab.com with 5 users (team members). Just select existing users. | This is not a test, just a required step for the test.|N/A|N/A|
|T-2| Purchase Premium  | New SaaS Subscription Purchase for created group. SaaS Premium link: https://customers.staging.gitlab.com/subscriptions/new?plan_id=2c92a00d76f0d5060176f2fb0a5029ff <br> <br> This is not a test, just a required step for the test.                                                        | (1) Purchase flow is successful for 5 users. <br> <br> (2) Completes with email confirmation.                                                                                                                                                                                                                                                                                                                                                                                                   |N/A | N/A|
|T-3 | Confirm billing details|This is not a test, just a required step for the test.                                                        |In staging.gitlab.com, navigate to Group > Settings > Billing, Validate Subscription `start date`, `end date`, `Seats in subscription` = `5`|N/A | N/A|
| T-4 | Add 2 users (team members) to you group with the role of `developer`. Just select existing users. | https://docs.gitlab.com/ee/user/group/#add-users-to-a-group    <br> <br> This is not a test, just a required step for the test.                                                                | Expected outcome is to have 7 billable users on the instance.                                                                                                                                                                                                                                                                                                                                                                                                                                    | N/A| N/A|
| T-5     | View subscription details | This is not a test, just a required step for the test. (note the values are updated every 24 hours).<br/><br/>To force the refresh, SSH into staging.gitlab.com, open the rails console and run the following steps:<br/>`ns = Namespace.find <Group ID> # Find this in the top of group main page (i.e. https://staging.gitlab.com/<Group Name>)`<br/>`gs = ns.gitlab_subscription`<br/>`gs # Verify that it reflects what is shown on the Billing page`<br/>`gs.refresh_seat_attributes! # This will force the refresh`<br/>`gs # Verify that it reflects the updated seat numbers`<br/>`gs.save!`                                                       | Navigate to Group > Settings > Billing. `Seats in subscription` s/b `5`, `Seats currently in use` s/b `7`, `Max seats used` s/b `7`, `Seats owed` s/b `2` (note these values are updated every 24 hours).                                                                                                                                                                                                                                                                                                                                                                                                                                                        |N/A |N/A |
|T-6 | Make a purchase in CustomersDot |Select `Add seats`, purchase `3` additional seats|Expect that the purchase will prorate the charge for the remainder of the subscription period. This is not a test, just a required step for the test. |N/A|
| T-7     | Q1 reconciliation occurs| This reconciliation step will need to be simulated/forced by development.<br/><br/>SSH into customers.staging.gitlab.com (requires access), open a fresh rails console and run the following steps:<br/>`ns_id = <Group ID> # Find this in the top of group main page (i.e. https://staging.gitlab.com/<Group Name>)`<br/>`order = Order.where(gl_namespace_id: ns_id).first # Verify that gl_namespace_name is the name of the group you created`<br/>`order.reconciliations.count # Verify that one exists for this order`<br/>`r = order.reconciliations.first # Check that the reconciliation_done_at is nil.`<br/> `Reconciliations::UpdateSaasUserCountService.new(r).execute`<br/>`ReconciliationService.new(r).execute`<br/>`r.reload # Verify that user count is the expected amount and no errors`                                                                    | This subscription is eligible for quarterly reconciliation because they are sending us data, but there is no overage, so no add-on is processed.      |Y | |
| T-8     | Remove 2 users (team members) from the group| https://docs.gitlab.com/ee/user/group/#remove-a-member-from-the-group                                                                                        | This is not a test, just a required step for the test.|N/A |N/A |
| T-9     | View subscription details| This is not a test, just a required step for the test.                                                        | Navigate to Group > Settings > Billing. `Seats in subscription` s/b `8`, `Seats currently in use` s/b `5`, `Max seats used` s/b `7`, `Seats owed` s/b `0` (note these values are updated every 24 hours)                   |N/A |N/A |
| T-10    | Q2 reconciliation occurs                                                             | This reconciliation step will need to be simulated/forced by development.  <br> <br>Open a fresh rails console to avoid any caching problems from the last reconciliation event and follow the steps from the previous reconciliation above.                                                                  | (1) The customer is not charged for any overage <br> <br> (2) No email is sent to the customer <br> <br> (3)The system records that while usage data was received, there was no overage so no reconciliation was performed.                                                                                                                                                                                                                                                                                                                                                   |Y| |
| T-11    | Add 4 new users (team members) and assign them to projects with role "maintainer". Just select existing users.                   | https://docs.gitlab.com/ee/user/project/members/#add-users-to-a-project | This is not a test, just a required step for the test.|N/A |N/A |
| T-12     | View subscription details| This is not a test, just a required step for the test.                                                        |Navigate to Group > Settings > Billing. `Seats in subscription` s/b `8`, `Seats currently in use` s/b `9`, `Max seats used` s/b `9`, `Seats owed` s/b `1` (note these values are updated every 24 hours)                   |N/A |N/A |
| T-13    | Q3 reconciliation occurs| This reconciliation step will need to be simulated/forced by development.  <br> <br>Open a fresh rails console to avoid any caching problems from the last reconciliation event and follow the steps from the previous reconciliation above. <br> After the `Reconciliations::UpdateSaasUserCountService.new(r).execute` step, run the commands in [this comment](https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/3824#note_758028735) | (1a) Where cc on file: the customer is charged for an overage of `1` Premium user for the remaining 1 quarter of the agreement. <br> - ALTERNATE TEST- <br> (1b) Where cc not on file: [tbd] <br> <br>(2a) Where reconciliation successful + payment successful: email sent to customer <br> - ALTERNATE TEST- <br> (2b) Where reconciliation was not successful due to payment error: [tbd] <br> - ALTERNATE TEST- <br> (2c) Where reconciliation was success + no cc on file: [tbd] <br> <br>(3) The system records that reconciliation was performed <br> <br>(4) Data sent to SFDC - [opp created? details?] | Y| |
| T-14     | View subscription details|This is not a test, just a required step for the test.                                                        | Navigate to Group > Settings > Billing. `Seats in subscription` s/b `9`, `Seats currently in use` s/b `9`, `Max seats used` s/b `9`, `Seats owed` s/b `0` (note these values are updated every 24 hours)                   |N/A |N/A |
| T-15    | Add 4 new users (team members) and assign them to projects with role "guest". Just select existing users.                        | https://docs.gitlab.com/ee/user/project/members/#add-users-to-a-project | This is not a test, just a required step for the test.|N/A |N/A|
| T-16     | View subscription details| This is not a test, just a required step for the test.                                                        |Navigate to Group > Settings > Billing. `Seats in subscription` s/b `9`, `Seats currently in use` s/b `13`, `Max seats used` s/b `13`, `Seats owed` s/b `4` (note these values are updated every 24 hours)                  |N/A |N/A |
| T-17   | No Q4 reconciliation occurs. | This reconciliation step will need to be simulated/forced by development.  <br> <br>Open a fresh rails console to avoid any caching problems from the last reconciliation event and follow the steps from the previous reconciliation above.  | The reconciliation should be skipped with a skip reason of `last_quarter`. | N/A |

</details>

<details>
<summary>UC-12.c: SaaS subscription with an Add-on </summary>

Follow the same steps as above, but after purchasing the plan at step T-2, also purchase additional storage. 

Link to purchase storage package: https://customers.staging.gitlab.com/subscriptions/new?plan_id=2c92a00f7279a6f5017279d299d01cf9.

</details>

Story points: 1.0