Repository: saleor/saleor
Pull request: Add database query count tests for API
Closes #3946.

### Screenshots

<!-- If your changes affect the UI, providing "before" and "after" screenshots will
greatly reduce the amount of work needed to review your work. -->

### Pull Request Checklist

<!-- Please keep this section. It will make maintainer's life easier. -->

1. [ ] Privileged views and APIs are guarded by proper permission checks.
1. [ ] All visible strings are translated with proper context.
1. [ ] All data-formatting is locale-aware (dates, numbers, and so on).
1. [ ] Database queries are optimized and the number of queries is constant.
1. [ ] Database migration files are up to date.
1. [x] The changes are tested.
1. [ ] The code is documented (docstrings, project documentation).
1. [ ] GraphQL schema and type definitions are up to date.
1. [ ] Changes are mentioned in the changelog.


Problem statement:
Tests for counting db queries in GraphQL queries and mutations
### What I'm trying to achieve
I want to have some tests that will give me a better perspective for usage of DB queries under graphql API.  Now we don't have any information about numbers of queries hitting DB. Most of the fields have optimization but even this I would like to have explicit logic to validate that it works as expected. Also, it would be helpful during any refactor of any graphQL logic.

### Describe a proposed solution
It would be great to have some basic tests to validate number of DB queries executed under a given graphql view.

Hints from issue discussion:
Do you have an idea on how we would check if the query was performed in a correct number of queries? Would we check that the number of queries is lower than a given number? If so, how would we determine the way to do so?

Anyway, there are two available fixtures:
- `django_assert_num_queries`;
- `django_assert_max_num_queries`.

The other way would be to retrieve the query count through the `CaptureQueriesContext` context manager (`len(...)` of the context will give the number of queries).
We still don't have a clear idea on how to "test" the query count. I started working on something, I will discuss my ideas and current progress in our next meeting.

Overall, I feel like the best way to "test" the performances, would be to define some benchmark tests (a new page, outside of `tests/` maybe a name close from golang with a `tests_bench/` or something like that).

So instead of doing useless assertions saying "I want this queries to perform in N queries or less", we would generate a report and see if our changes are actually improving the query count or not.

So, at the end, we would have a table showing us every differences between two commits (e.g. the parent tree, master vs the latest commit of the current tree). We would have a huge list of tests, and some red or green colors on the results depending on how well they performed.

I would assume that we would run those tests only once per commit/build. Probably on the latest LTS django and latest supported python. As the python version shouldn't affect the number of queries.

Then, only problem, it may not be enough to only focus on the query count. We could do some time based tests, but that would be more time consuming. That's something that we could dig up some other day.

In final words, the pytest bench tests will be structured the following way:
```
tests_bench/
  |- conftest.py
  |- products/
        |- test_queries.py
        |- test_mutations.py
  |- orders/
        |- test_queries.py
        |- test_mutations.py
```

Then the rapport would look like something like this:

```
## Products
### Queries
| test name       | Queries count <parent name> (<hash number>) | current <hash> | diff |
| --------------- | ------------------------------------------- | -------------- | ---- |
| List products   |                                           5 |              3 |   -2 |
| Product details |                                          11 |             10 |   -1 |
```

This would, of course, support parametrized tests. Which could be really useful and powerful to test many cases.

Commit message: Add database query count tests for homepage and checkout

Patch to review:
diff --git a/tests/api/test_checkout_bench.py b/tests/api/test_checkout_bench.py
new file mode 100644
index 00000000000..9acd0508099
--- /dev/null
+++ b/tests/api/test_checkout_bench.py
@@ -0,0 +1,602 @@
+import pytest
+from django.conf import settings
+from graphene import Node
+
+from saleor.checkout.utils import add_variant_to_checkout
+from saleor.payment import ChargeStatus, TransactionKind
+from saleor.payment.models import Payment
+from tests.api.utils import get_graphql_content
+
+
+@pytest.fixture()
+def checkout_with_variant(checkout, variant):
+    add_variant_to_checkout(checkout, variant, 1)
+    checkout.save()
+    return checkout
+
+
+@pytest.fixture()
+def checkout_with_shipping_method(checkout_with_variant, shipping_method):
+    checkout = checkout_with_variant
+
+    checkout.shipping_method = shipping_method
+    checkout.save()
+
+    return checkout
+
+
+@pytest.fixture()
+def checkout_with_billing_address(checkout_with_shipping_method, address):
+    checkout = checkout_with_shipping_method
+
+    checkout.billing_address = address
+    checkout.save()
+
+    return checkout
+
+
+@pytest.fixture()
+def checkout_with_charged_payment(checkout_with_billing_address):
+    checkout = checkout_with_billing_address
+
+    payment = Payment.objects.create(
+        gateway=settings.DUMMY,
+        is_active=True,
+        total=checkout.get_total().gross.amount,
+        currency="USD",
+    )
+
+    payment.charge_status = ChargeStatus.FULLY_CHARGED
+    payment.captured_amount = payment.total
+    payment.checkout = checkout_with_billing_address
+    payment.save()
+
+    payment.transactions.create(
+        amount=payment.total,
+        kind=TransactionKind.CAPTURE,
+        gateway_response={},
+        is_success=True,
+    )
+
+    return checkout
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_create_checkout(api_client, graphql_address_data, variant):
+    query = """
+        fragment Price on TaxedMoney {
+          gross {
+            amount
+            localized
+            __typename
+          }
+          currency
+          __typename
+        }
+
+        fragment ProductVariant on ProductVariant {
+          id
+          name
+          price {
+            amount
+            currency
+            localized
+            __typename
+          }
+          product {
+            id
+            name
+            thumbnail {
+              url
+              alt
+              __typename
+            }
+            thumbnail2x: thumbnail(size: 510) {
+              url
+              __typename
+            }
+            __typename
+          }
+          __typename
+        }
+
+        fragment CheckoutLine on CheckoutLine {
+          id
+          quantity
+          totalPrice {
+            ...Price
+            __typename
+          }
+          variant {
+            stockQuantity
+            ...ProductVariant
+            __typename
+          }
+          quantity
+          __typename
+        }
+
+        fragment Address on Address {
+          id
+          firstName
+          lastName
+          companyName
+          streetAddress1
+          streetAddress2
+          city
+          postalCode
+          country {
+            code
+            country
+            __typename
+          }
+          countryArea
+          phone
+          __typename
+        }
+
+        fragment ShippingMethod on ShippingMethod {
+          id
+          name
+          price {
+            currency
+            amount
+            localized
+            __typename
+          }
+          __typename
+        }
+
+        fragment Checkout on Checkout {
+          availablePaymentGateways
+          token
+          id
+          user {
+            email
+            __typename
+          }
+          totalPrice {
+            ...Price
+            __typename
+          }
+          subtotalPrice {
+            ...Price
+            __typename
+          }
+          billingAddress {
+            ...Address
+            __typename
+          }
+          shippingAddress {
+            ...Address
+            __typename
+          }
+          email
+          availableShippingMethods {
+            ...ShippingMethod
+            __typename
+          }
+          shippingMethod {
+            ...ShippingMethod
+            __typename
+          }
+          shippingPrice {
+            ...Price
+            __typename
+          }
+          lines {
+            ...CheckoutLine
+            __typename
+          }
+          __typename
+        }
+
+        mutation createCheckout($checkoutInput: CheckoutCreateInput!) {
+          checkoutCreate(input: $checkoutInput) {
+            errors {
+              field
+              message
+              __typename
+            }
+            checkout {
+              ...Checkout
+              __typename
+            }
+            __typename
+          }
+        }
+    """
+    variables = {
+        "checkoutInput": {
+            "email": "test@example.com",
+            "shippingAddress": graphql_address_data,
+            "lines": [
+                {
+                    "quantity": 1,
+                    "variantId": Node.to_global_id("ProductVariant", variant.pk),
+                }
+            ],
+        }
+    }
+    get_graphql_content(api_client.post_graphql(query, variables))
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_add_shipping_to_checkout(
+    api_client, graphql_address_data, variant, checkout_with_variant, shipping_method
+):
+    query = """
+        fragment Price on TaxedMoney {
+          gross {
+            amount
+            localized
+            __typename
+          }
+          currency
+          __typename
+        }
+
+        fragment ProductVariant on ProductVariant {
+          id
+          name
+          price {
+            amount
+            currency
+            localized
+            __typename
+          }
+          product {
+            id
+            name
+            thumbnail {
+              url
+              alt
+              __typename
+            }
+            thumbnail2x: thumbnail(size: 510) {
+              url
+              __typename
+            }
+            __typename
+          }
+          __typename
+        }
+
+        fragment CheckoutLine on CheckoutLine {
+          id
+          quantity
+          totalPrice {
+            ...Price
+            __typename
+          }
+          variant {
+            stockQuantity
+            ...ProductVariant
+            __typename
+          }
+          quantity
+          __typename
+        }
+
+        fragment Address on Address {
+          id
+          firstName
+          lastName
+          companyName
+          streetAddress1
+          streetAddress2
+          city
+          postalCode
+          country {
+            code
+            country
+            __typename
+          }
+          countryArea
+          phone
+          __typename
+        }
+
+        fragment ShippingMethod on ShippingMethod {
+          id
+          name
+          price {
+            currency
+            amount
+            localized
+            __typename
+          }
+          __typename
+        }
+
+        fragment Checkout on Checkout {
+          availablePaymentGateways
+          token
+          id
+          user {
+            email
+            __typename
+          }
+          totalPrice {
+            ...Price
+            __typename
+          }
+          subtotalPrice {
+            ...Price
+            __typename
+          }
+          billingAddress {
+            ...Address
+            __typename
+          }
+          shippingAddress {
+            ...Address
+            __typename
+          }
+          email
+          availableShippingMethods {
+            ...ShippingMethod
+            __typename
+          }
+          shippingMethod {
+            ...ShippingMethod
+            __typename
+          }
+          shippingPrice {
+            ...Price
+            __typename
+          }
+          lines {
+            ...CheckoutLine
+            __typename
+          }
+          __typename
+        }
+
+        mutation updateCheckoutShippingOptions(
+          $checkoutId: ID!
+          $shippingMethodId: ID!
+        ) {
+          checkoutShippingMethodUpdate(
+            checkoutId: $checkoutId
+            shippingMethodId: $shippingMethodId
+          ) {
+            errors {
+              field
+              message
+              __typename
+            }
+            checkout {
+              ...Checkout
+              __typename
+            }
+            __typename
+          }
+        }
+    """
+    variables = {
+        "checkoutId": Node.to_global_id("Checkout", checkout_with_variant.pk),
+        "shippingMethodId": Node.to_global_id("ShippingMethod", shipping_method.pk),
+    }
+    get_graphql_content(api_client.post_graphql(query, variables))
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_add_billing_address_to_checkout(
+    api_client, graphql_address_data, checkout_with_shipping_method
+):
+    query = """
+        fragment Price on TaxedMoney {
+          gross {
+            amount
+            localized
+            __typename
+          }
+          currency
+          __typename
+        }
+
+        fragment ProductVariant on ProductVariant {
+          id
+          name
+          price {
+            amount
+            currency
+            localized
+            __typename
+          }
+          product {
+            id
+            name
+            thumbnail {
+              url
+              alt
+              __typename
+            }
+            thumbnail2x: thumbnail(size: 510) {
+              url
+              __typename
+            }
+            __typename
+          }
+          __typename
+        }
+
+        fragment CheckoutLine on CheckoutLine {
+          id
+          quantity
+          totalPrice {
+            ...Price
+            __typename
+          }
+          variant {
+            stockQuantity
+            ...ProductVariant
+            __typename
+          }
+          quantity
+          __typename
+        }
+
+        fragment Address on Address {
+          id
+          firstName
+          lastName
+          companyName
+          streetAddress1
+          streetAddress2
+          city
+          postalCode
+          country {
+            code
+            country
+            __typename
+          }
+          countryArea
+          phone
+          __typename
+        }
+
+        fragment ShippingMethod on ShippingMethod {
+          id
+          name
+          price {
+            currency
+            amount
+            localized
+            __typename
+          }
+          __typename
+        }
+
+        fragment Checkout on Checkout {
+          availablePaymentGateways
+          token
+          id
+          user {
+            email
+            __typename
+          }
+          totalPrice {
+            ...Price
+            __typename
+          }
+          subtotalPrice {
+            ...Price
+            __typename
+          }
+          billingAddress {
+            ...Address
+            __typename
+          }
+          shippingAddress {
+            ...Address
+            __typename
+          }
+          email
+          availableShippingMethods {
+            ...ShippingMethod
+            __typename
+          }
+          shippingMethod {
+            ...ShippingMethod
+            __typename
+          }
+          shippingPrice {
+            ...Price
+            __typename
+          }
+          lines {
+            ...CheckoutLine
+            __typename
+          }
+          __typename
+        }
+
+        mutation updateCheckoutBillingAddress(
+          $checkoutId: ID!
+          $billingAddress: AddressInput!
+        ) {
+          checkoutBillingAddressUpdate(
+            checkoutId: $checkoutId
+            billingAddress: $billingAddress
+          ) {
+            errors {
+              field
+              message
+              __typename
+            }
+            checkout {
+              ...Checkout
+              __typename
+            }
+            __typename
+          }
+        }
+    """
+    variables = {
+        "checkoutId": Node.to_global_id("Checkout", checkout_with_shipping_method.pk),
+        "billingAddress": graphql_address_data,
+    }
+    get_graphql_content(api_client.post_graphql(query, variables))
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_checkout_payment_charge(
+    api_client, graphql_address_data, checkout_with_billing_address
+):
+    query = """
+        mutation createPayment($input: PaymentInput!, $checkoutId: ID!) {
+          checkoutPaymentCreate(input: $input, checkoutId: $checkoutId) {
+            errors {
+              field
+              message
+              __typename
+            }
+            __typename
+          }
+        }
+    """
+
+    variables = {
+        "checkoutId": Node.to_global_id("Checkout", checkout_with_billing_address.pk),
+        "input": {
+            "billingAddress": graphql_address_data,
+            "amount": 1000,  # 10.00 USD * 100
+            "gateway": settings.DUMMY.upper(),
+            "token": "charged",
+        },
+    }
+    get_graphql_content(api_client.post_graphql(query, variables))
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_complete_checkout(api_client, checkout_with_charged_payment):
+    query = """
+        mutation completeCheckout($checkoutId: ID!) {
+          checkoutComplete(checkoutId: $checkoutId) {
+            errors {
+              field
+              message
+              __typename
+            }
+            order {
+              id
+              token
+              __typename
+            }
+            __typename
+          }
+        }
+    """
+
+    variables = {
+        "checkoutId": Node.to_global_id("Checkout", checkout_with_charged_payment.pk)
+    }
+
+    get_graphql_content(api_client.post_graphql(query, variables))
diff --git a/tests/api/test_homepage_bench.py b/tests/api/test_homepage_bench.py
new file mode 100644
index 00000000000..c8ac4c5b9ba
--- /dev/null
+++ b/tests/api/test_homepage_bench.py
@@ -0,0 +1,180 @@
+import pytest
+
+from tests.api.utils import get_graphql_content
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_retrieve_main_menu(api_client):
+    query = """
+        fragment MainMenuSubItem on MenuItem {
+          id
+          name
+          category {
+            id
+            name
+            __typename
+          }
+          url
+          collection {
+            id
+            name
+            __typename
+          }
+          page {
+            slug
+            __typename
+          }
+          parent {
+            id
+            __typename
+          }
+          __typename
+        }
+
+        query MainMenu {
+          shop {
+            navigation {
+              main {
+                id
+                items {
+                  ...MainMenuSubItem
+                  children {
+                    ...MainMenuSubItem
+                    children {
+                      ...MainMenuSubItem
+                      __typename
+                    }
+                    __typename
+                  }
+                  __typename
+                }
+                __typename
+              }
+              __typename
+            }
+            __typename
+          }
+        }
+    """
+
+    get_graphql_content(api_client.post_graphql(query))
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_retrieve_secondary_menu(api_client):
+    query = """
+        fragment SecondaryMenuSubItem on MenuItem {
+          id
+          name
+          category {
+            id
+            name
+            __typename
+          }
+          url
+          collection {
+            id
+            name
+            __typename
+          }
+          page {
+            slug
+            __typename
+          }
+          __typename
+        }
+
+        query SecondaryMenu {
+          shop {
+            navigation {
+              secondary {
+                items {
+                  ...SecondaryMenuSubItem
+                  children {
+                    ...SecondaryMenuSubItem
+                    __typename
+                  }
+                  __typename
+                }
+                __typename
+              }
+              __typename
+            }
+            __typename
+          }
+        }
+    """
+    get_graphql_content(api_client.post_graphql(query))
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_retrieve_shop(api_client):
+    query = """
+        query getShop {
+          shop {
+            defaultCountry {
+              code
+              country
+              __typename
+            }
+            countries {
+              country
+              code
+              __typename
+            }
+            geolocalization {
+              country {
+                code
+                country
+                __typename
+              }
+              __typename
+            }
+            __typename
+          }
+        }
+    """
+
+    get_graphql_content(api_client.post_graphql(query))
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_retrieve_product_list(api_client):
+    query = """
+        query ProductsList {
+          shop {
+            description
+            name
+            homepageCollection {
+              id
+              backgroundImage {
+                url
+                __typename
+              }
+              name
+              __typename
+            }
+            __typename
+          }
+          categories(level: 0, first: 4) {
+            edges {
+              node {
+                id
+                name
+                backgroundImage {
+                  url
+                  __typename
+                }
+                __typename
+              }
+              __typename
+            }
+            __typename
+          }
+        }
+    """
+    get_graphql_content(api_client.post_graphql(query))
diff --git a/tests/api/test_payment_bench.py b/tests/api/test_payment_bench.py
new file mode 100644
index 00000000000..f29134fb57f
--- /dev/null
+++ b/tests/api/test_payment_bench.py
@@ -0,0 +1,12 @@
+import pytest
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_get_payment_token(api_client):
+    query = """
+        query getPaymentToken($gateway: GatewaysEnum!) {
+          paymentClientToken(gateway: $gateway)
+        }
+    """
+    assert query is not None  # so flake stops being upset about the unused variable


Code review comments:
diff_hunk: @@ -0,0 +1,602 @@
+import pytest
+from django.conf import settings
+from graphene import Node
+
+from saleor.checkout.utils import add_variant_to_checkout
+from saleor.payment import ChargeStatus, TransactionKind
+from saleor.payment.models import Payment
+from tests.api.utils import get_graphql_content
+
+
+@pytest.fixture()
+def checkout_with_variant(checkout, variant):
+    add_variant_to_checkout(checkout, variant, 1)
+    checkout.save()
+    return checkout
+
+
+@pytest.fixture()
+def checkout_with_shipping_method(checkout_with_variant, shipping_method):
+    checkout = checkout_with_variant
+
+    checkout.shipping_method = shipping_method
+    checkout.save()
+
+    return checkout
+
+
+@pytest.fixture()
+def checkout_with_billing_address(checkout_with_shipping_method, address):
+    checkout = checkout_with_shipping_method
+
+    checkout.billing_address = address
+    checkout.save()
+
+    return checkout
+
+
+@pytest.fixture()
+def checkout_with_charged_payment(checkout_with_billing_address):
+    checkout = checkout_with_billing_address
+
+    payment = Payment.objects.create(
+        gateway=settings.DUMMY,
+        is_active=True,
+        total=checkout.get_total().gross.amount,
+        currency="USD",
+    )
+
+    payment.charge_status = ChargeStatus.FULLY_CHARGED
+    payment.captured_amount = payment.total
+    payment.checkout = checkout_with_billing_address
+    payment.save()
+
+    payment.transactions.create(
+        amount=payment.total,
+        kind=TransactionKind.CAPTURE,
+        gateway_response={},
+        is_success=True,
+    )
+
+    return checkout
+
+
+@pytest.mark.django_db
+@pytest.mark.count_queries
+def test_create_checkout(api_client, graphql_address_data, variant):
+    query = """
+        fragment Price on TaxedMoney {
+          gross {
+            amount
+            localized
+            __typename
path: tests/api/test_checkout_bench.py
text: @user1:
We can safely drop all `__typename` occurrences - they're used by Apollo internally and don't impact the database performance.