Project: Dapr (App Definition and Development / Application Definition & Image Build)
File: running-e2e-test.md

E2E tests are designed for verifying the functional correctness by replicating end-user behavior from app deployment. This describes how to run e2e tests. Set up Get a Docker container registry: If using Docker Hub, create your Docker Hub ID Other options include Azure Container Registry, GitHub Container Registry, etc Set the environment variables ```bash export DAPRREGISTRY=docker.io/yourdockerhub_id export DAPR_REGISTRY=myregistry.azurecr.io export DAPR_TAG=dev export DAPR_NAMESPACE=dapr-tests export DAPRMTLSENABLED=true export DAPRTESTENV=minikube export MINIKUBENODEIP=yourk8smaster_ip ``` If you need to create the `DAPRTESTREGISTRY_SECRET` variable, you can use this command: ```sh DOCKER_REGISTRY="<url of the registry, such as myregistry.azurecr.io>" DOCKER_USERNAME="<your username>" DOCKER_PASSWORD="<your password>" DOCKER_EMAIL="<your email (leave empty if not required)>" export DAPRTESTREGISTRY_SECRET=$( kubectl create secret docker-registry --dry-run=client docker-regcred \ --docker-server="${DOCKER_REGISTRY}" \ --docker-username="${DOCKER_USERNAME}" \ --docker-password="${DOCKER_PASSWORD}" \ --docker-email=${DOCKER_EMAIL} \ -o json | \ jq -r '.data.".dockerconfigjson"' ) ``` If you are starting from scratch and just want to build dapr, deploy it, and run the e2e tests to your kubernetes cluster, do the following: Uninstall dapr, dapr-kafka, dapr-redis, dapr-postgres, if they exist Make sure you have DAPR_NAMESPACE set properly before you do this! ```sh helm uninstall dapr dapr-kafka dapr-redis dapr-postgres -n $DAPR_NAMESPACE ``` Remove the test namespace, if it exists ```bash make delete-test-namespace ``` > Note: please make sure that you have executed helm uninstall command before you deleted dapr test namespace. Otherwise if you directly deleted the dapr test namespace without helm uninstall command and re-installed dapr control plane, the dapr sidecar injector won't work and fail for "bad certificate". And you have already run into this problem, you can recover by helm uninstall command. See https://github.com/dapr/dapr/issues/4612 Build, deploy, run tests from start to finish ```bash make e2e-build-deploy-run ``` We also have individual targets to allow for quick iteration on parts of deployment and testing. To follow all or part of these steps individually, do the following: Create dapr-tests namespace ```bash make create-test-namespace ``` Install redis and kafka for state, pubsub, and binding building block ```bash make setup-helm-init make setup-test-env-redis make setup-test-env-postgres make setup-test-env-kafka ``` Run the below commands to build and deploy dapr from your local disk ```bash make build-linux make docker-build make docker-push make docker-deploy-k8s ``` ```bash make setup-disable-mtls ``` ```bash make setup-test-components ``` Build docker images from apps and push the images to test docker hub ```bash make build-e2e-app-all make push-e2e-app-all ``` Run end-to-end tests ```bash make test-e2e-all ``` If you'd rather run a subset of end-to-end test, set the environmental variable `DAPRE2ETEST` with the name(s) of the test(s) (space-separated). These are the names of folders within the `tests/e2e`

```sh DAPRE2ETEST="actor_reminder" make test-e2e-all ``` To completely remove Dapr, test dependencies, and any lingering e2e test apps: Make sure you have DAPR_NAMESPACE set properly before you do this! ```bash helm uninstall dapr -n $DAPR_NAMESPACE || true helm uninstall dapr-kafka -n $DAPR_NAMESPACE || true helm uninstall dapr-redis -n $DAPR_NAMESPACE || true helm uninstall dapr-postgres -n $DAPR_NAMESPACE || true kubectl delete deployment dapr-zipkin -n $DAPR_NAMESPACE || true make delete-test-namespace ``` To keep the build infrastructure simple, Dapr uses to run e2e tests using one of . A separate workflow also runs E2E in clusters. Once a contributor creates a pull request, E2E tests on KinD clusters are automatically executed for faster feedback. In order to run the E2E tests on AKS, ask a maintainer or approver to add `/ok-to-test` comment to the Pull Request. This repository's automated tests (CI) use an Azure Kubernetes Service (AKS) cluster to run E2E tests. If you want to run the tests in a similar environment, you can deploy the test infrastructure on your own using the Bicep templates in . Here are the scripts you could use. If you want to deploy AKS and the Azure Container Registry only (without Azure Cosmos DB and Azure Service Bus), you can use this script: ```sh AZURE_REGION="eastus2" export TEST_PREFIX="mydapraks42" ENABLE_WINDOWS="false" export TESTRESOURCEGROUP="MyDaprTest" az group create \ --resource-group "${TESTRESOURCEGROUP}" \ --location "${AZURE_REGION}" az deployment group create \ --resource-group "${TESTRESOURCEGROUP}" \ --template-file ./tests/test-infra/azure-aks.bicep \ --parameters namePrefix=${TESTPREFIX} location=${AZUREREGION} enableWindows=${ENABLE_WINDOWS} az acr login --name "${TEST_PREFIX}acr" az aks get-credentials -n "${TESTPREFIX}-aks" -g "${TESTRESOURCE_GROUP}" export DAPRREGISTRY="${TESTPREFIX}acr.azurecr.io" export DAPR_NAMESPACE=dapr-tests make create-test-namespace ``` After this, run the E2E tests as per instructions above, making sure to use the newly-created Azure Container Registry as Docker registry (make sure you maintain the environmental variables set in the steps above). This is the setup that our E2E tests use in GitHub Actions, which includes AKS, Azure Cosmos DB, and Azure Service Bus, in addition to an Azure Container Registry. To replicate the same setup, run: NOTE: This deploys an Azure Service Bus instance with ultra-high performance and it is very expensive. If you deploy this, don't forget to shut it down after you're done! ```sh AZURE_REGION="eastus2" export TEST_PREFIX="mydapraks42" ENABLE_WINDOWS="false" export TESTRESOURCEGROUP="MyDaprTest" az group create \ --resource-group "${TESTRESOURCEGROUP}" \ --location "${AZURE_REGION}" az deployment group create \ --resource-group "${TESTRESOURCEGROUP}" \ --template-file ./tests/test-infra/azure.bicep \ --parameters namePrefix=${TESTPREFIX} location=${AZUREREGION} enableWindows=${ENABLE_WINDOWS} az acr login --name "${TEST_PREFIX}acr" az aks get-credentials -n "${TESTPREFIX}-aks" -g "${TESTRESOURCE_GROUP}" export DAPRREGISTRY="${TESTPREFIX}acr.azurecr.io" export DAPR_NAMESPACE=dapr-tests make create-test-namespace ./tests/test-infra/setup_azure.sh true true false ``` After this, run the E2E tests as per instructions above, making sure to use the newly-created Azure Container Registry as Docker registry (make sure you maintain the environmental variables set in the steps above). The command in can also run in AKS. Here is a sample script. ```sh export ONLYDAPRIMAGE=true export TEST_PREFIX="testprefix" export DAPRREGISTRY="${TESTPREFIX}acr.azurecr.io" export DAPR_TAG=dev export DAPR_NAMESPACE=dapr-tests export DAPRTESTNAMESPACE="dapr-tests" export DAPRTESTTAG="${DAPR_TAG}-linux-amd64" export DAPRTESTREGISTRY=${DAPR_REGISTRY} export DEBUG_LOGGING=true az acr login --name ${DAPR_REGISTRY} make build DOCKERFILE=Dockerfile-mariner make docker-build docker-push make build-e2e-app-all make push-e2e-app-all make create-test-namespace make docker-deploy-k8s make setup-3rd-party setup-test-components make test-clean DAPRE2ETEST="pubsub" make test-e2e-all make test-e2e-all ``` You can optionally configure AKS to collect certain logs, including: All container logs, sent to Azure Log Analytics Diagnostic logs (`kube-apiserver` and `kube-controller-manager`), sent to Azure Log Analytics Audit logs (`kube-audit`), sent to Azure Storage To do that, first provision the required resources by deploying the `azure-aks-diagnostic.bicep` template (this template is not part of the `azure.bicep` or

templates, as it's considered shared infrastructure): ```sh DIAGRESOURCEGROUP="MyDaprTestLogs" DIAGNAMEPREFIX="mydaprdiag42" az group create \ --resource-group "${DIAGRESOURCEGROUP}" \ --location "${AZURE_REGION}" az deployment group create \ --resource-group "${DIAGRESOURCEGROUP}" \ --template-file ./tests/test-infra/azure-aks-diagnostic.bicep \ --parameters name=${DIAGNAMEPREFIX} location=${AZURE_REGION} ``` The output of the last command includes two values that are resource IDs: `diagLogAnalyticsWorkspaceResourceId`, for example: `/subscriptions/<subscription>/resourcegroups/<resource group>/providers/Microsoft.OperationalInsights/workspaces/<workspace name>` `diagStorageResourceId`, for example: `/subscriptions/<subscription>/resourcegroups/<resource group>/providers/Microsoft.Storage/storageAccounts/<storage account name>` Use those values as parameters when deploying the `azure.bicep` or `azure-all.bicep` templates. For example: ```sh az deployment group create \ --resource-group "${TESTRESOURCEGROUP}" \ --template-file ./tests/test-infra/azure.bicep \ --parameters namePrefix=${TESTPREFIX} location=${AZUREREGION} enableWindows=${ENABLE_WINDOWS} diagLogAnalyticsWorkspaceResourceId=... diagStorageResourceId=... ``` ```sh export DAPR_NAMESPACE=dapr-tests make test-clean helm uninstall dapr -n $DAPR_NAMESPACE || true helm uninstall dapr-kafka -n $DAPR_NAMESPACE || true helm uninstall dapr-redis -n $DAPR_NAMESPACE || true helm uninstall dapr-mongodb -n $DAPR_NAMESPACE || true helm uninstall dapr-temporal -n $DAPR_NAMESPACE || true kubectl delete deployment dapr-zipkin -n $DAPR_NAMESPACE || true kubectl delete namespace $DAPR_NAMESPACE || true make delete-test-namespace ``` is a zero-config VPN that provides NAT traversal out-of-the-box allowing our services and pods to be called directly - using its ClusterIP - without needed to be exposed using a loadbalancer. This provides a few advantages including the decrease of the total test duration. if you want to run the tests using tailscale as your network, few things are necessary: , this will be necessary since we're going to use personal keys. the tailscale client for your OS. When you're logged in, navigate to the menu `Access Controls` and two things are necessary, edit the ACL definition with: Create a new tag that will be used later to assign permissions to keys. ```json {... "tagOwners": { "tag:dapr-tests": ["youremailcomeshere@yourdomain.com"], } } ``` Assign permissions to the created tag. Since we are going to use the , it is much convenient that the subnet router should auto approve the registered routes, for that, use the following acl. ```json {... "autoApprovers": { "routes": { "10.0.0.0/8": ["tag:dapr-tests"], }, } } ``` > Warning: as we are using `10.0.0.0/8` we must guarantee that our CIDR block used in the kubernetes cluster must be a subset of it Now, go to the Settings > Personal Settings > Keys. Once in the keys section, generate a new ephemeral key by clicking in `Generate auth key`. Mark as `reusable`, `ephemeral` and add the created tag `dapr-tests` and do not forget to copy out the value. Now, we're almost set. The next step will be install the tailscale subnet router in your kubernetes cluster, for that, run ```sh TAILSCALEAUTHKEY=yourkeygoes_here make setup-tailscale ``` TIP: for security reasons you could run `unset HISTFILE` before the tailscale command so that will discard your history file Now, you have to login on tailscale client using your personal account, to verify if the subnet router deployment works, browse to the `Machines` on the tailscale portal, the subnet router should show up there. One more config is necessary, `TESTE2EUSEINTERNALIP=true`, you can use it as a variable when running tests as the following: ```sh TESTE2EUSEINTERNALIP=true make test-e2e-all ```