text: # Streaming at Scale integration tests
  
  ## Integration test suite
  
  The suite is designed to run in a fully automated manner in Azure DevOps. It
  deploys each scenario from the streaming-at-scale project in a new resource
  group, then runs a verification job in Azure Databricks that reads the entire
  set of events generated by the scenario, and tests assertions on the throughput
  and latency. For instance, after running the eventhubs-functions-cosmosdb
  scenario at 5000 events/second, the verification job downloads the output data
  stored in CosmosDB and computes the end-to-end latency and throughput
  aggregated at one-minute interval. To account for some runtime variability, the
  test asserts that a throughput of at least 90% of the expected throughput (in
  that case, 270,000 events in a one-minute interval) was reached at least once.
  
  Since the provisioning of an Azure Databricks workspace cannot be fully
  automated at present, you must generate a PAT token of a preexisting workspace
  and supply it to the pipeline.
  
  ## Installing the build agent
  
  As the integration tests can run for more than 6 hours, they must be run on self-hosted Azure DevOps agents.
  
  Create a project in Azure DevOps. Create an agent pool named "streaming-at-scale".
  
  In the Azure portal, create an Azure VM with Ubuntu 18.04 LTS.
  
  SSH to the VM and run the following commands interactively one a time.
  
  ```bash
  # Install Azure DevOps agent. When prompted enter the Azure DevOps host and a PAT token with Agent Pool management permissions.
  mkdir agent
  cd agent
  wget https://vstsagentpackage.azureedge.net/agent/2.155.1/vsts-agent-linux-x64-2.155.1.tar.gz
  tar zxvf vsts-agent-linux-x64-2.155.1.tar.gz 
  ./config.sh 
  sudo ./svc.sh install
  sudo ./svc.sh start
  # Install jq
  sudo apt update
  sudo apt install jq
  # Install zip
  sudo apt install zip
  # Install az (Azure CLI)
  curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
  # Install dotnet SDK
  wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
  sudo dpkg -i packages-microsoft-prod.deb
  sudo add-apt-repository universe
  sudo apt-get install apt-transport-https
  sudo apt-get update
  sudo apt-get install dotnet-sdk-2.2
  # Install maven
  sudo apt install maven
  ```
  
  
  ## Creating the integration test pipeline in Azure DevOps
  
  * Create a Databricks workspace in the Azure region of your choice:
    * tier: standard
    * make sure the workspace is deployed with a custom VNET (as the HDInsight
      Kafka setup will need to peer VNETs).
    * You can use the [Databricks VNET
    template](https://azure.microsoft.com/en-us/resources/templates/101-databricks-all-in-one-template-for-vnet-injection/),
    changing the tier to standard on the deployment screen.
  * Install a build agent (instructions below).
  * In Azure AD, create a service principal. Grant the service principal
    *Owner* permission on your subscription (we can't use the automatic
    service connection dialog in Azure DevOps, as that would only grant
    *Contributor* permissions. We need *Owner* in order to grant Azure Kubernetes
    Service permissions to pull containers from Azure Container Registry).
  * In your Azure DevOps project settings, navigate to service connection and
    create an ARM service connection to your Azure subscription named
    'ARMConnection'. *Use the full version* of the dialog to use your service
    principal.
  * Create a build pipeline:
    * As pipeline source, use https://github.com/Azure-Samples/streaming-at-scale.git.
    * As pipeline content, reference integration-tests/azure-pipelines.yaml.
    * Define the following  pipeline variables:
  
  | Variable name          | Description                                    | Required? | Example    |
  | --------------------   | ---------------------------------------------- | --------- | ---------- |
  | LOCATION               | Azure region in which to deploy infrastructure | required  | eastus     |
  | DATABRICKS_PAT_TOKEN   | (secret variable) Databricks PAT token for a Databricks workspace deployed in $LOCATION | required | dapi012345... |
  | DATABRICKS_VNET | Databricks VNET name (default value: 'databricks-vnet') | optional | databricks-vnet |
  | DATABRICKS_VNET_RESOURCE_GROUP | Resource Group containing the Databricks VNET | required | streamingitests |
  | RESOURCE_GROUP_PREFIX  | Prefix used to name deployed resources. Must be globally unique, use a sufficiently unique string  | required | xyzzy0x4 |
  | AGENT_VM_RESOURCE_GROUP | Resource group of the build agent VM  | required | streamingitests |
  | AGENT_VM_NAME          | Name of the build agent VM  | required | streamingbuildagent |
  
  
  ## Running the integration tests
  
  Running the pipeline will generate multiple runner jobs, and execute them in
  sequence. Each scenario is deployed in a newly created resource group, that is
  immediately deleted after the test is run.
  
  At the end of the job output of the 'Run test' task is a link to the Databricks
  job run URL for the verification job. You can open this URL to view the run
  notebook output. At the end of the main notebook is a call to the
  'verify-common' notebook job, open that notebook run to view detailed
  statistics collected by the verification job.
  
  