Repository: RasaHQ/rasa
Pull request: Model regression tests - show last performance on master
Solves #6393 

Problem statement:
Model regression tests: Show last performance on master
**Description of Problem**:
Currently the model regression tests displays the performance of a PR on the defined set of configurations and datasets. To validate if the PR improves/degrades the performance, the user needs to know the last recorded performance of the scheduled run on the master branch. While internally, this is viewable on Metabase, it's time taking to check for each configuration what numbers have changed from the last run. It would be useful if the numbers from the last scheduled run are shown in the comment of the tests.

**Overview of the Solution**:
1. Fetch the results of the latest scheduled run from segment
2. Show the difference between numbers captured in the PR run v/s the numbers extracted from the last scheduled run.

**Examples** (if relevant):
The resultant results table could look like - 

Dataset: `Carbon Bot`

| Configuration | Intent Classification Micro F1 | Entity Recognition Micro F1 | Response Selection Micro F1 |
|---------------|-----------------|-----------------|-------------------|
| `BERT + DIET(bow) + ResponseSelector(bow)`<br> test: `1m1s`, train: `2m29s`, total: `3m29s` |0.7709(0.2)|0.6260(0.0)|0.5133(-0.1)|

where the numbers in the brackets are the difference between numbers captured in the PR run v/s the numbers extracted from the last scheduled run.

cc @tczekajlo

Hints from issue discussion:

Commit message: Revert changes related to tests

Patch to review:
diff --git a/.github/templates/model_regression_test_results.tmpl b/.github/templates/model_regression_test_results.tmpl
index 3fca8c42b8ff..0f3f8ecac98c 100644
--- a/.github/templates/model_regression_test_results.tmpl
+++ b/.github/templates/model_regression_test_results.tmpl
@@ -5,7 +5,9 @@ as an artifact in the model regression tests workflow) and returns
 a markdown table with a summary of the tests.
 
 */ -}}
+{{- $results_master := (datasource "results_master") -}}
 {{ range $dataset, $config := (datasource "data")}}
+{{- $dataset_master := (index $results_master $dataset) -}}
 Dataset: `{{$dataset}}`
 
 | Configuration | Intent Classification Micro F1 | Entity Recognition Micro F1 | Response Selection Micro F1 |
@@ -13,20 +15,35 @@ Dataset: `{{$dataset}}`
 {{ range $config_name, $config_data := $config -}}
 | `{{ $config_name }}`<br> test: `{{ $config_data.test_run_time }}`, train: `{{ $config_data.train_run_time }}`, total: `{{ $config_data.total_run_time }}` |
 {{- if has $config_data "intent_classification" -}}
+{{- /* Use results from branch if there is lack of data doesn't exist in results from scheduled version */ -}}
+{{- $intent_class_master := $config_data.intent_classification -}}
+{{- if has $dataset_master $config_name -}}
+{{- $intent_class_master = (index $dataset_master $config_name).intent_classification -}}
+{{- end -}}
 {{- $intent_class := index $config_data.intent_classification -}}
-{{- if (has (index $intent_class "micro avg") "f1-score") -}}{{ printf "%.4f" (index (index $intent_class "micro avg") "f1-score") }}{{- else if (has $intent_class "accuracy") -}}{{ printf "%.4f" $intent_class.accuracy }}{{- else -}}`no data`{{- end -}} |
+{{- if (has (index $intent_class "micro avg") "f1-score") -}}{{ printf "%.4f" (index (index $intent_class "micro avg") "f1-score") }} ({{ printf "%.2f" ((index (index $intent_class_master "micro avg") "f1-score") | math.Sub (index (index $intent_class "micro avg") "f1-score")) }}){{- else if (has $intent_class "accuracy") -}}{{ printf "%.4f" $intent_class.accuracy }} ({{ printf "%.2f" ($intent_class_master.accuracy | math.Sub $intent_class.accuracy) }}){{- else -}}`no data`{{- end -}} |
 {{- else -}}
 `no data`|
 {{- end -}}
 {{- if has $config_data "entity_prediction" -}}
+{{- /* Use results from branch if there is lack of data doesn't exist in results from scheduled version */ -}}
+{{- $entity_class_master := index $config_data.entity_prediction -}}
+{{- if has $dataset_master $config_name -}}
+{{- $entity_class_master = (index $dataset_master $config_name).entity_prediction -}}
+{{- end -}}
 {{- $entity_class := $config_data.entity_prediction -}}
-{{- if (has (index $entity_class "micro avg") "f1-score") -}}{{ printf "%.4f" (index (index $entity_class "micro avg") "f1-score") }}{{- else if (has $entity_class "accuracy") -}}{{ printf "%.4f" $entity_class.accuracy }}{{- else -}}`no data`{{- end -}} |
+{{- if (has (index $entity_class "micro avg") "f1-score") -}}{{ printf "%.4f" (index (index $entity_class "micro avg") "f1-score") }} ({{ printf "%.2f" ((index (index $entity_class_master "micro avg") "f1-score") | math.Sub (index (index $entity_class "micro avg") "f1-score")) }}){{- else if (has $entity_class "accuracy") -}}{{ printf "%.4f" $entity_class.accuracy }} ({{ printf "%.2f" ($entity_class_master.accuracy | math.Sub $entity_class.accuracy) }}){{- else -}}`no data`{{- end -}} |
 {{- else -}}
 `no data`|
 {{- end -}}
 {{- if has $config_data "response_selection" -}}
+{{- /* Use results from branch if there is lack of data doesn't exist in results from scheduled version */ -}}
+{{- $response_class_master := index $config_data.response_selection -}}
+{{- if has $dataset_master $config_name -}}
+{{- $response_class_master = (index $dataset_master $config_name).response_selection -}}
+{{- end -}}
 {{- $response_class := $config_data.response_selection -}}
-{{- if (has (index $response_class "micro avg") "f1-score") -}}{{ printf "%.4f" (index (index $response_class "micro avg") "f1-score") }}{{- else if (has $response_class "accuracy") -}}{{ printf "%.4f" $response_class.accuracy }}{{- else -}}`no data`{{- end -}} |
+{{- if (has (index $response_class "micro avg") "f1-score") -}}{{ printf "%.4f" (index (index $response_class "micro avg") "f1-score") }} ({{ printf "%.2f" ((index (index $response_class_master "micro avg") "f1-score") | math.Sub (index (index $response_class "micro avg") "f1-score")) }}){{- else if (has $response_class "accuracy") -}}{{ printf "%.4f" $response_class.accuracy }} ({{ printf "%.2f" ($response_class_master.accuracy | math.Sub $response_class.accuracy) }}){{- else -}}`no data`{{- end -}} |
 {{- else -}}
 `no data`|
 {{- end }}
diff --git a/.github/workflows/ci-model-regression.yml b/.github/workflows/ci-model-regression.yml
index c46bd7f0a8c2..b90afca16d1e 100644
--- a/.github/workflows/ci-model-regression.yml
+++ b/.github/workflows/ci-model-regression.yml
@@ -460,6 +460,22 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v2
 
+      - name: Download an artifact with the results for the schedule version
+        run: |
+          # Schedule workflows ID - 1498790
+
+          ARTIFACT_URL=$(curl -s -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}'  -H "Accept: application/vnd.github.v3+json" \
+            "https://api.github.com/repos/RasaHQ/rasa/actions/workflows/1498790/runs?event=schedule&status=completed&branch=master&per_page=1" | jq -r .workflow_runs[0].artifacts_url)
+
+          DOWNLOAD_URL=$(curl -s -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' -H "Accept: application/vnd.github.v3+json" "${ARTIFACT_URL}" \
+            | jq -r '.artifacts[] | select(.name="report.json") | .archive_download_url')
+
+          # Download the artifact
+          curl -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' -LJO  -H "Accept: application/vnd.github.v3+json" $DOWNLOAD_URL
+
+          # Unzip and change name
+          unzip report.json.zip && mv report.json report_master.json
+
       - name: Download the report
         uses: actions/download-artifact@v2
         with:
@@ -473,7 +489,7 @@ jobs:
       - name: Render a comment to add
         id: get_results
         run: |
-          OUTPUT="$(gomplate -d data=report.json -f .github/templates/model_regression_test_results.tmpl)"
+          OUTPUT="$(gomplate -d data=report.json -d results_master=report_master.json -f .github/templates/model_regression_test_results.tmpl)"
           OUTPUT="${OUTPUT//$'\n'/'%0A'}"
           OUTPUT="${OUTPUT//$'\r'/'%0D'}"
           echo "::set-output name=result::$OUTPUT"
@@ -517,6 +533,22 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v2
 
+      - name: Download an artifact with the results for the schedule version
+        run: |
+          # Schedule workflows ID - 1498790
+
+          ARTIFACT_URL=$(curl -s -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}'  -H "Accept: application/vnd.github.v3+json" \
+            "https://api.github.com/repos/RasaHQ/rasa/actions/workflows/1498790/runs?event=schedule&status=completed&branch=master&per_page=1" | jq -r .workflow_runs[0].artifacts_url)
+
+          DOWNLOAD_URL=$(curl -s -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' -H "Accept: application/vnd.github.v3+json" "${ARTIFACT_URL}" \
+            | jq -r '.artifacts[] | select(.name="report.json") | .archive_download_url')
+
+          # Download the artifact
+          curl -H 'Authorization: token ${{ secrets.GITHUB_TOKEN }}' -LJO  -H "Accept: application/vnd.github.v3+json" $DOWNLOAD_URL
+
+          # Unzip and change name
+          unzip report.json.zip && mv report.json report_master.json
+
       - name: Download the report
         uses: actions/download-artifact@v2
         with:
@@ -530,7 +562,7 @@ jobs:
       - name: Render a comment to add
         id: get_results
         run: |
-          OUTPUT="$(gomplate -d data=report.json -f .github/templates/model_regression_test_results.tmpl)"
+          OUTPUT="$(gomplate -d data=report.json -d results_master=report_master.json -f .github/templates/model_regression_test_results.tmpl)"
           OUTPUT="${OUTPUT//$'\n'/'%0A'}"
           OUTPUT="${OUTPUT//$'\r'/'%0D'}"
           echo "::set-output name=result::$OUTPUT"


Code review comments:
diff_hunk: @@ -460,6 +460,22 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v2
 
+      - name: Download an artifact with the results for the schedule version
+        run: |
+          # Schedule workflows ID - 1498790
path: .github/workflows/ci-model-regression.yml
text: @user1:
is there a way we can fetch the ID for the _latest_ or _current_ version of `ci-model-regression-on-schedule.yml`? this is not ideal as it needs to be manually updated if the other file changes.

if it has to happen via ID, we need to add comments here that the ID has to be updated if `ci-model-regression-on-schedule.yml` changes, and also in `ci-model-regression-on-schedule.yml` to update the ID in this file

@user1:
also we need to think about the case where another PR changes `ci-model-regression-on-schedule.yml`, so simply getting the latest ID wouldn't be right. can we fetch the latest workflow ID of `ci-model-regression-on-schedule.yml` for the current branch?

@author:
The IDs for workflows are pretty static, they don't change even if there are several branches that have different versions of the same workflow, additionally, schedule workflows can exist only in the master branch.

I've changed a bit this part and now, we fetch ID via GH API for the scheduled workflow.