Repository: Lightning-AI/pytorch-lightning
Pull request: adding CI for e2e on Azure
## What does this PR do?

REsolves #13927

## Before submitting

- [x] Was this **discussed/approved** via a GitHub issue? (not for typos and docs)
- [x] Did you read the [contributor guideline](https://github.com/Lightning-AI/lightning/blob/master/.github/CONTRIBUTING.md), **Pull Request** section?
- [x] Did you make sure your **PR does only one thing**, instead of bundling different changes together?
- [ ] Did you make sure to **update the documentation** with your changes? (if necessary)
- [ ] Did you write any **new necessary tests**? (not for typos and docs)
- [ ] Did you verify new and **existing tests pass** locally with your changes?
- [ ] Did you list all the **breaking changes** introduced by this pull request?
- [ ] Did you **update the CHANGELOG**? (not for typos, docs, test updates, or minor internal changes/refactors)

<!-- In the CHANGELOG, separate each item in the unreleased section by a blank line to reduce collisions -->

## PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the [review guidelines](https://github.com/Lightning-AI/lightning/wiki/Review-guidelines). In short, see the following bullet-list:

- [x] Is this pull request ready for review? (if not, please submit in draft mode)
- [x] Check that all items from **Before submitting** are resolved
- [x] Make sure the title is self-explanatory and the description concisely explains the PR
- [x] Add labels and milestones (and optionally projects) to the PR so it can be classified

## Did you have fun?

Make sure you had fun coding 🙃


cc @carmocca @akihironitta @borda

Problem statement:
Run app cloud CI e2e tests on PRs from forks
## 🚀 Feature

<!-- A clear and concise description of the feature proposal -->

Run app cloud CI e2e tests on PRs from forks.

### Motivation

Currently, the cloud e2e tests require secrets to run and because the secrets (e.g. lightning_user_id/ lightning_api_key) are not shared with the forks, they run only on PRs from lightning branches only.  This is necessary to allow all contributers to run e2e cloud tests on their PRs 🎉 


<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link it here -->

### Pitch

<!-- A clear and concise description of what you want to happen. -->

### Alternatives

Ideas:
- Move tests to Azure to allow sharing secrets.

<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->

### Additional context

<!-- Add any other context or screenshots about the feature request here. -->


cc: @Borda 
______________________________________________________________________



cc @carmocca @akihironitta @borda @tchaton @rohitgr7

Hints from issue discussion:
Let's do it! We would like to start merging community PRs as soon as possible. Example:  https://github.com/Lightning-AI/lightning/pull/13531

Commit message: cache

Patch to review:
diff --git a/.azure/app-cloud-e2e.yml b/.azure/app-cloud-e2e.yml
new file mode 100644
index 0000000000000..dba64eba28a12
--- /dev/null
+++ b/.azure/app-cloud-e2e.yml
@@ -0,0 +1,127 @@
+# Python package
+# Create and test a Python package on multiple Python versions.
+# Add steps that analyze code, save the dist with the build record, publish to a PyPI-compatible index, and more:
+# https://docs.microsoft.com/azure/devops/pipelines/languages/python
+
+trigger:
+  tags:
+    include:
+      - '*'
+  branches:
+    include:
+      - "master"
+      - "release/*"
+      - "refs/tags/*"
+
+pr:
+  - "master"
+  - "release/*"
+
+variables:
+  # variables are automatically exported as environment variables so this will override pip's default cache dir
+  - name: pip_cache_dir
+    value: $(Pipeline.Workspace)/.pip
+
+jobs:
+  - job: App_cloud_e2e_testing
+    pool:
+      vmImage: 'ubuntu-latest'
+    timeoutInMinutes: "30"
+    cancelTimeoutInMinutes: "2"
+    strategy:
+      matrix:
+        'App: v0_app':
+          repo: "v0_app"
+        'App: boring_app':
+          repo: "boring_app"
+        'App: template_streamlit_ui':
+          repo: "template_streamlit_ui"
+        'App: template_react_ui':
+          repo: "template_react_ui"
+        'App: template_jupyterlab':
+          repo: "template_jupyterlab"
+        'App: idle_timeout':
+          repo: "idle_timeout"
+        'App: collect_failures':
+          repo: "collect_failures"
+        'App: custom_work_dependencies':
+          repo: "custom_work_dependencies"
+        'App: drive':
+          repo: "drive"
+        'App: payload':
+          repo: "payload"
+        'App: commands_and_api':
+          repo: "commands_and_api"
+    workspace:
+      clean: all
+    steps:
+    - task: UsePythonVersion@0
+      inputs:
+        versi`onSpec: '3.8'
+
+    # TODO: parse the PR number
+
+    - task: Cache@2
+      inputs:
+        key: 'pip | "$(repo)" | requirements/app/base.txt'
+        restoreKeys: |
+          pip | "$(Agent.OS)"
+        path: $(pip_cache_dir)
+      displayName: Cache pip
+
+    - bash: |
+        python --version
+        pip --version
+      displayName: 'Info'
+
+    - bash: python -m pip install -r requirements/app/devel.txt --no-cache --quiet --find-links https://download.pytorch.org/whl/cpu/torch_stable.html
+      displayName: 'Install dependencies'
+
+    - bash: |
+        python -m pip install playwright
+        python -m playwright install --with-deps
+      displayName: 'Install Playwright system dependencies'
+
+    - bash: pip install -e .
+      displayName: 'Install lightning'
+
+#    - checkout: git://MyProject/MyToolsRepo@features/mytools  # TODO: Lightning-AI/lightning-template-react
+#      path: examples/app_template_react_ui
+#      displayName: 'Clone Template React UI Repo'
+
+#    - checkout: git://MyProject/MyToolsRepo@features/mytools  # TODO: Lightning-AI/lightning-template-jupyterlab
+#      path: examples/app_template_jupyterlab
+#      displayName: 'Clone Template Jupyter Lab Repo'
+
+#    - bash: cp examples/app_template_jupyterlab/tests/test_template_jupyterlab.py tests/tests_app_examples/test_template_jupyterlab.py
+#      displayName: 'Copy Template Jupyter Lab Repo tests'
+
+    - bash: |
+        mkdir -p ${VIDEO_LOCATION}
+        ls -l tests/tests_app_examples
+        python -m pytest tests/tests_app_examples/test_${TEST_APP_NAME}.py::test_${TEST_APP_NAME}_example_cloud --timeout=900 --capture=no -v --color=yes
+      env:
+        HEADLESS: "1"
+        PACKAGE_LIGHTNING: "1"
+        CLOUD: "1"
+        VIDEO_LOCATION: ./artifacts/videos
+        PR_NUMBER: $(date +%s)  # TODO: use PR number
+        TEST_APP_NAME: $(repo)
+        HAR_LOCATION: ./artifacts/hars
+        SLOW_MO: 50
+        LAI_USER: $(LAI_USER)
+        LAI_PASS: $(LAI_PASS)
+        LIGHTNING_USER_ID: $(LIGHTNING_USER_ID)
+        LIGHTNING_API_KEY: $(LIGHTNING_API_KEY)
+        LIGHTNING_USERNAME: $(LIGHTNING_USERNAME)
+        LIGHTNING_CLOUD_URL: $(LIGHTNING_CLOUD_URL)
+      workingDirectory: tests/tests_pytorch
+      displayName: 'Run the tests'
+
+    - task: PublishTestResults@2
+      displayName: 'Publish test results'
+      inputs:
+        testResultsFiles: '$(Build.StagingDirectory)/artifacts/videos'
+        testRunTitle: 'App $(repo)'
+
+    # TODO: Clean Previous Apps
diff --git a/.github/workflows/ci-app-cloud-e2e-test.yml b/.github/workflows/ci-app-cloud-e2e-test.yml
deleted file mode 100644
index b2281389e7358..0000000000000
--- a/.github/workflows/ci-app-cloud-e2e-test.yml
+++ /dev/null
@@ -1,187 +0,0 @@
-name: cloud-testing
-
-# Used to run the e2e tests on lightning.ai
-
-on: # Trigger the workflow on push or pull request, but only for the master branch
-  push:
-    branches: [master, "release/*"]
-  pull_request:
-    branches: [master, "release/*"]
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref }}
-  cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
-
-jobs:
-  # This is job should once only once per PR to detect file changes so run required jobs.
-  # see .github/file-filters.yml to define file filters and run the jobs based on the output of each filter.
-  # More info: https://github.com/marketplace/actions/paths-changes-filter
-
-  changes:
-    runs-on: ubuntu-latest
-    # Set job outputs to the values from filter step
-    outputs:
-      app_examples: ${{ steps.filter.outputs.app_examples }}
-    steps:
-      - uses: actions/checkout@v2
-      - name: Set up Python 3.8
-        uses: actions/setup-python@v2
-        with:
-          python-version: "3.8"
-
-      - uses: dorny/paths-filter@v2
-        id: filter
-        with:
-          filters: .github/file-filters.yml
-
-  cloud-test:
-    name: Cloud Test
-    needs: changes
-    if: ${{ needs.changes.outputs.app_examples == 'true' }}
-    runs-on: ubuntu-20.04
-    strategy:
-      fail-fast: false
-      matrix:
-        app_name:
-          - v0_app
-          - boring_app
-#          - quick_start  # TODO: fix this
-          - template_streamlit_ui
-          - template_react_ui
-          - template_jupyterlab
-          - idle_timeout
-          - collect_failures
-          - custom_work_dependencies
-          - drive
-          - payload
-          - commands_and_api
-    timeout-minutes: 35
-    steps:
-      - uses: actions/checkout@v2
-      - name: Set up Python 3.8
-        uses: actions/setup-python@v4
-        with:
-          python-version: "3.8"
-
-      - name: Get PR ID
-        id: PR
-        run: |
-          if [ -z ${{github.event.number}} ]; then
-            echo "::set-output name=ID::$(date +%s)"
-          else
-            echo "::set-output name=ID::${{github.event.number}}"
-          fi
-
-# TODO: Enable cache
-#      - name: Cache virtualenv
-#        id: cache-venv
-#        uses: actions/cache@v3
-#        with:
-#          path: ./.venv/
-#          key: ${{ runner.os }}-pip-${{ matrix.app_name }}-${{ hashFiles('requirements/app/base.txt', 'requirements/app/*.txt', 'src/lightning_app/__version__.py') }}
-#          restore-keys: ${{ runner.os }}-venv-${{ matrix.app_name }}-
-
-      - name: Install dependencies
-        shell: bash
-        run: |
-          pip --version
-          python -m pip install -r requirements/app/devel.txt --no-cache --quiet --find-links https://download.pytorch.org/whl/cpu/torch_stable.html
-#        if: steps.cache-venv.outputs.cache-hit != 'true'  # TODO: Enable cache
-
-      - name: Cache Playwright dependencies
-        id: playwright-cache
-        uses: actions/cache@v3
-        with:
-          path: ~/.cache/ms-playwright
-          key: ${{ runner.os }}-playwright-${{ matrix.app_name }}-${{ hashFiles('requirements/app/base.txt', 'requirements/app/*.txt', 'src/lightning_app/__version__.py') }}
-          restore-keys: ${{ runner.os }}-playwright-${{ matrix.app_name }}-
-
-      - name: Install Playwright system dependencies
-        shell: bash
-        run: |
-          python -m pip install playwright
-          python -m playwright install --with-deps
-#        if: steps.playwright-cache.outputs.cache-hit != 'true'  # TODO: Enable cache
-
-      - name: Install lightning
-        run: |
-          pip install -e .
-        shell: bash
-
-      - name: Lightning Install quick-start
-        if: ${{ (matrix.app_name == 'quick_start') }}
-        shell: bash
-        run: |
-          python -m lightning install app lightning/quick-start -y
-
-      - name: Clone Template React UI Repo
-        uses: actions/checkout@v3
-        with:
-          repository: Lightning-AI/lightning-template-react
-          token: ${{ secrets.PAT_GHOST }}
-          ref: 'master'
-          path: examples/app_template_react_ui
-
-      - name: Clone Template Jupyter Lab Repo
-        uses: actions/checkout@v3
-        with:
-          repository: Lightning-AI/lightning-template-jupyterlab
-          token: ${{ secrets.PAT_GHOST }}
-          ref: 'master'
-          path: examples/app_template_jupyterlab
-
-      - name: Copy Template Jupyter Lab Repo tests
-        shell: bash
-        run: cp examples/app_template_jupyterlab/tests/test_template_jupyterlab.py tests/tests_app_examples/test_template_jupyterlab.py
-
-      - name: List pip dependency
-        shell: bash
-        run: |
-          pip list
-
-      - name: Run the tests
-        env:
-          LAI_USER: ${{ secrets.LAI_USER }}
-          LAI_PASS: ${{ secrets.LAI_PASS }}
-          LIGHTNING_USER_ID: ${{ secrets.LIGHTNING_USER_ID }}
-          LIGHTNING_API_KEY: ${{ secrets.LIGHTNING_API_KEY }}
-          LIGHTNING_USERNAME: ${{ secrets.LIGHTNING_USERNAME }}
-          LIGHTNING_CLOUD_URL: ${{ secrets.LIGHTNING_CLOUD_URL }}
-          CLOUD: "1"
-          VIDEO_LOCATION: ./artifacts/videos
-          PR_NUMBER: ${{ steps.PR.outputs.ID }}
-          TEST_APP_NAME: ${{ matrix.app_name }}
-          HAR_LOCATION: ./artifacts/hars
-          SLOW_MO: 50
-        shell: bash
-        run: |
-          mkdir -p ${VIDEO_LOCATION}
-          HEADLESS=1 PACKAGE_LIGHTNING=1 python -m pytest tests/tests_app_examples/test_${{ matrix.app_name }}.py::test_${{ matrix.app_name }}_example_cloud --timeout=900 --capture=no -v --color=yes
-          # Delete the artifacts if successful
-          rm -r ${VIDEO_LOCATION}/${{ matrix.app_name }}
-
-      - uses: actions/upload-artifact@v2
-
-        if: ${{ always() }}
-        with:
-          name: test-artifacts
-          path: ./artifacts/videos
-
-      - name: Clean Previous Apps
-        if: ${{ always() }}
-        env:
-          LAI_USER: ${{ secrets.LAI_USER }}
-          LAI_PASS: ${{ secrets.LAI_PASS }}
-          LIGHTNING_USER_ID: ${{ secrets.LIGHTNING_USER_ID }}
-          LIGHTNING_API_KEY: ${{ secrets.LIGHTNING_API_KEY }}
-          LIGHTNING_USERNAME: ${{ secrets.LIGHTNING_USERNAME }}
-          LIGHTNING_CLOUD_URL: ${{ secrets.LIGHTNING_CLOUD_URL }}
-          PR_NUMBER: ${{ steps.PR.outputs.ID }}
-          TEST_APP_NAME: ${{ matrix.app_name }}
-          GRID_USER_ID: ${{ secrets.LIGHTNING_USER_ID }}
-          GRID_USER_KEY: ${{ secrets.LIGHTNING_API_KEY }}
-          GRID_URL: ${{ secrets.LIGHTNING_CLOUD_URL }}
-          _GRID_USERNAME: ${{ secrets.LIGHTNING_USERNAME }}
-        shell: bash
-        run: |
-          time python -c "from lightning.app import testing; testing.delete_cloud_lightning_apps()"


Code review comments:
diff_hunk: @@ -0,0 +1,127 @@
+# Python package
+# Create and test a Python package on multiple Python versions.
+# Add steps that analyze code, save the dist with the build record, publish to a PyPI-compatible index, and more:
+# https://docs.microsoft.com/azure/devops/pipelines/languages/python
+
+trigger:
+  tags:
+    include:
+      - '*'
+  branches:
+    include:
+      - "master"
+      - "release/*"
+      - "refs/tags/*"
+
+pr:
+  - "master"
+  - "release/*"
+
+variables:
+  # variables are automatically exported as environment variables so this will override pip's default cache dir
+  - name: pip_cache_dir
+    value: $(Pipeline.Workspace)/.pip
+
+jobs:
+  - job: App_cloud_e2e_testing
+    pool:
+      vmImage: 'ubuntu-latest'
+    timeoutInMinutes: "30"
+    cancelTimeoutInMinutes: "2"
+    strategy:
+      matrix:
+        'App: v0_app':
+          repo: "v0_app"
+        'App: boring_app':
+          repo: "boring_app"
+        'App: template_streamlit_ui':
+          repo: "template_streamlit_ui"
+        'App: template_react_ui':
+          repo: "template_react_ui"
+        'App: template_jupyterlab':
+          repo: "template_jupyterlab"
+        'App: idle_timeout':
+          repo: "idle_timeout"
+        'App: collect_failures':
+          repo: "collect_failures"
+        'App: custom_work_dependencies':
+          repo: "custom_work_dependencies"
+        'App: drive':
+          repo: "drive"
+        'App: payload':
+          repo: "payload"
+        'App: commands_and_api':
+          repo: "commands_and_api"
+    workspace:
+      clean: all
+    steps:
+    - task: UsePythonVersion@0
+      inputs:
+        versi`onSpec: '3.8'
+
+    # TODO: parse the PR number
+
+    - task: Cache@2
+      inputs:
+        key: 'pip | "$(repo)" | requirements/app/base.txt'
+        restoreKeys: |
+          pip | "$(Agent.OS)"
+        path: $(pip_cache_dir)
+      displayName: Cache pip
+
+    - bash: |
+        python --version
+        pip --version
+      displayName: 'Info'
+
+    - bash: python -m pip install -r requirements/app/devel.txt --no-cache --quiet --find-links https://download.pytorch.org/whl/cpu/torch_stable.html
+      displayName: 'Install dependencies'
+
+    - bash: |
+        python -m pip install playwright
+        python -m playwright install --with-deps
+      displayName: 'Install Playwright system dependencies'
+
+    - bash: pip install -e .
+      displayName: 'Install lightning'
+
+#    - checkout: git://MyProject/MyToolsRepo@features/mytools  # TODO: Lightning-AI/lightning-template-react
+#      path: examples/app_template_react_ui
+#      displayName: 'Clone Template React UI Repo'
+
+#    - checkout: git://MyProject/MyToolsRepo@features/mytools  # TODO: Lightning-AI/lightning-template-jupyterlab
+#      path: examples/app_template_jupyterlab
+#      displayName: 'Clone Template Jupyter Lab Repo'
+
+#    - bash: cp examples/app_template_jupyterlab/tests/test_template_jupyterlab.py tests/tests_app_examples/test_template_jupyterlab.py
+#      displayName: 'Copy Template Jupyter Lab Repo tests'
+
+    - bash: |
+        mkdir -p ${VIDEO_LOCATION}
+        ls -l tests/tests_app_examples
+        python -m pytest tests/tests_app_examples/test_${TEST_APP_NAME}.py::test_${TEST_APP_NAME}_example_cloud --timeout=900 --capture=no -v --color=yes
+      env:
+        HEADLESS: "1"
+        PACKAGE_LIGHTNING: "1"
+        CLOUD: "1"
+        VIDEO_LOCATION: ./artifacts/videos
+        PR_NUMBER: $(date +%s)  # TODO: use PR number
+        TEST_APP_NAME: $(repo)
+        HAR_LOCATION: ./artifacts/hars
+        SLOW_MO: 50
+        LAI_USER: $(LAI_USER)
+        LAI_PASS: $(LAI_PASS)
+        LIGHTNING_USER_ID: $(LIGHTNING_USER_ID)
+        LIGHTNING_API_KEY: $(LIGHTNING_API_KEY)
+        LIGHTNING_USERNAME: $(LIGHTNING_USERNAME)
+        LIGHTNING_CLOUD_URL: $(LIGHTNING_CLOUD_URL)
+      workingDirectory: tests/tests_pytorch
path: .azure/app-cloud-e2e.yml
text: @user1:
This doesn't look right

@author:
it was bug, fixed :)