Repository: pandas-dev/pandas-stubs (language: python)

Issue:
Create a way to test nightly locally, or as part of a PR that fixes nightly
Right now, if we get a PR that fixes an issue for the nightly tests, the only way to test it is to approve the PR and merge it, and see what happens as a result of the merge.

We should figure out a way that we can run the nightly tests on demand, either locally, or by doing something in the PR that signals that the nightly tests should be run as part of the PR.

Maybe @twoertwein has an idea here.



Discussion / hints:
@twoertwein I looked into it and why you were having troubles creating that.
Here is what happened:
- The workflows you see in the Actions tabs are the ones that gets triggered when you push to the branch.
- When you use a comment, you actually use a different trigger (the `on` section of the yaml workflow).
- So even if you commented and it actually triggered the workflow you were hoping for, it would not display in the `Checks` tab of the PR.

Here is a workaround that we could use (I actually tried it out, PR on my own repo is below since you need to have the workflow file on main branch to have it applied in a PR):
- You create a comment command like @twoertwein you did on your PR.
- The Action will get triggered and run.
- Once the Action is complete you actually create a check run (https://docs.github.com/en/rest/checks/runs?apiVersion=2022-11-28) where you will list if your test ran without error or not.
- From there, this will be displayed a little differently than a typical Actions but you will still get most of the information:
<img width="800" alt="image" src="https://github.com/user-attachments/assets/649b168f-478f-440a-bb20-b5be97d2fcf4">
- We can decide what we want to add in the messages there.

Of course the content of the message will be a little different than if we were using a more standard Actions that is triggered on push but that is a good workaround not to have to run the nightly checks on merge to main branch (mostly to save CI credits and have the status check appearing in green).
PR I used is there https://github.com/loicdiridollou/pandas-stubs/pull/2, feel free to trigger the workflow with `/nightly`, I have not implemented the other commands yet but wanted to get a good proof-of-concept to share first and discuss if this is a route that you would like to take.
The code for the workflow is there https://github.com/loicdiridollou/pandas-stubs/blob/main/.github/workflows/comment_commands.yml it is a little tricky as I did not comment anything but we can discuss what is going on there if you need (only focus on the `nightly` job since I did not implement the other ones yet).

Please let me know what you think and I can spend a little bit more time working on improvements/changes.
Great, thank you @loicdiridollou for figuring it out!

Feel free to open a PR!



Fix patch:
diff --git a/.github/workflows/comment_commands.yml b/.github/workflows/comment_commands.yml
new file mode 100644
index 00000000..c85cdca8
--- /dev/null
+++ b/.github/workflows/comment_commands.yml
@@ -0,0 +1,71 @@
+name: Comment Commands to Trigger CI
+on:
+  issue_comment:
+    types: created
+
+permissions:
+  checks: write
+
+env:
+  # store mapping of commands to use with poetry
+  RUN_COMMAND: '{"/pandas_nightly": "pytest --nightly", "/pyright_strict": "pyright_strict", "/mypy_nightly": "mypy --mypy_nightly"}'
+  # store mapping of labels to display in the check runs
+  DISPLAY_COMMAND: '{"/pandas_nightly": "Pandas nightly tests", "/pyright_strict": "Pyright strict tests", "/mypy_nightly": "Mypy nightly tests"}'
+
+jobs:
+  optional_tests:
+    name: "Optional tests run"
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    # if more commands are added, they will need to be added here too as we don't have access to env at this stage
+    if: (github.event.issue.pull_request) && contains(fromJSON('["/pandas_nightly", "/pyright_strict", "/mypy_nightly"]'), github.event.comment.body)
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install project dependencies
+        uses: ./.github/setup
+        with:
+          os: ubuntu-latest
+          python-version: "3.11"
+
+      - name: Run ${{ fromJSON(env.DISPLAY_COMMAND)[github.event.comment.body] }}
+        # run the tests based on the value of the comment
+        id: tests-step
+        run: poetry run poe ${{ fromJSON(env.RUN_COMMAND)[github.event.comment.body] }}
+
+      - name: Get head sha and store value
+        # get the sha of the last commit to attach the results of the tests
+        if: always()
+        id: get-sha
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          script: |
+            const pr = await github.rest.pulls.get({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              pull_number: ${{ github.event.issue.number }}
+            })
+            core.setOutput('sha', pr.data.head.sha)
+
+      - name: Report results of the tests and publish
+        # publish the results to a check run no matter the pass or fail
+        if: always()
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          script: |
+            github.rest.checks.create({
+              name: '${{ fromJSON(env.DISPLAY_COMMAND)[github.event.comment.body] }}',
+              head_sha: '${{ steps.get-sha.outputs.sha }}',
+              status: 'completed',
+              conclusion: '${{ steps.tests-step.outcome }}',
+              output: {
+                title: 'Run ${{ fromJSON(env.DISPLAY_COMMAND)[github.event.comment.body] }}',
+                summary: 'Results: ${{ steps.tests-step.outcome }}',
+                text: 'See the actions run at ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}',
+              },
+              owner: context.repo.owner,
+              repo: context.repo.repo
+            })


Test patch:
diff --git a/docs/tests.md b/docs/tests.md
index fdf2dd5e..4101bc83 100644
--- a/docs/tests.md
+++ b/docs/tests.md
@@ -5,7 +5,7 @@
 Here are the most important options. Fore more details, please use `poe --help`.
 
 - Run all tests (against both source and installed stubs): `poe test_all`
-- Run tests against the source code: `poe test` 
+- Run tests against the source code: `poe test`
   - Run only mypy: `poe mypy`
   - Run only pyright: `poe pyright`
   - Run only pytest: `poe pytest`
@@ -20,3 +20,9 @@ The following tests are **optional**. Some of them are run by the CI but it is o
 - Use mypy nightly to validate the annotations: `poe mypy --mypy_nightly`
 - Use pyright in full strict mode: `poe pyright_strict`
 - Run stubtest to compare the installed pandas-stubs against pandas (this will fail): `poe stubtest`. If you have created an allowlist to ignore certain errors: `poe stubtest path_to_the_allow_list`
+
+Among the tests above, the following can be run directly during a PR by commenting in the discussion.
+
+- Run pytest against pandas nightly by commenting `/pandas_nightly`
+- Use mypy nightly to validate the annotations by commenting `/mypy_nightly`
+- Use pyright in full strict mode by commenting `/pyright_strict`