Repository: Lightning-AI/pytorch-lightning
Pull request: Replace usage of `grep -P` with `perl` in `run_standalone_tests.sh`
## What does this PR do?
Fixes #19938, a "bug" where on macOS, which does not use GNU `grep` by default and does not recognize the `-P` flag, `run_standalone_tests.sh` errors out (earlier than expected).

<!--
Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context.
List any dependencies that are required for this change.

If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

The following links the related issue to the PR (https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)
-->

<!-- Does your PR introduce any breaking changes? If yes, please list them. -->

<details>
  <summary><b>Before submitting</b></summary>

- [x] Was this **discussed/agreed** via a GitHub issue? (not for typos and docs)
- [x] Did you read the [contributor guideline](https://github.com/Lightning-AI/lightning/blob/master/.github/CONTRIBUTING.md), **Pull Request** section?
- [x] Did you make sure your **PR does only one thing**, instead of bundling different changes together?
- Did you make sure to **update the documentation** with your changes? (if necessary)
- Did you write any **new necessary tests**? (not for typos and docs)
- [x] Did you verify new and **existing tests pass** locally with your changes?
- Did you list all the **breaking changes** introduced by this pull request?
- Did you **update the CHANGELOG**? (not for typos, docs, test updates, or minor internal changes/refactors)

<!-- In the CHANGELOG, separate each item in the unreleased section by a blank line to reduce collisions -->

</details>

## PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the [review guidelines](https://github.com/Lightning-AI/lightning/wiki/Review-guidelines). In short, see the following bullet-list:

<details>
  <summary>Reviewer checklist</summary>

- [ ] Is this pull request ready for review? (if not, please submit in draft mode)
- [ ] Check that all items from **Before submitting** are resolved
- [ ] Make sure the title is self-explanatory and the description concisely explains the PR
- [ ] Add labels and milestones (and optionally projects) to the PR so it can be classified

</details>

<!--

Did you have fun?

Make sure you had fun coding 🙃

-->


<!-- readthedocs-preview pytorch-lightning start -->
----
📚 Documentation preview 📚: https://pytorch-lightning--19942.org.readthedocs.build/en/19942/

<!-- readthedocs-preview pytorch-lightning end -->

Problem statement:
`grep: Invalid option -- P` when running `./tests/run_standalone_tests.sh` on macOS
### Bug description

BSD `grep` no longer supports the `-- P` option on macOS.

One fix would be to use GNU `grep` instead, as `ggrep`.

### What version are you seeing the problem on?

master

### How to reproduce the bug

_No response_

### Error messages and logs

```
# Error messages and logs here please
```


### Environment

<details>
  <summary>Current environment</summary>

```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```

</details>


### More info

_No response_

cc @borda

Hints from issue discussion:
Hey @Peiffap 
You are welcome to send a PR for this if you'd like. 
We don't have any tests in `run_standalone_tests.sh` that would run on macos, but for completeness, this fix would be good IMO.

Commit message: Fix CI trigger

Patch to review:
diff --git a/.azure/gpu-tests-pytorch.yml b/.azure/gpu-tests-pytorch.yml
index 156513d604210..dd1a3d4abcf6a 100644
--- a/.azure/gpu-tests-pytorch.yml
+++ b/.azure/gpu-tests-pytorch.yml
@@ -31,6 +31,7 @@ pr:
       - "src/lightning/pytorch/**"
       - "src/pytorch_lightning/*"
       - "tests/tests_pytorch/**"
+      - "tests/run_standalone_*.sh"
       - "pyproject.toml" # includes pytest config
       - "requirements/fabric/**"
       - "src/lightning/fabric/**"
diff --git a/.github/checkgroup.yml b/.github/checkgroup.yml
index 0c5e7e4579ccb..c4c88b12a2598 100644
--- a/.github/checkgroup.yml
+++ b/.github/checkgroup.yml
@@ -60,6 +60,7 @@ subprojects:
       - "src/lightning/pytorch/**"
       - "src/pytorch_lightning/*"
       - "tests/tests_pytorch/**"
+      - "tests/run_standalone_*.sh"
       - "pyproject.toml" # includes pytest config
       - "requirements/fabric/**"
       - "src/lightning/fabric/**"
@@ -201,7 +202,6 @@ subprojects:
       - ".azure/gpu-tests-fabric.yml"
       - "examples/fabric/**"
       - "examples/run_fabric_examples.sh"
-      - "tests/run_standalone_*.sh"
       - "requirements/fabric/**"
       - "src/lightning/__init__.py"
       - "src/lightning/__setup__.py"
@@ -209,6 +209,7 @@ subprojects:
       - "src/lightning/fabric/**"
       - "src/lightning_fabric/*"
       - "tests/tests_fabric/**"
+      - "tests/run_standalone_*.sh"
       - "pyproject.toml" # includes pytest config
       - "!requirements/*/docs.txt"
       - "!*.md"
diff --git a/tests/run_standalone_tests.sh b/tests/run_standalone_tests.sh
index 0de781b0c47c7..075f602dac1ae 100755
--- a/tests/run_standalone_tests.sh
+++ b/tests/run_standalone_tests.sh
@@ -26,18 +26,17 @@ export PL_RUN_STANDALONE_TESTS=1
 defaults=" -m coverage run --source ${source} --append -m pytest --no-header -v -s --timeout 120 "
 echo "Using defaults: ${defaults}"
 
-# get the testing location as the fist argument
+# get the testing location as the first argument
 test_path=$1
 printf "source path: $test_path\n"
 
 # collect all tests with parametrization based filtering with PL_RUN_STANDALONE_TESTS
 standalone_tests=$(python3 -m pytest $test_path -q --collect-only --pythonwarnings ignore)
-printf "Collected tests: \n $standalone_tests"
+printf "Collected tests: \n $standalone_tests\n"
 # match only lines with tests
-parametrizations=$(grep -oP '\S+::test_\S+' <<< "$standalone_tests")
+parametrizations=$(perl -nle'print $& while m{\S+::test_\S+}g' <<< "$standalone_tests")
 # convert the list to be array
 parametrizations_arr=($parametrizations)
-
 report=''
 
 rm -f standalone_test_output.txt  # in case it exists, remove it
@@ -47,7 +46,7 @@ function show_batched_output {
   if [ -f standalone_test_output.txt ]; then  # if exists
     cat standalone_test_output.txt
     # heuristic: stop if there's mentions of errors. this can prevent false negatives when only some of the ranks fail
-    if grep -iE 'error|exception|traceback|failed' standalone_test_output.txt | grep -vE 'on_exception|xfailed' | grep -qv -f testnames.txt; then
+    if perl -nle'print if /error|(?<!(?-i)on_)exception|traceback|(?<!(?-i)x)failed/i' standalone_test_output.txt | grep -qv -f testnames.txt; then
       echo "Potential error! Stopping."
       rm standalone_test_output.txt
       exit 1


Code review comments:
diff_hunk: @@ -47,7 +46,7 @@ function show_batched_output {
   if [ -f standalone_test_output.txt ]; then  # if exists
     cat standalone_test_output.txt
     # heuristic: stop if there's mentions of errors. this can prevent false negatives when only some of the ranks fail
-    if grep -iE 'error|exception|traceback|failed' standalone_test_output.txt | grep -vE 'on_exception|xfailed' | grep -qv -f testnames.txt; then
+    if perl -nle'print if /error|(?<!(?-i)on_)exception|traceback|(?<!(?-i)x)failed/i' standalone_test_output.txt | grep -qv -f testnames.txt; then
path: tests/run_standalone_tests.sh
text: @author:
```suggestion
    if perl -nle 'print if /error|(?<!(?-i)on_)exception|traceback|(?<!(?-i)x)failed/i' standalone_test_output.txt | grep -qv -f testnames.txt; then
```

diff_hunk: @@ -26,18 +26,17 @@ export PL_RUN_STANDALONE_TESTS=1
 defaults=" -m coverage run --source ${source} --append -m pytest --no-header -v -s --timeout 120 "
 echo "Using defaults: ${defaults}"
 
-# get the testing location as the fist argument
+# get the testing location as the first argument
 test_path=$1
 printf "source path: $test_path\n"
 
 # collect all tests with parametrization based filtering with PL_RUN_STANDALONE_TESTS
 standalone_tests=$(python3 -m pytest $test_path -q --collect-only --pythonwarnings ignore)
-printf "Collected tests: \n $standalone_tests"
+printf "Collected tests: \n $standalone_tests\n"
 # match only lines with tests
-parametrizations=$(grep -oP '\S+::test_\S+' <<< "$standalone_tests")
+parametrizations=$(perl -nle'print $& while m{\S+::test_\S+}g' <<< "$standalone_tests")
path: tests/run_standalone_tests.sh
text: @user1:
This looks odd, shouldn't there be a space here between argument and value?
```suggestion
parametrizations=$(perl -nle 'print $& while m{\S+::test_\S+}g' <<< "$standalone_tests")
```

@author:
Good catch, thanks. I did the same below for the other call.