Repository: NVIDIA/spark-rapids
Pull request #12394: Add a way to override the PLUGIN_JARS [databricks]
Author: login: razajafri
Labels: name: test
description: Only impacts tests
Created: 2025-03-26T17:37:45Z | Merged: 2025-04-26T01:44:21Z

This PR adds a way to override the plugin jar when running integration tests. 

By setting `PLUGIN_JAR` env variable you can override the default which would use the plugin jar from the dist module. 

fixes #12393 

<!--

Thank you for contributing to RAPIDS Accelerator for Apache Spark!

Here are some guidelines to help the review process go smoothly.

1. Please write a description in this text box of the changes that are being
   made.

2. Please ensure that you have written units tests for the changes made/features
   added.

3. If you are closing an issue please use one of the automatic closing words as
   noted here: https://help.github.com/articles/closing-issues-using-keywords/

4. If your pull request is not ready for review but you want to make use of the
   continuous integration testing facilities please label it with `[WIP]`.

5. If your pull request is ready to be reviewed without requiring additional
   work on top of it, then remove the `[WIP]` label (if present).

6. Once all work has been done and review has taken place please do not add
   features or make changes out of the scope of those requested by the reviewer
   (doing this just add delays as already reviewed code ends up having to be
   re-reviewed/it is hard to tell what is new etc!). Further, please avoid
   rebasing your branch during the review process, as this causes the context
   of any comments made by reviewers to be lost. If conflicts occur during
   review then they should be resolved by merging into the branch used for
   making the pull request.

Many thanks in advance for your cooperation!

-->


Linked issue #12393: [FEA] Provide a way to override the plugin jar path only when running integration tests
While testing old nightly builds on Databricks, I want to be able to test a plugin jar without replacing any of the other dependencies e.g. `spark-avro*.jar` 

There is a `LOCAL_JAR_PATH` env variable that can be set but that needs to be a directory which assumes to have all the other dependencies as well. I couldn't find a way only to replace the plugin jar. 

 



Diff:
diff --git a/integration_tests/README.md b/integration_tests/README.md
index accd3d7c6d0..49cb5920f95 100644
--- a/integration_tests/README.md
+++ b/integration_tests/README.md
@@ -330,8 +330,9 @@ under YARN and even though it is off by default you probably want to be sure it
 
 In case your test jars and resources are downloaded to the `local-path` from dependency Repo, and you want to run tests with them
 using the shell-script [run_pyspark_from_build.sh](run_pyspark_from_build.sh), then the `LOCAL_JAR_PATH=local-path` must be set to point
-to the `local-path`, e.g. `LOCAL_JAR_PATH=local-path bash [run_pyspark_from_build.sh](run_pyspark_from_build.sh)`.By setting `LOCAL_JAR_PATH=local-path`
-the shell-script [run_pyspark_from_build.sh](run_pyspark_from_build.sh) can find the test jars and resources in the alternate path.
+to the `local-path`, e.g. `LOCAL_JAR_PATH=local-path bash [run_pyspark_from_build.sh](run_pyspark_from_build.sh)`. By setting `LOCAL_JAR_PATH=local-path`
+the shell-script [run_pyspark_from_build.sh](run_pyspark_from_build.sh) can find the test jars and resources in the alternate path. If you only want to override the 
+plugin jar then you can just set `PLUGIN_JAR=<path to the jar>`.
 
 When running the shell-script [run_pyspark_from_build.sh](run_pyspark_from_build.sh) under YARN or Kubernetes, the `$SCRIPTPATH` in the python options
 `--rootdir $SCRIPTPATH ...` and `--std_input_path $SCRIPTPATH ...` will not work, as the `$SCRIPTPATH` is a local path, you need to overwrite it to the cloud paths.
diff --git a/integration_tests/run_pyspark_from_build.sh b/integration_tests/run_pyspark_from_build.sh
index 8943a366b66..95d3af123e3 100755
--- a/integration_tests/run_pyspark_from_build.sh
+++ b/integration_tests/run_pyspark_from_build.sh
@@ -12,6 +12,50 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+# Documentation generated using a generative AI tool
+# ============================================================================
+# run_pyspark_from_build.sh
+# ============================================================================
+#
+# Description:
+#   This script runs PySpark integration tests from a build environment.
+#
+# Usage:
+#   To run this script, ensure you have Apache Spark installed and configured
+#   properly, along with the necessary dependencies (e.g., findspark, xdist).
+#
+# Parameters and Environment Variables:
+#   - SPARK_HOME: Path to your Apache Spark installation.
+#   - SKIP_TESTS: If set to true, skips running the Python integration tests.
+#   - INCLUDE_SPARK_AVRO_JAR: If set to true, includes Avro tests.
+#   - TEST: Specifies a specific test to run.
+#   - TEST_TAGS: Allows filtering tests based on tags.
+#   - TEST_TYPE: Specifies the type of tests to run.
+#   - LOCAL_JAR_PATH: Path to local jars if not building from source.
+#   - PLUGIN_JAR: Path to a built spark-rapids plugin jar, the default points to the target directory
+#   - INTEGRATION_TEST_VERSION_OVERRIDE: Overrides the auto-detected shim version.
+#
+# Script Flow:
+#   1. Setup and Checks: Validates environment and detects Spark/Scala versions.
+#   2. Jars and Dependencies: Identifies necessary jars based on Spark version.
+#   3. Test Configuration: Determines parallelism and checks for necessary tools.
+#   4. Running Tests: Executes tests using runtests.py with specified options.
+#
+# Example Usage:
+#   To run all tests, including Avro tests:
+#     INCLUDE_SPARK_AVRO_JAR=true ./run_pyspark_from_build.sh
+#
+#   To run a specific test:
+#     TEST=my_test ./run_pyspark_from_build.sh
+#
+# Troubleshooting:
+#   - Ensure SPARK_HOME is correctly set.
+#   - Check for missing dependencies like findspark or xdist.
+#   - Verify that the necessary jars are correctly linked.
+#
+# ============================================================================
+
 set -ex
 
 SCRIPTPATH="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
@@ -56,7 +100,7 @@ else
     # support alternate local jars NOT building from the source code
     if [ -d "$LOCAL_JAR_PATH" ]; then
         AVRO_JARS=$(echo "$LOCAL_JAR_PATH"/spark-avro*.jar)
-        PLUGIN_JARS=$(echo "$LOCAL_JAR_PATH"/rapids-4-spark_*.jar)
+        PLUGIN_JAR=$(echo "$LOCAL_JAR_PATH"/rapids-4-spark_*.jar)
         if [ -f $(echo $LOCAL_JAR_PATH/parquet-hadoop*.jar) ]; then
             export INCLUDE_PARQUET_HADOOP_TEST_JAR=true
             PARQUET_HADOOP_TESTS=$(echo $LOCAL_JAR_PATH/parquet-hadoop*.jar)
@@ -80,7 +124,7 @@ else
         # Make sure we have Parquet version >= 1.12 in the dependency
         LOWEST_PARQUET_JAR=$(echo -e "$MIN_PARQUET_JAR\n$PARQUET_HADOOP_TESTS" | sort -V | head -1)
         export INCLUDE_PARQUET_HADOOP_TEST_JAR=$([[ "$LOWEST_PARQUET_JAR" == "$MIN_PARQUET_JAR" ]] && echo true || echo false)
-        PLUGIN_JARS=$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)
+        PLUGIN_JAR=${PLUGIN_JAR:-$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)}
         # the integration-test-spark3xx.jar, should not include the integration-test-spark3xxtest.jar
         TEST_JARS=$(echo "$TARGET_DIR"/rapids-4-spark-integration-tests*-$INTEGRATION_TEST_VERSION.jar)
     fi
@@ -99,7 +143,7 @@ else
 
     # ALL_JARS includes dist.jar integration-test.jar avro.jar parquet.jar if they exist
     # Remove non-existing paths and canonicalize the paths including get rid of links and `..`
-    ALL_JARS=$(readlink -e $PLUGIN_JARS $TEST_JARS $AVRO_JARS $PARQUET_HADOOP_TESTS || true)
+    ALL_JARS=$(readlink -e $PLUGIN_JAR $TEST_JARS $AVRO_JARS $PARQUET_HADOOP_TESTS || true)
     # `:` separated jars
     ALL_JARS="${ALL_JARS//$'\n'/:}"
 


Reviews:
user: login: tgravescs
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T17:43:02Z
user: login: razajafri
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T20:22:07Z
user: login: tgravescs
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T20:25:44Z
user: login: razajafri
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T21:10:08Z
user: login: tgravescs
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-03-31T12:58:42Z
user: login: gerashegalov
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-09T06:05:26Z
user: login: razajafri
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-09T21:31:00Z
user: login: gerashegalov
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-10T00:53:33Z
user: login: gerashegalov
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-10T04:51:43Z
user: login: razajafri
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-17T18:44:21Z
user: login: razajafri
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-17T18:48:46Z
user: login: gerashegalov
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-17T23:10:46Z
user: login: razajafri
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-23T19:08:43Z
user: login: gerashegalov
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-24T21:53:12Z

Review comments:
diff_hunk: @@ -80,7 +80,11 @@ else
         # Make sure we have Parquet version >= 1.12 in the dependency
         LOWEST_PARQUET_JAR=$(echo -e "$MIN_PARQUET_JAR\n$PARQUET_HADOOP_TESTS" | sort -V | head -1)
         export INCLUDE_PARQUET_HADOOP_TEST_JAR=$([[ "$LOWEST_PARQUET_JAR" == "$MIN_PARQUET_JAR" ]] && echo true || echo false)
-        PLUGIN_JARS=$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)
+        if [ -e "$PLUGIN_JAR_PATH" ]; then
path: integration_tests/run_pyspark_from_build.sh
user: login: tgravescs
body: why not just use PLUGIN_JARS and check if its set first? Though we can probably rename to PLUGIN_JAR at this point.

Also it would be nice to document, in the very least at the top of the file (which i realize is lacking any docs but good to make it better)
created_at: 2025-03-26T17:43:02Z
author_association: COLLABORATOR
diff_hunk: @@ -80,7 +80,11 @@ else
         # Make sure we have Parquet version >= 1.12 in the dependency
         LOWEST_PARQUET_JAR=$(echo -e "$MIN_PARQUET_JAR\n$PARQUET_HADOOP_TESTS" | sort -V | head -1)
         export INCLUDE_PARQUET_HADOOP_TEST_JAR=$([[ "$LOWEST_PARQUET_JAR" == "$MIN_PARQUET_JAR" ]] && echo true || echo false)
-        PLUGIN_JARS=$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)
+        if [ -e "$PLUGIN_JAR_PATH" ]; then
path: integration_tests/run_pyspark_from_build.sh
user: login: razajafri
body: Fixed. PTAL
created_at: 2025-03-26T20:22:07Z
author_association: COLLABORATOR
diff_hunk: @@ -12,6 +12,54 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+# Documentation generated using a generative AI tool
+# ============================================================================
+# run_pyspark_from_build.sh
+# ============================================================================
+#
+# Description:
+#   This script runs PySpark integration tests from a build environment.
+#
+# Usage:
+#   To run this script, ensure you have Apache Spark installed and configured
+#   properly, along with the necessary dependencies (e.g., findspark, xdist).
+#
+# Parameters and Environment Variables:
+#   - SPARK_HOME: Path to your Apache Spark installation.
+#   - SKIP_TESTS: If set to true, skips running the Python integration tests.
+#   - INCLUDE_SPARK_AVRO_JAR: If set to true, includes Avro tests.
+#   - TEST: Specifies a specific test to run.
+#   - TEST_TAGS: Allows filtering tests based on tags.
+#   - TEST_TYPE: Specifies the type of tests to run.
+#   - LOCAL_JAR_PATH: Path to local jars if not building from source.
+#   - PLUGIN_JAR: Path to a built spark-rapids plugin jar if available
path: integration_tests/run_pyspark_from_build.sh
user: login: tgravescs
body: you might add the default is to the target directory one
created_at: 2025-03-26T20:25:44Z
author_association: COLLABORATOR
diff_hunk: @@ -12,6 +12,54 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+# Documentation generated using a generative AI tool
+# ============================================================================
+# run_pyspark_from_build.sh
+# ============================================================================
+#
+# Description:
+#   This script runs PySpark integration tests from a build environment.
+#
+# Usage:
+#   To run this script, ensure you have Apache Spark installed and configured
+#   properly, along with the necessary dependencies (e.g., findspark, xdist).
+#
+# Parameters and Environment Variables:
+#   - SPARK_HOME: Path to your Apache Spark installation.
+#   - SKIP_TESTS: If set to true, skips running the Python integration tests.
+#   - INCLUDE_SPARK_AVRO_JAR: If set to true, includes Avro tests.
+#   - TEST: Specifies a specific test to run.
+#   - TEST_TAGS: Allows filtering tests based on tags.
+#   - TEST_TYPE: Specifies the type of tests to run.
+#   - LOCAL_JAR_PATH: Path to local jars if not building from source.
+#   - PLUGIN_JAR: Path to a built spark-rapids plugin jar if available
path: integration_tests/run_pyspark_from_build.sh
user: login: razajafri
body: Done
created_at: 2025-03-26T21:10:08Z
author_association: COLLABORATOR
diff_hunk: @@ -80,7 +128,7 @@ else
         # Make sure we have Parquet version >= 1.12 in the dependency
         LOWEST_PARQUET_JAR=$(echo -e "$MIN_PARQUET_JAR\n$PARQUET_HADOOP_TESTS" | sort -V | head -1)
         export INCLUDE_PARQUET_HADOOP_TEST_JAR=$([[ "$LOWEST_PARQUET_JAR" == "$MIN_PARQUET_JAR" ]] && echo true || echo false)
-        PLUGIN_JARS=$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)
+        PLUGIN_JAR=${PLUGIN_JAR:-$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)}
path: integration_tests/run_pyspark_from_build.sh
user: login: gerashegalov
body: This is actually part of the bigger classpath mess we have. Our code should check that if we use spark.jars or spark.packages to specify the plugin artifact it should not include the one from TARGET_DIR
created_at: 2025-04-09T06:05:26Z
author_association: COLLABORATOR
diff_hunk: @@ -80,7 +128,7 @@ else
         # Make sure we have Parquet version >= 1.12 in the dependency
         LOWEST_PARQUET_JAR=$(echo -e "$MIN_PARQUET_JAR\n$PARQUET_HADOOP_TESTS" | sort -V | head -1)
         export INCLUDE_PARQUET_HADOOP_TEST_JAR=$([[ "$LOWEST_PARQUET_JAR" == "$MIN_PARQUET_JAR" ]] && echo true || echo false)
-        PLUGIN_JARS=$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)
+        PLUGIN_JAR=${PLUGIN_JAR:-$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)}
path: integration_tests/run_pyspark_from_build.sh
user: login: razajafri
body: Is there an issue created to overhaul this "classpath mess"? I'd rather not try to handle that mess in this PR. 
created_at: 2025-04-09T21:31:00Z
author_association: COLLABORATOR
diff_hunk: @@ -80,7 +128,7 @@ else
         # Make sure we have Parquet version >= 1.12 in the dependency
         LOWEST_PARQUET_JAR=$(echo -e "$MIN_PARQUET_JAR\n$PARQUET_HADOOP_TESTS" | sort -V | head -1)
         export INCLUDE_PARQUET_HADOOP_TEST_JAR=$([[ "$LOWEST_PARQUET_JAR" == "$MIN_PARQUET_JAR" ]] && echo true || echo false)
-        PLUGIN_JARS=$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)
+        PLUGIN_JAR=${PLUGIN_JAR:-$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)}
path: integration_tests/run_pyspark_from_build.sh
user: login: gerashegalov
body: 
We can defer the class path cleanup to a dedicated issue. But the previous ask to update the "Running with Alternate Paths" section in the integration tests README.md is essential for this PR 

created_at: 2025-04-10T00:53:33Z
author_association: COLLABORATOR
diff_hunk: @@ -12,6 +12,54 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+# Documentation generated using a generative AI tool
+# ============================================================================
+# run_pyspark_from_build.sh
+# ============================================================================
+#
+# Description:
+#   This script runs PySpark integration tests from a build environment.
+#
+# Usage:
+#   To run this script, ensure you have Apache Spark installed and configured
+#   properly, along with the necessary dependencies (e.g., findspark, xdist).
+#
+# Parameters and Environment Variables:
+#   - SPARK_HOME: Path to your Apache Spark installation.
+#   - SKIP_TESTS: If set to true, skips running the Python integration tests.
+#   - INCLUDE_SPARK_AVRO_JAR: If set to true, includes Avro tests.
+#   - TEST: Specifies a specific test to run.
+#   - TEST_TAGS: Allows filtering tests based on tags.
+#   - TEST_TYPE: Specifies the type of tests to run.
+#   - LOCAL_JAR_PATH: Path to local jars if not building from source.
+#   - PLUGIN_JAR: Path to a built spark-rapids plugin jar, the default points to the target directory
+#   - INTEGRATION_TEST_VERSION_OVERRIDE: Overrides the auto-detected shim version.
+#
+# Script Flow:
+#   1. Setup and Checks: Validates environment and detects Spark/Scala versions.
+#   2. Jars and Dependencies: Identifies necessary jars based on Spark version.
+#   3. Test Configuration: Determines parallelism and checks for necessary tools.
+#   4. Running Tests: Executes tests using runtests.py with specified options.
+#
+# Example Usage:
+#   To run all tests, including Avro tests:
+#     INCLUDE_SPARK_AVRO_JAR=true ./run_pyspark_from_build.sh
+#
+#   To run a specific test:
+#     TEST=my_test ./run_pyspark_from_build.sh
+#
+# Troubleshooting:
+#   - Ensure SPARK_HOME is correctly set.
+#   - Check for missing dependencies like findspark or xdist.
+#   - Verify that the necessary jars are correctly linked.
+#
+# Contributing:
+#   Contributions should follow the NVIDIA Spark-Rapids project guidelines.
+#   Ensure changes are tested thoroughly to maintain compatibility.
path: integration_tests/run_pyspark_from_build.sh
user: login: gerashegalov
body: Are we going to repeat this for every file? 
created_at: 2025-04-10T04:51:43Z
author_association: COLLABORATOR
diff_hunk: @@ -12,6 +12,54 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+# Documentation generated using a generative AI tool
+# ============================================================================
+# run_pyspark_from_build.sh
+# ============================================================================
+#
+# Description:
+#   This script runs PySpark integration tests from a build environment.
+#
+# Usage:
+#   To run this script, ensure you have Apache Spark installed and configured
+#   properly, along with the necessary dependencies (e.g., findspark, xdist).
+#
+# Parameters and Environment Variables:
+#   - SPARK_HOME: Path to your Apache Spark installation.
+#   - SKIP_TESTS: If set to true, skips running the Python integration tests.
+#   - INCLUDE_SPARK_AVRO_JAR: If set to true, includes Avro tests.
+#   - TEST: Specifies a specific test to run.
+#   - TEST_TAGS: Allows filtering tests based on tags.
+#   - TEST_TYPE: Specifies the type of tests to run.
+#   - LOCAL_JAR_PATH: Path to local jars if not building from source.
+#   - PLUGIN_JAR: Path to a built spark-rapids plugin jar, the default points to the target directory
+#   - INTEGRATION_TEST_VERSION_OVERRIDE: Overrides the auto-detected shim version.
+#
+# Script Flow:
+#   1. Setup and Checks: Validates environment and detects Spark/Scala versions.
+#   2. Jars and Dependencies: Identifies necessary jars based on Spark version.
+#   3. Test Configuration: Determines parallelism and checks for necessary tools.
+#   4. Running Tests: Executes tests using runtests.py with specified options.
+#
+# Example Usage:
+#   To run all tests, including Avro tests:
+#     INCLUDE_SPARK_AVRO_JAR=true ./run_pyspark_from_build.sh
+#
+#   To run a specific test:
+#     TEST=my_test ./run_pyspark_from_build.sh
+#
+# Troubleshooting:
+#   - Ensure SPARK_HOME is correctly set.
+#   - Check for missing dependencies like findspark or xdist.
+#   - Verify that the necessary jars are correctly linked.
+#
+# Contributing:
+#   Contributions should follow the NVIDIA Spark-Rapids project guidelines.
+#   Ensure changes are tested thoroughly to maintain compatibility.
path: integration_tests/run_pyspark_from_build.sh
user: login: razajafri
body: Are you worried that we aren't consistent across the scripts? I am OK removing this instead of adding it to other files. 
created_at: 2025-04-17T18:44:17Z
author_association: COLLABORATOR
diff_hunk: @@ -80,7 +128,7 @@ else
         # Make sure we have Parquet version >= 1.12 in the dependency
         LOWEST_PARQUET_JAR=$(echo -e "$MIN_PARQUET_JAR\n$PARQUET_HADOOP_TESTS" | sort -V | head -1)
         export INCLUDE_PARQUET_HADOOP_TEST_JAR=$([[ "$LOWEST_PARQUET_JAR" == "$MIN_PARQUET_JAR" ]] && echo true || echo false)
-        PLUGIN_JARS=$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)
+        PLUGIN_JAR=${PLUGIN_JAR:-$(echo "$TARGET_DIR"/../../dist/target/rapids-4-spark_*.jar)}
path: integration_tests/run_pyspark_from_build.sh
user: login: razajafri
body: I already made the changes to the README file. PTAL. Let me know if you feel we want to add more to it. 
created_at: 2025-04-17T18:48:46Z
author_association: COLLABORATOR
diff_hunk: @@ -12,6 +12,54 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+# Documentation generated using a generative AI tool
+# ============================================================================
+# run_pyspark_from_build.sh
+# ============================================================================
+#
+# Description:
+#   This script runs PySpark integration tests from a build environment.
+#
+# Usage:
+#   To run this script, ensure you have Apache Spark installed and configured
+#   properly, along with the necessary dependencies (e.g., findspark, xdist).
+#
+# Parameters and Environment Variables:
+#   - SPARK_HOME: Path to your Apache Spark installation.
+#   - SKIP_TESTS: If set to true, skips running the Python integration tests.
+#   - INCLUDE_SPARK_AVRO_JAR: If set to true, includes Avro tests.
+#   - TEST: Specifies a specific test to run.
+#   - TEST_TAGS: Allows filtering tests based on tags.
+#   - TEST_TYPE: Specifies the type of tests to run.
+#   - LOCAL_JAR_PATH: Path to local jars if not building from source.
+#   - PLUGIN_JAR: Path to a built spark-rapids plugin jar, the default points to the target directory
+#   - INTEGRATION_TEST_VERSION_OVERRIDE: Overrides the auto-detected shim version.
+#
+# Script Flow:
+#   1. Setup and Checks: Validates environment and detects Spark/Scala versions.
+#   2. Jars and Dependencies: Identifies necessary jars based on Spark version.
+#   3. Test Configuration: Determines parallelism and checks for necessary tools.
+#   4. Running Tests: Executes tests using runtests.py with specified options.
+#
+# Example Usage:
+#   To run all tests, including Avro tests:
+#     INCLUDE_SPARK_AVRO_JAR=true ./run_pyspark_from_build.sh
+#
+#   To run a specific test:
+#     TEST=my_test ./run_pyspark_from_build.sh
+#
+# Troubleshooting:
+#   - Ensure SPARK_HOME is correctly set.
+#   - Check for missing dependencies like findspark or xdist.
+#   - Verify that the necessary jars are correctly linked.
+#
+# Contributing:
+#   Contributions should follow the NVIDIA Spark-Rapids project guidelines.
+#   Ensure changes are tested thoroughly to maintain compatibility.
path: integration_tests/run_pyspark_from_build.sh
user: login: gerashegalov
body: yes. what is so special about this file to have dedicated contribution guidelines?
created_at: 2025-04-17T23:10:46Z
author_association: COLLABORATOR
diff_hunk: @@ -12,6 +12,54 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+# Documentation generated using a generative AI tool
+# ============================================================================
+# run_pyspark_from_build.sh
+# ============================================================================
+#
+# Description:
+#   This script runs PySpark integration tests from a build environment.
+#
+# Usage:
+#   To run this script, ensure you have Apache Spark installed and configured
+#   properly, along with the necessary dependencies (e.g., findspark, xdist).
+#
+# Parameters and Environment Variables:
+#   - SPARK_HOME: Path to your Apache Spark installation.
+#   - SKIP_TESTS: If set to true, skips running the Python integration tests.
+#   - INCLUDE_SPARK_AVRO_JAR: If set to true, includes Avro tests.
+#   - TEST: Specifies a specific test to run.
+#   - TEST_TAGS: Allows filtering tests based on tags.
+#   - TEST_TYPE: Specifies the type of tests to run.
+#   - LOCAL_JAR_PATH: Path to local jars if not building from source.
+#   - PLUGIN_JAR: Path to a built spark-rapids plugin jar, the default points to the target directory
+#   - INTEGRATION_TEST_VERSION_OVERRIDE: Overrides the auto-detected shim version.
+#
+# Script Flow:
+#   1. Setup and Checks: Validates environment and detects Spark/Scala versions.
+#   2. Jars and Dependencies: Identifies necessary jars based on Spark version.
+#   3. Test Configuration: Determines parallelism and checks for necessary tools.
+#   4. Running Tests: Executes tests using runtests.py with specified options.
+#
+# Example Usage:
+#   To run all tests, including Avro tests:
+#     INCLUDE_SPARK_AVRO_JAR=true ./run_pyspark_from_build.sh
+#
+#   To run a specific test:
+#     TEST=my_test ./run_pyspark_from_build.sh
+#
+# Troubleshooting:
+#   - Ensure SPARK_HOME is correctly set.
+#   - Check for missing dependencies like findspark or xdist.
+#   - Verify that the necessary jars are correctly linked.
+#
+# Contributing:
+#   Contributions should follow the NVIDIA Spark-Rapids project guidelines.
+#   Ensure changes are tested thoroughly to maintain compatibility.
path: integration_tests/run_pyspark_from_build.sh
user: login: razajafri
body: Removed
created_at: 2025-04-23T19:08:42Z
author_association: COLLABORATOR

PR comments:
user: login: razajafri
created_at: 2025-03-26T20:22:13Z
body: build
author_association: COLLABORATOR
user: login: ttnghia
created_at: 2025-04-02T19:45:18Z
body: Something is wrong with your branch: huge change list. You may need to rebase.
author_association: COLLABORATOR
user: login: razajafri
created_at: 2025-04-02T20:17:17Z
body: I am trying to figure out why github is showing this change to be so big. 
```
> git diff origin/branch-25.06 --name-only
integration_tests/run_pyspark_from_build.sh
(END)
```
author_association: COLLABORATOR
user: login: abellina
created_at: 2025-04-02T20:26:37Z
body: I'd fetch from your local repo, then do: `git rebase -i origin/branch-25.06` provided origin points to the NVIDIA fork. I'd be curious if you don't see a similar list of commits show up.
author_association: COLLABORATOR
user: login: ttnghia
created_at: 2025-04-02T20:28:18Z
body: Oh you are merging to 23.06, not 25.06.
author_association: COLLABORATOR
user: login: razajafri
created_at: 2025-04-03T02:14:55Z
body: > Oh you are merging to 23.06, not 25.06.

Need to get my eyes checked again
author_association: COLLABORATOR
user: login: razajafri
created_at: 2025-04-03T03:00:26Z
body: build
author_association: COLLABORATOR
user: login: gerashegalov
created_at: 2025-04-04T20:53:30Z
body: Please update https://github.com/NVIDIA/spark-rapids/tree/branch-25.04/integration_tests#running-with-alternate-paths
author_association: COLLABORATOR
user: login: razajafri
created_at: 2025-04-09T04:47:15Z
body: build
author_association: COLLABORATOR
user: login: razajafri
created_at: 2025-04-10T01:19:39Z
body: @tgravescs, I was having some problems rebasing this. Can you PTAL 
author_association: COLLABORATOR
user: login: razajafri
created_at: 2025-04-24T02:29:48Z
body: @gerashegalov do you have any more concerns here? 
author_association: COLLABORATOR
user: login: razajafri
created_at: 2025-04-25T00:59:56Z
body: build
author_association: COLLABORATOR