Repository: mlflow/mlflow

Issue:
Remove mleap / spark from sagemaker tests
## What changes are proposed in this pull request?

Remove mleap / spark from sagemaker tests

## How is this patch tested?

Cross version tests

## Release Notes

### Is this a user-facing change?

- [ ] No. You can skip the rest of this section.
- [X] Yes. Give a description of this change to be included in the release notes for MLflow users.

### What component(s), interfaces, languages, and integrations does this PR affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [X] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

<!--
Insert an empty named anchor here to allow jumping to this section with a fragment URL
(e.g. https://github.com/mlflow/mlflow/pull/123#user-content-release-note-category).
Note that GitHub prefixes anchor names in markdown with "user-content-".
-->
<a name="release-note-category"></a>
### How should the PR be classified in the release notes? Choose one:

- [ ] `rn/breaking-change` - The PR will be mentioned in the "Breaking Changes" section
- [X] `rn/none` - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
- [ ] `rn/feature` - A new user-facing feature worth mentioning in the release notes
- [ ] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes
- [ ] `rn/documentation` - A user-facing documentation change worth mentioning in the release notes



Discussion:
@dbczumar Thanks for the contribution! The DCO check failed. Please sign off your commits by following the instructions here: https://github.com/mlflow/mlflow/runs/3395075529. See https://github.com/mlflow/mlflow/blob/master/CONTRIBUTING.rst#sign-your-work for more details.

Resolving patch:
diff --git a/conftest.py b/conftest.py
--- a/conftest.py
+++ b/conftest.py
@@ -77,6 +77,7 @@ def pytest_ignore_collect(path, config):
             "tests/sagemaker",
             "tests/sklearn",
             "tests/spark",
+            "tests/mleap",
             "tests/tensorflow",
             "tests/azureml",
             "tests/onnx",
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -97,7 +97,6 @@ def package_files(directory):
             "pyarrow",
             # Required to log artifacts and models to AWS S3 artifact locations
             "boto3",
-            "mleap",
             # Required to log artifacts and models to GCS artifact locations
             "google-cloud-storage",
             "azureml-core>=1.2.0",


Test patch:
diff --git a/tests/spark/test_mleap_model_export.py b/tests/mleap/test_mleap_model_export.py
similarity index 88%
rename from tests/spark/test_mleap_model_export.py
rename to tests/mleap/test_mleap_model_export.py
--- a/tests/spark/test_mleap_model_export.py
+++ b/tests/mleap/test_mleap_model_export.py
@@ -1,11 +1,13 @@
 import json
 import os
 from unittest import mock
+from packaging.version import Version
 
 import numpy as np
 import pyspark
 from pyspark.ml.pipeline import Pipeline
 from pyspark.ml.wrapper import JavaModel
+import mleap.version
 import pytest
 
 import mlflow
@@ -26,16 +28,26 @@
 )
 
 
+def get_mleap_jars():
+    mleap_ver = Version(
+        mleap.version if isinstance(mleap.version, str) else mleap.version.__version__
+    )
+    scala_ver = "2.11" if mleap_ver < Version("0.18.0") else "2.12"
+    jar_ver = f"{mleap_ver.major}.{mleap_ver.minor}.0"
+    return ",".join(
+        [
+            f"ml.combust.mleap:mleap-spark-base_{scala_ver}:{jar_ver}",
+            f"ml.combust.mleap:mleap-spark_{scala_ver}:{jar_ver}",
+        ]
+    )
+
+
 @pytest.fixture(scope="session", autouse=True)
 def spark_context():
     conf = pyspark.SparkConf()
-    conf.set(
-        key="spark.jars.packages",
-        value=(
-            "ml.combust.mleap:mleap-spark-base_2.11:0.12.0,"
-            "ml.combust.mleap:mleap-spark_2.11:0.12.0"
-        ),
-    )
+    conf.set(key="spark.jars.packages", value=get_mleap_jars())
+    # Exclude `net.sourceforge.f2j` to avoid `java.io.FileNotFoundException`
+    conf.set(key="spark.jars.excludes", value="net.sourceforge.f2j:arpack_combined_all")
     spark_session = get_spark_session(conf)
     return spark_session.sparkContext