Repository: mlflow/mlflow

Issue:
[FR] support pyspark.ml.autolog read log_model_allowlist file from storage
### Willingness to contribute

Yes. I would be willing to contribute this feature with guidance from the MLflow community.

### Proposal Summary

Update pyspark.ml.autolog's behavior to accept log_model_allowlist file that's stored in storage like S3 or Azure blob storage. 

### Motivation

#### What is the use case for this feature?
Let pyspark.ml.autolog accept log_model_allowlist file from most storage including S3, Azure blob storage, etc.

#### Why is this use case valuable to support for MLflow users in general?
Take databricks for example, we need to set cluster spark configuration of `spark.mlflow.pysparkml.autolog.logModelAllowlistFile` to point to the file, and if we support reading from whatever storage then this config can accept link to files stored outside of dbfs.

#### Why is this use case valuable to support for your project(s) or organization?
Make the updating log_model_allowlist file process easier, and for specific libraries we can provide a default link that users could point to without downloading & uploading themselves.

#### Why is it currently difficult to achieve this use case?
Currently pyspark.ml.autolog is reading log_model_allowlist file using python's `open` function, which only supports reading files from local disk. This behavior makes updating log_model_allowlist file a bit complicated. 

### Details

1. Within the source code we're reading the file using python's built-in `open` method to read the log_model_allowlist file, one option would be that we replace this `open` with `smart_open.open()`. This `smart_open` library supports reading files from lots of storages (https://github.com/RaRe-Technologies/smart_open) But this need mlflow to depend on smart_open library.
2. We check the format of the config link, and it should be a public-accessible link that we don't need extra confidential to download, then if it's a downloadable link we can download the file and read it.

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
Support reading the config from a downloadable link (http protocol) looks reasonable. We don't need to introduce a new dependency library.

CC @dbczumar @harupy Thoughts ? :)

@dbczumar @harupy Kindly ping on this 😃 
@serena-ruan That approach sounds great! I'll go ahead and close https://github.com/mlflow/mlflow/pull/5857 in favor of the new approach you've outlined. Apologies for the delay!
Thanks @serena-ruan . Sorry for late response. I will help review your new PR.

Resolving patch:
diff --git a/mlflow/pyspark/ml/__init__.py b/mlflow/pyspark/ml/__init__.py
--- a/mlflow/pyspark/ml/__init__.py
+++ b/mlflow/pyspark/ml/__init__.py
@@ -2,12 +2,15 @@
 import logging
 import numpy as np
 import time
+import os
 from pkg_resources import resource_filename
+from urllib.parse import urlparse
 import weakref
 
 import mlflow
 from mlflow.tracking.client import MlflowClient
 from mlflow.entities import Metric, Param
+from mlflow.exceptions import MlflowException
 from mlflow.utils import (
     _chunk_dict,
     _truncate_dict,
@@ -23,6 +26,11 @@
 from mlflow.utils.autologging_utils import get_method_call_arg_value
 from mlflow.utils.file_utils import TempDir
 from mlflow.utils.mlflow_tags import MLFLOW_AUTOLOGGING, MLFLOW_PARENT_RUN_ID
+from mlflow.utils.rest_utils import (
+    augmented_raise_for_status,
+    http_request,
+    MlflowHostCreds,
+)
 from mlflow.utils.validation import (
     MAX_PARAMS_TAGS_PER_BATCH,
     MAX_PARAM_VAL_LENGTH,
@@ -39,14 +47,36 @@
 
 
 def _read_log_model_allowlist_from_file(allowlist_file):
-    allowlist = set()
-    with open(allowlist_file) as f:
-        for line in f:
+    def _parse_allowlist_file(line_iter):
+        allowlist = set()
+        for line in line_iter:
             stripped = line.strip()
             is_blankline_or_comment = stripped == "" or stripped.startswith("#")
             if not is_blankline_or_comment:
                 allowlist.add(stripped)
-    return allowlist
+        return allowlist
+
+    url_parsed = urlparse(allowlist_file)
+    scheme = url_parsed.scheme
+    path = url_parsed.path
+    if os.name == "nt" and not url_parsed.hostname:
+        path = scheme + "://" + path
+        scheme = ""
+    if scheme in ("file", ""):
+        if not os.path.exists(path):
+            raise MlflowException.invalid_parameter_value(f"{allowlist_file} does not exist")
+
+        with open(allowlist_file) as f:
+            return _parse_allowlist_file(f)
+    else:
+        host_creds = MlflowHostCreds(
+            host=scheme + "://" + (url_parsed.hostname or ""),
+            username=url_parsed.username,
+            password=url_parsed.password,
+        )
+        response = http_request(host_creds=host_creds, endpoint=path, method="GET")
+        augmented_raise_for_status(response)
+        return _parse_allowlist_file(response.iter_lines(decode_unicode=True))
 
 
 def _read_log_model_allowlist():


Test patch:
diff --git a/tests/spark/autologging/ml/test_pyspark_ml_autologging_custom_allowlist.py b/tests/spark/autologging/ml/test_pyspark_ml_autologging_custom_allowlist.py
--- a/tests/spark/autologging/ml/test_pyspark_ml_autologging_custom_allowlist.py
+++ b/tests/spark/autologging/ml/test_pyspark_ml_autologging_custom_allowlist.py
@@ -27,3 +27,58 @@ def test_custom_log_model_allowlist(tmpdir):
     }
 
     spark_session.stop()
+
+
+def test_log_model_allowlist_from_url():
+
+    allowlist_file_path = "https://raw.githubusercontent.com/mlflow/mlflow/v1.26.0/mlflow/pyspark/ml/log_model_allowlist.txt"
+
+    spark_session = (
+        SparkSession.builder.config(
+            "spark.mlflow.pysparkml.autolog.logModelAllowlistFile", allowlist_file_path
+        )
+        .master("local[*]")
+        .getOrCreate()
+    )
+
+    mlflow.pyspark.ml.autolog()
+
+    assert mlflow.pyspark.ml._log_model_allowlist == {
+        "pyspark.ml.classification.LinearSVCModel",
+        "pyspark.ml.classification.DecisionTreeClassificationModel",
+        "pyspark.ml.classification.GBTClassificationModel",
+        "pyspark.ml.classification.LogisticRegressionModel",
+        "pyspark.ml.classification.RandomForestClassificationModel",
+        "pyspark.ml.classification.NaiveBayesModel",
+        "pyspark.ml.clustering.BisectingKMeansModel",
+        "pyspark.ml.clustering.KMeansModel",
+        "pyspark.ml.clustering.GaussianMixtureModel",
+        "pyspark.ml.regression.AFTSurvivalRegressionModel",
+        "pyspark.ml.regression.DecisionTreeRegressionModel",
+        "pyspark.ml.regression.GBTRegressionModel",
+        "pyspark.ml.regression.GeneralizedLinearRegressionModel",
+        "pyspark.ml.regression.LinearRegressionModel",
+        "pyspark.ml.regression.RandomForestRegressionModel",
+        "pyspark.ml.feature.BucketedRandomProjectionLSHModel",
+        "pyspark.ml.feature.ChiSqSelectorModel",
+        "pyspark.ml.feature.CountVectorizerModel",
+        "pyspark.ml.feature.IDFModel",
+        "pyspark.ml.feature.ImputerModel",
+        "pyspark.ml.feature.MaxAbsScalerModel",
+        "pyspark.ml.feature.MinHashLSHModel",
+        "pyspark.ml.feature.MinMaxScalerModel",
+        "pyspark.ml.feature.OneHotEncoderModel",
+        "pyspark.ml.feature.RobustScalerModel",
+        "pyspark.ml.feature.RFormulaModel",
+        "pyspark.ml.feature.StandardScalerModel",
+        "pyspark.ml.feature.StringIndexerModel",
+        "pyspark.ml.feature.VarianceThresholdSelectorModel",
+        "pyspark.ml.feature.VectorIndexerModel",
+        "pyspark.ml.feature.UnivariateFeatureSelectorModel",
+        "pyspark.ml.classification.OneVsRestModel",
+        "pyspark.ml.pipeline.PipelineModel",
+        "pyspark.ml.tuning.CrossValidatorModel",
+        "pyspark.ml.tuning.TrainValidationSplitModel",
+    }
+
+    spark_session.stop()