Repository: mlflow/mlflow
Pull request: Use `pyarrow.fs.HadoopFileSystem` in pyarrow > 2.0.0
[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/maksboyarin/mlflow/pull/9878?quickstart=1)

### Related Issues/PRs

<!-- Uncomment 'Resolve' if this PR can close the linked items. -->
Resolve #8213

### What changes are proposed in this pull request?
Fixed a deprecation warning: 
FutureWarning: pyarrow.hdfs.connect is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.
<!-- Please fill in changes proposed in this PR. -->

### How is this PR tested?

- [ ] Existing unit/integration tests
- [ ] New unit/integration tests
- [ ] Manual tests

<!-- Attach code, screenshot, video used for manual testing here. -->

### Does this PR require documentation update?

- [x] No. You can skip the rest of this section.
- [ ] Yes. I've updated:
  - [ ] Examples
  - [ ] API references
  - [ ] Instructions

### Release Notes

#### Is this a user-facing change?

- [ ] No. You can skip the rest of this section.
- [ ] Yes. Give a description of this change to be included in the release notes for MLflow users.

<!-- Details in 1-2 sentences. You can just refer to another PR with a description if this PR is part of a larger change. -->

#### What component(s), interfaces, languages, and integrations does this PR affect?

Components

- [x] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/gateway`: AI Gateway service, Gateway client APIs, third-party Gateway integrations
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

<!--
Insert an empty named anchor here to allow jumping to this section with a fragment URL
(e.g. https://github.com/mlflow/mlflow/pull/123#user-content-release-note-category).
Note that GitHub prefixes anchor names in markdown with "user-content-".
-->

<a name="release-note-category"></a>

#### How should the PR be classified in the release notes? Choose one:

- [x] `rn/none` - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
- [ ] `rn/breaking-change` - The PR will be mentioned in the "Breaking Changes" section
- [ ] `rn/feature` - A new user-facing feature worth mentioning in the release notes
- [ ] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes
- [ ] `rn/documentation` - A user-facing documentation change worth mentioning in the release notes


Problem statement:
[BUG] FutureWarning: pyarrow.hdfs.connect is deprecated
### Willingness to contribute

Yes. I can contribute a fix for this bug independently.

### MLflow version

2.2.2

### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: CentOS 7
- **Python version**: 3.9.6
- **pyarrow version**: 5.0.0


### Describe the problem

When loading a model from the MLflow Model Registry (backend store is HDFS) I get the following warning:

```
legobricks/lib/python3.9/site-packages/mlflow/store/artifact/hdfs_artifact_repo.py:188: FutureWarning: pyarrow.hdfs.connect is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.
  connected = pa.hdfs.connect(
```

### Steps to reproduce the bug

1. Create a conda env with mlflow and pyarrow installed
2. Start Mlflow with hdfs as backend store
3. Train a model and save it
4. Store the trained model in the model registry
5. Load model from the registry

### Code to generate data required to reproduce the bug

_No response_

### Is the console panel in DevTools showing errors relevant to the bug?

_No response_

### Does the network panel in DevTools contain failed requests relevant to the bug?

_No response_

Hints from issue discussion:
Suggested change:

* Switch to new API in this line: https://github.com/mlflow/mlflow/blob/master/mlflow/store/artifact/hdfs_artifact_repo.py#L188
* Require at least `2.0.0` pyarrow version in the setup.py
<!-- assign-maintainer -->
@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.
@julcsii The suggestion makes sense to me! Looking forward to your PR :)
<!-- assign-maintainer -->
@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.

Commit message: made the changes version-specific

Signed-off-by: maksboyarin <99800578+maksboyarin@users.noreply.github.com>

Patch to review:
diff --git a/mlflow/store/artifact/hdfs_artifact_repo.py b/mlflow/store/artifact/hdfs_artifact_repo.py
index cacbe2f8a3b41..4ceeef7bcb3aa 100644
--- a/mlflow/store/artifact/hdfs_artifact_repo.py
+++ b/mlflow/store/artifact/hdfs_artifact_repo.py
@@ -173,7 +173,7 @@ def hdfs_system(scheme, host, port):
     :param host: hostname or when relaying on the core-site.xml config use 'default'
     :param port: port or when relaying on the core-site.xml config use 0
     """
-    import pyarrow as pa
+    import pyarrow
 
     kerb_ticket = MLFLOW_KERBEROS_TICKET_CACHE.get()
     kerberos_user = MLFLOW_KERBEROS_USER.get()
@@ -183,14 +183,22 @@ def hdfs_system(scheme, host, port):
         host = scheme + "://" + host
     else:
         host = "default"
-
-    connected = pa.hdfs.connect(
-        host=host,
-        port=port or 0,
-        user=kerberos_user,
-        kerb_ticket=kerb_ticket,
-        extra_conf=extra_conf,
-    )
+    if pyarrow.__version__ < "2.0.0":
+        connected = pyarrow.fs.HadoopFileSystem(
+            host=host,
+            port=port or 0,
+            user=kerberos_user,
+            kerb_ticket=kerb_ticket,
+            extra_conf=extra_conf,
+        )
+    else:
+        connected = pyarrow.hdfs.connect(
+            host=host,
+            port=port or 0,
+            user=kerberos_user,
+            kerb_ticket=kerb_ticket,
+            extra_conf=extra_conf,
+        )
     yield connected
     connected.close()
 


Code review comments:
diff_hunk: @@ -183,14 +183,22 @@ def hdfs_system(scheme, host, port):
         host = scheme + "://" + host
     else:
         host = "default"
-
-    connected = pa.hdfs.connect(
-        host=host,
-        port=port or 0,
-        user=kerberos_user,
-        kerb_ticket=kerb_ticket,
-        extra_conf=extra_conf,
-    )
+    if pyarrow.__version__ < "2.0.0":
path: mlflow/store/artifact/hdfs_artifact_repo.py
text: @user1:
can we use `packaging.version.Version` here?

@user1:
/ai suggest code

@user2:
@user1

<!---->
Yes, using `packaging.version.Version` could be beneficial for proper version comparison. Here's how you could modify the code to include it:

```python
from packaging import version

if version.parse(pyarrow.__version__) < version.parse("2.0.0"):
```

This method provides reliable version comparison even for complex version numbers. For example, it would correctly handle a version like "2.0.0rc1". 

Please note that you need to add the `packaging` library to your project dependencies if it's not already there.
<!---->

<details><summary>Click here to see the usage data</summary>

`````json
{
  "prompt_tokens": 71,
  "completion_tokens": 113,
  "total_tokens": 184,
  "estimated_cost_in_usd": 0.00891
}
`````

</details>

@user1:
@author can you apply the suggestion?