Repository: mlflow/mlflow

Issue:
[BUG] FutureWarning: pyarrow.hdfs.connect is deprecated
### Willingness to contribute

Yes. I can contribute a fix for this bug independently.

### MLflow version

2.2.2

### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: CentOS 7
- **Python version**: 3.9.6
- **pyarrow version**: 5.0.0


### Describe the problem

When loading a model from the MLflow Model Registry (backend store is HDFS) I get the following warning:

```
legobricks/lib/python3.9/site-packages/mlflow/store/artifact/hdfs_artifact_repo.py:188: FutureWarning: pyarrow.hdfs.connect is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.
  connected = pa.hdfs.connect(
```

### Steps to reproduce the bug

1. Create a conda env with mlflow and pyarrow installed
2. Start Mlflow with hdfs as backend store
3. Train a model and save it
4. Store the trained model in the model registry
5. Load model from the registry

### Code to generate data required to reproduce the bug

_No response_

### Is the console panel in DevTools showing errors relevant to the bug?

_No response_

### Does the network panel in DevTools contain failed requests relevant to the bug?

_No response_


Discussion:
Suggested change:

* Switch to new API in this line: https://github.com/mlflow/mlflow/blob/master/mlflow/store/artifact/hdfs_artifact_repo.py#L188
* Require at least `2.0.0` pyarrow version in the setup.py
<!-- assign-maintainer -->
@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.
@julcsii The suggestion makes sense to me! Looking forward to your PR :)
<!-- assign-maintainer -->
@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.

Resolving patch:
diff --git a/mlflow/store/artifact/hdfs_artifact_repo.py b/mlflow/store/artifact/hdfs_artifact_repo.py
--- a/mlflow/store/artifact/hdfs_artifact_repo.py
+++ b/mlflow/store/artifact/hdfs_artifact_repo.py
@@ -4,6 +4,8 @@
 import urllib.parse
 from contextlib import contextmanager
 
+import packaging.version
+
 from mlflow.entities import FileInfo
 from mlflow.environment_variables import (
     MLFLOW_KERBEROS_TICKET_CACHE,
@@ -173,7 +175,7 @@ def hdfs_system(scheme, host, port):
     :param host: hostname or when relaying on the core-site.xml config use 'default'
     :param port: port or when relaying on the core-site.xml config use 0
     """
-    import pyarrow as pa
+    import pyarrow
 
     kerb_ticket = MLFLOW_KERBEROS_TICKET_CACHE.get()
     kerberos_user = MLFLOW_KERBEROS_USER.get()
@@ -183,14 +185,22 @@ def hdfs_system(scheme, host, port):
         host = scheme + "://" + host
     else:
         host = "default"
-
-    connected = pa.hdfs.connect(
-        host=host,
-        port=port or 0,
-        user=kerberos_user,
-        kerb_ticket=kerb_ticket,
-        extra_conf=extra_conf,
-    )
+    if packaging.version.parse(pyarrow.__version__) < packaging.version.parse("2.0.0"):
+        connected = pyarrow.fs.HadoopFileSystem(
+            host=host,
+            port=port or 0,
+            user=kerberos_user,
+            kerb_ticket=kerb_ticket,
+            extra_conf=extra_conf,
+        )
+    else:
+        connected = pyarrow.hdfs.connect(
+            host=host,
+            port=port or 0,
+            user=kerberos_user,
+            kerb_ticket=kerb_ticket,
+            extra_conf=extra_conf,
+        )
     yield connected
     connected.close()
 


Test patch: