Repository: mlflow/mlflow

Issue:
[BUG]  Failed to score model on payload. Error: 'Endpoint' object has no attribute 'get'
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Where did you encounter this bug?

Local machine

### Willingness to contribute

No. I cannot contribute a bug fix at this time.

### MLflow version

- Client: 2.9.2
- Tracking server: 2.9.2


### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS 13.6
- **Python version**: Python 3.8.6

### Describe the problem

The llm evaluation metric generated with mlflow.evaluate() shows the following error
> Failed to score model on payload. Error: 'Endpoint' object has no attribute 'get'
![endpoint_error](https://github.com/mlflow/mlflow/assets/106637777/509d14ca-251f-403c-964d-47adf6484884)

I was able to avoid the ahead error with the following source code change. In <virtual_environment>/lib/python3.8/site-packages/mlflow/metrics/genai/model_utils.py, I changed line 122 from endpoint_type = endpoint.get("task", endpoint.get("endpoint_type")) to endpoint_type = endpoint.endpoint_type. 

### Tracking information

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```shell
MLflow version: 2.9.2
Tracking URI: http://127.0.0.1:5200/
Artifact URI: mlflow-artifacts:/587337650981774112/1149ed5fd34841f88e66c8303b25cf6e/artifacts
System information: Darwin Darwin Kernel Version 22.6.0: Tue Nov  7 21:48:06 PST 2023; root:xnu-8796.141.3.702.9~2/RELEASE_X86_64
Python version: 3.8.6
MLflow version: 2.9.2
MLflow module location: /Users/ys/dev/env/mlflow_jupyter/lib/python3.8/site-packages/mlflow/__init__.py
Tracking URI: http://127.0.0.1:5200/
Registry URI: http://127.0.0.1:5200/
Active experiment ID: 587337650981774112
Active run ID: 1149ed5fd34841f88e66c8303b25cf6e
Active run artifact URI: mlflow-artifacts:/587337650981774112/1149ed5fd34841f88e66c8303b25cf6e/artifacts
MLflow dependencies: 
  Flask: 3.0.0
  Jinja2: 3.1.2
  aiohttp: 3.9.1
  alembic: 1.13.1
  boto3: 1.34.15
  click: 8.1.7
  cloudpickle: 3.0.0
  databricks-cli: 0.18.0
  docker: 6.1.3
  entrypoints: 0.4
  fastapi: 0.108.0
  gitpython: 3.1.40
  gunicorn: 21.2.0
  importlib-metadata: 7.0.1
  markdown: 3.5.1
  matplotlib: 3.7.4
  numpy: 1.24.4
  packaging: 23.2
  pandas: 2.0.3
  protobuf: 4.25.1
  pyarrow: 14.0.2
  pydantic: 2.5.3
  pytz: 2023.3.post1
  pyyaml: 6.0.1
  querystring-parser: 1.2.4
  requests: 2.31.0
  scikit-learn: 1.3.2
  scipy: 1.10.1
  sqlalchemy: 2.0.25
  sqlparse: 0.4.4
  tiktoken: 0.5.2
  uvicorn: 0.25.0
  watchfiles: 0.21.0
```


### Code to reproduce issue

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
import pandas as pd
import mlflow
from mlflow.deployments import set_deployments_target
from mlflow.metrics.genai.metric_definitions import (
    answer_similarity,
)

df = pd.DataFrame(
    {
        "question": [
            "How does useEffect() work?",
            "What does the static keyword in a function mean?",
            "What does the 'finally' block in Python do?",
            "What is the difference between multiprocessing and multithreading?",
        ],
        "ground_truth": [
            "The useEffect() hook tells React that your component needs to do something after render. React will remember the function you passed (we’ll refer to it as our “effect”), and call it later after performing the DOM updates.",
            "Static members belongs to the class, rather than a specific instance. This means that only one instance of a static member exists, even if you create multiple objects of the class, or if you don't create any. It will be shared by all objects.",
            "'Finally' defines a block of code to run when the try... except...else block is final. The finally block will be executed no matter if the try block raises an error or not.",
            "Multithreading refers to the ability of a processor to execute multiple threads concurrently, where each thread runs a process. Whereas multiprocessing refers to the ability of a system to run multiple processors in parallel, where each processor can run one or more threads.",
        ],
    }
)

logged_model = 'runs:/397d4d9c82474edd8b0e8fe8a73b0650/model'
mlflow.set_tracking_uri('http://127.0.0.1:5200/')
mlflow.gateway.set_gateway_uri('http://127.0.0.1:5000/')
set_deployments_target("http://localhost:5000")
my_answer_similarity = answer_similarity(
    model="endpoints:/completions"
)

with mlflow.start_run():
    results = mlflow.evaluate(
        model=logged_model,
        data=df,
        targets="ground_truth",
        model_type="text-summarization",
        extra_metrics=[my_answer_similarity],
        evaluator_config={'col_mapping': {"inputs": "question"}}
    )
eval_table = results.tables["eval_results_table"]
print(f"See evaluation table below: \n{eval_table}")
```


### Stack trace

No error from stack trace

### Other info / logs

_No response_

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
@prithvikannan @sunishsheth2009 thoughts on this?
could you also send the config.yaml file?
<!-- assign-maintainer -->
@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.
any update on this?

sample config file that leads to the same issue on my end:

```
endpoints:
  - name: chat
    endpoints_type: llm/v1/chat
    model:
      provider: openai
      name: gpt-4
      config:
        openai_api_type: "azure"
        openai_api_key: $AZURE_OPENAI_KEY
        openai_deployment_name: ".."
        openai_api_base: "..."
        openai_api_version: "2023-07-01-preview"

  - name: completions
    endpoints_type: llm/v1/completions
    model:
      provider: openai
      name: gpt-4
      config:
        openai_api_type: "azure"
        openai_api_key: $AZURE_OPENAI_KEY
        openai_deployment_name: "..."
        openai_api_base: "..."
        openai_api_version: "2023-07-01-preview"
```
Similar config file to @alistairwgillespie above with same error message. 

Hardcoding `endpoint_type` in `metrics/genai/model_utils.py` like below seems to resolve the issue. Not sure of the explanation for why it is expecting `Endpoint` to have a `get` method.
 
```
client = get_deploy_client()

endpoint = client.get_endpoint(deployment_uri)
# endpoint_type = endpoint.get("task", endpoint.get("endpoint_type"))
endpoint_type = "llm/v1/completions"
```

Resolving patch:
diff --git a/mlflow/metrics/genai/model_utils.py b/mlflow/metrics/genai/model_utils.py
--- a/mlflow/metrics/genai/model_utils.py
+++ b/mlflow/metrics/genai/model_utils.py
@@ -114,11 +114,15 @@ def _call_openai_api(openai_uri, payload, eval_parameters):
 
 
 def _call_deployments_api(deployment_uri, payload, eval_parameters):
+    from pydantic import BaseModel
+
     from mlflow.deployments import get_deploy_client
 
     client = get_deploy_client()
 
     endpoint = client.get_endpoint(deployment_uri)
+    # TODO: Standardize the return type of `get_endpoint` and remove this check
+    endpoint = endpoint.dict() if isinstance(endpoint, BaseModel) else endpoint
     endpoint_type = endpoint.get("task", endpoint.get("endpoint_type"))
 
     if endpoint_type == "llm/v1/completions":


Test patch: