Repository: mlflow/mlflow

Issue:
[BUG] When using a custom LLM as judge model (endpoints:/endpoint_name), response is NaN
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Where did you encounter this bug?

Local machine

### Willingness to contribute

No. I cannot contribute a bug fix at this time.

### MLflow version

- Client: 2.10.0
- Tracking server: 2.10.0


### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: WSL (Ubuntu)
- **Python version**: 3.10.13
- **yarn version, if running the dev UI**:


### Describe the problem

I am trying to use a [custom LLM as judge model](https://mlflow.org/docs/latest/llms/llm-evaluate/index.html#selecting-the-llm-as-judge-model), but using `endpoints:/chat` doesn't work. The resulting metrics are NaN.
![image](https://github.com/mlflow/mlflow/assets/158079614/6c6579d0-c30d-4e42-ab9f-3bfd3f0384e1)



By default, is the chat or completions version of gpt-4 being used?

`config.yaml`
```
endpoints:
  - name: completions
    endpoint_type: llm/v1/completions
    model:
      provider: openai
      name: gpt-3.5-turbo
      config:
        openai_api_key: $OPENAI_API_KEY
    limit:
      renewal_period: minute
      calls: 10

  - name: chat
    endpoint_type: llm/v1/chat
    model:
      provider: openai
      name: gpt-3.5-turbo
      config:
        openai_api_key: $OPENAI_API_KEY

  - name: azure-chat
    endpoint_type: llm/v1/chat
    model:
      provider: openai
      name: gpt-4
      config:
        openai_api_type: "azure"
        openai_api_key: $AZURE_OPENAI_API_KEY
        openai_deployment_name: "gpt-4"
        openai_api_base: "_"
        openai_api_version: "2023-05-15"

  - name: azure-completions
    endpoint_type: llm/v1/completions
    model:
      provider: openai
      name: gpt-4
      config:
        openai_api_type: "azure"
        openai_api_key: $AZURE_OPENAI_API_KEY
        openai_deployment_name: "gpt-4"
        openai_api_base: "_"
        openai_api_version: "2023-05-15"

  - name: embeddings
    endpoint_type: llm/v1/embeddings
    model:
      provider: openai
      name: text-embedding-ada-002
      config:
        openai_api_key: $OPENAI_API_KEY
```
Create deployments server
```bash
mlflow deployments start-server --config-path config.yaml --port 7000
```
Connect tracking server and deployments server
```bash
export MLFLOW_DEPLOYMENTS_TARGET="http://127.0.0.1:7000"
mlflow server --port 5000
```

I can use the endpoints via `get_deploy_client`, so I am not sure what went wrong.
```python
from mlflow.deployments import get_deploy_client

client = get_deploy_client("http://localhost:7000")
response = client.predict(
    endpoint="chat",
    inputs={"messages": [{"role": "user", "content": "Tell me a joke about rabbits"}]},
)
print(response)
```
```
{'id': 'chatcmpl-8qR3CU9RyUMeOPQZJsfr2WPbBaX5Q', 'object': 'chat.completion', 'created': 1707507730, 'model': 'gpt-3.5-turbo-0613', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': 'Why did the rabbit go to the barber?\n\nBecause it wanted a hare-cut!'}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 13, 'completion_tokens': 16, 'total_tokens': 29}}
```
### Tracking information

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```shell
System information: Windows 10.0.19045
Python version: 3.10.13
MLflow version: 2.10.0
MLflow module location: [c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\__init__.py](file:///C:/Users/_/miniconda3/envs/ml-3.10/lib/site-packages/mlflow/__init__.py)
Tracking URI: http://127.0.0.1:5000/
Registry URI: http://127.0.0.1:5000/
MLflow dependencies: 
  Flask: 3.0.1
  Jinja2: 3.1.3
  aiohttp: 3.9.2
  alembic: 1.13.1
  boto3: 1.34.37
  botocore: 1.34.37
  click: 8.1.7
  cloudpickle: 3.0.0
  databricks-cli: 0.18.0
  docker: 7.0.0
  entrypoints: 0.4
  fastapi: 0.109.0
  gitpython: 3.1.41
  importlib-metadata: 6.11.0
  kubernetes: 29.0.0
  markdown: 3.5.2
  matplotlib: 3.8.2
  numpy: 1.26.3
  packaging: 23.2
  pandas: 2.2.0
  protobuf: 4.23.4
  pyarrow: 15.0.0
  pydantic: 2.6.0
  pytz: 2023.4
  pyyaml: 6.0.1
  querystring-parser: 1.2.4
  requests: 2.31.0
  scikit-learn: 1.4.0
  scipy: 1.12.0
  slowapi: 0.1.8
  sqlalchemy: 2.0.25
  sqlparse: 0.4.4
  tiktoken: 0.5.2
  uvicorn: 0.27.0.post1
  waitress: 2.1.2
  watchfiles: 0.21.0
```

### Code to reproduce issue

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```python
from datasets import load_dataset

dataset = load_dataset("squad")
eval_df = dataset["train"].to_pandas().sample(n=10, random_state=0)
def get_answer(row):
    answers = row["answers"]["text"]
    return answers[0]

eval_df["answer"] = eval_df.apply(get_answer, axis=1)
eval_df = eval_df.drop(columns=["answers", "id", "title"])

import mlflow

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("test-deployment")


from langchain_core.prompts import ChatPromptTemplate, PromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_core.runnables import RunnableLambda, RunnablePassthrough
from langchain_openai import ChatOpenAI

template = """Answer the question based only on the following context:
{context}

Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)

llm = ChatOpenAI()
chain = (
    RunnablePassthrough()
    | prompt 
    | llm
    | StrOutputParser()
)
chain.invoke({"question": "What is 1 + 1?", "context": "1 + 1 is 3"})

from mlflow.metrics.genai import EvaluationExample, answer_similarity

# Create an example to describe what answer_similarity means like for this problem.
example = EvaluationExample(
    input="What is MLflow?",
    output="MLflow is an open-source platform for managing machine "
    "learning workflows, including experiment tracking, model packaging, "
    "versioning, and deployment, simplifying the ML lifecycle.",
    score=4,
    justification="The definition effectively explains what MLflow is "
    "its purpose, and its developer. It could be more concise for a 5-score.",
    grading_context={
        "targets": "MLflow is an open-source platform for managing "
        "the end-to-end machine learning (ML) lifecycle. It was developed by Databricks, "
        "a company that specializes in big data and machine learning solutions. MLflow is "
        "designed to address the challenges that data scientists and machine learning "
        "engineers face when developing, training, and deploying machine learning models."
    },
)

# Construct the metric using OpenAI GPT-4 as the judge
mlflow.deployments.set_deployments_target("http://localhost:7000")

judge_model = "endpoints:/chat"
# judge_model = "openai:/gpt-4"

answer_similarity_metric = answer_similarity(model=judge_model, examples=[example])

mlflow.deployments.get_deployments_target() # wasn't set even if in env variable

def model(input_df):
    answers = []
    for _, row in input_df.iterrows():
        answer = chain.invoke({"question": row["question"], "context": row["context"]})
        answers.append(answer)
    return answers

with mlflow.start_run(run_name="mistral"):
    mlflow.evaluate(
        model,
        eval_df,
        # model_type="question-answering",
        # evaluators="default",
        targets="answer",
        extra_metrics=[answer_similarity_metric], # faithfulness_metric, relevance_metric],
        evaluator_config={
            "col_mapping": {
                "inputs": "question",
            }
        },
    )
```


### Stack trace

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
[c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\data\digest_utils.py:29](file:///C:/Users/_/miniconda3/envs/ml-3.10/lib/site-packages/mlflow/data/digest_utils.py:29): FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
  string_columns = trimmed_df.columns[(df.applymap(type) == str).all(0)]
[c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\models\evaluation\base.py:414](file:///C:/Users/_/miniconda3/envs/ml-3.10/lib/site-packages/mlflow/models/evaluation/base.py:414): FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
  data = data.applymap(_hash_array_like_element_as_bytes)
[c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\models\evaluation\base.py:414](file:///C:/Users/_/miniconda3/envs/ml-3.10/lib/site-packages/mlflow/models/evaluation/base.py:414): FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
  data = data.applymap(_hash_array_like_element_as_bytes)
2024/02/09 15:08:14 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.
2024/02/09 15:08:14 INFO mlflow.models.evaluation.default_evaluator: Computing model predictions.
2024/02/09 15:08:23 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...
100%
1/1 [00:02<00:00, 2.03s/it]
[c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\numpy\core\fromnumeric.py:3504](file:///C:/Users/_/miniconda3/envs/ml-3.10/lib/site-packages/numpy/core/fromnumeric.py:3504): RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
[c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\numpy\core\_methods.py:129](file:///C:/Users/_/miniconda3/envs/ml-3.10/lib/site-packages/numpy/core/_methods.py:129): RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
[c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\numpy\core\fromnumeric.py:3787](file:///C:/Users/_/miniconda3/envs/ml-3.10/lib/site-packages/numpy/core/fromnumeric.py:3787): RuntimeWarning: Degrees of freedom <= 0 for slice
  return _methods._var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,
[c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\numpy\core\_methods.py:163](file:///C:/Users/_/miniconda3/envs/ml-3.10/lib/site-packages/numpy/core/_methods.py:163): RuntimeWarning: invalid value encountered in divide
  arrmean = um.true_divide(arrmean, div, out=arrmean,
[c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\numpy\core\_methods.py:198](file:///C:/Users/_/miniconda3/envs/ml-3.10/lib/site-packages/numpy/core/_methods.py:198): RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
2024/02/09 15:08:25 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: answer_similarity
100%
10/10 [00:00<00:00, 42.78it/s]
```


### Other info / logs

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
```


### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [x] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [x] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

Discussion:
github-actions[bot]: <!-- documentation preview -->


Documentation preview for 274e5e46d43cefd09f9b409f8cfef067c2b502a8 will be available when [this CircleCI job](https://circleci.com/gh/mlflow/mlflow/90071) completes successfully.

- [Top page](https://output.circle-artifacts.com/output/job/703c86af-c508-4412-a47d-e3a0de1d7cfb/artifacts/0/docs/build/html/index.html)
- [Changed pages](https://output.circle-artifacts.com/output/job/703c86af-c508-4412-a47d-e3a0de1d7cfb/artifacts/0/docs/build/html/diff.html)

<details>
<summary>More info</summary>

- Ignore this comment if this PR does not change the documentation.
- It takes a few minutes for the preview to be available.
- The preview is updated when a new commit is pushed to this PR.
- This comment was created by https://github.com/mlflow/mlflow/actions/runs/7881643054.

</details>

harupy: @B-Step62 It's possible but if we do that, we'd need to inspect the object type in `_call_deployments_api` because `DatabricksDeploymentClient.get_endpoint` returns `AttrDict`. The code would look like:

```
endpoint_dict = endpoint.dict() if is_pydantic_model(endpoint) else endpoint
endpoint_dict.get("task", endpoint_dict.get("endpoint_type"))
```
harupy: The real issue is each client returns different objects.
WeichenXu123: What about adding a method like `mlflow.deployments.server.config.Endpoint.to_attr_dict()` ?
This way looks less ambiguous.
B-Step62: > The real issue is each client returns different objects.

100%, and I assume making change to Databricks client is too dangerous here? 

I think the solution [here](https://github.com/mlflow/mlflow/pull/11096#issuecomment-1970539657) is safer, requiring just minimum hack to resolve the issue. Pathing Endpoint class might cause some long-term drift... like `from mlflow.deployment import Endpoint` and `from mlflow.deployments.server.config import Endpoint` returns different object, and once more ppl rely on these classes it becomes hard to unify.

Alternatively we could add a getter method to the Endpoint pydantic model, but it feels less clean than the solution above.

harupy: @B-Step62 Let's go with [this solution](https://github.com/mlflow/mlflow/pull/11096#issuecomment-1970539657)!

Patch:
diff --git a/docs/source/_static/images/logos/mistral-ai-logo.svg b/docs/source/_static/images/logos/mistral-ai-logo.svg
new file mode 100644
index 000000000..dd739e465
--- /dev/null
+++ b/docs/source/_static/images/logos/mistral-ai-logo.svg
@@ -0,0 +1,38 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<svg width="512px" height="164px" viewBox="0 0 512 164" version="1.1" xmlns="http://www.w3.org/2000/svg" preserveAspectRatio="xMidYMid">
+    <title>Mistral AI</title>
+    <g>
+        <g transform="translate(213.3167, 27.4293)" fill="#000000">
+            <path d="M3.54926427,0.866543136 L14.7228067,0.866543136 L19.3327713,17.5381056 L20.9144735,24.0433432 L21.204218,24.0433432 L22.7664598,17.5381056 L27.3980472,0.866543136 L38.5715896,0.866543136 L38.5715896,47.9106495 L29.5614007,47.9106495 L29.5614007,33.3473459 L29.5670358,32.5879187 C29.6450324,27.0205397 30.5137834,16.3474271 30.9885004,11.494286 L30.7722732,11.494286 L27.1612784,24.1211468 L23.498389,35.4534436 L18.3305581,35.4534436 L14.6038817,24.1211468 L11.2361425,11.494286 L10.9864,11.494286 L11.101213,12.6193443 C11.6427988,18.1141469 12.4134998,28.4113913 12.4134998,33.3473459 L12.4134998,47.9106495 L3.54926427,47.9106495 L3.54926427,0.866543136 Z M48.0762959,38.9834424 L59.0682074,38.9834424 L59.0682074,9.79416023 L48.0762959,9.79416023 L48.0762959,0.86620785 L80.7103914,0.86620785 L80.7103914,9.79416023 L69.7217233,9.79416023 L69.7217233,38.9834424 L80.7103914,38.9834424 L80.7103914,47.9113948 L48.0762959,47.9113948 L48.0762959,38.9834424 Z M89.7022114,41.9836366 L95.8160364,34.6711876 C99.3740556,37.5834161 103.76455,39.5803727 108.328025,39.5803727 C112.848256,39.5803727 114.93593,37.8308744 114.93593,35.5313486 C114.93593,32.1436542 112.137949,31.3245562 107.684749,29.4605139 L101.552545,26.844371 C96.3382252,24.8636234 91.5758204,20.8664682 91.5758204,14.018679 C91.5758204,6.22752234 98.5977998,0 108.826429,0 C114.487258,0 120.20755,2.12230676 124.457496,6.15944427 L119.090736,12.8710773 C115.774891,10.4915867 112.841769,9.19594223 108.242616,9.19594223 C104.73541,9.19594223 102.353667,10.6504356 102.353667,13.3973316 C102.353667,16.2274341 105.508422,17.2637336 110.023247,18.9548793 L115.781378,21.3938031 C121.946017,23.7030544 125.835945,27.5456831 125.835945,34.1849157 C125.835945,42.0052487 119.272367,48.776315 107.620962,48.776315 C101.260638,48.776315 94.6927359,46.4508546 89.7022114,41.9836366 Z M145.739032,9.79417006 L131.628043,9.79417006 L131.628043,0.866217681 L170.501375,0.866217681 L170.501375,9.79417006 L156.392548,9.79417006 L156.392548,47.9114046 L145.739032,47.9114046 L145.739032,9.79417006 Z M188.420134,9.31643532 L193.613913,9.31643532 C199.208792,9.31643532 202.211107,10.8779085 202.211107,15.5234262 C202.211107,20.1289615 199.208792,22.5365478 193.613913,22.5365478 L188.420134,22.5365478 L188.420134,9.31643532 Z M214.913376,47.911297 L204.26851,29.0904126 C209.293631,26.8189825 212.655964,22.4360516 212.655964,15.5234262 C212.655964,4.44245627 204.614496,0.936171221 194.779555,0.867153617 L177.746077,0.866110028 L177.746077,47.911297 L188.420134,47.911297 L188.420134,30.9868731 L194.011771,30.9868731 L202.998175,47.911297 L214.913376,47.911297 Z M232.595264,28.4272429 L233.708834,24.0929392 L237.360912,8.92125547 L237.649576,8.92125547 C238.458111,12.4671812 239.420543,16.2915278 240.339083,19.8992811 L242.513607,28.4272429 L232.595264,28.4272429 Z M247.484672,47.9105377 L258.779301,47.9105377 L244.119095,0.866431332 L231.350876,0.866431332 L216.69067,47.9105377 L227.590685,47.9105377 L230.464345,36.7230418 L244.630472,36.7230418 L247.484672,47.9105377 Z M267.100055,0.866543136 L277.731949,0.866543136 L277.731949,38.9837777 L298.683287,38.9837777 L298.683287,47.9106495 L267.100055,47.9106495 L267.100055,0.866543136 Z"></path>
+            <path d="M15.9208113,103.897061 L17.0343816,99.5627576 L20.6864596,84.3899932 L20.975123,84.3899932 C21.7836584,87.9359189 22.7460902,91.7608169 23.6646304,95.3688852 L25.8391546,103.897061 L15.9208113,103.897061 Z M30.8058942,123.398726 L42.1048484,123.398726 L27.4392362,76.3351691 L14.6656122,76.3351691 L0,123.398726 L10.9043395,123.398726 L13.7801617,112.204747 L27.9506136,112.204747 L30.8058942,123.398726 Z M48.0762959,114.453272 L59.0682074,114.453272 L59.0682074,85.2629094 L48.0762959,85.2629094 L48.0762959,76.334957 L80.7103914,76.334957 L80.7103914,85.2629094 L69.7217233,85.2629094 L69.7217233,114.453272 L80.7103914,114.453272 L80.7103914,123.381225 L48.0762959,123.381225 L48.0762959,114.453272 Z M90.1825178,128.063257 L125.049168,128.063257 L125.049168,136.17178 L90.1825178,136.17178 L90.1825178,128.063257 Z"></path>
+        </g>
+        <g>
+            <rect fill="#000000" x="122.925214" y="0" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#F7D046" x="138.290866" y="0" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#000000" x="0" y="0" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#000000" x="0" y="30.7313035" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#000000" x="0" y="61.462607" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#000000" x="0" y="92.1939105" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#000000" x="0" y="122.925214" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#F7D046" x="15.3656518" y="0" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#F2A73B" x="138.290866" y="30.7313035" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#F2A73B" x="15.3656518" y="30.7313035" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#000000" x="92.1939105" y="30.7313035" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#F2A73B" x="107.559562" y="30.7313035" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#F2A73B" x="46.0969553" y="30.7313035" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#EE792F" x="76.8282588" y="61.462607" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#EE792F" x="107.559562" y="61.462607" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#EE792F" x="46.0969553" y="61.462607" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#000000" x="61.462607" y="92.1939105" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#EB5829" x="76.8282588" y="92.1939105" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#EE792F" x="138.290866" y="61.462607" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#EE792F" x="15.3656518" y="61.462607" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#000000" x="122.925214" y="92.1939105" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#EB5829" x="138.290866" y="92.1939105" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#000000" x="122.925214" y="122.925214" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#EB5829" x="15.3656518" y="92.1939105" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#EA3326" x="138.290866" y="122.925214" width="30.7313035" height="30.7313035"></rect>
+            <rect fill="#EA3326" x="15.3656518" y="122.925214" width="30.7313035" height="30.7313035"></rect>
+        </g>
+    </g>
+</svg>
diff --git a/docs/source/new-features/index.rst b/docs/source/new-features/index.rst
index 7eaa9dfd4..3420294d2 100644
--- a/docs/source/new-features/index.rst
+++ b/docs/source/new-features/index.rst
@@ -9,6 +9,25 @@ Find out about the details of major features, changes, and deprecations below.
 
     <section>
         <article class="new-content-grid">
+            <div class="grid-card">
+                <div class="content-container">
+                    <div class="header">
+                        MLflow Transformers Feature Enhancements
+                    </div>
+                    <img class="card-image" src="../_static/images/logos/huggingface-logo.svg" alt="HuggingFace transformers"></img>
+                    <div class="body">
+                        <p>The <a href="../llms/transformers/index.html">transformers flavor</a> 
+                        in MLflow has gotten a significant feature overhaul.</p>
+                        <ul>
+                            <li>All supported pipeline types can now be logged without restriction</li>
+                            <li>Pipelines using foundation models can now be logged without copying the large model weights</li>
+                        </ul>
+                    </div>
+                    <div class="tag">
+                        <a href="https://github.com/mlflow/mlflow/releases/tag/v2.11.0">released in 2.11.0</a>
+                    </div>
+                </div>
+            </div>
             <div class="grid-card">
                 <div class="content-container">
                     <div class="header">
@@ -16,7 +35,10 @@ Find out about the details of major features, changes, and deprecations below.
                     </div>
                     <img class="card-image" src="../_static/images/logos/huggingface-logo.svg" alt="HuggingFace Logo"></img>
                     <div class="body">
-                        MLflow now natively supports PEFT (Parameter-Efficient Fine-Tuning) models in Transformers flavor. PEFT unlocks significantly efficient model fine-tuning like LoRA, QLoRA, Prompt Tuning. Check it out to build your own cutting-edge models with MLflow and PEFT!
+                        MLflow now natively supports <a href="../llms/transformers/guide/index.html#peft-models-in-mlflow-transformers-flavor">PEFT (Parameter-Efficient Fine-Tuning)</a>
+                        models in the Transformers flavor. PEFT unlocks significantly more efficient model fine-tuning processes such as LoRA, QLoRA, and Prompt Tuning. Check out 
+                        <a href="../llms/transformers/tutorials/fine-tuning/transformers-peft.html">the new QLoRA fine-tuning tutorial</a> to learn how to 
+                        build your own cutting-edge models with MLflow and PEFT!
                     </div>
                     <div class="doc"><a class="icon bell" href="../llms/transformers/guide/index.html#peft-models-in-mlflow-transformers-flavor">Learn more</a></div>
                     <div class="tag">
@@ -46,6 +68,83 @@ Find out about the details of major features, changes, and deprecations below.
                     </div>
                 </div>
             </div>
+            <div class="grid-card">
+                <div class="content-container">
+                    <div class="header">
+                        Overhaul of MLflow Tracking UI for Deep Learning workflows
+                    </div>
+                    <img class="card-image" src="../_static/images/logos/mlflow-logo.svg" alt="MLflow"></img>
+                    <div class="body">
+                        We've listened to your feedback and have put in a huge amount of new UI features designed to empower and 
+                        simplify the process of evaluating DL model training runs. Be sure to upgrade your tracking server and 
+                        benefit from all of the new UI enhancements today!
+                    </div>
+                    <div class="tag">
+                        <a href="https://github.com/mlflow/mlflow/releases/tag/v2.11.0">released in 2.11.0</a>
+                    </div>
+                </div>
+            </div>
+            <div class="grid-card">
+                <div class="content-container">
+                    <div class="header">
+                        Automated model checkpointing for Deep Learning model training
+                    </div>
+                    <img class="card-image" src="../_static/images/logos/mlflow-logo.svg" alt="MLflow"></img>
+                    <div class="body">
+                        When performing training of Deep Learning models with <a href="../python_api/mlflow.pytorch.html#mlflow.pytorch.autolog">PyTorch Lightning</a> 
+                        or <a href="../python_api/mlflow.tensorflow.html#mlflow.tensorflow.autolog">Tensorflow with Keras</a>, model checkpoint saving 
+                        is enabled, allowing for state storage during long-running training events and the ability to resume if 
+                        an issue is encountered during training. 
+                    </div>
+                    <div class="tag">
+                        <a href="https://github.com/mlflow/mlflow/releases/tag/v2.11.0">released in 2.11.0</a>
+                    </div>
+                </div>
+            </div>
+            <div class="grid-card">
+                <div class="content-container">
+                    <div class="header">
+                        Mistral AI added as an MLflow Deployments Provider
+                    </div>
+                    <img class="card-image" src="../_static/images/logos/mistral-ai-logo.svg" alt="Mistral AI" style="max-height: 5rem"></img>
+                    <div class="body">
+                        The <a href="../llms/deployments/index.html">MLflow Deployments Server</a> can now 
+                        accept <a href="https://mistral.ai/">Mistral AI</a> endpoints. Give their models a try today! 
+                    </div>
+                    <div class="tag">
+                        <a href="https://github.com/mlflow/mlflow/releases/tag/v2.11.0">released in 2.11.0</a>
+                    </div>
+                </div>
+            </div>
+            <div class="grid-card">
+                <div class="content-container">
+                    <div class="header">
+                        Keras 3 is now supported in MLflow
+                    </div>
+                    <img class="card-image" src="../_static/images/logos/keras-logo.svg" alt="Keras"></img>
+                    <div class="body">
+                        You can now log and deploy models in the new <a href="https://keras.io/keras_3/">Keras 3 format</a>, allowing you 
+                        to work with TensorFlow, Torch, or JAX models with a new high-level, easy-to-use suite of APIs.
+                    </div>
+                    <div class="tag">
+                        <a href="https://github.com/mlflow/mlflow/releases/tag/v2.11.0">released in 2.11.0</a>
+                    </div>
+                </div>
+            </div>
+            <div class="grid-card">
+                <div class="content-container">
+                    <div class="header">
+                        MLflow now has support for OpenAI SDK 1.x
+                    </div>
+                    <img class="card-image" src="../_static/images/logos/openai-logo.svg" alt="OpenAI" style="max-height: 5rem"></img>
+                    <div class="body">
+                        We've updated flavors that interact with the OpenAI SDK, bringing full support for the API changes with the 1.x release.
+                    </div>
+                    <div class="tag">
+                        <a href="https://github.com/mlflow/mlflow/releases/tag/v2.11.0">released in 2.11.0</a>
+                    </div>
+                </div>
+            </div>
             <div class="grid-card">
                 <div class="content-container">
                     <div class="header">
diff --git a/mlflow/metrics/genai/model_utils.py b/mlflow/metrics/genai/model_utils.py
index 832735149..6ea4dce3b 100644
--- a/mlflow/metrics/genai/model_utils.py
+++ b/mlflow/metrics/genai/model_utils.py
@@ -114,11 +114,15 @@ def _call_openai_api(openai_uri, payload, eval_parameters):
 
 
 def _call_deployments_api(deployment_uri, payload, eval_parameters):
+    from pydantic import BaseModel
+
     from mlflow.deployments import get_deploy_client
 
     client = get_deploy_client()
 
     endpoint = client.get_endpoint(deployment_uri)
+    # TODO: Standardize the return type of `get_endpoint` and remove this check
+    endpoint = endpoint.dict() if isinstance(endpoint, BaseModel) else endpoint
     endpoint_type = endpoint.get("task", endpoint.get("endpoint_type"))
 
     if endpoint_type == "llm/v1/completions":
diff --git a/mlflow/pyfunc/model.py b/mlflow/pyfunc/model.py
index 726178469..d1de09486 100644
--- a/mlflow/pyfunc/model.py
+++ b/mlflow/pyfunc/model.py
@@ -284,8 +284,33 @@ def _save_model_with_class_artifacts_params(
     if callable(python_model):
         python_model = _FunctionPythonModel(python_model, hints, signature)
     saved_python_model_subpath = "python_model.pkl"
-    with open(os.path.join(path, saved_python_model_subpath), "wb") as out:
-        cloudpickle.dump(python_model, out)
+
+    try:
+        with open(os.path.join(path, saved_python_model_subpath), "wb") as out:
+            cloudpickle.dump(python_model, out)
+    except Exception as e:
+        # cloudpickle sometimes raises TypeError instead of PicklingError.
+        # catching generic Exception and checking message to handle both cases.
+        if "cannot pickle" in str(e).lower():
+            raise MlflowException(
+                "Failed to serialize Python model. Please audit your "
+                "class variables (e.g. in `__init__()`) for any "
+                "unpicklable objects. If you're trying to save an external model "
+                "in your custom pyfunc, Please use the `artifacts` parameter "
+                "in `mlflow.pyfunc.save_model()`, and load your external model "
+                "in the `load_context()` method instead. For example:\n\n"
+                "class MyModel(mlflow.pyfunc.PythonModel):\n"
+                "    def load_context(self, context):\n"
+                "        model_path = context.artifacts['my_model_path']\n"
+                "        // custom load logic here\n"
+                "        self.model = load_model(model_path)\n\n"
+                "For more information, see our full tutorial at: "
+                "https://mlflow.org/docs/latest/traditional-ml/creating-custom-pyfunc/index.html"
+                f"\n\nFull serialization error: {e}"
+            ) from None
+        else:
+            raise e
+
     custom_model_config_kwargs[CONFIG_KEY_PYTHON_MODEL] = saved_python_model_subpath
 
     if artifacts:
diff --git a/mlflow/tensorflow/__init__.py b/mlflow/tensorflow/__init__.py
index 06775851d..2719dc769 100644
--- a/mlflow/tensorflow/__init__.py
+++ b/mlflow/tensorflow/__init__.py
@@ -1089,6 +1089,24 @@ def autolog(
         log_every_epoch: If True, training metrics will be logged at the end of each epoch.
         log_every_n_steps: If set, training metrics will be logged every `n` training steps.
             `log_every_n_steps` must be `None` when `log_every_epoch=True`.
+        checkpoint: Enable automatic model checkpointing.
+        checkpoint_monitor: In automatic model checkpointing, the metric name to monitor if
+            you set `model_checkpoint_save_best_only` to True.
+        checkpoint_save_best_only: If True, automatic model checkpointing only saves when
+            the model is considered the "best" model according to the quantity
+            monitored and previous checkpoint model is overwritten.
+        checkpoint_mode: one of {"min", "max"}. In automatic model checkpointing,
+            if save_best_only=True, the decision to overwrite the current save file is made based on
+            either the maximization or the minimization of the monitored quantity.
+        checkpoint_save_weights_only: In automatic model checkpointing, if True, then
+            only the model’s weights will be saved. Otherwise, the optimizer states,
+            lr-scheduler states, etc are added in the checkpoint too.
+        checkpoint_save_freq: `"epoch"` or integer. When using `"epoch"`, the callback
+            saves the model after each epoch. When using integer, the callback
+            saves the model at end of this many batches. Note that if the saving isn't aligned to
+            epochs, the monitored metric may potentially be less reliable (it
+            could reflect as little as 1 batch, since the metrics get reset
+            every epoch). Defaults to `"epoch"`.
     """
     import tensorflow as tf
 


Test patch:
diff --git a/tests/pyfunc/test_pyfunc_exceptions.py b/tests/pyfunc/test_pyfunc_exceptions.py
new file mode 100644
index 000000000..ba4f9f9bf
--- /dev/null
+++ b/tests/pyfunc/test_pyfunc_exceptions.py
@@ -0,0 +1,19 @@
+import pytest
+
+import mlflow
+from mlflow.exceptions import MlflowException
+
+
+class UnpicklableModel(mlflow.pyfunc.PythonModel):
+    def __init__(self, path):
+        with open(path, "w+") as f:
+            pass
+
+        self.not_a_file = f
+
+
+def test_pyfunc_unpicklable_exception(tmp_path):
+    model = UnpicklableModel(tmp_path / "model.pkl")
+
+    with pytest.raises(MlflowException, match="Please audit your class variables"):
+        mlflow.pyfunc.save_model(python_model=model, path=tmp_path / "model")