Repository: mlflow/mlflow

Issue:
[BUG] XGBoost autologging breaks when metric with @ is used
Thank you for submitting an issue. Please refer to our [issue policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md) for additional information about bug reports. For help with debugging your code, please refer to [Stack Overflow](https://stackoverflow.com/questions/tagged/mlflow).

**Please fill in this bug report template to ensure a timely and thorough response.**

### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [x] Yes. I can contribute a fix for this bug independently.
- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [ ] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: yes
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS 11.6.3
- **MLflow installed from (source or binary)**: binary
- **MLflow version (run ``mlflow --version``)**: 1.23.1
- **Python version**: 3.7.9
- **npm version, if running the dev UI**:
- **Exact command to reproduce**: `python xgb_metrics_bug.py`

### Describe the problem
XGBoost comes with a few metrics with @ in their name, e.g. `ndcg@2`, see [XGBoost parameters](https://xgboost.readthedocs.io/en/stable/parameter.html#learning-task-parameters). 

Expected behavior: MLflow should be able to autolog all built-in XGBoost metrics.

Actual behavior: when using a metric like this, the following warning is shown:

> WARNING mlflow.utils.autologging_utils: Encountered unexpected error during xgboost autologging: Invalid metric name: 'train-ndcg@2'. Names may only contain alphanumerics, underscores (_), dashes (-), periods (.), spaces ( ), and slashes (/).

but not only this metric is discarded, instead autologging completely breaks, i.e. other metrics, the model artifact and the feature importance plot also are not recorded.

![image](https://user-images.githubusercontent.com/1640908/154653567-1de2e15b-9ad7-4f8b-a9fe-5a7808602916.png)

I would propose to string-replace `@` with `_at_` in the XGBoost autologging metrics callback so that MLflow is compatible with all XGBoost built-in metrics.

### Code to reproduce issue

`xgb_metrics_bug.py`:

```python
import mlflow
import numpy as np
import xgboost as xgb

mlflow.set_experiment("xgb_autolog_metrics_bug")

for metrics in [["ndcg"], ["ndcg", "ndcg@2"]]:
    with mlflow.start_run():
        mlflow.xgboost.autolog()

        train_matrix = xgb.DMatrix(np.array([[0], [1]]), label=[1, 0])

        params = {
            "objective": "rank:pairwise",
            "eval_metric": metrics,
        }
        xgb.train(params, train_matrix, evals=[(train_matrix, "train")])
```


### Other info / logs
- I tried this with xgboost versions 1.3.3 and 1.5.1


### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [x] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations



Discussion:
Hi @maxfriedrich thank you for the idea! 
Please feel free to submit a PR that patches the metric names from XGBoost's core metrics library within the callback logic in autologging (https://github.com/mlflow/mlflow/blob/b81a1d3ca377ff8ac47d33c9e6111172a31bd145/mlflow/xgboost/_autolog.py#L42-L46). 
Please let us know if you have any questions and we'll be sure to review the PR when you file it! Thank you again.

Resolving patch:
diff --git a/mlflow/xgboost/_autolog.py b/mlflow/xgboost/_autolog.py
--- a/mlflow/xgboost/_autolog.py
+++ b/mlflow/xgboost/_autolog.py
@@ -6,9 +6,15 @@
 from mlflow.utils.autologging_utils import ExceptionSafeAbstractClass
 
 
+def _patch_metric_names(metric_dict):
+    # XGBoost provides some metrics with "@", e.g. "ndcg@3" that are not valid MLflow metric names
+    return {metric_name.replace("@", "_at_"): value for metric_name, value in metric_dict.items()}
+
+
 def autolog_callback(env, metrics_logger, eval_results):
-    metrics_logger.record_metrics(dict(env.evaluation_result_list), env.iteration)
-    eval_results.append(dict(env.evaluation_result_list))
+    metric_dict = _patch_metric_names(dict(env.evaluation_result_list))
+    metrics_logger.record_metrics(metric_dict, env.iteration)
+    eval_results.append(metric_dict)
 
 
 IS_TRAINING_CALLBACK_SUPPORTED = Version(xgboost.__version__.replace("SNAPSHOT", "dev")) >= Version(
@@ -41,10 +47,11 @@ def after_iteration(self, model, epoch, evals_log):
             # }
             evaluation_result_dict = {}
             for data_name, metric_dict in evals_log.items():
+                metric_dict = _patch_metric_names(metric_dict)
                 for metric_name, metric_values_on_each_iter in metric_dict.items():
                     key = "{}-{}".format(data_name, metric_name)
                     # The last element in `metric_values_on_each_iter` corresponds to
-                    # the meric on the current iteration
+                    # the metric on the current iteration
                     evaluation_result_dict[key] = metric_values_on_each_iter[-1]
 
             self.metrics_logger.record_metrics(evaluation_result_dict, epoch)


Test patch:
diff --git a/tests/xgboost/test_xgboost_autolog.py b/tests/xgboost/test_xgboost_autolog.py
--- a/tests/xgboost/test_xgboost_autolog.py
+++ b/tests/xgboost/test_xgboost_autolog.py
@@ -141,6 +141,19 @@ def test_xgb_autolog_logs_specified_params(bst_params, dtrain):
         assert param not in params
 
 
+@pytest.mark.large
+def test_xgb_autolog_atsign_metrics():
+    mlflow.xgboost.autolog()
+    xgb_metrics = ["ndcg@2", "map@3-", "error@0.4"]
+    expected_metrics = {"train-ndcg_at_2", "train-map_at_3-", "train-error_at_0.4"}
+
+    params = {"objective": "rank:pairwise", "eval_metric": xgb_metrics}
+    dtrain = xgb.DMatrix(np.array([[0], [1]]), label=[1, 0])
+    xgb.train(params, dtrain, evals=[(dtrain, "train")], num_boost_round=1)
+    run = get_latest_run()
+    assert set(run.data.metrics) == expected_metrics
+
+
 @pytest.mark.large
 def test_xgb_autolog_sklearn():