mlflow can log a maximum of 100 parameters on Azure ML
Author: nbroad1881
Pull request: no

When trying to use the Trainer's mlflow integration within Azure ML, it will fail at the `on_train_begin` callback because it tries to log all of the TrainingArguments and the model config which will total more than 100 parameters. 

> RestException: INVALID_PARAMETER_VALUE: Response: {'Error': {'Code': 'ValidationError', 'Severity': None, 'Message': 'A field of the entity is over the size limit. FieldName=Parameters, Limit=100, Size=175. See https://aka.ms/azure-machine-learning-limits for service limits documentation.', 'MessageFormat': None, 'MessageParameters': None, 'ReferenceCode': None, 'DetailsUri': None, 'Target': None, 'Details': [], 'InnerError': None, 'DebugInfo': None, 'AdditionalInfo': None}, 'Correlation': {'operation': '9816ae760b843120b907ea5121aeb911', 'request': '4ac415478d4966af'}, 'Environment': 'eastus2', 'Location': 'eastus2', 'Time': '2022-09-02T14:33:48.3242911+00:00', 'ComponentName': 'mlflow', 'error_code': 'INVALID_PARAMETER_VALUE'}

Doing this outside of Azure ML does not produce this error. 

I know there is a separate AzureML callback, but I believe this is for the older version of Azure ML and the newer version just uses mlflow. I could not get it to work using only the Azure ML callback.

There are many values in TrainingArguments that are not super important and are typically never set, so the easy workaround is to limit the number of TrainingArguments that get logged. After going through all arguments, I identified 40 or so that are the most important for logging. The rest of the arguments can still be saved to the output directory by saving all arguments as a json file. 

If you want to see this error for yourself, run the following code inside Azure ML:

```python
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential
import mlflow

from transformers import AutoModel, TrainingArguments

ml_client = MLClient.from_config(credential=DefaultAzureCredential())
azureml_mlflow_uri = ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri
mlflow.set_tracking_uri(azureml_mlflow_uri)

with mlflow.start_run():
    targs  = TrainingArguments("output")
    model = AutoModel.from_pretrained("albert-base-v2")
    mlflow.log_params(targs.to_dict()) 
    mlflow.log_params(model.config.to_dict())
```


Questions:
1. Should a change be made to the mlflow callback?
2. Alternatively, should Azure ML + mlflow users not use the mlflow callback and instead do their own custom logging?
3. Should an updated Azure ML callback be created?

@sgugger 

Comments:
Note that we are not maintaining the external logging platforms ourselves, their creators are supposed to do this work. Since there is no one in Microsoft that is pushing this forward, some community contributed callbacks have been created, but it's not guaranteed that they work. I would recommend using other trackers that are better maintained until some folks at MlFlow/Azure put the work for a nice integration :-)

I'll just make my own callback. Thanks!

Hi nbroad1881, 

I am facing the same problem that you have mentioned at the beginning, however, I am not sure about the workaround and its bit urgent. Can you please help me how were you able to have your own callback? This will be of great help.

Regards,
Dev

@dks198 Something you might be able to do for a custom AzureMLCallback once you've defined your trainers:

from transformers import TrainerCallback
import importlib.util

------------------------------------------------

def is_azureml_available():
    if importlib.util.find_spec("azureml") is None:
        return False
    if importlib.util.find_spec("azureml.core") is None:
        return False
    return importlib.util.find_spec("azureml.core.run") is not None

class AzureMLCallback(TrainerCallback):
    """
    A [`TrainerCallback`] that sends the logs to [AzureML](https://pypi.org/project/azureml-sdk/).
    """

    def __init__(self, azureml_run=None):
        if not is_azureml_available():
            raise RuntimeError("AzureMLCallback requires azureml to be installed. Run `pip install azureml-sdk`.")
        self.azureml_run = azureml_run

    def on_init_end(self, args, state, control, **kwargs):
        from azureml.core.run import Run

        if self.azureml_run is None and state.is_world_process_zero:
            self.azureml_run = Run.get_context()

    def on_log(self, args, state, control, logs=None, **kwargs):
        if self.azureml_run and state.is_world_process_zero:
            for k, v in logs.items():
                if isinstance(v, (int, float)):
                    self.azureml_run.log(k, v, description=k)

-----------------------------------------------

trainer.add_callback(AzureMLCallback)

trainer.train()

I believe AzureML logging is being deprecated for mlflow. 

@dks198, you can use the [answer](https://github.com/huggingface/accelerate/pull/675#discussion_r974482489) I gave in the other thread. You can just limit the number of parameters logged.

Hi,

I tried implementing the code you have shared for my Bert model training
however, I still get the same error message. I am trying to figure out how
to implement this so that I can get through this error. I have attached the
error message for your reference.

Regards,
Dev

On Tue, Sep 20, 2022 at 1:52 PM giozinzi ***@***.***> wrote:

> @dks198 <https://github.com/dks198> Something you might be able to do for
> a custom AzureMLCallback once you've defined your trainers:
>
> from transformers import TrainerCallback
> import importlib.util
> ------------------------------
>
> def is_azureml_available():
> if importlib.util.find_spec("azureml") is None:
> return False
> if importlib.util.find_spec("azureml.core") is None:
> return False
> return importlib.util.find_spec("azureml.core.run") is not None
>
> class AzureMLCallback(TrainerCallback):
> """
> A [TrainerCallback] that sends the logs to AzureML
> <https://pypi.org/project/azureml-sdk/>.
> """
>
> def __init__(self, azureml_run=None):
>     if not is_azureml_available():
>         raise RuntimeError("AzureMLCallback requires azureml to be installed. Run `pip install azureml-sdk`.")
>     self.azureml_run = azureml_run
>
> def on_init_end(self, args, state, control, **kwargs):
>     from azureml.core.run import Run
>
>     if self.azureml_run is None and state.is_world_process_zero:
>         self.azureml_run = Run.get_context()
>
> def on_log(self, args, state, control, logs=None, **kwargs):
>     if self.azureml_run and state.is_world_process_zero:
>         for k, v in logs.items():
>             if isinstance(v, (int, float)):
>                 self.azureml_run.log(k, v, description=k)
>
> ------------------------------
>
> trainer.add_callback(AzureMLCallback)
>
> trainer.train()
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/18870#issuecomment-1252898595>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A3FF72LYYCFYTOIPM2LYXS3V7IPW7ANCNFSM6AAAAAAQDK7Y2I>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


AzureML recently raised the limit to the number of parameters that can be logged per mlflow run to 200. This should unblock using HF autolog in the issue raised initially. That change has been rolled out earlier in October. By now, you should be able to drop the workaround and just use HF autolog with mlflow in AzureML.