TypeError: cannot pickle 'module' object
Author: pathikg
Pull request: no

### System Info

transformers: 4.30.0
platform: Ubuntu 20.04.6 LTS x86_64
python: 3.8.5


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hello team, 
I was following this [tutorial](https://huggingface.co/docs/transformers/tasks/object_detection) on huggingface for object detection using DETR model on my custom dataset that has same dataset structure as `cpp5` (one used in tutorial)

I've used slightly different training arguments:
```
from transformers import TrainingArguments
from transformers.integrations import MLflowCallback, AzureMLCallback

training_args = TrainingArguments(
    output_dir="detr-resnet-50_finetuned_loss-run",
    per_device_train_batch_size=4,
    num_train_epochs=30,
    fp16=True,
    save_steps=200,
    logging_steps=50,
    learning_rate=1e-5,
    weight_decay=1e-4,
    save_total_limit=2,
    remove_unused_columns=False,
    push_to_hub=False,
    # dataloader_num_workers=4,  # Adjust the number of dataloader workers according to your system
    logging_dir="logs",
    report_to="mlflow",  # Report metrics to MLflow
    # load_best_model_at_end=True,
    metric_for_best_model="loss",
    greater_is_better=False,
)

# Create the MLflow callback
mlflow_callback = MLflowCallback()
azureml_callback = AzureMLCallback()

# Integrate the MLflow callback in TrainingArguments
training_args.callbacks = [mlflow_callback, azureml_callback]

from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=collate_fn,
    train_dataset=dataset["train"],
    eval_dataset=dataset["valid"],
    tokenizer=image_processor,
)

trainer.train()
```
but I am receiving at 200th step
```
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:12                                                                                   │
│                                                                                                  │
│    9 │   tokenizer=image_processor,                                                              │
│   10 )                                                                                           │
│   11                                                                                             │
│ ❱ 12 trainer.train()                                                                             │
│   13                                                                                             │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer.py:1645 in train    │
│                                                                                                  │
│   1642 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1643 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1644 │   │   )                                                                                 │
│ ❱ 1645 │   │   return inner_training_loop(                                                       │
│   1646 │   │   │   args=args,                                                                    │
│   1647 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1648 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer.py:2011 in          │
│ _inner_training_loop                                                                             │
│                                                                                                  │
│   2008 │   │   │   │   │   self.state.epoch = epoch + (step + 1 + steps_skipped) / steps_in_epo  │
│   2009 │   │   │   │   │   self.control = self.callback_handler.on_step_end(args, self.state, s  │
│   2010 │   │   │   │   │                                                                         │
│ ❱ 2011 │   │   │   │   │   self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_k  │
│   2012 │   │   │   │   else:                                                                     │
│   2013 │   │   │   │   │   self.control = self.callback_handler.on_substep_end(args, self.state  │
│   2014                                                                                           │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer.py:2323 in          │
│ _maybe_log_save_evaluate                                                                         │
│                                                                                                  │
│   2320 │   │   │   │   self.lr_scheduler.step(metrics[metric_to_check])                          │
│   2321 │   │                                                                                     │
│   2322 │   │   if self.control.should_save:                                                      │
│ ❱ 2323 │   │   │   self._save_checkpoint(model, trial, metrics=metrics)                          │
│   2324 │   │   │   self.control = self.callback_handler.on_save(self.args, self.state, self.con  │
│   2325 │                                                                                         │
│   2326 │   def _load_rng_state(self, checkpoint):                                                │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer.py:2380 in          │
│ _save_checkpoint                                                                                 │
│                                                                                                  │
│   2377 │   │                                                                                     │
│   2378 │   │   run_dir = self._get_output_dir(trial=trial)                                       │
│   2379 │   │   output_dir = os.path.join(run_dir, checkpoint_folder)                             │
│ ❱ 2380 │   │   self.save_model(output_dir, _internal_call=True)                                  │
│   2381 │   │   if self.is_deepspeed_enabled:                                                     │
│   2382 │   │   │   # under zero3 model file itself doesn't get saved since it's bogus! Unless d  │
│   2383 │   │   │   # config `stage3_gather_16bit_weights_on_model_save` is True                  │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer.py:2878 in          │
│ save_model                                                                                       │
│                                                                                                  │
│   2875 │   │   │   │   │   self.model_wrapped.save_checkpoint(output_dir)                        │
│   2876 │   │                                                                                     │
│   2877 │   │   elif self.args.should_save:                                                       │
│ ❱ 2878 │   │   │   self._save(output_dir)                                                        │
│   2879 │   │                                                                                     │
│   2880 │   │   # Push to the Hub when `save_model` is called by the user.                        │
│   2881 │   │   if self.args.push_to_hub and not _internal_call:                                  │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer.py:2944 in _save    │
│                                                                                                  │
│   2941 │   │   │   self.tokenizer.save_pretrained(output_dir)                                    │
│   2942 │   │                                                                                     │
│   2943 │   │   # Good practice: save your training arguments together with the trained model     │
│ ❱ 2944 │   │   torch.save(self.args, os.path.join(output_dir, TRAINING_ARGS_NAME))               │
│   2945 │                                                                                         │
│   2946 │   def store_flos(self):                                                                 │
│   2947 │   │   # Storing the number of floating-point operations that went into the model        │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/torch/serialization.py:441 in save       │
│                                                                                                  │
│    438 │                                                                                         │
│    439 │   if _use_new_zipfile_serialization:                                                    │
│    440 │   │   with _open_zipfile_writer(f) as opened_zipfile:                                   │
│ ❱  441 │   │   │   _save(obj, opened_zipfile, pickle_module, pickle_protocol)                    │
│    442 │   │   │   return                                                                        │
│    443 │   else:                                                                                 │
│    444 │   │   with _open_file_like(f, 'wb') as opened_file:                                     │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/torch/serialization.py:653 in _save      │
│                                                                                                  │
│    650 │   data_buf = io.BytesIO()                                                               │
│    651 │   pickler = pickle_module.Pickler(data_buf, protocol=pickle_protocol)                   │
│    652 │   pickler.persistent_id = persistent_id                                                 │
│ ❱  653 │   pickler.dump(obj)                                                                     │
│    654 │   data_value = data_buf.getvalue()                                                      │
│    655 │   zip_file.write_record('data.pkl', data_value, len(data_value))                        │
│    656                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: cannot pickle 'module' object
```


### Expected behavior

Artifact should've been saved at 200th step

Comments:
cc @sgugger 

It is possible that either of those callbacks (MlFlow or Azure) is inserting something in the state that cannot be serialized with `pickle`. We do not maintain those integrations ourselves, so I suggest you ping the author of the callback making your code fail (after trying to remove one or the other) :-)

Thanks @sgugger 
I commented `MLFlowCallBack()` added by @noise-field in [#8016](https://github.com/huggingface/transformers/pull/8016)
and the code worked fine till 350 steps but I received a new error at the end due to `AzureMLCallback()` added by @davidefiocco in [#8062](https://github.com/huggingface/transformers/pull/8062#issue-729809630)

```
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:12                                                                                   │
│                                                                                                  │
│    9 │   tokenizer=image_processor,                                                              │
│   10 )                                                                                           │
│   11                                                                                             │
│ ❱ 12 trainer.train()                                                                             │
│   13                                                                                             │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer.py:1645 in train    │
│                                                                                                  │
│   1642 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1643 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1644 │   │   )                                                                                 │
│ ❱ 1645 │   │   return inner_training_loop(                                                       │
│   1646 │   │   │   args=args,                                                                    │
│   1647 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1648 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer.py:2011 in          │
│ _inner_training_loop                                                                             │
│                                                                                                  │
│   2008 │   │   │   │   │   self.state.epoch = epoch + (step + 1 + steps_skipped) / steps_in_epo  │
│   2009 │   │   │   │   │   self.control = self.callback_handler.on_step_end(args, self.state, s  │
│   2010 │   │   │   │   │                                                                         │
│ ❱ 2011 │   │   │   │   │   self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_k  │
│   2012 │   │   │   │   else:                                                                     │
│   2013 │   │   │   │   │   self.control = self.callback_handler.on_substep_end(args, self.state  │
│   2014                                                                                           │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer.py:2324 in          │
│ _maybe_log_save_evaluate                                                                         │
│                                                                                                  │
│   2321 │   │                                                                                     │
│   2322 │   │   if self.control.should_save:                                                      │
│   2323 │   │   │   self._save_checkpoint(model, trial, metrics=metrics)                          │
│ ❱ 2324 │   │   │   self.control = self.callback_handler.on_save(self.args, self.state, self.con  │
│   2325 │                                                                                         │
│   2326 │   def _load_rng_state(self, checkpoint):                                                │
│   2327 │   │   # Load RNG states from `checkpoint`                                               │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer_callback.py:386 in  │
│ on_save                                                                                          │
│                                                                                                  │
│   383 │                                                                                          │
│   384 │   def on_save(self, args: TrainingArguments, state: TrainerState, control: TrainerCont   │
│   385 │   │   control.should_save = False                                                        │
│ ❱ 386 │   │   return self.call_event("on_save", args, state, control)                            │
│   387 │                                                                                          │
│   388 │   def on_log(self, args: TrainingArguments, state: TrainerState, control: TrainerContr   │
│   389 │   │   control.should_log = False                                                         │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/trainer_callback.py:397 in  │
│ call_event                                                                                       │
│                                                                                                  │
│   394 │                                                                                          │
│   395 │   def call_event(self, event, args, state, control, **kwargs):                           │
│   396 │   │   for callback in self.callbacks:                                                    │
│ ❱ 397 │   │   │   result = getattr(callback, event)(                                             │
│   398 │   │   │   │   args,                                                                      │
│   399 │   │   │   │   state,                                                                     │
│   400 │   │   │   │   control,                                                                   │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/integrations.py:1055 in     │
│ on_save                                                                                          │
│                                                                                                  │
│   1052 │   │   │   ckpt_dir = f"checkpoint-{state.global_step}"                                  │
│   1053 │   │   │   artifact_path = os.path.join(args.output_dir, ckpt_dir)                       │
│   1054 │   │   │   logger.info(f"Logging checkpoint artifacts in {ckpt_dir}. This may take time  │
│ ❱ 1055 │   │   │   self._ml_flow.pyfunc.log_model(                                               │
│   1056 │   │   │   │   ckpt_dir,                                                                 │
│   1057 │   │   │   │   artifacts={"model_path": artifact_path},                                  │
│   1058 │   │   │   │   python_model=self._ml_flow.pyfunc.PythonModel(),                          │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/mlflow/pyfunc/__init__.py:1578 in        │
│ log_model                                                                                        │
│                                                                                                  │
│   1575 │   :return: A :py:class:`ModelInfo <mlflow.models.model.ModelInfo>` instance that conta  │
│   1576 │   │   │    metadata of the logged model.                                                │
│   1577 │   """                                                                                   │
│ ❱ 1578 │   return Model.log(                                                                     │
│   1579 │   │   artifact_path=artifact_path,                                                      │
│   1580 │   │   flavor=mlflow.pyfunc,                                                             │
│   1581 │   │   loader_module=loader_module,                                                      │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/mlflow/models/model.py:487 in log        │
│                                                                                                  │
│   484 │   │   │   run_id = mlflow.tracking.fluent._get_or_start_run().info.run_id                │
│   485 │   │   │   mlflow_model = cls(artifact_path=artifact_path, run_id=run_id, metadata=meta   │
│   486 │   │   │   flavor.save_model(path=local_path, mlflow_model=mlflow_model, **kwargs)        │
│ ❱ 487 │   │   │   mlflow.tracking.fluent.log_artifacts(local_path, mlflow_model.artifact_path)   │
│   488 │   │   │   try:                                                                           │
│   489 │   │   │   │   mlflow.tracking.fluent._record_logged_model(mlflow_model)                  │
│   490 │   │   │   except MlflowException:                                                        │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/mlflow/tracking/fluent.py:810 in         │
│ log_artifacts                                                                                    │
│                                                                                                  │
│    807 │   │   │   mlflow.log_artifacts("data", artifact_path="states")                          │
│    808 │   """                                                                                   │
│    809 │   run_id = _get_or_start_run().info.run_id                                              │
│ ❱  810 │   MlflowClient().log_artifacts(run_id, local_dir, artifact_path)                        │
│    811                                                                                           │
│    812                                                                                           │
│    813 def log_text(text: str, artifact_file: str) -> None:                                      │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/mlflow/tracking/client.py:1048 in        │
│ log_artifacts                                                                                    │
│                                                                                                  │
│   1045 │   │   │   artifact: states                                                              │
│   1046 │   │   │   is_dir: True                                                                  │
│   1047 │   │   """                                                                               │
│ ❱ 1048 │   │   self._tracking_client.log_artifacts(run_id, local_dir, artifact_path)             │
│   1049 │                                                                                         │
│   1050 │   @contextlib.contextmanager                                                            │
│   1051 │   def _log_artifact_helper(self, run_id, artifact_file):                                │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client │
│ .py:448 in log_artifacts                                                                         │
│                                                                                                  │
│   445 │   │   :param local_dir: Path to the directory of files to write.                         │
│   446 │   │   :param artifact_path: If provided, the directory in ``artifact_uri`` to write to   │
│   447 │   │   """                                                                                │
│ ❱ 448 │   │   self._get_artifact_repo(run_id).log_artifacts(local_dir, artifact_path)            │
│   449 │                                                                                          │
│   450 │   def list_artifacts(self, run_id, path=None):                                           │
│   451 │   │   """                                                                                │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/mlflow/_store/artifact/artifact_ │
│ repo.py:88 in log_artifacts                                                                      │
│                                                                                                  │
│    85 │   │   if artifact_path is None:                                                          │
│    86 │   │   │   dest_path = ""                                                                 │
│    87 │   │                                                                                      │
│ ❱  88 │   │   self.artifacts.upload_dir(local_dir, dest_path)                                    │
│    89 │                                                                                          │
│    90 │   def list_artifacts(self, path):                                                        │
│    91 │   │   """                                                                                │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/mlflow/_client/artifact/run_arti │
│ fact_client.py:90 in upload_dir                                                                  │
│                                                                                                  │
│    87 │   │   │   │   local_paths.append(local_file_path)                                        │
│    88 │   │                                                                                      │
│    89 │   │   # Make batch request to create empty artifacts                                     │
│ ❱  90 │   │   empty_artifact_res = self._create_empty_artifacts(paths=remote_paths, run_id=sel   │
│    91 │   │                                                                                      │
│    92 │   │   result = self._upload_files(                                                       │
│    93 │   │   │   local_paths=local_paths, remote_paths=remote_paths, empty_artifact_content=e   │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/mlflow/_client/artifact/run_arti │
│ fact_client.py:146 in _create_empty_artifacts                                                    │
│                                                                                                  │
│   143 │   │                                                                                      │
│   144 │   │   artifacts = [ArtifactPath(path=path) for path in paths]                            │
│   145 │   │                                                                                      │
│ ❱ 146 │   │   response = self._client.run_artifacts.batch_create_empty_artifacts(                │
│   147 │   │   │   subscription_id=self._service_context.subscription_id,                         │
│   148 │   │   │   resource_group_name=self._service_context.resource_group_name,                 │
│   149 │   │   │   workspace_name=self._service_context.workspace_name,                           │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/mlflow/_restclient/run_artifact/ │
│ operations/_run_artifacts_operations.py:1116 in batch_create_empty_artifacts                     │
│                                                                                                  │
│   1113 │   │   │   body_content = None                                                           │
│   1114 │   │   body_content_kwargs['content'] = body_content                                     │
│   1115 │   │   request = self._client.post(url, query_parameters, header_parameters, **body_con  │
│ ❱ 1116 │   │   pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)   │
│   1117 │   │   response = pipeline_response.http_response                                        │
│   1118 │   │                                                                                     │
│   1119 │   │   if response.status_code not in [200]:                                             │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:205 in run  │
│                                                                                                  │
│   202 │   │   │   if self._impl_policies                                                         │
│   203 │   │   │   else _TransportRunner(self._transport)                                         │
│   204 │   │   )                                                                                  │
│ ❱ 205 │   │   return first_node.send(pipeline_request)  # type: ignore                           │
│   206                                                                                            │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:69 in send  │
│                                                                                                  │
│    66 │   │   """                                                                                │
│    67 │   │   _await_result(self._policy.on_request, request)                                    │
│    68 │   │   try:                                                                               │
│ ❱  69 │   │   │   response = self.next.send(request)                                             │
│    70 │   │   except Exception:  # pylint: disable=broad-except                                  │
│    71 │   │   │   _await_result(self._policy.on_exception, request)                              │
│    72 │   │   │   raise                                                                          │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:69 in send  │
│                                                                                                  │
│    66 │   │   """                                                                                │
│    67 │   │   _await_result(self._policy.on_request, request)                                    │
│    68 │   │   try:                                                                               │
│ ❱  69 │   │   │   response = self.next.send(request)                                             │
│    70 │   │   except Exception:  # pylint: disable=broad-except                                  │
│    71 │   │   │   _await_result(self._policy.on_exception, request)                              │
│    72 │   │   │   raise                                                                          │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:69 in send  │
│                                                                                                  │
│    66 │   │   """                                                                                │
│    67 │   │   _await_result(self._policy.on_request, request)                                    │
│    68 │   │   try:                                                                               │
│ ❱  69 │   │   │   response = self.next.send(request)                                             │
│    70 │   │   except Exception:  # pylint: disable=broad-except                                  │
│    71 │   │   │   _await_result(self._policy.on_exception, request)                              │
│    72 │   │   │   raise                                                                          │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:69 in send  │
│                                                                                                  │
│    66 │   │   """                                                                                │
│    67 │   │   _await_result(self._policy.on_request, request)                                    │
│    68 │   │   try:                                                                               │
│ ❱  69 │   │   │   response = self.next.send(request)                                             │
│    70 │   │   except Exception:  # pylint: disable=broad-except                                  │
│    71 │   │   │   _await_result(self._policy.on_exception, request)                              │
│    72 │   │   │   raise                                                                          │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:69 in send  │
│                                                                                                  │
│    66 │   │   """                                                                                │
│    67 │   │   _await_result(self._policy.on_request, request)                                    │
│    68 │   │   try:                                                                               │
│ ❱  69 │   │   │   response = self.next.send(request)                                             │
│    70 │   │   except Exception:  # pylint: disable=broad-except                                  │
│    71 │   │   │   _await_result(self._policy.on_exception, request)                              │
│    72 │   │   │   raise                                                                          │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/mgmt/core/policies/_base.py:47 in  │
│ send                                                                                             │
│                                                                                                  │
│    44 │   def send(self, request):                                                               │
│    45 │   │   # type: (PipelineRequest[HTTPRequestType], Any) -> PipelineResponse[HTTPRequestT   │
│    46 │   │   http_request = request.http_request                                                │
│ ❱  47 │   │   response = self.next.send(request)                                                 │
│    48 │   │   if response.http_response.status_code == 409:                                      │
│    49 │   │   │   rp_name = self._check_rp_not_registered_err(response)                          │
│    50 │   │   │   if rp_name:                                                                    │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/policies/_redirect.p │
│ y:160 in send                                                                                    │
│                                                                                                  │
│   157 │   │   retryable = True                                                                   │
│   158 │   │   redirect_settings = self.configure_redirects(request.context.options)              │
│   159 │   │   while retryable:                                                                   │
│ ❱ 160 │   │   │   response = self.next.send(request)                                             │
│   161 │   │   │   redirect_location = self.get_redirect_location(response)                       │
│   162 │   │   │   if redirect_location and redirect_settings["allow"]:                           │
│   163 │   │   │   │   retryable = self.increment(                                                │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/policies/_retry.py:5 │
│ 02 in send                                                                                       │
│                                                                                                  │
│   499 │   │   │   │   │   │   else:                                                              │
│   500 │   │   │   │   │   │   │   is_response_error = True                                       │
│   501 │   │   │   │   │   │   continue                                                           │
│ ❱ 502 │   │   │   │   raise err                                                                  │
│   503 │   │   │   finally:                                                                       │
│   504 │   │   │   │   end_time = time.time()                                                     │
│   505 │   │   │   │   if absolute_timeout:                                                       │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/policies/_retry.py:4 │
│ 74 in send                                                                                       │
│                                                                                                  │
│   471 │   │   │   try:                                                                           │
│   472 │   │   │   │   start_time = time.time()                                                   │
│   473 │   │   │   │   self._configure_timeout(request, absolute_timeout, is_response_error)      │
│ ❱ 474 │   │   │   │   response = self.next.send(request)                                         │
│   475 │   │   │   │   if self.is_retry(retry_settings, response):                                │
│   476 │   │   │   │   │   retry_active = self.increment(retry_settings, response=response)       │
│   477 │   │   │   │   │   if retry_active:                                                       │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentica │
│ tion.py:117 in send                                                                              │
│                                                                                                  │
│   114 │   │   """                                                                                │
│   115 │   │   self.on_request(request)                                                           │
│   116 │   │   try:                                                                               │
│ ❱ 117 │   │   │   response = self.next.send(request)                                             │
│   118 │   │   │   self.on_response(request, response)                                            │
│   119 │   │   except Exception:  # pylint:disable=broad-except                                   │
│   120 │   │   │   self.on_exception(request)                                                     │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:69 in send  │
│                                                                                                  │
│    66 │   │   """                                                                                │
│    67 │   │   _await_result(self._policy.on_request, request)                                    │
│    68 │   │   try:                                                                               │
│ ❱  69 │   │   │   response = self.next.send(request)                                             │
│    70 │   │   except Exception:  # pylint: disable=broad-except                                  │
│    71 │   │   │   _await_result(self._policy.on_exception, request)                              │
│    72 │   │   │   raise                                                                          │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:69 in send  │
│                                                                                                  │
│    66 │   │   """                                                                                │
│    67 │   │   _await_result(self._policy.on_request, request)                                    │
│    68 │   │   try:                                                                               │
│ ❱  69 │   │   │   response = self.next.send(request)                                             │
│    70 │   │   except Exception:  # pylint: disable=broad-except                                  │
│    71 │   │   │   _await_result(self._policy.on_exception, request)                              │
│    72 │   │   │   raise                                                                          │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:69 in send  │
│                                                                                                  │
│    66 │   │   """                                                                                │
│    67 │   │   _await_result(self._policy.on_request, request)                                    │
│    68 │   │   try:                                                                               │
│ ❱  69 │   │   │   response = self.next.send(request)                                             │
│    70 │   │   except Exception:  # pylint: disable=broad-except                                  │
│    71 │   │   │   _await_result(self._policy.on_exception, request)                              │
│    72 │   │   │   raise                                                                          │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:69 in send  │
│                                                                                                  │
│    66 │   │   """                                                                                │
│    67 │   │   _await_result(self._policy.on_request, request)                                    │
│    68 │   │   try:                                                                               │
│ ❱  69 │   │   │   response = self.next.send(request)                                             │
│    70 │   │   except Exception:  # pylint: disable=broad-except                                  │
│    71 │   │   │   _await_result(self._policy.on_exception, request)                              │
│    72 │   │   │   raise                                                                          │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/_base.py:100 in send │
│                                                                                                  │
│    97 │   │   """                                                                                │
│    98 │   │   return PipelineResponse(                                                           │
│    99 │   │   │   request.http_request,                                                          │
│ ❱ 100 │   │   │   self._sender.send(request.http_request, **request.context.options),            │
│   101 │   │   │   context=request.context,                                                       │
│   102 │   │   )                                                                                  │
│   103                                                                                            │
│                                                                                                  │
│ /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_ │
│ basic.py:376 in send                                                                             │
│                                                                                                  │
│   373 │   │   │   error = ServiceRequestError(err, error=err)                                    │
│   374 │   │                                                                                      │
│   375 │   │   if error:                                                                          │
│ ❱ 376 │   │   │   raise error                                                                    │
│   377 │   │   if _is_rest(request):                                                              │
│   378 │   │   │   from azure.core.rest._requests_basic import RestRequestsTransportResponse      │
│   379                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ServiceResponseError: HTTPSConnectionPool(host='centralindia.api.azureml.ms', port=443): Read timed out. (read 
timeout=300)
```

I think this has something to do with the azureml not registering the request in given time. In past, I had faced a similar issue in yolov5 while logging these artifacts to azureml I used to get this similar error so I had added another a retry loop with some wait time and it worked
So please let me know if you found any resolution for the same @davidefiocco :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.