mlflow parameter overflow when training a language adapter 
Author: JackyXiangcheng
Pull request: no

When I train a german adapter, I met a problem, which has been talked a lot in the issues, but there are not very clear solutions.



Traceback (most recent call last):
  File "/mnt/localdata/cao/run_de_modeling.py", line 512, in <module>
    main()
  File "/mnt/localdata/cao/run_de_modeling.py", line 476, in main
    trainer.train(model_path=model_path)
  File "/home/cao/miniconda3/lib/python3.8/site-packages/transformers/trainer.py", line 748, in train
    self.control = self.callback_handler.on_train_begin(self.args, self.state, self.control)
  File "/home/cao/miniconda3/lib/python3.8/site-packages/transformers/trainer_callback.py", line 335, in on_train_begin
    return self.call_event("on_train_begin", args, state, control)
  File "/home/cao/miniconda3/lib/python3.8/site-packages/transformers/trainer_callback.py", line 373, in call_event
    result = getattr(callback, event)(
  File "/home/cao/miniconda3/lib/python3.8/site-packages/transformers/integrations.py", line 502, in on_train_begin
    self.setup(args, state, model)
  File "/home/cao/miniconda3/lib/python3.8/site-packages/transformers/integrations.py", line 497, in setup
    mlflow.log_params(dict(combined_dict_items[i : i + MLflowCallback.MAX_LOG_SIZE]))
  File "/home/cao/miniconda3/lib/python3.8/site-packages/mlflow/tracking/fluent.py", line 475, in log_params
    MlflowClient().log_batch(run_id=run_id, metrics=[], params=params_arr, tags=[])
  File "/home/cao/miniconda3/lib/python3.8/site-packages/mlflow/tracking/client.py", line 838, in log_batch
    self._tracking_client.log_batch(run_id, metrics, params, tags)
  File "/home/cao/miniconda3/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 245, in log_batch
    self.store.log_batch(run_id=run_id, metrics=metrics, params=params, tags=tags)
  File "/home/cao/miniconda3/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 852, in log_batch
    _validate_batch_log_data(metrics, params, tags)
  File "/home/cao/miniconda3/lib/python3.8/site-packages/mlflow/utils/validation.py", line 232, in _validate_batch_log_data
    _validate_param(param.key, param.value)
  File "/home/cao/miniconda3/lib/python3.8/site-packages/mlflow/utils/validation.py", line 112, in _validate_param
    _validate_length_limit("Param value", MAX_PARAM_VAL_LENGTH, value)
  File "/home/cao/miniconda3/lib/python3.8/site-packages/mlflow/utils/validation.py", line 180, in _validate_length_limit
    raise MlflowException(
mlflow.exceptions.MlflowException: Param value '{'adapters': {'de': (text_lang, 'bb1c8efb82510bed')}, 'config_map': {text_lang: AdapterConfig(original_ln_before=True, original_ln_after=True, residual_before_ln=True, adapter_residual_before_ln=False, ln_before=False, ln_after=False, mh_adapter=Fals' had length 786, which exceeded length limit of 250




My codes are like this:

CUDA_VISIBLE_DEVICES="4" python3 /mnt/localdata/cao/run_de_modeling.py \
--output_dir=/mnt/localdata/cao/output_language_adapter_de/ \
--model_type=bert \
--model_name_or_path=bert-base-multilingual-cased \
--do_train \
--train_data_file=/mnt/localdata/cao/data_for_model/DE_train.txt \
--do_eval \
--eval_data_file=/mnt/localdata/cao/data_for_model/DE_valid.txt \
--mlm \
--language de \
--train_adapter \
--adapter_config pfeiffer \
--per_gpu_train_batch_size 3 \
--learning_rate 5e-5 \
--cache_dir /mnt/localdata/cao/de_cache_dir/

Comments: