Question: Object of type EncoderDecoderConfig is not JSON serializable
Author: Captainr22
Pull request: no

Hi.
An error occurred when I used Trainer to train and save EncoderDecoderModel.

```python
  File "/home/jwli/ljw/study/hotpotqa/roberta_seq2seq/roberta_for_seq2seq.py", line 482, in <module>
    run(model_args, data_args, training_args)
  File "/home/jwli/ljw/study/hotpotqa/roberta_seq2seq/roberta_for_seq2seq.py", line 465, in run
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/site-packages/transformers/trainer.py", line 1391, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/site-packages/transformers/trainer.py", line 1495, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/site-packages/transformers/trainer.py", line 1557, in _save_checkpoint
    self.save_model(output_dir)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/site-packages/transformers/trainer.py", line 1961, in save_model
    self._save(output_dir)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/site-packages/transformers/trainer.py", line 2009, in _save
    self.model.save_pretrained(output_dir, state_dict=state_dict)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1053, in save_pretrained
    model_to_save.config.save_pretrained(save_directory)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/site-packages/transformers/configuration_utils.py", line 416, in save_pretrained
    self.to_json_file(output_config_file, use_diff=True)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/site-packages/transformers/configuration_utils.py", line 739, in to_json_file
    writer.write(self.to_json_string(use_diff=use_diff))
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/site-packages/transformers/configuration_utils.py", line 725, in to_json_string
    return json.dumps(config_dict, indent=2, sort_keys=True) + "\n"
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/home/jwli/anaconda3/envs/study/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type EncoderDecoderConfig is not JSON serializable
```
My model and Config define the following code. 
```python
    tokenizer = RobertaTokenizerFast.from_pretrained(model_args.tokenizer_name)
    encoder_config = RobertaConfig.from_pretrained(model_args.encoder_model_name_or_path)
    decoder_config = RobertaConfig.from_pretrained(model_args.decoder_model_name_or_path)
    encoder_decoder_config = EncoderDecoderConfig.from_encoder_decoder_configs(encoder_config, decoder_config)
    model = RobertaForSeq2Seq.from_encoder_decoder_pretrained(model_args.encoder_model_name_or_path,
                                                              model_args.decoder_model_name_or_path,
                                                              config=encoder_decoder_config, tie_encoder_decoder=True)
    model.config.decoder_start_token_id = tokenizer.bos_token_id
    model.config.eos_token_id = tokenizer.eos_token_id
    model.config.max_length = 64
    model.config.early_stopping = True
    model.config.no_repeat_ngram_size = 3
    model.config.length_penalty = 2.0
    model.config.num_beams = 4
    model.config.pad_token_id = tokenizer.pad_token_id
```
This error occurred because EncoderDecoderConfig cannot be converted to json format. But I don't know how to modify it.
```python
ERROR OCCURRED:

        if use_diff is True:
            config_dict = self.to_diff_dict()
        else:
            config_dict = self.to_dict()
        return json.dumps(config_dict, indent=2, sort_keys=True) + "\n"
```

I look forward to your help! Thanks!
 @jplu  @patrickvonplaten 

Comments:
Hey @Captainr22,

Thanks for reporting this issue here. I now know how you create the model. Could you also provide the code to run this model with the Trainer? A bash command is enough in case you are using one of the official examples :-)

Hi @patrickvonplaten ,

Here is my code to run my model. It looks very ordinary.
```test
CUDA_VISIBLE_DEVICES=1,2 python roberta_for_seq2seq.py --output_dir=./seq2seq_output --train_dataset_name=../datasets/seq2seq_hotpotqa_train.csv --eval_dataset_name=../datasets/seq2seq_hotpotqa_eval.csv --do_train --do_eval --per_device_train_batch_size=4 --per_device_eval_batch_size=4 --evaluation_strategy=epoch --save_strategy=no --overwrite_output_dir --num_train_epochs=10 --gradient_accumulation_steps=2 --warmup_steps=5000 
```

And here is my Trainer code.
```python
trainer = Seq2SeqHotpotQuestionAnsweringTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset if training_args.do_train else None,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        eval_examples=eval_examples if training_args.do_eval else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
        post_process_function=post_processing_function,
        compute_metrics=None,
    )


class Seq2SeqHotpotQuestionAnsweringTrainer(Seq2SeqTrainer):
    def __init__(self, *args, eval_examples=None, post_process_function=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.eval_examples = eval_examples
        self.post_process_function = post_process_function

    def evaluate(self, eval_dataset=None, eval_examples=None, ignore_keys=None, metric_key_prefix: str = 'eval'):
        pass

```

Besides that, I found that if I don't add the config when defining the model, the problem doesn't occur. Like this:
```python
model = RobertaForSeq2Seq.from_encoder_decoder_pretrained(model_args.encoder_model_name_or_path,
                                                              model_args.decoder_model_name_or_path,
                                                              tie_encoder_decoder=True)
```

I think this may be a bug in transformers. 

The problem with this issue is probably the same as mine. #5459



Hey @Captainr22,

I don't know what `roberta_for_seq2seq.py` is. Could you please provide all the code that is needed to reproduce the error.

Hi @patrickvonplaten ,
I will provide you with another code which is using my cnn_dm dataset to train the EncoderDecoderModel. This code also has the above error. The roberta_decoder model is the RobertaForCausalLM which has 6 Transformer layers. Thank you for your attention!
```python
import nltk
import numpy as np
import os
import logging
import datasets
import transformers
import sys

from typing import Optional
from dataclasses import dataclass, field
from datasets import load_dataset, load_metric
from transformers import HfArgumentParser,Seq2SeqTrainingArguments, set_seed, BartTokenizer, BartForConditionalGeneration,\
    Seq2SeqTrainer, DataCollatorForSeq2Seq, RobertaTokenizerFast, EncoderDecoderModel, EncoderDecoderConfig, RobertaConfig
from transformers.trainer_utils import get_last_checkpoint

logger = logging.getLogger(__name__)

@dataclass
class ModelArguments:
    model_name_or_path: str = field(
        default="/home/jwli/models/facebook/bart-base",
        metadata={"help": "Path to pretrained model or model identifier from huggingface.co/models"}
    )
    tokenizer_name: Optional[str] = field(
        default="/home/jwli/models/facebook/bart-base",
        metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}
    )


@dataclass
class DataTrainingArguments:
    dataset_name: Optional[str] = field(
        default="xsum", metadata={"help": "The name of the dataset to use (via the datasets library)."}
    )
    max_source_length: Optional[int] = field(
        default=1024,
        metadata={
            "help": "The maximum total input sequence length after tokenization. Sequences longer "
                    "than this will be truncated, sequences shorter will be padded."
        },
    )
    max_target_length: Optional[int] = field(
        default=256,
        metadata={
            "help": "The maximum total sequence length for target text after tokenization. Sequences longer "
                    "than this will be truncated, sequences shorter will be padded."
        },
    )
    pad_to_max_length: bool = field(
        default=False,
        metadata={
            "help": "Whether to pad all samples to model maximum sentence length. "
                    "If False, will pad the samples dynamically when batching to the maximum length in the batch. More "
                    "efficient on GPU but very bad for TPU."
        },
    )
    val_max_target_length: Optional[int] = field(
        default=256,
        metadata={
            "help": "The maximum total sequence length for validation target text after tokenization. Sequences longer "
                    "than this will be truncated, sequences shorter will be padded. Will default to `max_target_length`."
                    "This argument is also used to override the ``max_length`` param of ``model.generate``, which is used "
                    "during ``evaluate`` and ``predict``."
        },
    )
    max_train_samples: Optional[int] = field(
        default=None,
        metadata={
            "help": "For debugging purposes or quicker training, truncate the number of training examples to this "
                    "value if set."
        },
    )
    max_eval_samples: Optional[int] = field(
        default=None,
        metadata={
            "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
                    "value if set."
        },
    )
    max_predict_samples: Optional[int] = field(
        default=None,
        metadata={
            "help": "For debugging purposes or quicker training, truncate the number of prediction examples to this "
                    "value if set."
        },
    )
    num_beams: Optional[int] = field(
        default=4,
        metadata={
            "help": "Number of beams to use for evaluation. This argument will be passed to ``model.generate``, "
                    "which is used during ``evaluate`` and ``predict``."
        },
    )
    ignore_pad_token_for_loss: bool = field(
        default=True,
        metadata={
            "help": "Whether to ignore the tokens corresponding to padded labels in the loss computation or not."
        },
    )


if __name__ == '__main__':
    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, Seq2SeqTrainingArguments))
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    set_seed(training_args.seed)

    dataset = load_dataset('csv', data_files={'train':"/home/jwli/ljw/study/use_trainer/xsum/cnn_data/train.csv",
                                              'eval':"/home/jwli/ljw/study/use_trainer/xsum/cnn_data/valid.csv"})

    # bart
    # tokenizer = BartTokenizer.from_pretrained(model_args.tokenizer_name)
    # model = BartForConditionalGeneration.from_pretrained(model_args.model_name_or_path)
    # model.resize_token_embeddings(len(tokenizer))

    # encoder decoder model
    tokenizer = RobertaTokenizerFast.from_pretrained("/home/jwli/models/roberta-large")
    encoder_config = RobertaConfig.from_pretrained("/home/jwli/models/roberta-large")
    decoder_config = RobertaConfig.from_pretrained("/home/jwli/models/roberta-decoder")
    encoder_decoder_config = EncoderDecoderConfig.from_encoder_decoder_configs(encoder_config, decoder_config)
    model = EncoderDecoderModel.from_encoder_decoder_pretrained("/home/jwli/models/roberta-large",
                                                              "/home/jwli/models/roberta-decoder",
                                                              config=encoder_decoder_config,
                                                              tie_encoder_decoder=True)
    model.config.decoder_start_token_id = tokenizer.bos_token_id
    model.config.eos_token_id = tokenizer.eos_token_id
    model.config.max_length = 64
    model.config.early_stopping = True
    model.config.no_repeat_ngram_size = 3
    model.config.length_penalty = 2.0
    model.config.num_beams = 4
    model.config.pad_token_id = tokenizer.pad_token_id

    padding = "max_length" if data_args.pad_to_max_length else False

    def _preprocess_function(examples):
        inputs = examples['document']
        targets = examples['summary']

        model_inputs = tokenizer(inputs, max_length=data_args.max_source_length, padding=padding, truncation=True)
        model_targets = tokenizer(targets, max_length=data_args.max_target_length, padding=padding, truncation=True)

        model_targets['input_ids'] = [
            [(l if l != tokenizer.pad_token_id else -100) for l in model_target] for model_target in model_targets['input_ids']
        ]

        model_inputs['labels'] = model_targets['input_ids']
        return model_inputs

    train_dataset, eval_dataset, predict_dataset = None, None, None

    if training_args.do_train:
        train_dataset = dataset['train']
        if data_args.max_train_samples is not None:
            train_dataset = train_dataset.select(range(data_args.max_train_samples))

        with training_args.main_process_first(desc="train dataset map pre-processing"):

            train_dataset = train_dataset.map(
                _preprocess_function,  # 传入function
                batched=True,
                remove_columns=['document', 'summary'],  # map后原来数据集中的数据就不需要了，只需要保留input_ids,attention_mask和labels
                num_proc=16,
                desc="Running Tokenizer on train dataset",
            )

    if training_args.do_eval:
        eval_dataset = dataset['eval']
        if data_args.max_eval_samples is not None:
            eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
        with training_args.main_process_first(desc="validation dataset map pre-processing"):
            eval_dataset = eval_dataset.map(
                _preprocess_function,
                batched=True,
                remove_columns=['document', 'summary'],
                num_proc=16,
                desc="Running Tokenizer on eval dataset",
            )

    if training_args.do_predict:
        predict_dataset = dataset['test']
        if data_args.max_predict_samples is not None:
            predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
        with training_args.main_process_first(desc="prediction dataset map pre-processing"):
            predict_dataset = predict_dataset.map(
                _preprocess_function,
                batched=True,
                remove_columns=['document', 'summary'],
                num_proc=16,
                desc="Running Tokenizer on predict dataset",
            )

    data_collator = DataCollatorForSeq2Seq(
        tokenizer,
        model=model,
        label_pad_token_id=-100,
        pad_to_multiple_of=8 if training_args.fp16 else None,
    )

    metric = load_metric("rouge")

    def _postprocess_text(preds, labels):
        preds = [pred.strip() for pred in preds]
        labels = [label.strip() for label in labels]

        # rougeLSum expects newline after each sentence
        preds = ["\n".join(nltk.sent_tokenize(pred)) for pred in preds]
        labels = ["\n".join(nltk.sent_tokenize(label)) for label in labels]

        return preds, labels

    def _computer_metric(eval_preds):
        preds, labels = eval_preds
        if isinstance(preds, tuple):
            preds = preds[0]
        decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
        if data_args.ignore_pad_token_for_loss:
            # Replace -100 in the labels as we can't decode them.
            labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
        decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

        # Some simple post-processing
        decoded_preds, decoded_labels = _postprocess_text(decoded_preds, decoded_labels)

        result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
        # Extract a few results from ROUGE
        result = {key: value.mid.fmeasure * 100 for key, value in result.items()}

        prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
        result["gen_len"] = np.mean(prediction_lens)
        result = {k: round(v, 4) for k, v in result.items()}
        return result

    trainer = Seq2SeqTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset if training_args.do_train else None,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=_computer_metric if training_args.predict_with_generate else None,
    )

    if training_args.do_train:

        checkpoint = None
        if training_args.resume_from_checkpoint is not None:
            checkpoint = training_args.resume_from_checkpoint
        elif last_checkpoint is not None:
            checkpoint = last_checkpoint

        train_result = trainer.train(resume_from_checkpoint=checkpoint)

        metrics = train_result.metrics
        max_train_samples = (
            data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)
        )
        metrics["train_samples"] = min(max_train_samples, len(train_dataset))

        trainer.log_metrics("train", metrics)
        trainer.save_metrics("train", metrics)
        trainer.save_state()

    results = {}
    max_length = (
        training_args.generation_max_length
        if training_args.generation_max_length is not None
        else data_args.val_max_target_length
    )
    num_beams = data_args.num_beams if data_args.num_beams is not None else training_args.generation_num_beams
```

Ok I think I can finally reproduce the error :-) 

This doesn't work: 

```python
#!/usr/bin/env python3
from transformers import RobertaConfig, EncoderDecoderConfig, EncoderDecoderModel

model_id = "roberta-base"

encoder_config = RobertaConfig.from_pretrained(model_id)
decoder_config = RobertaConfig.from_pretrained(model_id)
encoder_decoder_config = EncoderDecoderConfig.from_encoder_decoder_configs(encoder_config, decoder_config)
model = EncoderDecoderModel.from_encoder_decoder_pretrained(model_id, model_id, config=encoder_decoder_config, tie_encoder_decoder=True)
model.config.decoder_start_token_id = 0
model.config.eos_token_id = 0
model.config.max_length = 64
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 4
model.config.pad_token_id = 0

model.save_pretrained("./")
```

since it fails with:

```bash
  File "/usr/lib/python3.8/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/lib/python3.8/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.8/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type EncoderDecoderConfig is not JSON serializable
```

no?

The problem in your code is that you pass an `EncoderDecoderConfig` into the `from_encoder_decoder_pretrained(...)` method, which is not allowed. You should instead pass an `encoder_config` and a `decoder_config` argument as follows:

```python
#!/usr/bin/env python3
from transformers import RobertaConfig, EncoderDecoderConfig, EncoderDecoderModel

model_id = "roberta-base"

encoder_config = RobertaConfig.from_pretrained(model_id)
decoder_config = RobertaConfig.from_pretrained(model_id)

model = EncoderDecoderModel.from_encoder_decoder_pretrained(model_id, model_id, encoder_config=encoder_config, decoder_config=decoder_config, tie_encoder_decoder=True)
model.config.decoder_start_token_id = 0
model.config.eos_token_id = 0
model.config.max_length = 64
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 4
model.config.pad_token_id = 0

model.save_pretrained("./")
```

👍 I will try your suggestion. 
I am sorry to have taken you so long because of my problems. :-(
Thank you very much!

No worries! Hope it works now for your use case