Bug when overwriting model checkpoint callback and using Trainer.from_argparse_args
🐛 Bug
Im experiencing a weird bug when overwriting the model checkpoint callback, this only happens when initializing the trainer with the function: Trainer.from_argparse_args. If done this way the script will get stuck on initialization and not start training (at least stuck more than 10~15 mins)
To Reproduce
I failed to create a reproducible snapshot of code based on your templates, so my guess is that is a bug that occurs when something else I added on my code is present (my guess is that it try's to write the binary file for my model and its too big and gets blocked).
The way I overwrite the checkpoint_callback to send it to the trainer constructor is this one:
    # Checkpoint manager
    checkpoint_callback = ModelCheckpoint(
        verbose=True,
        # save_top_k=5,  # Save 5 Top models
        # monitor="wer",
        # mode="min",
        # period=1,
    )

    # Logger
    logger = TensorBoardLogger(save_dir=args.loggs_path, name=args.experiment_name)

    args.checkpoint_callback = checkpoint_callback # IF I REMOVE THIS THEN IT WILL WORK
    args.early_stop_callback = early_stop
    args.logger = logge

    # ---------------------
    # RUN TRAINING
    # ---------------------
    main(args)
This one will get stuck on trainer.fit(model) unless I remove args.checkpoint_callback = checkpoint_callback
Since there are a lot of extra pytorch lightning functionalities included in my code than in the basic pl template I also link my repository to reproduce the error: https://github.com/jiwidi/DeepSpeech-pytorch/blob/lightning/train.py#L70-L86.
A sidebug from this behaviour is that the hparams.yaml file will be very long and unfinished.
It usually looks like:
auto_scale_batch_size: false
auto_select_gpus: true
batch_size: 4
data_root: data/
data_test:
- test-clean
data_train:
- train-clean-100
- train-clean-360
- train-other-500
dropout: 0.1
early_stop_callback: !!python/object:pytorch_lightning.callbacks.early_stopping.EarlyStopping
  best_score: !!python/object/apply:torch._utils._rebuild_tensor_v2
  - !!python/object/apply:torch.storage._load_from_bytes
    - !!binary |
      gAKKCmz8nEb5IGqoUBkugAJN6QMugAJ9cQAoWBAAAABwcm90b2NvbF92ZXJzaW9ucQFN6QNYDQAA
      AGxpdHRsZV9lbmRpYW5xAohYCgAAAHR5cGVfc2l6ZXNxA31xBChYBQAAAHNob3J0cQVLAlgDAAAA
      aW50cQZLBFgEAAAAbG9uZ3EHSwR1dS6AAihYBwAAAHN0b3JhZ2VxAGN0b3JjaApGbG9hdFN0b3Jh
      Z2UKcQFYDgAAADk0Mzg3MzMzMjcyOTkycQJYAwAAAGNwdXEDSwFOdHEEUS6AAl1xAFgOAAAAOTQz
      ODczMzMyNzI5OTJxAWEuAQAAAAAAAAAAAIB/
  - 0
  - !!python/tuple []
  - !!python/tuple []
  - false
  - !!python/object/apply:collections.OrderedDict
    - []
  min_delta: -0.0
  mode: min
  monitor: wer
  patience: 3
  stopped_epoch: 0
  strict: true
  verbose: true
  wait_count: 0
  warned_result_obj: false
early_stop_metric: wer
early_stop_patience: 3
epochs: 10
experiment_name: DeepSpeech
When overriding the checkpoint callback it looks like:
auto_scale_batch_size: false
auto_select_gpus: true
batch_size: 4
checkpoint_callback: &id002 !!python/object:pytorch_lightning.callbacks.model_checkpoint.ModelCheckpoint
  _rank: 0
  best_k_models: {}
  best_model_path: ''
  best_model_score: 0
  dirpath: null
  epoch_last_check: null
  filename: null
  kth_best_model_path: ''
  kth_value: !!python/object/apply:torch._utils._rebuild_tensor_v2
  - !!python/object/apply:torch.storage._load_from_bytes
    - !!binary |
      gAKKCmz8nEb5IGqoUBkugAJN6QMugAJ9cQAoWBAAAABwcm90b2NvbF92ZXJzaW9ucQFN6QNYDQAA
      AGxpdHRsZV9lbmRpYW5xAohYCgAAAHR5cGVfc2l6ZXNxA31xBChYBQAAAHNob3J0cQVLAlgDAAAA
      aW50cQZLBFgEAAAAbG9uZ3EHSwR1dS6AAihYBwAAAHN0b3JhZ2VxAGN0b3JjaApGbG9hdFN0b3Jh
      Z2UKcQFYDgAAADkzODI2ODYwODU0NDE2cQJYAwAAAGNwdXEDSwFOdHEEUS6AAl1xAFgOAAAAOTM4
      MjY4NjA4NTQ0MTZxAWEuAQAAAAAAAAAAAIB/
  - 0
  - !!python/tuple []
  - !!python/tuple []
  - false
  - !!python/object/apply:collections.OrderedDict
    - []
   .
   .
   .
   . #many many parameters
   .
   .
   .
   .
   .
        optimizer: &id007 !!python/object:torch.optim.adamw.AdamW
          defaults:
            amsgrad: false
            betas: !!python/tuple
            - 0.9
            - 0.999
            eps: 1.0e-08
            lr: 0.0005
            weight_decay: 0.01
          param_groups:
          - amsgrad: false
            base_momentum: 0.85
            betas: !!python/tuple
            - 0.95
            - 0.999
            eps: 1.0e-08
            initial_lr: 2.0e-05
            lr: 2.0e-05
            max_lr: 0.0005
            max_momentum: 0.95
            min_lr: 2.0e-09
            params:
            - !!python/object/apply:torch._utils._rebuild_parameter
              - !!python/object/apply:torch._utils._rebuild_tensor_v2
                - !!python/object/apply:torch.storage._load_from_bytes
                  - !!binary |
            - !!python/object/apply:torch._utils._rebuild_parameter
              - !!python/object/apply:torch._utils._rebuild_tensor_v2
                - !!python/object/apply:torch.storage._load_from_bytes
                  - !!binary |
                       ## a Binary string of more than 500.000 lines that is unfinished to write (the rest of hparams are missing on the file)
                       
The parameters that go after checkpoint callback are missing since my guess is that the script got stuck writting the binary for the optimizer parameters. Since my model is very big this will take a lot of time.
Expected behavior
The training loop to start :)
Environment
* CUDA:
	- GPU:
		- GeForce RTX 2060
	- available:         True
	- version:           10.1
* Packages:
	- numpy:             1.18.1
	- pyTorch_debug:     False
	- pyTorch_version:   1.6.0+cu101
	- pytorch-lightning: 0.9.0
	- tqdm:              4.46.0
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- 
	- processor:         x86_64
	- python:            3.6.10
	- version:           #51-Ubuntu SMP Fri Sep 4 19:50:52 UTC 2020