save checkpoint issue in nested trainer
Hi all, I use multi-GPU to train the model, this error happens when saving the checkpoint. But using a single GPU to train the model, everything works fine.
TypeError: cannot pickle '_thread.lock' object

I wonder if it is because I nest the trainer inside another model, the codes for a nested trainer is as followed:
self.model.trainer = self.trainer

So many thanks in advance.
Hi @eddiecong, the trainer is not pickleable and so you're probably correct that it's to do with your nested trainer code. You could override the on_save_checkpoint hook and then delete the trainer from the checkpoint as suggested by this answer: #3444 (comment)
Hope that helps 😃