how to merge mutiple parts pt file to one pt file？
I have tried cat checkpoint_best_part_1.pt checkpoint_best_part_2.pt checkpoint_best_part_3.pt >checkpoint_best.pt
but when i load the model, one error arised:
Traceback (most recent call last):
File "main.py", line 9, in
inference = Inference(model_path, data_path, eet_batch_size)
File "/root/workspace/inference.py", line 81, in init
state = torch.load(args.path, map_location=torch.device("cpu"))
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 771, in load
with _open_file_like(f, 'rb') as opened_file:
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 270, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 251, in init
super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'transformer_lm_gpt2_xxl_dialogue/checkpoint_best_part*.pt'
then how should i merge these sub part files?
Thank you for your concern.
I will check it, and give you a response as soon as possible.
Thank you for your reply.
em... i pasted a wrong error info, the corresponding error info is:
Traceback (most recent call last):
File "main.py", line 9, in
inference = Inference(model_path, data_path, eet_batch_size)
File "/root/workspace/inference.py", line 81, in init
state = torch.load(args.path, map_location=torch.device("cpu"))
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 777, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 282, in init
super(_open_zipfile_reader, self).init(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: invalid header or archive is corrupted
The former version missed the codes for parts merging. We have fix the mistake and test the code. You can try it again.
The merging code is simple, from line 81 to line 85 of inference.py
If you meet new questions, please feel free to contact me.