bigcode

/

tiny_starcoder_py

like 71

You can clone this repo: https://github.com/bigcode-project/transformers/ and use this code

You can clone this repo: https://github.com/bigcode-project/transformers/ and use this code

Thanks,
But this code just support 1-way tensor/pipeline parallelism model converted, how to convert when tensor/pipline parallelism > 1?

you need to merge the partitions with Megatron-LM before the conversion, with something like this

python Megatron-LM/tools/checkpoint_util.py \
        --model-type GPT  \
        --load-dir CKPT_DIR \
        --save-dir OUTPUT_PATH \
        --target-tensor-parallel-size 1 \
        --target-pipeline-parallel-size 1