CPM LM
Author: buildcharger
Pull request: no
Labels: name: New model

# 🌟 New model addition （2.6B params）

## Model description

CPM(Chinese Pre-Trained Language Models), which has 2.6B parameters, can be used for zero shot、one shot、few shot learning.
Code and model is available.

## Open source status

* [✅] the model implementation is available:   [CPM-Generate pytorch](https://github.com/TsinghuaAI/CPM-Generate) [CPM-LM-TF2](https://github.com/qhduan/CPM-LM-TF2)
* [✅] the model weights are available: can be found [here](https://github.com/TsinghuaAI/CPM-Generate)
* [✅] who are the authors: (Research team of Beijing Zhiyuan Institute of artificial intelligence and Tsinghua University @ TsinghuaAI)


Comments:
@JetRunner might be interested in that!

Hi, I didn't notice this before I finished the translation to `Transformer` just now. May [this script](https://github.com/mymusise/CPM-TF2Transformer/blob/main/transfor_CMP.ipynb) help.

BTW, I met some problems when uploading. #8662

> @JetRunner might be interested in that!

Yes I was working on it but it seems @mymusise has already worked it out!

@mymusise I will assist you through the uploading process!

@mymusise I think the generated result from your repo is a little buggy here. Any idea why? 
```
[{'generated_text': '你好 ▁ , ▁我 ▁是 ▁ 个 ▁ <unk> ▁ <unk> ▁ <unk> ▁ <unk> ▁ <unk> ▁ <unk>'}]
```
https://github.com/mymusise/CPM-TF2Transformer/blob/e5ea4799603f19ab7f92596f7ad7472198c505c6/transfor_CMP.ipynb#L881

|```
|[{'generated_text': '你好 ▁ , ▁我 ▁是 ▁ 个 ▁ <unk> ▁ <unk> ▁ <unk> ▁ <unk> ▁ <unk> ▁ <unk>'}]
|```

@JetRunner  Hi, because I use  `BertTokenizer` here, didn't use the `bpe` method. 
And seems `GPT2Tokenizer` does not yet support other languages such as Chinese, the [byte_encoder](https://github.com/huggingface/transformers/blob/dd52804f5fce0a568ffbb3dc7fd088d2de0a0e56/src/transformers/models/gpt2/tokenization_gpt2.py#L246) here will encode other languages to unknown token. Any advice?

I see. It's not a big problem since we can now specify the tokenizer type in the configuration file. I can take care of those once you've uploaded the model file. Let's wait for @julien-c to solve the big file uploading problem first.

OK, thank you, guy.

Hi, the model has been uploaded, see: https://huggingface.co/mymusise/CPM-Third-Party

@mymusise Awesome news! Let me take care of the rest and I will keep you updated.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


😄 Is there any news? 
I have tested this before and the result is different from the official repo. 
Has this problem been solved? (I don't have a card now so I can't test it, sorry😢.)
If so, I will close this issue. @mymusise @JetRunner 

> I have tested this before and the result is different from the official repo.

Hi, there are a few problems with the old one. I've recreated the model card, please try the new one: [mymusise/CPM-GPT2](https://huggingface.co/mymusise/CPM-GPT2)

and the FP16 version has been uploaded already: [mymusise/CPM-GPT2-FP16](https://huggingface.co/mymusise/CPM-GPT2-FP16)


Thanks, it works perfectly! 😄