Edit model card

YuYan-10b

YuYan is a series of natural language processing models developed by Fuxi AI lab, Netease.Inc, including text generation models, natural language understanding models, and more. YuYan-10b is a natural language understanding model trained on high-quality Chinese corpus.

YuYan-10b is similar to BERT in that it is trained on large-scale pre-training corpora using unsupervised learning. However, it differs in that it incorporates various tasks such as sentence order and word deletion in addition to the MLM task during training to enhance the model's semantic representation ability and improve its understanding of Chinese.

CLUE result

Score AFQMC TNEWS1.1 IFLYTEK OCNLI_50k WSC1.1 CSL
YuYan-10b 84.685 86.45 74.04 67.96 86.33 95.73 97.6
HUMAN 84.1 81 71 80.3 90.3 98 84
HunYuan-NLP 1T 83.632 85.11 70.44 67.54 86.5 96 96.2

How to use

Our model is trained based on the Megatron. As a result, the inference and finetuning depend on it.

Below are the install tutorial. We have packaged all the required dependencies for the model. Use the following command to obtain the model running environment.

pip install -r requirements.txt

Finetuning script

We provide multiple scripts for finetuning on the CLUE benchmark, which is a Chinese language understanding evaluation leaderboard that covers various tasks such as natural language understanding, reading comprehension, and semantic understanding. For any given CLUE task, use the following command to start finetuning.

# finetuning afqmc task
sh finetune_afqmc_distributed.sh

# finetuning csl task
sh finetune_csl_distributed.sh
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference API
Inference API (serverless) has been turned off for this model.