Joelzhang commited on
Commit
d9a552e
1 Parent(s): 03746ad

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -4
README.md CHANGED
@@ -13,9 +13,9 @@ widget:
13
 
14
  ## 简介 Brief Introduction
15
 
16
- 善于处理长文本,采用旋转位置编码的中文版1.1亿参数的Longformer
17
 
18
- The Chinese version of Longformer, which uses rotating positional encoding and has 110M parameters, is adept at handling lengthy text.
19
 
20
  ## 模型分类 Model Taxonomy
21
 
@@ -25,9 +25,9 @@ The Chinese version of Longformer, which uses rotating positional encoding and h
25
 
26
  ## 模型信息 Model Information
27
 
28
- 遵循Longformer的设计,我们基于[chinese_roformer_L-12_H-768_A-12](https://github.com/ZhuiyiTechnology/roformer),在悟道语料库(180 GB版本)上进行了继续预训练。特别的,我们采用旋转位置嵌入(RoPE)来避免预训练语料库的不均匀序列长度问题。
29
 
30
- Following the design of Longformer, we performed continual pre-training on the WuDao corpus (180 GB) based on [chinese_roformer_L-12_H-768_A-12](https://github.com/ZhuiyiTechnology/roformer). Particularly, we employed rotational position embedding (RoPE) to avoid the uneven sequence length of the pre-trained corpus.
31
 
32
  ## 使用 Usage
33
 
 
13
 
14
  ## 简介 Brief Introduction
15
 
16
+ 善于处理长文本,采用旋转位置编码的中文版1.1亿参数的Longformer-base
17
 
18
+ The Chinese Longformer-base (110M), which uses rotating positional encoding, is adept at handling lengthy text.
19
 
20
  ## 模型分类 Model Taxonomy
21
 
 
25
 
26
  ## 模型信息 Model Information
27
 
28
+ 遵循Longformer-base的设计,我们基于[chinese_roformer_L-12_H-768_A-12](https://github.com/ZhuiyiTechnology/roformer),在悟道语料库(180 GB版本)上进行了继续预训练。特别的,我们采用旋转位置嵌入(RoPE)来避免预训练语料库的不均匀序列长度问题。
29
 
30
+ Following the design of Longformer-base, we performed continual pre-training on the WuDao corpus (180 GB) based on [chinese_roformer_L-12_H-768_A-12](https://github.com/ZhuiyiTechnology/roformer). Particularly, we employed rotational position embedding (RoPE) to avoid the uneven sequence length of the pre-trained corpus.
31
 
32
  ## 使用 Usage
33