Repository: FlagAI-Open/FlagAI (language: python)

Issue:
edit tutorial 8 12 13
Signed-off-by: wchh-2000 <2371156095@qq.com>


Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index 8c549251..46037518 100644
--- a/README.md
+++ b/README.md
@@ -80,7 +80,7 @@ Host 127.0.0.1
 ```
 
 ## Quick Start
-We provide many models which are trained to perform different tasks. You can load these models by AutoLoader to make prediction.
+We provide many models which are trained to perform different tasks. You can load these models by AutoLoader to make prediction. See more in `FlagAI/quickstart`.
 ## Load model and tokenizer
 We provide the AutoLoad class to load the model and tokenizer quickly, for example:
 ```python
@@ -124,12 +124,13 @@ for text in test_data:
 * [NER with RoBerta-WWM](/docs/TUTORIAL_17_BERT_EXAMPLE_NER.md)
 * [Writing with GPT-2](/docs/TUTORIAL_18_GPT2_WRITING.md)
 * [Title generation with T5](/docs/TUTORIAL_19_T5_EXAMPLE_TITLE_GENERATION.md)
-* [Supported tasks](/docs/TUTORIAL_20_SUPPORTED_TASKS.md)
+
+[//]: # (* [Supported tasks]&#40;/docs/TUTORIAL_20_SUPPORTED_TASKS.md&#41;)
 
 
 This session explains how the base NLP classes work, how you can load pre-trained models to tag your
 text, how you can embed your text with different word or document embeddings, and how you can train your own
-language models, sequence labeling models, and text classification models. Let us know if anything is unclear.
+language models, sequence labeling models, and text classification models. Let us know if anything is unclear. See more in `FlagAI/examples`.
 
 
 
diff --git a/README_zh.md b/README_zh.md
index 3e01ee80..c525b02d 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -5,16 +5,16 @@
 
 --------------------------------------------------------------------------------
 
-FlagAI 是一个快速、易于使用和可扩展的大模型工具包。 我们的目标是支持在多模态的各种下游任务上训练、微调和部署大规模模型。 目前，我们专注于 NLP 模型和任务。 在不久的将来，我们将支持其他模态。
+飞智是一个快速、易于使用和可扩展的大模型工具包。 我们的目标是支持在多模态的各种下游任务上训练、微调和部署大规模模型。 目前，我们专注于 NLP 模型和任务。 在不久的将来，我们将支持其他模态。
 <br><br>
 
-* 现在它支持最高百亿参数的**WUDAO GLM**(详见[GLM介绍](/doc_zh/GLM.md))。它同时也支持**BERT**、**RoBERTa**、**GPT2**、**T5** 模型和 Huggingface Transformers 的模型。
+* 现在它支持最高百亿参数的**悟道GLM**(详见[GLM介绍](/doc_zh/GLM.md))。它同时也支持**BERT**、**RoBERTa**、**GPT2**、**T5** 模型和 Huggingface Transformers 的模型。
 
 * 它提供 API 以快速下载并在给定（中/英文）文本上使用这些预训练模型，在您自己的数据集上对其进行微调(fine-tuning)或者应用[提示学习(prompt-tuning)](/doc_zh/TUTORIAL_7_PROMPT_LERANING.md)，然后在我们的模型中心与社区共享它们。 
 
 * 这些模型可以应用于文本，用于文本分类、信息提取、问答、摘要、文本生成等任务，尤其是中文。
 
-* FlagAI 由三个最流行的数据/模型并行库（[PyTorch](https://pytorch.org/)/[Deepspeed](https://www.deepspeed.ai/)/[Megatron-LM](https://github.com/NVIDIA/Megatron-LM)）提供支持，它们之间实现了无缝集成。 你可以用不到十行代码来并行你的训练/测试过程。
+* 飞智由三个最流行的数据/模型并行库（[PyTorch](https://pytorch.org/)/[Deepspeed](https://www.deepspeed.ai/)/[Megatron-LM](https://github.com/NVIDIA/Megatron-LM)）提供支持，它们之间实现了无缝集成。 你可以用不到十行代码来并行你的训练/测试过程。
 
 
 本项目的部分代码基于[GLM](https://github.com/THUDM/GLM),[Transformers](https://github.com/huggingface/transformers) 和 [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples).
@@ -80,7 +80,7 @@ Host 127.0.0.1
 
 
 # 快速上手
-我们提供了精选的中英文预训练模型，以及经过训练可以执行不同任务的模型权重。 您可以通过 AutoLoader 加载这些模型以进行训练和预测。
+我们提供了精选的中英文预训练模型，以及经过训练可以执行不同任务的模型权重。 您可以通过 AutoLoader 加载这些模型以进行训练和预测。更多样例见 `FlagAI/quickstart`。
 
 ## 加载模型和分词器
 我们提供 `AutoLoad` 类来快速加载模型和分词器，例如：
@@ -190,23 +190,23 @@ for text_pair in test_data:
 * [RoBERTa-base-ch用于命名实体识别](/doc_zh/TUTORIAL_17_BERT_EXAMPLE_NER.md)
 * [GPT-2用于文本续写](/doc_zh/TUTORIAL_18_GPT2_WRITING.md)
 * [T5用于标题生成](/doc_zh/TUTORIAL_19_T5_EXAMPLE_TITLE_GENERATION.md)
-* [所有支持的任务](doc_zh/TUTORIAL_20_SUPPORTED_TASKS.md)
+* [用GLM10b模型在TNEWS短文本分类数据集上微调](doc_zh/TUTORIAL_20_GLM_TNEWS.md)
 
 
-本节解释了本项目中基础NLP类是如何工作的，如何加载预先训练的模型来标记您的文本，如何使用不同的词或文档嵌入来得到表示，以及如何训练自己的语言模型、序列标注模型和文本分类模型。
+本节解释了本项目中基础NLP类是如何工作的，如何加载预先训练的模型来标记您的文本，如何使用不同的词或文档嵌入来得到表示，以及如何训练自己的语言模型、序列标注模型和文本分类模型。更多样例见 `FlagAI/examples`。
 
 
 # 教程
 我们提供了一组教程来帮助您快速上手使用本库：
-* [Tutorial 1: 构建和应用分词器](/doc_zh/TUTORIAL_1_TOKENIZER.md)
+* [Tutorial 1: 如何构建和应用分词器](/doc_zh/TUTORIAL_1_TOKENIZER.md)
 * [Tutorial 2: 数据集预处理流程](/doc_zh/TUTORIAL_2_DATASET.md)
 * [Tutorial 3: 模型的主要功能及相关结构](/doc_zh/TUTORIAL_3_MODEL.md)
-* [Tutorial 4: 模型训练(支持并行化)](/doc_zh/TUTORIAL_4_TRAINER.md)
-* [Tutorial 5: 使用AutoLoader工具快速构建模型](/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md)
-* [Tutorial 6: 使用Predictor工具进行预测](/doc_zh/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md)
-* [Tutorial 7: FlagAI提示学习功能](/doc_zh/TUTORIAL_7_PROMPT_LERANING.md)
-* [Tutorial 8: 数据/模型并行的环境设置](/doc_zh/TUTORIAL_8_ENVIRONMENT_SETUP.md)
-* [Tutorial 9: 使用**编码器/解码器/编解码**器模型进行文本生成](/doc_zh/TUTORIAL_9_SEQ2SEQ_METHOD.md)
+* [Tutorial 4: 为模型和数据并行训练定制训练器](/doc_zh/TUTORIAL_4_TRAINER.md)
+* [Tutorial 5: 使用 Autoloader 简化模型和分词器初始化过程](/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md)
+* [Tutorial 6: 将现成的推理算法与 Predictor 结合使用](/doc_zh/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md)
+* [Tutorial 7: 使用飞智提示学习工具包来提高在SuperGLUE任务上的表现](/doc_zh/TUTORIAL_7_PROMPT_LERANING.md)
+* [Tutorial 8: 多机训练模型搭建环境](/doc_zh/TUTORIAL_8_ENVIRONMENT_SETUP.md)
+* [Tutorial 9: 使用encoder/decoder/encoder-decoder模型进行文本生成](/doc_zh/TUTORIAL_9_SEQ2SEQ_METHOD.md)
 * [Tutorial 10: 转化一个模型为Megatron-LM的模型并行版本](/doc_zh/TUTORIAL_10_MEGATRON.md)
 
 
diff --git a/doc_zh/GLM.md b/doc_zh/GLM.md
index 1458d6e6..a42ae357 100644
--- a/doc_zh/GLM.md
+++ b/doc_zh/GLM.md
@@ -12,6 +12,7 @@
 | Autoencoding    | ✅   | ×         | ×          |
 | Encoder-Decoder | -   | ✅         | -          |
 | GLM             | ✅   | ✅         | ✅          |
+
 GLM的主要功能包括：
 
 - 任务一：文本的一些区间会被屏蔽（参照自动编码的做法）。 这些区间将被随机重新排列，并以自动回归方式进行预测。屏蔽的区间覆盖原始文本的15%。
diff --git a/doc_zh/TUTORIAL_10_METATRON.md b/doc_zh/TUTORIAL_10_METATRON.md
index 55c89bfa..ae837dac 100644
--- a/doc_zh/TUTORIAL_10_METATRON.md
+++ b/doc_zh/TUTORIAL_10_METATRON.md
@@ -1,15 +1,16 @@
-# 转化model为Megatron-LM版本
-三个步骤：
-- [1] 将`MLP`层转化为`column/rowParallel`版本
-- [2] 将`self-Attention` 中的两个`Linear layer`分别转化为`column/rowParallel `版本
-- [3] 将`cross-Attention`中的两个`Linear Layer` 转化为`column/rowParallel`版本
+# 转化一个模型为Megatron-LM的模型并行版本
+- [转化model为Megatron-LM版本](#转化model为megatron-lm版本)
+  - [1.将MLP层转化为column/rowParallel版本](#1将mlp层转化为columnrowparallel版本)
+  - [2.转化`self-Attention`](#2转化self-attention)
+    - [2.1 将`x` proj 到  `q, k, v` 的`linear `层，转化为 `columnParalleLinear`](#21-将x-proj-到--q-k-v-的linear-层转化为-columnparallelinear)
+    - [2.2 将输出的`dense`层转化为`rowparallelLinear`](#22-将输出的dense层转化为rowparallellinear)
+  - [3.将`cross-Attention`中的两个`Linear Layer `转化为`column/rowParallel`版本](#3将cross-attention中的两个linear-layer-转化为columnrowparallel版本)
 
-`parallel`版本中主要的操作都是从`Megatron-LM`中拿出来的，放在了`mpu`模块中。
+飞智支持了BERT，GLM，GPT2 以及T5模型的megatron-lm模型并行. 其主要的操作都是从`Megatron-LM`中拿出来的，放在了`mpu`模块中。
 
 ## 1.将MLP层转化为column/rowParallel版本
-代码位置：`flagai/model/layers/embeddings_mpu.py` 【名字有点奇怪】
-核心思想
-将两个1linear1层的1forward1过程，按照特定顺序进行拆分（先列后行）：
+代码位置：`flagai/model/layers/embeddings_mpu.py` 
+核心思想:将两个`linear`层的`forward`过程，按照特定顺序进行拆分（先列后行）：
 ```python
 intermediate_parallel = self.dense_h_to_4h(hidden_states)
 intermediate_parallel = gelu(intermediate_parallel)
@@ -38,178 +39,10 @@ self.hidden_size_per_attention_head = divide(hidden_size,
 self.num_attention_heads_per_partition = divide(
     num_attention_heads, world_size)
 ```
-下面是两个`parallel linear`的代码实现
-### ColumnParallelLinear
-```python
-
-class ColumnParallelLinear(torch.nn.Module):
-    """Linear layer with column parallelism.
-
-    The linear layer is defined as Y = XA + b. A is parallelized along
-    its second dimension as A = [A_1, ..., A_p].
-
-    Arguments:
-        input_size: first dimension of matrix A.
-        output_size: second dimension of matrix A.
-        bias: If true, add bias
-        gather_output: If true, call all-gether on output and make Y avaiable
-                       to all GPUs, otherwise, every GPU will have its output
-                       which is Y_i = XA_i
-        init_method: method to initialize weights. Note that bias is always set
-                     to zero.
-        stride: For the strided linear layers.
-        keep_master_weight_for_test: This was added for testing and should be
-                                     set to False. It returns the master weights
-                                     used for initialization.
-    """
-
-    def __init__(self,
-                 input_size,
-                 output_size,
-                 bias=True,
-                 gather_output=True,
-                 init_method=init.xavier_normal_,
-                 stride=1,
-                 keep_master_weight_for_test=False):
-        super(ColumnParallelLinear, self).__init__()
-
-        # Keep input parameters
-        self.input_size = input_size
-        self.output_size = output_size
-        self.gather_output = gather_output
-        # Divide the weight matrix along the last dimension.
-        world_size = get_model_parallel_world_size()
-        self.output_size_per_partition = divide(output_size, world_size)
-
-        # Parameters.
-        # Note: torch.nn.functional.linear performs XA^T + b and as a result
-        # we allocate the transpose.
-        self.weight = Parameter(
-            torch.Tensor(self.output_size_per_partition, self.input_size))
-        self.weight.model_parallel = True
-        if bias:
-            self.bias = Parameter(torch.Tensor(self.output_size_per_partition))
-            self.bias.model_parallel = True
-            # Always initialize bias to zero.
-            with torch.no_grad():
-                self.bias.zero_()
-        else:
-            self.register_parameter('bias', None)
-
-        # Initialize weight.
-        self.master_weight = _initialize_affine_weight(
-            self.weight,
-            self.output_size,
-            self.input_size,
-            self.output_size_per_partition,
-            0,
-            init_method,
-            stride=stride,
-            return_master_weight=keep_master_weight_for_test)
 
-    def forward(self, input_):
-        # Set up backprop all-reduce.
-        input_parallel = copy_to_model_parallel_region(input_)
-        # Matrix multiply.
-        output_parallel = F.linear(input_parallel, self.weight, self.bias)
-        if self.gather_output:
-            # All-gather across the partitions.
-            output = gather_from_model_parallel_region(output_parallel)
-            # gather_from_model_parallel_region 是mpu中提供的功能
-        else:
-            output = output_parallel
-        return output
-```
-### RowParallelLinear
-```python
-class RowParallelLinear(torch.nn.Module):
-    """Linear layer with row parallelism.
-
-    The linear layer is defined as Y = XA + b. A is parallelized along
-    its first dimension and X along its second dimension as:
-               -   -
-              | A_1 |
-              | .   |
-          A = | .   |        X = [X_1, ..., X_p]
-              | .   |
-              | A_p |
-               -   -
-    Arguments:
-        input_size: first dimension of matrix A.
-        output_size: second dimension of matrix A.
-        bias: If true, add bias. Note that bias is not parallelized.
-        input_is_parallel: If true, we assume that the input is already
-                           split across the GPUs and we do not split
-                           again.
-        init_method: method to initialize weights. Note that bias is always set
-                     to zero.
-        stride: For the strided linear layers.
-        keep_master_weight_for_test: This was added for testing and should be
-                                     set to False. It returns the master weights
-                                     used for initialization.
-    """
-
-    def __init__(self,
-                 input_size,
-                 output_size,
-                 bias=True,
-                 input_is_parallel=False,
-                 init_method=init.xavier_normal_,
-                 stride=1,
-                 keep_master_weight_for_test=False):
-        super(RowParallelLinear, self).__init__()
+## 2.转化`self-Attention` 
+`self-attention`中的两个Linear layer分别转化为`column/rowParallel` 版本
 
-        # Keep input parameters
-        self.input_size = input_size
-        self.output_size = output_size
-        self.input_is_parallel = input_is_parallel
-        # Divide the weight matrix along the last dimension.
-        world_size = get_model_parallel_world_size()
-        self.input_size_per_partition = divide(input_size, world_size)
-
-        # Parameters.
-        # Note: torch.nn.functional.linear performs XA^T + b and as a result
-        # we allocate the transpose.
-        self.weight = Parameter(
-            torch.Tensor(self.output_size, self.input_size_per_partition))
-        self.weight.model_parallel = True
-        if bias:
-            self.bias = Parameter(torch.Tensor(self.output_size))
-            # Always initialize bias to zero.
-            with torch.no_grad():
-                self.bias.zero_()
-        else:
-            self.register_parameter('bias', None)
-
-        # Initialize weight.
-        self.master_weight = _initialize_affine_weight(
-            self.weight,
-            self.output_size,
-            self.input_size,
-            self.input_size_per_partition,
-            1,
-            init_method,
-            stride=stride,
-            return_master_weight=keep_master_weight_for_test)
-
-    def forward(self, input_):
-        # Set up backprop all-reduce.
-        if self.input_is_parallel:
-            input_parallel = input_
-        else:
-            input_parallel = scatter_to_model_parallel_region(input_)
-        # Matrix multiply.
-        output_parallel = F.linear(input_parallel, self.weight)
-        # All-reduce across all the partitions.
-        output_ = reduce_from_model_parallel_region(output_parallel)
-        # reduce_from_model_parallel_region 是mpu提供的功能
-        if self.bias is not None:
-            output = output_ + self.bias
-        else:
-            output = output_
-        return output
-```
-## 2.将`self-Attention` 中的两个Linear layer分别转化为`column/rowParallel` 版本
 代码位置：`flagai/model/layers/attentions_mpu.py`
 ### 2.1 将`x` proj 到  `q, k, v` 的`linear `层，转化为 `columnParalleLinear`
 ### 2.2 将输出的`dense`层转化为`rowparallelLinear`
diff --git a/doc_zh/TUTORIAL_11_GLM_BLANK_FILLING_QA.md b/doc_zh/TUTORIAL_11_GLM_BLANK_FILLING_QA.md
index d73f2629..4324e35c 100644
--- a/doc_zh/TUTORIAL_11_GLM_BLANK_FILLING_QA.md
+++ b/doc_zh/TUTORIAL_11_GLM_BLANK_FILLING_QA.md
@@ -1,4 +1,3 @@
-
 # GLM 完形填空式生成
 
 ![img.png](./img/glm_blank_filling.png)
@@ -20,9 +19,9 @@ GLM 对下游任务进行微调，并将它们重新定义为空白填充生成
 
 
 在 GLM 中，共有三种 MASK 方法，分别对应三种预测格式。
-1. ```[MASK]```, 令牌级别。这种屏蔽方法只会屏蔽一个句子中的随机标记，
-   在三种遮罩方法中，它的掩码部分最少。所以，生成的内容也有限。例如： ``[CLS]北京故宫是中国[MASK]非物质文化遗产。

现存最大的古代宫殿建筑, 也是``.
-2. ```[sMASK]```: 实体级。我们限制遮罩
+1. ```[MASK]```, Token级别。这种屏蔽方法只会屏蔽一个句子中的随机标记，
+   在三种 MASK 方法中，它的掩码部分最少。所以，生成的内容也有限。例如： ``[CLS]北京故宫是中国[MASK]非物质文化遗产。

现存最大的古代宫殿建筑, 也是``.
+2. ```[sMASK]```: 句子级别。我们限制遮罩
    spans必须是完整的句子。多个跨度
    （句子）被抽样覆盖 15%
    原始令牌。该目标旨在
diff --git a/doc_zh/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md b/doc_zh/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md
index 99d2ba82..e3134e4c 100644
--- a/doc_zh/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md
+++ b/doc_zh/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md
@@ -22,25 +22,24 @@
 
 ## 模型训练（train.py）
 
-运行前修改训练数据路径src_dir, tgt_dir, 模型路径model_dir。在命令行运行此命令：
+在命令行运行此命令：
 ```commandline
 cd ./examples/glm_title_generation
 python ./train.py
 ```
 
 ### 1.数据加载
-样例数据在 /examples/bert_title_generation/data/
+样例数据在 /examples/glm_title_generation/data/
 
+需要针对数据格式定义数据加载方法，例如：定义文件读取函数，从文件中读取数据，得到src和tgt列表：
 1）定义加载过程
 ```python
 def read_file():
     src = []
     tgt = []
-
-    ## read data file to load src and tgt, for example:
-    ## src = ["article_1", "article_2", "article_3" ......]
-    ## tgt = ["title_1", "title_2", "title_3" ......]
-    ## no matter what data you use, you need to construct the right src and tgt.
+    # src = ["article_1", "article_2", "article_3" ......]
+    # tgt = ["title_1", "title_2", "title_3" ......]
+    # 如果换为其他数据，修改处理方式即可，只需要构造好src以及对应tgt列表
     with open(src_dir, 'r', encoding='utf-8') as f:
         lines = f.readlines()
         for line in lines:
@@ -53,12 +52,12 @@ def read_file():
     return src,tgt
 ```
 
-2）定义数据集处理过程（Dataset）：
+2）定义数据集处理过程：
 ```python
-class GLMSeq2seqDataset(Dataset):
+class GLMTitleGenerationDataset(Dataset):
 
     def __init__(self, sents_src, sents_tgt):
-        super(GLMSeq2seqDataset, self).__init__()
+        super(GLMTitleGenerationDataset, self).__init__()
         self.sents_src = sents_src
         self.sents_tgt = sents_tgt
 
@@ -74,7 +73,7 @@ class GLMSeq2seqDataset(Dataset):
 
 3）定义数据迭代器（DataLoader）中的批处理函数（collate_fn），用于将一批（batch）数据填充（padding）成统一大小
 ```python
-class GLMSeq2seqDynamicCollateFN():
+class GLMTitleGenerationCollateFN():
     def __init__(self, pad_id):
         self.pad_id = pad_id
 
@@ -119,8 +118,8 @@ class GLMSeq2seqDynamicCollateFN():
 ```python
 train_src, train_tgt = read_file()
 print('-----------train data length:', len(train_src))
-my_collate_fn = GLMSeq2seqDynamicCollateFN(pad_id=tokenizer.get_command('pad').Id)
-train_dataset = GLMSeq2seqDataset(train_src,
+my_collate_fn = GLMTitleGenerationCollateFN(pad_id=tokenizer.get_command('pad').Id)
+train_dataset = GLMTitleGenerationDataset(train_src,
                                    train_tgt)
 ```
 ### 2.加载模型和分词器
@@ -128,12 +127,12 @@ train_dataset = GLMSeq2seqDataset(train_src,
 ```python
 from flagai.auto_model.auto_loader import AutoLoader
 
-# the model dir, which contains the 1.config.json, 2.pytorch_model.bin, 3.vocab.txt,
-# or we will download these files from the model hub to this dir.
+# model_dir: 包含 1.config.json, 2.pytorch_model.bin, 3.vocab.txt,
+# 如果本地没有，则会在modelhub上进行查找并下载
+# Autoloader 能够自动构建模型与切词器
+# 'title-generation' 是task_name
 model_dir = "./state_dict/glm/"
-# Autoloader can build the model and tokenizer automatically.
-# 'seq2seq' is the task_name.
-AutoLoader("seq2seq",model_name="GLM-large-ch",model_dir=model_dir)
+AutoLoader("title-generation",model_name="GLM-large-ch",model_dir=model_dir)
 model = auto_loader.get_model()
 tokenizer = auto_loader.get_tokenizer()
 ```
@@ -149,7 +148,7 @@ from flagai.trainer import Trainer
 device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
 trainer = Trainer(
     env_type="pytorch",
-    experiment_name="roberta_seq2seq",
+    experiment_name="glm-title-generation",
     batch_size=1,
     gradient_accumulation_steps=1,
     lr=2e-4,
@@ -183,4 +182,4 @@ python ./generate.py
 print(predictor.predict_generate_randomsample(text,out_max_length=66, top_k=10, top_p=.1,
                                       repetition_penalty=4.0, temperature=1.2))
 print(predictor.predict_generate_beamsearch(text, out_max_length=66, beam_size=10))
-```
\ No newline at end of file
+```
diff --git a/doc_zh/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md b/doc_zh/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md
index 8d9c7fa4..dfb76ff7 100644
--- a/doc_zh/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md
+++ b/doc_zh/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md
@@ -1,15 +1,23 @@
 # GLM 例子：古诗生成
 
 ## 古诗背景介绍
-中国古代近体诗有两种体裁：绝句和律诗。
+中国古代近体诗有两种体裁：绝句和律诗。绝句规定全诗四句。律诗规定全诗八句。每句话含五个或七个汉字，称为五言或七言。总共有四种类型，见下表。
 
-绝句规定全诗四句。其中五言绝句（五绝）为每句五个字，共二十个字；七言绝句（七绝）为每句七个字，共二十八个字。
+|     | 绝句 | 律诗 |
+|  ----  | ---- | ---- |
+| 五言 | 五言绝句 | 五言律诗 |
+| 七言 | 七言绝句 | 七言律诗 |
 
-律诗规定全诗八句。其中五言律诗（五律）为每句五个字，共四十个字；七言律诗（七律）为每句七个字，共五十六个字。
+一个五言绝句的示例：
 
-绝句在格律上虽讲究押韵，但对平仄和对仗要求不严。律诗在格律上要求严谨，押韵严格、讲究平仄和要求对仗。
+**静夜思** 李白
 
-## 结果展示
+床前明月光，疑是地上霜。
+
+举头望明月，低头思故乡。
+
+
+## 生成结果展示
 #### 输入古诗标题与体裁
 ```
 "桃花：七言绝句"
@@ -20,31 +28,47 @@
 ```
 ## 模型训练（train.py）
 
-运行前修改训练数据路径src_dir, tgt_dir, 模型路径model_dir。在命令行运行此命令：
+在命令行运行此命令：
 ```commandline
 cd ./examples/glm_poetry_generation
 python ./train.py
 ```
 这里使用`GLM-large-ch`作为样例,如果想要使用`GLM-10b-ch`请点[这里](https://model.baai.ac.cn/model-detail/100001)。
 ### 1.准备训练数据
-1）定义文件读取函数，从文件中读取数据，得到src和tgt列表：
+1）从文件中读取数据
+
+样例数据在 FlagAI/examples/glm_poetry_generation/data/
+
+需要在 train.py 中定义数据加载过程，得到src和tgt列表：
 ```python
 def read_file():
     src = []
     tgt = []
-
-    ##TODO read data file to load src and tgt, for example:
-    ## src = ["春晓：五言绝句", "标题：五言律诗",......]
-    ## tgt = ["春眠不觉晓，处处闻啼鸟。夜来风雨声，花落知多少。", "诗句...", ......]
-    ## no matter what data you use, you need to construct the right src and tgt.
-  
-    return src,tgt
+    # src = ["春晓：五言绝句", "标题：五言律诗",......]
+    # tgt = ["春眠不觉晓，处处闻啼鸟。夜来风雨声，花落知多少。", "诗句...", ......]
+    # 如果换为其他数据，修改处理方式即可，只需要构造好src以及对应tgt列表
+    with open(src_dir, 'r', encoding='utf-8') as f:
+        for line in f:
+            line = line.strip()
+            if "：" in line:
+                l = line.split("：")  #line eg:"初夏：五言绝句"
+                #if there are more than one '：', get title before the first '：'
+                title, style = l[0], l[-1]
+                if len(title) > 20:
+                    title = title[:20]  #cut the longer title
+                line = "：".join([title, style])
+            src.append(line)
+    with open(tgt_dir, 'r', encoding='utf-8') as f:
+        for line in f:
+            tgt.append(line.strip())
+    assert len(src) == len(tgt), 'lines not equal!'
+    return src, tgt
 ```
 2）定义数据迭代器（DataLoader）：
 ```python
-class BertSeq2seqDataset(Dataset):
+class GLMPoetryDataset(Dataset):
     def __init__(self, sents_src, sents_tgt):
-        super(BertSeq2seqDataset, self).__init__()
+        super(GLMPoetryDataset, self).__init__()
         self.sents_src = sents_src
         self.sents_tgt = sents_tgt
 
@@ -58,9 +82,9 @@ class BertSeq2seqDataset(Dataset):
     def __len__(self):
         return len(self.sents_src)
 ```
-其中tokenizer.encode_plus()方法将源、目标字符串转换为GLM模型的输入token id等数据
+其中tokenizer.encode_plus()方法将源、目标字符串转换为GLM模型的输入token索引、位置编码等数据
 
-3）定义数据迭代器（DataLoader）中的批处理函数（collate_fn），用于将一批（batch）数据填充（padding）成统一大小
+3）定义数据迭代器（DataLoader）中的collate_fn，用于将一批（batch）数据填充（padding）成统一大小
 ```python
 class GLMPoetryDynamicCollateFN():
     def __init__(self, pad_id):
@@ -108,7 +132,7 @@ class GLMPoetryDynamicCollateFN():
 train_src, train_tgt = read_file()
 print('-----------train data length:', len(train_src))
 my_collate_fn = GLMPoetryDynamicCollateFN(pad_id=tokenizer.get_command('pad').Id)
-train_dataset = BertSeq2seqDataset(train_src,
+train_dataset = GLMPoetryDataset(train_src,
                                    train_tgt)
 ```
 ### 2.加载模型和分词器
@@ -116,12 +140,12 @@ train_dataset = BertSeq2seqDataset(train_src,
 ```python
 from flagai.auto_model.auto_loader import AutoLoader
 
-# the model dir, which contains the 1.config.json, 2.pytorch_model.bin, 3.vocab.txt,
-# or we will download these files from the model hub to this dir.
-model_dir = "./state_dict/glm/"
-# Autoloader can build the model and tokenizer automatically.
-# 'seq2seq' is the task_name.
-AutoLoader("seq2seq",model_name="GLM-large-ch",model_dir=model_dir)
+# model_dir: 包含 1.config.json, 2.pytorch_model.bin, 3.vocab.txt,
+# 如果本地没有，则会在modelhub上进行查找并下载
+# Autoloader 能够自动构建模型与切词器
+# 'poetry' 是task_name
+model_dir = "./state_dict/"
+AutoLoader("poetry",model_name="GLM-large-ch",model_dir=model_dir)
 model = auto_loader.get_model()
 tokenizer = auto_loader.get_tokenizer()
 ```
@@ -132,30 +156,22 @@ tokenizer = auto_loader.get_tokenizer()
 ```python
 from flagai.trainer import Trainer
 trainer = Trainer(
-    env_type="pytorch",#pytorch or deepspeed
-    experiment_name="glm_seq2seq",
-    batch_size=64,#96
+    env_type="pytorch",
+    experiment_name="glm_poetry",
+    batch_size=4,
     gradient_accumulation_steps=1,
-    lr=2e-4,#2e-4
-    weight_decay=2e-8,#1e-3
+    lr=2e-4,
+    weight_decay=2e-8,
     epochs=100,
     log_interval=10,  
     tensorboard_dir="tbsummary",
     eval_interval=2000000,
-    load_dir="",
+    load_dir=None,
     save_dir="checkpoints_poetry",
     save_epoch=1,
-    num_checkpoints=1,
-    master_ip='127.0.0.1',
-    master_port=17750,
-    num_nodes=1,
-    num_gpus=2,
-    hostfile='./hostfile',
-    deepspeed_config='./deepspeed.json',
-    training_script=__file__,
 )
 ```
-将模型、数据、批处理函数输入训练器开始训练：
+将模型、数据、collate_fn输入训练器开始训练：
 ```python
 trainer.train(model,
               train_dataset=train_dataset,
@@ -166,7 +182,7 @@ trainer.train(model,
 
 
 ## 生成（generate.py）
-运行前修改模型配置路径model_dir，训练好的模型路径model_save_path。在命令行运行此命令：
+在命令行运行此命令：
 ```commandline
 cd ./examples/glm_poetry_generation
 python ./generate.py
diff --git a/doc_zh/TUTORIAL_14_HUGGINGFACE_T5.md b/doc_zh/TUTORIAL_14_HUGGINGFACE_T5.md
index e0dad603..42f94f00 100644
--- a/doc_zh/TUTORIAL_14_HUGGINGFACE_T5.md
+++ b/doc_zh/TUTORIAL_14_HUGGINGFACE_T5.md
@@ -159,18 +159,7 @@ trainer = MyTrainer(
 在forward阶段不将中间结果保存。我们可以运行`batch size`=1的t5-11b。
 现在，我们可以用 `gradient_accumulation_steps` train/finetune 一个 t5-11b。
 ```python
-trainer = MyTrainer(
-    env_type='pytorch',
-    epochs=1,
-    batch_size=1,
-    eval_interval=10,
-    log_interval=10,
-    experiment_name='t5-11b',
-    pytorch_device='cuda:0',
-    load_dir=None,
-    lr=1e-4,
-    fp16=True
-    checkpoint_activations = True) # setting as `True`
+model.gradient_checkpointing = True
 ```  
 ### 第三步：数据并行(DDP)
 为了增加batch size，我们可以在多个GPU上使用数据并行。
diff --git a/doc_zh/TUTORIAL_1_TOKENIZER.md b/doc_zh/TUTORIAL_1_TOKENIZER.md
index 1ec6448f..b7c957b3 100644
--- a/doc_zh/TUTORIAL_1_TOKENIZER.md
+++ b/doc_zh/TUTORIAL_1_TOKENIZER.md
@@ -1,4 +1,4 @@
-# 分词器
+# 如何构建和应用分词器
 
 ## 什么是分词？
 分词是 NLP 项目预处理阶段的一个基本步骤，其目的是将散乱的原始文本转换为规整的数字序列， 以便机器学习系统使用。
@@ -26,7 +26,7 @@
 from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
 tokenizer = GLMLargeEnWordPieceTokenizer()
 ```
-在这一步里，Modelhub中的词表文件将被自动下载到`cache_dir`参数中指定的路径。默认设置为分词器文件下的 ./vocab 目录。
+在这一步里，模型仓库中的词表文件将被自动下载到`cache_dir`参数中指定的路径。默认设置为分词器文件下的 ./vocab 目录。
 
 ## 应用分词器
 让我们使用一个分词器将原始文本编码成数字序列，然后将数字序列恢复成原始文本：
diff --git a/doc_zh/TUTORIAL_20_SUPPORTED_TASKS.md b/doc_zh/TUTORIAL_20_GLM_TNEWS.md
similarity index 100%
rename from doc_zh/TUTORIAL_20_SUPPORTED_TASKS.md
rename to doc_zh/TUTORIAL_20_GLM_TNEWS.md
diff --git a/doc_zh/TUTORIAL_20_SUPPORTED_TASKS_backup.md b/doc_zh/TUTORIAL_20_SUPPORTED_TASKS_backup.md
new file mode 100644
index 00000000..c36eaf24
--- /dev/null
+++ b/doc_zh/TUTORIAL_20_SUPPORTED_TASKS_backup.md
@@ -0,0 +1,1 @@
+# 用GLM10b模型在TNEWS短文本分类数据集上微调
diff --git a/doc_zh/TUTORIAL_2_DATASET.md b/doc_zh/TUTORIAL_2_DATASET.md
index 2e5d742c..41b71a62 100644
--- a/doc_zh/TUTORIAL_2_DATASET.md
+++ b/doc_zh/TUTORIAL_2_DATASET.md
@@ -5,13 +5,36 @@
 
 目前项目里存在三种数据预处理的情况，即对分类任务的微调，预训练, 以及对生成任务的微调。我们在接下来会分别展开。
 
-## 数据处理：分类任务微调([prompt-learning模式](TUTORIAL_7_PROMPT_LEARNING.md))
-### 应用代码
+<!-- toc -->
+
+- [1 分类任务微调](#分类任务微调)
+  - [1.1 应用代码](#分类任务应用代码)
+  - [1.2 初步读取并处理数据集](#初步读取并处理数据集)
+    - [1.2a 加载数据集](#加载数据集)
+    - [1.2b 统一数据集结构](#统一数据集结构)
+  - [1.3 将数据整理成模型的输入](#将数据整理成模型的输入)
+    - [1.3a 构建完形填空模板](#构建完形填空模板)
+    - [1.3b 分词并构造输入样例](#分词并构造输入样例)
+  - [1.4 创建加载器](#创建加载器)
+- [2 预训练](#预训练)
+  - [2.1 预训练任务数据格式样例](#预训练任务数据格式样例)
+  - [2.2 预训练的任务处理实例代码](#预训练的任务处理实例代码)
+- [3 生成任务微调](#生成任务微调)
+  - [3.1 生成任务应用代码](#生成任务应用代码)
+  - [3.2 生成任务里将数据整理成模型的输入](#生成任务里将数据整理成模型的输入)
+    - [3.2a 生成任务里构建填空模板](#生成任务里构建填空模板)
+    - [3.2b 生成任务里分词并构造输入样例](#生成任务里分词并构造输入样例)
+
+  
+
+## 分类任务微调
+分类任务微调有着两种形式：一种是普通的微调，一种是提示学习的方法。提示学习需要为任务额外构建一个完形填空的模板，它更适用于低资源以及小样本的情况。接下来我们以提示学习为例来介绍分类任务中的数据处理方法：
+### 分类任务应用代码
 ```python
-import torch.utils.data
-from flagai.data.dataset import SuperGlueDataset
+import torch
 from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
-from tests.test_dataset_new_superglue import CollateArguments
+from flagai.data.dataset import SuperGlueDataset
+from flagai.test_utils import CollateArguments
 from flagai.data.dataset import ConstructSuperglueStrategy
 
 # 得到默认参数
@@ -47,10 +70,21 @@ dataset = SuperGlueDataset(task_name='cb',
                            dataset_type='train',
                            tokenizer=tokenizer)
 ```
-#### 1.加载数据集
 
-将`task_name`设置成数据集对应的简称后，运行以上代码，相关数据会在后台自动下载。FlagAI目前支持自动加载下列分类数据集：
+其中`SuperGlueDataset`是我们这一步的主函数，它的主要参数及其作用如下所示：
+
+`task_name`: 数据集的简称, FlagAI支持的数据集及其对应的简称如[加载数据集](#加载数据集)所示。
+
+`data_dir`: 数据集会被自动下载到`data_dir`对应的地址，默认为项目的`./dataset`目录。
 
+`dataset_type`: 可以为 train/dev/test, 分别代表要处理的是训练集/验证集/测试集
+
+`tokenizer`: 如[Tutorial1](/doc_zh/TUTORIAL_1_TOKENIZER.md)所介绍的构建完成的分词器
+
+
+#### 加载数据集
+
+FlagAI目前支持自动加载下列分类数据集：
 
 | 数据集名称                                     | 数据集简称    | 语言   | 所属评测基准   |
 |----------------------------------------------|----------|------|----------|
@@ -64,8 +98,6 @@ dataset = SuperGlueDataset(task_name='cb',
 | Ant Financial Question Matching Corpus       | afqmc    | 中文   | CLUE     |
 | Short Text Classificaiton for News           | tnews    | 中文   | CLUE     |
 
-数据集会被自动下载到`data_dir`对应的地址，默认为项目的`./dataset`目录。
-
 下载好的数据集目录下会包含三个文件，对应训练集数据，验证集数据，以及测试集数据, 以`CommitmentBank`数据集为例，目录下的`train.jsonl`对应训练集，`val.jsonl`对应验证集，`test.jsonl`对应测试集。一般来说训练集和测试集会包含标签信息，而测试集则没有。这些数据文件在接下来的流程中会分开处理。
 
 不同的数据集可能会有不同的文件格式，以及不同的结构。以`CommitmentBank`数据集为例，下面是其中的一个样例
@@ -83,7 +115,7 @@ dataset = SuperGlueDataset(task_name='cb',
 
 目前所有FlagAI支持数据集的具体结构可以在[这里](DATASET_EXAMPLE.md)查看。
 
-#### 2. 统一数据集结构
+#### 统一数据集结构
 在这一步里，我们会统一不同数据集的数据结构，以方便接下来处理。此结构的细节如下：
 
 
@@ -97,17 +129,16 @@ dataset = SuperGlueDataset(task_name='cb',
 | meta   | an optional dictionary to store arbitrary meta information                        | dict |
 | ids    | an optional numeric index                                                    | int  |
 
-例如上一步`CommitmentBank`的样例会被处理成如下的形式
+如果想要查看某个样例的数据结构，也可以直接在代码里通过索引的方式得到：
+```python
+example = dataset[3]  # 数据集里第3个样例
+```
+比如说，上一步`CommitmentBank`的样例会被处理成如下的形式:
 
 <div align=center><img src="img/dataset_figure_2.png" width="500px"></div>
 
-需要注意的是如果因为数据结构太复杂，导致text_a和text_b无法塞下背景文本信息的话，可以把剩下的信息放在`meta`里。
+需要注意的是如果因为数据结构太复杂，导致text_a和text_b无法塞下背景文本信息的话，剩下的信息会被放在`meta`里。
 
-当数据集被构造好以后，可以直接在代码里通过索引的方式查看其中某个样例：
-
-```python
-example = dataset[3]  # 数据集里第3个样例
-```
 
 ### 将数据整理成模型的输入
 
@@ -118,7 +149,7 @@ collate_fn = ConstructSuperglueStrategy(cl_args,
                                         task_name=task_name)
 ```
 
-#### 1.构建完形填空模板
+#### 构建完形填空模板
 
 一个完形填空模板包含了背景文本，空位，以及在提供给空位的选项。模型需要找到正确的选项，并填进空位里。
 
@@ -128,7 +159,7 @@ collate_fn = ConstructSuperglueStrategy(cl_args,
 
 可以看到，大体上可以分成两步：第一步是组合已有的文本，使其看上去符合完形填空格式；第二步是将原始的标签文本转化为新的标签，作为可能会填入空位的选项。        
 
-#### 2.分词并构造输入样例
+#### 分词并构造输入样例
 接下来，我们需要构造模型的输入，第一步是分词，而接下来则需要分成两种情况：
 
 第一种情况下，数据集包含的标签类别是有限的，比如`CommitmentBank`数据集里只会存在`entailment/contradiction/neutral`三种标签文本。第二类情况里每一段完型填空都会给出不同的选项(一般是一段长文本)。比如在一些阅读理解数据集里，每一个选项都是一段对于文本的不同理解。这两类情况的处理方法如下所示：
@@ -148,7 +179,7 @@ collate_fn = ConstructSuperglueStrategy(cl_args,
 
 <sup>1</sup>: seq_length代表规定的每个输入向量的最大长度
 
-<sup>2</sup>: 特殊字符添加流程：在句首添加[CLS]符号，句尾添加[EOS]符号，直到长度达到seq_length, 如果完形填空输出的文本有两段，则在中间添加[SEP]符号
+<sup>2</sup>: 特殊字符添加流程如下图所示：在句首添加[CLS]符号，句尾添加[EOS]符号，直到长度达到seq_length。
 <div align=center><img src="img/dataset_example_4.png" width="500px"></div>
 
 <sup>3</sup>: num_labels代表完形填空问题中选项的个数
@@ -189,9 +220,105 @@ for data_iterator in loader:
 ```
 加载器构造好之后即可用于接下来的训练和预测过程。<br /><br /><br />
 
+### 自定义数据处理
+如果要处理不支持的数据集，需要您手动实现以下功能
+
+1. 在`flagai/data/dataset/superglue/control.py`文件里, 如下所示创建自定义的processor函数， 然后把它与自定义数据集名字的映射关系添加到control.py下的`PROCESSOR_DICT`字典里。
+
+其中最主要的函数是`_create_example`, 需要在这里读取对应的数据集，并如[之前](#统一数据集结构)所示初步处理成统一的结构。
+
+```python
+class ExampleProcessor(DataProcessor):
+
+    def get_train_examples(self, data_dir):
+        # Assign the filename of train set
+        return self._create_examples(os.path.join(data_dir, "train.tsv"), "train")
+
+    def get_dev_examples(self, data_dir, for_train=False):
+        # Assign the filename of dev set
+        return self._create_examples(os.path.join(data_dir, "dev.tsv"), "dev")
+
+    def get_test_examples(self, data_dir) -> List[InputExample]:
+        # Assign the filename of test set
+        return self._create_examples(os.path.join(data_dir, "test.tsv"), "test")
+
+    def get_labels(self):
+        # Return all label categories
+        return ["0", "1"]
+
+    @staticmethod
+    def _create_examples(path: str, set_type: str) -> List[InputExample]:
+        """
+        InputExample包含下列信息
+        text_a (str, required): 文本1
+        text_b (str, optional): 文本2
+        label (str, required): 标签
+        guid (str, required): 每一个InputExample的唯一序号
+        """
+        examples = []
+        df = read_tsv(path)
+
+        for idx, row in df.iterrows():
+            guid = f"{set_type}-{idx}"
+            text_a = punctuation_standardization(row['sentence'])
+            label = row.get('label', None)
+            example = InputExample(guid=guid, text_a=text_a, label=label)
+            examples.append(example)
+        return examples
+
+
+
+```
+2. 在`flagai/data/dataset/superglue/control.py`文件里, 如下所示创建自定义的pvp函数， 然后把它与自定义数据集名字的映射关系添加到control.py下的`PVPS`字典里。
+
+此函数的主要功能是如[之前](#将数据整理成模型的输入)所说的构建完形填空模板，其主要函数有
+
+`get_parts`: 构造完形填空文本
+
+`verbalize`：把数据集的标签映射到完形填空的选项上
+
+
+```python
+class ExamplePVP(PVP):
+    # 把标签映射到对应的含义文本上
+    VERBALIZER = {
+        "contradiction": [" No"],
+        "entailment": [" Yes"],
+        "neutral": [" Maybe"]
+    }
+
+    @staticmethod
+    def available_patterns():
+        # 输出所有可用的模板
+        return [0]
+
+    @property
+    def is_multi_token(self):
+        # 如果标签包含大于一个token，就输出True
+        return False
+
+    def get_parts(self, example: InputExample) -> FilledPattern:
+       # 把InputExample里面的元素通过设计组合成一个完形填空模板  
+
+        text_a = self.shortenable(example.text_a)
+        text_b = self.shortenable(" " + example.text_b)
+        parts_a, parts_b = [
+            None, text_a, None, ' question:', text_b,
+            ' true, false or neither?', None, ' answer:', [self.mask]
+        ], []
+        parts_a, parts_b = self.replace_prompt_tokens(parts_a, parts_b)
+        return parts_a, parts_b
+    
+    def verbalize(self, label) -> List[str]:
+        # 借用类里自定义的VERBALIZER完成初始标签到完形填空选项的映射
+        return [' true'] if label == 'entailment' else [
+            ' false'
+        ] if label == 'contradiction' else [' neither']
+
+```
 
 ## 数据处理：GLM预训练任务
-预训练任务数据格式样例：
+### 预训练任务数据格式样例
 ```text
 {
     "RECORDS": [
@@ -216,14 +343,19 @@ for data_iterator in loader:
     ]
 }
 ```
-预训练的任务处理实例代码:
+### 预训练的任务处理实例代码
 ```python
+from flagai.data.tokenizer import GLMLargeChTokenizer
+from flagai.data.dataset import BlockDataset
+from flagai.data.dataset.block.data_utils import split_ds, get_dataset_lazy, add_args
+from flagai.test_utils import PretrainDatasetArguments
+
 tokenizer = GLMLargeChTokenizer(add_block_symbols=True,
                                 add_task_mask=True,
                                 add_decoder_mask=False,
                                 fix_command_token=True)
 
-ds_args = DatasetArguments()
+ds_args = PretrainDatasetArguments()
 
 tokenizer = GLMLargeChTokenizer(fix_command_token=True,
                                 add_block_symbols=True,
@@ -261,23 +393,24 @@ datasets = create_dataset(tokenizer, should_split=True)
 3. 预训练的时候，collate function会随机按照三种不同的模式处理数据: bert模式（遮挡随机区间），sentence模式(按照完整句子进行遮挡)以及gpt模式（只遮挡一段长区间）。模型的输入相比一般生成任务也会多一个`mode`键。
 4. 预训练不用添加模板，直接按下表构建模型输入即可
 
-| 键值             | 维度                                   | 含义         | 构造方法                                             |
-|----------------|--------------------------------------|------------|--------------------------------------------------|
-| input_ids      | torch.Size([seq_length<sup>1</sup>]) | 输入矩阵       | 由上一步的模板文本，加上一些特殊字符组成                             |
-| position_ids   | torch.Size([2， seq_length])          | 位置编码         | 参考[GLM流程](GLM.md)，第一行代表token的绝对位置，第二行代表遮挡部分的相对位置 |
-| attention_mask | torch.Size([1])                      | 分隔符位置         | 对于生成类的模式，得到源文本结束的位置；否则得到输入文本结束的位置                                          |
-| target_ids     | torch.Size([num_labels<sup>3</sup>]) | 全量标签列表     | 被遮挡住的文本                                          |
-| logit_mask     | torch.Size([seq_length])             | 通过遮挡使得模型只会处理目标文本部分的loss | 对于每个token, 如果是回答，则对应的地方为1，否则为0                   |                                                   
-| mode           | str                                  | 数据处理模式 |                    |  
+| 键值             | 维度                                   | 含义                            | 构造方法                                             |
+|----------------|--------------------------------------|-------------------------------|--------------------------------------------------|
+| input_ids      | torch.Size([seq_length<sup>1</sup>]) | 输入矩阵                          | 由上一步的模板文本，加上一些特殊字符组成                             |
+| position_ids   | torch.Size([2， seq_length])          | 位置编码                          | 参考[GLM流程](GLM.md)，第一行代表token的绝对位置，第二行代表遮挡部分的相对位置 |
+| attention_mask | torch.Size([1])                      | 分隔符位置                         | 对于生成类的模式，得到源文本结束的位置；否则得到输入文本结束的位置                                          |
+| target_ids     | torch.Size([num_labels<sup>3</sup>]) | 全量标签列表                        | 被遮挡住的文本                                          |
+| logit_mask     | torch.Size([seq_length])             | 通过遮挡使得模型只会处理目标文本部分的loss       | 对于每个token, 如果是回答，则对应的地方为1，否则为0                   |                                                   
+| mode           | str                                  | 数据处理模式，取值范围：bert/sentence/gpt |                    |  
+
 <br /><br /><br />
 
-## 数据处理：生成任务微调
-代码实现如下所示：
+## 数据处理：生成任务微调 
+### 生成任务应用代码
 ```python 
-import torch.utils.data
+import torch
 from flagai.data.dataset import Seq2SeqDataset
 from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
-from tests.test_dataset_new_superglue import Seq2SeqCollateArguments
+from flagai.test_utils import CollateArguments
 from flagai.data.dataset import ConstructSeq2seqStrategy
 
 # 得到默认参数
@@ -308,9 +441,9 @@ loader = torch.utils.data.DataLoader(dataset,
                                     collate_fn=collate_fn)
 ```
 
-### 初步读取并处理数据集
+### 生成任务里初步读取并处理数据集
 
-目前支持[CMRC2018](https://www.clue.ai/introduce.html)任务，CMRC是一个阅读理解类的任务，需要根据背景文本来回答一系列提问，其数据结构示例如下：
+以[CMRC2018](https://www.clue.ai/introduce.html)为例，CMRC是一个阅读理解类的任务，需要根据背景文本来回答一系列提问，其数据结构示例如下：
 
 ```text
 {'paragraphs': 
@@ -340,7 +473,7 @@ dataset = Seq2SeqDataset(task_name='cmrc', data_dir='./datasets/',
                             dataset_type='train', tokenizer=tokenizer) 
 ```
 
-### 将数据整理成模型的输入
+### 生成任务里将数据整理成模型的输入
 
 代码如下所示。与生成任务相比，同样是构造模板以及模型输入，区别在于构造的方式不一样
 ```python 
@@ -349,12 +482,12 @@ collate_fn = ConstructSeq2seqStrategy(cl_args,
                                         task_name=task_name) 
 ```
 
-#### 1.构建填空模板
+#### 生成任务里构建填空模板
 由于是阅读理解任务，所以在模板里需要体现出是针对指定的阅读理解问题进行回答的，参考如下构建方式
 <div align=center><img src="img/dataset_figure_4.png" width="500px"></div>
 
 
 
-#### 2.分词并构造输入样例
+#### 生成任务里分词并构造输入样例
 与预训练类似，区别在于没有mode这个键值。
 
diff --git a/doc_zh/TUTORIAL_3_MODEL.md b/doc_zh/TUTORIAL_3_MODEL.md
index d29cc6ee..de77e367 100644
--- a/doc_zh/TUTORIAL_3_MODEL.md
+++ b/doc_zh/TUTORIAL_3_MODEL.md
@@ -1,72 +1,93 @@
-# 基类
+# 模型的主要功能及相关结构
+- [模型的主要功能及相关结构](#模型的主要功能及相关结构)
+  - [基类](#基类)
+  - [From_pretrain](#from_pretrain)
+    - [从modelhub加载](#从modelhub加载)
+    - [从本地加载](#从本地加载)
+  - [所有支持模型](#所有支持模型)
+  - [支持的模型+任务](#支持的模型任务)
+  - [模型设计](#模型设计)
+  - [模型的forward 函数](#模型的forward-函数)
+    - [输入](#输入)
+    - [输出](#输出)
+  - [模型的init_from_json函数](#模型的init_from_json函数)
+    - [输入](#输入-1)
+    - [输出](#输出-1)
+## 基类
 
-基类 BaseModel 实现了从本地文件或目录或从库提供的预训练模型配置（从BAAI modelhub 的 金山 S3 存储库下载）加载/保存模型的常用方法。
-现在所有支持的模型，对三种最常见的模型类型【encoder，decoder和encoder-decoder】进行了支持。现在GLM模型可以加载所有GLM系列的模型，详见 https://github.com/THUDM/GLM
+基类 BaseModel 实现了从本地文件或目录或从库提供的预训练模型配置（从BAAI modelhub 的 金山 S3 存储库下载）加载/保存模型的常用方法。现在对三种最常见的模型类型`encoder`，`decoder`和`encoder-decoder`进行了支持。同一种结构的模型，可以用同一个`Class`来进行加载。比如,`GLMModel`可以加载所有`GLM`系列的模型，详见 https://github.com/THUDM/GLM。
 
 ## From_pretrain
 
-同一个模型结构的模型可以用同一个class进行加载，比如BERT-base 和Roberta-base模型都能用BertModel这个class进行加载。From_pretrain为了数据/模型并行的模型加载进行了特定优化，避免重复下载导致的资源浪费。
-通过调用ClassName.from_pretrian()来进行加载，现在我们的model hub中对以下的模型进行了支持，可以直接下载模型配置文件【config.json】，模型权重[pytorch_model.bin]，以及字典文件[vocab.txt]。例子：
+`From_pretrain` 函数用于加载模型。同一个模型结构的模型可以用同一个class进行加载，比如`BERT-base-ch` 和`Roberta-base-ch`模型都能用`BertModel`这个`Class`进行加载。`From_pretrain`为了数据/模型并行的模型加载进行了特定优化，避免重复下载导致的资源浪费。
+通过调用`ClassName.from_pretrian()`来进行加载.
+### 从modelhub加载
+现在我们支持从modelhub中下载[常用模型](#所有支持模型)，可以直接通过`from_pretrain`下载模型配置文件`config.json`，模型权重`pytorch_model.bin`，以及字典文件`vocab.txt`。例子：
 ```python
-from flagai.model.glm_model import GLMForSingleTokenCloze
-model = GLMForSingleTokenCloze.from_pretrain(download_path="./state_dict", model_name="GLM-large-ch")
+#从modelhub下载GLM-large-ch 模型
+from flagai.model.glm_model import GLMModel
+model = GLMModel.from_pretrain(download_path="./state_dict", model_name="GLM-large-ch")
 ```
-如果是从本地加载模型权重，也可以通过 ClassName.from_pretrain()进行加载。例子：
-从`./state_dict/GLM-large-ch`目录中加载模型文件 `pytorch_model.bin`
+### 从本地加载
+如果想从本地加载模型权重，也可以通过 `ClassName.from_pretrain()`进行加载。模型权重保存在`download_path/model_name/`中,其中`model_name`是模型相关所在的目录，而`download_path`是`model_name`所在的目录，例子：
+
 ```python
-from flagai.model.glm_model import GLMForSingleTokenCloze
-model = GLMForSingleTokenCloze.from_pretrain(download_path="./state_dict",
-                                               model_name="GLM-large-ch")
+#从`./state_dict/GLM-large-ch`目录中加载模型文件 `pytorch_model.bin`和模型配置文件 `config.json`。
+from flagai.model.glm_model import GLMModel
+model = GLMModel.from_pretrain(download_path="./state_dict", model_name="GLM-large-ch")
 ```
 ## 所有支持模型
+这些模型都可以通过`from_pretrain` 从model_hub下载。
 
 | ClassName                         | ModelName       | Language | Model Type |
 |-----------------------------------|-----------------|----------|------------|
-| flagai.model.glm_model.GLMModel   | GLM-10b-ch      | chinese  | encoder    |
-| flagai.model.glm_model.GLMModel   | GLM-large-ch    | chinese  | encoder    |
-| flagai.model.bert_model.BertModel | RoBERTa-base-ch | chinese  | encoder    |
-| flagai.model.gpt2_model.GPT2Model | GPT2_base_ch    | chinese  | decoder    |
-| flagai.model.t5_model.T5Model     | T5-base-ch      | chinese  | enc2dec    |
-| flagai.model.t5_model.T5Model     | T5-base-en      | chinese  | enc2dec    |
-| flagai.model.bert_model.BertModel | BERT-base-en    | english  | encoder    |
-| flagai.model.glm_model.GLMModel   | GLM-large-en    | english  | encoder    |
+| flagai.model.glm_model.GLMModel   | **GLM-10b-ch**      | chinese  | encoder    |
+| flagai.model.glm_model.GLMModel   | **GLM-large-ch**   | chinese  | encoder    |
+| flagai.model.bert_model.BertModel | **RoBERTa-base-ch** | chinese  | encoder    |
+| flagai.model.gpt2_model.GPT2Model | **GPT2_base_ch**    | chinese  | decoder    |
+| flagai.model.t5_model.T5Model     | **T5-base-ch**     | chinese  | enc2dec    |
+| flagai.model.t5_model.T5Model     | **T5-base-en**      | chinese  | enc2dec    |
+| flagai.model.bert_model.BertModel | **BERT-base-en**    | english  | encoder    |
+| flagai.model.glm_model.GLMModel   | **GLM-large-en**    | english  | encoder    |
 
 ## 支持的模型+任务
-
 同时，我们对在任务上finetune好的模型进行了支持，如下表所示，可以通过ClassName.from_pretrain()来加载模型权重，例如，我们自动下载并加载一个在title-generation任务上训练好的GLM-large-ch模型：
 ```python
 from flagai.model.glm_model import GLMForSeq2Seq
 model = GLMForSeq2Seq.from_pretrain(model_name='GLM-large-ch')
 ```
-我们也提供了AutoLoader类来帮助加载模型，比如GLM-large-ch模型用于seq2seq任务，这里我们采用了任务和模型独立的设计，理论上任务和模型可以自由更换。
+为了简化加载流程，我们也提供了AutoLoader类来帮助加载模型，比如GLM-large-ch模型用于seq2seq任务。由于我们采用了任务和模型独立的设计，理论上任务和模型可以自由更换。
 ```python
 from flagai.auto_model.auto_loader import AutoLoader
-auto_loader = AutoLoader("seq2seq",
+auto_loader = AutoLoader("title-generation",
                          model_name="GLM-large-ch",
                          model_dir= "./state_dict")
 model = auto_loader.get_model()
 ```
 | ClassName                                       | Model Name      | language | Task              |
 |-------------------------------------------------|-----------------|----------|-------------------|
-| flagai.model.glm_model.GLMForSeq2Seq            | GLM-large-ch    | chinese  | title generation  |
-| flagai.model.glm_model.GLMForSeq2Seq            | GLM-large-ch    | chinese  | poetry generation |
-| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese  | title generation  |
-| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese  | NER               |
-| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese  | semantic matching |
-| flagai.model.t5_model.T5Model                   | T5-base-ch      | chinese  | title generation  |
-| flagai.model.bert_model.BertForSequenceLabeling | BERT-base-en    | english  | title gneration   |
+| flagai.model.glm_model.GLMForSeq2Seq            | GLM-large-ch    | chinese  | **title generation**  |
+| flagai.model.glm_model.GLMForSeq2Seq            | GLM-large-ch    | chinese  | **poetry generation** |
+| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese  | **title generation**  |
+| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese  | **NER**               |
+| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese  | **semantic matching** |
+| flagai.model.t5_model.T5Model                   | T5-base-ch      | chinese  | **title generation**  |
+| flagai.model.bert_model.BertForSequenceLabeling | BERT-base-en    | english  | **title gneration**   |
 
 ## 模型设计
-模型主要的构建逻辑`layer->block>model`
+模型主要的构建逻辑`layer->block->model`
+
 `flagai.model.layer`: 包括mlp，layernorm, activation，attention等各种layer层
 
 `flagai.model.block`:通过组装各种layer来构建transformer block，比如BERT block等
 
 `flagai.model`: 通过embedding层和stacked blocks 来构建model
 
-## forward 函数
-Model 的forward函数：
-输入是 keyword arguments：包括 input_ids, position_ids, attention_mask等，对冗余的参数会自动忽略
+如果想要自定义一个新的模型结构，可以参考上述构建过程。
+
+## 模型的forward 函数
+### 输入
+输入都是 keyword arguments：包括 input_ids, position_ids, attention_mask等，对冗余的参数会自动忽略
 比如GLM的forward 函数：
 ```python
 def forward(self,
@@ -79,15 +100,19 @@ def forward(self,
             prompt_pos=None,
             **kwargs)
 ```
+### 输出
 输出是 dictionary，包括 logits 和hidden states，这两个是必须的，例如GLM forword函数的返回：
 ```python
 return {'loss': loss, 'logits': logits, 'hidden_states': mems}
 ```
-## init_from_json
-Model 的init_from json函数：
-输入是一个dictionary， 输出是一个初始化的model
+## 模型的init_from_json函数
+### 输入
+输入是一个`json`，以及预留参数 `**kwags`，为了兼容一些模型新增的初始化参数
+
 例如GLMModel的调用如下：
 ```python
-GLMModel.init_from_json(config_file = "./config.json", **kwargs)
+GLMModel.init_from_json(config_file = "./config.json", checkpoint_activations=True)
 ```
-**kwargs是预留参数，为了兼容一些模型新增的初始化参数
\ No newline at end of file
+`checkpoint_activations=True`是新增的参数，用于控制是否进行梯重计算
+### 输出
+输出是一个初始化的model
\ No newline at end of file
diff --git a/doc_zh/TUTORIAL_4_TRAINER.md b/doc_zh/TUTORIAL_4_TRAINER.md
index cefacefb..ebe39595 100644
--- a/doc_zh/TUTORIAL_4_TRAINER.md
+++ b/doc_zh/TUTORIAL_4_TRAINER.md
@@ -1,12 +1,37 @@
-# Trainer
+# 为模型和数据并行训练定制训练器
+- [Trainer](#trainer)
+  - [入门](#入门)
+  - [Trainer主要参数env_type](#trainer主要参数env_type)
+  - [自定义Trainer](#自定义trainer)
+  - [单节点cpu/gpu](#单节点cpugpu)
+  - [fp16](#fp16)
+  - [梯度重计算](#梯度重计算)
+    - [GLM-10b-ch](#glm-10b-ch)
+    - [huggingface t5-11b](#huggingface-t5-11b)
+  - [完整的用Trainer训练huggingface t5-11b例子](#完整的用trainer训练huggingface-t5-11b例子)
+  - [分布式训练](#分布式训练)
+    - [deepspeed](#deepspeed)
+    - [pytorchDDP](#pytorchddp)
+    - [deepspeed + megatron-lm](#deepspeed--megatron-lm)
+
 
 Trainer 类提供了API用于多种并行框架的训练。API 支持在多个 GPU上使用Pytorch DDP/Deepspeed进行分布式训练，同时支持Megatron-LM+Deepspeed的混合并行分布式训练，同时也通过 NVIDIA Apex 实现混合精度。
 ## 入门
 Trainer 包含支持上述功能的基本训练循环。Trainer使用分成两个步骤，初始化和调用. 参考 `examples/glm_superglue` 目录下的代码.
 
+## Trainer主要参数env_type
+Trainer 中有一个控制是否分布式训练的参数  env_type，
+```shell
+The enviroment type for training. Will default to 'pytorch'.
+env_type: `pytorch`, `pytorchDDP`, `deepspeed`, `deepspeed+mpu`
+            pytorch: single node cpu/gpu
+            pytorchDDP: single-/multi- node gpu <data parallel>
+            deepspeed: single-/multi- node gpu <data/pipline parallel>
+            deepspeed+mpu: single-/multi- node gpu <data parallel + model parallel>
+```            
+
 ## 自定义Trainer
-要自定义Trainer来快速支持自定义模型，您可以对它们进行子类化并覆盖forward_step方法：
-forward_step方法的返回是一个dict
+在使用自定义模型的时候，模型的输入输出与`FlagAI`框架中模型的行为不一致时（参考 [模型forward函数介绍](TUTORIAL_3_MODEL.md#模型的forward-函数)），需要自定义Trainer来进行训练。要自定义`Trainer`来快速支持自定义模型，您可以它们进行子类化并覆盖`forward_step`方法. 这里需要注意：`forward_step`方法的返回是一个dict
 ```python
 from flagai.trainer import Trainer
 class MyTrainer(Trainer):
@@ -21,23 +46,14 @@ class MyTrainer(Trainer):
         return output
 ```
 
-Trainer 中有一个控制是否分布式训练的参数  env_type，
-```shell
-The enviroment type for training. Will default to 'pytorch'.
-env_type: `pytorch`, `pytorchDDP`, `deepspeed`, `deepspeed+mpu`
-            pytorch: single node cpu/gpu
-            pytorchDDP: single-/multi- node gpu <data parallel>
-            deepspeed: single-/multi- node gpu <data/pipline parallel>
-            deepspeed+mpu: single-/multi- node gpu <data parallel + model parallel>
-```                          
-
 ## 单节点cpu/gpu
+可以看到这里的Trainer使用的是`pytorch` `cpu`的环境
 ```python
 trainer = MyTrainer(
     env_type='pytorch',
     epochs=1,
     batch_size=4,
-    eval_interval=100000,
+    eval_interval=10,
     log_interval=10,
     experiment_name='t5-11b',
     pytorch_device='cpu',
@@ -46,8 +62,8 @@ trainer = MyTrainer(
 ```
 指定显卡/cpu的设置为pytorch_device, 'cpu', 'cuda:0'等
 ## fp16 
+在模型参数大，而显存空间又很紧张的时候，可以通过将`fp32`的参数转化为`fp16`的参数来降低显存的使用量.FlagAI可以通过一个参数来启动。
 ```python
-Model parameters turned to fp16
 trainer = MyTrainer(
     env_type='pytorch',
     epochs=1,
@@ -61,37 +77,38 @@ trainer = MyTrainer(
     fp16=True) # change to `True`
 ```
 ## 梯度重计算
-在前向过程中不保存中间变量，节约GPU显存.  Paper: [Training Deep Nets with Sublinear Memory Cost](https://arxiv.org/abs/1604.06174v2)
+在前向过程中不保存中间变量，节约GPU显存.  Paper: [Training Deep Nets with Sublinear Memory Cost](https://arxiv.org/abs/1604.06174v2)。下面，我们举了两个百亿模型开启梯度重计算的例子，一个是从FlagAI加载百亿参数模型，一个是加载huggingface的百亿模型。
 
+### GLM-10b-ch
+```python
+#从modelhub下载GLM-10b-ch 模型并开启梯度重计算
+from flagai.model.glm_model import GLMModel
+model = GLMModel.from_pretrain(download_path="./state_dict", model_name="GLM-large-ch", checkpoint_activations=True)
+```
+### huggingface t5-11b
+t5-11b模型的 paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)
 ```python
 from transformers import T5ForConditionalGeneration, T5Tokenizer
 tokenizer = T5Tokenizer.from_pretrained('t5-11b')
 model = T5ForConditionalGeneration.from_pretrained('t5-11b')
 model.gradient_checkpointing = True
 ```
-## 支持huggingface model
-FlagAI 项目的example目录位置：examples/t5_huggingface
-t5-11b paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)
+为了展示FlagAI的Trainer的扩展能力，下面我们用训练T5-11b模型作为例子。**注意**: 百亿规模以上的模型权重在20G+，单块显卡需要V100 及以上的硬件。
+
+## 完整的用Trainer训练huggingface t5-11b例子
+FlagAI 项目的example目录位置：`examples/t5_huggingface
+t5-11b`
 
-The example train t5-11b on a toy dataset.  You need  a device > 30G memory to run the example. If your have multiple devices,  please check out the following session for speedup.
 ```python
-import sys
 from flagai.trainer import Trainer
 from transformers import T5ForConditionalGeneration, T5Tokenizer
 from torch.utils.data import Dataset
 import torch
 
-## Inheriant the Trainer
-## overload the forward_step function
+
 class MyTrainer(Trainer):
 
     def forward_step(self, data, model, mems):
-        """
-        Args:
-            data: a dict contains a batch of inputs
-        return:
-            output: a dict contains `loss`
-        """
         model_outputs = model(**data)
         output = {}
         output['loss'] = model_outputs.loss
@@ -99,28 +116,34 @@ class MyTrainer(Trainer):
         output['hidden_states'] = model_outputs.decoder_hidden_states
         return output
 
-# get a customized trainer instance
+
 trainer = MyTrainer(
-    env_type='pytorch',
+    env_type='deepspeed',
     epochs=1,
-    batch_size=4,
+    batch_size=1,
     eval_interval=10,
     log_interval=10,
     experiment_name='t5-11b',
-    pytorch_device='cuda:0',
     load_dir=None,
-    lr=1e-4,
-    fp16=False)
+    lr=1e-4
+    # parameters for pytorchDDP
+    master_ip='127.0.0.1',
+    master_port=17750,
+    num_nodes=1,
+    num_gpus=1,
+    training_script=__file__,
+    # deepspeed
+    deepspeed_config='deepspeed.json'
+)
 
-# using huggingface transformers to get tokenizer and models
 model_name = 't5-11b'
 tokenizer = T5Tokenizer.from_pretrained(model_name)
 model = T5ForConditionalGeneration.from_pretrained(model_name)
+model.gradient_checkpointing = True
 
 print("loading model & tokenizer is done!")
-src_dir = 'train_inputs.txt'
-tgt_dir = 'train_targets.txt'
-model_dir = "./t5-11b"  # 模型位置
+src_dir = './data/train.src'
+tgt_dir = './data/train.tgt'
 maxlen = 1024
 
 
@@ -140,10 +163,10 @@ def read_file():
     return src, tgt
 
 
-class BertSeq2seqDataset(Dataset):
+class T5Seq2seqDataset(Dataset):
 
     def __init__(self, sents_src, sents_tgt, tokenizer, maxlen=512):
-        super(BertSeq2seqDataset, self).__init__()
+        super(T5Seq2seqDataset, self).__init__()
         self.sents_src = sents_src
         self.sents_tgt = sents_tgt
         self.tokenizer = tokenizer
@@ -190,80 +213,101 @@ def seq2seq_collate_fn(batch):
 sents_src, sents_tgt = read_file()
 data_len = len(sents_tgt)
 train_size = int(data_len * 0.8)
-train_src = sents_src[:train_size]
-train_tgt = sents_tgt[:train_size]
+train_src = sents_src[:train_size][:200]
+train_tgt = sents_tgt[:train_size][:200]
 
 val_src = sents_src[train_size:]
 val_tgt = sents_tgt[train_size:]
 
-train_dataset = BertSeq2seqDataset(train_src,
-                                   train_tgt,
-                                   tokenizer=tokenizer,
-                                   maxlen=maxlen)
-val_dataset = BertSeq2seqDataset(val_src,
-                                 val_tgt,
+train_dataset = T5Seq2seqDataset(train_src,
+                                 train_tgt,
                                  tokenizer=tokenizer,
                                  maxlen=maxlen)
-## Training
+val_dataset = T5Seq2seqDataset(val_src,
+                               val_tgt,
+                               tokenizer=tokenizer,
+                               maxlen=maxlen)
+
 trainer.train(model,
               train_dataset=train_dataset,
               collate_fn=seq2seq_collate_fn)
 ```
 
 ## 分布式训练
-To accelerate your training/finetuning process, FlagAI integrate three popular parallel frameworks for deep neural networks.
-### pytorchDDP
-DistributedDataParallel (DDP) link
-我们可以通过修改几个参数就能使用分布式的训练，内部直接封装了launch的调用，直接python script_name.py  [script_name:是训练脚本的名字] 就可以直接运行。
-```python
-trainer = MyTrainer(
-    env_type='pytorchDDP',
-    epochs=1,
-    batch_size=4,
-    eval_interval=10,
-    log_interval=10,
-    experiment_name='t5-11b',
-    load_dir=None,
-    lr=1e-4
-    # parameters for pytorchDDP
-    master_ip='127.0.0.1',
-    master_port=17750,
-    num_nodes=1,
-    num_gpus=2,
-    hostfile='./hostfile',
-    training_script=__file__,
-)
-```
+为了加速模型训练，飞智支持三种模式的并行方式，但是在上述训练T5-11b的例子中，只能使用`deepspeed`框架的数据并行模式。
+
 ### deepspeed
-deepspeed相对于pytorch主要在优化器方面做了优化。
-其提供了cpu-offload的optimizer，可以极大的降低gpu显存的占用。
-对deepspeed优化器的相关设置，需要提供deepspeed.json的配置文件 
+deepspeed主要在优化器方面做了优化，其提供了cpu-offload的optimizer，可以极大的降低gpu显存的占用。对deepspeed优化器的相关设置，需要提供`deepspeed.json`的配置文件 
+```shell
+{
+  "train_micro_batch_size_per_gpu": 2,
+  "gradient_accumulation_steps": 1,
+  "steps_per_print": 100,
+  "gradient_clipping": 1.0,
+  "zero_optimization": {
+    "stage": 3,
+    "contiguous_gradients": false,
+    "overlap_comm": true,
+    "reduce_scatter": true,
+    "reduce_bucket_size": 5e7,
+    "allgather_bucket_size": 5e7,
+    "cpu_offload": true 
+  },
+  "zero_allow_untested_optimizer": true,
+  "fp16": {
+    "enabled": true,
+    "loss_scale": 0,
+    "loss_scale_window": 1000,
+    "hysteresis": 2,
+    "min_loss_scale": 1
+  },
+  "optimizer": {
+    "type": "Adam",
+    "params": {
+      "lr": 0.0004,
+      "weight_decay": 0.01,
+      "betas": [
+        0.9,
+        0.98
+      ],
+      "eps": 1e-6
+    }
+  },
+  "activation_checkpointing": {
+    "partition_activations": false,
+    "contiguous_memory_optimization": false
+  },
+  "wall_clock_breakdown": false
+}
+```
+`deepspeed_config`文件参考 `examples/t5_huggingface/deepspeed.json`.其中，主要的参数为 `stage` 和 `cpu_offload`.
+
+`hostfile`在单节点时可以省略
+### pytorchDDP
+DistributedDataParallel (DDP) 在模型的参数规模<1 billion,比如`t5-base`的时候可以使用。我们可以通过修改`env_type`参数就能完成分布式框架的切换.
+
 ```python
 trainer = MyTrainer(
     env_type='pytorchDDP',
     epochs=1,
-    batch_size=4,
+    batch_size=1,
     eval_interval=10,
     log_interval=10,
-    experiment_name='t5-11b',
+    experiment_name='t5-base',
     load_dir=None,
     lr=1e-4
     # parameters for pytorchDDP
     master_ip='127.0.0.1',
     master_port=17750,
     num_nodes=1,
-    num_gpus=2,
+    num_gpus=1,
     hostfile='./hostfile',
     training_script=__file__,
-    # deepspeed
-    deepspeed_config='deepspeed.json'
 )
 ```
-Deepspeed config文件参考 [examples/t5_huggingface/deepspeed.json]
-其中，主要的参数为 stage 和 cpu offload
 ### deepspeed + megatron-lm
-现在百亿级模型GLM采用了Megatron-LM的模型并行技术。在模型参数scaling到10b以上级别，单卡的显存就很难将单个模型以及训练时的中间变量全部加载进来。为此，Megatron-LM提供了Tensor的切分方法，主要思想是将矩阵按照行/列进行切分。 FlagAI link，将model转化为Megatron-LM版本。
-如下，FlagAI内部模型（GLM，T5，BERT【包括RoBERTa】，GPT2）支持了Megatron-LM，只要在配置文件中将环境变量修改为deepspeed+mpu，就能启动模型并行的功能。
+现在百亿级模型GLM-10-ch采用了`Megatron-LM`的模型并行技术加上`deepspeed`的数据并行技术。在模型参数scaling到10b以上级别，单卡的显存就很难将单个模型以及训练时的中间变量全部加载进来。为此，Megatron-LM提供了Tensor的切分方法，主要思想是将矩阵按照行/列进行切分。 FlagAI 将model转化为Megatron-LM版本。
+如下，飞智内部模型（GLM，T5，BERT【包括RoBERTa】，GPT2）支持了Megatron-LM，只要在配置文件中将环境变量修改为deepspeed+mpu，就能启动模型并行的功能。
 对于huggingface版本的模型，暂时没有提供模型并行的支持。
 ```python
 trainer = MyTrainer(
@@ -272,7 +316,7 @@ trainer = MyTrainer(
     batch_size=8,
     eval_interval=10,
     log_interval=10,
-    experiment_name='t5-11b',
+    experiment_name='GLM-10b-ch',
     load_dir=None,
     lr=1e-4,
     # parallel settings
diff --git a/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md b/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md
index 8c5cb5a8..3448e816 100644
--- a/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md
+++ b/doc_zh/TUTORIAL_5_INSTRUCTIONS_FOR_AutoLoader.md
@@ -1,4 +1,4 @@
-## AutoLoader
+## 使用 Autoloader 简化模型和分词器初始化过程
 
 ### 通过 AutoLoader 快速构建支持不同下游任务的大型模型。
 ![](./img/autoloader_map.png)
@@ -42,7 +42,7 @@ model = auto_loader.get_model()
 tokenizer = auto_loader.get_tokenizer()
 ```
 查看 https://github.com/BAAI-Open/FlagAI/blob/master/quickstart/ner_ch.py 了解更多.
-## All supported tasks and models
+## 所有支持的任务和模型
 
 ![model_and_task_table](../docs/img/model_task_table.png)
  **task_name** 参数支持:
@@ -69,8 +69,8 @@ tokenizer = auto_loader.get_tokenizer()
 6. chinese-word-segmentation (todo)
 7. ...
 
-## 所有支持的模型
-所有支持的模型都可以在 **model hub** 中找到。
+### 所有支持的模型
+所有支持的模型都可以在 **[这里](./TUTORIAL_3_MODEL.md#所有支持模型)** 中找到。
 不同的模型适应不同的任务。
 
 #### Transfomrer encoder:
diff --git a/doc_zh/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md b/doc_zh/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md
index 46b11b4b..b072ddb9 100644
--- a/doc_zh/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md
+++ b/doc_zh/TUTORIAL_6_INSTRUCTIONS_FOR_PREDICTOR.md
@@ -1,4 +1,4 @@
-## Predictor
+## 将现成的推理算法与 Predictor 结合使用
 
 ### 通过Predictor调用一行代码得到理想的预测结果
 
diff --git a/doc_zh/TUTORIAL_7_PROMPT_LEARNING.md b/doc_zh/TUTORIAL_7_PROMPT_LEARNING.md
index ada0e194..84a35c06 100644
--- a/doc_zh/TUTORIAL_7_PROMPT_LEARNING.md
+++ b/doc_zh/TUTORIAL_7_PROMPT_LEARNING.md
@@ -1,4 +1,4 @@
-# 提示学习介绍
+# 使用 FlagAI 提示学习工具包来提高在SuperGLUE任务上的表现
 
 ## 背景
 提示学习(prompt-learning)是一种自然语言处理 (NLP) 领域广泛使用的范式，尤其是在数据有限的情况下表现良好。它遵循预训练-微调范式的思想：即预训练好的语言模型将被适配到各种下游任务上。不过与微调不同的是，提示学习会将原本的任务转化为一个特定的模板，这个过程被称为提示工程。
@@ -19,43 +19,19 @@
 
 我们可以设计一个如下的模板：
 
-<div align=center><img src="img/prompt_figure_1.png" width="500px"></div>
-
-相关的代码实现如下：
-
-```python
-def get_parts(self, example: InputExample):  
-    '''A simplified version of the actual code'''
-    # Construct the template
-    cloze_ids = [
-        example.text_a,' question: is it true, false or neither that',
-         example.text_b, ' answer:', [self.mask]]
-    return cloze_ids
-```
-
-各种数据集中的标签可以有多种类型，包括布尔值、整数和字符串。我们需要实现一个语言器，将标签映射到完形填空问题中的选项。例如，这里的“entailment”、“contradiction”和“neutral”标签可以映射到“true”、“false”和“neither”，并且每次要填的答案总是在这些选项中选择。
-
-语言器本质上是一种映射关系, 所以我们可以用一个Python字典来定义它：
-```python
-VERBALIZER = {
-    "contradiction": [" false"],
-    "entailment": [" true"],
-    "neutral": [" neither"]
-}
-def verbalize(self, label) -> List[str]:
-    return CbPVP.VERBALIZER[label]
-```
+<div align=center><img src="img/dataset_example_3.png" width="500px"></div>
 
 ### 前缀提示
 前缀提示并没有用真正的自然语言设计模板，而是直接修改嵌入空间。如下图所示，它在Transformer里插入一个前缀向量，并冻结语言模型的其余参数。与完形填空提示对比，这么做相当于从离散的tokens变成了连续的向量，避免了人工设计模板的好坏对于模型性能的影响过大的问题。
 <div align=center><img src="img/prompt_figure_2.png" width="600px"></div>
 
 
-## Reference
-[Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://arxiv.org/abs/2107.13586)
+## 引用
 
-[OpenPrompt: An Open-source Framework for Prompt-learning](https://arxiv.org/abs/2111.01998)
+Liu, P. (2021, July 28). Pre-train, Prompt, and Predict: A Systematic Survey of Prompting. . . arXiv.Org. https://arxiv.org/abs/2107.13586 
 
-[Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://arxiv.org/abs/2001.07676)
+Ding, N. (2021, November 3). OpenPrompt: An Open-source Framework for Prompt-learning. arXiv.Org. https://arxiv.org/abs/2111.01998
 
-[Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://arxiv.org/abs/2101.00190)
\ No newline at end of file
+Schick, T. (2020, January 21). Exploiting Cloze Questions for Few Shot Text Classification and. . . arXiv.Org. https://arxiv.org/abs/2001.07676
+
+Li, X. L. (2021, January 1). Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv.Org. https://arxiv.org/abs/2101.00190
diff --git a/doc_zh/TUTORIAL_8_ENVIRONMENT_SETUP.md b/doc_zh/TUTORIAL_8_ENVIRONMENT_SETUP.md
index 8712c78f..78f27017 100644
--- a/doc_zh/TUTORIAL_8_ENVIRONMENT_SETUP.md
+++ b/doc_zh/TUTORIAL_8_ENVIRONMENT_SETUP.md
@@ -1,4 +1,41 @@
-# 预训练分布式环境搭建
+# 多机训练模型搭建环境
+
+- [多机训练模型搭建环境](#多机训练模型搭建环境)
+- [一.  Docker](#一--docker)
+  - [1.安装docker](#1安装docker)
+  - [2.Docker 换源](#2docker-换源)
+  - [3.安装显卡驱动（如已装可跳过）](#3安装显卡驱动如已装可跳过)
+  - [4.配置nvidia-docker源：](#4配置nvidia-docker源)
+  - [5.制作dockerfile](#5制作dockerfile)
+    - [a.拉取nvidia 基础镜像, 创建临时文件夹（容器内，镜像创建完成后，删除）](#a拉取nvidia-基础镜像-创建临时文件夹容器内镜像创建完成后删除)
+    - [b.配置apt 安装源,并安装一些linux 系统常用基础包](#b配置apt-安装源并安装一些linux-系统常用基础包)
+    - [c.  安装最新版git(创建镜像clone 安装包)](#c--安装最新版git创建镜像clone-安装包)
+    - [d. 安装  Mellanox OFED, 由于网络问题，推荐安装包下到本地后，再执行dockerfile](#d-安装--mellanox-ofed-由于网络问题推荐安装包下到本地后再执行dockerfile)
+    - [e. 安装 nv_peer_mem](#e-安装-nv_peer_mem)
+    - [f. 安装openmpi, 需先安装libevent 依赖包](#f-安装openmpi-需先安装libevent-依赖包)
+    - [g.安装 python](#g安装-python)
+    - [h.安装 magma-cuda](#h安装-magma-cuda)
+    - [i.配置路径](#i配置路径)
+    - [j.安装一些pip 包](#j安装一些pip-包)
+    - [k.安装mpi4py （需下载到本地安装，pip 安装可能因为版本兼容问题报错）](#k安装mpi4py-需下载到本地安装pip-安装可能因为版本兼容问题报错)
+    - [l.安装pytorch, 版本可替换， 需先下载项目到本地，国内安装容易因为网速原因，造成终止, pytorch git clone 过程中可能有些子包下载过程中会终止。可以多 git clone 几次](#l安装pytorch-版本可替换-需先下载项目到本地国内安装容易因为网速原因造成终止-pytorch-git-clone-过程中可能有些子包下载过程中会终止可以多-git-clone-几次)
+    - [m.安装apex](#m安装apex)
+    - [n.安装deepspeed](#n安装deepspeed)
+    - [o.安装NCCL(可选，pytorch 已自带)](#o安装nccl可选pytorch-已自带)
+    - [p.配置网络端口、公钥和ssh](#p配置网络端口公钥和ssh)
+  - [6.构建docker 镜像](#6构建docker-镜像)
+    - [a.方式一.  pull 镜像](#a方式一--pull-镜像)
+    - [b.方式二.  构建镜像](#b方式二--构建镜像)
+- [二. 在每个机器节点构建容器](#二-在每个机器节点构建容器)
+- [三. 互信机制设置](#三-互信机制设置)
+  - [1. 公钥生成默认docker 镜像创建时已生成，如不存在，则在shell 端输入](#1-公钥生成默认docker-镜像创建时已生成如不存在则在shell-端输入)
+  - [2.将各节点容器生成的公钥文件](#2将各节点容器生成的公钥文件)
+  - [3.免密登陆](#3免密登陆)
+  - [4.测试](#4测试)
+- [四.  分布式训练测试](#四--分布式训练测试)
+  - [a.配置hostfile（hostfile 中的V100-1 与~/.ssh/config 对应）:](#a配置hostfilehostfile-中的v100-1-与sshconfig-对应)
+  - [b. 配置glm 文件，各节点配置code 和数据，要求路径相同（也可共同访问云端共享文件）](#b-配置glm-文件各节点配置code-和数据要求路径相同也可共同访问云端共享文件)
+  - [c. cmd](#c-cmd)
 
 # 一.  Docker
 
@@ -29,11 +66,12 @@ apt-get install -y docker-ce
 
 ## 2.Docker 换源
 
-### (https://xxxx.mirror.aliyuncs.com) 为自己的docker源仓库
+(https://xxxx.mirror.aliyuncs.com) 为自己的docker源仓库
 
 ```shell
 mkdir -p /etc/docker
-tee /etc/docker/daemon.json <<-'EOF'
+tee /etc/docker/daemon.json 
+-'EOF'
 {
   "registry-mirrors": ["https://xxxx.mirror.aliyuncs.com"]
 }
@@ -80,7 +118,7 @@ apt-get update
 apt-get install -y nvidia-docker2
 ````
 
-### 修改/etc/docker/daemon.json，添加相关信息
+修改/etc/docker/daemon.json，添加相关信息
 
 ```text
 "runtimes": {
@@ -90,7 +128,7 @@ apt-get install -y nvidia-docker2
    }
 }
 ```
-### /etc/docker/daemon.json最终内容
+/etc/docker/daemon.json最终内容
 
 ```json
 {
@@ -104,7 +142,7 @@ apt-get install -y nvidia-docker2
 }
 ```
 
-### 重启docker服务
+重启docker服务
 
 ```shell
 systemctl daemon-reload
diff --git a/doc_zh/TUTORIAL_9_SEQ2SEQ_METHOD.md b/doc_zh/TUTORIAL_9_SEQ2SEQ_METHOD.md
index 52388174..2a85c838 100644
--- a/doc_zh/TUTORIAL_9_SEQ2SEQ_METHOD.md
+++ b/doc_zh/TUTORIAL_9_SEQ2SEQ_METHOD.md
@@ -1,18 +1,32 @@
-# Seq2seq方法
+# 使用encoder/decoder/encoder-decoder模型进行文本生成
 
-## 编码器模型
+支持三种方式进行序列到序列任务的生成，包括
+1. 只用Encoder模型进行序列到序列的生成任务 
+2. 只用Decoder模型进行序列到序列的生成任务 
+3. 用Encoder+Decoder模型进行序列到序列的生成任务。
+
+## Encoder 模型
 我们提供了encoder模型来执行seq2seq任务，例如，Bert、Roberta、GLM等等。
+
+1. [使用RoBERTa模型进行标题生成](https://github.com/BAAI-Open/FlagAI/tree/master/examples/roberta_title_generation)
+2. [使用GLM模型进行标题生成](https://github.com/BAAI-Open/FlagAI/tree/master/examples/glm_title_generation)
+   
 在训练过程中，我们在编码器模型中添加了一个特殊的 attention mask。(https://github.com/microsoft/unilm)
 ![encoder_mask](../docs/img/encoder_mask.png)
 该模型的输入是两个句子：[cls]句子1[sep]句子2[sep]。
 其中，句子_1不使用mask，而句子_2使用自回归mask。
 
-## 解码器模型
+## Decoder 模型
 我们还为seq2seq任务提供了decoder模型，例如gpt-2模型。
+1. [使用GPT-2模型进行文章续写](https://github.com/BAAI-Open/FlagAI/blob/master/quickstart/writing_ch.py)
+
 ![decoder_mask](../docs/img/decoder_mask.png)
 给出一个起始文本，这个模型可以很好地延续文本。
 
-## 编解码器模型
+## Encoder-Decoder 模型
 我们还为seq2seq任务提供encoder-decoder模型，例如T5模型。
+1. [使用T5模型进行标题生成](https://github.com/BAAI-Open/FlagAI/tree/master/examples/t5_title_generation)
+
+
 ![encoder_decoder_mask](../docs/img/encoder_decoder_mask.png)
 encoder只需编码一次即可获得特征编码，decoder根据自身和特征编码继续生成。
\ No newline at end of file
diff --git a/doc_zh/img/dataset_figure_3.png b/doc_zh/img/dataset_figure_3.png
index be869428..20e38840 100644
Binary files a/doc_zh/img/dataset_figure_3.png and b/doc_zh/img/dataset_figure_3.png differ
diff --git a/doc_zh/img/prompt_figure_1.png b/doc_zh/img/prompt_figure_1.png
index 28dbd5fd..c67ba6d1 100644
Binary files a/doc_zh/img/prompt_figure_1.png and b/doc_zh/img/prompt_figure_1.png differ
diff --git a/doc_zh/img/tokenize.png b/doc_zh/img/tokenize.png
new file mode 100644
index 00000000..71888a1a
Binary files /dev/null and b/doc_zh/img/tokenize.png differ
diff --git a/doc_zh/tokenization.md b/doc_zh/tokenization.md
index f22461f7..1ede7e80 100644
--- a/doc_zh/tokenization.md
+++ b/doc_zh/tokenization.md
@@ -1,5 +1,3 @@
-[TOC]
-
 # sub-word algorithm
 
 ## Tokenization
@@ -38,7 +36,7 @@
 
 ​		三种不同的切分方法效果对比如下：
 
-![3 different tokenization scheme: on rules, on punctuaction, on white spaces](tokenization.assets/tokenize.png)
+![3 different tokenization scheme: on rules, on punctuaction, on white spaces](img/tokenize.png)
 
 ​		上面说的三种分词方法，都是词级别的分词方法（word tokenization），简单来说，就是以词为单位对文本进行切分。尽管这是将文本切分为更小的单位的最直观的方式，但是这种方式在处理大量语料的时候会产生一个非常严重的问题——生成一个巨大的词表（因为大量的语料中会存在大量的只出现一次/少次的token）。
 
@@ -182,14 +180,17 @@
 
 ​		与 BPE 的主要区别在于，选择两个子词进行合并的规则：BPE 选择频数最高的相邻子词合并，而 WordPiece 使用的是通过语言模型来计算合并两个单词可能造成的影响，然后选择使得似然函数提升最大的字符对。这个提升是通过结合后的字符对减去结合前的字符对之和得到的。
 
-​		假设句子$S=（t1,t2,...,t_n）$由 n 个 token 组成，$t_i$表示第$i$个 token，假设各个 token 之间是相互独立的，则句子$S$的语言模型似然值等于所有子词概率的乘积
-$$
-logP(S) = \sum^n_{i=1}logP(t_i)
-$$
+​		假设句子$ S=（t1,t2,...,t_n）$由 n 个 token 组成，$t_i$表示第$i$个 token，假设各个 token 之间是相互独立的，则句子$S$的语言模型似然值等于所有子词概率的乘积
+
+<img src="https://render.githubusercontent.com/render/math?math={logP(S) = \sum^n_{i=1}logP(t_i)}} = -1">
+`$ logP(S) = \sum^n_{i=1}logP(t_i) $`
+
+<img src="https://render.githubusercontent.com/render/math?math={\color{white}\L = -\sum_{j}[T_{j}ln(O_{j})] + \frac{\lambda W_{ij}^{2}}{2} \rightarrow \text{one-hot} \rightarrow -ln(O_{c}) + \frac{\lambda W_{ij}^{2}}{2}}">
+
 ​		假设把相邻位置的两个 token $x$ 和 $y$ 进行合并，合并后的 token 记为 $z$，此时句子 $S$ 的似然值变化可以表示为：
-$$
+`$
 logP(t_z)-(logP(t_x)+logP(t_y)) = log(\frac{P(t_z)}{P(t_x)P(T_y)})
-$$
+$`
 ​		也就是说，判断`tokenization`相较于`token` + `ization`是否更适合出现。选择能够提升语言模型概率最大的相邻子词加入词表。
 
 
diff --git a/docs/GLM.md b/docs/GLM.md
index 4c1e7a68..598510a0 100644
--- a/docs/GLM.md
+++ b/docs/GLM.md
@@ -9,12 +9,14 @@ The **GLM model**, proposed in [GLM: General Language Model Pretraining
 with Autoregressive Blank Infilling](https://arxiv.org/abs/2103.10360), is based on a slightly different strategy: autoregressive blank infilling. 
 
 It claims to perform well in the three main categories of NLP taks: classification, unconditional generation, and conditional generation tasks.
+
 | Framwork        | NLU | Cond.Gen. | Uncond.Gen |
 |-----------------|-----|-----------|------------|
 | Augoregressive  | -   | -         | ✅          |
 | Autoencoding    | ✅   | ×         | ×          |
 | Encoder-Decoder | -   | ✅         | -          |
 | GLM             | ✅   | ✅         | ✅          |
+
 The key features of GLM include:
 
 - First task: Several spans of the text are masked following the idea of autoencoding. Those spans will be randomly rearranged and be predicted in an autoregressive manner. The masked spans covers 15% original tokens.
diff --git a/docs/TUTORIAL_10_MEGATRON.md b/docs/TUTORIAL_10_MEGATRON.md
index ce056f6e..c3915290 100644
--- a/docs/TUTORIAL_10_MEGATRON.md
+++ b/docs/TUTORIAL_10_MEGATRON.md
@@ -1,8 +1,5 @@
 # Turn model into Megatron-LM version
-Three steps：
-- [1] Turn`MLP`layer into`column/rowParallel` version
-- [2] Turn two `Linear layer` in `self-Attention` into `column/rowParallel `version
-- [3] Turn two `Linear Layer` in `cross-Attention`into `column/rowParallel`version
+
 
 Most of the process in `parallel` are taken from `Megatron-LM`，and is put in `mpu`module
 
@@ -40,204 +37,25 @@ self.num_attention_heads_per_partition = divide(
     num_attention_heads, world_size)
 ```
 Here is the code for code of two `parallel linear`
-### ColumnParallelLinear
-```python
-
-class ColumnParallelLinear(torch.nn.Module):
-    """Linear layer with column parallelism.
-
-    The linear layer is defined as Y = XA + b. A is parallelized along
-    its second dimension as A = [A_1, ..., A_p].
-
-    Arguments:
-        input_size: first dimension of matrix A.
-        output_size: second dimension of matrix A.
-        bias: If true, add bias
-        gather_output: If true, call all-gether on output and make Y avaiable
-                       to all GPUs, otherwise, every GPU will have its output
-                       which is Y_i = XA_i
-        init_method: method to initialize weights. Note that bias is always set
-                     to zero.
-        stride: For the strided linear layers.
-        keep_master_weight_for_test: This was added for testing and should be
-                                     set to False. It returns the master weights
-                                     used for initialization.
-    """
-
-    def __init__(self,
-                 input_size,
-                 output_size,
-                 bias=True,
-                 gather_output=True,
-                 init_method=init.xavier_normal_,
-                 stride=1,
-                 keep_master_weight_for_test=False):
-        super(ColumnParallelLinear, self).__init__()
-
-        # Keep input parameters
-        self.input_size = input_size
-        self.output_size = output_size
-        self.gather_output = gather_output
-        # Divide the weight matrix along the last dimension.
-        world_size = get_model_parallel_world_size()
-        self.output_size_per_partition = divide(output_size, world_size)
-
-        # Parameters.
-        # Note: torch.nn.functional.linear performs XA^T + b and as a result
-        # we allocate the transpose.
-        self.weight = Parameter(
-            torch.Tensor(self.output_size_per_partition, self.input_size))
-        self.weight.model_parallel = True
-        if bias:
-            self.bias = Parameter(torch.Tensor(self.output_size_per_partition))
-            self.bias.model_parallel = True
-            # Always initialize bias to zero.
-            with torch.no_grad():
-                self.bias.zero_()
-        else:
-            self.register_parameter('bias', None)
-
-        # Initialize weight.
-        self.master_weight = _initialize_affine_weight(
-            self.weight,
-            self.output_size,
-            self.input_size,
-            self.output_size_per_partition,
-            0,
-            init_method,
-            stride=stride,
-            return_master_weight=keep_master_weight_for_test)
-
-    def forward(self, input_):
-        # Set up backprop all-reduce.
-        input_parallel = copy_to_model_parallel_region(input_)
-        # Matrix multiply.
-        output_parallel = F.linear(input_parallel, self.weight, self.bias)
-        if self.gather_output:
-            # All-gather across the partitions.
-            output = gather_from_model_parallel_region(output_parallel)
-            # gather_from_model_parallel_region 是mpu中提供的功能
-        else:
-            output = output_parallel
-        return output
-```
-### RowParallelLinear
-```python
-class RowParallelLinear(torch.nn.Module):
-    """Linear layer with row parallelism.
-
-    The linear layer is defined as Y = XA + b. A is parallelized along
-    its first dimension and X along its second dimension as:
-               -   -
-              | A_1 |
-              | .   |
-          A = | .   |        X = [X_1, ..., X_p]
-              | .   |
-              | A_p |
-               -   -
-    Arguments:
-        input_size: first dimension of matrix A.
-        output_size: second dimension of matrix A.
-        bias: If true, add bias. Note that bias is not parallelized.
-        input_is_parallel: If true, we assume that the input is already
-                           split across the GPUs and we do not split
-                           again.
-        init_method: method to initialize weights. Note that bias is always set
-                     to zero.
-        stride: For the strided linear layers.
-        keep_master_weight_for_test: This was added for testing and should be
-                                     set to False. It returns the master weights
-                                     used for initialization.
-    """
 
-    def __init__(self,
-                 input_size,
-                 output_size,
-                 bias=True,
-                 input_is_parallel=False,
-                 init_method=init.xavier_normal_,
-                 stride=1,
-                 keep_master_weight_for_test=False):
-        super(RowParallelLinear, self).__init__()
-
-        # Keep input parameters
-        self.input_size = input_size
-        self.output_size = output_size
-        self.input_is_parallel = input_is_parallel
-        # Divide the weight matrix along the last dimension.
-        world_size = get_model_parallel_world_size()
-        self.input_size_per_partition = divide(input_size, world_size)
-
-        # Parameters.
-        # Note: torch.nn.functional.linear performs XA^T + b and as a result
-        # we allocate the transpose.
-        self.weight = Parameter(
-            torch.Tensor(self.output_size, self.input_size_per_partition))
-        self.weight.model_parallel = True
-        if bias:
-            self.bias = Parameter(torch.Tensor(self.output_size))
-            # Always initialize bias to zero.
-            with torch.no_grad():
-                self.bias.zero_()
-        else:
-            self.register_parameter('bias', None)
-
-        # Initialize weight.
-        self.master_weight = _initialize_affine_weight(
-            self.weight,
-            self.output_size,
-            self.input_size,
-            self.input_size_per_partition,
-            1,
-            init_method,
-            stride=stride,
-            return_master_weight=keep_master_weight_for_test)
-
-    def forward(self, input_):
-        # Set up backprop all-reduce.
-        if self.input_is_parallel:
-            input_parallel = input_
-        else:
-            input_parallel = scatter_to_model_parallel_region(input_)
-        # Matrix multiply.
-        output_parallel = F.linear(input_parallel, self.weight)
-        # All-reduce across all the partitions.
-        output_ = reduce_from_model_parallel_region(output_parallel)
-        # reduce_from_model_parallel_region 是mpu提供的功能
-        if self.bias is not None:
-            output = output_ + self.bias
-        else:
-            output = output_
-        return output
-```
-## 2.Turn  two Linear layerin `self-Attention`into`column/rowParallel` version
-Location：`flagai/model/layers/attentions_mpu.py`
+## 2. Transform `self-Attention`
+Turn  two Linear layer in `self-Attention`into`column/rowParallel` version. Location：`flagai/model/layers/attentions_mpu.py`
 ### 2.1 Turn the linear layer that projects`x` to  `q, k, v`  into `collumnParalleLinear`
-### 2.2 Turn the output`dense`layer into`rowparallelLinear`
 ```python
 self.query_key_value = ColumnParallelLinear(hidden_size,
                                                 3 * hidden_size,
                                                 stride=3,
                                                 gather_output=False,
                                         init_method=init_method)
-if relative_encoding:
-    self.relative = ColumnParallelLinear(hidden_size,
-                                         hidden_size,
-                                         gather_output=False,
-                                         init_method=init_method)
-# Dropout. Note that for a single iteration, this layer will generate
-# different outputs on different number of parallel partitions but
-# on average it should not be partition dependent.
-self.attention_dropout = torch.nn.Dropout(attention_dropout_prob)
-
-# Output.
+```
+### 2.2 Turn the output`dense`layer into`rowparallelLinear`
+```python
 self.dense = RowParallelLinear(hidden_size,
                                hidden_size,
                                input_is_parallel=True,
                                init_method=output_layer_init_method)
 ```
-## 3.Turn two `Linear Layer ` in `cross-Attention`into`column/rowParallel`version
-as above
+## 3.Transform two `Linear Layer` in `cross-Attention` into `column/rowParallel` version as above
 ```python
 self.query = ColumnParallelLinear(hidden_size,
                                   hidden_size,
@@ -248,11 +66,6 @@ self.key_value = ColumnParallelLinear(hidden_size,
                                       stride=2,
                                       gather_output=False,
                                       init_method=init_method)
-# Dropout. Note that for a single iteration, this layer will generate
-# different outputs on different number of parallel partitions but
-# on average it should not be partition dependent.
-self.attention_dropout = torch.nn.Dropout(attention_dropout_prob)
-
 # Output.
 self.dense = RowParallelLinear(hidden_size,
                                hidden_size,
diff --git a/docs/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md b/docs/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md
index 782933da..a16d8c90 100644
--- a/docs/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md
+++ b/docs/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md
@@ -22,14 +22,14 @@ The title generation task needs to input a piece of text, and the model outputs
 
 ## Model Train（train.py）
 
-Modify the training data path src_dir, tgt_dir, and model path model_dir before running. Run this command at the command line:
+Run this command at the command line:
 ```commandline
-cd ./examples/glm_title_generation
+cd FlagAI/examples/glm_title_generation
 python ./train.py
 ```
 
 ### 1.Load data
-Sample data is at /examples/bert_title_generation/data/
+Sample data is at /examples/glm_title_generation/data/
 
 1）Define the load function
 ```python
@@ -37,10 +37,10 @@ def read_file():
     src = []
     tgt = []
 
-    ## read data file to load src and tgt, for example:
-    ## src = ["article_1", "article_2", "article_3" ......]
-    ## tgt = ["title_1", "title_2", "title_3" ......]
-    ## no matter what data you use, you need to construct the right src and tgt.
+    # read data file to load src and tgt, for example:
+    # src = ["article_1", "article_2", "article_3" ......]
+    # tgt = ["title_1", "title_2", "title_3" ......]
+    # no matter what data you use, you need to construct the right src and tgt.
     with open(src_dir, 'r', encoding='utf-8') as f:
         lines = f.readlines()
         for line in lines:
@@ -55,10 +55,10 @@ def read_file():
 
 2）Define the data loading process
 ```python
-class GLMSeq2seqDataset(Dataset):
+class GLMTitleGenerationDataset(Dataset):
 
     def __init__(self, sents_src, sents_tgt):
-        super(GLMSeq2seqDataset, self).__init__()
+        super(GLMTitleGenerationDataset, self).__init__()
         self.sents_src = sents_src
         self.sents_tgt = sents_tgt
 
@@ -74,7 +74,7 @@ class GLMSeq2seqDataset(Dataset):
 
 3) Define the batch function (collate_fn) in the data iterator (DataLoader) to pad a batch of data into a uniform size
 ```python
-class GLMSeq2seqDynamicCollateFN():
+class GLMTitleGenerationCollateFN():
     def __init__(self, pad_id):
         self.pad_id = pad_id
 
@@ -119,8 +119,8 @@ class GLMSeq2seqDynamicCollateFN():
 ```python
 train_src, train_tgt = read_file()
 print('-----------train data length:', len(train_src))
-my_collate_fn = GLMSeq2seqDynamicCollateFN(pad_id=tokenizer.get_command('pad').Id)
-train_dataset = GLMSeq2seqDataset(train_src,
+my_collate_fn = GLMTitleGenerationCollateFN(pad_id=tokenizer.get_command('pad').Id)
+train_dataset = GLMTitleGenerationDataset(train_src,
                                    train_tgt)
 ```
 ### 2.Load model and tokenizer
@@ -133,7 +133,7 @@ from flagai.auto_model.auto_loader import AutoLoader
 model_dir = "./state_dict/glm/"
 # Autoloader can build the model and tokenizer automatically.
 # 'seq2seq' is the task_name.
-AutoLoader("seq2seq",model_name="GLM-large-ch",model_dir=model_dir)
+AutoLoader("title-generation",model_name="GLM-large-ch",model_dir=model_dir)
 model = auto_loader.get_model()
 tokenizer = auto_loader.get_tokenizer()
 ```
@@ -149,7 +149,7 @@ from flagai.trainer import Trainer
 device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
 trainer = Trainer(
     env_type="pytorch",
-    experiment_name="roberta_seq2seq",
+    experiment_name="glm-title-generation",
     batch_size=1,
     gradient_accumulation_steps=1,
     lr=2e-4,
@@ -176,7 +176,7 @@ trainer.train(model,
 Modify the model configuration path model_dir and the trained model path model_save_path before running. Run this command at the command line:
 
 ```commandline
-cd ./examples/glm_title_generation
+cd FlagAI/examples/glm_title_generation
 python ./generate.py
 ```
 You can choose between random sampling based on probability screening (random sample) or beam search (beamsearch) two generation methods:
diff --git a/docs/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md b/docs/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md
index 93caef47..339ed645 100644
--- a/docs/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md
+++ b/docs/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md
@@ -1,13 +1,12 @@
 # GLM example: Classical Chinese Poetry Generation
 
 ## Introduction to the Background of Classical Chinese Poetry
-There are two types of classical Chinese poetry: Jueju(绝句), Lvshi(律诗).
-
-The Jueju contains four lines in the whole poem. The five-character Jueju (wujue) has five characters per sentence, with a total of twenty characters; the seven-character Jueju (qijue) has seven characters per sentence, with a total of twenty-eight characters.
-
-There are eight lines in the Lvshi. The five-character Lvshi (wulv) has five characters per sentence, with a total of forty characters; the seven-character Lvshi (qilv) has seven characters per sentence, with a total of fifty-six characters.
-
+There are two types of classical Chinese poetry: Jueju(绝句), Lvshi(律诗).The Jueju contains four lines in the whole poem. The Lvshi contains eight lines in the whole poem.Each line contains five or seven Chinese characters. There are four types in total, see the table below.
 
+|     | Jueju | Lvshi |
+|  ----  | ---- | ---- |
+| five characters | wujue | wulv |
+| seven characters | qijue | qilv |
 
 ## Result show
 #### Input ancient poem title and type:
@@ -20,30 +19,47 @@ There are eight lines in the Lvshi. The five-character Lvshi (wulv) has five cha
 ```
 ## Model training（train.py）
 
-Modify the training data path (**src_dir**, **tgt_dir**) and **model_dir** before running the file. Input the code in commandline to train:
+Input the code in commandline to train:
 ```commandline
 cd ./examples/glm_poetry_generation
 python ./train.py
 ```
 ### 1.Prepare the training data
 1）Define the file reading function:
+
+The sample data is in FlagAI/examples/glm_poetry_generation/data/
+
+You need to define the data loading process in train.py, and get the list of **src** and **tgt**:
 ```python
 def read_file():
     src = []
     tgt = []
-
-    ##TODO read data file to load src and tgt, for example:
     ## src = ["春晓：五言绝句", "标题：五言律诗",......]
     ## tgt = ["春眠不觉晓，处处闻啼鸟。夜来风雨声，花落知多少。", "诗句...", ......]
     ## no matter what data you use, you need to construct the right src and tgt.
-  
+    with open(src_dir, 'r', encoding='utf-8') as f:
+        for line in f:
+            line = line.strip()
+            if "：" in line:
+                l = line.split("：")  #line eg:"初夏：五言绝句"
+                #if there are more than one '：', get title before the first '：'
+                title, style = l[0], l[-1]
+                if len(title) > 20:
+                    title = title[:20]  #cut the longer title
+                line = "：".join([title, style])
+            src.append(line)
+    with open(tgt_dir, 'r', encoding='utf-8') as f:
+        for line in f:
+            tgt.append(line.strip())
+    assert len(src) == len(tgt), 'lines not equal!'
+    return src, tgt
     return src,tgt
 ```
 2）Define the DataLoader:
 ```python
-class BertSeq2seqDataset(Dataset):
+class GLMPoetryDataset(Dataset):
     def __init__(self, sents_src, sents_tgt):
-        super(BertSeq2seqDataset, self).__init__()
+        super(GLMPoetryDataset, self).__init__()
         self.sents_src = sents_src
         self.sents_tgt = sents_tgt
 
@@ -107,7 +123,7 @@ class GLMPoetryDynamicCollateFN():
 train_src, train_tgt = read_file()
 print('-----------train data length:', len(train_src))
 my_collate_fn = GLMPoetryDynamicCollateFN(pad_id=tokenizer.get_command('pad').Id)
-train_dataset = BertSeq2seqDataset(train_src,
+train_dataset = GLMPoetryDataset(train_src,
                                    train_tgt)
 ```
 ### 2.Load model and tokenizer
@@ -131,27 +147,19 @@ Instantiate the Trainer, and set the training parameters.
 ```python
 from flagai.trainer import Trainer
 trainer = Trainer(
-    env_type="pytorch",#pytorch or deepspeed
-    experiment_name="glm_seq2seq",
-    batch_size=64,#96
+    env_type="pytorch",
+    experiment_name="glm_poetry",
+    batch_size=4,
     gradient_accumulation_steps=1,
-    lr=2e-4,#2e-4
-    weight_decay=2e-8,#1e-3
+    lr=2e-4,
+    weight_decay=2e-8,
     epochs=100,
     log_interval=10,  
     tensorboard_dir="tbsummary",
     eval_interval=2000000,
-    load_dir="",
+    load_dir=None,
     save_dir="checkpoints_poetry",
     save_epoch=1,
-    num_checkpoints=1,
-    master_ip='127.0.0.1',
-    master_port=17750,
-    num_nodes=1,
-    num_gpus=2,
-    hostfile='./hostfile',
-    deepspeed_config='./deepspeed.json',
-    training_script=__file__,
 )
 ```
 Pass the model, data, and collate_fn into the trainer to start training:
diff --git a/docs/TUTORIAL_14_HUGGINGFACE_T5.md b/docs/TUTORIAL_14_HUGGINGFACE_T5.md
index acd18f81..b78227fd 100644
--- a/docs/TUTORIAL_14_HUGGINGFACE_T5.md
+++ b/docs/TUTORIAL_14_HUGGINGFACE_T5.md
@@ -159,18 +159,7 @@ trainer = MyTrainer(
 Do not save the itermedia results in forward stage. Now you may run t5-11b with `batch size`=1.
 Now, we can train/finetune a t5-11b with `gradient_accumulation_steps`.
 ```python
-trainer = MyTrainer(
-    env_type='pytorch',
-    epochs=1,
-    batch_size=1,
-    eval_interval=10,
-    log_interval=10,
-    experiment_name='t5-11b',
-    pytorch_device='cuda:0',
-    load_dir=None,
-    lr=1e-4,
-    fp16=True
-    checkpoint_activations = True) # setting as `True`
+model.gradient_checkpointing = True
 ```  
 ### step3. data parallel (DDP)
 To multiply your batch size, we can use data paralle on multiple GPUs.
diff --git a/docs/TUTORIAL_1_TOKENIZER.md b/docs/TUTORIAL_1_TOKENIZER.md
index bc38f365..bb116385 100644
--- a/docs/TUTORIAL_1_TOKENIZER.md
+++ b/docs/TUTORIAL_1_TOKENIZER.md
@@ -13,7 +13,9 @@ input text `Jack is walking a dog.` is shown below:
 <div align=center><img src="img/tokenizer_example_1.png" width="500px"></div>
 
 It is noticeable that different tokenizers can have different ways to split text,
-and have different vocabulary files. An introduction to those algorithms can be viewed [here](tokenization.md).
+and have different vocabulary files. 
+
+[//]: # (An introduction to those algorithms can be viewed [here]&#40;tokenization.md&#41;.)
 
 Our projects currently support six tokenizers
 as listed below:
diff --git a/docs/TUTORIAL_2_DATASET.md b/docs/TUTORIAL_2_DATASET.md
index 230c419c..222ef66b 100644
--- a/docs/TUTORIAL_2_DATASET.md
+++ b/docs/TUTORIAL_2_DATASET.md
@@ -5,14 +5,16 @@ The process of constructing a dataset is the data preprocessing process of NLP.
 
 At present, there are three kinds of data preprocessing in the project, namely, fine-tuning for classification tasks, pre-training, and fine-tuning for generation tasks. We will expand on them separately in the following.
 
-## Data processing: fine-tuning for classification tasks ([prompt-learning mode](TUTORIAL_7_PROMPT_LEARNING.md))
+## Data processing: fine-tuning for classification tasks 
+
+There are two forms of tuning for classification tasks: one is fine-tuning, and the other is the [prompt-tuning](/docs/TUTORIAL_7_PROMPT_LEARNING.md). Prompt-tuning requires an additional cloze template for the task, which is more suitable for limited data. Let's take prompt learning as an example to introduce the data processing method in classification tasks:
 ### Application code
 
 ```python
-import torch.utils.data
-from flagai.data.dataset import SuperGlueDataset
+import torch
 from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
-from tests.test_dataset_new_superglue import CollateArguments
+from flagai.data.dataset import SuperGlueDataset
+from flagai.test_utils import CollateArguments
 from flagai.data.dataset import ConstructSuperglueStrategy
 
 # get default parameters
@@ -49,23 +51,34 @@ dataset = SuperGlueDataset(task_name='cb',
                            dataset_type='train',
                            tokenizer=tokenizer)
 ```
-#### 1. Load the dataset
+`SuperGlueDataset`is the function in this step，and its major parameters are introduced below：
+
+`task_name`: Identifier of dataset, Supported datasets and their identifiers are given at the table in [1.Load the dataset](#1.Load the dataset).
+
+`data_dir`: Data will be automatically downloaded to `data_dir` directory, which is `./dataset` by default.
+
+`dataset_type`: It can be train/dev/test, which represents train/validation/test set is going to be preprocessed.
+
+`tokenizer`: Constructed tokenizer as introduced in [Tutorial1](/docs/TUTORIAL_1_TOKENIZER.md).
+
+
 
-After setting `task_name` to the abbreviation of the task name, the relevant data will be automatically downloaded in the background. FlagAI currently supports automatic loading of the following classification datasets:
 
-| dataset name                                 | short name| Language | Benchmark   |
-|----------------------------------------------|----------|------|----------|
-| Broadcoverage Diagnostics                    | boolq    | Eng   | SuperGLUE |
-| CommitmentBank                               | cb       | Eng   | SuperGLUE |
-| Choice of Plausible Alternatives             | copa     | Eng   | SuperGLUE |
-| Multi-Sentence Reading Comprehension         | muiltirc | Eng   | SuperGLUE |
-| Recognizing Textual Entailment               | rte      | Eng   | SuperGLUE |
-| Words in Context                             | wic      | Eng   | SuperGLUE |                                                   
-| The Winograd Schema Challenge                | wsc      | Eng   | SuperGLUE |
-| Ant Financial Question Matching Corpus       | afqmc    | Zh    | CLUE     |
-| Short Text Classificaiton for News           | tnews    | Zh    | CLUE     |
+#### 1.Load the dataset
 
-The dataset will be automatically downloaded to the address corresponding to `data_dir`, which defaults to the `./dataset` directory of the project.
+FlagAI currently supports automatic loading of the following classification datasets:
+
+| dataset name                                 | Identifier | Language | Benchmark   |
+|----------------------------------------------|------------|------|----------|
+| Broadcoverage Diagnostics                    | boolq      | Eng   | SuperGLUE |
+| CommitmentBank                               | cb         | Eng   | SuperGLUE |
+| Choice of Plausible Alternatives             | copa       | Eng   | SuperGLUE |
+| Multi-Sentence Reading Comprehension         | muiltirc   | Eng   | SuperGLUE |
+| Recognizing Textual Entailment               | rte        | Eng   | SuperGLUE |
+| Words in Context                             | wic        | Eng   | SuperGLUE |                                                   
+| The Winograd Schema Challenge                | wsc        | Eng   | SuperGLUE |
+| Ant Financial Question Matching Corpus       | afqmc      | Zh    | CLUE     |
+| Short Text Classificaiton for News           | tnews      | Zh    | CLUE     |
 
 The downloaded dataset directory will contain three files, corresponding to the training set data, the validation set data, and the test set data. Take the CommitmentBank dataset as an example, the `train.jsonl` in the directory corresponds to the training set, `val.jsonl `corresponds to the validation set, `test.jsonl` corresponds to the test set. Generally, the training set and test set contain label information, but the test set does not. These data files will be processed separately in the next process.
 
@@ -98,18 +111,19 @@ In this step, we will unify the data structures of different datasets to facilit
 | meta   | an optional dictionary to store arbitrary meta information                        | dict |
 | ids    | an optional numeric index                                                    | int  |
 
-For example, the example of CommitBank in the previous step will be processed into the following form
+When the dataset is built, you can view one of the samples directly in the code by indexing:
+
+```python
+example = dataset[3]  # The third example in dataset 
+```
+
+For instance, the example of CommitBank in the previous step will be processed into the following form
 
 <div align=center><img src="img/dataset_figure_2.png" width="500px"></div>
 
 Noted that if text_a and text_b cannot be filled with background text information because the data structure is too complex, you can put the rest of the information in meta.
 
-When the dataset is constructed, you can view one of the samples directly in the code by indexing:
-
 
-```python
-example = dataset[3]  # The third example in dataset 
-```
 
 ### Organize the data into input to the model
 
@@ -148,7 +162,7 @@ In the first case, the label categories contained in the dataset are limited. Fo
 
 <sup>1</sup>: seq_length represents the specified maximum length of each input vector
 
-<sup>2</sup>: The process of adding special characters: add the [CLS] symbol at the beginning of the sentence and the [EOS] symbol at the end of the sentence until the length reaches seq_length. If the text output by cloze has two paragraphs, it will be in the middle Add [SEP] symbol
+<sup>2</sup>: As illustrated in the following figure, the process of adding special characters: add the [CLS] symbol at the beginning of the sentence and the [EOS] symbol at the end of the sentence until the length reaches seq_length. If the text output by cloze has two paragraphs, it will be in the middle Add [SEP] symbol
 <div align=center><img src="img/dataset_figure_5.png" width="500px"></div>
 
 <sup>3</sup>: num_labels represents the number of options in the cloze problem
diff --git a/docs/TUTORIAL_3_MODEL.md b/docs/TUTORIAL_3_MODEL.md
index 90f71fdb..f1f62715 100644
--- a/docs/TUTORIAL_3_MODEL.md
+++ b/docs/TUTORIAL_3_MODEL.md
@@ -1,54 +1,73 @@
-# Base class
+# Major Function of Model Module 
+- [Major Function of Model Module](#major-function-of-model-module)
+  - [BaseModel](#basemodel)
+  - [from_pretrain](#from_pretrain)
+    - [load model from modelhub](#load-model-from-modelhub)
+    - [load local model weights](#load-local-model-weights)
+  - [All supported models](#all-supported-models)
+  - [Supported models + tasks](#supported-models--tasks)
+  - [Model design](#model-design)
+  - [forward function](#forward-function)
+    - [input](#input)
+    - [output](#output)
+  - [init_from_json](#init_from_json)
+    - [input](#input-1)
+    - [output](#output-1)
+
+## BaseModel
 
 The base class BaseModel implements common methods for loading/saving models from a local file or directory or from a library-provided pretrained model configuration (downloaded from BAAI modelhub's Kingsoft S3 repository).
 All supported models now support the three most common model types [encoder, decoder and encoder-decoder]. GLM models can now load all GLM series models, see https://github.com/THUDM/GLM
 
-## From_pretrain
+## from_pretrain
 
-Models with the same model structure can be loaded with the same class. For example, BERT-base and Roberta-base models can be loaded with the BertModel class. From_pretrain is optimized for data/model parallel model loading to avoid resource waste caused by repeated downloads.
-By calling ClassName.from_pretrian() to load, now our model hub supports the following models, you can directly download the model configuration file [config.json], model weights [pytorch_model.bin], and dictionary files [vocab .txt]. example:
+`from_pretrain` is used for load models. Models with the same model structure can be loaded with the same class. For example, `BERT-base` and `Roberta-base` models can be loaded with the `BertModel` class. `from_pretrain` is optimized for data/model parallel model loading to avoid resource waste caused by repeated downloads.
 
-````python
-from flagai.model.glm_model import GLMForSingleTokenCloze
-model = GLMForSingleTokenCloze.from_pretrain(download_path="./state_dict", model_name="GLM-large-ch")
-````
+### load model from modelhub
 
-If the model weights are loaded locally, they can also be loaded through ClassName.from_pretrain(). example:
-Load the model file `pytorch_model.bin` from the `./state_dict/GLM-large-ch` directory
+By calling `ClassName.from_pretrian()` to load following [supported models](#all-supported-models), it will automatically download the model configuration file `config.json`, model weights `pytorch_model.bin`, and dictionary files `vocab .txt`.
 
-````python
-from flagai.model.glm_model import GLMForSingleTokenCloze
-model = GLMForSingleTokenCloze.from_pretrain(download_path="./state_dict",
-                                               model_name="GLM-large-ch")
-````
+```python
+# Downloading GLM-large-ch from modelhub
+from flagai.model.glm_model import GLMModel
+model = GLMModel.from_pretrain(download_path="./state_dict", model_name="GLM-large-ch")
+```
+### load local model weights
+If the model weights are stored in `download_path/model_name/`, where `model_name` is the directory where the model is located, and `download_path` is the directory where `model_name` is located,
 
-## All supported models
+```python
+# load  `pytorch_model.bin` and `config.json` from `./state_dict/GLM-large-ch`
+from flagai.model.glm_model import GLMModel
+model = GLMModel.from_pretrain(download_path="./state_dict", model_name="GLM-large-ch")
+```
 
+## All supported models
+We can use `from_pretrain` to automatically download following models
 | ClassName | ModelName | Language | Model Type |
 |-----------------------------------|------------- ----|----------|------------|
-| flagai.model.glm_model.GLMModel | GLM-10b-ch | chinese | encoder |
-| flagai.model.glm_model.GLMModel | GLM-large-ch | chinese | encoder |
-| flagai.model.bert_model.BertModel | RoBERTa-base-ch | chinese | encoder |
-| flagai.model.gpt2_model.GPT2Model | GPT2_base_ch | chinese | decoder |
-| flagai.model.t5_model.T5Model | T5-base-ch | chinese | enc2dec |
-| flagai.model.t5_model.T5Model | T5-base-en | chinese | enc2dec |
-| flagai.model.bert_model.BertModel | BERT-base-en | english | encoder |
-| flagai.model.glm_model.GLMModel | GLM-large-en | english | encoder |
+| flagai.model.glm_model.GLMModel | ** GLM-10b-ch** | chinese | encoder |
+| flagai.model.glm_model.GLMModel | **GLM-large-ch** | chinese | encoder |
+| flagai.model.bert_model.BertModel | **RoBERTa-base-ch** | chinese | encoder |
+| flagai.model.gpt2_model.GPT2Model | **GPT2_base_ch** | chinese | decoder |
+| flagai.model.t5_model.T5Model | **T5-base-ch** | chinese | enc2dec |
+| flagai.model.t5_model.T5Model | **T5-base-en** | chinese | enc2dec |
+| flagai.model.bert_model.BertModel | **BERT-base-en** | english | encoder |
+| flagai.model.glm_model.GLMModel | **GLM-large-en** | english | encoder |
 
 ## Supported models + tasks
 
-At the same time, we support the finetuned model on the task, as shown in the table below, the model weights can be loaded through ClassName.from_pretrain(), for example, we automatically download and load a GLM trained on the title-generation task -large-ch model:
+At the same time, we support the finetuned model on the task, as shown in the table below, the model weights can be loaded through `ClassName.from_pretrain()`, for example, we automatically download and load a `GLM-large-ch` model finetuned on the title-generation task:
 
 ````python
 from flagai.model.glm_model import GLMForSeq2Seq
 model = GLMForSeq2Seq.from_pretrain(model_name='GLM-large-ch')
 ````
 
-We also provide the AutoLoader class to help load models. For example, the GLM-large-ch model is used for seq2seq tasks. Here we adopt a task- and model-independent design. In theory, tasks and models can be freely replaced.
+We also provide the `AutoLoader` class to help load models. For example, the `GLM-large-ch` model is used for `seq2seq` tasks. Here we adopt a task- and model-independent design. In theory, tasks and models can be freely replaced.
 
 ````python
 from flagai.auto_model.auto_loader import AutoLoader
-auto_loader = AutoLoader("seq2seq",
+auto_loader = AutoLoader("title-generation",
                          model_name="GLM-large-ch",
                          model_dir= "./state_dict")
 model = auto_loader.get_model()
@@ -56,30 +75,33 @@ model = auto_loader.get_model()
 
 | ClassName | Model Name | language | Task |
 |------------------------------------------------- |-----------------|----------|-------------------|
-| flagai.model.glm_model.GLMForSeq2Seq | GLM-large-ch | chinese | title generation |
-| flagai.model.glm_model.GLMForSeq2Seq | GLM-large-ch | chinese | poetry generation |
-| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | title generation |
-| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | NER |
-| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | semantic matching |
-| flagai.model.t5_model.T5Model | T5-base-ch | chinese | title generation |
-| flagai.model.bert_model.BertForSequenceLabeling | BERT-base-en | english | title gneration |
+| flagai.model.glm_model.GLMForSeq2Seq | GLM-large-ch | chinese | **title generation** |
+| flagai.model.glm_model.GLMForSeq2Seq | GLM-large-ch | chinese | **poetry generation** |
+| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | **title generation** |
+| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | **NER** |
+| flagai.model.bert_model.BertForSequenceLabeling | RoBERTa-base-ch | chinese | **semantic matching** |
+| flagai.model.t5_model.T5Model | T5-base-ch | chinese | **title generation** |
+| flagai.model.bert_model.BertForSequenceLabeling | BERT-base-en | english | **title gneration** |
 
 ## Model design
 
-The main construction logic of the model `layer->block>model`
+The main construction logic of the model `layer->block->model`
+
 `flagai.model.layer`: including mlp, layernorm, activation, attention and other layers
 
 `flagai.model.block`: Build a transformer block by assembling various layers, such as BERT block, etc.
 
 `flagai.model`: build the model by embedding layers and stacked blocks
 
+If you want to customize a new model structure, you can refer to the above construction process.
+
 ## forward function
 
-Model's forward function:
-Input is keyword arguments: including input_ids, position_ids, attention_mask, etc., redundant parameters will be automatically ignored
+### input
+Input parameters are keyword arguments: including input_ids, position_ids, attention_mask, etc., redundant parameters will be automatically ignored
 For example, GLM's forward function:
 
-````python
+```python
 def forward(self,
             input_ids=None,
             position_ids=None,
@@ -89,21 +111,25 @@ def forward(self,
             detach_memory=True,
             prompt_pos=None,
             **kwargs)
-````
+```
 
+### output
 The output is a dictionary, including logits and hidden states, which are required, such as the return of the GLM forword function:
 
-````python
+```python
 return {'loss': loss, 'logits': logits, 'hidden_states': mems}
-````
+```
 
 ## init_from_json
-Model's init_from json function:
-The input is a dictionary, the output is an initialized model
-For example, the invocation of GLMModel is as follows:
 
-````python
-GLMModel.init_from_json(config_file = "./config.json", **kwargs)
-````
+### input
+The input is a json and `**kwargs` which is used for the new initialization parameters of models.
+For example, the initialization of GLMModel is as follows:
+
+```python
+GLMModel.init_from_json(config_file = "./config.json", checkpoint_activations=True)
+```
+`checkpoint_activations=True` is a new parameter used to control whether to perform gradient recomputation.
 
-**kwargs are reserved parameters, in order to be compatible with new initialization parameters of some models
+### output
+An instance of GLM model.
\ No newline at end of file
diff --git a/docs/TUTORIAL_4_TRAINER.md b/docs/TUTORIAL_4_TRAINER.md
index d12d0415..f78526b4 100644
--- a/docs/TUTORIAL_4_TRAINER.md
+++ b/docs/TUTORIAL_4_TRAINER.md
@@ -1,11 +1,36 @@
 # Trainer
-
+- [Trainer](#trainer)
+  - [Getting Started](#getting-started)
+  - [env_type in Trainer](#env_type-in-trainer)
+  - [customize a Trainer](#customize-a-trainer)
+  - [Single node cpu/gpu](#single-node-cpugpu)
+  - [fp16](#fp16)
+  - [Gradient recomputation](#gradient-recomputation)
+    - [GLM-10b-ch](#glm-10b-ch)
+    - [huggingface t5-11b](#huggingface-t5-11b)
+  - [Training huggingface t5-11b example with Trainer](#training-huggingface-t5-11b-example-with-trainer)
+  - [Parallel training](#parallel-training)
+    - [deepspeed](#deepspeed)
+    - [pytorchDDP](#pytorchddp)
+    - [deepspeed + megatron-lm](#deepspeed--megatron-lm)
 The Trainer class provides APIs for training with multiple parallel frameworks. The API supports distributed training with Pytorch DDP/Deepspeed on multiple GPUs, as well as mixed parallel distributed training with Megatron-LM+Deepspeed, and mixed precision via NVIDIA Apex.
 
 ## Getting Started
-Trainer includes basic training loops that support the above features. To customize the behavior, you can subclass them and override the forward_step method:
-The return of the forward_step method is a dict
+Trainer includes basic training loops that support the above features. Two steps to use a Trainer: initialization and excution. Refer to the code in the directory `examples/glm_superglue` .
+
+## env_type in Trainer
+There is a parameter env_type in Trainer that controls whether the training is distributed or not.
 
+```shell
+The enviroment type for training. Will default to 'pytorch'.
+env_type: `pytorch`, `pytorchDDP`, `deepspeed`, `deepspeed+mpu`
+            pytorch: single node cpu/gpu
+            pytorchDDP: single-/multi- node gpu <data parallel>
+            deepspeed: single-/multi- node gpu <data/pipline parallel>
+            deepspeed+mpu: single-/multi- node gpu <data parallel + model parallel>
+```                          
+## customize a Trainer
+When using a custom model, when the input and output of the model are inconsistent with the behavior of the model in the FlagAI framework (refer to the introduction of the [model forward function](TUTORIAL_3_MODEL.md#forward-function)), a custom Trainer is required for training. To customize Trainer to quickly support custom models, you can inherent Trainer and override the forward_step method. Note: the return of the forward_step method is a dict
 ```python
 from flagai.trainer import Trainer
 
@@ -21,16 +46,6 @@ class MyTrainer(Trainer):
         return output
 ```
 
-There is a parameter env_type in Trainer that controls whether the training is distributed or not.
-
-```shell
-The enviroment type for training. Will default to 'pytorch'.
-env_type: `pytorch`, `pytorchDDP`, `deepspeed`, `deepspeed+mpu`
-            pytorch: single node cpu/gpu
-            pytorchDDP: single-/multi- node gpu <data parallel>
-            deepspeed: single-/multi- node gpu <data/pipline parallel>
-            deepspeed+mpu: single-/multi- node gpu <data parallel + model parallel>
-```                          
 
 ## Single node cpu/gpu
 
@@ -48,7 +63,7 @@ trainer = MyTrainer(
 ```
 Specify the graphics card/cpu settings as pytorch_device, 'cpu', 'cuda:0', etc.
 ## fp16
-
+When the model parameters are large and the GPU memory space is very tight, the memory usage can be reduced by converting the fp32 parameters to fp16. FlagAI can realize such transformer with change of a parameter.
 ```python
 Model parameters turned to fp16
 trainer = MyTrainer(
@@ -66,38 +81,42 @@ trainer = MyTrainer(
 ## Gradient recomputation
 Do not save the Intermediate results in the forward stage. Paper: [Training Deep Nets with Sublinear Memory Cost](https://arxiv.org/abs/1604.06174v2)
  
-Now, we can train/finetune a t5-11b with gradient_accumulation_steps. 
+Below, we give two examples of tens of billions of models to enable gradient recalculation, one is to load the 10-billion parameter model from FlagAI, and the other is to load the tens of billions model of huggingface.
+
+### GLM-10b-ch
+```python
+#download model from modelhub and activate gradient recompuatation
+from flagai.model.glm_model import GLMModel
+model = GLMModel.from_pretrain(download_path="./state_dict", model_name="GLM-large-ch", checkpoint_activations=True)
+```
 
+### huggingface t5-11b
+t5-11b paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)
 ```python
 from transformers import T5ForConditionalGeneration, T5Tokenizer
 tokenizer = T5Tokenizer.from_pretrained('t5-11b')
 model = T5ForConditionalGeneration.from_pretrained('t5-11b')
 model.gradient_checkpointing = True
 ```
-## Support huggingface model
-The example directory location of the FlagAI project：examples/t5_huggingface
-t5-11b paper: [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)
+To demonstrate the scalability of FlagAI's Trainer, let's use the training `T5-11b` model as an example. Note: The weight of models with a scale of more than 10 billion is **20G+**, and a single `gpu >= V100` is required.
 
-The example train t5-11b on a toy dataset.  You need  a device > 30G memory to run the example. If your have multiple devices,  please check out the following session for speedup.
+## Training huggingface t5-11b example with Trainer 
+FlagAI example：`examples/t5_huggingface`
 
 ```python
-import sys
+# Copyright © 2022 BAAI. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License")
 from flagai.trainer import Trainer
 from transformers import T5ForConditionalGeneration, T5Tokenizer
 from torch.utils.data import Dataset
 import torch
 
-## Inheriant the Trainer
-## overload the forward_step function
+
 class MyTrainer(Trainer):
 
     def forward_step(self, data, model, mems):
-        """
-        Args:
-            data: a dict contains a batch of inputs
-        return:
-            output: a dict contains `loss`
-        """
+
         model_outputs = model(**data)
         output = {}
         output['loss'] = model_outputs.loss
@@ -105,28 +124,34 @@ class MyTrainer(Trainer):
         output['hidden_states'] = model_outputs.decoder_hidden_states
         return output
 
-# get a customized trainer instance
+
 trainer = MyTrainer(
-    env_type='pytorch',
+    env_type='deepspeed',
     epochs=1,
-    batch_size=4,
+    batch_size=1,
     eval_interval=10,
     log_interval=10,
     experiment_name='t5-11b',
-    pytorch_device='cuda:0',
     load_dir=None,
-    lr=1e-4,
-    fp16=False)
+    lr=1e-4
+    # parameters for pytorchDDP
+    master_ip='127.0.0.1',
+    master_port=17750,
+    num_nodes=1,
+    num_gpus=1,
+    training_script=__file__,
+    # deepspeed
+    deepspeed_config='deepspeed.json'
+)
 
-# using huggingface transformers to get tokenizer and models
 model_name = 't5-11b'
 tokenizer = T5Tokenizer.from_pretrained(model_name)
 model = T5ForConditionalGeneration.from_pretrained(model_name)
+model.gradient_checkpointing = True
 
 print("loading model & tokenizer is done!")
-src_dir = 'train_inputs.txt'
-tgt_dir = 'train_targets.txt'
-model_dir = "./t5-11b"  # 模型位置
+src_dir = './data/train.src'
+tgt_dir = './data/train.tgt'
 maxlen = 1024
 
 
@@ -146,10 +171,10 @@ def read_file():
     return src, tgt
 
 
-class BertSeq2seqDataset(Dataset):
+class T5Seq2seqDataset(Dataset):
 
     def __init__(self, sents_src, sents_tgt, tokenizer, maxlen=512):
-        super(BertSeq2seqDataset, self).__init__()
+        super(T5Seq2seqDataset, self).__init__()
         self.sents_src = sents_src
         self.sents_tgt = sents_tgt
         self.tokenizer = tokenizer
@@ -196,87 +221,103 @@ def seq2seq_collate_fn(batch):
 sents_src, sents_tgt = read_file()
 data_len = len(sents_tgt)
 train_size = int(data_len * 0.8)
-train_src = sents_src[:train_size]
-train_tgt = sents_tgt[:train_size]
+train_src = sents_src[:train_size][:200]
+train_tgt = sents_tgt[:train_size][:200]
 
 val_src = sents_src[train_size:]
 val_tgt = sents_tgt[train_size:]
 
-train_dataset = BertSeq2seqDataset(train_src,
-                                   train_tgt,
-                                   tokenizer=tokenizer,
-                                   maxlen=maxlen)
-val_dataset = BertSeq2seqDataset(val_src,
-                                 val_tgt,
+train_dataset = T5Seq2seqDataset(train_src,
+                                 train_tgt,
                                  tokenizer=tokenizer,
                                  maxlen=maxlen)
-## Training
+val_dataset = T5Seq2seqDataset(val_src,
+                               val_tgt,
+                               tokenizer=tokenizer,
+                               maxlen=maxlen)
+
 trainer.train(model,
               train_dataset=train_dataset,
               collate_fn=seq2seq_collate_fn)
 ```
 
-## Distributed training
-To accelerate your training/finetuning process, FlagAI integrate three popular parallel frameworks for deep neural networks.
-### pytorchDDP
-DistributedDataParallel (DDP) link
-We can use distributed training by modifying a few parameters. The launch call is directly encapsulated inside, and python script_name.py [script_name: is the name of the training script] can be run directly.
+## Parallel training
+For speedup model training, FlagAI supports three types of paralleled training, but in the example of training T5-11b, only the the `deepspeed` framework can be used.
 
-```python
-trainer = MyTrainer(
-    env_type='pytorchDDP',
-    epochs=1,
-    batch_size=4,
-    eval_interval=10,
-    log_interval=10,
-    experiment_name='t5-11b',
-    load_dir=None,
-    lr=1e-4
-    # parameters for pytorchDDP
-    master_ip='127.0.0.1',
-    master_port=17750,
-    num_nodes=1,
-    num_gpus=2,
-    hostfile='./hostfile',
-    training_script=__file__,
-)
-```
 ### deepspeed
-Compared with pytorch, deepspeed mainly optimizes the optimizer.
-It provides an optimizer of cpu-offload, which can greatly reduce the occupation of gpu video memory.
-For the relevant settings of the deepspeed optimizer, the configuration file of deepspeed.json needs to be provided
+Deepspeed provides the cpu-offload optimizer , which can greatly reduce the occupation of gpu memory.  The configuration file of `deepspeed.json` is as follows,
+```shell
+{
+  "train_micro_batch_size_per_gpu": 2,
+  "gradient_accumulation_steps": 1,
+  "steps_per_print": 100,
+  "gradient_clipping": 1.0,
+  "zero_optimization": {
+    "stage": 3,
+    "contiguous_gradients": false,
+    "overlap_comm": true,
+    "reduce_scatter": true,
+    "reduce_bucket_size": 5e7,
+    "allgather_bucket_size": 5e7,
+    "cpu_offload": true 
+  },
+  "zero_allow_untested_optimizer": true,
+  "fp16": {
+    "enabled": true,
+    "loss_scale": 0,
+    "loss_scale_window": 1000,
+    "hysteresis": 2,
+    "min_loss_scale": 1
+  },
+  "optimizer": {
+    "type": "Adam",
+    "params": {
+      "lr": 0.0004,
+      "weight_decay": 0.01,
+      "betas": [
+        0.9,
+        0.98
+      ],
+      "eps": 1e-6
+    }
+  },
+  "activation_checkpointing": {
+    "partition_activations": false,
+    "contiguous_memory_optimization": false
+  },
+  "wall_clock_breakdown": false
+}
+```
+`deepspeed_config` can be find in  `examples/t5_huggingface/deepspeed.json`. `stage` and `cpu_offload` are two key parameters.
+
+`hostfile` can be is ignored in single node setting.
+
+### pytorchDDP
+DistributedDataParallel (DDP) can be used when the size of model parameters <1 billion, e.g., `t5-base`. We can activate the framework by setting `env_type` = `pytorchDDP`.
 
 ```python
 trainer = MyTrainer(
     env_type='pytorchDDP',
     epochs=1,
-    batch_size=4,
+    batch_size=1,
     eval_interval=10,
     log_interval=10,
-    experiment_name='t5-11b',
+    experiment_name='t5-base',
     load_dir=None,
     lr=1e-4
     # parameters for pytorchDDP
     master_ip='127.0.0.1',
     master_port=17750,
     num_nodes=1,
-    num_gpus=2,
+    num_gpus=1,
     hostfile='./hostfile',
     training_script=__file__,
-    # deepspeed
-    deepspeed_config='deepspeed.json'
 )
 ```
-
-Deepspeed config file reference [examples/t5_huggingface/deepspeed.json]
-Among them, the main parameters are stage and cpu offload
-
 ### deepspeed + megatron-lm
-
-Now the tens of billions of model GLM adopts the model parallel technology of Megatron-LM. When the model parameter scaling is above 10b, it is difficult to load a single model and all the intermediate variables during training in the video memory of a single card. To this end, Megatron-LM provides a Tensor segmentation method. The main idea is to segment the matrix according to rows/columns. FlagAI link, convert the model to the Megatron-LM version.
-As follows, FlagAI's internal models (GLM, T5, BERT [including RoBERTa], GPT2) support Megatron-LM. As long as the environment variable is changed to deepspeed+mpu in the configuration file, the model parallel function can be started.
-For the huggingface version of the model, there is no support for model parallelism.
-
+Now the 10-billion model GLM-10-ch adopts the model-parallel technology of `Megatron-LM` and the data-parallel technology of `deepspeed`. When the size of model parameters is above 10-billion, it is difficult to load a model and all the intermediate variables during training in a single gpu. To this end, `Megatron-LM` provides a model-parallel method. The main idea is to segment the matrix according to rows/columns. FlagAI converts the model to the `Megatron-LM` version.
+As follows, FlagAI support Megatron-LM version of models (GLM, T5, BERT [including RoBERTa], GPT2), as long as the environment variable is modified to `deepspeed+mpu` in the configuration file, the model-parallel version can be activated.
+For the huggingface models, there is no `Megatron-LM` support in FlagAI.
 ```python
 trainer = MyTrainer(
     env_type="deepspeed+mpu", # env_type
@@ -284,7 +325,7 @@ trainer = MyTrainer(
     batch_size=8,
     eval_interval=10,
     log_interval=10,
-    experiment_name='t5-11b',
+    experiment_name='GLM-10b-ch',
     load_dir=None,
     lr=1e-4,
     # parallel settings
diff --git a/docs/TUTORIAL_7_PROMPT_LERANING.md b/docs/TUTORIAL_7_PROMPT_LERANING.md
index 144b5100..86c342ce 100644
--- a/docs/TUTORIAL_7_PROMPT_LERANING.md
+++ b/docs/TUTORIAL_7_PROMPT_LERANING.md
@@ -19,32 +19,6 @@ We can design a meaningful cloze template as below
 
 <div align=center><img src="img/prompt_figure_1.png" width="500px"></div>
 
-And we can implement the template through the following code:
-
-```python
-def get_parts(self, example: InputExample):  
-    '''A simplified version of the actual code'''
-    # Construct the template
-    cloze_ids = [
-        example.text_a,' question: is it true, false or neither that',
-         example.text_b, ' answer:', [self.mask]]
-    return cloze_ids
-```
-
-The label in various datasets can have multiple types including boolean, integer and string. We need to implement a verbalizer that maps the labels to the text string of choices in cloze question. For example, here the 'entailment', 'contradiction' and 'neutral' labels can be mapped to 'true' ,'false' and 'neither', and each time the answer to be filled is always selected among those choices. 
-
-Verbalizer is essentially a mapping relationship, and we can define it in a Python dictionary as shown below:
-
-```python
-VERBALIZER = {
-    "contradiction": [" false"],
-    "entailment": [" true"],
-    "neutral": [" neither"]
-}
-def verbalize(self, label) -> List[str]:
-    return CbPVP.VERBALIZER[label]
-```
-
 ### Prefix Prompts
 Instead of designing a prompt in real natural language, prefix prompts directly modifies the embedding space. It inserts a sequence of task-specific vectors and freezes the LM parameters. An advantage of prefix prompts is that embeddings of templates are no longer limited by parameters in pre-trained LMs. 
 
@@ -54,10 +28,10 @@ Here is an example that compares prefix tuning and fine-tuning in the embedding
 
 
 ## Reference
-[Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://arxiv.org/abs/2107.13586)
+Liu, P. (2021, July 28). Pre-train, Prompt, and Predict: A Systematic Survey of Prompting. . . arXiv.Org. https://arxiv.org/abs/2107.13586 
 
-[OpenPrompt: An Open-source Framework for Prompt-learning](https://arxiv.org/abs/2111.01998)
+Ding, N. (2021, November 3). OpenPrompt: An Open-source Framework for Prompt-learning. arXiv.Org. https://arxiv.org/abs/2111.01998
 
-[Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://arxiv.org/abs/2001.07676)
+Schick, T. (2020, January 21). Exploiting Cloze Questions for Few Shot Text Classification and. . . arXiv.Org. https://arxiv.org/abs/2001.07676
 
-[Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://arxiv.org/abs/2101.00190)
\ No newline at end of file
+Li, X. L. (2021, January 1). Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv.Org. https://arxiv.org/abs/2101.00190
diff --git a/docs/TUTORIAL_8_ENVIRONMENT_SETUP.md b/docs/TUTORIAL_8_ENVIRONMENT_SETUP.md
index 49f02aa7..d6ef4585 100644
--- a/docs/TUTORIAL_8_ENVIRONMENT_SETUP.md
+++ b/docs/TUTORIAL_8_ENVIRONMENT_SETUP.md
@@ -1,4 +1,40 @@
 #  Pre-training distributed environment setup
+- [Pre-training distributed environment setup](#pre-training-distributed-environment-setup)
+- [一.  Docker](#一--docker)
+  - [1.install docker](#1install-docker)
+  - [2.Docker source change](#2docker-source-change)
+  - [3.Install the graphics card driver (skip if installed)](#3install-the-graphics-card-driver-skip-if-installed)
+  - [4.Configure NVIDIA-docker source：](#4configure-nvidia-docker-source)
+  - [5.Make dockerfile](#5make-dockerfile)
+    - [a. Pull the NVIDIA basic image and create a temporary folder (in the container, delete the image after it is created)](#a-pull-the-nvidia-basic-image-and-create-a-temporary-folder-in-the-container-delete-the-image-after-it-is-created)
+    - [b.Configure apt installation source and install some common basic packages of Linux](#bconfigure-apt-installation-source-and-install-some-common-basic-packages-of-linux)
+    - [c. Install the latest version of GIT (create an image clone installation package)](#c-install-the-latest-version-of-git-create-an-image-clone-installation-package)
+    - [d. Install Mellanox OFED. Due to network problems, it is recommended to download the installation package locally and then execute dockerfile](#d-install-mellanox-ofed-due-to-network-problems-it-is-recommended-to-download-the-installation-package-locally-and-then-execute-dockerfile)
+    - [e. Install nv_peer_mem](#e-install-nv_peer_mem)
+    - [f. Install openmpi, You need to install the libevent dependency package first](#f-install-openmpi-you-need-to-install-the-libevent-dependency-package-first)
+    - [g.Install python](#ginstall-python)
+    - [h.Install magma-cuda](#hinstall-magma-cuda)
+    - [i.Configuration path](#iconfiguration-path)
+    - [j.Install some packages](#jinstall-some-packages)
+    - [k.Install mpi4py (need to download to local installation, pip installation may report an error due to version compatibility)](#kinstall-mpi4py-need-to-download-to-local-installation-pip-installation-may-report-an-error-due-to-version-compatibility)
+    - [l.Install pytorch, the version can be replaced, need to download locally first. Installation is easy to be terminated due to network speed. Some sub packages may be terminated during the download process of pytorch git clone. You can try few more times.](#linstall-pytorch-the-version-can-be-replaced-need-to-download-locally-first-installation-is-easy-to-be-terminated-due-to-network-speed-some-sub-packages-may-be-terminated-during-the-download-process-of-pytorch-git-clone-you-can-try-few-more-times)
+    - [m.Install apex](#minstall-apex)
+    - [n.Install deepspeed](#ninstall-deepspeed)
+    - [o.Install NCCL(optional)](#oinstall-nccloptional)
+    - [p.Configure network port, public key and SSH](#pconfigure-network-port-public-key-and-ssh)
+  - [6.Build docker image](#6build-docker-image)
+    - [a.Method 1. Pull image](#amethod-1-pull-image)
+    - [b.Method 2. Build image](#bmethod-2-build-image)
+- [二. Build containers at each machine node](#二-build-containers-at-each-machine-node)
+- [三. Mutual trust mechanism setting](#三-mutual-trust-mechanism-setting)
+  - [1. The default docker image for public key has been generated when it is created. If it does not exist, enter flow on the shell](#1-the-default-docker-image-for-public-key-has-been-generated-when-it-is-created-if-it-does-not-exist-enter-flow-on-the-shell)
+  - [2.The public key file generated by each node container](#2the-public-key-file-generated-by-each-node-container)
+  - [3.login without password](#3login-without-password)
+  - [4.test](#4test)
+- [四.  Distributed training test](#四--distributed-training-test)
+  - [a.Configure hostfile (~/SSH/config and V100-1 in the hostfile correspondence):](#aconfigure-hostfile-sshconfig-and-v100-1-in-the-hostfile-correspondence)
+  - [b. Configure glm files. Each node is configured with code and data. The path is required to be the same (you can also access cloud shared files together)](#b-configure-glm-files-each-node-is-configured-with-code-and-data-the-path-is-required-to-be-the-same-you-can-also-access-cloud-shared-files-together)
+  - [c. cmd](#c-cmd)
 
 # 一.  Docker
 
@@ -30,7 +66,7 @@ apt-get install -y docker-ce
 
 ## 2.Docker source change
 
-### (https://xxxx.mirror.aliyuncs.com) is your own docker source
+(https://xxxx.mirror.aliyuncs.com) is your own docker source
 
 ```shell
 mkdir -p /etc/docker
@@ -81,7 +117,7 @@ apt-get update
 apt-get install -y nvidia-docker2
 ````
 
-### Modify /etc/docker/daemon.json，Add relevant information
+Modify /etc/docker/daemon.json，Add relevant information
 
 ```text
 "runtimes": {
@@ -91,7 +127,7 @@ apt-get install -y nvidia-docker2
    }
 }
 ```
-### final content of /etc/docker/daemon.json
+final content of /etc/docker/daemon.json
 
 ```json
 {
@@ -105,7 +141,7 @@ apt-get install -y nvidia-docker2
 }
 ```
 
-### reboot docker service
+reboot docker service
 
 ```shell
 systemctl daemon-reload
diff --git a/docs/TUTORIAL_9_SEQ2SEQ_METHOD.md b/docs/TUTORIAL_9_SEQ2SEQ_METHOD.md
index a0e0be5e..e06c5716 100644
--- a/docs/TUTORIAL_9_SEQ2SEQ_METHOD.md
+++ b/docs/TUTORIAL_9_SEQ2SEQ_METHOD.md
@@ -1,8 +1,17 @@
 # Seq2seq Method
 
+Three ways are supported for sequence-to-sequence task generation, including:
+1. Sequence-to-sequence generation tasks using only encoder models
+2. Sequence-to-sequence generation tasks using only the decoder model
+3. Sequence-to-sequence generation task with encoder+decoder model
+
+
 ## Encoder model
 We provide the encoder model to perform the seq2seq task, for example, Bert, Roberta, GLM, and so on.
 
+1. [Title Generation with RoBERTa model](https://github.com/BAAI-Open/FlagAI/tree/master/examples/roberta_title_generation)
+2. [Title Generation with GLM model](https://github.com/BAAI-Open/FlagAI/tree/master/examples/glm_title_generation)
+
 We add a special attention mask in the encoder model at training process. (https://github.com/microsoft/unilm)
 
 ![encoder_mask](./img/encoder_mask.png)
@@ -16,12 +25,16 @@ Where, sentence_1 does not use mask, and sentence_2 uses autoregressive mask.
 
 We also provide the decoder model for seq2seq task, such as gpt-2 models.
 
+1. [Writing text with GPT-2 model](https://github.com/BAAI-Open/FlagAI/blob/master/quickstart/writing_ch.py)
+
 ![decoder_mask](./img/decoder_mask.png)
 
 Giving a start text, this model can be a good continuation of the text.
 
 ## Encoder-Decoder model
 We also provide the encoder-decoder model for seq2seq task, such as T5 models.
+1. [Title Generation with T5 model](https://github.com/BAAI-Open/FlagAI/tree/master/examples/t5_title_generation)
+
 
 ![encoder_decoder_mask](./img/encoder_decoder_mask.png)
 
diff --git a/docs/tokenization.md b/docs/tokenization.md
index f22461f7..8a582dc1 100644
--- a/docs/tokenization.md
+++ b/docs/tokenization.md
@@ -1,5 +1,3 @@
-[TOC]
-
 # sub-word algorithm
 
 ## Tokenization
@@ -38,7 +36,7 @@
 
 ​		三种不同的切分方法效果对比如下：
 
-![3 different tokenization scheme: on rules, on punctuaction, on white spaces](tokenization.assets/tokenize.png)
+![3 different tokenization scheme: on rules, on punctuaction, on white spaces](img/tokenize.png)
 
 ​		上面说的三种分词方法，都是词级别的分词方法（word tokenization），简单来说，就是以词为单位对文本进行切分。尽管这是将文本切分为更小的单位的最直观的方式，但是这种方式在处理大量语料的时候会产生一个非常严重的问题——生成一个巨大的词表（因为大量的语料中会存在大量的只出现一次/少次的token）。
 
@@ -174,22 +172,20 @@
 
 ***
 
-
-
 ## WordPiece
 
 ​		WordPiece 算法最早发表于 [Japanese and Korean Voice Search (Schuster et al., 2012)](https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf)，它和 BPE 算法非常相似。也有一些预训练模型的分词器用的是这个算法，比如 BERT、DistilBERT 和 Electra。与 BPE 算法类似，WordPiece 算法也是初始化一个基础词表，然后每次从词表中选出两个子词合并成新的子词。
 
 ​		与 BPE 的主要区别在于，选择两个子词进行合并的规则：BPE 选择频数最高的相邻子词合并，而 WordPiece 使用的是通过语言模型来计算合并两个单词可能造成的影响，然后选择使得似然函数提升最大的字符对。这个提升是通过结合后的字符对减去结合前的字符对之和得到的。
 
-​		假设句子$S=（t1,t2,...,t_n）$由 n 个 token 组成，$t_i$表示第$i$个 token，假设各个 token 之间是相互独立的，则句子$S$的语言模型似然值等于所有子词概率的乘积
-$$
-logP(S) = \sum^n_{i=1}logP(t_i)
-$$
-​		假设把相邻位置的两个 token $x$ 和 $y$ 进行合并，合并后的 token 记为 $z$，此时句子 $S$ 的似然值变化可以表示为：
-$$
-logP(t_z)-(logP(t_x)+logP(t_y)) = log(\frac{P(t_z)}{P(t_x)P(T_y)})
-$$
+​		假设句子<img src="https://latex.codecogs.com/png.latex? S=(t1,t2,...,t_n)"> 由 n 个 token 组成，<img src="https://latex.codecogs.com/png.latex? t_i">表示第i个 token，假设各个 token 之间是相互独立的，则句子`S`的语言模型似然值等于所有子词概率的乘积
+
+<img src="https://latex.codecogs.com/png.latex? logP(S) = \sum^n_{i=1}logP(t_i)">
+
+​		假设把相邻位置的两个 token `x` 和 `y` 进行合并，合并后的 token 记为 `z`，此时句子 `S` 的似然值变化可以表示为：
+
+<img src="https://latex.codecogs.com/png.latex? logP(t_z)-(logP(t_x)+logP(t_y)) = log(\frac{P(t_z)}{P(t_x)P(T_y)})">
+
 ​		也就是说，判断`tokenization`相较于`token` + `ization`是否更适合出现。选择能够提升语言模型概率最大的相邻子词加入词表。
 
 
@@ -200,20 +196,20 @@ $$
 
 ​		并没有预训练模型单独使用 ULM 作为分词的算法，但是它常与 SentencePiece 一起使用。
 
-​		对于句子$S$，$\mathop{x}\limits ^{\rightarrow} = (x_1,x_2,...,x_m)$为句子的一个分词结果，由 m 个子词组成。所以，当前分词下句子$S$的似然值可以表示为：
-$$
-P(\mathop{x}\limits ^{\rightarrow}) = \prod^m_{i=1}P(x_i)
-$$
+​		对于句子S，<img src="https://latex.codecogs.com/png.latex? \mathop{x}\limits ^{\rightarrow} = (x_1,x_2,...,x_m)">为句子的一个分词结果，由 m 个子词组成。所以，当前分词下句子$S$的似然值可以表示为：
+
+<img src="https://latex.codecogs.com/png.latex? P(\mathop{x}\limits ^{\rightarrow}) = \prod^m_{i=1}P(x_i)">
+
 ​		对于句子$S$，挑选似然值最大的作为分词结果，则可以表示为：
-$$
-x^* = \mathop{argmax}\limits_{x \in U(x)}~P(\mathop{x}\limits ^{\rightarrow})
-$$
-​		这里的$U(x)$包含了句子的所有分词结果。在实际应用中，词表大小有上万个，直接罗列所有的分词组合不具有操作性。针对这个问题，可以通过维特比算法得到 $x^*$来解决。
-
-​		ULM 通过 EM 算法来估计每个子词的概率$P(x_i)$，假设当前词表为V，训练数据为|D|，则当前训练数据中所有句子的所有分词组合形成的概率相加，表示为：
-$$
-L = \sum^{|D|}_{s=1}log(P(X^{(s)}))=\sum^{|D|}_{s=1}log(\mathop{\sum_{x \in U(X^{(s)})}}P(x))
-$$
+
+<img src="https://latex.codecogs.com/png.latex? x^* = \mathop{argmax}\limits_{x \in U(x)}~P(\mathop{x}\limits ^{\rightarrow})">
+
+​		这里的$U(x)$包含了句子的所有分词结果。在实际应用中，词表大小有上万个，直接罗列所有的分词组合不具有操作性。针对这个问题，可以通过维特比算法得到 <img src="https://latex.codecogs.com/png.latex? x^*">来解决。
+
+​		ULM 通过 EM 算法来估计每个子词的概率<img src="https://latex.codecogs.com/png.latex? P(x_i)">，假设当前词表为V，训练数据为|D|，则当前训练数据中所有句子的所有分词组合形成的概率相加，表示为：
+
+<img src="https://latex.codecogs.com/png.latex? L = \sum^{|D|}_{s=1}log(P(X^{(s)}))=\sum^{|D|}_{s=1}log(\mathop{\sum_{x \in U(X^{(s)})}}P(x))">
+
 ​		在训练的开始阶段，ULM 初始化一个巨大的基础词表，然后针对当前词表，用上面的算法计算每个子词在训练数据上的概率，然后计算移除每个子词时，语言模型整体的损失上升了多少，记为这个子词的损失值。然后通过排序，按照一定比例移除掉词表中损失值最低的一部分子词（这个比例一般是10%或者20%），这些子词的损失值最低，意味着他们对语言模型的影响最小。每训练一步，都会移除掉部分的子词，直到词表大小达到预设值为止。（ULM 算法训练的过程中会保留基础的字符以保证任何词语都能被处理）。
 
 ​		训练完成的 UML 模型对一个句子给出多种的分词可能，比如说一个训练好的 ULM 分词器的词表如下：
diff --git a/examples/glm_pretrain/train.py b/examples/glm_pretrain/train.py
index 5e72f260..cad5e9d2 100644
--- a/examples/glm_pretrain/train.py
+++ b/examples/glm_pretrain/train.py
@@ -1,6 +1,7 @@
 # Copyright © 2022 BAAI. All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License")
+
 from flagai.data.tokenizer import GLMLargeChTokenizer
 from flagai.model.glm_model import GLMForSeq2Seq
 from flagai.trainer import Trainer
@@ -8,20 +9,7 @@
 from flagai.data.dataset import BlockDataset
 from flagai.data.dataset.block.data_utils import split_ds, get_dataset_lazy, add_args
 from flagai.data.dataset.superglue.control import DEFAULT_METRICS
-
-
-class DatasetArguments:
-
-    def __init__(self):
-        self.task_mask = True  # Distinguished the generation and gap-sentence mask
-        self.block_mask_prob = 0.1
-        self.block_lm = True  # Whether do masking
-        self.masked_lm = False  # Whether do simple masking (same symbol among masks)
-
-        self.pre_tokenize = True
-        self.no_lazy_loader = True
-        self.half_lazy_loader = False
-        self.sentinel_token = False
+from flagai.test_utils import PretrainDatasetArguments
 
 
 if __name__ == '__main__':
@@ -39,7 +27,7 @@ def __init__(self):
 
     model = GLMForSeq2Seq.from_pretrain(model_name='GLM-large-ch')
 
-    ds_args = DatasetArguments()
+    ds_args = PretrainDatasetArguments()
 
     tokenizer = GLMLargeChTokenizer()
 
@@ -73,35 +61,9 @@ def create_dataset(tokenizer, should_split):
         collate_fn = ConstructBlockStrategy(
             tokenizer, 512, eod_token=tokenizer.get_command('eos').Id)
     metric_methods = DEFAULT_METRICS['pretrain']
-    # trainer.train(model,
-    #               collate_fn=collate_fn,
-    #               train_dataset=datasets[0],
-    #               valid_dataset=datasets[1],
-    #               metric_methods=metric_methods)
-    import torch
-
-    train_loader = torch.utils.data.DataLoader(datasets[0],
-                                               batch_size=16,
-                                               shuffle=False,
-                                               num_workers=1,
-                                               drop_last=False,
-                                               pin_memory=False,
-                                               collate_fn=collate_fn)
-    for data_iterator in train_loader:
-        dct = data_iterator
-        '''
-        input_ids torch.Size([16, 2, 256])
-        labels torch.Size([16])
-        position_ids torch.Size([16, 2, 2, 256])
-        attention_mask torch.Size([16, 2])
-        target_ids torch.Size([16, 2, 256])
-        logit_mask torch.Size([16, 2, 256])
-        loss_mask torch.Size([16, 2])
+    trainer.train(model,
+                  collate_fn=collate_fn,
+                  train_dataset=datasets[0],
+                  valid_dataset=datasets[1],
+                  metric_methods=metric_methods)
 
-        '''
-        for key, value in dct.items():
-            try:
-                print(key, value.size())
-            except:
-                print(key, len(value))
-        break
diff --git a/examples/glm_seq2seq/train.py b/examples/glm_seq2seq/train.py
index ffc4b185..91d9ec43 100644
--- a/examples/glm_seq2seq/train.py
+++ b/examples/glm_seq2seq/train.py
@@ -25,7 +25,7 @@ def test_init_trainer_pytorch(self):
                           eval_interval=5,
                           log_interval=50,
                           experiment_name='glm_large',
-                          pytorch_device='cpu',
+                          pytorch_device='cuda',
                           load_dir=None,
                           lr=1e-4)
         print("downloading...")
diff --git a/examples/glm_superglue/train.py b/examples/glm_superglue/train.py
deleted file mode 100644
index d1cdb2d2..00000000
--- a/examples/glm_superglue/train.py
+++ /dev/null
@@ -1,63 +0,0 @@
-# Copyright © 2022 BAAI. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License")
-from flagai.trainer import Trainer
-from flagai.model.glm_model import GLMForSingleTokenCloze, GLMForMultiTokenCloze, GLMForMultiTokenClozeFast
-from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer, GLMLargeChTokenizer
-
-from flagai.data.dataset import SuperGlueDataset
-from flagai.test_utils import CollateArguments
-from flagai.data.dataset.superglue.control import DEFAULT_METRICS, MULTI_TOKEN_TASKS, CH_TASKS
-import unittest
-from flagai.data.dataset import ConstructSuperglueStrategy
-
-
-# task_name options: ['boolq', 'cb', 'copa', 'multirc', 'rte', 'wic', 'wsc', 'afqmc', 'tnews']
-task_name = "afqmc"
-
-trainer = Trainer(env_type='pytorch',
-                    epochs=10,
-                    batch_size=4,
-                    eval_interval=100,
-                    log_interval=50,
-                    experiment_name='glm_large',
-                    pytorch_device='cuda',
-                    load_dir=None,
-                    lr=1e-4)
-print("downloading...")
-
-cl_args = CollateArguments()
-cl_args.multi_token = task_name in MULTI_TOKEN_TASKS
-
-if task_name in CH_TASKS:
-    model_name = 'GLM-large-ch'
-    tokenizer = GLMLargeChTokenizer()
-else:
-    model_name = 'GLM-large-en'
-    tokenizer = GLMLargeEnWordPieceTokenizer()
-
-if cl_args.multi_token:
-    model = GLMForMultiTokenCloze.from_pretrain(model_name=model_name)
-else:
-    model = GLMForSingleTokenCloze.from_pretrain(model_name=model_name)
-
-train_dataset = SuperGlueDataset(task_name=task_name,
-                                    data_dir='./datasets/',
-                                    dataset_type='train',
-                                    tokenizer=tokenizer)
-
-collate_fn = ConstructSuperglueStrategy(cl_args,
-                                        tokenizer,
-                                        task_name=task_name)
-
-valid_dataset = SuperGlueDataset(task_name=task_name,
-                                    data_dir='./datasets/',
-                                    dataset_type='dev',
-                                    tokenizer=tokenizer)
-
-metric_methods = DEFAULT_METRICS[task_name]
-trainer.train(model,
-                collate_fn=collate_fn,
-                train_dataset=train_dataset,
-                valid_dataset=valid_dataset,
-                metric_methods=metric_methods)
diff --git a/examples/glm_superglue/train_10b_clue.py b/examples/glm_superglue/train_10b_clue.py
index fbfd6882..d97f331a 100644
--- a/examples/glm_superglue/train_10b_clue.py
+++ b/examples/glm_superglue/train_10b_clue.py
@@ -1,19 +1,21 @@
 # Copyright © 2022 BAAI. All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License")
+import os
 from flagai.trainer import Trainer
 from flagai.model.glm_model import GLMForSingleTokenCloze
 from flagai.data.tokenizer import GLMLargeChTokenizer
 from flagai.metrics import accuracy_metric
 from flagai.data.dataset import SuperGlueDataset
 from flagai.test_utils import CollateArguments
+from flagai.data.dataset import ConstructSuperglueStrategy
 
 
 task_name = 'tnews'
 trainer = Trainer(env_type='deepspeed',
                   epochs=2,
-                  batch_size=1,
-                  eval_interval=1000,
+                  batch_size=4,
+                  eval_interval=10,
                   checkpoint_activations=False,
                   fp16=True,
                   log_interval=1,
@@ -33,28 +35,24 @@
 
 tokenizer =  GLMLargeChTokenizer()
 train_dataset = SuperGlueDataset(task_name=task_name,
-                                 data_dir='/mnt/datasets/yan/',
+                                 data_dir='./datasets/',
                                  dataset_type='train',
                                  tokenizer=tokenizer,
                                  cloze_eval=True)
 valid_dataset = SuperGlueDataset(task_name=task_name,
-                                 data_dir='/mnt/datasets/yan/',
+                                 data_dir='./datasets/',
                                  dataset_type='dev',
                                  tokenizer=tokenizer,
                                  cloze_eval=True)
 
 cl_args = CollateArguments()
 cl_args.cloze_eval = True
-if task_name in ['copa', 'wsc', 'record']:
-    cl_args.multi_token = True
-
-from flagai.data.dataset import ConstructSuperglueStrategy
+cl_args.multi_token = False
 
 collate_fn = ConstructSuperglueStrategy(cl_args,
                                         tokenizer,
                                         task_name=task_name)
 trainer.train(model,
-
               train_dataset=train_dataset,
               valid_dataset=valid_dataset,
               collate_fn=collate_fn,
diff --git a/examples/glm_superglue/train_10b_superglue.py b/examples/glm_superglue/train_10b_superglue.py
index 49ce9414..082c86a6 100644
--- a/examples/glm_superglue/train_10b_superglue.py
+++ b/examples/glm_superglue/train_10b_superglue.py
@@ -3,12 +3,13 @@
 # Licensed under the Apache License, Version 2.0 (the "License")
 from flagai.trainer import Trainer
 from flagai.model.glm_model import GLMForSingleTokenCloze
-from flagai.data.tokenizer import GLMLargeEnWordPieceTokenizer
+from flagai.data.tokenizer import GLM10bENBPETokenizer
 from flagai.metrics import accuracy_metric
 from flagai.data.dataset import SuperGlueDataset
 from flagai.test_utils import CollateArguments
 
 
+
 task_name = 'boolq'
 trainer = Trainer(env_type='deepspeed+mpu',
                   epochs=2,
@@ -30,14 +31,15 @@
 model = GLMForSingleTokenCloze.from_pretrain(download_path="/mnt/test_10b_models",
                                              model_name="GLM-10b-en")
 
-tokenizer = GLMLargeEnWordPieceTokenizer()
+tokenizer = GLM10bENBPETokenizer()
+
 train_dataset = SuperGlueDataset(task_name=task_name,
-                                 data_dir='/mnt/datasets/yan/',
+                                 data_dir='./datasets/',
                                  dataset_type='train',
                                  tokenizer=tokenizer,
                                  cloze_eval=True)
 valid_dataset = SuperGlueDataset(task_name=task_name,
-                                 data_dir='/mnt/datasets/yan/',
+                                 data_dir='./datasets/',
                                  dataset_type='dev',
                                  tokenizer=tokenizer,
                                  cloze_eval=True)
diff --git a/examples/glm_title_generation/train.py b/examples/glm_title_generation/train.py
index 78841045..6c68375d 100644
--- a/examples/glm_title_generation/train.py
+++ b/examples/glm_title_generation/train.py
@@ -29,8 +29,9 @@
     num_checkpoints=1,
 )
 
-src_dir = './data/train.src'
-tgt_dir = './data/train.tgt'
+cur_dir = os.path.dirname(os.path.abspath(__file__))
+src_dir = cur_dir + '/data/train.src'
+tgt_dir = cur_dir + '/data/train.tgt'
 
 maxlen = 256
 auto_loader = AutoLoader("seq2seq",
diff --git a/examples/t5_huggingface/deepspeed.json b/examples/t5_huggingface/deepspeed.json
index 89ef65e7..67a24ee9 100644
--- a/examples/t5_huggingface/deepspeed.json
+++ b/examples/t5_huggingface/deepspeed.json
@@ -1,16 +1,16 @@
 {
-  "train_micro_batch_size_per_gpu": 4,
+  "train_micro_batch_size_per_gpu": 2,
   "gradient_accumulation_steps": 1,
-  "steps_per_print": 50,
+  "steps_per_print": 100,
   "gradient_clipping": 1.0,
   "zero_optimization": {
-    "stage": 1,
+    "stage": 3,
     "contiguous_gradients": false,
     "overlap_comm": true,
     "reduce_scatter": true,
     "reduce_bucket_size": 5e7,
     "allgather_bucket_size": 5e7,
-    "cpu_offload": true
+    "cpu_offload": true 
   },
   "zero_allow_untested_optimizer": true,
   "fp16": {
@@ -23,26 +23,18 @@
   "optimizer": {
     "type": "Adam",
     "params": {
-      "lr": 5e-6,
+      "lr": 0.0004,
+      "weight_decay": 0.01,
       "betas": [
         0.9,
-        0.95
+        0.98
       ],
-      "eps": 1e-8,
-      "weight_decay": 1e-2
+      "eps": 1e-6
     }
   },
   "activation_checkpointing": {
-    "partition_activations": true,
+    "partition_activations": false,
     "contiguous_memory_optimization": false
   },
-  "wall_clock_breakdown": false,
-  "flops_profiler": {
-    "enabled": false,
-    "profile_step": 1,
-    "module_depth": -1,
-    "top_modules": 1,
-    "detailed": true,
-    "output_file": "/data/liuguang/t5_profiler_2gpu"
-    }
-}
+  "wall_clock_breakdown": false
+}
\ No newline at end of file
diff --git a/examples/t5_huggingface/train_t5_11b.py b/examples/t5_huggingface/train_t5_11b.py
index a70c85e8..ed4f0d66 100644
--- a/examples/t5_huggingface/train_t5_11b.py
+++ b/examples/t5_huggingface/train_t5_11b.py
@@ -5,7 +5,7 @@
 from transformers import T5ForConditionalGeneration, T5Tokenizer
 from torch.utils.data import Dataset
 import torch
-
+import os
 
 class MyTrainer(Trainer):
 
@@ -36,8 +36,9 @@ def forward_step(self, data, model, mems):
 model.gradient_checkpointing = True
 
 print("loading model & tokenizer is done!")
-src_dir = './data/train.src'
-tgt_dir = './data/train.tgt'
+cur_dir = os.path.dirname(os.path.abspath(__file__))
+src_dir = cur_dir + '/data/train.src'
+tgt_dir = cur_dir + '/data/train.tgt'
 maxlen = 1024
 
 
diff --git a/flagai/trainer.py b/flagai/trainer.py
index 151ed339..ea3e28c8 100644
--- a/flagai/trainer.py
+++ b/flagai/trainer.py
@@ -392,7 +392,7 @@ def train(self,
             optimizer = get_optimizer(param_groups=param_groups,
                                       lr=self.lr,
                                       weight_decay=self.weight_decay,
-                                      cpu_optimizer=True,
+                                      cpu_optimizer=False,
                                       cpu_torch_adam=False,
                                       fp16=self.fp16)
 
@@ -568,7 +568,6 @@ def train_step(self,
             step_output = self.forward_step(data, model, mems)
             self.timers('forward').stop()
             lm_loss = step_output['loss']
-
             if 'deepspeed' not in self.env_type:
                 lm_loss /= self.gradient_accumulation_steps
 
@@ -583,7 +582,6 @@ def train_step(self,
             if 'deepspeed' in self.env_type:
                 reduced_loss.data = reduced_loss.data / \
                     (self.world_size / self.model_parallel_size)
-
             if not DynamicLossScaler._has_inf_or_nan(reduced_loss):
                 lm_loss_total += reduced_loss
                 count += 1
@@ -625,6 +623,7 @@ def train_step(self,
                 mems = []
             if single_step:
                 break
+
         # if train_args.deepspeed:
         if 'deepspeed' in self.env_type:
             lm_loss_total = lm_loss_total / count


Test patch:
diff --git a/flagai/test_utils.py b/flagai/test_utils.py
index 62af5cf2..3473fd53 100644
--- a/flagai/test_utils.py
+++ b/flagai/test_utils.py
@@ -153,7 +153,9 @@ def __init__(self):
         #
         # self.few_superglue = False
         # self.pattern_text = False
-class DatasetArguments:
+
+
+class PretrainDatasetArguments:
 
     def __init__(self):
         self.task_mask = True  # Distinguished the generation and gap-sentence mask
@@ -164,4 +166,5 @@ def __init__(self):
         self.pre_tokenize = True
         self.no_lazy_loader = True
         self.half_lazy_loader = False
-        self.sentinel_token = False
\ No newline at end of file
+        self.sentinel_token = False
+