|
--- |
|
language: zh |
|
tags: |
|
- summarization |
|
--- |
|
|
|
IDEA-CCNL/Randeng_Pegasus_238M_Summary_Chinese model (Chinese) has 238M million parameter, pretrained on 180G Chinese data with GSG task which is stochastically sample important sentences with sampled gap sentence ratios by 25%. The pretraining task just as same as the paper PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization mentioned. |
|
|
|
Different from the English version of pegasus, considering that the Chinese sentence piece is unstable, we use jieba and Bertokenizer as the tokenizer in chinese pegasus model. |
|
|
|
After pre-training, We use 8 summary datasets which we collect on the internet to do the supervised training. The 8 datasets include education_data, new2016zh_data, nlpcc, shence_data, sohu_data, thucnews_data and weibo_data, four million training samples in all. |
|
|
|
|
|
Task: Summarization |
|
## Usage |
|
```python |
|
from typing import List, Optional |
|
import jieba_fast as jieba |
|
jieba.initialize() |
|
from transformers import PegasusForConditionalGeneration,BertTokenizer |
|
# Need to download tokenizers_pegasus.py and other Python script from Fengshenbang-LM github repo in advance, |
|
# or you can download tokenizers_pegasus.py and data_utils.py in https://huggingface.co/IDEA-CCNL/Randeng_Pegasus_523M/tree/main |
|
# Strongly recommend you git clone the Fengshenbang-LM repo: |
|
# 1. git clone https://github.com/IDEA-CCNL/Fengshenbang-LM |
|
# 2. cd Fengshenbang-LM/fengshen/examples/pegasus/ |
|
# and then you will see the tokenizers_pegasus.py and data_utils.py which are needed by pegasus model |
|
|
|
|
|
# from tokenizers_pegasus import PegasusTokenizer |
|
|
|
class PegasusTokenizer(BertTokenizer): |
|
model_input_names = ["input_ids", "attention_mask"] |
|
def __init__(self, **kwargs): |
|
super().__init__(pre_tokenizer=lambda x: jieba.cut(x, HMM=False), **kwargs) |
|
self.add_special_tokens({'additional_special_tokens':["<mask_1>"]}) |
|
|
|
def build_inputs_with_special_tokens( |
|
self, |
|
token_ids_0: List[int], |
|
token_ids_1: Optional[List[int]] = None) -> List[int]: |
|
|
|
if token_ids_1 is None: |
|
return token_ids_0 + [self.eos_token_id] |
|
return token_ids_0 + token_ids_1 + [self.eos_token_id] |
|
|
|
def _special_token_mask(self, seq): |
|
all_special_ids = set( |
|
self.all_special_ids) # call it once instead of inside list comp |
|
# all_special_ids.remove(self.unk_token_id) # <unk> is only sometimes special |
|
return [1 if x in all_special_ids else 0 for x in seq] |
|
|
|
def get_special_tokens_mask( |
|
self, |
|
token_ids_0: List[int], |
|
token_ids_1: Optional[List[int]] = None, |
|
already_has_special_tokens: bool = False) -> List[int]: |
|
if already_has_special_tokens: |
|
return self._special_token_mask(token_ids_0) |
|
elif token_ids_1 is None: |
|
return self._special_token_mask(token_ids_0) + [self.eos_token_id] |
|
else: |
|
return self._special_token_mask(token_ids_0 + |
|
token_ids_1) + [self.eos_token_id] |
|
|
|
model = PegasusForConditionalGeneration.from_pretrained("IDEA-CCNL/Randeng-Pegasus-238M-Summary-Chinese") |
|
tokenizer = PegasusTokenizer.from_pretrained("IDEA-CCNL/Randeng-Pegasus-238M-Summary-Chinese") |
|
|
|
text = "在北京冬奥会自由式滑雪女子坡面障碍技巧决赛中，中国选手谷爱凌夺得银牌。祝贺谷爱凌！今天上午，自由式滑雪女子坡面障碍技巧决赛举行。决赛分三轮进行，取选手最佳成绩排名决出奖牌。第一跳，中国选手谷爱凌获得69.90分。在12位选手中排名第三。完成动作后，谷爱凌又扮了个鬼脸，甚是可爱。第二轮中，谷爱凌在道具区第三个障碍处失误，落地时摔倒。获得16.98分。网友：摔倒了也没关系，继续加油！在第二跳失误摔倒的情况下，谷爱凌顶住压力，第三跳稳稳发挥，流畅落地！获得86.23分！此轮比赛，共12位选手参赛，谷爱凌第10位出场。网友：看比赛时我比谷爱凌紧张，加油！" |
|
inputs = tokenizer(text, max_length=1024, return_tensors="pt") |
|
|
|
# Generate Summary |
|
summary_ids = model.generate(inputs["input_ids"]) |
|
tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] |
|
|
|
# model Output: 滑雪女子坡面障碍技巧决赛谷爱凌获银牌 |
|
``` |
|
|
|
## Citation |
|
If you find the resource is useful, please cite the following website in your paper. |
|
``` |
|
@misc{Fengshenbang-LM, |
|
title={Fengshenbang-LM}, |
|
author={IDEA-CCNL}, |
|
year={2022}, |
|
howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}}, |
|
} |
|
``` |
