Package: longformer-chinese
Longformer-chineseAll work is based onLongformer(https://github.com/allenai/longformer)Longformer-chinese提供了：基于BERT的中文预训练模型、在分类任务上的实现WHAT'S DIFFERENTLongformer-chinese基于BERT框架进行修改，在embedding层会与原版的稍有区别。加载时使用longformer.longformer：from longformer.longformer import *
config = LongformerConfig.from_pretrained('schen/longformer-chinese-base-4096')
model = Longformer.from_pretrained('schen/longformer-chinese-base-4096', config=config)使用嵌入fromlongformer.longformerimportLongformer,LongformerConfig,LongformerEmbedding# model=LongformerEmbedding('schen/longformer-chinese-base-4096',attention_mode='n2')model=LongformerEmbedding('schen/longformer-chinese-base-4096')# inputs = model.tokenizer("Hello, my dog is cute", return_tensors="pt",padding="max_length",truncation=True,max_length=model.tokenizer.model_max_length)inputs=model.tokenizer("Hello, my dog is cute",return_tensors="pt",padding="max_length",truncation=True,max_length=40)outputs=model(**inputs)print("outputs",outputs)print("outputs",outputs.keys())print("outputs",outputs['last_hidden_state'].size())使用schen/longformer-chinese-base-4096会自动从transformers下载预训练模型，也可以自行下载后替换成所在目录：https://huggingface.co/schen/longformer-chinese-base-4096How to useDownload pretrained modellongformer-base-4096longformer-large-4096Install environment and codecondacreate--namelongformerpython=3.7
condaactivatelongformer
condainstallcudatoolkit=10.0
pipinstallgit+https://github.com/allenai/longformer.gitRun the modelimporttorchfromlongformer.longformerimportLongformer,LongformerConfigfromlongformer.sliding_chunksimportpad_to_window_sizefromtransformersimportRobertaTokenizerconfig=LongformerConfig.from_pretrained('longformer-base-4096/')# choose the attention mode 'n2', 'tvm' or 'sliding_chunks'# 'n2': for regular n2 attantion# 'tvm': a custom CUDA kernel implementation of our sliding window attention# 'sliding_chunks': a PyTorch implementation of our sliding window attentionconfig.attention_mode='sliding_chunks'model=Longformer.from_pretrained('longformer-base-4096/',config=config)tokenizer=RobertaTokenizer.from_pretrained('roberta-base')tokenizer.model_max_length=model.config.max_position_embeddingsSAMPLE_TEXT=' '.join(['Hello world! ']*1000)# long input documentinput_ids=torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)# batch of size 1# TVM code doesn't work on CPU. Uncomment this if `config.attention_mode = 'tvm'`# model = model.cuda(); input_ids = input_ids.cuda()# Attention mask values -- 0: no attention, 1: local attention, 2: global attentionattention_mask=torch.ones(input_ids.shape,dtype=torch.long,device=input_ids.device)# initialize to local attentionattention_mask[:,[1,4,21,]]=2# Set global attention based on the task. For example,# classification: the 
 token# QA: question tokens# padding seqlen to the nearest multiple of 512. Needed for the 'sliding_chunks' attentioninput_ids,attention_mask=pad_to_window_size(input_ids,attention_mask,config.attention_window[0],tokenizer.pad_token_id)output=model(input_ids,attention_mask=attention_mask)[0]Model pretrainingThis notebookdemonstrates our procedure for training Longformer starting from the RoBERTa checkpoint. The same procedure can be followed to get a long-version of other existing pretrained models.TriviaQATraining scripts:scripts/triviaqa.pyPretrained large model:here(replicates leaderboard results)Instructions:scripts/cheatsheet.txtCUDA kernelOur custom CUDA kernel is implemented in TVM.  For now, the kernel only works on GPUs and Linux. We tested it on Ubuntu, Python 3.7, CUDA10, PyTorch >= 1.2.0. If it doesn't work for your environment, please create a new issue.Compiling the kernel: We already include the compiled binaries of the CUDA kernel, so most users won't need to compile it, but if you are intersted, checkscripts/cheatsheet.txtfor instructions.Known issuesPlease check the repoissuesfor a list of known issues that we are planning to address soon. If your issue is not discussed, please create a new one.CitingIf you useLongformerin your research, please citeLongformer: The Long-Document Transformer.@article{Beltagy2020Longformer,
  title={Longformer: The Long-Document Transformer},
  author={Iz Beltagy and Matthew E. Peters and Arman Cohan},
  journal={arXiv:2004.05150},
  year={2020},
}Longformeris an open-source project developed bythe Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.