Package: asian-bart
Asian Bartasian-bartis package of Bart model for Asian languages.asian-bartsupports English, Chinese, Korean, japanese, Total (=ECJK)We madeasian-bartusingmBartby embedding layer pruning.Installationpip install asian-bartModel specificationECJK modelvocab size: 57kmodel size: 413Mlanguages: En, Zh, Ja, Koarchitecture: Transformer 12 Encoder + 12 Decodername:hyunwoongko/asian-bart-ecjkEnglish modelvocab size: 32kmodel size: 387Mlanguages: English (en_XX)architecture: Transformer 12 Encoder + 12 Decodername:hyunwoongko/asian-bart-enChinese modelvocab size: 20kmodel size: 375Mlanguages: Chinese (zh_CN)architecture: Transformer 12 Encoder + 12 Decodername:hyunwoongko/asian-bart-zhJapanese modelvocab size: 13kmodel size: 368Mlanguages: Japanese (ja_XX)architecture: Transformer 12 Encoder + 12 Decodername:hyunwoongko/asian-bart-jaKorean modelvocab size: 8kmodel size: 363Mlanguages: Korean (ko_KR)architecture: Transformer 12 Encoder + 12 Decodername:hyunwoongko/asian-bart-koUsageTheasian-bartis made using mbart, so you have to follow mbart's input rules:source:text+
+lang_codetarget:lang_code+text+
For more details, please check the content of thembart paper.Usage of tokenizertokenization of(single language, single text)>>>fromasian_bartimportAsianBartTokenizer>>>tokenizer=AsianBartTokenizer.from_pretrained("hyunwoongko/asian-bart-ecjk")>>>tokenizer.prepare_seq2seq_batch(...src_texts="hello.",...src_langs="en_XX",...){
  'input_ids': tensor([[37199, 35816,     2, 57521]]), 
  'attention_mask': tensor([[1, 1, 1, 1]])
}batch tokenization of(single language, mutiple texts)>>>fromasian_bartimportAsianBartTokenizer>>>tokenizer=AsianBartTokenizer.from_pretrained("hyunwoongko/asian-bart-ecjk")>>>tokenizer.prepare_seq2seq_batch(...src_texts=["hello.","how are you?","good."],...src_langs="en_XX",...){
  'input_ids': tensor([[37199, 35816,     2, 57521,     1,     1],
                       [38248, 46819, 39446, 36209,     2, 57521],
                       [40010, 39539,     2, 57521,     1,     1]]), 

  'attention_mask': tensor([[1, 1, 1, 1, 0, 0],
                            [1, 1, 1, 1, 1, 1],
                            [1, 1, 1, 1, 0, 0]])
}batch tokenization of(multiple languages, multiple texts)>>>fromasian_bartimportAsianBartTokenizer>>>tokenizer=AsianBartTokenizer.from_pretrained("hyunwoongko/asian-bart-ecjk")>>>tokenizer.prepare_seq2seq_batch(...src_texts=["hello.","반가워","你好","こんにちは"],...src_langs=["en_XX","ko_KR","zh_CN","ja_XX"],...){
  'input_ids': tensor([[37199, 35816, 39539,     2, 57521,     1,     1,     1],
                       [22880, 49591,  3901,     2, 57523,     1,     1,     1],
                       [50356,  7929,     2, 57524,     1,     1,     1,     1],
                       [42990, 19092, 51547, 36821, 33899, 37382,     2, 57522]]), 

   'attention_mask': tensor([[1, 1, 1, 1, 1, 0, 0, 0],
                             [1, 1, 1, 1, 1, 0, 0, 0],
                             [1, 1, 1, 1, 0, 0, 0, 0],
                             [1, 1, 1, 1, 1, 1, 1, 1]])
}seq2seq tokenization of(source text, target text)>>>fromasian_bartimportAsianBartTokenizer>>>tokenizer=AsianBartTokenizer.from_pretrained("hyunwoongko/asian-bart-ecjk")>>>tokenizer.prepare_seq2seq_batch(...src_texts="반가워",...src_langs="ko_KR",...tgt_texts="hello.",...tgt_langs="en_XX",...){
  'input_ids': tensor([[22880, 49591,  3901,     2, 57523]]), 
  'attention_mask': tensor([[1, 1, 1, 1, 1]]), 
  'labels': tensor([[37199, 35816, 39539,     2, 57521]])
}all above batch tokenization settings work the same about target texts>>>fromasian_bartimportAsianBartTokenizer>>>tokenizer=AsianBartTokenizer.from_pretrained("hyunwoongko/asian-bart-ecjk")>>>tokenizer.prepare_seq2seq_batch(...src_texts=["hello.","반가워","你好","こんにちは"],...src_langs=["en_XX","ko_KR","zh_CN","ja_XX"],...tgt_texts=["hello.","반가워","你好","こんにちは"],...tgt_langs=["en_XX","ko_KR","zh_CN","ja_XX"],...){
  'input_ids': tensor([[37199, 35816, 39539,     2, 57521,     1,     1,     1],
                      [22880, 49591,  3901,     2, 57523,     1,     1,     1],
                      [50356,  7929,     2, 57524,     1,     1,     1,     1],
                      [42990, 19092, 51547, 36821, 33899, 37382,     2, 57522]]), 

  'attention_mask': tensor([[1, 1, 1, 1, 1, 0, 0, 0],
                            [1, 1, 1, 1, 1, 0, 0, 0],
                            [1, 1, 1, 1, 0, 0, 0, 0],
                            [1, 1, 1, 1, 1, 1, 1, 1]]), 

  'labels': tensor([[37199, 35816, 39539,     2, 57521,     1,     1,     1],
                    [22880, 49591,  3901,     2, 57523,     1,     1,     1],
                    [50356,  7929,     2, 57524,     1,     1,     1,     1],
                    [42990, 19092, 51547, 36821, 33899, 37382,     2, 57522]])
}Usage of modelsInterfaces of all functions are the same as mbart model on Huggingface transformers.Here is an example of using a asian bart model. (ecjk model)Other language work the same way. change both model and tokenizer'sfrom_pretrained.English only:from_pretrained("hyunwoongko/asian-bart-en")Chinese only:from_pretrained("hyunwoongko/asian-bart-zh")Japanese only:from_pretrained("hyunwoongko/asian-bart-ja")Korean only:from_pretrained("hyunwoongko/asian-bart-ko")# import modules>>>importtorch>>>fromasian_bartimportAsianBartTokenizer,AsianBartForConditionalGeneration>>>fromtransformers.models.bart.modeling_bartimportshift_tokens_right# create model and tokenizer>>>tokenizer=AsianBartTokenizer.from_pretrained("hyunwoongko/asian-bart-ecjk")>>>model=AsianBartForConditionalGeneration.from_pretrained("hyunwoongko/asian-bart-ecjk")# tokenize texts>>>tokens=tokenizer.prepare_seq2seq_batch(...src_texts="Kevin is the <mask> man in the world.",...src_langs="en_XX",...tgt_texts="Kevin is the most kind man in the world.",...tgt_langs="en_XX",...)>>>input_ids=tokens["input_ids"]>>>attention_mask=tokens["attention_mask"]>>>labels=tokens["labels"]>>>decoder_input_ids=shift_tokens_right(labels,tokenizer.pad_token_id)# forwarding model for training>>>output=model(...input_ids=input_ids,...attention_mask=attention_mask,...decoder_input_ids=decoder_input_ids,...)# compute loss>>>lm_logits=outputs[0]>>>loss_function=torch.nn.CrossEntropyLoss(...ignore_index=tokenizer.pad_token_id...)>>>loss=loss_function(...lm_logits.view(-1,lm_logits.shape[-1]),...labels.view(-1)...)# generate text>>>output=model.generate(...input_ids=input_ids,...attention_mask=attention_mask,...decoder_start_token_id=tokenizer.lang_code_to_id["en_XX"],...)Downstream tasksYou can train various downstream tasks with asian bart.All interfaces have the same usage as the Huggingface transformers.Supported classes:AsianBartTokenizerAsianBartModelAsianBartForCausalLMAsianBartForQuestionAnsweringAsianBartForConditionalGenerationAsianBartForSequenceClassificationLicenseCopyright 2021 Hyunwoong Ko.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.