y-tag commited on
Commit
ed066e8
1 Parent(s): f411ce0

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +76 -0
README.md ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: ja
3
+ license: mit
4
+ ---
5
+
6
+ # Japanese DistilBERT Pretrained Model
7
+ A Japanese DistilBERT pretrained model, which was trained on [Wikipedia](https://ja.wikipedia.org/).
8
+
9
+ Find [here](https://github.com/BandaiNamcoResearchInc/DistilBERT-base-jp/blob/master/docs/GUIDE.md) for a quickstart guidance in Japanese.
10
+
11
+ ## Table of Contents
12
+
13
+ 1. [Introduction](#Introduction)
14
+ 1. [Requirements](#Requirements)
15
+ 1. [Usage](#Usage)
16
+ 1. [License](#License)
17
+
18
+ ## Introduction
19
+ DistilBERT is a small, fast, cheap and light Transformer model based on Bert architecture. It has 40% less parameters than BERT-base, runs 60% faster while preserving 97% of BERT's performance as measured on the GLUE language understanding benchmark.
20
+
21
+ This model was trained with the official Hugging Face implementation from [here](https://github.com/huggingface/transformers/tree/master/examples/distillation) for 2 weeks on AWS p3dn.24xlarge instance.
22
+
23
+ More details about distillation can be found in following paper.
24
+ ["DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter"](https://arxiv.org/abs/1910.01108) by Sanh et al. (2019).
25
+
26
+ The teacher model is [the pretrained Japanese BERT models from TOHOKU NLP LAB](https://www.nlp.ecei.tohoku.ac.jp/news-release/3284/).
27
+
28
+ Currently only PyTorch compatible weights are available. Tensorflow checkpoints can be generated by following the [official guide](https://github.com/huggingface/transformers).
29
+
30
+ ## Requirements
31
+
32
+ ```
33
+ torch>=1.3.1
34
+ torchvision>=0.4.2
35
+ transformers>=2.5.0
36
+ tensorboard>=1.14.0
37
+ tensorboardX==1.8
38
+ scikit-learn>=0.21.0
39
+ mecab-python3
40
+ ```
41
+
42
+ ## Usage
43
+
44
+ ### Download model
45
+
46
+ Please download and unzip [DistilBERT-base-jp.zip](https://github.com/BandaiNamcoResearchInc/DistilBERT-base-jp/releases).
47
+
48
+ ### Use model
49
+
50
+ ```python
51
+ # Read from local path
52
+ from transformers import AutoModel, AutoTokenizer
53
+ tokenizer = AutoTokenizer.from_pretrained("bert-base-japanese-whole-word-masking")
54
+ model = AutoModel.from_pretrained("LOCAL_PATH")
55
+ ```
56
+
57
+ LOCAL_PATH means the path which above file is unzipped. 3 files should be included:
58
+
59
+ - pytorch_model.bin
60
+ - config.json
61
+ - vocal.txt
62
+
63
+ or
64
+ ```python
65
+ # Download from model library from huggingface.co
66
+ from transformers import AutoModel, AutoTokenizer
67
+ tokenizer = AutoTokenizer.from_pretrained("bert-base-japanese-whole-word-masking")
68
+ model = AutoModel.from_pretrained("bandainamco-mirai/distilbert-base-japanese")
69
+ ```
70
+
71
+ ## License
72
+ Copyright (c) 2020 BANDAI NAMCO Research Inc.
73
+
74
+ Released under the MIT license
75
+
76
+ https://opensource.org/licenses/mit-license.php