Create README.md

README.md

@@ -0,0 +1,76 @@
+---
+language: ja
+license: mit
+---
+
+# Japanese DistilBERT Pretrained Model
+A Japanese DistilBERT pretrained model, which was trained on [Wikipedia](https://ja.wikipedia.org/).
+
+Find [here](https://github.com/BandaiNamcoResearchInc/DistilBERT-base-jp/blob/master/docs/GUIDE.md) for a quickstart guidance in Japanese.
+
+## Table of Contents
+
+1. [Introduction](#Introduction)
+1. [Requirements](#Requirements)
+1. [Usage](#Usage)
+1. [License](#License)
+
+## Introduction
+DistilBERT is a small, fast, cheap and light Transformer model based on Bert architecture. It has 40% less parameters than BERT-base, runs 60% faster while preserving 97% of BERT's performance as measured on the GLUE language understanding benchmark.
+
+This model was trained with the official Hugging Face implementation from [here](https://github.com/huggingface/transformers/tree/master/examples/distillation) for 2 weeks on AWS p3dn.24xlarge instance.
+
+More details about distillation can be found in following paper.
+["DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter"](https://arxiv.org/abs/1910.01108) by Sanh et al. (2019).
+
+The teacher model is [the pretrained Japanese BERT models from TOHOKU NLP LAB](https://www.nlp.ecei.tohoku.ac.jp/news-release/3284/). 
+
+Currently only PyTorch compatible weights are available. Tensorflow checkpoints can be generated by following the [official guide](https://github.com/huggingface/transformers).
+
+## Requirements
+
+```
+torch>=1.3.1
+torchvision>=0.4.2
+transformers>=2.5.0
+tensorboard>=1.14.0
+tensorboardX==1.8
+scikit-learn>=0.21.0
+mecab-python3
+```
+
+## Usage
+
+### Download model
+
+Please download and unzip [DistilBERT-base-jp.zip](https://github.com/BandaiNamcoResearchInc/DistilBERT-base-jp/releases).
+
+### Use model
+
+```python
+# Read from local path
+from transformers import AutoModel, AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("bert-base-japanese-whole-word-masking")
+model = AutoModel.from_pretrained("LOCAL_PATH")                                     
+```
+
+LOCAL_PATH means the path which above file is unzipped. 3 files should be included:
+
+- pytorch_model.bin
+- config.json
+- vocal.txt
+
+or
+```python
+# Download from model library from huggingface.co
+from transformers import AutoModel, AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("bert-base-japanese-whole-word-masking")
+model = AutoModel.from_pretrained("bandainamco-mirai/distilbert-base-japanese")     
+```
+
+## License
+Copyright (c) 2020 BANDAI NAMCO Research Inc.
+
+Released under the MIT license
+
+https://opensource.org/licenses/mit-license.php