---
language: ja
license: mit
---

# Japanese DistilBERT Pretrained Model
A Japanese DistilBERT pretrained model, which was trained on [Wikipedia](https://ja.wikipedia.org/).

Find [here](https://github.com/BandaiNamcoResearchInc/DistilBERT-base-jp/blob/master/docs/GUIDE.md) for a quickstart guidance in Japanese.

## Table of Contents

1. [Introduction](#Introduction)
1. [Requirements](#Requirements)
1. [Usage](#Usage)
1. [License](#License)

## Introduction
DistilBERT is a small, fast, cheap and light Transformer model based on Bert architecture. It has 40% less parameters than BERT-base, runs 60% faster while preserving 97% of BERT's performance as measured on the GLUE language understanding benchmark.

This model was trained with the official Hugging Face implementation from [here](https://github.com/huggingface/transformers/tree/master/examples/distillation) for 2 weeks on AWS p3dn.24xlarge instance.

More details about distillation can be found in following paper.
["DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter"](https://arxiv.org/abs/1910.01108) by Sanh et al. (2019).

The teacher model is [the pretrained Japanese BERT models from TOHOKU NLP LAB](https://www.nlp.ecei.tohoku.ac.jp/news-release/3284/). 

Currently only PyTorch compatible weights are available. Tensorflow checkpoints can be generated by following the [official guide](https://github.com/huggingface/transformers).

## Requirements

```
torch>=1.3.1
torchvision>=0.4.2
transformers>=2.5.0
tensorboard>=1.14.0
tensorboardX==1.8
scikit-learn>=0.21.0
mecab-python3
```

## Usage

### Download model

Please download and unzip [DistilBERT-base-jp.zip](https://github.com/BandaiNamcoResearchInc/DistilBERT-base-jp/releases).

### Use model

```python
# Read from local path
from transformers import AutoModel, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-japanese-whole-word-masking")
model = AutoModel.from_pretrained("LOCAL_PATH")                                     
```

LOCAL_PATH means the path which above file is unzipped. 3 files should be included:

- pytorch_model.bin
- config.json
- vocal.txt

or
```python
# Download from model library from huggingface.co
from transformers import AutoModel, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-japanese-whole-word-masking")
model = AutoModel.from_pretrained("bandainamco-mirai/distilbert-base-japanese")     
```

## License
Copyright (c) 2020 BANDAI NAMCO Research Inc.

Released under the MIT license

https://opensource.org/licenses/mit-license.php