sbert-base-ja / README.md
1
---
2
language: ja
3
pipeline_tag: sentence-similarity
4
tags:
5
- sentence-transformers
6
- feature-extraction
7
- sentence-similarity
8
widget:
9
    source_sentence: "走るのが趣味です"
10
    sentences:
11
        - 外をランニングするのが好きです
12
        - 運動はそこそこです
13
        - 走るのは嫌いです
14
license: cc-by-sa-4.0
15
---
16
17
# Sentence BERT base Japanese model
18
19
This repository contains a Sentence BERT base model for Japanese.
20
21
## Pretrained model
22
23
This model utilizes a Japanese BERT model [colorfulscoop/bert-base-ja](https://huggingface.co/colorfulscoop/bert-base-ja) v1.0 released under [Creative Commons Attribution-ShareAlike 3.0](https://creativecommons.org/licenses/by-sa/3.0/) as a pretrained model.
24
25
## Training data
26
27
[Japanese SNLI dataset](https://nlp.ist.i.kyoto-u.ac.jp/index.php?%E6%97%A5%E6%9C%AC%E8%AA%9ESNLI%28JSNLI%29%E3%83%87%E3%83%BC%E3%82%BF%E3%82%BB%E3%83%83%E3%83%88) released under [Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/) is used for training.
28
29
Original training dataset is splitted into train/valid dataset. Finally, follwoing data is prepared.
30
31
* Train data: 523,005 samples
32
* Valid data: 10,000 samples
33
* Test data: 3,916 samples
34
35
## Model description
36
37
This model utilizes `SentenceTransformer` model from the [sentence-transformers](https://github.com/UKPLab/sentence-transformers) .
38
The model detail is as below.
39
40
```py
41
>>> from sentence_transformers import SentenceTransformer
42
>>> SentenceTransformer("colorfulscoop/sbert-base-ja")
43
SentenceTransformer(
44
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
45
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
46
)
47
```
48
49
## Training
50
51
This model finetuned [colorfulscoop/bert-base-ja](https://huggingface.co/colorfulscoop/bert-base-ja) with Softmax classifier of 3 labels of SNLI. AdamW optimizer with learning rate of 2e-05 linearly warmed-up in 10% of train data was used. The model was trained in 1 epoch with batch size 8.
52
53
Note: in a original paper of [Sentence BERT](https://arxiv.org/abs/1908.10084), a batch size of the model trained on SNLI and Multi-Genle NLI was 16. In this model, the dataset is around half smaller than the origial one, therefore the batch size was set to half of the original batch size of 16.
54
55
Trainind was conducted on Ubuntu 18.04.5 LTS with one RTX 2080 Ti.
56
57
After training, test set accuracy reached to 0.8529.
58
59
Training code is available in [a GitHub repository](https://github.com/colorfulscoop/sbert-ja).
60
61
## Usage
62
63
First, install dependecies.
64
65
```sh
66
$ pip install sentence-transformers==2.0.0
67
```
68
69
Then initialize `SentenceTransformer` model and use `encode` method to convert to vectors.
70
71
```py
72
>>> from sentence_transformers import SentenceTransformer
73
>>> model = SentenceTransformer("colorfulscoop/sbert-base-ja")
74
>>> sentences = ["外をランニングするのが好きです", "海外旅行に行くのが趣味です"]
75
>>> model.encode(sentences)
76
```
77
78
## License
79
80
Copyright (c) 2021 Colorful Scoop
81
82
All the models included in this repository are licensed under [Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
83
84
**Disclaimer:** Use of this model is at your sole risk. Colorful Scoop makes no warranty or guarantee of any outputs from the model. Colorful Scoop is not liable for any trouble, loss, or damage arising from the model output.
85
86
---
87
88
This model utilizes the folllowing pretrained model.
89
90
* **Name:** bert-base-ja
91
* **Credit:** (c) 2021 Colorful Scoop
92
* **License:** [Creative Commons Attribution-ShareAlike 3.0](https://creativecommons.org/licenses/by-sa/3.0/)
93
* **Disclaimer:** The model potentially has possibility that it generates similar texts in the training data, texts not to be true, or biased texts. Use of the model is at your sole risk. Colorful Scoop makes no warranty or guarantee of any outputs from the model. Colorful Scoop is not liable for any trouble, loss, or damage arising from the model output.
94
* **Link:** https://huggingface.co/colorfulscoop/bert-base-ja
95
96
---
97
98
This model utilizes the following data for fine-tuning.
99
100
* **Name:** 日本語SNLI(JSNLI)データセット
101
* **Credit:** [https://nlp.ist.i.kyoto-u.ac.jp/index.php?日本語SNLI(JSNLI)データセット](https://nlp.ist.i.kyoto-u.ac.jp/index.php?%E6%97%A5%E6%9C%AC%E8%AA%9ESNLI%28JSNLI%29%E3%83%87%E3%83%BC%E3%82%BF%E3%82%BB%E3%83%83%E3%83%88)
102
* **License:** [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)
103
* **Link:** [https://nlp.ist.i.kyoto-u.ac.jp/index.php?日本語SNLI(JSNLI)データセット](https://nlp.ist.i.kyoto-u.ac.jp/index.php?%E6%97%A5%E6%9C%AC%E8%AA%9ESNLI%28JSNLI%29%E3%83%87%E3%83%BC%E3%82%BF%E3%82%BB%E3%83%83%E3%83%88)