英語のデータセットをつかった日本語の文章の分類

### 目的

今回は[empathetic dialogue dataset](https://github.com/facebookresearch/EmpatheticDialogues)を使って, 日本語の文章の分類をしていきたいと思います.

### 問題点

まず, このempathtic dialogue dataset は英語のデータセットです. しかしながら, facebook の研究だけあって, 文章自体の質はとてもいいように感じます. なぜ, このdatasetを選んだかというと文章の感情分析を pos / neg よりも詳細に行いたかったためです. それはおいておいて, 英語のデータセットをに日本語で利用したい場合, 2通りの方法が考えられます.

1. 英語のデータセットのまま学習し, 利用時には日本語->英語の翻訳を利用する.
2. データセットを日本語に変換して学習し, 利用時には日本語のまま利用する.

1と2では直感的には1のほうがいいように感じますが, 実際には不明です. 今回はfastTextを使って検証しました.

###　下準備

#### データセット
英語->日本語のデータセットへの変換には[deepl](https://www.deepl.com/ja/translator)を利用させていただきました.

### 正規化

文章の正規化には英語・日本語についてそれぞれ

1. stop words による正規化あり + 単語の normalization あり
2. stop words による正規化あり + 単語の normalization なし
3. 記号のみの stop words による正規化あり

の3 パターンについて調べました.

英語の正規化のスクリプトはこれ

```en_regularizer.py
import cleantext
import nltk

try:
    nltk.download('stopwords')
    nltk.download('wordnet')
except:
    pass

from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from nltk.stem import WordNetLemmatizer

stop = stopwords.words('english')
steamer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

def en_clean(sentence, lemma=False, use_normalize=True):
    words = cleantext.clean_words(sentence,
                                clean_all=False,
                                extra_spaces= True,
                                stemming= False,
                                stopwords= False,
                                lowercase = True,
                                numbers= True,
                                punct = True)
    words = [word for word in words if word not in stop]
    if use_normalize:
        if not lemma:
            words = [steamer.stem(word, to_lowercase=True) for word in words]
        else:
            words = [lemmatizer.lemmatize(word) for word in words]
    return ' '.join(words)
```

日本語の正規化のスクリプトはこれ

```ja_regularizer.py
import re
import unicodedata
import MeCab
mecab = MeCab.Tagger()
from sudachipy import dictionary, tokenizer
tokenizer_obj = dictionary.Dictionary().create()
mode = tokenizer.Tokenizer.SplitMode.C
from ja_stopword_remover.remover import StopwordRemover
remover = StopwordRemover()
mecab_wakachi = MeCab.Tagger('-Owakati')


def clean_text(text):
    # replaced_text = '\n'.join(s.strip() for s in text.splitlines()[2:] if s != '')  # skip header by [2:]
    replaced_text = text.lower()
    replaced_text = re.sub(r'[【】]', ' ', replaced_text)       # 【】の除去
    replaced_text = re.sub(r'[（）()]', ' ', replaced_text)     # （）の除去
    replaced_text = re.sub(r'[［］\[\]]', ' ', replaced_text)   # ［］の除去
    replaced_text = re.sub(r'[@＠]\w+', '', replaced_text)  # メンションの除去
    replaced_text = re.sub(r'https?:\/\/.*?[\r\n ]', '', replaced_text)  # URLの除去
    replaced_text = re.sub(r'　', ' ', replaced_text)  # 全角空白の除去
    return replaced_text

def clean_url(html_text):
    """
    \S+ matches all non-whitespace characters (the end of the url)
    :param html_text:
    :return:
    """
    clean_text = re.sub(r'http\S+', '', html_text)
    return clean_text

def normalize(text):
    normalized_text = normalize_unicode(text)
    normalized_text = normalize_number(normalized_text)
    normalized_text = lower_text(normalized_text)
    return normalized_text


def lower_text(text):
    return text.lower()


def normalize_unicode(text, form='NFKC'):
    normalized_text = unicodedata.normalize(form, text)
    return normalized_text


def remove_stopwords(sentence, stopwords="、。！!#$%&'()-=~|▁^-[]{}「」【】『』〔〕《》（）〖〗〚〛〘〙～＾＝・ー.,…_\\/<>＜＞《》≪≫〈〉“\""):
    res_sentence = ''
    for ch in sentence:
        if ch in stopwords:
            res_sentence += ' '
        else:
            res_sentence += ch
    res_sentence = ' '.join(res_sentence.split())
    return res_sentence

def normalize_number(text):
    """
    pattern = r'\d+'
    replacer = re.compile(pattern)
    result = replacer.sub('0', text)
    """
    # 連続した数字を0で置換
    replaced_text = re.sub(r'\d+', '0', text)
    return replaced_text

def mecab_parser(sentence, normlaize_):
    parsed = mecab.parse(sentence).split('\n')[:-2]
    if normlaize_:
        normalized_tokens = [p.split(',')[7].split('-')[0] for p in parsed]
    else:
        normalized_tokens = [p.split('\t')[0] for p in parsed]
    return normalized_tokens

def normalize_word(text, sudachi=False, normalize_=True):
    if not sudachi:
        return mecab_parser(text, normalize_)
    else:
        words = tokenizer_obj.tokenize(text, mode=mode)
        if normalize_:
            words  = [word.normalized_form() for word in words]
        else:
            words = [word.surface() for word in words]
        return words

def ja_clean(text, sudachi=True, normalize_=True):
    text = clean_text(text)
    text = clean_url(text)
    text = normalize(text)
    words = normalize_word(text, sudachi, normalize_)
    words = remover.remove([words])[0]
    return ' '.join(words)

def ja_wakachi(text):
    text = clean_text(text)
    text = clean_url(text)
    text = normalize(text)
    text = remove_stopwords(text)
    words = mecab_wakachi.parse(text)
    words = words.replace('\n', '')
    return words
```

英語・日本語それぞれ [cleantext](https://pypi.org/project/clean-text/)や[ja_stopword_remover](https://pypi.org/project/ja-stopword-remover/)といった大変便利なライブラリを利用させて頂きました. 本当に助かりました.

次にempathetic dialogue dataset をutters(発言)/label(感情のラベル)/speaker_id(話者id)に分割するスクリプトはこれになります.

```empathetic_dialogues.py
import os
import pandas as pd
import string
from tqdm import tqdm

def get_files():
    files = ['train.csv', 'test.csv', 'valid.csv']
    utterances, labels, speaker_ids = list(), list(), list()
    for file in files:
        with open(file, 'r', encoding='utf-8') as f:
            rows = f.readlines()
        print(f'length of {file} at {len(rows) -1 }')
        for row in rows[1:]:
            row = row.split(',')
            if "<NUMERAL>" in row[-1]:
                label, line, speaker_id = row[2], row[3], row[4]
            else:
                label, line, speaker_id = row[2], row[5], row[4]
            line = line.replace('_comma_', ' ,')
            line = line.replace('&lt;', '<').replace('&gt;', '>').replace('&amp;', '&').replace('&quot;', "'")
                utterances.append(line)
                labels.append(label)
                speaker_ids.append(speaker_id)
            else:
                pass
    assert len(labels) == len(utterances) == len(speaker_ids)
    return utterances, labels, speaker_ids

def main():
    utters, emotions, speaker_ids = get_files()
    with open('resources/empatheticdialogues_utterances.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(utters))
    with open('resources/empatheticdialogues_sentiments.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(emotions))
    with open('resources/empatheticdialogues_emotions.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(list(set(emotions))))
    with open('resources/empatheticdialogues_speaker_ids.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(speaker_ids))

if __name__=='__main__':
    main()
```

このスクリプトでは "\_\_comma\_\_" を "," に置換する作業と, &ltなどの特殊文字を置換する作業を行っています.


- empatheticdailogues_utterances.txt : 発言のファイル
- empatheticdialogues_sentiments.txt : 感情ラベルのファイル
- empatheticdialogues_emotions.txt : 感情ラベルの一覧(32種類存在)
- empatheticdialogues_speaker_ids.txt :話者のidのファイル

以上が保存するファイルになります.

次にデータセットを作成するスクリプトはこれになります.
ただし, empatheticdialogues_utterances_translated.txtはempatheticdialogues_utterances.txtを日本語に翻訳したものとします.

```create_dataset.py
from en_regularizer import en_clean
from ja_regularizer import ja_clean, ja_wakachi
import random
from tqdm import tqdm
import re

en_base_path = 'resources/empatheticdialogues_utterances.txt'
ja_base_path = 'resources/empatheticdialogues_utterances_translated.txt'
label_path = 'resources/empatheticdialogues_sentiments.txt'
speaker_ids_path = 'resources/empatheticdialogues_speaker_ids.txt'

train_length, test_length, valid_length = 84169, 10973, 12078

def _split_words(sentence):
    words = re.sub(r'[^\w]', ' ', sentence).split()
    return words

def _split_dataset(dataset):
    assert len(dataset) == train_length + test_length + valid_length
    train_dataset = dataset[: train_length] + dataset[train_length + test_length : ]
    test_dataset = dataset[train_length : train_length + test_length]
    assert len(train_dataset) == valid_length + train_length and len(test_dataset) == test_length
    return train_dataset, test_dataset

def get_datasets(datas0, datas1, datas2, labels, speaker_ids):
    dataset0, dataset1, dataset2 = list(), list(), list() #normalize + stopwords, only stopwords, None
    start_idx = 0
    main_id = speaker_ids[0]
    for idx, label, id in tqdm(zip(range(len(datas0)),labels, speaker_ids), total=len(datas0)):
        data0 = datas0[id]
        data1 = datas1[id]
        data2 = datas2[idx]
        if idx != 0 and labels[idx-1] != labels[idx]:
            main_id = speaker_ids[idx]
            start_idx = idx
        if id == main_id: #奇数番目のラベル
            dataset0.append('__label__' + label + ' ' + ' '.join(data0))
            dataset1.append('__label__' + label + ' ' + ' '.join(data1))
            dataset2.append('__label__' + label + ' ' + ' '.join(data2))
            if idx != start_idx:
                dataset0.append('__label__' + label + ' ' + ' '.join(datas0[idx-1]) + ' ' + ' '.join(data0))
                dataset1.append('__label__' + label + ' '  + ' '.join(datas1[idx-1]) + ' ' + ' '.join(data1))
                dataset2.append('__label__' + label + ' ' + ' '.join(datas2[idx-1]) + ' ' + ' '.join(data2))
        else:  #偶数番目のラベル
            n = random.randint(1, 16)
            if n == 1:
                dataset0.append('__label__normal' +'  ' +' '.join(data0))
                dataset1.append('__label__normal' + ' ' + ' '.join(data1))
                dataset2.append('__label__normal' + ' ' + ' '.join(data2))
    return dataset0, dataset1, dataset2,

def create_en_dataset():
    with open(en_base_path, 'r', encoding='utf-8') as f:
        sentences = f.readlines()
    sentences = [sent.rstrip('\n') for sent in sentences]
    with open(label_path, 'r', encoding='utf-8') as f:
        labels = f.readlines()
    with open(speaker_ids_path, 'r', encoding='utf-8') as f:
        speaker_ids = f.readlines()
    assert len(labels) == len(sentences) == len(speaker_ids), f"len labels {len(labels)} : len sentences {len(sentences)} : len speaker ids {len(speaker_ids)}"
    labels = [label.rstrip('\n') for label in labels]
    sentences0 = [_split_words(sent) for sent in sentences]
    sentences1 = [en_clean(sent, lemma=False) for sent in sentences]
    sentences2 = [en_clean(sent, lemma=False, use_normalize=False) for sent in sentences]
    speaker_ids = [int(id.rstrip('\n')) for id in speaker_ids ]
    assert len(labels) == len(sentences) == len(speaker_ids), f"len labels {len(labels)} : len sentences {len(sentences)} : len speaker ids {len(speaker_ids)}"
    
    trains0, tests0 = _split_dataset(sentences0)
    trains1, tests1 = _split_dataset(sentences1)
    trains2, tests2 = _split_dataset(sentences2)
    train_labels, test_labels = _split_dataset(labels)
    train_ids, test_ids = _split_dataset(speaker_ids)
    
    dataset0, dataset1, dataset2 = get_datasets(trains0, trains1, trains2, train_labels, train_ids)

    with open('resources/en_train.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset0))
    with open('resources/en_stop_normalize_train.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset1))
    with open('resources/en_stop_train.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset2))

    dataset0, dataset1, dataset2 = get_datasets(tests0, tests1, tests2, test_labels, test_ids)

    with open('resources/en_test.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset0))
    with open('resources/en_stop_normalize_test.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset1))
    with open('resources/en_stop_test.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset2))


def create_ja_dataset():
    with open(ja_base_path, 'r', encoding='utf-8') as f:
        sentences = f.readlines()
    sentences = [sent.rstrip('\n') for sent in sentences]
    with open(label_path, 'r', encoding='utf-8') as f:
        labels = f.readlines()
    with open(speaker_ids_path, 'r', encoding='utf-8') as f:
        speaker_ids = f.readlines()
    assert len(labels) == len(sentences) == len(speaker_ids), f"len labels {len(labels)} : len sentences {len(sentences)} : len speaker ids {len(speaker_ids)}"
    labels = [label.rstrip('\n') for label in labels]
    sentences0 = [ja_wakachi(sent) for sent in sentences]
    sentences1 = [ja_clean(sent, normalize_=True) for sent in sentences]
    sentences2 = [ja_clean(sent, normalize_=False) for sent in sentences]
    speaker_ids = [int(id.rstrip('\n')) for id in speaker_ids ]
    assert len(labels) == len(sentences) == len(speaker_ids), f"len labels {len(labels)} : len sentences {len(sentences)} : len speaker ids {len(speaker_ids)}"

    trains0, tests0 = _split_dataset(sentences0)
    trains1, tests1 = _split_dataset(sentences1)
    trains2, tests2 = _split_dataset(sentences2)
    train_labels, test_labels = _split_dataset(labels)
    train_ids, test_ids = _split_dataset(speaker_ids)

    dataset0, dataset1, dataset2 = get_datasets(trains0, trains1, trains2, train_labels, train_ids)

    with open('resources/ja_train.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset0))
    with open('resources/ja_stop_normalize_train.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset1))
    with open('resources/ja_stop_train.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset2))

    dataset0, dataset1, dataset2 = get_datasets(tests0, tests1, tests2, test_labels, test_ids)

    with open('resources/ja_test.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset0))
    with open('resources/ja_stop_normalize_test.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset1))
    with open('resources/ja_stop_test.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(dataset2))


if __name__=='__main__':
    create_en_dataset()
    create_ja_dataset()
```

empathetic dialogue dataset では最初に発言した話者の感情をラベルとして設定しています. そのため, 発言のセットが始まった時点でのspeaker_idをmain_idとして保存しておき, speaker_idとmain_idが一致した場合発言をdatasetに追加しています. ただし, データ量を増やすために, ひとつ前の発言のラベルと現在の発言のラベルが一致した場合にもdatasetを追加しています. また, 特に感情なしという意味で normal　というラベルも追加しいます. empathetic dialogue datasetでは各ラベルの数が大体一定なので normal だけが数的に突出しないように 1 / 16 の確率で　dataset　に追加するようにしています. (16 = 32パターン /　2通りのデータの追加方法)

また, 英語・日本語のでータセットは以下のような形になります.

```en_train.txt
__label__sentimental I remember going to see the fireworks with my best friend It was the first time we ever spent time alone together Although there was a lot of people we felt like the only people in the world
__label__sentimental This was a best friend I miss her
__label__sentimental Was this a friend you were in love with or just a best friend This was a best friend I miss her
__label__sentimental We no longer talk
__label__sentimental Where has she gone We no longer talk
__label__afraid it feels like hitting to blank wall when i see the darkness
__label__afraid dont you feel so its a wonder
__label__afraid Oh ya I don t really see how dont you feel so its a wonder
__label__afraid i virtually thought so and i used to get sweatings
__label__afraid I do actually hit blank walls a lot of times but i get by i virtually thought so and i used to get sweatings
```
```en_stop_normalize_train.txt
__label__sentimental rememb go see firework best friend first time ever spent time alon togeth although lot peopl felt like peopl world
__label__sentimental best friend miss
__label__sentimental friend love best friend best friend miss
__label__sentimental longer talk
__label__sentimental gone longer talk
__label__afraid feel like hit blank wall see dark
__label__afraid dont feel wonder
__label__afraid oh ya dont realli see dont feel wonder
__label__afraid virtual thought use get sweat
__label__afraid actual hit blank wall lot time get virtual thought use get sweat
```
```en_stop_train.txt
__label__sentimental remember going see fireworks best friend first time ever spent time alone together although lot people felt like people world
__label__sentimental best friend miss
__label__sentimental friend love best friend best friend miss
__label__sentimental longer talk
__label__sentimental gone longer talk
__label__afraid feels like hitting blank wall see darkness
__label__afraid dont feel wonder
__label__afraid oh ya dont really see dont feel wonder
__label__afraid virtually thought used get sweatings
__label__afraid actually hit blank walls lot times get virtually thought used get sweatings
```
```ja_train.txt
__label__sentimental 親友 花火 覚え 二人 過ごす 初めて 私たち 世界 たった 一人 人間
__label__sentimental 親友 恋しい
__label__sentimental 友人 恋 親友 親友 恋しい
__label__sentimental 話す なくなり
__label__sentimental ? 話す なくなり
__label__afraid 闇 目の当たり 真っ白 壁 ぶつかる 気分
__label__afraid 不思議
__label__afraid ? やっ 不思議
__label__afraid 思っ 昔 よく 汗 かき
__label__afraid 実際 何度 真っ白 壁 ぶつかり 乗り切り 思っ 昔 よく 汗 かき
```
```ja_stop_normalize_train.txt
__label__sentimental 親友 花火 行く 覚える 居る 二人 過ごす 初めて 沢山 居る 私達 世界 唯 一人 人間 感じる
__label__sentimental 親友 恋しい
__label__sentimental 其の 友人 恋 為る 居る 其れ 只 親友 親友 恋しい
__label__sentimental 話す なくなる
__label__sentimental 行く 仕舞う ？ 話す なくなる
__label__afraid 闇 目の当たり 為る 真っ白 壁 ぶつかる 気分 成る
__label__afraid 感じる 不思議
__label__afraid ？ 遣る 感じる 不思議
__label__afraid 思う 居る 昔 汗
__label__afraid 実際 何度 真っ白 壁 ぶつかる 乗り切る 思う 居る 昔 汗
```
```ja_stop_train.txt
__label__sentimental 親友 花火 覚え 二人 過ごす 初めて 私たち 世界 たった 一人 人間
__label__sentimental 親友 恋しい
__label__sentimental 友人 恋 親友 親友 恋しい
__label__sentimental 話す なくなり
__label__sentimental ? 話す なくなり
__label__afraid 闇 目の当たり 真っ白 壁 ぶつかる 気分
__label__afraid 不思議
__label__afraid ? やっ 不思議
__label__afraid 思っ 昔 よく 汗 かき
__label__afraid 実際 何度 真っ白 壁 ぶつかり 乗り切り 思っ 昔 よく 汗 かき
```

まず, これらのデータセットを精度で比較します.

```fastText_classifier.py
import fasttext
import numpy as np
from collections import defaultdict
import pandas as pd
import pprint
from tqdm import tqdm

'''
    preprocessing the data ;
    changing the number of epochs (using the option -epoch, standard range [5 - 50]) ;
    changing the learning rate (using the option -lr, standard range [0.1 - 1.0]) ;
    using word n-grams (using the option -wordNgrams, standard range [1 - 5]).
'''


en_train0_path, en_test0_path = 'empatheticdialogues/resources/en_train.txt', 'empatheticdialogues/resources/en_test.txt'
en_train1_path, en_test1_path = 'empatheticdialogues/resources/en_stop_normalize_train.txt', 'empatheticdialogues/resources/en_stop_normalize_test.txt'
en_train2_path, en_test2_path = 'empatheticdialogues/resources/en_stop_train.txt', 'empatheticdialogues/resources/en_stop_test.txt'

ja_train0_path, ja_test0_path = 'empatheticdialogues/resources/ja_train.txt', 'empatheticdialogues/resources/ja_test.txt'
ja_train1_path, ja_test1_path = 'empatheticdialogues/resources/ja_stop_normalize_train.txt', 'empatheticdialogues/resources/ja_stop_normalize_test.txt'
ja_train2_path, ja_test2_path = 'empatheticdialogues/resources/ja_stop_train.txt', 'empatheticdialogues/resources/ja_stop_test.txt'

def _train1(train_path, test_path, default_args, n=10):
    scores = np.zeros(n + 2)
    print(f'train path {train_path} test path {test_path}\n')
    for i in range(n):
        model = fasttext.train_supervised(input=train_path, **default_args)
        print('\n')
        samples, precision, recall = model.test(test_path)
        scores[i] = precision
    scores[-2] = np.sum(scores)
    scores[-1] = scores[-2] / n
    return scores

def train1():
    default_args = {'minCount' : 5, 'dim' : 300, 'ws' : 15, 'neg' : 15, 'epoch' : 20}
    trains, tests = [en_train0_path, en_train1_path, en_train2_path, ja_train0_path, ja_train1_path, ja_train2_path], \
                    [en_test0_path, en_test1_path, en_test2_path, ja_test0_path, ja_test1_path, ja_test2_path]
    save_names = ['_'.join(train.split('/')[-1].split('_')[:-1]) + '.bin' for train in trains]
    assert save_names ==  ['_'.join(test.split('/')[-1].split('_')[:-1]) + '.bin' for test in tests]
    df_scores = defaultdict()
    for train, test, save_name in zip(trains, tests, save_names):
        scores = _train1(train, test, default_args)
        name = save_name.split('.')[0]
        df_scores[name] = scores

    df = pd.DataFrame(df_scores, index=[f'score {i}' for i in range(1, 11)] + ['sum', 'mean'])
    df.to_csv('models/FastText_scores.csv')
```

fastTextのハイパーパラメータとして,

- minCount : クラス分類に用いる単語における文章中の個数の基準値
- dim : 出力される単語ベクトルの次元
- ws : 文脈として考慮する前後の単語数
- neg : ネガティブサンプリング時の単語数
- epoch :　データセットの訓練回数

結果としてこうなります.

| |en|en_stop_normalize|en_stop|ja|ja_stop_normalize|ja_stop|
|:----|:----|:----|:----|:----|:----|:----|
|score1|0.26351723378482544|0.2409426274639357|0.2308115846272437|0.2578193832599119|0.23359030837004405|0.21376651982378855|
|score2|0.26373747384649265|0.23951106706309877|0.23158242484307895|0.25903083700440527|0.2341409691629956|0.21145374449339208|
|score3|0.26219579341482213|0.24149322761810374|0.23312410527474947|0.2605726872246696|0.2327092511013216|0.21200440528634362|
|score4|0.26351723378482544|0.2421539478031054|0.2318026649047462|0.2593612334801762|0.23469162995594714|0.21211453744493391|
|score5|0.2624160334764894|0.24138310758727013|0.23125206475057813|0.2593612334801762|0.23403083700440527|0.21156387665198237|
|score6|0.26439819403149434|0.24182358771060455|0.23147230481224534|0.2579295154185022|0.23149779735682818|0.208920704845815|
|score7|0.26329699372315823|0.24149322761810374|0.2322431450280806|0.2592511013215859|0.23359030837004405|0.2133259911894273|
|score8|0.2629666336306574|0.24050214734060127|0.23070146459641008|0.2599118942731278|0.23491189427312775|0.21288546255506607|
|score9|0.26318687369232463|0.24204382777227176|0.2323532650589142|0.2588105726872247|0.23425110132158591|0.21354625550660794|
|score10|0.263076753661491|0.2424843078956062|0.23279374518224866|0.2579295154185022|0.23116740088105728|0.21255506607929514|
|sum|2.6323092170465805|2.413831075872701|2.3181367690782957|2.589977973568282|2.3345814977973567|2.122136563876652|
|mean|0.26323092170465806|0.2413831075872701|0.23181367690782956|0.2589977973568282|0.23345814977973567|0.21221365638766518|


なんと, 正規化していない方が精度が高かったんです. 残念. 

ちなみに若干, 英語の方が精度が高いですが, これは英語の翻訳ミスが約10万7000件中4000件ほど. つまり, 3.7%のデータが質が悪かったためなので, 実質はどっこいどっこいだと思われます. fastTextすごいです.

###　パラメータ探索による最適化

次にパラメータ探索を行いました。

スクリプトはこれになります。使用したデータセットは最も精度のよかった en になります。

```fastText_classifier.py
def _train2(train_path, test_path, args_dict):
    num_list = list()
    best_model, best_precision, best_pram = None, 0.0, None
    log = list()
    for key, value in args_dict.items():
        print(key, len(value))
        temp_num_list = list()
        for num in range(len(value)):
            temp_num_list.append(num)
        if len(num_list) == 0:
            num_list = [[item] for item in temp_num_list]
        else:
            new_num_list = list()
            for temp_list in num_list:
                for n in temp_num_list:
                    new_temp_list = temp_list + [n]
                    new_num_list.append(new_temp_list)
            num_list = new_num_list
    print(f"length num list {len(num_list)}")
    for nums in tqdm(num_list, total=len(num_list)):
        args_temp_dict = defaultdict()
        assert len(nums) == len(args_dict)
        for i, key_value, num in zip(range(len(nums)), args_dict.items(), nums):
            key, value = key_value
            args_temp_dict[key] = value[num]
        print('args', args_temp_dict)
        ## train_phase
        model = fasttext.train_supervised(input=train_path, dim=300, **args_temp_dict)
        samples, precision, recall = model.test(test_path)
        print('precision : ', precision)
        if precision > best_precision:
            best_precision = precision
            best_model = model
            best_param = ' -----*------'.join([f"key : {str(key)} _ value : {str(value)}" for key, value in args_temp_dict.items()])
        print('best Precision : ' ,best_precision)
        print('best Pram : ', best_param)
        log.append(f'Precision : {precision} -----' +  '-----,----- '.join([f"key  :  {str(key)} _ value  :  {str(value)}" for key, value in args_temp_dict.items()]))
    return best_model, best_precision, log

def train2(train_path, test_path):
    args_list_dict = {'minCount' : [1, 5, 10  ,30, 50], 'ws': [15, 10, 15, 20], 'neg': [5, 10, 15, 20], 'epoch' : [5, 10,  25, 50], 'lr' : [1, 1e-1], 'wordNgrams' : [5, 10, 15, 20, 30]}
    train, test = train_path, test_path
    save_name = train.split('/')[-1].split('.')[0] + '.bin'
    model, precision, log =  _train2(train, test, args_list_dict)
    name = save_name.split('.')[0]
    model.save_model('models' + '/' + save_name)
    with open(f'models/FastText_log_{name}.txt', 'w', encoding='utf-8') as f:
        f.write('\n'.join(log))
```

ちなみに個々の部分は再起を使わずに、探索リストを作ることができ、結構便利に使えます。

```_.py
    for key, value in args_dict.items():
        print(key, len(value))
        temp_num_list = list()
        for num in range(len(value)):
            temp_num_list.append(num)
        if len(num_list) == 0:
            num_list = [[item] for item in temp_num_list]
        else:
            new_num_list = list()
            for temp_list in num_list:
                for n in temp_num_list:
                    new_temp_list = temp_list + [n]
                    new_num_list.append(new_temp_list)
            num_list = new_num_list
```

結果をまとめると

```
mean
          precision    ws   neg  epoch    lr  wordNgrams
minCount                                                
1.0        0.224930  15.0  12.5   22.5  0.55        16.0
5.0        0.224807  15.0  12.5   22.5  0.55        16.0
10.0       0.224392  15.0  12.5   22.5  0.55        16.0
30.0       0.222991  15.0  12.5   22.5  0.55        16.0
50.0       0.221474  15.0  12.5   22.5  0.55        16.0


mean
      precision  minCount   neg  epoch    lr  wordNgrams
ws                                                      
10.0   0.223706      19.2  12.5   22.5  0.55        16.0
15.0   0.223733      19.2  12.5   22.5  0.55        16.0
20.0   0.223703      19.2  12.5   22.5  0.55        16.0


mean
      precision  minCount    ws  epoch    lr  wordNgrams
neg                                                     
5.0    0.223762      19.2  15.0   22.5  0.55        16.0
10.0   0.223681      19.2  15.0   22.5  0.55        16.0
15.0   0.223654      19.2  15.0   22.5  0.55        16.0
20.0   0.223778      19.2  15.0   22.5  0.55        16.0


mean
       precision  minCount    ws   neg    lr  wordNgrams
epoch                                                   
5.0     0.170419      19.2  15.0  12.5  0.55        16.0
10.0    0.209479      19.2  15.0  12.5  0.55        16.0
25.0    0.253635      19.2  15.0  12.5  0.55        16.0
50.0    0.261341      19.2  15.0  12.5  0.55        16.0


mean
     precision  minCount    ws   neg  epoch  wordNgrams
lr                                                     
0.1   0.185118      19.2  15.0  12.5   22.5        16.0
1.0   0.262319      19.2  15.0  12.5   22.5        16.0


mean
            precision  minCount    ws   neg  epoch    lr
wordNgrams                                              
5.0          0.248441      19.2  15.0  12.5   22.5  0.55
10.0         0.226233      19.2  15.0  12.5   22.5  0.55
15.0         0.217456      19.2  15.0  12.5   22.5  0.55
20.0         0.214330      19.2  15.0  12.5   22.5  0.55
30.0         0.212133      19.2  15.0  12.5   22.5  0.55
```

minCountは小さいほど精度がよく、ws, negはあまり関係なく, epochは多いほど, lrは0.1, Ngramsは小さいほどいいという結果になりました。

### 再度、データセットづくり

ここで更なる疑問が生じます。

では、本当に英語のテクストをrawのままで処理したら?類義語で置換したら?

ということで、rawのままでの処理と類義語置換の処理を行いました。

まずはrawについて、英語は本当にそのままで入れました。
日本語は、[rinnaのtokenizer](https://github.com/rinnakk/japanese-pretrained-models)をちょっと変えて、tokenizerのパースのちに先頭のトークンの_を除いたものを記号の置換とかせずにそのまま入れました。

```dataset3.py
            if type == 'en':
                data0 = current_sentence
                data1 = en_synonym(current_sentence)
            else:
                data0 =ja_split(current_sentence)
                data1 = ja_synonym(current_sentence)
```

類義語の置換について、基本的にword2vecか別にトレーニングしたfastTextを用いて置換を行っています。英語については、[google](https://code.google.com/archive/p/word2vec/)のword2vecのpre-trainモデルを使用しました。日本語については [jpcc100](https://data.statmt.org/cc-100/)をrinnaのtokenizerで分かち書きした後で、fastTextのunsupervisedでトレーニングを行った。(パラメータは次元を300にした以外変更なし)

すべての単語について、0.8以上の類似度を持つ単語があれば、その中からランダムに一つ選び置換している。
詳しくはコード参照。

```replace_synonim.py
import re

import fasttext
import unicodedata
import gensim
import gensim.downloader as api
from stop_words import Stopword
import copy
from pymagnitude import Magnitude
from Synonyms_Replacement.tokenizer import rinna_tokenzier

from nltk.corpus import stopwords
en_stop = stopwords.words('english')

ja_stop = Stopword().words()

import random

print('load jananese')
jpcc100_fastText_path = "../jpcc100.bin"
ja_w2v = fasttext.load_model(jpcc100_fastText_path)

print('load english')
# en_w2v =  api.load('word2vec-google-news-300')
en_w2v = Magnitude('../GoogleNews-vectors-negative300.magnitude')

def ja_split(text):
    words = rinna_tokenzier(text)
    return ' '.join(words)

#.get_nearest_neighbors('hello')

def find_most_similar(model, wrd, threshold=0.8, topn=10):
    try:
        res = [item[0] for item in model.most_similar(wrd, topn=topn) if item[1] > threshold]
    except:
        res = []
    return res

def find_nearest_neighbors(model, word, threshold=0.8):
    try:
        res = [item[1] for item in model.get_nearest_neighbors(word)  if item[0] > threshold]
    except:
        res = []
    return res

def en_synonym(sentence):
    sentence = sentence.lower()
    words = re.sub(r'[^\w]', ' ', sentence).split()
    res_list = [['BOS']]
    for word in words:
        if word not in en_stop:
            candidates = find_most_similar(en_w2v, word)
            if len(candidates) > 0:
                # print(f"word {word} candidates {' : '.join(candidates)}")
                new_res_list = list()
                for res_item in res_list:
                    item0, item1 = copy.copy(res_item), copy.copy(res_item)
                    item0.append(random.choice(candidates))
                    item1.append(word)
                    new_res_list.append(item0)
                    new_res_list.append(item1)
                res_list = new_res_list
            else:
                for res_item in res_list:
                    res_item.append(word)
        else:
            for res_item in res_list:
                res_item.append(word)
    if len(res_list) > 1:
        print(f'length of res_list {len(res_list)}')
    assert len(res_list) > 0
    return [' '.join(res_item[1: ]) for res_item in res_list]

def ja_synonym(sentence):
    words = rinna_tokenzier(sentence)
    res_list = [['BOS']]
    for word in words:
        if word not in ja_stop:
            candidates = find_nearest_neighbors(ja_w2v, word)
            if len(candidates) > 0:
                # print(f"word {word} candidates {' : '.join(candidates)}")
                new_res_list = list()
                for res_item in res_list:
                    item0, item1 = copy.copy(res_item), copy.copy(res_item)
                    item0.append(random.choice(candidates))
                    item1.append(word)
                    new_res_list.append(item0)
                    new_res_list.append(item1)
                res_list = new_res_list
            else:
                for res_item in res_list:
                    res_item.append(word)
        else:
            for res_item in res_list:
                res_item.append(word)
    if len(res_list) > 1:
        print(f'length of res_list {len(res_list)}')
    assert len(res_list) > 0
    return [' '.join(res_item[1: ]) for res_item in res_list]
```

結果は

| |en_raw|en_synonym|ja_raw|ja_synonym|
|:----|:----|:----|:----|:----|
|score 1|0.24623336632574508|0.22564917397644907|0.24210990951224895|0.19913397860417728|
|score 2|0.24898273397118661|0.22229719434952006|0.24332376958728757|0.1966887417218543|
|score 3|0.24656329044319805|0.22660688244128593|0.24354447141911278|0.20231788079470198|
|score 4|0.24942263279445728|0.23222253662146572|0.24155815493268595|0.20061130922058074|
|score 5|0.2501924557351809|0.23511742811744987|0.244316927830501|0.2010188487009679|
|score 6|0.24799296161882767|0.2233854994231983|0.24266166409181197|0.20257259296994395|
|score 7|0.24777301220719233|0.2304812485035805|0.24299271683954976|0.19847172694854814|
|score 8|0.24920268338282195|0.22042530962279347|0.2436548223350254|0.20183392766174224|
|score 9|0.24865280985373364|0.22495265872929499|0.24233061134407416|0.20084055017829852|
|score 10|0.24865280985373364|0.2321790044185186|0.24409622599867578|0.20277636271013755|
|sum|2.4836687561860775|2.273316936203557|2.430589273890973|2.0062659195109527|
|mean|0.24836687561860776|0.22733169362035568|0.2430589273890973|0.20062659195109528|

まず、英語に関しては最低限記号は取り除こうということがわかりました。日本語に関しては、sentence piece　よりも unidic + mecab の方が fastText とは相性がいいみたいです。

置換に関しては、データセットが増えてもゴミが増えただけという悲しい結果になりました。
##　残念!

#### データセットの作り方
これまでは

発言
前の発言 + 発言　

でデータセットを作っていましたが、本当に最適化調べました。
試したのは

パターン1
発言

パターン2
発言
前の前の発言 + 発言

パターン3
発言
前の発言 + 発言
前の前の発言 + 前の発言 + 発言

これまでのサイズが8.81 MB, パターン1が 4.82 MB, パターン2が11.12 MB, パターン3が 15.18 MB になります。

結果は

| |en|en_turn1|en_turn1_turn321|en_turn1_turn321_turn21|ja|ja_turn1|ja_turn1_turn321|ja_turn1_turn321_turn21|
|:----|:----|:----|:----|:----|:----|:----|:----|:----|
|score 1|0.2639577139081599|0.2957983193277311|0.3003747795414462|0.27488928981466293|0.2593612334801762|0.2827924273157539|0.23511838902412038|0.2131578947368421|
|score 2|0.2623059134456558|0.2956302521008403|0.29993386243386244|0.27488928981466293|0.2605726872246696|0.2836375929682218|0.23766319982296968|0.2131578947368421|
|score 3|0.2609844730756525|0.2963025210084034|0.2993827160493827|0.27333114646547485|0.26002202643171807|0.2838066260987154|0.23600354060632883|0.20986842105263157|
|score 4|0.263076753661491|0.29680672268907565|0.29971340388007056|0.27357716909955715|0.26222466960352425|0.2838066260987154|0.23611418455410488|0.20953947368421053|
|score 5|0.263076753661491|0.2961344537815126|0.30081569664902996|0.27349516155486303|0.26024229074889865|0.28498985801217036|0.23390130559858377|0.21282894736842106|
|score 6|0.2628565135998238|0.2984873949579832|0.3019179894179894|0.2745612596358865|0.2567180616740088|0.2834685598377282|0.23722062403186545|0.21422697368421054|
|score 7|0.26384759387732626|0.29764705882352943|0.29817019400352734|0.27324913892078073|0.26068281938325993|0.28431372549019607|0.23290551006859925|0.21636513157894738|
|score 8|0.26329699372315823|0.30134453781512605|0.2996031746031746|0.27398720682302774|0.25903083700440527|0.2841446923597025|0.23910157114405842|0.21200657894736843|
|score 9|0.2616451932606541|0.2969747899159664|0.29894179894179895|0.27431523700180416|0.25903083700440527|0.28262339418526033|0.23644611639743307|0.2136513157894737|
|score 10|0.26417795396982713|0.29764705882352943|0.2998236331569665|0.2741512219124159|0.2598017621145374|0.28076402974983095|0.23699933613631335|0.21077302631578948|
|sum|2.6292258561832402|2.9727731092436978|2.998677248677249|2.740446121043136|2.5976872246696034|2.834347532116295|2.361473777384377|2.125575657894737|
|mean|0.26292258561832405|0.29727731092436976|0.2998677248677249|0.2740446121043136|0.25976872246696037|0.28343475321162953|0.2361473777384377|0.21255756578947368|

英語は、前の前の発言 + 発言 or 発言 が最も精度がよく、日本語は、発言のみが最も精度がいいという結果になりました。つまり、感情を判定したい人の発言以外の情報は入れるな！っていうことですね。

### 再びパラメータ探索

```
args_list_dict = {'minCount':  [1, 3, 5], 'epoch' : [50, 75, 100, 150], 'lr' : [1], 'wordNgrams' : [1, 3, 5]}
```
前回の結果を踏まえて、この中でパラメータ探索を行いました。

結果は

```
mean
          precision  epoch   lr  wordNgrams
minCount                                   
1.0        0.272122  93.75  1.0         3.0
3.0        0.272887  93.75  1.0         3.0
5.0        0.273502  93.75  1.0         3.0


mean
       precision  minCount   lr  wordNgrams
epoch                                      
50.0    0.274106       3.0  1.0         3.0
75.0    0.273513       3.0  1.0         3.0
100.0   0.272384       3.0  1.0         3.0
150.0   0.271345       3.0  1.0         3.0


mean
            precision  minCount  epoch   lr
wordNgrams                                 
1.0          0.222582       3.0  93.75  1.0
3.0          0.295111       3.0  93.75  1.0
5.0          0.300817       3.0  93.75  1.0
```

minCountは5が最もよく、epochは50から過学習気味、wordNgramsも5が最もよい。やはり、デフォルトの値は強かった。