Rasa NLU で自然言語理解のおためし

NLU (Natural Language Understanding=自然言語理解) をやってみたくて、

https://tech.mof-mof.co.jp/blog/rasa-nlu-japanese.html

を参考にしながら、dockerで環境を作って、実際に動くところまでやってみる。



## Docker環境構築

ホストマシン上でrasa用のディレクトリを作る。

```sh
$ mkdir rasa
$ cd rasa
$ vi Dockerfile
```

Dockerfile に、まずは python が動くimageのみを記述してみる。

```Dockerfile
FROM python:3.7-slim-stretch
```

docker-composeで動かしたいので docker-compose.yml を作成

```sh
$ vi docker-compose.yml
```

```docker-compose.yml
version: '2'
services:
  rasa:
    container_name: rasa
    build:
      context: .
    volumes:
      - .:/app
```

buildする

```sh
$ docker-compose build 
Building rasa
Step 1/1 : FROM python:3.7-slim-stretch
 ---> c9ec5ac0f580
Successfully built c9ec5ac0f580
Successfully tagged rasa-test_rasa:latest

$
```

python動くか確認。

```sh
$ docker-compose run --rm rasa bash
Creating network "rasa-test_default" with the default driver
root@c02085da1f5f:/# python --version
Python 3.7.7
root@c02085da1f5f:/#
```

動いた。

## Rasaをインストール

https://tech.mof-mof.co.jp/blog/rasa-nlu-tutorial.html

を参考にrasa環境を作っていく

```sh
root@c02085da1f5f:/# pip install rasa
...(なんかいろいろインストールされる)

Successfully built sanic-jwt absl-py mattermostwrapper colorclass webexteamssdk SQLAlchemy terminaltables future gast termcolor wrapt PyYAML docopt pyrsistent
Failed to build ujson
Installing collected packages: ujson, certifi, six, pycparser, cffi, cryptography, future, python-telegram-bot, tqdm, tabulate, python-crfsuite, sklearn-crfsuite, dnspython, pymongo, ruamel.yaml, numpy, h5py, keras-applications, opt-einsum, grpcio, protobuf, absl-py, google-pasta, gast, termcolor, scipy, wrapt, keras-preprocessing, werkzeug, cachetools, pyasn1, pyasn1-modules, rsa, google-auth, chardet, urllib3, idna, requests, markdown, oauthlib, requests-oauthlib, google-auth-oauthlib, tensorboard, tensorflow-estimator, astor, tensorflow, python-dateutil, PyYAML, docopt, pykwalify, zipp, importlib-metadata, pyrsistent, attrs, jsonschema, greenlet, gevent, redis, PyJWT, sanic-jwt, cloudpickle, kafka-python, rocketchat-API, joblib, scikit-learn, pysocks, pytz, twilio, cycler, pyparsing, kiwisolver, matplotlib, mattermostwrapper, multidict, psycopg2-binary, colorclass, aiofiles, rfc3986, hstspreload, hpack, hyperframe, h2, h11, sniffio, httpx, httptools, uvloop, websockets, sanic, requests-toolbelt, webexteamssdk, httplib2, oauth2client, python-engineio, tzlocal, apscheduler, async-generator, wcwidth, prompt-toolkit, questionary, fbmessenger, sanic-plugins-framework, sanic-cors, humanfriendly, coloredlogs, colorhash, jsonpickle, SQLAlchemy, async-timeout, yarl, aiohttp, pydot, packaging, rasa-sdk, python-socketio, tensorflow-hub, decorator, tensorflow-probability, typeguard, tensorflow-addons, terminaltables, networkx, jmespath, docutils, botocore, s3transfer, boto3, slackclient, pika, rasa
    Running setup.py install for ujson ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ingvl92n/ujson/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ingvl92n/ujson/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-46isinhm/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/ujson
         cwd: /tmp/pip-install-ingvl92n/ujson/
    Complete output (12 lines):
    Warning: 'classifiers' should be a list, got type 'filter'
    running install
    running build
    running build_ext
    building 'ujson' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/python
    creating build/temp.linux-x86_64-3.7/lib
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I./python -I./lib -I/usr/local/include/python3.7m -c ./python/ujson.c -o build/temp.linux-x86_64-3.7/./python/ujson.o -D_GNU_SOURCE
    unable to execute 'gcc': No such file or directory
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ingvl92n/ujson/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ingvl92n/ujson/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-46isinhm/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/ujson Check the logs for full command output.
root@c02085da1f5f:/# 
```

なんかエラー出た。

`unable to execute 'gcc': No such file or directory` ってことなので、それっぽいのをインストールする

```sh
root@c02085da1f5f:/# apt-get update -q -y && apt-get install -q -y build-essential
...(なんかいろいろインストールされる)

update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up build-essential (12.3) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
root@c02085da1f5f:/#
```

もっかいrasaのインストールにチャレンジ

```sh
root@c02085da1f5f:/# pip install rasa
...(なんかいろいろインストールされる)

Successfully built ujson
Installing collected packages: six, absl-py, cycler, python-dateutil, kiwisolver, numpy, pyparsing, matplotlib, docopt, PyYAML, pykwalify, protobuf, tensorflow-hub, kafka-python, pytz, multidict, pysocks, urllib3, certifi, idna, chardet, requests, PyJWT, twilio, fbmessenger, ruamel.yaml, yarl, attrs, async-timeout, aiohttp, slackclient, terminaltables, aiofiles, httptools, websockets, ujson, h11, hpack, hyperframe, h2, sniffio, rfc3986, hstspreload, httpx, uvloop, sanic, sanic-plugins-framework, sanic-cors, humanfriendly, coloredlogs, rasa-sdk, tzlocal, apscheduler, packaging, typeguard, tensorflow-addons, greenlet, gevent, pika, scipy, google-pasta, astor, termcolor, wrapt, grpcio, pyasn1, pyasn1-modules, rsa, cachetools, google-auth, werkzeug, markdown, oauthlib, requests-oauthlib, google-auth-oauthlib, tensorboard, h5py, keras-applications, tensorflow-estimator, keras-preprocessing, opt-einsum, gast, tensorflow, cloudpickle, decorator, tensorflow-probability, dnspython, pymongo, python-engineio, joblib, scikit-learn, httplib2, oauth2client, SQLAlchemy, requests-toolbelt, future, webexteamssdk, sanic-jwt, wcwidth, prompt-toolkit, colorhash, colorclass, tqdm, rocketchat-API, jmespath, docutils, botocore, s3transfer, boto3, questionary, zipp, importlib-metadata, pyrsistent, jsonschema, mattermostwrapper, jsonpickle, python-socketio, redis, pydot, networkx, pycparser, cffi, cryptography, python-telegram-bot, psycopg2-binary, async-generator, python-crfsuite, tabulate, sklearn-crfsuite, rasa
Successfully installed PyJWT-1.7.1 PyYAML-5.3.1 SQLAlchemy-1.3.15 absl-py-0.9.0 aiofiles-0.4.0 aiohttp-3.6.2 apscheduler-3.6.3 astor-0.8.1 async-generator-1.10 async-timeout-3.0.1 attrs-19.3.0 boto3-1.12.30 botocore-1.15.30 cachetools-4.0.0 certifi-2019.11.28 cffi-1.14.0 chardet-3.0.4 cloudpickle-1.2.2 colorclass-2.2.0 coloredlogs-10.0 colorhash-1.0.2 cryptography-2.8 cycler-0.10.0 decorator-4.4.2 dnspython-1.16.0 docopt-0.6.2 docutils-0.15.2 fbmessenger-6.0.0 future-0.18.2 gast-0.2.2 gevent-1.4.0 google-auth-1.12.0 google-auth-oauthlib-0.4.1 google-pasta-0.2.0 greenlet-0.4.15 grpcio-1.27.2 h11-0.8.1 h2-3.2.0 h5py-2.10.0 hpack-3.0.0 hstspreload-2020.3.25 httplib2-0.17.0 httptools-0.1.1 httpx-0.9.3 humanfriendly-8.1 hyperframe-5.2.0 idna-2.9 importlib-metadata-1.5.2 jmespath-0.9.5 joblib-0.14.1 jsonpickle-1.3 jsonschema-3.2.0 kafka-python-1.4.7 keras-applications-1.0.8 keras-preprocessing-1.1.0 kiwisolver-1.1.0 markdown-3.2.1 matplotlib-3.1.3 mattermostwrapper-2.2 multidict-4.7.5 networkx-2.4 numpy-1.18.2 oauth2client-4.1.3 oauthlib-3.1.0 opt-einsum-3.2.0 packaging-19.0 pika-1.1.0 prompt-toolkit-2.0.10 protobuf-3.11.3 psycopg2-binary-2.8.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pykwalify-1.7.0 pymongo-3.8.0 pyparsing-2.4.6 pyrsistent-0.16.0 pysocks-1.7.1 python-crfsuite-0.9.7 python-dateutil-2.8.1 python-engineio-3.11.2 python-socketio-4.4.0 python-telegram-bot-11.1.0 pytz-2019.3 questionary-1.5.1 rasa-1.9.2 rasa-sdk-1.9.0 redis-3.4.1 requests-2.23.0 requests-oauthlib-1.3.0 requests-toolbelt-0.9.1 rfc3986-1.3.2 rocketchat-API-0.6.36 rsa-4.0 ruamel.yaml-0.15.100 s3transfer-0.3.3 sanic-19.12.2 sanic-cors-0.10.0.post3 sanic-jwt-1.3.2 sanic-plugins-framework-0.9.2 scikit-learn-0.22.2.post1 scipy-1.4.1 six-1.14.0 sklearn-crfsuite-0.3.6 slackclient-2.5.0 sniffio-1.1.0 tabulate-0.8.7 tensorboard-2.1.1 tensorflow-2.1.0 tensorflow-addons-0.8.3 tensorflow-estimator-2.1.0 tensorflow-hub-0.7.0 tensorflow-probability-0.7.0 termcolor-1.1.0 terminaltables-3.1.0 tqdm-4.31.1 twilio-6.26.3 typeguard-2.7.1 tzlocal-2.0.0 ujson-1.35 urllib3-1.25.8 uvloop-0.14.0 wcwidth-0.1.9 webexteamssdk-1.1.1 websockets-8.1 werkzeug-1.0.0 wrapt-1.12.1 yarl-1.4.2 zipp-3.1.0
root@c02085da1f5f:/#
```

インストールできた。

次はrasaのinit。

の前に、作業ディレクトリを移動する。/ 直下でなんかファイルができると気持ち悪いので。

```sh
root@c02085da1f5f:/# pwd
/
root@c02085da1f5f:/# ls -al
total 88
drwxr-xr-x   1 root root 4096 Mar 27 08:25 .
drwxr-xr-x   1 root root 4096 Mar 27 08:25 ..
-rwxr-xr-x   1 root root    0 Mar 27 08:25 .dockerenv
drwxr-xr-x   4 root root  128 Mar 27 08:22 app
drwxr-xr-x   1 root root 4096 Mar 27 08:33 bin
drwxr-xr-x   2 root root 4096 Feb  1 17:09 boot
drwxr-xr-x   5 root root  360 Mar 27 08:25 dev
drwxr-xr-x   1 root root 4096 Mar 27 08:33 etc
drwxr-xr-x   2 root root 4096 Feb  1 17:09 home
drwxr-xr-x   1 root root 4096 Mar 27 08:33 lib
drwxr-xr-x   2 root root 4096 Feb 24 00:00 lib64
drwxr-xr-x   2 root root 4096 Feb 24 00:00 media
drwxr-xr-x   2 root root 4096 Feb 24 00:00 mnt
drwxr-xr-x   2 root root 4096 Feb 24 00:00 opt
dr-xr-xr-x 228 root root    0 Mar 27 08:25 proc
drwx------   1 root root 4096 Mar 27 08:27 root
drwxr-xr-x   3 root root 4096 Feb 24 00:00 run
drwxr-xr-x   2 root root 4096 Feb 24 00:00 sbin
drwxr-xr-x   2 root root 4096 Feb 24 00:00 srv
dr-xr-xr-x  13 root root    0 Mar 27 08:25 sys
drwxrwxrwt   1 root root 4096 Mar 27 08:36 tmp
drwxr-xr-x   1 root root 4096 Feb 24 00:00 usr
drwxr-xr-x   1 root root 4096 Feb 24 00:00 var
root@c02085da1f5f:/# cd app/
root@c02085da1f5f:/app# ls -al
total 12
drwxr-xr-x 4 root root  128 Mar 27 08:22 .
drwxr-xr-x 1 root root 4096 Mar 27 08:25 ..
-rw-r--r-- 1 root root   29 Mar 27 08:20 Dockerfile
-rw-r--r-- 1 root root  112 Mar 27 08:22 docker-compose.yml
root@c02085da1f5f:/app#
```

/app で作業する。

rasa init する

```sh
root@c02085da1f5f:/app# rasa init --no-prompt
Welcome to Rasa! 🤖

To get started quickly, an initial project will be created.
If you need some help, check out the documentation at https://rasa.com/docs/rasa.

Created project directory at '/app'.
Finished creating project structure.
Training an initial model...
Training Core model...

(なんやかんや)

2020-03-27 08:46:25 INFO     rasa.nlu.model  - Successfully saved model into '/tmp/tmp4p2k58s6/nlu'
NLU model training completed.
Your Rasa model is trained and saved at '/app/models/20200327-084527.tar.gz'.
If you want to speak to the assistant, run 'rasa shell' at any time inside the project directory.
root@c02085da1f5f:/app#
```

init できたっぽい。

ディレクトリ内を見てみる。

```sh
root@c02085da1f5f:/app# ls -al
total 32
drwxr-xr-x 14 root root  448 Mar 27 08:46 .
drwxr-xr-x  1 root root 4096 Mar 27 08:25 ..
-rw-r--r--  1 root root   29 Mar 27 08:20 Dockerfile
-rw-r--r--  1 root root    0 Mar 27 08:36 __init__.py
drwxr-xr-x  4 root root  128 Mar 27 08:45 __pycache__
-rw-r--r--  1 root root  757 Mar 27 08:36 actions.py
-rw-r--r--  1 root root  622 Mar 27 08:36 config.yml
-rw-r--r--  1 root root  938 Mar 27 08:36 credentials.yml
drwxr-xr-x  4 root root  128 Mar 27 08:45 data
-rw-r--r--  1 root root  112 Mar 27 08:22 docker-compose.yml
-rw-r--r--  1 root root  549 Mar 27 08:36 domain.yml
-rw-r--r--  1 root root 1456 Mar 27 08:36 endpoints.yml
drwxr-xr-x  3 root root   96 Mar 27 08:46 models
drwxr-xr-x  3 root root   96 Mar 27 08:45 tests
root@c02085da1f5f:/app#
```

なんか色々増えてる。

参考にした記事と同様に、まずは会話してみる。

```sh
root@c02085da1f5f:/app# rasa shell
2020-03-27 08:54:07 INFO     root  - Connecting to channel 'cmdline' which was specified by the '--connector' argument. Any other channels will be ignored. To connect to all given channels, omit the '--connector' argument.
2020-03-27 08:54:07 INFO     root  - Starting Rasa server on http://localhost:5005
2020-03-27 08:54:08.593814: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
Bot loaded. Type a message and press enter (use '/stop' to exit):
Your input ->  hello
Hey! How are you?
Your input ->  /stop
2020-03-27 08:54:31 INFO     root  - Killing Sanic server now.
root@c02085da1f5f:/app#
```

おお。できてる。

## 日本語に対応する

https://tech.mof-mof.co.jp/blog/rasa-nlu-japanese.html

次にこの記事に戻って日本語対応してみる。

まずは `data/nlu.md` の一番下にデータを追加してみる。

```sh
$ vi data/nlu.md 
```

```data/nlu.md
## intent:restaurant_ja
- 渋谷で美味しいイタリアンない？
- 和食食べたいんだけど、六本木におすすめある？
- 今度麻布行くんだけど、フレンチのお店教えて
```

上記記事だとmecabをやめてspaCyを使っているようなので、mecabを飛ばしてspaCyでやってみる。

config.yml にデフォルト設定があるようなので、一旦全部消して以下を書いてみる。

```config.yml
pipeline:
  # - name: “tokenizer_whitespace”
  - name: “nlp_spacy”
  - name: “tokenizer_spacy”
  - name: “CRFEntityExtractor”
  - name: “ner_crf”
  - name: “ner_synonyms”
  - name: “intent_featurizer_count_vectors”
  - name: “intent_classifier_tensorflow_embedding”
```

学習させてみる。

```sh
root@c02085da1f5f:/app# rasa train nlu
The config file 'config.yml' is missing mandatory parameters: 'language'. Add missing parameters to config file and try again.
root@c02085da1f5f:/app#
```

config.yml に languageが無いよって。

上記記事だとja_ginzaにしているようなので、端折って config.yml の一番上に ja_ginza を追加してみる。

```config.yml
language: ja_ginza
```

もういっかいコマンドを叩いてみる。

```sh
root@c02085da1f5f:/app# rasa train nlu
Training NLU model...
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/rasa/nlu/registry.py", line 173, in get_component_class
    return class_from_module_path(component_name)
  File "/usr/local/lib/python3.7/site-packages/rasa/utils/common.py", line 211, in class_from_module_path
    raise ImportError(f"Cannot retrieve class from path {module_path}.")
ImportError: Cannot retrieve class from path “nlp_spacy”.

(なんやかんや)
```

nlp_spacyが云々で怒られている。

spaCy入れてないもんね。

spaCy入れてみる。

```sh
root@c02085da1f5f:/app# pip install spacy

(なんやかんや)

Successfully installed blis-0.4.1 catalogue-1.0.0 cymem-2.0.3 murmurhash-1.0.2 plac-1.1.3 preshed-3.0.2 spacy-2.2.4 srsly-1.0.2 thinc-7.4.0 tqdm-4.43.0 wasabi-0.6.0
root@c02085da1f5f:/app#
```

入った。

GiNZAの日本語処理ライブラリ？があるようなので、インストールする。

```sh
root@c02085da1f5f:/app# pip install "https://github.com/megagonlabs/ginza/releases/download/latest/ginza-latest.tar.gz"
Collecting https://github.com/megagonlabs/ginza/releases/download/latest/ginza-latest.tar.gz

(なんやかんや)

Successfully built ginza ja-ginza SudachiDict-core
Installing collected packages: ja-ginza, sortedcontainers, Cython, dartsclone, SudachiPy, SudachiDict-core, ginza
Successfully installed Cython-0.29.16 SudachiDict-core-20190927 SudachiPy-0.4.3 dartsclone-0.9.0 ginza-2.2.1 ja-ginza-2.2.0 sortedcontainers-2.1.0
root@c02085da1f5f:/app#
```

インストールできた。

これで準備できたか？

もう一回コマンドを叩く。

```sh
root@c02085da1f5f:/app# rasa train nlu
Training NLU model...
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/rasa/nlu/registry.py", line 173, in get_component_class
    return class_from_module_path(component_name)
  File "/usr/local/lib/python3.7/site-packages/rasa/utils/common.py", line 211, in class_from_module_path
    raise ImportError(f"Cannot retrieve class from path {module_path}.")
ImportError: Cannot retrieve class from path “nlp_spacy”.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/rasa/__main__.py", line 91, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/usr/local/lib/python3.7/site-packages/rasa/cli/train.py", line 140, in train_nlu
    persist_nlu_training_data=args.persist_nlu_data,
  File "/usr/local/lib/python3.7/site-packages/rasa/train.py", line 414, in train_nlu
    persist_nlu_training_data,
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.7/site-packages/rasa/train.py", line 445, in _train_nlu_async
    persist_nlu_training_data=persist_nlu_training_data,
  File "/usr/local/lib/python3.7/site-packages/rasa/train.py", line 474, in _train_nlu_with_validated_data
    persist_nlu_training_data=persist_nlu_training_data,
  File "/usr/local/lib/python3.7/site-packages/rasa/nlu/train.py", line 74, in train
    trainer = Trainer(nlu_config, component_builder)
  File "/usr/local/lib/python3.7/site-packages/rasa/nlu/model.py", line 142, in __init__
    components.validate_requirements(cfg.component_names)
  File "/usr/local/lib/python3.7/site-packages/rasa/nlu/components.py", line 51, in validate_requirements
    component_class = registry.get_component_class(component_name)
  File "/usr/local/lib/python3.7/site-packages/rasa/nlu/registry.py", line 199, in get_component_class
    raise ModuleNotFoundError(exception_message)
ModuleNotFoundError: Cannot find class '“nlp_spacy”' from global namespace. Please check that there is no typo in the class name and that you have imported the class into the global namespace.
root@c02085da1f5f:/app#
```

なんか怒られた。

`Cannot find class '“nlp_spacy”'` ってなんだろ。

`ModuleNotFoundError: Cannot find class '“nlp_spacy”' from global namespace. Please check that there is no typo in the class name and that you have imported the class into the global namespace.` でググってみる。

https://github.com/RasaHQ/rasa/blob/master/rasa/nlu/registry.py

このページが一番上に出てきた。

https://github.com/RasaHQ/rasa/blob/master/rasa/nlu/registry.py#L101

`old_style_names` ？

```python
"nlp_spacy": "SpacyNLP",
```

こんな感じの対応付けが定義されている。

さっきconfig.ymlに書いたのが、`nlp_spacy` だから `SpacyNLP` に変更してみる。

```config.yml
language: ja_ginza

pipeline:
  # - name: “tokenizer_whitespace”
  - name: "SpacyNLP" # ←ここを変更
  - name: "tokenizer_spacy"
  - name: "CRFEntityExtractor"
  - name: "ner_crf"
  - name: "ner_synonyms"
  - name: "intent_featurizer_count_vectors"
  - name: "intent_classifier_tensorflow_embedding"
```

再度コマンドを叩く。

```sh
root@c02085da1f5f:/app# rasa train nlu
Training NLU model...

(なんやかんや)

ModuleNotFoundError: Cannot find class '“SpacyNLP”' from global namespace. Please check that there is no typo in the class name and that you have imported the class into the global namespace.
```

`“SpacyNLP”` ？？

あ、コピペしたらダブルクオートがおかしいのか。

ダブルクオートを治す。

```config.yml
language: ja_ginza

pipeline:
  # - name: “tokenizer_whitespace”
  - name: "nlp_spacy"
  - name: "tokenizer_spacy"
  - name: "CRFEntityExtractor"
  - name: "ner_crf"
  - name: "ner_synonyms"
  - name: "intent_featurizer_count_vectors"
  - name: "intent_classifier_tensorflow_embedding"
```

もういっかいコマンド叩く。

```sh
root@c02085da1f5f:/app# rasa train nlu
Training NLU model...

(なんやかんや)

oblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
2020-03-27 09:15:20 INFO     rasa.nlu.model  - Successfully saved model into '/tmp/tmpn4xtb3h7/nlu'
NLU model training completed.
Your Rasa model is trained and saved at '/app/models/nlu-20200327-091520.tar.gz'.
root@c02085da1f5f:/app#
```

お、やっと動いた。

`rasa shell nlu` で intent が取れれば成功？

```sh
root@c02085da1f5f:/app# rasa shell nlu
2020-03-27 09:16:26 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-ja_ginza'.
/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
2020-03-27 09:16:26.359929: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
/usr/local/lib/python3.7/site-packages/rasa/nlu/classifiers/diet_classifier.py:864: FutureWarning: 'EmbeddingIntentClassifier' is deprecated and will be removed in version 2.0. Use 'DIETClassifier' instead.
  model=model,
NLU model loaded. Type a message and press enter to parse it.
Next message:
六本木のイタリアン教えて
{
  "intent": {
    "name": "restaurant_ja",
    "confidence": 0.48260971903800964
  },
  "entities": [],
  "intent_ranking": [
    {
      "name": "restaurant_ja",
      "confidence": 0.48260971903800964
    },
    {
      "name": "affirm",
      "confidence": 0.1126396581530571
    },
    {
      "name": "greet",
      "confidence": 0.09804855287075043
    },
    {
      "name": "goodbye",
      "confidence": 0.08541827648878098
    },
    {
      "name": "mood_great",
      "confidence": 0.07586738467216492
    },
    {
      "name": "deny",
      "confidence": 0.06578702479600906
    },
    {
      "name": "bot_challenge",
      "confidence": 0.05436018109321594
    },
    {
      "name": "mood_unhappy",
      "confidence": 0.025269268080592155
    }
  ],
  "text": "六本木のイタリアン教えて"
}
Next message:

```

お、それっぽい。intentのスコア低いけど、こんなもん？

固有表現もやってみよう。

`data/nlu.md` の日本語の箇所を下記のように書き換えて、

```data/nlu.md
## intent:restaurant_ja
- [渋谷](location)で美味しい[イタリアン](restaurant_type)ない？
- [和食](restaurant_type)食べたいんだけど、[六本木](location)におすすめある？
- 今度[麻布](location)行くんだけど、[フレンチ](restaurant_type)のお店教えて
```

学習させて、

```sh
root@c02085da1f5f:/app# rasa train nlu
Training NLU model...

(なんやかんや)

2020-03-27 13:46:37 INFO     rasa.nlu.model  - Finished training component.
/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
2020-03-27 13:46:37 INFO     rasa.nlu.model  - Successfully saved model into '/tmp/tmpy7tqyrsb/nlu'
NLU model training completed.
Your Rasa model is trained and saved at '/app/models/nlu-20200327-134637.tar.gz'.
root@c02085da1f5f:/app#
```

shell を叩いてみる。

```sh
root@c02085da1f5f:/app# rasa shell nlu
2020-03-27 13:47:07 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-ja_ginza'.
/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
2020-03-27 13:47:07.924760: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
/usr/local/lib/python3.7/site-packages/rasa/nlu/classifiers/diet_classifier.py:864: FutureWarning: 'EmbeddingIntentClassifier' is deprecated and will be removed in version 2.0. Use 'DIETClassifier' instead.
  model=model,
NLU model loaded. Type a message and press enter to parse it.
Next message:
渋谷で美味しいイタリアンない？
{
  "intent": {
    "name": "restaurant_ja",
    "confidence": 0.9823814034461975
  },
  "entities": [
    {
      "start": 0,
      "end": 2,
      "value": "渋谷",
      "entity": "location",
      "confidence": 0.7429793877231764,
      "extractor": "CRFEntityExtractor"
    },
    {
      "start": 7,
      "end": 12,
      "value": "イタリアン",
      "entity": "restaurant_type",
      "confidence": 0.7588623133724827,
      "extractor": "CRFEntityExtractor"
    },
    {
      "start": 0,
      "end": 2,
      "value": "渋谷",
      "entity": "location",
      "confidence": 0.7429793877231764,
      "extractor": "CRFEntityExtractor"
    },
    {
      "start": 7,
      "end": 12,
      "value": "イタリアン",
      "entity": "restaurant_type",
      "confidence": 0.7588623133724827,
      "extractor": "CRFEntityExtractor"
    }
  ],
  "intent_ranking": [
    {
      "name": "restaurant_ja",
      "confidence": 0.9823814034461975
    },
    {
      "name": "mood_great",
      "confidence": 0.008284788578748703
    },
    {
      "name": "deny",
      "confidence": 0.006644146051257849
    },
    {
      "name": "greet",
      "confidence": 0.001571052591316402
    },
    {
      "name": "bot_challenge",
      "confidence": 0.0006475948612205684
    },
    {
      "name": "affirm",
      "confidence": 0.00034828390926122665
    },
    {
      "name": "mood_unhappy",
      "confidence": 8.46567636472173e-05
    },
    {
      "name": "goodbye",
      "confidence": 3.805106462095864e-05
    }
  ],
  "text": "渋谷で美味しいイタリアンない？"
}
Next message:

```

おお。記事みたいな出力になった。

記事とは違って日本語は文字化けしてないけど。

環境ができて動くところまで見れたので、今回はここまで。
今後は教師データを増やす方法を考える。


## 追記

ここまでの成果として、pip で install したライブラリを requirements.txt にして、build したら勝手にインストールされるように Dockerfile に処理を追記しておいた。

```requirements.txt
rasa==1.9.2
spacy==2.2.4
```

```Dockerfile
FROM python:3.7-slim-stretch

RUN apt-get update -q -y && \
    apt-get install -q -y \
    build-essential

ADD ./requirements.txt requirements.txt

RUN pip install -r requirements.txt
RUN pip install "https://github.com/megagonlabs/ginza/releases/download/latest/ginza-latest.tar.gz"

ADD . /app
WORKDIR /app
```

んで、build して

```sh
$ docker-compose build
```

動かす

```sh
$ docker-compose run --rm rasa bash
root@56ff2c2f2f38:/app#
```

## 追記2

github で公開した

https://github.com/booink/rasa-trial

## 追記3

ginzaのインストールはpip経由で(3.0から)できるようになったようなので、requirements.txtにginzaを追記してDockerfileから該当箇所を削除した。

https://github.com/booink/rasa-trial/commit/eb8e020d70ad8d92e29c628008d550d214bcc14c