Tensorflow-Tacotron-2をtensorflow-rocmで動くかどうかの検証メモ

https://github.com/Rayhane-mamah/Tacotron-2

pythochでのTacotron2はうまく動かなかったのでなるべくTensorflowでは動いてほしかったのですが大変厳しかったです
その時の動作検証メモになります. 


##環境

CPU Xeon E5-2603 v4
GPU RadeonⅦ
GPU GTX1080Ti
RAM DDR4 48GB

OS Ubuntu 18.04.3 LTS
Kernel Linux rocm 5.0.0-31-generic 
Python環境 miniconda 4.7.10
ROCm vesion 2.9.6
CUDA V10.1.243
NVIDIA Driver Version: 430.26 

上記の環境を整えた上で本テストを行いました.


#ネイティブ環境での動作検証

```
 git clone https://github.com/Rayhane-mamah/Tacotron-2.git
cd ./Tacotron-2
wget https://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2
tar xf ./LJSpeech-1.1.tar.bz2 LJSpeech-1.1/
```

miniconda を使用してpython3.6環境を立ち上げてから






```
pip install -r ./requirements.txt 
pip install -r tensorflow-rocm==1.13.4
python preprocess.py
```
pyaudioのみ、インストールがpipで出来なかったので ``` conda install pyaudio```で行った、あまりpipとcondaの混合環境は推奨できないのでできればrequirements.txtでインストールできるならしたほうがよいです.

python preprocess.pyの処理はそれなりに時間がかかるのでしばらく待ちます



```
python train.py --model='Tacotron-2'
```

これで学習が開始できるはずなのですがCPU TFでは学習ができたのに対し、ROCｍでは
```ValueError: operands could not be broadcast together with shapes (1,1025) (0,) ```
となってしまい学習が出来ませんでした.



https://qiita.com/Tarooo000/items/dcce992672c7ea539049
>[備忘録] tensorflow-rocmでimport時にnumpyのFutureWaringが出た話

これを参考に1.16.4にダウングレードしましたが動かなかった.


rocmが悪いのか設定とかが悪いのかわからないのNVIDIA-Dockerでnumpyのバージョン等を検討してから再検証することにしました.

#NVIDIA-Dockerでの動作検証

そこで元々NVIDIA-CUDA用のDockerfileがあるのでこれで動作検証をして動くようならtensorflow-rocm imageにdocker fileを置き換えて実験するのが良いのではないかと考えた


RepositoryのDockerfileは以下のようになっているが

```
FROM continuumio/anaconda3:latest
FROM tensorflow/tensorflow:latest-gpu-py3

RUN apt-get update
RUN apt-get install -y libasound-dev portaudio19-dev libportaudio2 libportaudiocpp0 ffmpeg libav-tools wget git vim

RUN wget http://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2
RUN tar -jxvf LJSpeech-1.1.tar.bz2

RUN git clone https://github.com/Rayhane-mamah/Tacotron-2.git


WORKDIR Tacotron-2
RUN ln -s ../LJSpeech-1.1 .
RUN pip install -r requirements.txt
```

```RUN apt-get install -y libasound-dev portaudio19-dev libportaudio2 libportaudiocpp0 ffmpeg libav-tools wget git vim```

この部分の```libhav-tools```を消してください　ubuntu18.04ではすでに廃止されたパッケージです
またなんのためのDockerなんだって感じはあるんですがデフォルトで最新のtensorflow-gpuがinstall
されるようになっているのが仇になってしまいtf2.0系ではうまく動作しないのでコンテナに入ったら
```pip3 install tensorflow-gpu==1.15.0```をしてTFを1.X系に戻す作業が必要になります.



```
sudo docker build -t tacotron2:cuda  .
sudo docker run -it --name tacotron2 tacotron2:cuda```
```





```
#python preprocess.py
#python train.py --model='Tacotron-2'

Step       1 [63.983 sec/step, loss=24.80627, avg_loss=24.80627]
Saving Model Character Embeddings visualization..
Tacotron Character embeddings have been updated on tensorboard!
Step       4 [49.477 sec/step, loss=10.18475, avg_loss=15.85305]
Step       5 [45.228 sec/step, loss=12.18279, avg_loss=15.11899]
```

無事学習自体はできるようなのでこれを元にROCｍ Dockerで動かしてみます


#ROCm-Dockerで動作確認
なるべく独立した環境で再検証するべくDockerでやってみます

rocm2.9-tf1.15-devのimageが配布されているのでコレを使います
https://github.com/RadeonOpenCompute/ROCm-docker
https://hub.docker.com/r/rocm/tensorflow/tags

```
FROM  rocm/tensorflow:rocm2.9-tf1.15-dev


RUN apt-get update
RUN apt-get install -y libasound-dev portaudio19-dev libportaudio2 libportaudiocpp0 ffmpeg libav-tools wget git vim

RUN wget http://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2
RUN tar -jxvf LJSpeech-1.1.tar.bz2

RUN git clone https://github.com/Rayhane-mamah/Tacotron-2.git


WORKDIR Tacotron-2
RUN ln -s ../LJSpeech-1.1 .
RUN pip install -r requirements.txt
```

dockerfileを保存してビルドする.

```
sudo docker build -t tacotron2:rocm  .
```

build自体は成功したのですが

```
ERROR: tensorflow 1.15.0rc2 has requirement numpy<2.0,>=1.16.0, but you'll have numpy 1.14.0 which is incompatible.
```

やはりTensorflow-rocmの仕様でnumpy1.16.0以上を使わないとダメなようでそこがネックになりそうです
動かしてみないとわからないのでひとまずDocker runしてみます


```
sudo docker run -it --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --group-add video tacotron2:rocm
```

ひとますnumpy==1.14.0でうごかしてみます

```
python3 preprocess.py 
ImportError: No module named 'numpy.core._multiarray_umath'
ImportError: numpy.core.multiarray failed to import

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
SystemError: PyEval_EvalFrameEx returned a result with an error set
ImportError: numpy.core._multiarray_umath failed to import
ImportError: numpy.core.umath failed to import
2019-10-24 18:03:18.669564: F tensorflow/python/lib/core/bfloat16.cc:675] Check failed: PyBfloat16_Type.tp_base != nullptr 
Aborted (core dumped)
```

これだとtensorflow-rocm=1.15.0では動かないようなので
1.16.0にしてみます.

```
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/root/Tacotron-2/datasets/preprocessor.py", line 117, in _process_utterance
    mel_spectrogram = audio.melspectrogram(preem_wav, hparams).astype(np.float32)
  File "/root/Tacotron-2/datasets/audio.py", line 73, in melspectrogram
    S = _amp_to_db(_linear_to_mel(np.abs(D)**hparams.magnitude_power, hparams), hparams) - hparams.ref_level_db
  File "/root/Tacotron-2/datasets/audio.py", line 228, in _linear_to_mel
    _mel_basis = _build_mel_basis(hparams)
  File "/root/Tacotron-2/datasets/audio.py", line 246, in _build_mel_basis
    fmin=hparams.fmin, fmax=hparams.fmax)
  File "/usr/local/lib/python3.5/dist-packages/librosa/filters.py", line 247, in mel
    lower = -ramps[i] / fdiff[i]
ValueError: operands could not be broadcast together with shapes (1,1025) (0,) 
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "preprocess.py", line 110, in <module>
    main()
  File "preprocess.py", line 106, in main
    run_preprocess(args, modified_hp)
  File "preprocess.py", line 83, in run_preprocess
    preprocess(args, input_folders, output_folder, hparams)
  File "preprocess.py", line 17, in preprocess
    metadata = preprocessor.build_from_path(hparams, input_folders, mel_dir, linear_dir, wav_dir, args.n_jobs, tqdm=tqdm)
  File "/root/Tacotron-2/datasets/preprocessor.py", line 42, in build_from_path
    return [future.result() for future in tqdm(futures) if future.result() is not None]
  File "/root/Tacotron-2/datasets/preprocessor.py", line 42, in <listcomp>
    return [future.result() for future in tqdm(futures) if future.result() is not None]
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
ValueError: operands could not be broadcast together with shapes (1,1025) (0,) 
```

1.16.0ではやはりダメなようで

https://github.com/Rayhane-mamah/Tacotron-2/issues/427
>check numpy version and scipy version, make sure they are same as requirements

1.14.0を使うとtensorflow-rocmが未対応で1.16.0を使うとTactron２が対応していないと言う挟み撃ち状態になってしまいました

#最後にネイティブ環境でTF-ROCm1.12.0で試してみる

ダメな原因がわかったので古いTFなら古いnumpyでも動くのではないかと思ったので最初に環境を流用して実行してみます
参考までにpip listは以下の通り

```
------------------ ---------
absl-py             0.8.1    
astor               0.8.0    
audioread           2.1.5    
certifi             2019.9.11
cffi                1.12.3   
cget                0.1.8    
click               6.6      
cycler              0.10.0   
decorator           4.4.0    
falcon              1.2.0    
gast                0.3.2    
grpcio              1.24.3   
h5py                2.10.0   
inflect             0.2.5    
joblib              0.14.0   
Keras               2.3.1    
Keras-Applications  1.0.8    
Keras-Preprocessing 1.1.0    
kiwisolver          1.1.0    
librosa             0.5.1    
llvmlite            0.30.0   
lws                 1.2.5    
Markdown            3.1.1    
matplotlib          2.0.2    
numba               0.46.0   
numpy               1.14.0   
pbr                 5.4.3    
pip                 19.3.1   
protobuf            3.10.0   
PyAudio             0.2.11   
pycparser           2.19     
pyparsing           2.4.2    
python-dateutil     2.8.0    
python-mimeparse    1.6.0    
pytz                2019.3   
PyYAML              5.1.2    
rbuild              0.0.1    
resampy             0.2.2    
scikit-learn        0.21.3   
scipy               1.0.0    
setuptools          41.2.0   
six                 1.12.0   
sounddevice         0.3.10   
SoundFile           0.10.2   
tensorboard         1.12.2   
tensorflow-rocm     1.12.0   
termcolor           1.1.0    
tqdm                4.11.2   
Unidecode           0.4.20   
Werkzeug            0.16.0   
wheel               0.33.6 
```

```
$ python ./preprocess.py 
Traceback (most recent call last):
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: /home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: undefined symbol: _ZN8hip_impl22hipLaunchKernelGGLImplEmRK4dim3S2_jP12ihipStream_tPPv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./preprocess.py", line 5, in <module>
    from datasets import preprocessor
  File "/home/rocm/Tacotron-2/datasets/preprocessor.py", line 6, in <module>
    from datasets import audio
  File "/home/rocm/Tacotron-2/datasets/audio.py", line 4, in <module>
    import tensorflow as tf
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/rocm/miniconda3/envs/tf13/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: /home/rocm/miniconda3/envs/tf13/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: undefined symbol: _ZN8hip_impl22hipLaunchKernelGGLImplEmRK4dim3S2_jP12ihipStream_tPPv


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

```

どうやら足りない機能があるらしくTF-ROCm1.12では動かないようだった
TF-ROCm 1.13でも同様の結果に終わったので結論としては```preprocess.py ```による前処理はROCｍでは難しいと言うのが実情だと思います.

preprocess.pyだけCUDAにやらせてみたいと思います




```
pip install tensorflow-gpu==1.15.0
python ./preprocess.py 
pip install tensorflow-rocm==1.14.0
pip uninstall tensorflow-gpu
```

rocm1.14.xを入れた時点で自動的にnumpy1.17.0がinstallされるのは注意が必要です

1.14.1を入れるとこのように

```
python train.py --model='Tacotron-2'
Traceback (most recent call last):
  File "train.py", line 7, in <module>
    from hparams import hparams
  File "/home/rocm/Tacotron-2/hparams.py", line 5, in <module>
    hparams = tf.contrib.training.HParams(
AttributeError: module 'tensorflow' has no attribute 'contrib'
```

TF-ROCm1.14.1ではなぜかTF-ROCm2.0と同じエラーが出ます


ただ1.14.0でも同様に

```
Traceback (most recent call last):
  File "train.py", line 138, in <module>
    main()
  File "train.py", line 132, in main
    train(args, log_dir, hparams)
  File "train.py", line 52, in train
    checkpoint = tacotron_train(args, log_dir, hparams)
  File "/home/rocm/Tacotron-2/tacotron/train.py", line 399, in tacotron_train
    return train(log_dir, args, hparams)
  File "/home/rocm/Tacotron-2/tacotron/train.py", line 176, in train
    GLGPU_mel_outputs = audio.inv_mel_spectrogram_tensorflow(GLGPU_mel_inputs, hparams)
  File "/home/rocm/Tacotron-2/datasets/audio.py", line 142, in inv_mel_spectrogram_tensorflow
    S = _mel_to_linear_tensorflow(S, hparams)  # Convert back to linear
  File "/home/rocm/Tacotron-2/datasets/audio.py", line 240, in _mel_to_linear_tensorflow
    _inv_mel_basis = np.linalg.pinv(_build_mel_basis(hparams))
  File "/home/rocm/Tacotron-2/datasets/audio.py", line 246, in _build_mel_basis
    fmin=hparams.fmin, fmax=hparams.fmax)
  File "/home/rocm/.local/lib/python3.6/site-packages/librosa/filters.py", line 247, in mel
    lower = -ramps[i] / fdiff[i]
ValueError: operands could not be broadcast together with shapes (1,1025) (0,) 
```

以上のようになってしまうのでやはりROCmでは厳しいかなと言う結論に至りました

numpy==1.14.0 TF-ROCm==1.12.0などの組み合わせでも動作しない為厳しいなと言った感じでした

ただTF1.13.4+numpy1.14.0の組み合わせでは一応Train自体は走るのですが

```
Tacotron training set to a maximum of 100000 steps
*** stack smashing detected ***: <unknown> terminated
Aborted (core dumped)
```
途中でコアダンプで落ちてしまってるのでなんか問題がありそうです.