AIを使って不労所得の道へ(2)～MLPでFXの学習させてみた～【AIncomeプロジェクト】

## はじめに

### 素人でも戦略的に取引したい

株やFXの知識がないため，戦略的な取引ができない素人が多いと思います．私もその一人です．
そこで，強化学習の力を借りて戦略的に取引をしようと思います．



### AIncomeプロジェクト

AIで所得（Income）を得るため「AIncome」と名付けました．

![](https://hamaruki.com/wp-content/uploads/2022/08/スクリーンショット-2022-08-21-173848-300x119.jpg)

### 前回までの取り組み

前回は強化学習とそれをシミュレーションするOpenAI Gymの説明をしました．さらには，ランダムで行動を決定して売り買いしてみたところ，ランダムでも利益が出るような結果になっていました．

https://hamaruki.com/2022/08/21/the_ai_unemployment_income_path_01/

### 今回の取り組みの概要

今回はFXの取引を学習させていきたいと思います．


## Google Drive setting

Google Driveと連携します．

```python
from google.colab import drive
drive.mount('/content/drive')
```

    Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
    

プロジェクトフォルダに移動します．

```python
%cd /content/drive/MyDrive/AIncome
```

    /content/drive/MyDrive/AIncome
    

現在地はここにいます．

```python
!pwd
```

    /content/drive/MyDrive/AIncome
    

### 作業フォルダ

作業用フォルダを作成します．

```python
!mkdir AutoTrade02
```

    mkdir: cannot create directory ‘AutoTrade02’: File exists
    

作業用フォルダに移動します．

```python
%cd  AutoTrade02
```

    /content/drive/MyDrive/AIncome/AutoTrade02
    


```python
!pwd
```

    /content/drive/MyDrive/AIncome/AutoTrade02
    

## 必要パッケージのインストール


```python
#!pip install gym[all] -U
!pip install "gym==0.19.0"
!pip install stable-baselines[mpi]
!pip install tensorflow==1.14.0
!pip install pyqt5
!pip install imageio
!pip install gym-anytrading
```

    Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
    Requirement already satisfied: gym==0.19.0 in /usr/local/lib/python3.7/dist-packages (0.19.0)
    Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym==0.19.0) (1.5.0)
    Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.7/dist-packages (from gym==0.19.0) (1.21.6)
    Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
    Requirement already satisfied: stable-baselines[mpi] in /usr/local/lib/python3.7/dist-packages (2.10.2)
    Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (1.1.0)
    Requirement already satisfied: cloudpickle>=0.5.5 in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (1.5.0)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (1.21.6)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (3.2.2)
    Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (4.6.0.66)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (1.7.3)
    Requirement already satisfied: gym[atari,classic_control]>=0.11 in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (0.19.0)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (1.3.5)
    Requirement already satisfied: mpi4py in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (3.1.3)
    Requirement already satisfied: atari-py==0.2.6 in /usr/local/lib/python3.7/dist-packages (from gym[atari,classic_control]>=0.11->stable-baselines[mpi]) (0.2.6)
    Requirement already satisfied: pyglet>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym[atari,classic_control]>=0.11->stable-baselines[mpi]) (1.5.26)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from atari-py==0.2.6->gym[atari,classic_control]>=0.11->stable-baselines[mpi]) (1.15.0)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]) (2.8.2)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]) (3.0.9)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]) (0.11.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]) (1.4.4)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib->stable-baselines[mpi]) (4.1.1)
    Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->stable-baselines[mpi]) (2022.2.1)
    Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
    Requirement already satisfied: tensorflow==1.14.0 in /usr/local/lib/python3.7/dist-packages (1.14.0)
    Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (0.2.0)
    Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (1.2.0)
    Requirement already satisfied: tensorboard<1.15.0,>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (1.14.0)
    Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (1.1.0)
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (1.47.0)
    Requirement already satisfied: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (1.14.0)
    Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (1.21.6)
    Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (0.37.1)
    Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (3.17.3)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (1.15.0)
    Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (1.14.1)
    Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (0.8.1)
    Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (1.0.8)
    Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (1.1.2)
    Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==1.14.0) (0.5.3)
    Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from keras-applications>=1.0.6->tensorflow==1.14.0) (3.1.0)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (1.0.1)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.4.1)
    Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (57.4.0)
    Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (4.12.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (4.1.1)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.8.1)
    Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py->keras-applications>=1.0.6->tensorflow==1.14.0) (1.5.2)
    Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
    Requirement already satisfied: pyqt5 in /usr/local/lib/python3.7/dist-packages (5.15.7)
    Requirement already satisfied: PyQt5-sip<13,>=12.11 in /usr/local/lib/python3.7/dist-packages (from pyqt5) (12.11.0)
    Requirement already satisfied: PyQt5-Qt5>=5.15.0 in /usr/local/lib/python3.7/dist-packages (from pyqt5) (5.15.2)
    Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
    Requirement already satisfied: imageio in /usr/local/lib/python3.7/dist-packages (2.21.1)
    Requirement already satisfied: pillow>=8.3.2 in /usr/local/lib/python3.7/dist-packages (from imageio) (9.2.0)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from imageio) (1.21.6)
    Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
    Requirement already satisfied: gym-anytrading in /usr/local/lib/python3.7/dist-packages (1.3.1)
    Requirement already satisfied: gym>=0.12.5 in /usr/local/lib/python3.7/dist-packages (from gym-anytrading) (0.19.0)
    Requirement already satisfied: numpy>=1.16.4 in /usr/local/lib/python3.7/dist-packages (from gym-anytrading) (1.21.6)
    Requirement already satisfied: pandas>=0.24.2 in /usr/local/lib/python3.7/dist-packages (from gym-anytrading) (1.3.5)
    Requirement already satisfied: matplotlib>=3.1.1 in /usr/local/lib/python3.7/dist-packages (from gym-anytrading) (3.2.2)
    Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym>=0.12.5->gym-anytrading) (1.5.0)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.1->gym-anytrading) (2.8.2)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.1->gym-anytrading) (3.0.9)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.1->gym-anytrading) (1.4.4)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.1->gym-anytrading) (0.11.0)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib>=3.1.1->gym-anytrading) (4.1.1)
    Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24.2->gym-anytrading) (2022.2.1)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib>=3.1.1->gym-anytrading) (1.15.0)
    


```python

```

## インポート



```python
import os, gym
import gym_anytrading
import matplotlib.pyplot as plt
from gym_anytrading.envs import TradingEnv, ForexEnv, StocksEnv, Actions, Positions
from gym_anytrading.datasets import FOREX_EURUSD_1H_ASK, STOCKS_GOOGL
from stable_baselines.common.vec_env import DummyVecEnv
from stable_baselines import PPO2
from stable_baselines import ACKTR
from stable_baselines.bench import Monitor
from stable_baselines.common import set_global_seeds


```

## 設定


```python
# ログフォルダの生成
log_dir = './logs/'
os.makedirs(log_dir, exist_ok=True)

idx1 = 10
idx2 = 300

span = idx2 - idx1


```

## 環境の生成


```python
# 環境の生成
env = gym.make('forex-v0', frame_bound=(idx1, idx2), window_size=10)
env = Monitor(env, log_dir, allow_early_resets=True)

# シードの指定
env.seed(0)
set_global_seeds(0)

# ベクトル化環境の生成
env = DummyVecEnv([lambda: env])

# モデルの生成
model = PPO2('MlpPolicy', env, verbose=1)
#model = ACKTR('MlpPolicy', env, verbose=1)

# モデルの読み込み
# model = PPO2.load('trading_model')
```

    WARNING:tensorflow:Entity <bound method Flatten.call of <tensorflow.python.layers.core.Flatten object at 0x7f35963e6fd0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Flatten.call of <tensorflow.python.layers.core.Flatten object at 0x7f35963e6fd0>>: AttributeError: module 'gast' has no attribute 'Index'
    WARNING:tensorflow:Entity <bound method Flatten.call of <tensorflow.python.layers.core.Flatten object at 0x7f35952adfd0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Flatten.call of <tensorflow.python.layers.core.Flatten object at 0x7f35952adfd0>>: AttributeError: module 'gast' has no attribute 'Index'
    

    WARNING: Entity <bound method Flatten.call of <tensorflow.python.layers.core.Flatten object at 0x7f35963e6fd0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Flatten.call of <tensorflow.python.layers.core.Flatten object at 0x7f35963e6fd0>>: AttributeError: module 'gast' has no attribute 'Index'
    WARNING: Entity <bound method Flatten.call of <tensorflow.python.layers.core.Flatten object at 0x7f35952adfd0>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: converting <bound method Flatten.call of <tensorflow.python.layers.core.Flatten object at 0x7f35952adfd0>>: AttributeError: module 'gast' has no attribute 'Index'
    

## 学習


```python
# モデルの学習
model.learn(total_timesteps=128000)

# モデルの保存
model.save('trading_model')
```

    [1;30;43mストリーミング出力は最後の 5000 行に切り捨てられました。[0m
    | serial_timesteps   | 85376          |
    | time_elapsed       | 89.8           |
    | total_timesteps    | 85376          |
    | value_loss         | 699.543        |
    ---------------------------------------
    ---------------------------------------
    | approxkl           | 4.803229e-06   |
    | clipfrac           | 0.0            |
    | ep_len_mean        | 289            |
    | ep_reward_mean     | 166            |
    | explained_variance | -3.58e-07      |
    | fps                | 1011           |
    | n_updates          | 668            |
    | policy_entropy     | 0.5915557      |
    | policy_loss        | -0.00012180582 |
    | serial_timesteps   | 85504          |
    | time_elapsed       | 90             |
    | total_timesteps    | 85504          |
    | value_loss         | 534.26917      |
    ---------------------------------------
    
	.....................
	
    ---------------------------------------
    | approxkl           | 1.32921905e-05 |
    | clipfrac           | 0.0            |
    | ep_len_mean        | 289            |
    | ep_reward_mean     | 141            |
    | explained_variance | 1.79e-07       |
    | fps                | 976            |
    | n_updates          | 1000           |
    | policy_entropy     | 0.21772377     |
    | policy_loss        | -0.00046240288 |
    | serial_timesteps   | 128000         |
    | time_elapsed       | 133            |
    | total_timesteps    | 128000         |
    | value_loss         | 180.04169      |
    ---------------------------------------
    

## モデルのテスト


```python
env = gym.make('forex-v0', frame_bound=(idx1, idx2), window_size=10)
env.seed(0)
state = env.reset()
while True:
    # 行動の取得
    action, _ = model.predict(state)
    # 1ステップ実行
    state, reward, done, info = env.step(action)
    # エピソード完了
    if done:
        print('info:', info)
        break
```

    info: {'total_reward': -363.3000000000041, 'total_profit': 0.9551622411792473, 'position': 1}
    

累計報酬（total_reward），純利益（total_profit），ポジション（position：0がショート、1がロング）が表示されています．

## グラフのプロット


```python
plt.cla()
env.render_all()
plt.show()
```


![](https://hamaruki.com/wp-content/uploads/2022/08/output_23_0-300x216.png)

赤●が「0:Sell」で落ちる予想，緑●が「1:Buy」で上がる予想を表します．

思ったよりもいい感じの結果ですね．



# 最新情報はツイッターにて！！
▼▼▼▼
▼▼▼
▼▼
▼


<blockquote class="twitter-tweet"><p lang="ja" dir="ltr">AIを使って不労所得の道へ<br>(10)～348億!? ビットコイン(仮想通貨)のデータセットを食わせて強化<br>学習を回してみた～<a href="https://twitter.com/hashtag/%E4%BB%8A%E6%97%A5%E3%81%AE%E7%A9%8D%E3%81%BF%E4%B8%8A%E3%81%92?src=hash&amp;ref_src=twsrc%5Etfw">#今日の積み上げ</a><a href="https://twitter.com/hashtag/AI?src=hash&amp;ref_src=twsrc%5Etfw">#AI</a><a href="https://twitter.com/hashtag/python?src=hash&amp;ref_src=twsrc%5Etfw">#python</a><a href="https://twitter.com/hashtag/%E5%BC%B7%E5%8C%96%E5%AD%A6%E7%BF%92?src=hash&amp;ref_src=twsrc%5Etfw">#強化学習</a><a href="https://twitter.com/hashtag/%E4%B8%8D%E5%8A%B4%E6%89%80%E5%BE%97?src=hash&amp;ref_src=twsrc%5Etfw">#不労所得</a><a href="https://twitter.com/hashtag/%E6%8A%95%E8%B3%87?src=hash&amp;ref_src=twsrc%5Etfw">#投資</a><a href="https://twitter.com/hashtag/%EF%BD%8D%EF%BD%8Dbot?src=hash&amp;ref_src=twsrc%5Etfw">#ｍｍbot</a><a href="https://twitter.com/hashtag/%E8%87%AA%E5%8B%95%E5%A3%B2%E8%B2%B7?src=hash&amp;ref_src=twsrc%5Etfw">#自動売買</a><a href="https://twitter.com/hashtag/%E3%82%AA%E3%83%BC%E3%83%88%E3%83%88%E3%83%AC%E3%83%BC%E3%83%89?src=hash&amp;ref_src=twsrc%5Etfw">#オートトレード</a><a href="https://twitter.com/hashtag/AIncome?src=hash&amp;ref_src=twsrc%5Etfw">#AIncome</a><a href="https://twitter.com/hashtag/%E4%B8%BB%E5%A4%AB?src=hash&amp;ref_src=twsrc%5Etfw">#主夫</a><a href="https://twitter.com/hashtag/%E3%82%A8%E3%83%B3%E3%82%B8%E3%83%8B%E3%82%A2?src=hash&amp;ref_src=twsrc%5Etfw">#エンジニア</a> <a href="https://t.co/cvqEnWUloT">https://t.co/cvqEnWUloT</a></p>&mdash; はまる木@エンジニア夫婦CH (@hAru_mAki_ch) <a href="https://twitter.com/hAru_mAki_ch/status/1562959371318923267?ref_src=twsrc%5Etfw">August 26, 2022</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>


# まとまった情報はブログにて

▼▼▼▼
▼▼▼
▼▼
▼

https://hamaruki.com/category/ai%e3%82%92%e4%bd%bf%e3%81%a3%e3%81%a6%e4%b8%8d%e5%8a%b4%e6%89%80%e5%be%97/