強化学習の系図

強化学習を一旦掘り下げられるよう整理したかったので，

- いつ
- だれが
- (どんな問題を解いたアルゴリズムで)
- 何の略称で
- 親は誰なのか

系図を作ってまとめてみました．(取り違えているかもしれません)

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/d16c909e-cf73-9d38-767f-5022f9d50ddb.png)
図: 主要な強化学習アルゴリズムの系図(左上の数字は誕生年)

#### [TD(λ)](https://www.researchgate.net/publication/225264698_Learning_to_Predict_by_the_Method_of_Temporal_Differences)(Sutton, 1984;1988)

- Temporal Differences
- Sutton, Richard S. "Learning to predict by the methods of temporal differences." Machine learning 3.1 (1988): 9-44.

#### [Q学習](https://link.springer.com/content/pdf/10.1007%2FBF00992698.pdf)(Watkins, 1989)

- Watkins, Christopher JCH, and Peter Dayan. "Q-learning." Machine learning 8.3-4 (1992): 279-292.

#### [REINFORCE](http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf)(Williams, 1992)

- REward Increment = Nonnegative Factor Offset Reinforcement
- Williams, Ronald J. "Simple statistical gradient-following algorithms for connectionist reinforcement learning." Machine learning 8.3-4 (1992): 229-256.

#### [SARSA](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.17.2539&rep=rep1&type=pdf)(Rummery, 1994)

- State-Action-Reward-State-Actionの略称(論文の注釈で登場)
- Rummery, Gavin A., and Mahesan Niranjan. On-line Q-learning using connectionist systems. Vol. 37. Cambridge, England: University of Cambridge, Department of Engineering, 1994.
- cf. sugulu, [Qiita -【強化学習初心者向け】シンプルな実装例で学ぶSARSA法およびモンテカルロ法【CartPoleで棒立て：1ファイルで完結】](https://qiita.com/sugulu/items/7a14117bbd3d926eb1f2)

#### [CEM](https://www.sciencedirect.com/science/article/abs/pii/S0377221796003852)(Rubenstein, 1997)

- Cross Entropy Method
- 進化的アルゴリズム
- 参考1: http://web.mit.edu/6.454/www/www_fall_2003/gew/CEtutorial.pdf
- 参考2: https://esc.fnwi.uva.nl/thesis/centraal/files/f2110275396.pdf
- 参考3: http://learning.mpi-sws.org/mlss2016/slides/2016-MLSS-RL.pdf

#### [AL](http://reports-archive.adm.cs.cmu.edu/anon/1999/CMU-CS-99-132.pdf)(Baird Ⅲ, 1999)

- Advantage Learning
- Baird III, Leemon C. Reinforcement learning through gradient descent. No. CMU-CS-99-132. CARNEGIE-MELLON UNIV PITTSBURGH PA DEPT OF COMPUTER SCIENCE, 1999.

#### [AC](https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf)(Sutton et al., 2000)

- Actor-Critic
- Sutton, Richard S., et al. "Policy gradient methods for reinforcement learning with function approximation." Advances in neural information processing systems. 2000.

#### [NFQ](http://ml.informatik.uni-freiburg.de/former/_media/publications/rieecml05.pdf)(Riedmiller, 2005)

- Neural Fitted Q Iteration
- Riedmiller, Martin. "Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method." European Conference on Machine Learning. Springer, Berlin, Heidelberg, 2005.

#### [REPS](https://www.aaai.org/ocs/index.php/AAAI/AAAI10/paper/viewFile/1851/2264)(Peters et al., 2010)

- Relative Entropy Policy Search
- Peters, Jan, Katharina Mülling, and Yasemin Altun. "Relative Entropy Policy Search." AAAI. 2010.

---

#### [DQN](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf)(Mnih et al., 2013)

- Deep Q-Networks
- Mnih, Volodymyr, et al. "Playing atari with deep reinforcement learning." arXiv preprint arXiv:1312.5602 (2013).
- cf. sugulu, [Qiita -【強化学習初心者向け】シンプルな実装例で学ぶQ学習、DQN、DDQN【CartPoleで棒立て：1ファイルで完結、Kearas使用】](https://qiita.com/sugulu/items/bc7c70e6658f204f85f9)

#### [DPG](http://proceedings.mlr.press/v32/silver14.pdf)(Silver et al., 2014)

- Deterministic Policy Gradient
- Silver, David, et al. "Deterministic policy gradient algorithms." ICML. 2014.

#### [DDQN](https://arxiv.org/abs/1509.06461)(Hasselt et al., 2015)

- Double DQN
- Van Hasselt, Hado, Arthur Guez, and David Silver. "Deep Reinforcement Learning with Double Q-Learning." AAAI. Vol. 2. 2016.
- cf. sugulu, [Qiita -【強化学習中級者向け】実装例から学ぶDueling Network DQN 【CartPoleで棒立て：1ファイルで完結】](https://qiita.com/sugulu/items/6c4d34446d4878cde61a)

#### [Dueling DQN](https://arxiv.org/abs/1511.06581)(Wang et al., 2015)

- Double DQNの進化版
- Wang, Ziyu, et al. "Dueling network architectures for deep reinforcement learning." arXiv preprint arXiv:1511.06581 (2015).
![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/95b5ebb0-62d1-c3ae-5042-cdc0e92467df.png)

#### [PAL](http://arxiv.org/abs/1512.04860)(Bellemare et al., 2015)

- Persistent Average Learning
- Wang, Ziyu, et al. "Dueling network architectures for deep reinforcement learning." arXiv preprint arXiv:1511.06581 (2015).

#### [TRPO](https://arxiv.org/abs/1502.05477)(Schulman et al., 2015)

- Trust Region Policy Optimization
- Schulman, John, et al. "Trust region policy optimization." International Conference on Machine Learning. 2015.

#### [DDPG](https://arxiv.org/abs/1509.02971)(Lillicrap et al., 2015)

- Deep DPG
- Lillicrap, Timothy P., et al. "Continuous control with deep reinforcement learning." arXiv preprint arXiv:1509.02971 (2015).

#### [PER](https://arxiv.org/abs/1511.05952)(Schual et al., 2015)

- Prioritized Experience Replay
- Schaul, Tom, et al. "Prioritized experience replay." arXiv preprint arXiv:1511.05952 (2015).
- sugulu, [【強化学習中級者向け】実装例から学ぶ優先順位付き経験再生 prioritized experience replay DQN 【CartPoleで棒立て：1ファイルで完結】](https://qiita.com/sugulu/items/10ac7ce53de40d4c8891)

#### [Gorila](https://arxiv.org/pdf/1507.04296.pdf)(Nair et al., 2016)

- General Reinforcement Learning Architecture
- Mnih, Volodymyr, et al. "Asynchronous methods for deep reinforcement learning." International conference on machine learning. 2016.

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/7adb78fc-5a81-df0e-1d91-969833c1a560.png)

#### [NAF](https://arxiv.org/abs/1603.00748)(Gu et al., 2016)

- Normalized Advantage Function
- 行動空間が連続な問題でも解けるようDouble DQNを拡張
- Gu, Shixiang, et al. "Continuous deep q-learning with model-based acceleration." International Conference on Machine Learning. 2016.

#### [A3C](https://arxiv.org/pdf/1602.01783)(Mnih et al., 2016)

- Asynchronous Advantage Actor-Critic
- Mnih, Volodymyr, et al. "Asynchronous methods for deep reinforcement learning." International conference on machine learning. 2016.
- cf. sugulu, [Qiita -【強化学習】実装しながら学ぶA3C【CartPoleで棒立て：1ファイルで完結】](https://qiita.com/sugulu/items/acbc909dd9b74b043e45)

#### [A2C](https://arxiv.org/abs/1602.01783)(Mnih et al., 2016)

- Advantage Actor-Critic
- アイデア: https://blog.openai.com/baselines-acktr-a2c/
- Mnih, Volodymyr, et al. "Asynchronous methods for deep reinforcement learning." International conference on machine learning. 2016.

#### [UNREAL](https://arxiv.org/abs/1611.05397)(Jarderberg et al., 2016)

- UNsupervised REinforcement and Auxiliary Learning
- Jaderberg, Max, et al. "Reinforcement learning with unsupervised auxiliary tasks." arXiv preprint arXiv:1611.05397 (2016).

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/230e15a5-09d4-5e18-9da3-6722c48743d5.png)

#### [GAIL](https://arxiv.org/pdf/1606.03476.pdf)(Ho et al., 2016)

- Generative Adversarial Imitation Learning
- Ho, Jonathan, and Stefano Ermon. "Generative adversarial imitation learning." Advances in Neural Information Processing Systems. 2016.

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/8136ee1c-886f-d6db-de53-707745ab8af9.png)

#### [ACER](https://arxiv.org/abs/1611.01224)(Wang et al., 2016)

- Actor-Critic with Experience Replay
- Wang, Ziyu, et al. "Sample efficient actor-critic with experience replay." arXiv preprint arXiv:1611.01224 (2016).

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/14e19b4f-b5e5-934b-24e7-272ca1b4d684.png)

#### [HER](https://arxiv.org/abs/1707.01495)(Andrychowicz et al., 2017)

- Hindsight Experience Replay
- Andrychowicz, Marcin, et al. "Hindsight experience replay." Advances in Neural Information Processing Systems. 2017.
- cf. ishizakiiii, [Qiita - 失敗からも学ぶ 強化学習 HERのアルゴリズムを理解して、OpenAI Gymの新しいロボットで試してみた](https://qiita.com/ishizakiiii/items/c02422087920961b2518)

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/39dce650-8d51-d065-92a6-7731bd98b835.png)

#### [C51](https://arxiv.org/abs/1707.06887)(Bellemare et al., 2017)

- Rainbowの特殊系
- A Distributional Perspective on Reinforcement Learning
- Bellemare, Marc G., Will Dabney, and Rémi Munos. "A distributional perspective on reinforcement learning." arXiv preprint arXiv:1707.06887 (2017).

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/012d7aa3-75d4-9ddf-4bad-ebe0928b0e1d.png)

#### [Rainbow](https://arxiv.org/abs/1710.02298) (Hessel et al., 2017)

- Hessel, Matteo, et al. "Rainbow: Combining improvements in deep reinforcement learning." arXiv preprint arXiv:1710.02298 (2017).
- cf. Jun Okumura, [SlideShare - DQNからRainbowまで 〜深層強化学習の最新動向〜](https://www.slideshare.net/juneokumura/dqnrainbow)

<img alt="figure1" src="https://qiita-image-store.s3.amazonaws.com/0/107399/f9036be7-c09c-3781-1216-ece4de88973f.png" width="50%">

<img alt="table1" src="https://qiita-image-store.s3.amazonaws.com/0/107399/c0e53a4f-ca3f-c104-42b0-736b9e42f045.png" width="50%">

<img alt="figure2" src="https://qiita-image-store.s3.amazonaws.com/0/107399/982c58db-3ec1-5912-e0d6-7a41e9e3cd92.png" width="100%">

<img alt="table2" src="https://qiita-image-store.s3.amazonaws.com/0/107399/631f9339-cf2f-301c-9cf9-f3c97753452c.png" width="50%">

<img alt="figure3" src="https://qiita-image-store.s3.amazonaws.com/0/107399/09600aea-b5a0-b88f-93fc-f1c97415c5c3.png" width="50%">

<img alt="table4" src="https://qiita-image-store.s3.amazonaws.com/0/107399/52a94d6e-8916-5393-b8c8-2956162ddabd.png" width="100%">

#### [PPO](https://arxiv.org/abs/1707.06347)(Schulman et al., 2017)

- Proximal Policy Optimization
- Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).
- cf. sugulu, [Qiita -【強化学習】実装しながら学ぶPPO【CartPoleで棒立て：1ファイルで完結】](https://qiita.com/sugulu/items/8925d170f030878d6582)

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/e5a47eaa-3624-063a-f239-b29f03298246.png)
![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/a1ed3aae-2225-6f0f-378e-2adf34fe8923.png)
[Song et al., 2018](https://arxiv.org/pdf/1801.05566.pdf)より

#### [ACKTR](https://arxiv.org/abs/1708.05144)(Wu et al., 2017)

- Actor Critic using Kronecker-Factored Trust Region
- Wu, Yuhuai, et al. "Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation." Advances in neural information processing systems. 2017.

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/a50aa90f-9f92-6b09-b227-9d4f1cefa15a.png)

#### [SDQN](https://arxiv.org/pdf/1705.05035.pdf)(Metz et al., 2017)

- Sequential DQN
- Metz, Luke, et al. "Discrete sequential prediction of continuous actions for deep RL." arXiv preprint arXiv:1705.05035 (2017).

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/f4b4b96d-1853-ebe9-0248-93ce8fc0b486.png)
![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/44d9ad11-b045-d79d-9daa-54a737ce0b3e.png)

#### [QR-DQN](https://arxiv.org/abs/1710.10044)(Dabney et al., 2017)

- Quantile Regression DQN
- Dabney, Will, et al. "Distributional reinforcement learning with quantile regression." arXiv preprint arXiv:1710.10044 (2017).

#### [IQN](https://arxiv.org/abs/1806.06923)(Dabney et al., 2018)

- Implicit Quantile Network
- Dabney, Will, et al. "Implicit Quantile Networks for Distributional Reinforcement Learning." arXiv preprint arXiv:1806.06923 (2018).

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/3df2b209-5f08-e307-b6f7-f3a82d0b58a5.png)
![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/1384b349-d57d-0717-c57d-12fc60b791ee.png)
![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/038bc9c7-daf3-67e7-407f-d9933e64d181.png)
![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/043ae22b-90a8-8ef1-2e51-84a43927ac3b.png)

#### [Ape-X](https://arxiv.org/abs/1803.00933)(Horgan et al. 2018)

- DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY
- 参考: [【深層強化学習】『2018年最強手法(?)』Ape-X 実装・解説](https://qiita.com/omuram/items/bb7d463d8177cb395bb7), Qiita.

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/b88d9a7b-554a-5c2c-2698-d50533f348bb.png)

![image.png](https://qiita-image-store.s3.amazonaws.com/0/107399/fc0b7998-91ba-9d7d-8225-35c629de2b05.png)


## References

- [GitHub - Deep Reinforcement Learning Papers](https://github.com/muupan/deep-reinforcement-learning-papers)