56014 %56.7 2 Kazuki Yoshizoe5614&)   ! "-/$ ,'(+-/   !-AIP2019922 #3"*201978$ !
"  ! 
,1"2102

-
(4%SAT
socialnetwork)&5/*+(6'1(3 -
(3
#.

AlphaGo = DeepLearning + 78

2!0
Google DeepMind ,26

&3 +.
/("$(2006)34

Deep Learning( )/("$)
(2006)%5' #1(*-)https://deepmind.com/research/dqn/

Arcade Learning
Environment[Coulom 2006][Silver, Huang et al. 2016] Fig. 1b

https://github.com/mgbellemare/Arcade-Learning-Environment
https://www.youtube.com/watch?v=nzUiEkasXZI[Coulom 2006] CrazyStone

Monte Carlo Tree Search$.10",%(  ,%# & )

UCB1 algorithm!
2ln'+""()

'+

game tree - *.1

2!/,!-0 ,!
#%) $( *./

*"-0+
21$'&4

[Brügmann 1993]

6;#&! $%LPQCI*B
D*BI/%$"D/%$"
4/1>

3/2>

2/3>

(B/*2@ +1N=?,-O 5H*) @ +1 AE 3<LP
G9.7 :  0FK4J:0'8M5)0)0

$"#89-1$"#89





(&*2&/6.37/  &
-1#! CrazyStone [Coulom 2006]#"+:4,%

5'67[Robbins 1952][Lai and Robbins 1984]

Multi-Armed Bandit Problem[Auer, Cesa-Bianchi, Fischer 2002]

UCB1MAB  %+-3

cumulative regret$#"

w
2 ln t+ssmean/ + bias/w : 1(2s : ' t : s )10&regret $#"!

*.?

$,!

[Kocsis, Szepesvári 2006]

UCT algorithm

Upper Confidence bound applied to Trees
4&+5UCB1$ w
2 ln t+ss.

' n ,24&/#w
æ log n ö+ Oç÷
sè n ø,2

1"

1!-(* 0 )3()%0 8UCT UCT() {while(time_remaining) {iter()}}iter() {node = root
while (true) {node = SelectBestChild(node)
if(node is leaf) break}if(visit_count(node) > expand_threshold) {expand(node)
node =SelectBestChild(node)}
reward = rollout(node)
backup(reward)}    backup

UCB1, UCT: 8+)-?>6F [0, 1]  c=18+3*[Auer, Cesa-Bianchi and fischer 2002]

UCB18+Chernoff bound /[Kocsis, Szepesvári 2006]

UCT8+<&#59=;C 
Hoeffding-Azuma 1D:/
BA [0, 1] 7(c=1 ) 8+13*1( ! 
E' 8+13*

exploration constantw
2 ln t+css,G%24"c 0@  $"
 c .

10


1#

(+%9

single player
multiplayer
two player<,A$) 
schedulingplanning
biometric security

MCTS A$)6."*/   :!52=> 3  &0MCTS
4;?-'7@8

11

MCTS 
single player games(real time games)

12



[Yang, Zhang, Yoshizoe, Terayama, Tsuda 2017]“ChemTS: an efficient python library for de novomolecular generation”.

 



Canadian Traveler Problem[Z. Bnaya, A. Felner, D. Fried, O. Maksin and S. E. Shimony]“Repeated-Task Canadian Traveler Problem”, SoCS 2011

multi player games

biometric security[Tanabe, Yoshizoe and Imai 2009]

two playergames“A study on security evaluation methodology for image-based
biometrics authentication systems”[Cazenave, Balbo, Pinson 2009]“Monte-Carlo bus regulation”[Chevelu, Putois, Lepage 2010]scheduling“The true score of statistical paraphrase generation”

MCTSB:6B:1

0.&(*V7RWPF3A 2KCE#
TN95D8MX-1),+/'
2KCEL%Q #SH0.&(*?U
B!$# ;>I2K

#H

G:MCTS MX
@J "<@J#O34:=…

…

13

MCTS+('+(214MCTS=)

9!7*/2#?A(trap, 45)&1#,@6 [Coquelin, Munos 2007]
-#9!*.
 0>;

%E#,@6 [Ramanujan, Selman 2011]" B:+(:9!6<C+(8D9!

3333

$

3

15.5("24ARS Anytime ?

12' -%-O=Ms. Pac-Man AI Competition,AI Competition

MCTS =+4$0@BDMP< [Ikehata, Itoh 2011]

WGC#&/5%#45FAI67VNE42!)F MCTS

8:+3 T;L(Turing Track)
A*RS MCTS IU >H

Ms. Pac-Man

1/15K

9>

,

http://cswww.essex.ac.uk/staff/sml/pacman/PacManContest.html+0 *&
JQ 

1/24K

9>

http://www.platformersai.com/

AlphaGo

Deep Neural Network") *+

!
")*+#$'≒ &%(1617MCTS -3,0: Root ,0. /124+ 13 *#" $ '  20,0 ")
24,0(   &%!: SAT solver  portfolio ,0-.$ 59;>4
radfbeghcikjlnm%'" ;>;>2!?@+) #1o
6<;>2!
0 59&#,/(,/!

0 1 2 387 3=:%*

4

18

! signature
011101001011
DF

Hash tablehash key

101000111100
001101010101• DF! ,8
– Transposition Table : TT I<, A – -@E0 signature 39'1:/• .6)&– DFGB+%>*$(• 7=?*, History Heuristics,– #;45H"392C

[Evett et al. 1995] PRA*

[Romein et al. 1999] TDSHash driven parallel search3

311 01001011

10 Hash Table
Hash Table

Hash Key2210 11010001

1101 10001110

32 Hash Table
Hash Table" (Hash table driven parallel search) *•••    
 1 1!$( !$,' %#!$ &)  +,(%.U4MQSV4>

2

354

16

T9$;?+* @6
" ,(%.2:RL"6314521&/-

&/-GO

E7T9;?5 +*NC

BAIT9$@6";?+* J T9$@6 ")';?+*P1# "&/- F4"
6K&/-<:;?+
*!=   3H;?+*: x 320 D
&/-$08 "Hash driven Parallel Search 
HJ3F2</HDA*(Parallel A*)

TDS algorithm(Parallel IDA*)

TDS-df-UCT algorithm(Parallel MCTS)
4,800

hash driven

work stealing[Romein+ 1999] Fig. 4 (c)'"-(+.%.8=sequence alignment
A460 2400 ?41,000I;/0F3[Kishimoto+ 2012]synthesized game tree
(benchmark problem)

4,000
3,200
2,400
1,600

Go program(Fuego)800
0800 1600 2400 3200 4000 4800

[Yoshizoe+ 2011]C7 9>-!,"*@E &)#$5GD:  B1 2003 nicolas p. rougier (CC BY-SA 4.0)

22

virtual loss %"+DBC2  Virtual Lossrfcadge

UCT,*A=@ 3
<>5. 7EH  EH(F8 VirtualLoss $!& :1EH(F8;K6?G/ virtual loss bd F8EH(#'9D IJF8 d *<#2ln)+$$#2ln() + ,)+$+*$+*[-0)40 2007?]

virtual loss + .0UCT
(!12' (P-game) 3" ,$) 
0.1 millisecond rollout

14

3&#+100

Speed up

10Speed up$)/$) 
1.0 millisecond rollout

120122486

branch 8

4branch 40

2branch 150

0200

400

600

800

Number of Cores

Virtual Loss

1000

1200

8060

branch 840

branch 40
branch 150

20-.0UCT "%0200400600$)-*

800

Number of Cores

10001200

25+#!&)

$ 0rr

fcadgfebcadgeb(vanilla) UCT

Depth First UCT

%3 root 4
 /

%-,' [(, 1, , *5 2007].2

"!$+ 0.
#( 

1#(

0.1 millisecond rollout

350

26

1.0 millisecond rollout

800
700

300

600

200
150

branch 8100

branch 40

50

branch 150

2')

-

2
')

500

Speed upSpeed up

250

400
300

branch 8200

branch 40

100

branch 150

0200

400

600

800

Number of Cores10001200

0200/*&,-  "#(%400600800

Number of Cores

10001200

TDS + depth first UCT = TDS-df-UCT[Yoshizoe, Kishimoto, Kaneko, Yoshimoto, Ishikawa 2011]

EMMCTS: LRSpeedup• P-Game (5%3)

3,000

close to 19x19 Go
1.0 ms playout
branch 150

2,500
2,000
1,500

500
0800

– 6/>DS• EMLR

1600 2400 3200 4000 4800

2' - 48'

Hardware: TSUBAME2
CPU: Xeon 2.93GHz x 2 (12 cores)
Network: Infiniband QDR x 2
Library: MVAPICH2 library

QK- 9K9K" rollout =• A)2 ?CNumber of Cores
12 cores3– MP-Fuego (FuegoEMN)

close to 9x9 Go
0.1 ms playout
branch 40y=x

1,000

/8600 cores (50 nodes)– rollout #$– &1A)
–G

9.J:7
AlphaGo HOEM*F+PI,(04
40CPU+8GPU ! 1202CPU+176GPU
;DN'<75%@<BMCTS • MCTS !)$') (rollout, playout) _K

28kD

– 412_K3F:• QH@ UCT (9R2bZ'"% Multi-Armed Bandit – ?`JPj8M

• IA=IA– d*EJ-g/i (], AlphaGoVW)• >l<c-5;BX
– kD_KOT• ah7

Y.+,^0S– Hash driven parallel search  ah7UN–6e\– LfG:UNahMCTS&#&'6C[