Road toWinning at Horse Racingwith Data Science
2018/06/27(Wed)AlphaImpact
NUKUI Shun

How to win?

2

Profile●●●●●

Name: NUKUI ShunSpeciality: Machine Learning
Experience of horse racing: 11 years
Bio○ The president of horse racing club in Tokyo Tech ~2017/03○ Participated in 電脳賞(春) 2016/03○ AlphaImpact (development of horse racing AI) 2016/06~
Favorites: Watching training progress of LightGBM, Kaggle(Home Credit Default Risk)

3

AlphaImpact Project●●Developing horse racing AIMembers○○○●

NUKUI Shun : Machine Learning, Horse racing domain knowledge
OMOTO Tsukasa : Machine Learning, System Architect, One of committers of LightGBM
HARA Tomonori : Horse racing hacker

Activities○○○

HP: https://alphaimpact.jp/
Published AI scores for free (~2018/06/24)
Sell predictions on netkeiba ウマい馬券■ http://yoso.netkeiba.com/

4Road to winning
1.
2.
3.
4.
5.

Introduction to the system of horse racing
Definition of objective
Feature engineering
Prediction model
Evaluation

5

Road to winning
1.
2.
3.
4.
5.

Introduction to the system of horse racing
Definition of objective
Feature engineering
Prediction modelEvaluation

6

What is the Purpose of Horse Racing Prediction?
●Hit?

7

What is the Purpose of Horse Racing Prediction?
●Hit?

● MAKE A PROFIT!!

8

Which should we bet on?

win proba. (勝率)

1

2

370%

20%

10%

best choice?

9

Which should we bet on?
1

2

3

win proba. (勝率)

70%

20%

10%

odds (オッズ)

x1.1

x5.2

x8.0

exp. (期待値)

1.1 * 0.70 = 0.77

5.2 * 0.20 = 1.04

8.0 * 0.10 = 0.8

best choice!!

10

House Edge (控除率)House edge

total sales
Refund as payoff

70%~80% = mean of return ratio

11

Betting Typehouse edge

●Win (単勝)20.0%●Place (複勝)●Bracket Quinella (枠連)●Quinella (馬連)●Quinella Place (ワイド)●Exacta (馬単)●Trio (3連複)●Trifecta (3連単)

22.5%

25.0%

27.5%
12

Not Impossible to Win●●●

Mr. 卍 (Manji) made a profit of 140 million in a 3 years
His theory and analytical skill is great, but we believe it is
not impossible to exceed him by machine learning
Not hitting tickets are acknowledged as expenses undercertain conditions

13

Road to winning
1.
2.
3.
4.
5.

Introduction to the system of horse racing
Definition of objective
Feature engineering
Prediction model
Evaluation

14

What should we solve?
●classification○●●

imbalance problemregression○

better choice○

important to design smoothed objective

ranking○seems to be a natural choice too○

but not better performance than regression

15

Objective of Regression●

Strength
○
○
○
○
○

●

place of order (着順)
standardized time (標準化走破タイム )standardized velocity (標準化平均速度 )prize (賞金)
speed index (スピード指数)

Profit○○place (=within top3) payoff (複勝払い戻し )■ 1st 120 yen < 3rd 540 yen
dark score (business secret)■ transform strength score to be high correlated with return ratio
16

Two types of Objective vs Hit Ratio
all turf races in 2017

over-popular

standardized velocity

dark score17

Two types of Objective vs Return Ratio
all turf races in 2017

standardized velocity

dark score

18

Road to winning
1.
2.
3.
4.
5.

Introduction to the system of horse racing
Definition of objective
Feature engineering
Prediction model
Evaluation

19

Feature Engineering in Horse Racing●Most important and most time-consuming part●

Necessary to collect data by ourselves, unlike Kaggle●

Difficult to handle complicated structured data

●Requires deep domain knowledge to horse racing

20

Horse Table (馬柱)race info

odds
horse attributejockeyrace histories
21

Many Types of Historyhorse
jockeytrainer

22

Automated Achievement Features●

We made more than 1500 achievement featureshorse
jockeytrainerownersire (父馬)…

track typecourselengthcourse×length
weatherfield condition
pace
…

count
win hit ratio
place hit ratio
win return ratioplace return ratio

subject

race condition

statisticrequire domain knowledge

23Smoothing Ratio Features●

Eliminate noise of unpopular subject×condition

unconfident

where α=0.1

count=2
hit ratio=0.5

count=100
hit ratio=0.4

hit ratio=0.2

ABaverage

hit ratio=0.254

hit ratio=0.399
24

Road to winning
1.
2.
3.
4.
5.

Introduction to the system of horse racing
Definition of objective
Feature engineering
Prediction model
Evaluation

25

LightGBM
●
●
●
●
●
●

One of the most popular frameworks in Kaggle
Very strong to structured dataNot affected by scales of features
Can handle the missing value
Robust to meaningless features
We have a committer of LightGBM

26

Training Architecture
hyperopt
update params
Raceshuffle

LightGBM

Race1

score
feedback

average

Race2
Race・・・

LightGBM

valid score

RaceN
Race

LightGBM
split into k-folds
by the race(not by the horse)

cross validation(the valid set is used forearly stopping)27

Notice of LightGBM●●

Early stopping is important to avoid overfitting
Dummied features are better than categorical ones○●

Sensitive to random_state○●

Depends on dataset
May result from subsample/colsample_bytree?

pred_contrib is useful for feature analysis

28

Feature Analysis with LightGBM# of features# of horses

feature contribution
matrix= predict(data, pred_contrib=True)

29

Feature Analysis with LightGBM

30

Road to winning
1.
2.
3.
4.
5.

Introduction to the system of horse racing
Definition of objective
Feature engineering
Prediction model
Evaluation

31

nDCG
●
●

Used in ranking evaluation
Higher relevant score (reli) should be positioned at higher rank
non-negative, “the higher, the better”

32The Relevant Score of nDCG●

inverse of order
○
○

●

prize
○
○

●

15000, 6000, 3800, 0, 0, 0, 0, ..., 0

place payoff○●

15000, 6000, 3800, 2300, 1500, 0, 0, ..., 0
consider only top 5

prize@3○●

1/1, 1/2, …, 1/N
consider the whole ranking

emphasize the dark horses

win betting share (単勝支持率)○

how close to popularity

33

The Relevant Score of nDCG
all turf races in 2017
strength model1 (standardized velocity)●●●strength model2 (standardized velocity)model1 is more accurate than model2
model2 is closer to win betting sharemodel1 is able to hit the horses that the public cannot predict

34

Evaluation of Top-N Box Betting
all turf races in 2017
storength model (standardized velocity)

profit model (dark score)

the lower hit,
the higher return35

Summary●●●●●

The purpose of horse racing prediction is making a profit
Design of objective is critical to performance
Feature engineering requires domain knowledge too
LightGBM is coolnDCG is useful for model evaluation

36

Future work●●

Calculate expectation with predicted probability and real-time oddsAuto buying system with investment strategies

37

Enjoy horse racing life!!

38