国立天文台将来シンポジウム / NAOJ Future Planning Symposium

Public version

10 November, 2021

Data Sciencein Astronomy

Feel free to ask questions. Comments are also welcome.

khattori
あ@20.
ism.ac.jp
Kohei Hattori (服部公平)

No community of Data Science !?

λ

3月17日
（水）
宇宙電波懇談会 (radio)

C

月日
D

G
会場
H

3月15日
（月）

E

宇宙線研究者会議(cosmic ray)高エネルギー宇宙物理連絡会

(Sun)

A
A
B
B
C
C
D

（水）
3月16日

D
E

（火）
H

太陽研究者連絡会

AB

objectsCD
3月18日 3月18日

理論天文学宇宙物理学懇談会 (theory)

E
F
F
G
G
H
H
A
A
B
B
C
C
D

（木）
D
E
3月17日
[ …… ]（木）E(Data Science
Community??)（水）
E
F

Method

1

月日

3月17日

G

光学赤外線天文連絡会 (optical / IR)

日本天文学会
春季年会
S. 活動銀河核
S. 活動銀河核
Z3.
計算宇宙惑星2021年
Z1.
会場

9

10

11

12

(光赤・重) 3 月
V2. 観測機器(光赤・重)
3月16日
期
日 D2021 年 3V2.
月 観測機器16 日（火）〜昼 19
休日（金）
み 3月15日（月）
ポス
BW.
コンパクト天体W.
2021年(X線
春季年会
場（火）
所
ン開催
11:40~13:00
Z3. 計算宇宙惑星
・γ線)(X線・γ線)
E オ ン ラ イV3.
観測機器
V3.A 観測機器
コンパクト天 体
電 話
090 - 4387 - 6893 ( 学会事務局 ） ＜使用期間B 2021W.年
3 月 15 日（FP2. 原始惑星系円盤
P2.C 原始惑星系円盤
S. 活動銀河核
銀
河
R.
C
R.
nenkai-committee@asj.or.jp ( 年会実行委員会） V2. 観測機器(光赤・重) 昼
E-Mail
3月16日 DF

ＶＬＢＩ懇談会 (radio)

(HEP)

A

F

F
G
G
H

9

星間現象
Q. 年会プログラム
重)
11 (光赤・12
V2.10観測機器

於

Q. 星間現象11:
E
V3. 観測機器(X線・γ線)
13V2.
P2. 原始惑星系円盤
昼 F 休14
み 15（火）

東京工業大学 ( オンライン開催 )G

Q. 星間現象A

Z3. 計算宇宙惑星

V3. 観測機器(X線・γ線)

記者会見
Z1.H天文データ科学
11:40~13:00

P2.
W.
体
銀原始惑/P1.星形成
河
R. コンパクト天

W.
R. Cコンパクト天銀 河 R. 銀 河 体

Z3. 計算宇宙惑星

Z3.
計算宇宙惑星体
W. コンパクト天

2021 年 3 月 16 日（火）〜 3 月 19 日（金）S.
観測機器(光赤・重)V2. 活動銀河核

Q. 星間現象昼休

Z3.
計算宇宙惑星W. B コンパクト天体
W. コンパクト天 体
3月17日

D

V3.
P1.

V2. 観測機器(光赤・重)

昼

(X線
・γ線)
V3. 観測機器(光赤・重)V2.E 活動銀河核
観測機器
み （水）S.

(X線・・
γ線)
重)
V2.
V3. 観測機器(光赤11:40~13:00
昼
休 み
Data Science Session
11:40~13:00
・γ線) I guess.)
教育・広報・他
Y.
V3.
観測機器
P2. time
原始惑/P1.星形成(1stin (X線
history,P2
星間現象
Q. . 原始惑星系円盤

F

P2. 原始惑/P1.星形成

(X線・・
γ線)
(光赤
重)
V2.
V3.G観測機器
Q. 星間現象

Q.
教育・広報・他
Y. 星間現象Y. 教育・広報・他

A

Z1. 天文データ科学

C

X. 銀河形成・進化

Q.
Y. D星間現象
教育・広報・他
V1. 観測機器 (電波)3月18日

Z1. 天文データ科学
W.
コンパクト天 体（木）E

T. 銀河団F

P1. 星 形 成Z1. 天文データ科学

Instrument
Sessions
Z3.
計算宇宙惑星W. コンパクト天体(long
銀河形成・進化X. 日history)
本 天 文 学 会

M. 太 陽Z1.
N . G天文データ科学
恒星進化
H

W.
体
銀河形成・進化X. コンパクト天

ポス

V3.
P1. 観測機器
星 形 成 (X線・γ線) Y.
H

W. コンパクト天 体
P2
Q. .B 原始惑星系円盤
星間現象

Z1. 天文データ科学

Q.

11:

Z1.
昼

N.

11:

X.

W.
体
X. Aコンパクト天
銀河形成・進化Z2. ngVLA の天文学

(電波) (電波)
(電波)
河
R.
河
R.
観測機器
V1. 銀
観測機器
V1. 銀
昼 休 み
観測機器
V1.
V1.
昼 休 み
ポス

V2.
T. 観測機器
銀河団 (光赤・重)

T.
V3.
星銀河団
形 成 (X線・γ線)
P1. 観測機器
P1. 星 形 成P2.
太陽
M. 原始惑/P1.星形成

Q. 星間現象休

N . 恒星進化

C

X. 銀河形成・進化

V2.
. 観測機器宇宙論 (光赤・重)み （金）U

11:40~13:00 3月19日

昼

B

11:40~13:00

D

V1. 観測機器 (電波)

昼

11:

V3.
P1. 観測機器
星 形 成 (X線・γ線) U .

E
11:40~13:00
FG

M. 太 陽

P1.

P1.
M. H星
太形
陽成P3. 惑 星 系
3月20日
（土）

Q.
惑星系
P3. 星間現象
9

10

(代議

11

12

Institute of Statistical Mathematics (ISM = 統計数理研究所)
三鷹

立川

My oﬃce- Masato Shirasaki- Kohei Hattori (myself)
We are trying to connect
astronomers and statisticians.

ISM webpage — https://www.ism.ac.jp

An ideal workplace for astro data scientists.
Astronomers can use various resources.

・共同研究スタートアップ
Data analysis consulting. (First consulting is free.)Any questions from astronomers are welcome!

・Seminars- 天文観測におけるビッグデータ解析と宇宙論パラメータの推定(hosted by M. Shirasaki)- 統計物理と統計科学のセミナー (hosted by ISM people)

・受託研究員制度(Need to pay. Visiting scholar / visiting PhD student)(I started mentoring an astro PhD student.)

https://www.ism.ac.jp/shikoin/startup/index.html
https://www.ism.ac.jp/shikoin/overview/index.html

Most importantly, people at ISM
are supportive, andare HIGHLY interested in ASTRONOMY data.

Trends in ADS papers
ADS papers with “Bayesian” in the abstract.
ADS papers with “Milky Way” in the abstract.
ADS papers with “neural net” in the abstract.

Exponential growth. Time scale ~7 years

Unfamiliar with Bayes? See, e.g.,[1] Data analysis recipes: Fitting a model to data
https://arxiv.org/abs/1008.4686[2] Data analysis recipes: Probability calculus for inference
https://arxiv.org/abs/1205.4446

Bayesian
papersTrends in ADS papers
ADS papers with “Bayesian” in the abstract.
ADS papers with “Milky Way” in the abstract.
ADS papers with “neural net” in the abstract.

Milky Way papers

Bayesian
papersExponential growth. Time scale ~7 years

Unfamiliar with Bayes? See, e.g.,[1] Data analysis recipes: Fitting a model to data
https://arxiv.org/abs/1008.4686[2] Data analysis recipes: Probability calculus for inference
https://arxiv.org/abs/1205.4446

Trends in ADS papers
ADS papers with “Bayesian” in the abstract.
ADS papers with “Milky Way” in the abstract.
ADS papers with “neural net” in the abstract.

Neural net
papers
Bayesian
papersExponential growth. Time scale ~7 years
Exponential growth.
After 2015, the time scale is ~2.5 years

Trends in ADS papers
ADS papers with “Bayesian” in the abstract.
ADS papers with “Milky Way” in the abstract.
ADS papers with “neural net” in the abstract.

Exponential growth.
After 2015, the time scale is ~2.5 years

Inflation of ML universe.
Neural net
papers
Bayesian
papersJapanese PhD course ~ 5 years
Are we ready to train PhD student
in this expansion of the ML world?

Trends in ADS papers
ADS papers with “Bayesian” in the abstract.
ADS papers with “Milky Way” in the abstract.
ADS papers with “neural net” in the abstract.
(1)

(2)

Neural net
papersExponential growth.
After 2015, the time scale is ~2.5 years(1) Back propagation method [誤差逆伝播法 ]Bayesian
papers(2) Deep learning (AlexNet) [深層学習]

Newtechniques

Speed-up of
scienceHistory of science
1. Empirical sciencein the last ~1000s of years
2. Theoretical sciencein the last ~100s of years
3. Computational sciencein the last several decades
4. Data explorationNowToday’s talk・(1) Era of big data
・(2) Dimensionality reduction
・(3) Sparsity
・(4) Bayesian analysis
・(5) Machine learning・(6) Neural network
・(7) Data challenge

Setting the stage

・(1) Era of big data
・(2) Dimensionality reduction
・(3) Sparsity
・(4) Bayesian analysis
・(5) Machine learning・(6) Neural network
・(7) Data challenge

Era of Big Data
Keywords for astro observations- wide-field- deep- high-resolution- high-dimensional (many bands)- time-domain- high-precision- large data volume (e.g., radio telescope)
Data are becoming larger / more complex.
Most data will never be “seen” by eye.
Need to automate
data acquisition / reduction / analysis… Data science challenge

Main part・(1) Era of big data
Trends instatistical mathematics

・(2) Dimensionality reduction
・(3) Sparsity
・(4) Bayesian analysis

Trends inmachine learning

Robustness / reliabilityof new methods

・(5) Machine learning・(6) Neural network
・(7) Data challenge

・(1) Era of big data
Trends instatistical mathematics

・(2) Dimensionality reduction
・(3) Sparsity
・(4) Bayesian analysis
・(5) Machine learning・(6) Neural network
・(7) Data challengeDimensionality reduction“特徴量”
N-dimensional “data” → low-dimensional “feature”
Simpler description“Manifold learning”
2D manifold in 3D-space

UMAP

2D representation
1 8
528x28 = 784 pixels… 784-dim data

7

4Techniques

2

3
9
6

0

- Self-Organizing Map(SOM)- Isomap- UMAP・(1) Era of big data
Trends instatistical mathematics

・(2) Dimensionality reduction
・(3) Sparsity
・(4) Bayesian analysis
・(5) Machine learning・(6) Neural network
・(7) Data challenge

−1Sparse Modeling−10Easy case:

x

1

0

M =3
1 Example: Polynomial Curve Fitting
1.1.

(# of model params) ≤ (# of data points)
1

N≤M

t

t
0
1

M =0

t

0

M =1

t
−1
0

0

−1

0

−1

x

1

0

−1 1.4 Plots of polynomials having various orders M , shown as red curves, fitte
FigureFigure 1.2.

What if …
(# of model params) > (# of data points)
0

1

x

1

0

M =3

1
t

0

0

−1

−1

0

In general, it fails (degenerate params).
But there is a hope, if the model is sparse.

xx(RMS) error defined by

t

N>M

7 1

1

0

1

!
ERMS = 2 E(w⋆ )/Nin which the division
by9 N allows us to compare differM =an equal footing, and the square root ensures that ERMS
scale (and in the same units) as the target variable t. Gr
test set RMS errors are shown, for various values of M ,set error is a measure of how well we are doing in predi
new data observations of x. We note from Figure 1.5 tha
relatively large values of the test set error, and this can be
the corresponding polynomials are rather inflexible and ar
the oscillations in the function sin(2 πx). Values of M ingive small values for the test set error, and these also give re1of the generating functionsin(2πx), as can be seen, for thx
Figure 1.4.

Figure 1.4 Plots of polynomials having various orders M , shown as red curves, fitted to the data set shown in
Figure 1.2.

(RMS) error defined by!
ERMS = 2 E(w⋆ )/N(1.3)

the division by N allows us to compare different sizes of data sets on
Tibshiraniinan which(1996)is measured on the sameequal footing, and the square root ensures that E
RMS

scale (and in the same units) as the target variable t. Graphs of the training andtest set RMS errors are shown, for various values of M , in Figure 1.5. The test
set error is a measure of how well we are doing in predicting the values of t for[ ]• Zeeman effect → magnetic field strength

Sparse Modeling foroptical
image
data
starlight
M81 group @ HI 21 cm"
VLA
Fourier transformationof a radio image
atomic hydrogen"
Visibility
21cm “spin-flip”
Spectral Lines
Model brightness (β)• discrete, low energy transitions of atoms and molecules
has N pixels.
• gas chemical composition; temperatures, densities
Doppler effectLines→ line-of-sight velocities•Spectralmolecular lines:
e.g. COv(U, V) =•Zeeman effect → magnetic field strength

Dirty
Beam
and
Dirty
Image
• discrete,
lowenergytransitions ofM81
atoms
and
molecules
opticalstarlight
group @ HI 21 cm"
incomplete sampling•••gas chemical composition; temperatures, densities
Doppler effect → line-of-sight velocities
Zeeman effect → magnetic field strength

VLA

10

atomic hydrogen"21cm “spin-flip”

βs(l,m)"
1
optical starlight
M81 group @ HI 21 cm"“dirtybeam”
VLA⋮
Often,
we
have
M
<
N.
v1
atomic hydrogen"⋮No
uniquesolutionforβ.
21cm “spin-flip”⋮ =molecular lines:⋮
e.g. COvM⋮
M data points forthe observed v(U,V)
βN

v)

10

molecular lines:
e.g. CO•

Zeeman effect → magnetic field strengthβ
1 @ HI 21 cm"
M81 group
VLA
⋮
⋮
=
⋮
⋮
βN

Sparseoptical
Modeling
starlight

v1⋮
vM

21cm “spin-flip”

atomic hydrogen"21cm “spin-flip”
molecular lines:
e.g. CO10

molecular lines:
e.g. COIf we assume that the image is sparse (many zeros in beta),
we can obtain a LASSO solution:
penaltyresidual12̂| βi |∥v − Fβ∥2 + λ
β = arg min
∑
[2
]
i
10

By adopting a large lambda, we obtain a sparse solution.
(Adequate lambda depends on the science case. Need cross-validation.)Sparse Modeling

Ground truth

We conducted simulations for two cases of the blackhole shadow of M 87: a ring-like shadow and a crescentshadow (both with a diameter of 20 µas). Here, in order
to emulate real observations, we assume that visibilities are

LASSO

Honma, Akiyama, Uemura, Ikeda (2014)

ps://academic.oup.com/pasj/article/66/5/95/2337725 by guest on 03 November 2021

Fig. 5. Simulated UV coverage of M 87 with six-station sub-mm VLBI
array of EHT. Here it is assumed that observations are conducted at anelevation larger than 20◦ at each station.

Sparse Modeling

To believe or not to believe; blind test is important.

M87’s black hole shadow
Visibility data

Data challenge (Mock analysis)
Ground truth

Reconstructed image
Reconstructed image

Katherine L. Bouman PhD thesis (2017)

EHT Collaboration, Akiyama et al. (2019)

Radio telescopes opened a new window
into SMBHs and possibly IMBHs (ALMA, SKA, ngVLA).

Sparse Modeling for time-series data

flux: y(t) =∑jKato & Uemura (2012)*

Aj exp(iωjt)

2| Aj |

least-square fit(fitting the noise !!)ωjRegular variable stars (e.g., Cepheids, RR Lyrae)
have only few frequencies with non-zero amplitude.
4

Bellinger, Wysocki & Kanbur| Aj |

LASSO

-0.5

LASSO in power-spectrum space.
>> Simplest
representation
light200curve.
Least Squares
200 observations of the
LASSO
observations

2

0.5

0.0

Bellinger, Wysocki, Kanbur (2016)

Successful LASSO for RR Lyrae-0.5

75 observations

LASSO

75 observations

ωj

0.0
0.5

Magnitude

Least Squares* Data taken from VSNET.

Sparse Modeling

森田耕一郎 教授
(1954 - 2012)

One of the first persons in astronomy
who envisioned the importance of sparse modeling in radio interferometry.

1.1 Observation Equation and Design Matrix

Sparse matrix … Tough
In general the example
function f will be non-linear. If an estimate p̂ for the unP

knowns p exists, the problem P can, however, be linearised around p̂ using
a Taylor expansion and then neglecting all terms of second order and higher.

JASMINE will stare at the MW center region.
In the simple one-dimensional case this gives
JASMINE will(3)
o = f (p) with p = p̂ + p̂X 1 d
- observe 1e5 stars =f (p) (p p̂)
(4)
n! dp
- take photo for 80 x=2000
star(Paparazzi!)
dtimes for 1eachdf (p̂) +f (p) (p p̂) +f (p) (p p̂) + . . .
(5)
dp
2 dp
- measure N=1e7 params d(including satellite attitude)
P
`
1

`

n

P
n `

n

p̂

n=0
P
`

2

P
`

P
2 `

p̂

= f`P (p̂) +dpf`P (p)

p̂

p̂

2

p̂

(6)

+ O( p̂2 )dmost of JASMINE,
If we are to make
the
f (p)
p̂
(7)
' ô +
dp
we need to solve
problemf (p̂) is the matrix
estimate for theinversion
observation o computed
fromwhere c = ôa= huge
P
`

`

p̂

`

P
`

`

10

7

10 × 10 matrix

the estimate parameters p̂.

10

10

`

p̂ is the update of the estimate unknown p̂.

In the full multi-dimensional case this kind of linearisation yields
0 1 0 1 0 @ P
1 0
1
@
@
P
P
f
· · · @p f1 · · · @p⇤ f1
o1
p1 p̂1ô1@p1 1
B .. C B .. C B ..
C..
.. C B.
.
.
.
.
.
B.C B.C B .
B
C
.
.
.
. C
.
B C B C B @ P
B
C
C
@
@
P
P
B o` C ' B ô` C + B f` · · ·
C Bp
C
f
·
·
·
f
p̂
`
`
@p
@p
@p
B C B C B `C BC⇤B .. C B .. C B ..
B
C
C
..
..
..
.
.
.
.
@.A @.A @ .
A
.
.
.
.
. A @
oL
o

ôL

'

ô

+

|

@
P
f
@pL L

···

@
@p

fLP · · ·
{z
DP

@
P
f
@p⇤ L

p̂

}

p⇤

10

(8)

7

p̂⇤

p̂

(9)

where c = ô = F P (p̂) are the estimates for the observations o computed fromthe estimate parameters p̂. p̂ are the updates of the estimate unknowns p̂.
The matrix DP holding the Jacobian derivatives of the functions f`P withrespect to the unknowns p is called the design matrix of the problem.

Although the design matrix is sparse, it will be tough to solve it quickly.
>> Need a Parallel Computing LSQR (Least Squares with QR-factorization) method?

・(1) Era of big data
Trends instatistical mathematics

・(2) Dimensionality reduction
・(3) Sparsity
・(4) Bayesian analysis
・(5) Machine learning・(6) Neural network
・(7) Data challengeTrends in ADS papers
ADS papers with “Bayesian” in the abstract.
ADS papers with “Milky Way” in the abstract.
ADS papers with “neural net” in the abstract.

Exponential growth. Time scale ~7 years

Unfamiliar with Bayes? See, e.g.,[1] Data analysis recipes: Fitting a model to data
https://arxiv.org/abs/1008.4686[2] Data analysis recipes: Probability calculus for inference
https://arxiv.org/abs/1205.4446

Bayesian
papersBayesian analysis
Only 3 topics (to save time):(1) MCMC in high-dimensional space is tricky,
especially if the posterior distribution is multi-modal.
>> Try nested sampling (Skilling 2004). [e.g., Hikage et al. 2019](2) Bottleneck in MCMC is the computational cost in the likelihood.
>> Simplify the likelihood function (e.g., interpolation) [e.g., Nishimichi et al. 2019]>> Reduce the eﬀective data size [e.g., Hattori et al. 2021](3) “Likelihood” is sometimes hard to define (e.g., likelihood of N-body model?)>> Try ABC “Approximate Bayesian Computation”= MCMC-like analysis for the summary statistics

・(1) Era of big data
・(2) Dimensionality reduction
・(3) Sparsity
・(4) Bayesian analysis
Trends inmachine learning・(5) Machine learning・(6) Neural network
・(7) Data challenge

Machine learning
Machine learning is the science of getting computers to act
without being explicitly programmed.
— Andrew Ng

A set of methods that can automatically detect patterns in data,
and then use the uncovered patterns to predict future data,or to perform other kinds of decision making under uncertainty.
— Kevin P. Murphy
Machine learning is the study of computer algorithms that can
improve automatically through experience and by the use of data.
— Wikipedia

Machine learning

3-body problem and deep neural networks

3

Supervised learning:

Based on the “input-output” pairs (test data),
find the function that maps input to output.

Classification

Parameter inference

Figure 2. Newton and the machine. Image of sir Isaac Newton alongside a schematic of a 10-layer deep neural network. In
each layer (apart from the input layer), a node takes the weighted
input from the previous layer’s nodes (plus a bias) and then applies an activation function before passing data to the next node.
The weights (and bias) are free parameters which are updated
during training.

Identification
Find strong
lensing images

Spectral analysis(Subaru PFS / GA)

Transients / Variables /counters, and computationof converged solutions in these
Flares / Micro-lensing/
SNesituations is costly1 (Boekholt et al 2019).

We used a feed-forward ANN consisting of 10 hidden
layers of 128 interconnected nodes (Fig. 2 and Appendix
B). Training was performed using the adaptive moment estimation optimization algorithm ADAM (20) with 10000
passes over the data, in which each epoch was separated into
batches of 5000, and setting the rectified linear unit (ReLU)
activation function to max(0, x) (Glorot, Bordes & Bengio,2011). By entering a time t and the initial location of particle x2 into the input layer, the ANN returns the locations of
the particles x1 and x2 at time t, thereby approximating thelatent analytical solution to the general three-body problem.
To assess performance of the trained ANN across a
range of time intervals, we partitioned the training and validation datasets into three segments: t . 3.9, t . 7.8 andt . 10 (which includes all data). For each scenario, we assessed the loss-function (taken as the mean absolute error

Radio galaxy types

PLASTiCC challenge

Figure 3. Mean Absolute Error (MAE) vs epoch. The
ANN has the same training structure in each time interval. Solids
lines are the loss on the training set and dashed are the loss on thevalidation set. T  3.9 corresponds to 1000 labels per simulation,
similarly T  7.8 to 2000 labels and T  10.0 to 2561 labels/timepoints (the entire dataset).The results illustrate a typical occurrence in ANN training, there is an initial phase of rapid learning,e.g. âL’š100 epochs, followed by a stage of much slower learningin which relative prediction gains are smaller with each epoch.

Photometric redshift

Model generation
Chaotic
3-body model… etc.

Machine learning
Unsupervised learning:
Clustering

Density estimation

16

anderson et al.

5l (deg)l (deg)Figure 1. Comparison of HDBSCAN outputs using di↵erent clustering methods and di↵erent cut-o↵ parallaxes. Left: Orion,
sources shown only up to ⇡ =2 mas. Right: Upper Sco and CrA, sources shown only up to ⇡ =5 mas. Both panels are shown in
Galactic coordinates. Di↵erent symbols indicate di↵erent products of di↵erent clustering runs. The structures they trace varydepending on the cut-o↵ parallax, including the persistence of various structures and their specific membership. Note the edgee↵ects at l = 0 in the runs shown in the right panel.

Find open clusters /Deriving
precise/Figure 4. The CMD prior, modeled as a Gaussian
mixture, inferredby running XDCMD
on allOver-density
ofleftgalaxiesof the data. The
panel shows a sampling of the prior, with the black lines showing the
1 and 2 contours. The right panel shows the 1contours of the3D
individualcomponents.
Deriving
dust
map
The latter are Gaussian in the transformed magnitude space to make the XD inference
tractable. Thus, they appear as slightly deformed ellipses in color–magnitude space.

b (deg)Anomaly detectionb (deg)b (deg)Find the pattern in the data without external information.

dataset and includes the 1 uncertainties on the posterior PDFs. Compared with theraw data shown in Figure 2, it is clear the precision of the posterior PDFs is greater,
especially for the red giant branch stars.

Pair-instability SNe /Hypervelocity stars /Gravitational lensing… etc.

Machine learning
Classical methods
andersonet al.
K-means
Support Vector Machine (SVM)
Random Forest
Gaussian Mixture Model (GMM)

Neural network

Clustering

Figure credit: https://en.wikipedia.org/wiki/DBSCAN

One of the fast / successful / easy-to-implement methods:
DBSCAN (Density-based spatial clustering of applications with noise)ϵϵ

One of the most widely used clustering method.
Finding data points within a given radius.
Circles which encloses less than N_min are ignored.
# of clusters is automatically determined.

N_min = 4.
Only red circles form the cluster.

Similar to “friend-of-friend” method in N-body simulation.

b (deg)b (deg)Clustering

Discovering open clusters with Gaia data
Stars in an open cluster have similar position and velocity (x, v)>> DBSCAN can discover open clusters with Gaia’s astrometric data.
>> There are ~2000 open clusters within 1 kpc from the Sun.
l (deg)l (deg)l (deg)Figure 2. Spatial distribution of sources in the final catalog in Galactic coordinates. Sources in common with Cantat-Gaudin
et al. (2018) are shown in yellow.

b (deg)b (deg)b (deg)Figure 1. Comparison of HDBSCAN outputs using di↵erent clustering methods and di↵erent cut-o↵ parallaxes. Left: Orion,
sources shown only up to ⇡ =2 mas. Right: Upper Sco and CrA, sources shown only up to ⇡ =5 mas. Both panels are shown in
Galactic coordinates. Di↵erent symbols indicate di↵erent products of di↵erent clustering runs. The structures they trace varydepending on the cut-o↵ parallax, including the persistence of various structures and their specific membership. Note the edgee↵ects at l = 0 in the runs shown in the right panel.

l (deg)Kounkel & Covey (2019)

Figure 1. Comparison of HDBSCAN outputs using di↵erent clustering methods ansources shown only up to ⇡ =2 mas. Right: Upper Sco and CrA, sources shown only
Galactic coordinates. Di↵erent symbols indicate di↵erent products of di↵erent cluste
depending on the cut-o↵ parallax, including the persistence of various structures andDensity estimation
improving gaia parallax precision
13
Color-magnitude diagram (CMD) of nearby stars in Gaia (DR1) data.

Raw data… Blurred CMD due to distance error

16

anderson et al.

Figure 2. The observed CMD: To visualize the full dataset, in the left panel we use thepoint estimates of the 2MASS J Ks color, the J band apparent magnitude, and the TGAS
parallax. The grey lines represent the 1 and 2 contours for the distribution. To give asense of the uncertainties, in the right panel we subsample the dataset and include the1 uncertainties. The uncertainties are dominated by the parallax noise, with many stars
diverging to infinitely far away and therefore very intrinsically bright.

with f ($true ) being a window function to insure $true is positive and to put $true
a similar grid for all the posteriors.

Reconstructed CMD… Gaussian Mixture model
after
deconvolution
of
error
on

4. DATA AND RESULTS

We use stars crossmatched in TGAS and 2MASS. The match was done using anearest-neighbor algorithm with a search radius of 4 arcsec2 . We also required thatthe stars lie within the observing footprint of PS1 to access the Green et al. (2015)

Anderson et al. (2017)[see also Leistedt et al. 2017]

・(1) Era of big data
・(2) Dimensionality reduction
・(3) Sparsity
・(4) Bayesian analysis
Trends inmachine learning・(5) Machine learning・(6) Neural network
・(7) Data challenge
23Machine learning
Classical methods

Neural network

Artificial Neural Network
Convolutional Neural Network
Auto-encoder
Generative Adversarial Network (GAN*)* Masato Shirasaki is an expert of GAN

Neural network
weighted sum of
input nodes

y=f

(

b+

∑
i

)

wi xi

f: activation function

input
layer

N

ℝ

hiddenlayersNMhidden layers = g : ℝ → ℝ
outputlayer
Mℝ

Many parameters (weights)
allow large flexibility.
We find the optimal weights by training.

If N>M, this can be seen as
a dimensionality reduction.

Figure credit: https://www.knime.com/blog/a-friendly-introduction-to-deep-neural-networks

hs, followed by a stage of much slower learning
prediction gains are smaller with each epoch.

Model generation

Newton vs the machine: solving the chaotic three-bodyproblem using deep neural networks

3-body problem in math.

x1(0)
x1(T)
x2(0)
x2(T)
dation of the trained ANN. Presented arex (0)m the training set (left) and two from the vali-3. All examplesx
wererandomly chosen from their3(T)llets indicate the initial conditions.
The curves
=
f
(0)
v
1
its of the three bodies (red, blue and green, the(T)vrom symmetry). 1The solution from the trained
urves) is hardly distinguishable from the con(0)v2(dashes, acquired
using Brutus (Boekholt &(T)v2015)). The two 2
scenarios presented to the rightd in the training dataset.
v3(0)
v3(T)
rticle trajectories reflect a series of complex T

v:1910.07291v1 [astro-ph.GA] 16 Oct 2019

Philip G. Breen1? †, Christopher N. Foley2 ? ‡, Tjarda Boekholt3
4
and
Simon
Portegies
Zwart
1

School of Mathematics and Maxwell Institute for Mathematical Sciences, University of Edinburgh, Kings Buildings, Edinburgh
2 MRC Biostatistics Unit, University of Cambridge, Cambridge, CB2 0SR, UK.
3 Instituto de Telecomunicações, Campus Universitário de Santiago, 3810-193, Aveiro, Portugal
4 Leiden Observatory, Leiden University, PO Box 9513, 2300 RA, Leiden, The Netherlands.

Accepted XXX. Received YYY; in original form ZZZABSTRACT

Since its formulation by Sir Isaac Newton, the problem of solving the equatio
motion for three bodies under their own gravitational force has remained pract
unsolved. Currently, the solution for a given initialization can only be found byforming laborious iterative calculations that have unpredictable and potentially incomputational cost, due to the system’s chaotic nature. We show that an ens
of solutions obtained using an arbitrarily precise numerical integrator can be us
train a deep artificial neural network (ANN) that, over a bounded time intervalvides accurate solutions at fixed computational cost and up to 100 million times
than a state-of-the-art solver. Our results provide evidence that, for computa
ally challenging regions of phase-space, a trained ANN can replace existing num
solvers, enabling fast and scalable simulations of many-body systems to shedon outstanding phenomena such as the formation of black-hole binary systems oorigin of the core collapse in dense star clusters.

Breen et al. (2020)

Figure 2.
Newtonand Key
thewords:
machine.
Imageof sir methods:
Isaac New(step1) Solve
3-bodyproblemstars:
kinematics
and dynamics,
numerical, statistical
ton alongside a schematicof a 10-layer
neural network. In
with various
initial deep
conditions.
each layer (apart from the input layer), a node takes the weighted
d the trained ANN reproduced these satis(step 2) Train the NN.
1 INTRODUCTION
The focus of the present study is therefore the timely
 0.1). The ANN also closely matched the
input
from
the
previous
layer’s
nodes
(plus
a
bias)
and
then
apputationofaccurate
solutionstothe three-body prob
haviour of the converged solutions in all the
Newton’sequationsofmotionsolves
describe the evolutionof many problem
(step
3)
NN
3-bodyDespiteits agenext
and interest
from numerous dOperatorofplies anbodies
activation
beforepassing data
to
the
node.
were not included in its training.
Moreover,in space underfunction
the influence of theirown gravitational
guished scientists (de Lagrange 1772; Heggie 1975;
H
Fi
force (Newton 1687).with
The equations
haveacentralrole
in
is in fixed computational time (t ⇠ 10 3 secany
initialconditionquickly.
Bahcall
1983;
Montgomery
1998; Stone & Leigh 2019
The
weights
(and
bias)
are
free
parameters
which
are
updated5
manyclassical
problemsinPhysics.
For
example,theequaon average
about 10 (andequation
sometimes even“solvingof motion”
problem of solving the equations of motion for three-bA
tions explain the dynamical evolution of globular star clusduring training.
r than Brutus.

CNN — Convolutional Neural Networkedge-detecting filter

raw image

-1

-1

-1

-1

-1

Convolution

-1

1.78

1.78

1.78

-1

＊

-1

1.78

1.78

1.78

-1

-1

1.78

1.78

1.78

-1

-1

-1

-1

-1

-1

CNN — Convolutional Neural Networkedge-detecting filter

raw image

-1

-1

-1

-1

-1

Convolution

-1

1.78

1.78

1.78

-1

＊

-1

1.78

1.78

1.78

-1

-1

1.78

1.78

1.78

-1

-1

-1

-1

-1

-1

Smooth components disappear.

CNN — Convolutional Neural Networkedge-detecting filter

raw image

-1

-1

-1

-1

-1

Convolution

-1

1.78

1.78

1.78

-1

＊

-1

1.78

1.78

1.78

-1

-1

1.78

1.78

1.78

-1

-1

-1

-1

-1

-1

Smooth components disappear.

Convolutions extract the local pattern (e.g., “edge”) in the data.
The extracted information is translation invariant.

CNN — Convolutional Neural Networkedge-detecting filter

raw image

-1

-1

-1

-1

-1

Convolution

-1

1.78

1.78

1.78

-1

＊

-1

1.78

1.78

1.78

-1

-1

1.78

1.78

1.78

-1

-1

-1

-1

-1

-1

Smooth components disappear.

Convolutions extract the local pattern (e.g., “edge”) in the data.
The extracted information is translation invariant.

＊

-1

-1

-1

-1

-1

-1

1.78

1.78

1.78

-1

-1

1.78

1.78

1.78

-1

-1

1.78

1.78

1.78

-1

CNN — Convolutional Neural Network

input
image
(c) いらすとや

AlexNet 2012

Krizhevsky et. al. (2012)
Hassan & Gousev (2019)

output
array

0.01
0.00
⋮
P(dog)
0.94 = 0.94
0.02

CNN — Convolutional Neural Network

AlexNet 2012

Krizhevsky et. al. (2012)
Hassan & Gousev (2019)

output
array

input
image
(c) いらすとや

0.01
0.00
⋮
P(dog)
0.94 = 0.94
0.02

Convolution = Extract “local patterns” that are translation invariant.
Various filters are used = Various patterns are extracted.
-1

-1

-1

-1

0

1

-1

-1

-1

-1

8

-1

-1

0

1

0

0

0

-1

-1

-1

-1

0

1

1

1

1

(original work) input = 2D image … (N x N x 3) array for RGB color image(astro works) input = 2D image … (N x N x 1) array for gray-scale image(astro works) input = 1D vector … 1D spectra

CNN — Convolutional Neural Network

AlexNet 2012
Spin Parity of Spiral Galaxies II

Krizhevsky et. al. (2012)
Hassan & Gousev (2019)9output
array

input
image

0.01
0.00
⋮
P(spiral)
0.94 = 0.94
0.02

Convolution = Extract “local patterns” that are translation invariant.
Various filters are used = Various patterns are extracted.
-1

-1

-1

-1

0

1

-1

-1

-1

-1

8

-1

-1

0

1

0

0

0

-1

-1

-1

-1

0

1

1

1

1

(original work) input = 2D image … (N x N x 3) array for RGB color image(astro works) input = 2D image … (N x N x 1) array for gray-scale image(astro works) input = 1D vector … 1D spectra

CNN — Convolutional Neural Network
Majewski et al.

65Normalized flux + offset

The APOGEE field targeting strategy was designed
around several motivations and requirements:

Metallicity Sequence4

Teff=4686 K

log(g)=2.39[Fe/H]=+0.39

Teff=4636 K

log(g)=2.38[Fe/H]=+0.19

Teff=4633 K

log(g)=2.41[Fe/H]=+0.01

Teff=4664 K

log(g)=2.38[Fe/H]=−0.20• A desire to sample, with minimal bias, all stellar
populations of the Galaxy, from the bulge, acrossthe disk, and into the halo.

log(g)=2.42[Fe/H]=−0.40

Teff=4576 K

log(g)=2.32[Fe/H]=−0.60

Teff=4667 Klog(g)=2.43[Fe/H]=−0.88

Teff=4664 K

log(g)=2.31[Fe/H]=−1.20

Teff=4895 K

log(g)=2.23[Fe/H]=−1.82

Metallicity

2D image (galaxy)
Teff=4666 K

3

2

1
Fe I

15200

Fe I

Si I

Si I Fe I

Mg I

15400

15600

15800

Si I

Si I

Si I

Fe I

16000
16200
Wavelength [Å]Si I + Fe I Fe I

16400

Si I Al I

16600

19

CI

16800

Fig. 8.— An image showing continuum normalized APOGEE
spectra as a function of metallicity for giant stars of similar temperature. Some of the strongest metal lines seen are identified at
the bottom of the figure.

• The need to probe fields to a variety of magnitude
limits to access stars over a wide range of distancein all parts of the Galaxy.
• The ability to calibrate efficiently against stars withwell-established physical properties, such as thechemical abundances and radial velocities that areoften well established for star cluster members, orthe masses and gravities that can be derived forasteroseismology targets.

CNN

Spin Parity of Spiral Galaxies IIclassification

9

Tadaki et al. (2020)

galaxymorphology• The need to coordinate with the other SDSS-III
bright time program, MARVELS, which relied on
frequent visits to a relatively limited number offields.

1D stellar spectrum (stellar physics)

In the end, changes in the latter two requirements as wellas the realities of the actual distribution of clear weather
and several other considerations led to the evolution of
the APOGEE target selection over the three year observing campaign.
4.1.2. Field Selection Evolution

inference

Leung & Bovy (2019)

Initial Survey Design: For its expansion into brighttime observing the SDSS-III collaboration planned tocapitalize on the existence of two new fiber-fed instruments that could operate simultaneously from shared
plugplates, thereby doubling the e↵ectiveness of the
Sloan Telescope. Because the MARVELS project required many visits to each of its target fields, whereas
Fig. 9.— Comparison of a section of the APOGEE spectra for
APOGEE had always planned at least some deep field
two stars of the same temperature (approximately 4060 K) withprobes, the original SDSS-III plan was for 75% of theabout a 100⇥ ratio in abundance of iron. The red spectrum is for abright time to be in co-observing mode, whereas the restar that has [Fe/H] = -1.8 and log g = 0.158. The black spectrum
is for a star that has [Fe/H] = 0.365 and log g = 1.5.
maining 25% of bright time would be given to APOGEE
to observe fields of no interest to MARVELS and to fill
Examples of the appearance of stellar spectra as obout its sky coverage. Moreover, because MARVELS tartained by the APOGEE spectrograph are shown in Figgets were relatively bright, relatively nearby stars, bothures 6-9. Figure 6 shows stars ranging from spectral typesurveys could make good use of many visits to fields at
O to M; the primary APOGEE science targets are ofhigh latitude (in APOGEE’s case, for accumulating sigtype G and K, whereas most of the early spectral types
nal on faint, distant halo stars) as well as in the disk
were observed as telluric standards and some M types(where APOGEE could both accumulate flux on highly
are selected by the random sampling of the parent disdust-extinguished stars across the disk as well as cycle
tribution (§4.2). Across the temperature range of thethrough large numbers of brighter stars).
primary survey target types (G-K stars), it is still possiThe baseline for co-observed fields was to accumulate
ble to discern line strength variations, as shown in Figurea total of 24, approximately one hour visits. Under these
7. A primary driver of the APOGEE project is the explooverriding restrictions, the initial APOGEE field selecration of chemical abundance variations among its latetion plan focused on fulfilling the other principles detype stellar sample; Figure 8 demonstrates the appearscribed in §4.1.1. The 75% shared survey time was disance of RGB stars of similar temperature but a 2.2 dex
tributed in a series of 24- and 12-visit fields across themetallicity spread. To show greater detail and a broader
disk and halo (the latter used for fields that MARVELS
array of chemical species, Figure 9 highlights the blue
began observing before APOGEE came on line). The
array spectra for two giant stars separated by about 2.2
disk plan included, a regular “picket fence” Galactic londex in [Fe/H].
distribution of these deep fields, and with multiple
Figure 8. Examples of HSC images ofgitude
S-spirals
and Z-spirals(right)withvisitsateach (left)
picket broken
up into a seriesof platede-the predicted probability of >0.95. They are randomly
3
4. SURVEY
DESIGNsignsgalaxies
that enableat
stars
of di↵erent(i.e., mean
= 0.2 −magnitudes
0.3.
selectedfrom spectroscopically-confirmed
zspec
distances) to be cycled through for di↵erent numbers of
4.1. Field Selection
total visits. The adopted distributions of Galactic lati4.1.1. Field Selection Principles
tude and cycling of stars were based on modeling stellar

Asteroseismic classification with convnets

chemicalabundances

4579

1D power spectrum (asteroseismology)

Hon et al. (2017)

1D light curve (dimming stars)

the Dipper

C.E.R acknowledges Anupreeta More for providing atool for visual inspection of images.

stellarparameters

identification

Tajiri et al. (2020)

Gehrels N., 1986, ApJ, 303, 336
Goldreich P., Lynden-Bell D., 1965, MNRAS, 130, 125
Goodfellow I., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley
Figure 2. (a) Appended and normalized folded spectrum of KIC 10790301
D., Ozair S., Courville A., Bengio Y., 2014, in Ghahramaniwith mode
identification.
s isN.
theD.,shapeparameter
Z., Welling
M., Cortes
C., Lawrence
Weinberger
K. Q.,or order (=2 for standard
eds, Gaussian),, Advances inwhile
Neural
Processing
Systems
27. super-Gaussian weight.
σ Informationis the standard
deviationof theREFERENCES
Figure 1. Comparisons
between an RGB star KIC 11293804 (top), with
Curran Associates, Inc., pp 2672–2680
The solid vertical line separates the original image from its appended copy.
S., Aniyan
A. K., Kembhavi
A. K.,
PhilipN. S., Vaghan HeB starAbraham
KIC 5810333(bottom),
both having
large
frequency
spacing Hart R. E., et al., 2017, MNRAS, 472, 2263

Downloaded fr

a support vector
dged to be true,
used thundersvm
tor analysis. Usr neural networkr a negative deight curves. Thenary candidates.

inference

Find a specific
type of stars

CNN — Convolutional Neural Network
Majewski et al.

65Normalized flux + offset

The APOGEE field targeting strategy was designed
around several motivations and requirements:

Metallicity Sequence4

Teff=4686 K

log(g)=2.39[Fe/H]=+0.39

Teff=4636 K

log(g)=2.38[Fe/H]=+0.19

Teff=4633 K

log(g)=2.41[Fe/H]=+0.01

Teff=4664 K

log(g)=2.38[Fe/H]=−0.20• A desire to sample, with minimal bias, all stellar
populations of the Galaxy, from the bulge, acrossthe disk, and into the halo.

log(g)=2.42[Fe/H]=−0.40

Teff=4576 K

log(g)=2.32[Fe/H]=−0.60

Teff=4667 Klog(g)=2.43[Fe/H]=−0.88

Teff=4664 K

log(g)=2.31[Fe/H]=−1.20

Teff=4895 K

log(g)=2.23[Fe/H]=−1.82

Metallicity

2D image (galaxy)
Teff=4666 K

3

2

1
Fe I

15200

Fe I

Si I

Si I Fe I

Mg I

15400

15600

15800

Si I

Si I

Si I

Fe I

16000
16200
Wavelength [Å]Si I + Fe I Fe I

16400

Si I Al I

16600

19

CI

16800

Fig. 8.— An image showing continuum normalized APOGEE
spectra as a function of metallicity for giant stars of similar temperature. Some of the strongest metal lines seen are identified at
the bottom of the figure.

• The need to probe fields to a variety of magnitude
limits to access stars over a wide range of distancein all parts of the Galaxy.

CNN

Spin Parity of Spiral Galaxies II

9• The ability to calibrate efficiently against stars withwell-established physical properties, such as thechemical abundances and radial velocities that areoften well established for star cluster members, orthe masses and gravities that can be derived forasteroseismology targets.
• The need to coordinate with the other SDSS-III
bright time program, MARVELS, which relied on
frequent visits to a relatively limited number offields.

1D stellar spectrum (stellar physics)

In the end, changes in the latter two requirements as wellas the realities of the actual distribution of clear weather
and several other considerations led to the evolution of
the APOGEE target selection over the three year observing campaign.
4.1.2. Field Selection Evolution

Why CNN works for various data?
— “Translational invariance”
Leung & Bovy (2019)

Initial Survey Design: For its expansion into brighttime observing the SDSS-III collaboration planned tocapitalize on the existence of two new fiber-fed instruments that could operate simultaneously from shared
plugplates, thereby doubling the e↵ectiveness of the
Sloan Telescope. Because the MARVELS project required many visits to each of its target fields, whereas
Fig. 9.— Comparison of a section of the APOGEE spectra for
APOGEE had always planned at least some deep field
two stars of the same temperature (approximately 4060 K) withprobes, the original SDSS-III plan was for 75% of theabout a 100⇥ ratio in abundance of iron. The red spectrum is for abright time to be in co-observing mode, whereas the restar that has [Fe/H] = -1.8 and log g = 0.158. The black spectrum
is for a star that has [Fe/H] = 0.365 and log g = 1.5.
maining 25% of bright time would be given to APOGEE
to observe fields of no interest to MARVELS and to fill
Examples of the appearance of stellar spectra as obout its sky coverage. Moreover, because MARVELS tartained by the APOGEE spectrograph are shown in Figgets were relatively bright, relatively nearby stars, bothures 6-9. Figure 6 shows stars ranging from spectral typesurveys could make good use of many visits to fields at
O to M; the primary APOGEE science targets are ofhigh latitude (in APOGEE’s case, for accumulating sigtype G and K, whereas most of the early spectral types
nal on faint, distant halo stars) as well as in the disk
were observed as telluric standards and some M types(where APOGEE could both accumulate flux on highly
are selected by the random sampling of the parent disdust-extinguished stars across the disk as well as cycle
tribution (§4.2). Across the temperature range of thethrough large numbers of brighter stars).
primary survey target types (G-K stars), it is still possiThe baseline for co-observed fields was to accumulate
ble to discern line strength variations, as shown in Figurea total of 24, approximately one hour visits. Under these
7. A primary driver of the APOGEE project is the explooverriding restrictions, the initial APOGEE field selecration of chemical abundance variations among its latetion plan focused on fulfilling the other principles detype stellar sample; Figure 8 demonstrates the appearscribed in §4.1.1. The 75% shared survey time was disance of RGB stars of similar temperature but a 2.2 dex
tributed in a series of 24- and 12-visit fields across themetallicity spread. To show greater detail and a broader
disk and halo (the latter used for fields that MARVELS
array of chemical species, Figure 9 highlights the blue
began observing before APOGEE came on line). The
array spectra for two giant stars separated by about 2.2
disk plan included, a regular “picket fence” Galactic londex in [Fe/H].
distribution of these deep fields, and with multiple
Figure 8. Examples of HSC images ofgitude
S-spirals
and Z-spirals(right)withvisitsateach (left)
picket broken
up into a seriesof platede-the predicted probability of >0.95. They are randomly
3
4. SURVEY
DESIGNsignsgalaxies
that enableat
stars
of di↵erent(i.e., mean
= 0.2 −magnitudes
0.3.
selectedfrom spectroscopically-confirmed
zspec
distances) to be cycled through for di↵erent numbers of
4.1. Field Selection
total visits. The adopted distributions of Galactic lati4.1.1. Field Selection Principles
tude and cycling of stars were based on modeling stellar

2D image: Freedom to choose the origin

Asteroseismic classification with convnets

1D power spectrum (asteroseismology)

4579

1D stellar spectrum: Doppler shift
Hon
et
al.
(2017)
1D power spectrum: Δν

1D light curve: Freedom to choose the origin

1D light curve (dimming stars)

the Dipper

C.E.R acknowledges Anupreeta More for providing atool for visual inspection of images.

Gehrels N., 1986, ApJ, 303, 336
Goldreich P., Lynden-Bell D., 1965, MNRAS, 130, 125
Goodfellow I., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley
Figure 2. (a) Appended and normalized folded spectrum of KIC 10790301
D., Ozair S., Courville A., Bengio Y., 2014, in Ghahramaniwith mode
identification.
s isN.
theD.,shapeparameter
Z., Welling
M., Cortes
C., Lawrence
Weinberger
K. Q.,or order (=2 for standard
eds, Gaussian),, Advances inwhile
Neural
Processing
Systems
27. super-Gaussian weight.
σ Informationis the standard
deviationof theREFERENCES
Figure 1. Comparisons
between an RGB star KIC 11293804 (top), with
Curran Associates, Inc., pp 2672–2680
The solid vertical line separates the original image from its appended copy.
S., Aniyan
A. K., Kembhavi
A. K.,
PhilipN. S., Vaghan HeB starAbraham
KIC 5810333(bottom),
both having
large
frequency
spacing Hart R. E., et al., 2017, MNRAS, 472, 2263

“Inflation of ML universe”
Downloaded fr

a support vector
dged to be true,
used thundersvm
tor analysis. Usr neural networkr a negative deight curves. Thenary candidates.

CNN is suited for natural science.
— Reason forSome concerns about neural network
NN is sometimes described as a “black box.”
・Bias in the test data
・Interpretability of NN — “Explainable AI” (XAI)・Uncertainty quantification — Bayesian NN
・High degrees of freedom — No unique solution.

Bias in the test data
ExampleMachineMachine learning on stellar
spectralearning
2987on stellar spectra

2987

Fabbro et al. (2018)

Figure 1. The current StarNet CNN model composed of seven layers. The first layer is solely the input data; followed by two convolutional layers with 4 an
16 filters (in successive order), then a max pooling layer with a window length of four units, followed by three fully connected layers with 256, 128, andnodes (again, in successive order). The final layer is the output layer.

(1) Traina NN
to learn
the stellar
relationship
Figure
11. Partial
derivativesof the three
output parameters from the StarNet model – trained on APOGEE spectra – with respect to input wavelengthbins
for a section[X/H]of the green
The partial
derivativesof stars fromdifferent ranges of the parameter space were compared against each other. Stars with
betweenandchip.
spectrausing
APOGEE
spectranetwork parameters), the stellar parameters can be predicted forconvolutional layers. Fully connected layers are the classical NN[Fe/H] > 0.0 were compared to those with [Fe/H] < −1.2 (top). Similarly,
stars
with
5000 Kwewere
to thosewithmultiple
Teff <
4300
K (bottom).
the rest of thesample.
The T
stellarconsidercompared
for thislayersthat computelinearcombinationsof all of theAninpu
eff >parametersspectral analysis are the effective temperature (Teff ), surface gravity
values to produce an output vector. In the case of a convolutiona
Machineonstellarspectra2987
average absolute-valued Jacobian was calculated from2000 stars learningin eachparameter
range.
Note
the
scale
differences
when comparing the partial derivatives.
(log g), and metallicity ([Fe/H]).
layer, a series of filters are applied, extracting local information from

Result:NNs prioritize diﬀerent
lines
to
infer
the previous layer.
Throughout
the training
phase[X/H],
the network learn
the filters that are activated most strongly when detecting specifi
because synthetic
spectraAPOGEE
features,thus producing a≠collectionof feature maps. Using two

In this paper, we present StarNet: a convolutional neural network(CNN) model applied to the analysis of stellar spectra. We introduce
our machine learning methods in Section 2 and evaluate our model
for a set of synthetic data in Section 3. As an exercise of its effectiveness, we apply StarNet to the APOGEE survey in Section 4 (DR13and earlier data releases when appropriate) and compare the stellar
parameters to those from the APOGEE pipeline(s). In Section 5,
we discuss the success of our StarNet results to other stellar analyses and confirm that NNs can significantly increase the robustness,
efficiency, and scientific impact of spectroscopic surveys.

successive convolutional layers, the second of the two convolve
across the previous layer’s feature map, which allows the model tolearn higher order features.
The combination of convolutional layers and fully connected lay
ers in our StarNet implementation means that the output parameterare not only affected by individual features in the input spectrum, bu
also, combinations of features found in different areas of the spec
trum are utilized. This technique strengthens the ability of StarNe
to generalize its predictions on spectra with a wide range of signal
to-noise ratios (S/Ns) across a larger stellar parameter space. Mor
details of the StarNet model itself are discussed in Section 4.3.

(2) Train the same NN to learn the relationshipbetween [X/H] and spectra using synthetic2 spectra
M AC H I N E L E A R N I N G M E T H O D O L O G Y

Supervised learning has been shown to be well adapted for continuous variable regression problems. Given a training set in which, foreach input spectrum, there are known stellar parameters, a supervised learning model is then capable of approximating a functionthat transforms the input spectra to the output values. The learned
function can then ideally be used to predict the output values of adifferent data set. The particular form of this function and how it
is learned depends on the NN architecture. Summarized below is
the CNN that we have implemented for the analysis of stellar spectra; we provide more details about deep NNs and the mathematicaloperations of our selected architecture in Appendix A.

Downloaded from https://acadD

A bias in test data will aﬀect performance.

When designing stellar
2.2 Trainingspectra
and testing of thepipeline,
model
Beforetraining themodel,
the reference set is split into a training se
this bias should be
kept
in
mind.
and a cross-validation set. Training is performed through a serie

of forward and back propagation iterations, using batches of th
training set. The forward propagation is the model function itself
at each layer, weights are applied to all of the input values, and at th
output layer, a prediction is computed. These predictions are thencompared to the target values through a loss function. In our StarNe

Interpretability of NN

Modified in the public version

A funny example
Train NN to learn the relationshipbetween chemistry ([Fe/H]) and observation data using(1) observation fits file (spectra, date, observer’s name etc..)(2) [X/H] from another catalog… Supervised learning.
Result:NN learned how to measure [Fe/H] from fits file.
This result might be wrong!
NN might learn the correlation between observer’s name and chemistry.)(“Dr. AAA only observes low-[Fe/H] stars,” etc.)
Do not use NN as a black-box.

nerate theira certain model is best suited in a particular situation and how
Trade-oﬀ:
interpretability
vsaccuracyle in [172],the selection of the algorithm is related to the given use case,he researcha depth understanding of the trade-off between accuracy andfrom many
interpretability becomes convenient (Fig. 11).
accuracynd analyze

that if weevery MLn inferring
lgorithm isd when weions in real
ML is not to
we need toour specificas the bestinterpretability

FIGURE 11. The trade-off between interpretability and accuracy of somerelevant ML models. Highly interpretable algorithms such as classification
rules,
or linear
are often
inaccurate.
Very accurate DNNs
are a Limitations, and Future Directions“
From “Machine
Learning
for regression,
5G/B5G Mobile
and WirelessCommunications:
Potential,

Toward explainable AI: Layer-wise Relevance Propagation
input
hidden10. Layer-Wise Relevance
Propagation: An Overview
layer
layers203
Input

outputlayer

Uniform LRP
LRP-0

LRP-

LRP-°

N

ℝ

ℝ

M

“Castle”

LRP-°

Input

1x1 @ 1000

1x1 @ 4096

7x1 @ 4096

3x3 @ 512

3x3 @ 512

3x3 @ 512

3x3 @ 512

3x3 @ 512

3x3 @ 512

3x3 @ 2563x3 @ 2563x3 @ 2563x3 @ 128

3x3 @ 128

3x3 @ 64

3x3 @ 64

Composite LRP

10. Layer-Wise Relevance Propagation: An Overview

LRP-

LRP-0

203

Uniform LRP

LRP-0

LRP-

Fig. 10.4. Input image and pixel-wise explanations of the output neuron ‘castle’ obtained with various LRP procedures. Parameters are = 0.25 std and ⇥ = 0.25.

LRP-°

The reason why Composite LRP delivers a better explanation can be traced
to the qualitative di⇥erences between the various layers of the VGG-16 neuralnetwork:

LRP-°

Middle layers have a more disentangled representation, however, the stackingof many layers and the weight sharing in convolutions introduces spurious

LRP-

1x1 @ 1000

1x1 @ 4096

7x1 @ 4096

3x3 @ 512

3x3 @ 512

3x3 @ 512

3x3 @ 512

3x3 @ 512

3x3 @ 512

3x3 @ 2563x3 @ 2563x3 @ 128

3x3 @ 128

3x3 @ 64

3x3 @ 64

Upper layers have only approximately 4 000 neurons (i.e. on average 4 neurons
per class), making it likely that the many concepts forming the di⇥erent
classes are entangled. Here, a propagation rule close to the function and its
gradient (e.g. LRP-0) will be insensitive to these entanglements.

3x3 @ 256

“Red part of the image
was useful forclassification”“Castle”

Composite LRP

LRP-0

Montavon, Binder, Lapuschkin et al. (2019)・(1) Era of big data
・(2) Dimensionality reduction
・(3) Sparsity
・(4) Bayesian analysis
・(5) Machine learning・(6) Deep learning
Robustness / reliabilityof new methods

・(7) Data challenge

Data Challenge (Test to validate methods)• Photometric LSST Astronomical Time-Series Classification Challenge (• Exoplanet: ARIEL Mission Data Challenges ()

https://www.ariel-datachallenge.space/ML/documentation/description• Radio astronomy: SKA Data Challenge Competition #1 (• Microlensing Data Challenge ()

https://www.kaggle.com/c/PLAsTiCC-2018https://microlensing-source.org/data-challenge/

https://astronomers.skatelescope.org/ska-science-data-challenge-1/)• Galaxy Zoo: galaxy morphology classification challenge (https://www.kaggle.com/c/galaxy-zoo-the-galaxy-challenge• Mapping dark matter competition ()• Strong lensing data challenge (• Gaia challenge (https://www.kaggle.com/c/mdm/overview

https://bolognalensfactory.wordpress.com/home-2/blfkids-lens-finding-challenge/

http://astrowiki.ph.surrey.ac.uk/dokuwiki/doku.php

)

)

)

)

Photometric LSST Astronomical Time-Series Classification Challenge
Kaggle-based competition of classifying mock LSST datawith generous prize money :)1st prize: Kyle Boone(Astro PhD student)
2nd prize: Mike & Silogram(Non-astro group)
3rd prize: Major Tom, mamas & nyanp (Non-astro group)

Inflation of ML universe.

Is our community open to data scientists,
given the need for ML talents?
Data challenges for- mock ULTIMATE-Subaru data?
- mock JASMINE data?
- mock [your favorite project] data?
… These will galvanize young/enthusiastic
members of astro/non-astro community!!

Concluding remarks: Era of Big Data

Test Accuracy

sky, 1994), and aNatural
speech. language
The memory-basedlearner
used
only
tasks
s (Gale et al., 1993,
the word before and word after as features.
[自然言語処理=機械英文チェックのテスト]
Schabes, 1996). In
the problem is
1.00
Given a specificScoreo}), all occurrences
0.95
in thetest
set
are(Accuracy)
ywhere the system
0.90
ide which member.
iguation is one of a
0.85
roblems involving
ively small set of
0.80
string context inMemory-Basedpears. Other such
Winnow
0.75
Perceptron
se disambiguation,Naïve Bayesme formulations of
0.70
ntageous aspect of
0.1
1
10
100
1000
Data
size
Millions of Words
n, which allows us
rge
data& Brillsets(2001)
on “Scaling to Very Very Large Corpora for Natural Language Disambiguation”
Banko
Figure 1. Learning Curves for Confusion Set

Concluding remarks: Era of Big Data

Test Accuracy

sky, 1994), and aspeech. The memory-based learner used onlys (Gale et al., 1993,
the word before and word after as features.
It’s NOT who has the best algorithms that wins.
Schabes, 1996). In
the problem is
It’s
who
has
the
most
data.
1.00
Given a specificScoreo}), all occurrences
0.95
in thetest
set
are(Accuracy)(A) 0.93
ywhere the system
0.90
ide which member.
(A) Poor algorithm + Big data
iguation is one of a
0.85
vsroblems involving
ively small set of(B) Good algorithm + Small data
0.80
string context inMemory-Basedpears. Other such
Winnow
0.75(B) 0.75
Perceptron
se disambiguation,Naïve Bayesme formulations of
0.70
ntageous aspect of
0.1
1
10
100
1000
Data
size
Millions of Words
n, which allows us
rge
data& Brillsets(2001)
on “Scaling to Very Very Large Corpora for Natural Language Disambiguation”
Banko
Figure 1. Learning Curves for Confusion Set

Concluding remarks: Era of Big Data
We have the biggest data.
speech. The memory-based learner used only
We
need
a
scope
for
fosteringdata
scientists,
the word before and word after as features.
because the future of NAOJ is on the shoulders of
grad
students,
postdocs,
andyoungresearchers.
1.00

Test Accuracy

sky, 1994), and as (Gale et al., 1993,
Schabes, 1996). In
the problem isGiven a specifico}), all occurrences
0.95
in the test set are(A) 0.93
ywhere the system
0.90
ide which member.
(A) Poor algorithm + Big data
iguation is one of a
0.85
vsroblems involving
ively small set of(B) Good algorithm + Small data
0.80
string context inMemory-Basedpears. Other such
Winnow
0.75(B) 0.75
Perceptron
se disambiguation,Naïve Bayesme formulations of
0.70
ntageous aspect of
0.1
1
10
100
1000
Millions of Words
n, which allows us
rge
data& Brillsets(2001)
on “Scaling to Very Very Large Corpora for Natural Language Disambiguation”
Banko
Figure 1. Learning Curves for Confusion Set

End