データ科学を利用した固体触媒開発
北海道大学 触媒科学研究所 清水研究室 助教
鳥屋尾 隆 (とやお たかし)Institute for Catalysis, Hokkaido University北海道大学 触媒科学研究所 清水研究室 (2015~)
教授
助教
研究員
秘書
博士学生
修士学生
学士学生
研究生

✓ CO2変換✓ 低級アルカン変換✓ DeNOx
Institute for Catalysis, Hokkaido University

実験に加えて、理論・データ科学を利用した研究

:清水研一
:鳥屋尾隆
:5名
:1名
:11名
:7名
:2名
:1名

Ichigaku Takigawa
2ML-aided research in chemistry (Material/Molecular Informatics)

Sanchez-Lengeling and Alán Aspuru-Guzik, Science 361, 360–365 (2018)Material/Molecular Informatics分野はすでに一定の成功を収めているが、真に革新的な物質・反応の開発は未だ達成が困難→ 機械学習は、“最適化研究”においては力を発揮するが、外挿領域に存在する真に革新的な物質を探索するには不向き
Institute for Catalysis, Hokkaido University3

Heterogeneous catalysis: we need to consider wide range of length and time scale

God made the bulk;
the surface was invented by the Devil.

Length, time

ACS Catalysis 2020 (Review)

触媒分野などの物質のダイナミクスが重要となる分野では実験と理論計算の乖離が大きい (演繹的な研究は難しい)→ データ科学+実験を用いた研究が求められている
Institute for Catalysis, Hokkaido University

4

Result

通常の機械学習予測の例
データセットに(ほとんど)登場しない元素は機械学習モデルの構築にも(ほとんど)貢献しない

Dataset

Cat-ABCCatalyst composition (mol%)ElementA
BCD90640E0F0G0H0I0…
…

Cat-ABCDE

105605200000…

10Cat-ABD

75

1001500000…

23Cat-BCDF

071070013000…

40Cat-ABD’

80

1001000000…

14Catalysts

Yield / %Prediction
25Problem：
通常の機械学習モデルでは元素を単なるシンボルとして扱うだけであり、各元素固有の性質(元素特徴量)がモデルに反映されない・探索空間は、データセット中の(頻出)元素に限定される・元素の持つ特徴を活かしきれない
Institute for Catalysis, Hokkaido University

5Feature engineering (New representation of elements)
元素を、ただの“シンボル”としてではなく、“多次元の特徴量ベクトル”として表現
Atomic radius,
electronegativity,
density, melting point,
Ionization energy, etc.

DatasetElement properties (p features)

Catalyst compositions

Composition [mol%]
Catalyst

Descriptors

ElementA

B

CDE

Cat-ABC1904600

Cat-BDE1011080

9Cat-AE175

00025

Cat-AE280

00020

Cat-ABCDE1 23151070

12…

pA

A1A2…

ApB

B1

B2…

Bp

C

C1C2…

Cp

D

D1

D2…

Dp

E

E1E2…

Ep

元素を基に触媒設計するのではなく、電子状態を基にした触媒設計

Elemental descriptor representation
1st feature

Catalyst
Cat-ABC1Cat-BDE1
Cat-AE1Cat-AE2
Cat-ABCDE1

190×A180×D175×A180×A190×E1

290×A2
80×D2
75×A2
80×A2
90×E2

… p
… 90×Ap… 80×Dp… 75×Ap… 80×Ap… 90×Ep

2nd feature
16×C1
11×B1
25×E1
20×E1
15×C1

2
6×C2
11×B2
25×E2
20×E2
15×C2

…
…
…
…
…
…

Kth feature

…
p
6×Cp
11×Bp
25×Ep
20×Ep
15×Cp

…
…
…
…
…
…

1
0×E1
0×C1
0×D1
0×D1
2×A1

2
0×E2
0×C2
0×D2
0×D2
2×A2

…
…
…
…
…
…

p
0×Ep
0×Cp
0×Dp
0×Dp
2×Ap

ChemCatChem, 2019, 11, 4537
ChemCatChem, 2021, 13, 3636

Institute for Catalysis, Hokkaido University

元素組成の代わりに、元素特徴量(原子半径、電気陰性度、密度等)と構成比の積を予測記述子として用いる

Prediction of literature data on
water gas shift (WGS) reaction(~200 datapoints)
CO + H2O ⇄ CO2 + H2

元素特徴量を用いることで高い予測精度6

ML prediction of promising catalyst candidates
Sequential model-based optimization (SMBO) and recovery procedure
The core procedure of our SMBO strategy

Recovery of actual catalyst compositions fromelemental descriptor representations

ChemCatChem, 2021, 13, 3636
Chemisty Letters 2022, 51, 269

Institute for Catalysis, Hokkaido University

Collaboration withDr. Takigawa

7ML prediction of promising catalyst candidates
Top 20 promising candidate catalysts for WGS worth testing next suggested by the explorative ML model

CO conversion*The EI scores correspond to the average of the improvement on 𝑦weighted by the probability for the red-color region.
(how much we can improve the CO conversion compared to thecurrent best)
Element composition(Oxygen is not shown)

Predictions made here include elements that do not appear frequently in the literature
Institute for Catalysis, Hokkaido University8

Can our ML methods predict catalysts that include elements unseen in the original dataset?
The number of each elements appeared(A) in the original dataset(B) in the ML-predicted catalyst candidates*Elements having the atomic number 3 (Li) through 83 (Bi) except O, Tc, Hg, and noble gasses were used as catalyst component candidates

元素そのものは記述子に使わないので、データセットに無い元素も高い精度で提案可能 +小さなデータセットや偏りのあるデータセットにも有効
化学・材料科学分野における外挿!?
※データ科学的には内挿
Institute for Catalysis, Hokkaido University

9Can we actually make high-performance and novel catalysts
using the developed extrapolative ML models?

Institute for Catalysis, Hokkaido University

10Reverse Water-Gas Shift (RWGS) reaction at low temperature (T ≤ 250 ˚C)Catalyst

CO2+Catalyst

H2/CO2T (˚C)

P (bar)CO2 conv.
(%)

CO selec.
(%)

CO formation rate
(mmol gcat-1 min-1)

ReferenceCu(10)-C/ZnO(5)/SiO2

3250

30

6.0

99.0

<0.05

Sustain. Energy Fuels 2020, 4, 2937–2949

Li(10)-Rh(5)/Zeolite

3250

30

13.0

87.0

0.12

Catalysts. 1998, 175, 67–81.3

Rh(0.4)/SiO23250

10

2.5

84.6

<0.05

J. Am. Chem. Soc. 2019, 141, 8482−8488

K80-Pt(0.05)/zeolite L

12501

1.01000.19

Appl. Catal. B Environ. 2017, 216, 95–105

Au(1)/TiO24250

111.5

1000.13ACS Catal. 2018, 8, 7455−7467

Pt(5)/TiO2

1250

14.0

97.50.07

Catal. Today 2017, 281, 312–318

NH2MPA/Pt(5)/TiO2

14250

1

8.4

95.1

0.06

J. Am. Chem. Soc. 2020, 142, 5184–5193

Pt(1)/CeO2

1250

10.6

1000.06

J. Energy Chem. 2016, 25, 1051–1057

In2O3-CeO2

1250

1

<0.1-<0.05

Catal. Today 2016, 259, 402–408

Pd(3)In/SiO2

1250

1<0.1-<0.05

Chem. Eng. Sci. 2015, 135, 193–201

Pt(3)/MoOx(10)/TiO2

4

250

19.4

1000.66

Our previous work (identified without use of ML)

H2T ≤ 250 ˚CCO( + CH4 )Through studies using in situ/operando spectroscopies and DFT calculations,
we have identified that MoOx species play important roles for progression of the reaction

Institute for Catalysis, Hokkaido University11

実触媒探索への応用: CO2水素化反応 (逆水性ガスシフト反応)
CO2+H2Pt(3)/M1(X1)-M2(X2)-M3(X3)-M4(X4)-M5(X5)/TiO2
250 ˚C, 1 atm

Original dataset (触媒数45: 全て実験により取得)
2種類の添加元素からなる触媒を含むオリジナルデータセットを作成
5種類の添加元素からなる触媒系にまで探索範囲を拡張

CO

先行研究(ML利用ナシ)で発見した世界最高活性を示すPt/Mo(10)/TiO2触媒を凌ぐ触媒を探索する

X = 0, 1, 2, ..., 10 wt%
検討した触媒組成の候補は膨大
505 x 205 = 約600兆(ca. 白金族元素や毒性・放射性元素を除く50種類)※4元素としても、504 x 204 = 約6000億

ML modelExtraTrees Regression (ETR)

記述子として選定した元素特徴量Electronegativity
Density
Enthalpy of fusion
Melting point
Group
Band gap in the most stable oxide form
Oxidation number in the most stable oxide form
Ead of CO2 on metallic surface (obtained using VASP)

Institute for Catalysis, Hokkaido University12ML-assisted catalysts discovery
Original

Pt(3)/Rb(1)-Ba(1)-Mo(0.6)-Nb(0.2)/TiO2
Pt/Al-Mo-Re/TiO2

Datapoint

The best CO form. rate at each iteration
Prediction accuracy (R2) at each iteration
Pt/V-Sr-Nb-Mo-Re/TiO2
Current
Pt/Mo-Cs-Ba-Tb-Lu/TiO2

機械学習
→推薦
データの拡張

触媒合成

触媒反応
(45)

(79)

Institute for Catalysis, Hokkaido University

(115)

(162)

(256)

(300)

試した触媒数JP Patent + Nature Communications, in press

13

Control reactions

Institute for Catalysis, Hokkaido University14

(Normal) machine learning vs deep learning
A rough schematic showing why we use “normal” ML models rather than deep learning approaches

Deep Learning(NeuralNet)
Prediction accuracy

LargeMediumSmall(Normal) Machine Learninginterpretable/explainable compared todeep learning approaches

Amount of data
Institute for Catalysis, Hokkaido University

15

Importance of descriptors and their contributions on the catalytic performance
記述子の重要度とその寄与の可視化

SHAP (SHapley Additive exPlanations)

Positive impact
重要記述子とRWGS活性の関係

Positive impact
Positive impact
Positive impact

Negative impact
Negative impact

Institute for Catalysis, Hokkaido University16

Associative mechanism vs redox mechanismi) Associative mechanismii) Redox mechanism― CO is produced through the decomposition of the surface-reactive
intermediates such as formate and carbonates― Oxygen vacancies are formed on the surface of the support oxide by H2,
while CO2 reoxidizes the partially reduced oxide to fill the formed oxygenvacancies, resulting in the formation of CO.

COMOx

CO2MOx-y

H2

H2O

L. F. Bobadilla, et al., ACS Catal. 2018, 8, 7455−7467

Institute for Catalysis, Hokkaido University

17

Mo K-edge XANES at SPring-8

Institute for Catalysis, Hokkaido University18

Operando Mo K-edge XANES measurements at SPring-8
Pretreatment
H2 red.

He purge

CO2/He(15 min)

H2/He(20 min)

CO2/H2×0.05

Reaction proceeds via redox of Mo species ((Reverse) Mars-van Krevlen type mechanism)* No change was observed for Rb, Ba, Nb, and Ti K-edge XANESInstitute for Catalysis, Hokkaido University

19

XAS at Swiss Light Source
Mo K edge
H2/He

CO2/He

H2/He

CO2/HeH2/He

Collaboration withAtsushi Urakawa@TU Delft= Oxidized Mo (VI)
= Reduced Mo (IV)

Institute for Catalysis, Hokkaido University

https://mcrals.wordpress.com/R. Tauler et al Journal of Chromatography A, 730 (1996) 177-183
J. Jaumot et al Chemometrics and Intelligent Laboratory Systems
140 (2015) 1–1220

XAS at Swiss Light Source
Pt L3 edge
H2/He

CO2/He

H2/He

CO2/HeH2/He= Metallic Pt= Partly oxidized Pt= Metallic Pt withCO adsorbed= Partly oxidized Pt(more oxidized thanthe component 2)

Institute for Catalysis, Hokkaido University

Operando IR (CO2 flow at 250 oC after the H2 reduction + He purge)
Time plot of each peak

Carbonate

FormateRb and Ba act as bases to generate adsorbed surface species such as carbonate and formatethat lead to the formation of CO
Institute for Catalysis, Hokkaido University22

Kinetic study
Arrhenius plot
Apparent reaction order and activation energy (Ea) of RWGS reaction
Catalyst (10 mg)

CO2 a

H2 b

CO c

Ea (kJ mol-1)

Pt(3)/Rb(1)-Ba(1)-Mo(0.6)-Nb(0.2)/TiO2

0.47

0.48

-0.80

45.6

Pt(3)/Rb(1)-Ba(1)-Mo(0.6)/TiO2

0.39

0.42

-1.12

48.7

Pt(3)/Mo(0.6)/TiO2

0.48

0.52

-1.01

52.8

Pt(3)/TiO2

0.53

0.57

-1.37

58.4

a 0.706 atm H , total flow rate of CO , H and N is 85 mL min-1, 250 ˚C.
2222b 0.235 atm CO , total flow rate of CO , H and N is 85 mL min-1, 250 ˚C.
2222c CO /H = 1/3, total flow rate of CO , H , N and CO (1.0-2.5 mL min-1) is 85 mL min-1, 250 ˚C.
2 2222【 Reaction order of CO 】
Pt(3)/Rb(1)-Ba(1)-Mo(0.6)-Nb(0.2)/TiO2＜ Pt(3)/Rb(1)-Ba(1)-Mo(0.6)/TiO2

CO poisoning is suppressed by the addition of Nb
Institute for Catalysis, Hokkaido University23Electronic state analysis of Pt
In situ IR spectra of CO adsorption

XPS (Pt4f region): H2 reduction at 300 ˚CAddition of Nb reduces the amountof CO adsorption and shifts thepeak to higher wavenumber.

Nb makes Pt electron-deficient, resulting in suppression of CO poisoning
Institute for Catalysis, Hokkaido University

24

Near ambient pressure X-ray photoelectron spectroscopy (NAP-XPS)

Ti4+
Collaboration with
Hiroshi Kondoh and Ryo Toyoshima@Keio University(i) H2Ti3+
250 °C(ii) CO2Temperature

300 °C

0.1 Torr H230 min(ii)

(i)

(iii)

(i) 0.1 Torr H2(ii) 0.1 Torr CO2(iii) 0.1 Torr H2Time(iii) H2

Gas condition(i) H2(ii) CO2(iii) H2

Institute for Catalysis, Hokkaido University

Peak concentration / %
Ti3+
Ti4+
595
298
49625

分光・速度論解析による反応機構研究
Ti4+(i) H2

Ti3+(ii) CO2(iii) H2

Carbonate

浦川篤教授@TU DelftFormate

近藤寛教授@慶應義塾大学

Davide Ferri博士@PSI

Pt(3)/Rb(1)-Ba(1)-Mo(0.6)-Nb(0.2)/TiO2
各種In situ/operando分光・速度論解析から、各元素の役割を解明
Element

Function

MethodMo

RedoxXASBa, Rb

BaseIRNbModification of electronic structure of Pt

IR, XPS, Kinetic study

Institute for Catalysis, Hokkaido University26

The desire for full optimization of catalysts in a more efficient way
Pt(3)/Rb(1)-Ba(1)-Mo(0.6)-Nb(0.2)/TiO2
PGM1(X1)-PGM2(X2)-PGM3(X3)/
M1(Z1)-M2(Z2)-M3(Z3)-M4(Z4)-M5(Z5)/Support

Pt/Al-Mo-Re/TiO2

Datapoint

The best CO form. rate at each iteration
Prediction accuracy (R2) at each iteration
Pt/V-Sr-Nb-Mo-Re/TiO2
Pt/Mo-Cs-Ba-Tb-Lu/TiO2

Previous study focused solely on exploring
the additive oxide (M element) of thecatalysts, with Pt and TiO2 fixed.

It is highly desired to achieve full optimizationof catalysts in a more efficient way(with small number of experiments)

(45)

(79)

Institute for Catalysis, Hokkaido University

(115)

(162)

(256)

(300)

27

Computational data (public databases) vs experimental data
Computational data (public databases)

Experimental data

・ Rapidly developed over the last decade・ Contains a large amount of data
However,・ Mostly only for known compounds and bulk properties→ Still large gap with experimental data・ Should represent “real catalytic activities”
However,・ Difficult to compare literature data
・ Takes too much cost/time to perform experiments→ Only small amount of data are available

Is there a method that effectively use the advantages of both data?
Institute for Catalysis, Hokkaido University28

Transfer learning
A technique that applies knowledge from one domain to the learning process in another domain.
Step 1 Pre-training
Randomly Initialized
Deep Learning (DL) model
Source dataset(Public databases)

Step 2 Fine-tuning

Transfer“learned knowledge”

Pre task

Replace (Initialize)
Fully connected layer

Target task
Pretrained DL model

Fully connectedlayer

Target dataset(Experimental data)

Using transfer learning techniques,
We can achieve high prediction accuracy even with a small experimental amount of data (experimental trials)
Institute for Catalysis, Hokkaido University29

Data from public databases (Formation energy)
Data distribution

Frequency

The total number of data points = ca. 600,000

Institute for Catalysis, Hokkaido University30

Model architecture based on transformer-based neural network
Public databases (ca. 600,000 datapoints)
Target1 : formation energy

Materials Project
OQMD
AFLOW

Back propagation(Parameter Update)

Transformer encoder×N

Embedding Layer

Loss1

Predict
Target1

Truth
Target1

Predict
Target2

Truth
Target2

Loss2
Experimental data for RWGS (362 datapoints)
Target2 : CO formation rate
Institute for Catalysis, Hokkaido University

Back propagation(Parameter Update): Pre task process: Fine-tuning process: Common process31

Prediction accuracy
Transformer(without pre-training)

Extra Trees

Transformer(with pre-training)

RWGS TL

RMSE

R2

RMSE

R2

RMSE

R2

Train

0.04

0.99

0.08

0.99

0.070.99

Test

0.52

0.68

0.78

0.31

0.51

0.72

The Transformer-based model (with pre-training)
showed the best accuracy

Institute for Catalysis, Hokkaido University32

Summary and future prospects✓ Extrapolative ML method has been developed using elemental descriptor representation
ChemCatChem 2019, 11, 4537
ChemCatChem, 2021, 13, 3636
Chemisty Letters 2022, 51, 269

XAFS measurements✓ The ML-assisted research has led to discovery of high-performance catalysts(Current best catalyst = Pt/Rb-Nb-Mo-Ba/TiO2)
Nature Communications, in press✓ Full optimization of RWGS catalysts were realized by using the transfer learning approach
Manuscript in preparation

将来展望✓ 新(novel)触媒の開発

Experiment－ PGM削減、実験条件の考慮、他の反応に対して等✓ 大規模(計算)データの利用－ 転移学習等

TheoryData science✓ 解釈性－ ブラックボックスとされる固体触媒分野の理解を進める

Institute for Catalysis, Hokkaido University33