Applying Reinforcement Learning to the 

Determination of Crystal Structures with 

Neutron Diffraction

August 8th, 2018

Abigail Wilson

NIST SURF Fellow, Tufts University Class of 2021

Dr. William Ratcliff

NIST Center For Neutron Research, Neutron Condensed Matter Science Group
Outline

•Background

• Crystallography

• Neutron Diffraction

•Problem

•Algorithms

•Performance

•Reinforcement Learning in this Problem Space

•Future Steps
https://opentextbc.ca/chemistry/chapter/10-6-lattice-structures-in-crystallinesolids/

https://www.researchgate.net/figure/Miller-indices-indicating-theplane-perpendicular-to-the-vector-given-for-the-cubic_fig7_302838100

Crystal Structure
https://www.researchgate.net/figure/Principle-of-a-4-circles-diffractometer-and-definition-of-rotationangles_fig5_320672016?_sg=2SkZLSPV7hSnuAUpNs3TXfaV8kakzml0dVVeB2cvI2IzZGyC1Ps1yU8kVyvZqnvICSSLG6P9JpuDMRMowYQU9g

Neutron Diffraction
Problem

•Determining atomic structure is key to understanding novel materials

•Beam time is highly limited

• Few neutron sources

•Need to optimize experiments to minimize required beam time for experiments
Can Reinforcement Learning be 

applied to optimize Neutron 

Diffraction experiments?
Reinforcement learning

Whiteson, Shimon. (2010). Adaptive Representations for Reinforcement Learning. 10.1007/978-3-642-13932-

1. 
http://www.geocities.jp/ohba

_lab_ob_page/structure6.html

https://www.researchgate.net/figure/Principle-of-a-4-circles-diffractometer-and-definition-of-rotationangles_fig5_320672016?_sg=2SkZLSPV7hSnuAUpNs3TXfaV8kakzml0dVVeB2cvI2IzZGyC1Ps1yU8kVyvZqnvICSSLG6P9JpuDMRMowYQU9g

https://publicdomainvectors.org/en/fr

ee-clipart/Green-robot/38871.html

Action: Take Measurement

Environment

Agent

State: Measurements Taken

Reward: +/-
Our Algorithms

ALGORITHMS

•Q Learning

•Deep Q Learning

•Epsilon Greedy

• Implemented by Ryan Cho and Telon Yan

•Actor Critic

• Implemented in collaboration with Ryan Cho and Telon

Yan

RESOURCES

•BLAND [1]

• Crystallographic library

•Training Data

• PPPP2NNNNOO4 – single crystal data from FullProf

[1] J. E. Lesniewski, S. M.T. Disseler, D. J. Quintanta, P. Kienzle, W. D. Ratcliff. “Bayesian Method for the Analysis of Diffraction Patterns using 

BLAND,” Journal of Applied Crystallography, vol. 49, December 2016.
Epsilon Greedy

•Implemented by Ryan Cho and Telon Yan

•Agent chooses the best action at each state, with a small likelihood of choosing 

a random action

• The “best” action is determined by recording the average reward earned from taking that 

action previously

•Preliminary results:

• The agent learned a preference for certain measurements

• Have not yet identified a pattern
Q Learning Deep Q Learning

•Adds a Deep Neural Network to 

standard Q Learning

•Implemented an Environment in 

TensorForce to represent this problem 

space

A sample Q table for Tic-Tac-Toe
Number of Episodes

Total Reward

Total Reward Over Time: Deep Q Learning Algorithm


Total Reward Over Time: Actor Critic Final Z Position of Pr Atom: Actor Critic
Reinforcement Learning in Crystallographic Problem Space

STRENGTHS

•Problem can be framed as a game

•Has the potential to identify patterns 

researchers wouldn’t find

•If effective, could optimize 

experiments

WEAKNESSES

•Has a large discrete action space

•Challenging to develop an effective 

reward
Future Steps

•Complete Actor Critic

•Continue exploring rewards functions

•Implement the Wolpertinger architecture

• Designed for large discrete action spaces

• Proposed by Dulac-Arnold et al in “Deep Reinforcement Learning in Large Discrete Action 

Spaces” (arXiv:1512.07679 [cs.AI])
Acknowledgements 

Dr. William Ratcliff, Paul Kienzle

NIST Center For Neutron Research, Neutron Condensed 

Matter Science Group

Ryan Cho and Telon Yan (SHIP)

Montgomery Blair High School

Tufts University

NIST SURF Program and Directors

Our Reinforcement Learning Game

•The environment contains an initial 

model of the crystal

•Agent chooses to make a certain 

measurement

• Model is updated

• Fit of the model is calculated

• Agent gets reward

•Repeat until the uncertainty drops 

below threshold

•The reward function

• A penalty each step, to encourage efficiency

• A measure of fit: χχ2

• A reward for improving the fit of the model

• Constant

• Proportional to how significantly the fit 

improved

• Inversely proportional to χχ2
https://nmi3.eu/neutron-research/techniques-for-/structural-research.html

Neutron 

Diffraction
Episode Reward Over Time: Actor Critic Final Z Value: Actor Critic