Spin-Interactions Using 

Reinforcement Learning

Justin Wang (Johns Hopkins University)

Collaborators: William Ratcliff, Paul Kienzle

1
Magnetic Materials

Why is understanding magnetic 

structures important? • Data Storage (RAM and hard 

drives)

• High-resolution imaging

• Spintronics • Superconductivity • Quantum computing

2
Neutron Scattering • Probing crystal structures and magnetic 

interactions by scattering neutrons off 

of samples

• Inelastic neutron scattering • Measures the difference in energy 

between the neutrons before and 

after contact with the sample

• Creates excitations in the magnetic 

structure of the crystal lattice and 

generates a spinwave

• Measures the energy and direction 

of the spinwave in a structure

3
Problems

• Limited supply of neutron beams worldwide

• Experiments take a long time to run

• Not all measurements are needed

4
Reinforcement 

Learning (RL) • Agent learns by interacting with the 

environment

• No data involved (unlike Active Learning) • Agents learn by trial and error with 

a reward system

• By maximizing rewards, the agent 

can successfully navigate the 

environment

5
RL Process

• Observe the environment

• Decide how to act using a given 

strategy

• Take an action

• Receive reward/penalty

• Refine/modify strategy based on 

outcome

• Iterate until optimized

6
Triple-Axis Spectrometer

• First axis

• Monochromator that filters the 

neutrons to a certain wavelength

• Second axis

• Sample that neutrons scatter on

• Third Axis

• Analyzer that selects the desired 

energy to be detected

• Detector

• Detects said energy

7
SpinW

• MATLAB Library used to simulate magnetic 

structures and excitations

• Plots dispersion graphs and neutron intensities

• RL Application

• Compiled into Python to better utilize 

machine learning

8
Challenges

Language/Translation

Utilizing new RL library

Hardware limitations

9
Language and

Translation

• Python-MATLAB communication

• PySpinW (python version of 

SpinW) is extremely buggy

• Cannot plot neutron intensities

• Potential Solution: Julia

• Trouble aligning ferromagnets

• Defined ferromagnetic structures 

came out antiferromagnetic

10

Utilized Solution:

• Compile MATLAB and SpinW into 

python

• Essentially calling MATLAB 

from Python

• Issues with sorting arrays

• Eigenvector sorting is 

scrambled when translating 

MATLAB to Python
MATLAB vs SpinW Neutron Intensity

11

MATLAB SpinW (calling from MATLAB)
Utilizing New RL Library

• Reinforcement Learning library

• Set of built in modules to assist with RL 

development

• Old RL Library is growing obsolete

• OpenAI Gym is no longer being updated 

and maintained

12

Solution

• Switched from OpenAI Gym to 

Gymnasium

• New fork of Gym library

• Reconstructed old SpinW RL 

algorithm to be used in 

Gymnasium
Hardware Limitations

• Experiments take extremely long 

time to run

• Memory intensive

• Computer frequently crashes 

on longer experiments

• Hard to obtain large episode 

datasets

13

• Solution 

• Parallel processing

• Divided processing over 8 

cores instead of 1

• Future Improvement: 

• Run experiments on cluster 

for larger datasets
Results • MATLAB and SpinW successfully 

compiled/translated into Python • Allows for future work to be able to 

plot neutron intensities in addition 

to spin dispersions

• RL learning process limited by 

hardware • Reward plot does not show an 

upward trend 

• Needs more episodes to see if it 

truly works

14
Reward/Episode • Expectation: • Upward trending graph with more 

weight on positive rewards than 

negative

• Shows that the AI is learning and 

slowly optimizing

• Result: • Relatively even all around • Shows that AI isn’t learning very 

fast

• Needs more episodes (hardware 

limitation)

15
Number of Steps/Episode

• Shows the direction to measure the 

spinwave dispersion

• More steps per episode means that 

direction is where measurements 

should be made

16
Future Steps • Apply Neutron Intensity 

measurements now that it is 

possible

• Run on cluster to maximize 

episodes used and time efficiency 

to better train RL agent

• Publishing

17
Acknowledgements

Thanks to the SURF program directors, 

interns, and mentors

Special thanks to William Ratcliff and 

Paul Kienzle as mentors

We are grateful for funding from the 

Center for High Resolution Neutron 

Scattering

18
Extra Slides

19
Elastic vs 

Inelastic 

Neutron 

Scattering

20

Scattered particles don’t lose any kinetic 

energy 

Can determine where atoms are from the 

way they scatter

Elastic

Neutron energy changes and is monitored

Excites the nucleus

Understand energy of interactions 

between atoms or spins

Inelastic
21