Published as a conference paper at ICLR 2022

G ROUP E QUIVARIANT N EURAL N ETWORKS FORS PECTROPOLARIMETRIC I NVERSIONS INS OLAR A STRONOMY
Michael Ito & Peter Sadowski
Department of Information and Computer Science
University of Hawai‘i at Mānoa
Honolulu, HI 96822{mi24,peter.sadowski}@hawaii.edu

Ian Cunnyngham & Xudong Sun
Institute for Astronomy
University of Hawai‘i at Mānoa
Honolulu, HI 96822{ianmc,xudongs}@hawaii.edu

A BSTRACTThe upcoming Daniel K. Inouye Solar Telescope (DKIST) will produce unprecedented high-cadence, high-resolution, and multi-line spectropolarimetric observations of the Sun. New computational techniques are needed to infer the state of theSun’s atmosphere from these observations. Deep learning is a promising approachto this spectropolarimetric inversion problem that can both provide real-time visualizations to astronomers and potentially improve upon existing algorithms bycombining spatial, temporal, and multi-spectral information. Here we investigate
group equivariant deep learning as a method for inferring the three-dimensional
photospheric structures, training on magnetohydrodynamic (MHD) simulationsof two types of solar features: sunspots and active regions. Our results demonstrate that including multiple lines improves the mean relative error from 18.6%
to 14.4%, averaged over all MHD state variables, and that using group equivariant
convolution architectures further improves the mean relative error to 12.5%.

1

I NTRODUCTION

The solar photosphere hosts a variety of interacting physical phenomena including magnetic fields,
convection, and energy transfer/conversion. These phenomena are generally well-described by the
MHD equations for the evolution of the photospheric state variables: the magnetic field B, velocity
field v, density ρ, and pressure p throughout the 3d volume. Because these quantities determine
how photons are emitted, absorbed, and re-emitted, solar physicists infer them from distinct spectral
bands and polarization patterns (Stokes profiles) observed by telescopes. Thus, interpreting solar
telescope data is an inverse problem (Figure 1).
Current spectropolarimetric inversion techniques are extremely computationally demanding. These
inversion algorithms generally start with an initial guess of the solar state variables, and a physics
model is used to calculate the resulting Stokes profiles that would be observed on Earth. These
profiles are compared to observation data, and the error is used to update the estimate of the solar
state variables —iterating until convergence. This inversion is essentially a non-convex optimization
problem with convergence difficulties, ambiguities due to multiple local minima, and computationally expensive evaluations of the forward model. Thus, these models rely on a number of simplifications to speed up calculation (del Toro Iniesta & Ruiz Cobo, 2016), and processing cannot keep
up with the DKIST data stream (20 TB per day on average) so astronomers cannot view solar conditions in real time. New computational methods are needed to meet the demands of modern solar
astronomy.
As in many other inverse problems in physics (Sadowski & Baldi, 2018; Brehmer et al., 2020),
deep learning is a promising approach because simulations can be used for training data, and approximate inference with neural networks is computationally efficient. MHD simulations are now
advanced enough to faithfully reproduce many solar features from first principles (Rempel, 2012;
Cheung et al., 2010; 2019) and provide the ground truth labels for training a neural network. The
1

Published as a conference paper at ICLR 2022

MHD Variables

Forward Model(Known Physics)

1024 x 1920 pixels x 7 depths x 7 MHD

Stokes Profiles
1024 x 1920 pixels x 256 wavelengths x 4 Stokes

Inversion
Algorithm

Deep Learning

Inverse Problem

Figure 1: Current Stokes inversion techniques randomly initialize the hidden MHD state variables(left) and make iterative updates by repeatedly simulating the forward physics to predict observed
Stokes profiles (right). This is done independently for each pixel, without taking advantage of spatial
correlations. Instead, we use a deep G-CNN to predict all the MHD variables in a 3-d volume directlyfrom the Stokes profiles in a single step. In our setup, the input to the G-CNN is a 4-d tensor of shape(16, 16, 272, 4) with two spatial dimensions of size 16, 272 wavelengths, and 4 stokes parameters.
The output is a 4-d tensor with three spatial dimensions and 7 MHD variables.

observations (inputs) are the observed Stokes parameters that would be observed on Earth, while theoutputs are the hidden MHD variables that describe the state of the Sun’s atmosphere.
The feasibility of deep learning for spectropolarimetric inversion was first demonstrated by the pioneering work of Ramos & Baso (2019) using a standard convolutional neural network. In this work,
we show that performance is improved by accounting for rotational symmetry with specialized neural network architectures. Enforcing this symmetry makes a model less likely to overfit (Gens &Domingos, 2014; Dieleman et al., 2016; Cohen & Welling, 2016; Cohen et al., 2019), with many
applications in medical imaging (Winkels & Cohen, 2018; Bekkers et al., 2018), remote sensing, andastronomy (Dieleman et al., 2015). In particular, we use the implementation of group equivariant
convolution networks (G-CNNs) proposed by Cohen & Welling (2016) to build models equivariant under 90◦ rotations and reflection transformations (the Dihedral group D4 ). In a D4 G-CNN,feature detectors at the initial convolutional layer are rotated and reflected, such that a rotation orreflection applied to the input image results in a permutation of the feature maps in the first hidden layer. Permutations of feature maps in the first hidden layer result in similar permutations insubsequent layers, until they are aggregated at the top, resulting in an image-to-image model that is
globally-equivariant — or an image classifier that is invariant — to rotations and reflections.
In experiments, we compare G-CNNs against standard CNNs using two simulation datasets which
capture solar features: a sunspot and an active region. Moreover, we show for the first time that performance improves if we include multiple spectral lines (more wavelengths) as inputs to the model.
These spectral lines probe different heights in the photosphere, which provide additional information that can be exploited by our deep learning approach. Together, our experiments demonstratethat deep learning is a promising approach to the Stokes inversion problem, and that applicationsof deep learning to astronomy can benefit by designing architectures that capture symmetries of the
data.

2

E XPERIMENTAL R ESULTS

2.1

DATASETS AND E VALUATION

Two separate models were trained and evaluated on the simulations of an active region (Rempel,2012) and a sunspot (Cheung et al., 2010). Stokes observations of both pairs of Fe I 630.2 and Fe
I 1565 nm lines were synthesized using the software framework of Ramos & Baso (2019). Thesesimulations both describe the state of the Sun’s atmosphere as a sequence of 3d snapshots of seven2

Published as a conference paper at ICLR 2022

variables. Each snapshot was split into non-overlapping 16 × 16 pixel patches along the latitude
and longitude dimensions, resulting in 76,800 different 16×16×7×7 tensors for the active region
and 9,216 tensors for the sunspot. For each tensor, we simulated the forward physics to obtain the
Stokes profiles — the data observed by the telescope — using the SIR algorithm (Ruiz Cobo & delToro Iniesta, 2012). This produced a labelled dataset for supervised learning, where the inputs are
4-d tensors of shape (16, 16, 4, 272) with two spatial dimensions each of size 16 and four Stokes
parameters (I, Q, U, V ) at 272 wavelengths (depending on which set of lines are used). The outputs
are 4-d tensors of shape (16, 16, 7, 7) with three spatial dimensions and 7 MHD variables
A problem that arises when predicting magnetic fields is that there are ambiguous solutions: thesame Stokes parameters can correspond to opposing magnetic fields, so it may be impossible for themodel to predict the true direction. We remove this ambiguity using the coordinate transformation
described in Ramos & Baso (2019) that maps the original field (Bx , By , Bz ) to field (BQ , BU , BV ):1

BQ = sign(Bx2 − By2 )|Bx2 − By2 | 2
BU = sign(Bx By )|Bx By |BV = Bz .

1
2

(1)
(2)
(3)

The input data was then standardized by subtracting the mean and dividing by the standard deviation
across heights, widths, time steps, and wavelengths. Outputs were standardized by subtracting themean and dividing by the standard deviation across heights, widths, and time steps.
Each dataset was split into 60/20/20% train/validation/test. For the sunspot, splits were madechronologically, keeping all patches within the same snapshot in the same split. Models were optimized with Adam using a batch size of 32 (Kingma & Ba, 2014). The initial learning rate was
1e−4 and was halved after 10 epochs if no improvement was observed in the validation loss. After
15 epochs of no improvement on the validation loss, training was stopped. Hyperparameter optimization was conducted with SHERPA to minimize the validation set loss (Hertel et al., 2020). Dataaugmentation of rotations and mirror reflections were used during evaluation on the test set, andwe report the median relative error for each variable across all heights, widths, and depths. These
choices were held constant for each model.
2.2

D O M ULTI - LINE O BSERVATIONS I MPROVE P ERFORMANCE ?

DKIST’s multi-line observations should enable us to cover a wider range of physical heights andbetter constrain the vertical gradient of MHD variables. The latter will improve our understandingof several important questions, for example, the horizontal electric current density and the degreeof force-freeness of the photosphere (Puschmann et al., 2010; Metcalf et al., 1995). Initial line
candidates include the well studied Fe I 630.2 and Fe I 1565 nm, which will be simultaneously
observed with DL-NIRSP, with possible extension to the He I/Si I lines near 1083 nm.
We confirm that observing multiple lines improves performance in the deep learning approach. A
fully convolutional neural network (CNN) was trained with two different sets of inputs: (1) only
the Fe I 1565 nm lines (160 wavelengths); (2) both the Fe I 630.2 and Fe I 1565 nm lines (272
wavelengths). The results show clear performance improvements using both sets of lines (Table 1).
2.3

D O G ROUP -E QUIVARIANT CNN S I MPROVE P ERFORMANCE ?

Enormous progress has been made in computer vision by leveraging two key properties in CNN
architectures: equivariance and invariance (Goodfellow et al., 2016). Convolutional layers provide
translational equivariance: when the image is translated, the activations of the convolutional layer
are translated too. When stacked with pooling layers, the CNNs can achieve translational invariance:
when the image is translated, the output does not change. Invariance to translations constrains thelearning problem and enables CNNs to learn from fewer examples. However, many applications inmedical imaging, remote sensing, and astronomy also have rotational equivariance or invariance,
which is harder to capture in neural network architectures. The G-CNN architecture introduced by
Cohen & Welling (2016) provides a computationally-efficient architecture that is provably equivariant to 90◦ rotations and reflection transformations (the Dihedral group, D4 ).
3

Published as a conference paper at ICLR 2022

Property
HeightTemperature
Pressure
Velocity Z
Magnetic field Q
Magnetic field U
Magnetic field V
Average

km
K
Pa
m/s
gauss
gauss
gauss
NA

Active Region
Single line Multi-line
0.19%
0.17%
1.27%
0.87%
4.43%
3.77%
27.38%
20.93%
31.56%
24.27%
32.53%
24.66%
33.08%
26.03%
18.63 %
14.39%

Sunspot Region
Single line Multi-line
1.28%
1.23%
1.46%
1.16%
6.78%
6.36%
49.18%
41.66%
32.95%
26.23%
33.07%
25.00%
30.19%
25.31%
22.13%
18.14%

Table 1: Relative error on the test set of two CNN inversion models trained using single (Fe I
1565nm) and multiple lines (Fe I 630.2nm and Fe I 1565nm). The multi-line model outperforms the
single-line model for each property.
In G-CNNs, feature detectors at the initial convolutional layer are rotated and reflected. Thus, arotation or reflection applied to the input image simply results in a reordering of the output featuremaps at the first layer. At intermediate layers, the order of feature detectors are permuted suchthat permutations of feature maps in one layer result in permutations of feature maps in subsequent
layers. Lastly, pooling at the output layer results in locally invariant and globally equivariant representations.
We construct a D4 -CNN model, a G-CNN on the D4 group, by stacking group convolution (Gconvolution) layers in D4 along with group pooling layers (G-pooling) after each G-convolution
layer. Next, we evaluate performance on a standard CNN as a baseline. We also evaluate performance on a CNN where the input-output pairs are augmented by random 90◦ rotations. The results
show that rotation equivariance is a useful inductive bias that cannot be obtained from only data
augmentation. The D4 -CNN overfits less to the training set, and the results on the test set show
clear performance improvements across each MHD variable (Table 2). The inversion examples in
Figure 2 also show that the D4 -CNN can accurately infer the features of the MHD variables.
Property
HeightTemperature
Pressure
Velocity Z
Mag. field Q
Mag. field U
Mag. field V
Average

km
K
Pa
m/s
gauss
gauss
gauss
NA

CNN
0.17%
0.87%
3.77%
20.93%
24.27%
24.66%
26.03%
14.39%

Active Region
CNN +R D4 -CNN
0.17%
0.14%
0.87%
0.70%
3.77%
3.20%
20.96%
17.97%
24.30%
21.35%
24.65%
22.03%
26.20%
22.16%
14.42 %
12.51%

CNN
1.23%
1.16%
6.36%
41.66%
26.23%
25.00%
25.31%
18.14%

Sunspot Region
CNN +R D4 -CNN
1.25%
1.14%
1.19%
1.00%
6.50%
6.15%
42.74%
38.31%
26.95%
23.61%
25.47%
22.06%
26.21%
22.74%
18.62%
16.43%

Table 2: Relative error on the test set for a CNN, a CNN with random rotations of 90◦ , 180◦ , or 270◦
(CNN+R), and a D4 -CNN. Each model was trained on the multi-line set of inputs, with an equivalent
number of parameters. The D4 -CNN outperforms both standard CNNs for every property

3

C ONCLUSIONDKIST will be the center of ground-based solar observation for several decades. The DKIST Level-2
effort, being implemented by the National Solar Observatory (NSO) for the community, will initiallyconcentrate on providing routine inversion data products for a fraction of DKIST observations focused on single snapshots in time, rather than at the high time cadence afforded by photospheric use
cases. This deep learning approach could accelerate scientific progress by providing faster, cheaper,larger and more accurate data products to the scientific community.
Here we have introduced a deep learning model equivariant to rotations and reflections for the spectropolarimetric inversion problem, and showed through experiments that this is a useful inductive4

Published as a conference paper at ICLR 2022

Figure 2: Stokes inversions for the active region using a D4 -CNN (left) vs. ground truth (right) on
the test set. The model simultaneously predicts seven different properties at seven depths. Shown
here from top to bottom are surface height, pressure, and magnetic field V .
bias, improving performance by 13%. Moreover, we demonstrate that deep learning is able to successfully integrate information from multiple spectral lines to improve performance, whereas iterative approaches are only able to invert lines independently. Once trained, D4 -CNN inversions areorders of magnitude faster (≈ 7.4 × 104 pixel/sec) in comparison to current inversion methods suchas VFSIV (≈ 6 × 102 pixel/sec) (Borrero et al., 2011).
Our collaborators are currently running large-scale MHD simulations at the NCAR supercomputing
facility to provide training data for this project. Future work will leverage the insights from thiswork for much larger models.
ACKNOWLEDGMENTS
We thank Curt Dodds, Sarah Jaeggli, Matthias Rempel, Lucas Tarr, and Tom Schad for valuable
discussions. This material is based upon work supported by the National Science Foundation under Grants No. 1848250 and 2008344. The work was made possible by hardware grants fromNVIDIA to PS, along with technical support and advanced computing resources from the University of Hawai’i Information Technology Services Cyberinfrastructure group.

R EFERENCES
Erik J Bekkers, Maxime W Lafarge, Mitko Veta, Koen AJ Eppenhof, Josien PW Pluim, and Remco
Duits. Roto-translation covariant convolutional networks for medical image analysis. In International conference on medical image computing and computer-assisted intervention, pp. 440–448.
Springer, 2018.
JM Borrero, S Tomczyk, M Kubo, H Socas-Navarro, J Schou, S Couvidat, and R Bogart. Vfisv:
Very fast inversion of the stokes vector for the helioseismic and magnetic imager. Solar Physics,
5

Published as a conference paper at ICLR 2022

273(1):267–293, 2011.
Johann Brehmer, Gilles Louppe, Juan Pavez, and Kyle Cranmer. Mining gold from implicit models
to improve likelihood-free inference. Proceedings of the National Academy of Sciences, 117(10):5242–5249, 2020.
M. C. M. Cheung, M. Rempel, A. M. Title, and M. Schüssler. Simulation of the Formation of a Solar
Active Region. Astrophys. J., 720(1):233–244, Sep 2010. doi: 10.1088/0004-637X/720/1/233.
M. C. M. Cheung, M. Rempel, G. Chintzoglou, F. Chen, P. Testa, J. Martı́nez-Sykora, A. Sainz
Dalda, M. L. DeRosa, A. Malanushenko, V. Hansteen, B. De Pontieu, M. Carlsson, B. Gudiksen, and S. W. McIntosh. A comprehensive three-dimensional radiative magnetohydrodynamic
simulation of a solar flare. Nat. Astron., 3:160–166, Jan 2019. doi: 10.1038/s41550-018-0629-3.
MCM Cheung, M Rempel, M Schüssler, et al. Simulation of the formation of a solar active region.
The Astrophysical Journal, 720(1):233, 2010.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990–2999. PMLR, 2016.
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral cnn. In International Conference on Machine Learning, pp.
1321–1330. PMLR, 2019.
Jose Carlos del Toro Iniesta and Basilio Ruiz Cobo. Inversion of the radiative transfer equation forpolarized light. Living Reviews in Solar Physics, 13(1):4, Nov 2016. ISSN 1614-4961. doi: 10.
1007/s41116-016-0005-2. URL https://doi.org/10.1007/s41116-016-0005-2.
Sander Dieleman, Kyle W Willett, and Joni Dambre. Rotation-invariant convolutional neural networks for galaxy morphology prediction. Monthly notices of the royal astronomical society, 450(2):1441–1459, 2015.
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convolutional neural networks. In International conference on machine learning, pp. 1889–1898.
PMLR, 2016.
Robert Gens and Pedro M Domingos.
Deep symmetry networks.
In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates,
Inc., 2014.
URL https://proceedings.neurips.cc/paper/2014/file/
f9be311e65d81a9ad8150a60844bb94c-Paper.pdf.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
Lars Hertel, Julian Collado, Peter Sadowski, Jordan Ott, and Pierre Baldi. Sherpa: Robust hyperparameter optimization for machine learning. SoftwareX, 12:100591, 2020. ISSN 2352-7110. doi:
https://doi.org/10.1016/j.softx.2020.100591. URL https://www.sciencedirect.com/
science/article/pii/S2352711020303046. arXiv:2005.04048 [cs.LG].
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
Thomas R Metcalf, Litao Jiao, Alexander N McClymont, Richard C Canfield, and Han Uitenbroek.
Is the solar chromospheric magnetic field force-free? The Astrophysical Journal, 439:474–481,
1995.
KG Puschmann, B Ruiz Cobo, and V Martı́nez Pillet. The electrical current density vector in the
inner penumbra of a sunspot. The Astrophysical Journal Letters, 721(1):L58, 2010.
A Asensio Ramos and CJ Dı́az Baso. Stokes inversion based on convolutional neural networks.
Astronomy & Astrophysics, 626:A102, 2019.
6

Published as a conference paper at ICLR 2022

Matthias Rempel. Numerical sunspot models: robustness of photospheric velocity and magnetic
field structure. The Astrophysical Journal, 750(1):62, 2012.
B Ruiz Cobo and JC del Toro Iniesta. Sir: Stokes inversion based on response functions. Astrophysics Source Code Library, pp. ascl–1212, 2012.
Peter Sadowski and Pierre Baldi. Deep learning in the natural sciences: applications to physics.
In Braverman Readings in Machine Learning. Key Ideas from Inception to Current State, pp.
269–297. Springer, 2018.
Marysia Winkels and Taco S Cohen. 3d g-cnns for pulmonary nodule detection. arXiv preprint
arXiv:1804.04656, 2018.

7

Published as a conference paper at ICLR 2022A

A PPENDIX

A.1

A RCHITECTURE A BLATION S TUDY

We perform an ablation study on the D4 -CNN trained on the sunspot region to measure the relative
contributions of the different components of the model (Table 3). We focus on the contributions
from G-convolution on different groups (D4 or C4 ) and G-pooling layers. A G-convolution layer
simply results in kernels and feature maps coming in groups of 4 or 8 corresponding to either the 4rotations in C4 , the Cyclic group, or 8 rotations and reflections in D4 (Figure 3). A G-pooling layer
pools over all rotated and reflected versions of feature maps from the same kernel.
Original

270°

G-Convolution

90°

180°

Figure 3: An example 3 × 3 kernel at the initial layer and its rotation in G-convolution on the C4group.
First, we replace all G-convolution layers equivariant in D4 with G-convolution layers equivariant
in C4 . This results in the model being equivariant to rotations, but not reflections. Next, we remove
all G-pooling layers except for one at the last layer — this results in equivariant but non-invariant
intermediate layers, despite the model as a whole retaining equivariance. We test this for the fullD4 model as well. Lastly, we remove all G-pooling layers completely, resulting in the model not
retaining equivariance.
Property
HeightTemperature
Pressure
Velocity Z
Mag. field Q
Mag. field U
Mag. field V
Average

km
K
Pa
m/s
gauss
gauss
gauss
NA

D4 -CNN
1.14%
1.00%
6.15%
38.31%
23.61%
22.06%
22.74%
16.43%

D4 -CNN −P
1.14%
1.00%
6.00%
38.60%
25.78%
24.28%
24.51%
17.33%

Architectures
C4 -CNN C4 -CNN −P
1.12%
1.13%
1.00%
0.99%
6.06%
5.89%
37.40%
37.11%
23.30%
24.54%
21.56%
23.21%
22.43%
23.19%
16.12%
16.54%

C4 -CNN −PP
1.13%
0.95%
5.90%
37.05%
25.89%
24.40%
23.95%
17.04%

Table 3: Relative error on the sunspot region testing set comparing a full D4 -CNN with grouppooling at each layer, a D4 -CNN with group pooling only at the output layer indicated by −P, a
C4 -CNN with group pooling at each layer, a C4 -CNN with group pooling only at the output layer,
and a C4 -CNN without group pooling indicated by −PP.
When removing G-pooling at intermediate layers we need to account for the difference in trainable
parameters between models. Compared to G-convolution in D4 using G-pooling, G-convolution
layers without G-pooling result in feature maps having 8 times the number of input channels. Thus,
keeping the number of unique filters the same between layers results in G-convolution layers without
G-pooling having 8 times the number of trainable parameters. To make a fair comparison, we needto divide the number of filters in each of the G-convolution layers without G-pooling by the square
8

Published as a conference paper at ICLR 2022√
root of the group size as suggested in Cohen & Welling (2016). The result is that there
are
1/
8
√
times the number of filters in G-convolution layers without pooling and therefore 1/ 8 times thenumber of output feature maps. Composing layers in this fashion results in 1/8 times the number oftrainable parameters in each G-convolution layer without pooling. Thus, when G-pooling is not used
the number of trainable parameters between models remains constant, allowing for fair comparisons.
The results indicate that the largest performance improvement over standard CNNs come from using
rotation-equivariant features in the C4 -CNN. The use of final pooling layers in C4 -CNN −P and theuse of intermediate pooling layers in D4 -CNN and C4 -CNN improve performance to a lesser extent
in comparison to their non-pooling counterparts. We expected the premature invariance achieved byintermediate G-pooling layers to reduce performance as found in Cohen & Welling (2016). However, the observed improvement may be due to the reduction in the number of filters in models notusing G-pooling at intermediate layers. Surprisingly, enforcing equivariance in D4 rather than C4slightly decreases performance; we suspect this is not significant and the result of using a small
training set.

9