Bessel Equivariant Networks for Inversion of Transmission Effects in Multi-Mode Optical Fibres
Authors: Joshua Mitton
Simon Peter Mekhail
Miles Padgett
Daniele Faccio
Marco Aversa
Roderick Murray-Smith
Venue: NeurIPS 2022 Accept (2022)
Keywords: physics
physics informed machine learning
inverse problems
optical fibre
optics
fibres
multi-mode fibre
equivariance
group theory

Abstract: We develop a new type of model for solving the task of inverting the transmission effects of multi-mode optical fibres through the construction of an $\mathrm{SO}^{+}(2,1)$-equivariant neural network. This model takes advantage of the of the azimuthal correlations known to exist in fibre speckle patterns and naturally accounts for the difference in spatial arrangement between input and speckle patterns. In addition, we use a second post-processing network to remove circular artifacts, fill gaps, and sharpen the images, which is required due to the nature of optical fibre transmission. This two stage approach allows for the inspection of the predicted images produced by the more robust physically motivated equivariant model, which could be useful in a safety-critical application, or by the output of both models, which produces high quality images. Further, this model can scale to previously unachievable resolutions of imaging with multi-mode optical fibres and is demonstrated on $256 \times 256$ pixel images. This is a result of improving the trainable parameter requirement from $\mathcal{O}(N^4)$ to $\mathcal{O}(m)$, where $N$ is pixel size and $m$ is number of fibre modes. Finally, this model generalises to new images, outside of the set of training data classes, better than previous models.

Bessel Equivariant Networks for Inversion of Transmission Effects in Multi-Mode Optical Fibres
Joshua Mitton"
Simon Peter Mekhail
School of Computing Science
School of Physics & Astronomy, Simon. Simon.Mekhail www.eeeeeglllaao......oooo ac. ac.
j mimtton..... 18research. waarrcch.g...a.. gla.ac.aa.......
Miles Padgett
Daniele Faccio
School of Physics & Astronomy, Miles. Padgett@@geet ac. ac. uk
School of Physics & Astronomy,
Daniele Daniele.Faccionn Faccio@glasgov ac uk
Marco Aversa
Roderick Murray-Smith
School of Computing Science
School of Computing Science
marco. aversa@glasgov.. .ww.ww.cc...o ac uk
Beterrickmuuray-aar sainrraa-smtthhglasssoo th@glasgoww...omooo ac.
University of Glasgow, Glasgow, Scotland, UK.
Abstract
We develop new type of model for solving the task of inverting the transmis- sion effects of multi-mode optical fibres through the construction of SO * (2. 1)- equivariant neural network. This model takes ad advantage of the of the azimuthal correlations known exist in fibre speckle patterns and naturally accounts for the difference in spatial arrangement between input and speckle patterns. In addition, we use a second post-processing network to remove circular artifacts, fill gaps, and sharpen the images, which is required due to the nature of optical fibre trans- mission. This two stage approach allows for the inspection the predicted im- ages produced by the more robust physically motivated equivariant model, which could be useful which produces high quality images. Further, this model scale to previously unachievable resolutions of imaging with multi-mode optical fibres and demon- strated 256 x 256 pixel images. This is result of improving the trainable parameter requirement from O(N))
safety-critical application, or by the output of both models,
) to O(m), where N is pixel size and m is number of fibre modes. Finally, this model generalises to images, outside of the set training data classes, better than previous models.
Introduction
Multi-mode fibres (MMF) have many potential applications in medical imaging, cryptography, and communications. In the medical domain, the use of multi-mode fibre imaging has potential to create hair-thin endoscopes for imaging sensitive the fibre transmission properties must be compensated for to return a clear image (Stasto P0L A different velocity. This leads
of the body. However, achieve these applications,
MMF has multiple different fibre modes, each of which propagates to an amplitude and phase mixing of the image as it propagates through the fibre (Mitschke, 2016). As result, an input image creates a complex-values speckled pattern on the output of the MMF. The ability to accurately and in a scalable way learn to invert the transmission effects would unlock MMF imaging as a useful tool across a range domains. This work concerns use of a single mudti-mode fibre and not fibre bundles.
MoitooshuaMitton
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Inverting a speckled image is challenging for multiple reasons. Firstly, the speckled images have a non-local relationship with respect to the original images. As a result, solely local patch-based models, such as convolutional neural networks, do not make sense solution without some dense mapping function. Therefore, the non-locality necessitates mapping the speckled images into a spa- tial arrangement similar to the original images before typical image-based deep learning techniques can be used, such as convolutions and pooling. In addition, the speckled images have circular corre- lation, which could be taken advantage of, although as noted by Moran et all (2ITS) finding these equivalent
requires solving the inversion, creating a chicken-and-egg problem. Finally, the fibre is e to unknown complex transmission matrix (TM) so, the inverse of this could be found using complex- valued linear model, although this presents challenges in terms of memory requirements. A mapping between 350 x 350 original and speckled images would result in TM with 350 x 15 billion entries, requiring a linear model with as many parameters.
Previous work in inverting the transmission effects of MMFs has either required extensive experi- mentation to characterise the TM of the fibre (Cižmár & Dholakia 2011 2012: Choi et all, 2012
Mahalati all 201: Papadopoulos et all 2012: Plöschner 2015; Leite atl 2021), where the number of experimental measurements required for re re-calibration was reduced by al (202T) by exploiting sparstiy in the TM; made use of dense linear models (Moran et all 2018; Fan el all, 2019% Caramazza et all, 2019;; or made use of convolutional models (Borhani et all, 2018; Rahmani et 2018). For the machine learning approaches to tackling the inversion task, those which make use of a dense linear model (Moran et al. 20R: Fan et all 2019: Caramazza et an 2019) can naturally account for the difference in spatial arrangement between speckled and original images, although they scale badly with the resolution of the images considered (O (O(N)) for N x N resolution). On the other hand, in theory, the convolutional neural network (CNN) models (Borham el all 2018 Rahmani et all 2013 improve upon the scalability issue, but in practice due to the need to approxi- mate the transmission matrix and its non-local effects, the models require a large number of layers in order to be able to effectively map every pixel in the speckled image every pixel in the original image, and hence in practice do not over come the scalability issues. All of these approaches are mostly expected to work for classes of objects that belong to the class that was used for training (Borhani et all 2018), with Rahmani et all (2018) making initial steps towards general imaging and Caramazza et (2(T)) demonstrating this for more diverse testing dataset. We provide further details on each of the previous methods in Appendix A.
In this work we present model which naturally accounts for the difference in spatial arrangement between speckled and original images and scales more efficiently than previous methods to higher resolution images. We believe this is the first method to demonstrate ability to invert 256 x 256 pixel speckled images into 256 x 256 pixel original images. Our approach also takes advantage of the circular correlations in the speckled images, and improves upon previous general imaging results. Concerning the equivariance literature, we develop a model comprising of cylindrical harmonic basis functions, basis set which has seen little attention in the equivariance literature, and make the connection between the transmission of light through a fibre and the group theoretic understanding used in developing equivariant neural networks. Our contributions are:
1. A more data-efficient, scalable model to solve the inversion of MMF transmission effects.
2. A model that provides better generalisation to out-of- of-tiaiiinn domain images.
3. A connection between group theoretic equivariant neural networks and the inversion of MMF transmission effects, providing new type of model to tackle the problem.
2 Background
2.1 Multi-Mode Fibres
Multi-mode fibres present a clear advantage over single-mode fibre bundles due to having 1-2 orders of magnitude greater density of modes than a fibre bundle (Choi all 2012). Despite this, the different propagation velocities of each mode, which result in the fibre producing scrambled images, presents a significant challenge. If each propagation mode and velocity was known the TM could be computed, providing a linear system that inverts the transmission effects, although in general this is not known. Further, mode specific losses and imperfect mapping between the input pixels and fibre modes can lead to information loss such that inverting the transmission does

---

not yield the true original input image. as is demonstrated in Figure II Therefore, model which correctly models the physics of inverting the transmission effects should produce the inverted image in Figure 0 and producing the original image requires addition information to be learned. We discuss the propagation of light through optical fibres and how we construct theoretical TMs in Section 2np and further details on the inversion of the TM in Appendix A.!
(a) Original
(b) Speckled
(c) Inverted
Figure 1: (a) Original image. (b) Speckled image from passing original images through theoretical
TM. (c) Inverted image created by passing speckled images through the inverted theoretical TMM.
2.2 Generation of Theoretical Transmission Matrices
Light propagation through MMF is characterised by the transmission modes of the fibre. The fi- bres considered this work have a step index profile with the refractive index within the core being constant and at a higher value than the cladding. Analytical solutions for the fibre modes can be determined by solving the Helmholtz equation cylindrical coordinates (details Appendix (.I.). The problem can therefore be expressed as an eigenvalue eigenfunction problem, where the eigen- functions are the fibre modes and the eigenvalues are the propagation constants, 3, of the modes. The solutions for the electric field within the fibre are comprised of Bessel functions of the first kind inside the core and modified Bessel functions of the second kind the cladding as follows:
Mrore (r. = Ji(u(B) fi) eii Ji(u(3))
pelad (r, = Ki(w(B) fi)
Ki(w(B)) e 1it0
(1)
where and O are the radial and azimuthal coordinates, respectively, / is the azimuthal index of the mode, the fibre core radius, and u and w are normalised frequencies defined as follows:
(B3) = R.nngoorr Pr..
w( RR B
k alad:
where Thcoee and Relad are the refractive indices of the core and cladding, respectively, ko is the vacuum wave number, and m is the radial mode index. Further details on the connection between propagation of light through MMFs and group theoretic equivariant neural networks are provided in Appendix A. Taking the derivative of Equations II, and equating the resulting functions asserts a smoothness condition on the electric field across the core-cladding boundary. Solving this for all values of 3 over all possible integer values of gives the propagation constants. These constants correspond to the fibre modes given by the now appropriately parameterised Equations Examples of the mode fields given in Figure Q
diagonal fibre propagation matrix, can then be made from the eigenvalues, 8, of the Helmholtz problem and the corresponding eigenfunctions can be recorded in matrix, M, which maps the image space to the fibre mode, or group space. We can find the inverse mapping bases from the fibre or group space back to the image space, M by taking the conjugate transpose. These three components allow us construct the transmission matrix as: TM MPM
2.3 Equivariance
Equivariance in neural networks concerns the study of symmetries and uses these build an induc- tive bias into model (Cohen & Welling 2016% [Bronstein et all 2021). Equivariance as a prop- erty of a neural network guarantees that transformation of the input data produces a predictable transformation of the predicted features (Worrall et all, 2017). Formally, we say that, given two transformations and 7" which are linear representations of group G and network or layer $ the network is equivariant if applying transform T3 to an input and then passing it through the

---

be chosen. To use Figure 2: Electric field amplitude profiles of the Bessel bases used within model. Here red is positive, blue negative and the colour saturation represents the electric field strength. These bases often known LP modes.
network & yields the same result as first passing x through &a and then transforming by T (Cohen & Welling 2016a); that is,
(T) (x)))
Convolutional neural networks (CNNs) are an early example of equivariance on images, where by design they are translationally equivariant (LeCunn et all, (198). More recently, equivariance has been considered for other group actions than that of the translation group. The majority works on images involve incorporating rotation reflection equivariance into the model, where variety of different group transformations have been considered (Cohen Welling, 2016a Weiler Cesa. 201; Murugan et af, 201% Wiersma et all 2020, De Haan et all 2020: Franzen & Wand. 2021).
We are not interested developing a group equivariant convolutional model here due the non- similar spatial arrangements between the speckled and original image domains. Despite this. the concept equivariance still applies, as the concept of learning model from fixed basis set, which guarantees symmetry properties are conserved, is relevant due to the nature of transmission through multi-mode optical fibres.
2.4 Circular Harmonies
Of the works that incorporate rotation equivariance into CNNs we are particularly interested in those using a continuous rotation group SO(2). the case of discrete rotation groups different feature space is associated each rotation angle. Storing features for an infinite number of rotation angles is not computationally tractable. One option to turn to a Fourier representation, whereby instead of choosing discrete number of rotations maximum frequency a Fourier representation for the weights of the model the domain IR? is decomposed into a radial profile and angular function. Solving for the kernel space of permissible filters that can be used within a CNN and ensuring that the model maintains rotation equivariance yields only the spectrally localised circular harmonics (Worrall all 2017: Weiler & Cesa 201% |ranzen & Wand, (021).. Therefore a rotation equivariant CNN can be created by solving for the circular harmonics up to a certain rotational frequency, combining this with radial profile function, and sampling basis of resolution given by the CNN filter size. This yields a set of bases which can be linearly combined with learnable weighting applied to each to form the kernel for the CNN.
2.4.1 Cylindrical Harmonics
While the circular harmonics have seen some attention in the deep learning community due to the use in constructing rotational equivariant CNNs, the cylindrical harmonies have seen less attention (Kliepera et all, 2020). The cylindrical harmonics appear as solution to Bessel functions for integer and are therefore of interest for problems where information transformation is characterised by such functions. For example Bessel functions are used when solving wave heat propagation.

---

Bessel functions are solutions for different complex numbers a of Bessel's ' differential equation:
d'
dy
x²
+ aa)- dx
(4)
dx²
For integer values of a the Bessel function solutions are a linearly independent set of functions ex- pressed cylindrical coordinates. Each function consists of the product of three functions. The radially dependent term is typically called the cylindrical harmonics. Further details the connec- tion between group theory and the Bessel basis functions provided in Appendix A.
Our Method
Our method comprises two models: a Bessel equivariant model and convolutional post-processing model. The Bessel equivariant model uses basis set comprising cylindrical harmonics and learns complex mapping function to use the known symmetries in optical fibres. The Bessel equivariant model is constrained by the basis choice to produce circular images, which can have circular artifacts also seen in a true inversion, see Figure M1 The post-processing model is used to fill gaps in the corners, remove circular artifacts, and sharpen the images. The two stage approach is useful in safety- critical applications users can inspect the output of both models, where the Bessel equivariant model output is less likely to depend on the training data due to the choice of inductive bias.
3.1 Bessel Basis Equivariant Model
The core of our model has an inductive bias that better replicates the physics of the task of inverting the transmission effects of MMF imaging. To achieve this we utilise prior knowledge of MMFs and the modes with which an image propagates through the fibre. Similar to the successes of rotation equivariant neural networks, which make use of circular harmonics achieve rotation equivariance, we utilise cylindrical harmonics as an inductive bias of the model. This is motivated by the prop- agation modes of the fibre being characterised by Bessel functions, where the radially dependent solution is given by the cylindrical harmonics. We make the connection to the group theoretic devel- opment of equivariant neural networks in Appendix IAI, demonstrating the connection between the group SOT (2, 1), the light cone, and Bessel function solution to the wave equation.
The first stage the model is the computation of two of basis functions. The first set transforms the input image into function that lives on the group and the second transforms back into the image domain. The model's weight matrix linearly transforms the function mapped by the first basis set. The basis filters are computed in accordance with equations in Appendix 2,2
These functions are sampled set and the original images for the second basis set. An example the bases produced is given in Figure A The Bessel function bases are computed in pre-training stage and therefore this step only has to be completed once when creating the model. Further, these hases are not trainable parameters and
grid given by the size of the speckled images for the first basis
updated during training of the model. These basis functions to those used in general rotation equivariant neural networks, which typically offer equivariance to the group SO(2), except they correctly model the symmetry group, SO* (2, 1) of the task of inverting transmission effects of a MMF due to the added time dimension.
alternative
The model trainable parameters are complex-valued valued weights of size equal the number of bases. The complex weight matrix is a diagonal matrix. The images transformed by the first basis set are multiplied by the weight matrix before being transformed by the second basis set to produce the predicted image. The overall model architecture is given in Figure C
3.2 Post-Processing Model
As demonstrated in Section A the limited number of modes means that the transmission effects of a MMF are typically not fully invertible leading to information loss. We would therefore not expect an equivariant model informed by the physics of the task to be able to fully invert the transmission effects. To overcome this we add second model which similar to a super-resolution model, except that it does not change the resolution and instead learns to remove circular artifacts, fill in gaps, and sharpen the images. It is a fully convolutional model comprising of convolutional layers and non- linearities. This model takes as input the output of our Bessel equivariant model, which we suspect
5

---

Speciled Image Resclution Besse)
Complex Weight Matrix
Ousput Image Rescution Bessel Panction Bases
Post Peet-Proossing Model
Function Bases
Our two stage model architecture comprising of Bessel equivariant model and post- processing model. Input (left) is speckled images. Output (right) predicted image. The speckled image resolution Bessel function bases transform the speckled image into mode space of the group SO (2, 1). The complex weight matrix is diagonal. The original image resolution Bessel function basis provides mapping from the mode space to the output image space. The trainable parameters of the Bessel equivariant model are the diagonal complex weight matrix only. The post-processing model is a convolutional model.
Figure 3:
will be similar what is achievable by passing the speckled images through an inverted TM, and predicts the original images. This model is composed of eight convolutional layers with ReLU non- linearities. The motivation behind the explicit separation of the inversion and enhancement models that in some applications, especially safety-critical applications, important to be able to see the results of inversion conditioned on the observations alone, regardless of prior expectations.
4 Experiments
We investigate performance with both data generated with a theoretical TM and experimental data collected using real MMF fibre. We provide details of the theoretical TM generation in Section 2.2 We use the data from a real fibre, as described in Moran et all (2018), to validate that our new approach model the same data, despite using significantly fewer trainable model parameters.
Further, we generate data using a theoretical TM as this allows us to flexibly adapt the parameters of the experiment and create new datasets validate the model design. Finally, the theoretical TM allows us to experiment with images with higher resolution than has been previously been considered. We provide details on the training of the models and datasets Appendix
4.1 Real Multimode Fibre
We compare our model to the complex-valued linear model developed by Moran et all (2018) by considering both MNIST and fMNIST data. For this train and test separate models for the MNIST and fMNIST datasets. In addition, we compare multiple different versions of our model where the diagonal restriction of the mapping between bases is relaxed to allow for a block diagonal structure. We do this to account for manufacturing defects, sharp bends, dopant diffusion, elliptical cores which causes the diagonal mapping of the fibre matrix within the TM to be block diagonal (Carpenter et all 2014). This relaxation allows for I-offfets above and below the main diagonal.
Table 01 shows that our model with full mapping matrix between bases, i.c. full relaxation of the di- agonal constraint, outperforms all other models. In addition, with a relaxation to allow for a 10 block diagonal structure our model performs comparably. On the other hand, the complex linear model outperforms our Bessel equivariant model when the diagonal restriction on the mapping function be- tween bases is enforced. Despite this, our model still provides clear and accurate reconstructions of
Table I: Comparison of the loss values of each model trained with MNIST IMNIST data.
MNIST
IMNIST
Model
Train Loss
Test Loss
Train Loss
Test Loss
Complex Linear
0.00396
0.00684
0.00509
0.01061
Bessel Equivariant Diag
0.03004
0.03125
0.02903
0,03140
Bessel Equivariant Diag Post Proc
0.01317
0.01453
0.01609
0.01749
Bessel Equivariant 10 Off Diag Post Proc
0,00488
0.00576
0.00943
0,01162
Bessel Equivariant Full
0.00300
0.00684
0.00548
0.01380
Bessel Equivariant Full Post Proc
0.00145
0.00378
0.00306
0.00964

---

9
9
9
9
(a) Input
(b) Target
(c) BEM
(d) BEM+PP
(c) Complex
Figure 4: Comparison of predicted images from inverting transmission effects of a MMF. The upper row is fMNIST data, lower row MNIST data. (a) Input speckled image, (b) the target original image to reconstruct, (c) Output of the Bessel equivariant model, (d) Output of the combination of Bessel equivariant and post- post-processing model, and (e) the output of the complex-valude linear model.
the target images whilst using orders of magnitude fewer trainable parameters. Additional results are presented in Tables R and Z Further to comparison of the loss values, we visually inspect the recon- struction quality in Figure Hl, and we provide further visualisations in Figures IG and DT. Thus despite our Bessel equivariant diagonal model achieving a larger loss value, the digit item is still clearly visible in the prediction and is correctly sharpened to a realistic prediction of the original image by our post-processing model. The complex linear model can also be seen predict a realistic looking output of the correct digit and object, although some noise does feature in the prediction. In addition, we visually compare each of our Bessel equivariant models with the diagonal restriction relaxed in Figures III and III which demonstrates that as the diagonal restriction is relaxed the model can better predict the original images with less noise in the prediction from the Bessel equivariant model.
We also compare the number of trainable parameters within the model in Table D This highlights that model requires orders of magnitude fewer parameters that the complex linear model when using the diagonal Bessel equivariant model. Therefore, our model has the potential to scale to higher resolution images where the complex linear model will run out of GPU memory. As the diagonal mapping between Bessel bases is relaxed to include 10-block diagonal structure the model still requires two order of magnitude fewer parameters. In the most flexible version of our model the number of trainable parameters is comparable with the complex Linear model, although we have demonstrated that this level of flexibility not required to achieve good image reconstruction.
Table 2: Comparison of trainable parameters in each model trained with MNIST IMNIST data.
Model
Number of Trainable Parameters (Millions)
Complex Linear
78.826
Bessel Equivariant Diag + Post Proc
0,500
Bessel Equivariant 10 Off Diag Post Proc
0.617
Bessel Equivariant Full + Post Proc
43.108
4.2 Theoretical TM
We compare our model the complex linear model developed by Moran et all (2018) with data created using a theoretical TM. We create multiple datasets one using 28 x 28 IMNIST images, with 180 x 180 speckled images, another with MNIST images, of the same size, and one with subset of images from the ImageNet dataset (Deng et all 2009) where we fix the resolution to be 256 x 256 of both the images and speckled images. Using fMNIST and MNIST with theory TM useful for developing understanding, due to the ease of creating different experiments and datasets, and for testing the generalisability of the model between the two datasets. We include the ImageNet-based dataset as these are higher resolution images containing more challenging information than FMNIST. To the best of our knowledge, demonstrating the ability to invert transmission effects of such high resolution images has not been previously achievable with a machine learning based approach.
7

---

Table Comparison of the loss values of each model trained with FMNIST data.
fMNIST
MNIST
Model
Train Loss
Test Loss
Test Loss
Complex Linear
0.0149
0,0146
0,0363
Bessel Equivariant
0.0141
0.0139
0.0026
Bessel Equivariant + Post Proc
0.0032
0.0032
0.0028
4.2.1 Generalisability of Models
We provide a comparison of the predictions of a complex valued linear model (Moran et all 2018). our equivariant model only, and our equivariant model coupled with post-processing model trained on IMNIST images and tested on both IMNIST and MNIST to analyse to what extent the models can generalise to new dataset. Table B presents the loss values for both the training and testing of each model. Considering the IMNIST data this shows that the complex linear and Bessel equivariant models achieve similar loss values. Although this is result of the Bessel equivariant model under- performing due to not being able to reconstruct pixel values close to the edge of the image as consequence of the circular nature of the Bessel bases functions, while the complex linear model due to failing to reconstruct higher frequency information and over-fitting to the general clothing categories. Finally, when our Bessel equivariant model combined with the post- processing model it outperforms all other models. The ability to generalise to a new data is assessed through the MNIST dataset. This shows that our Bessel equivariant model significantly out-performs the complex linear model in generalising to a new dataset.
Figure shows reconstructions produced by the three models, with further reconstructions provided in Figure IR This demonstrates visually that our model, even without the pooseprocsssing part, is able to better reconstruct the original images. The post-process post-processing model refines the output of the equivariant model and fills in gaps outside the fibre. Despite it being possible to estimate the general category of clothing from the outputs of the complex linear model, the higher frequency information has been lost, i.e. the pattern on the jumper or the curve in the trouser leg. On the other hand, our model is able reconstruct this higher frequency information.
Previous works have demonstrated some ability to generalise to new data classes (Rahman et all 2018; Caramazza allll 2019), although in each case the results are not perfect. Here, we compare the different methods trained on IMNIST and then tested on MNIST data. Figure l demonstrates that our Bessel equivariant model generalises well to new data domain, with further reconstructions provided in Figure Ig. On the other hand, the complex linear model somewhat predicts the correct digits. In this out of training domain our post-processing model does not add any value to the original Bessel equivariant model. This highlights the benefit of our two-stage modelling approach as we have one robust model and a second model to sharpen the images, as result if user believes the situation could be unusual they could reliably use the output of our Bessel equivariant model and not the post-processing part. Further to achieving strong generalisation results, we also consider the effect of noise in the speckled images in Appendix A.A.
Figure & shows how the number of trainable model parameters scale. Even for 28 x 28 images our model requires two orders of magnitude fewer trainable parameters than the complex linear model.
(a) Input
(b) Target
(c) BEM
(d) BEM + PP
(c) Complex
Figure 5: Comparison of predicted images from inverting transmission effects of MMF. (a) Input speckled image, (b) Target original image to reconstruct, (c) Output of the Bessel equivariant model, (d) Output of the Bessel equivariant and post-processing model, and (c) Output of the complex valued linear model.
8

---

7
7
7
(a) Input
(b) Target
(c) BEM
(d) BEM PP (e) Complex
Figure 6: Comparison of predicted images from inverting transmission effects of MMF. (a) Input speckled images, (b) Target original images, (c) Output of the Bessel equivariant model, (d) Output of Bessel equivariant and post-processee processing model, and (e) Output of the complex valued linear model.
4.2.2 Scaling to Larger Images ImageNet
We now experiment with a subset of images from the ImageNet dataset, where we fix the resolution to be 256 x 256. Inverting the transmission effects of such higher resolution images has not been previously achievable. We only compare our equivariant model with our equivariant model coupled with fully convolutional post-processing model. The complex-valee valued linear model cannot fit in the GPU memory of an A6000 with 48. 78BB of memory with this resolution of image.
(a) Input
(b) Target
(c) BEM
(d) BEM PP
Figure 7: Comparison of predicted images from inverting transmission effects MMF using high resolution ImageNet data. (a) Input speckled image.
(b) Target original image to reconstruct, (c) Output Bessel equivariant model (d) Output Bessel equivariant and post-processing model.
We analyse model visually, with Figure M showing that Bessel equivariant model produces a reconstruction from which one can determine the type of dog and its activity, although the recon- struction does capture all the high frequency information, and information in the corners is lost due to the circular nature of the fibre. Further reconstructions are provided in Figure 24 When we combine the two models some of the artifacts of the Bessel equivariant model are removed as the post-processing model fills in information towards the corners and sharpens the image.
We present the loss values for both models in Table g. This shows similar result as for fMNIST, that our Bessel equivariant model can solve the task well, but requires the post-processing model to sharpen the image and fill in gaps in the corners We compare the models' memory requirements in Figure S. Note how fewer trainable parameters translate into significantly less memory use than the complex linear model. This effect is more drastically when scaling to high resolution images, where the complex linear model runs out of memory on 24. 24.2GiB Titan RTX for original and speckled images of resolution 180 180 pixels. Our model, in contrast, has been tested 256 x 256 pixel images, requiring only 1.B. and can scale to megapixel images.
Finally, explore the impact each model of reducing the size of the training dataset. Requiring a reduced dataset size minimises the time taken in a lab collecting data. Figure & demonstrates that model outperforms the complex linear model with 10x-less training data, reducing the need to collect 12000 samples to 1200. The reconstructions are visualised in Appendix A.12
Table 4: Comparison of the loss values of each model trained with ImageNet data.
Model
Train Loss
Test Loss
Bessel Equivariant
0.0541
0.0574 0.0159
Bessel Equivariant Post Proc
0.0161

---

180x160 > 26x28
1500 2
180v180
256x256
512K512 > S12x512
1024x1024 1624x1024
180x1
1500200
2562256
Trainable Parameters
1012
Linear
2010
Complex Linear
Bessel Equivariant Full
10a
Equivariant
Dlog
10"
Be ssel Equivariant 5 Off Diag
Equivariant Diag
10
Bessel Equivariant: + PP
GPU Memory (Gib)
25
0.20
20
Test Loss
0.35
0.30
0.05
5-
0
10a
10a
10
10
1012
2000 4000 6000
10000 12000
Transmission Matrix Size
Train Dataset Size
Figure 8: (Upper Left) Comparison of number of trainable parameters TM size (dashed line indicates model cannot in practice be built due to large memory requirements). (Lower Left) Com- parison of required GPU memory against TM size. Vertical grey lines indicate given image size our model can process 1024 x 1024 images on a single consumer-level Titan RTX GPU. (Lower Right) Comparison of reducing the size of the training dataset. All plots for 1000 mode fibre.
Conclusions
We present a new type of model for solving the task of inverting the transmission effects of MMF through developing a SO+ (2, !)-equivariant neural network and combining this with a post- processing network. This improves upon complex linear models through incorporating useful physically informed inductive bias into the equivariant network. The equivariant network is shown perform well on new image classes, providing a general model for fibre inversion. The post- processing network is specific to an image domain, as it generalises to new image classes as well as a general convolutional network, although it improves the quality of returned images. The use of theoretical TM allows us to compare an 'ideal' inverse with the output of previous models based on learning a transmission matrix. This suggests that previous learned transmission matrices were combining elements of the actual transmission matrix with elements of the pest-processing network, which would affect their ability to generalise to new images. We also anticipate that interactive safety-critical applications users might want the ability to switch between modes, to be sure that the evidence was there, and not overly influenced by the priors in the training data.
This new model significantly improves the ability to scale to higher resolution images by improving the scaling law from O(N4) to O(m), where N is pixel size and m the number of fibre modes. We provide comparison between our model and prior works on both data created using real fibre and theoretical TM, demonstrating in both cases that our model solves the task while using significantly fewer trainable parameters than the complex and real linear models. In addition, we demonstrate results on high resolution 256 x 256 images, which has previously been unachievable due to the growth of parameters with previous models. Furthermore, we demonstrate the ability of our model to generalise to new data classes outside of the training data, outperforming prior works. The dramatic reduction in the number of parameters for each fibre configuration opens the way for future models which can learn mappings for high resh-resolution images, from a wider set of perturbed fibre poses and combine these using architectures such as VAEs.
Acknowledgments and Disclosure of Funding
JM is supported by University Glasgow Lord Kelvin Adam Smith Studentship. RM-S, MP. DF and SPM are grateful for EPSRC support through grants EP/T00097X/1 and RM-S for OP/RO18634/1. MA funded by dotPhoton and UofG scholarship. DF acknowledges funding through the Royal Academy of Engineering Chair in Emerging Technologies programme.
10

---

References
Navid Borhani, Eirini Kakkava, Christophe Moser, and Demetri Psaltis. Learning to see through multimode fibers. Optica, 5(8):960-966, 2018.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
Piergiorgio Caramazza, Oisín Moran, Roderick Murray-Smith, and Daniele Faccio. Transmission of natural scene images through a multimode fibre. Nature communications, 10(1):1-6, 2019.
Joel Carpenter, Benjamin J Eggleton, and Jochen Schröder. Maximally efficient imaging through multimode fiber. In CLEO: Science and Innovations, pp- STh1H-3. Optical Society of America, 2014.
Youngwoon Choi, Changhyeong Yoon, Moonseok Kim, Taeseok Daniel Yang, Christopher Fang- Yen, Ramachandra R Dasari, Kyoung Jin Lee, and Wonshik Choi. Scanner-free and wide-field endoscopic imaging by using single multimode optical fiber. Physical review letters, 109(20): 203901, 2012.
Tomáš Čižmár and Kishan Dholakia. Shaping the light transmission through a multimode optical fibre: complex transformation analysis and applications in biophotonics. Optics express, 19(20): 18871-18884 2011.
Tomáš Čižmár and Kishan Dholakia. Exploiting multimode waveguides for pure fibre-based imag- ing. Nature communications, 3(1):1-9, 2012.
Taco Cohen and Max Welling. Group equivariant convolutional networks. International confer- ence on machine learning. pp. 2990-2999. PMLR, 201
Taco S. Cohen and Max Welling. Steerable CNNs. arXiv preprint arXiv:1612.08490 2016b.
Pim De Haan, Maurice Weiler, Taco Cohen, and Max Welling. Gauge equivariant mesh CNNs: Anisotropic convolutions geometric graphs. arXiv preprint arXiv:2003.05444 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Icee, 2009.
Pengfei Fan, Tianrui Zhao, and Lei Su. Deep learning the high variability and randomness inside multimode fibers. Optics express, 27(15)22044 2019.
Daniel Franzen and Michael Wand. General nonlincarities in Equiivaqnnaaaiant CNNs. Advances in Neural Information Processing Systems, 34, 2021.
Johannes Klicpera, Janek GroB, and Stephan Günnemann. Directional message passing for molecu- lar graphs. arXiv preprint arXiv:2003.03123, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):227--233) 1998.
Ivo T. Leite, Sergey Turtaev, Dirk F. Boonzajer Flaes, and Tomáš Čižmár. Observing distant objects with a multimode fiber-based holographic endoscope. APL Photonics, 6(3):0361 12, 2021.
Shuhui Li, Charles Saunders, Daniel J Lum, John Murray-Bruce, Vivek K Goyal, Tomá Čižmár, and David B Phillips. Compressively sampling the optical transmission matrix of multimode fibre. Light: Science & Applications, 10(1):1-15, 2021.
Reza Nasiri Mahalati, Ruo Yu Gu, and Joseph M Kahn. Resolution limits for imaging through multi-mode fiber. Optics express, 21(2):1656-1668 2013.
Fedor Mitschke. Fiber optics. Springer, 2016.
Oisín Moran, Piergiorgio Caramazza, Daniele Faccio, and Roderick Murray-Smith. Deep, complex, invertible networks for inversion of transmission effects in multimode optical fibres. Advances in Neural Information Processing Systems, 31, 2018

---

Muthuvel Murugan, KV Subrahmanyam, and Siruseri IT Park. SO (2)-euuivariance in neural net- works using tensor nonlinearity. British Machine Vision Conference, 2019.
Ioannis N. Papadopoulos, Salma Farahi, Christophe Moser, and Demetri Psaltis. Focusing and scanning light through multimode optical fiber using digital phase conjugation. Optics express, 20(10):10583-10555 10590, 2012.
Martin Plöschner, Tomáš Tyc, and Tomáš Čižmár. Seeing through chaos in multimode fibres. Nature Photonics, 988):229-335 2015.
Babak Rahmani, Damien Loterie, Georgia Konstantinou, Demetri Psaltis, and Christophe Moser. Multimode optical fiber transmission with deep learning network. Light: Science & Applica- tions, 7(1):1-11, 2018.
Nicolino Stasio. Multimode fiber optical imaging using wavefront control. Technical report, EPFL. 2017.
Maurice Weiler and Gabriele Cesa. General equivvvrrat steerable CNNs. Advances in Neural Information Processing Systems, 32, 2019.
Ruben Wiersma, Elmar Eisemann, and Klaus Hildebrandt. CNNs on surfaces using rotation- equivariant features. ACM Transactions Graphics (TOG), 39(4):2-11, 2020.
M.K.F. Wong. Unitary representations of SO(n, 1). Journal of Mathematical Physics, 15(1):25-30, 1974.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Mushion-MNISTY novel image dataset for bench- marking machine learning algorithms. arXiv preprint 47X1v:7008.0747, 2017.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Sections 3.2 and A.4
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section A.14
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sec- tions 2 and A4
(b) Did you include complete proofs of all theoretical results? and A.
[Yes] See Sections
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section A
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] Due to compute constraints we did not run with multiple
random seeds, although we do note that training was very stable and so the results will be reproducible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section A.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section [.]
(b) Did you mention the license of the assets? [Yes] See Section Aw!
(c) Did you include any assets either in the supplemental material URL? Yes]]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] Consent provided by the authors who collected the data.
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [Yes] It does not contain any sensitive infor- mation.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "P1KOGy_Iy8J", "reviewer": "Reviewer_iftp", "summary": "This paper assumes that the transmission matrices (TMs) associated with multimodal fibers (MMF) are diagonizable using a Bessel Function Basis. Using this assumption, the authors propose a new learning-based technique for imaging through MMFs. The speckle image is first multiplied by the learned diagonal inverse TM and is then denoised by a post-processing CNN. The proposed method is tested on low-resolution toy (MNIST and FMNIST) experimentally captured data from (Mora et al. 2018). It is also tested on simulated TM data on higher-res image-net data.", "questions": "How would fitting a low rank approximation of the TM (or its inverse) compare to the proposed method?", "limitations": "Not discussed. Method is specific to a particular MMF. Moreover, if that fiber bends (like it would in endoscopy) that fibers TM is likely to change.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \n\nThe diagonal assumption massively reduced the number of parameters that need to be learned and could make imaging through MMF much easier. \n\n## Weaknesses\n\nThe paper makes extremely strong assumptions on the structure of the TM which are not adequately justified. The only experimental validation is on (F)MNIST which isn't convincing. The paper provides no evidence that Bessel functions accurately describe the structure of higher-dimensional TMs. The simulated speckle data (Fig 6) looks considerably more structured than the experimental data (Fig 5), suggesting the adopted model is inaccurate.\n\nSome details are missing. It seems the paper is considering only \"incoherent\" fiber bundles.\n\nThere's no evidence the proposed method generalizes across MMFs.\n\nThere are minor inaccuracies in the paper: Line 270 states reducing the calibration dataset size minimizes how long cells have to be imaged; cells are involved in the calibration of a TM.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HzDeNvC4t_h", "reviewer": "Reviewer_88iJ", "summary": "The paper reports on a novel NN architecture for image transmission through a multimode fiber. The approach is physics based, and rely on modeling a NN using bessel modes, that mimicks the pre-existing modes in a MMF. the main advantage is to reduce strongly the number of parameters to train (compared to a complex-valued fully connected layer, the current SoTA) and  the required number of examples, while maintaining similar performances and generalization ability. The concept is validated numerically and experimentally, at scale (256*256) previously unattainable. ", "questions": "-Modes of MMF are usually denoted LP modes in the literature, can the authors comments if they are the same as their Bessels? is it a different terminology, or is there a real difference? \n", "limitations": "I think it really doesn't apply here. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is overall clearly written, the results are sound, and the comparison to the previous litterature is qualitatively and quantitatively appropriate. The main advantage of the technique (less parameters) meaning less memory usage and less examples needed, means that it can scale to complex and large scale images, which is demonstrated. \n\nA minor criticism is that the authors could have cited a recent paper (Cite: Li, Shuhui, et al. \"Compressively sampling the optical transmission matrix of a multimode fibre.\" Light: Science & Applications 10.1 (2021): 1-15.) that also exploit the sparsity of the transmission matrix of a multimode fiber.  The paper in my opinion should be cited and properly put in perspective with their result.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4Jem9hOzTJu", "reviewer": "Reviewer_cTsz", "summary": "This paper proposes to use a bessel basis and a post-processing neural network for modeling the optical transmission through a multi-mode optical fiber. The key idea is that multi-mode fiber outputs are band-limited with respect to the bessel functions of different modes. Applying this representation for the multi-mode fiber output allows us to reduce the number of parameters significantly, reducing the computational burden of inverse estimation: from a speckle image to a target natural image.\n", "questions": "My concern mainly lies in the analysis of the representation capability of the bessel basis. How accurate the basis is for modeling the MM fiber outputs, especially w.r.t. the number of bases used? What are the remaining components that cannot be modeled? Probably visualizing the full weight matrix might be useful to answer these questions.\n\nSecond, many works using MM fibers for display applications modulate incident wavefront to generate high-fidelity images directly out of the fiber. Could the proposed method be also used for this application?", "limitations": "The authors adequately described the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- For the first time, a 256x256 image is reconstructed from a speckle image, thanks to the efficient parameterization.\n- Using the bessel representation is a neat idea for the neural modeling of the multi-mode fiber outputs.\n- The paper is well written.\n\nWeaknesses\n- The authors show that the bessel representation is effective in modeling the MM fiber output. However, there is no analysis on it's representation capability with respect to the number of bases. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept