Single-Photon Image Classification
Authors: Thomas Fischbacher
Luciano Sbaiz
Venue: ICLR 2021 Poster (2021)
Keywords: quantum mechanics
image classification
quantum machine learning
theoretical limits

Abstract: Quantum Computing based Machine Learning mainly focuses on quantum computing hardware that is experimentally challenging to realize due to requiring quantum gates that operate at very low temperature. We demonstrate the existence of a "quantum computing toy model" that illustrates key aspects of quantum information processing while being experimentally accessible with room temperature optics. Pondering the question of the theoretical classification accuracy performance limit for MNIST (respectively "Fashion-MNIST") classifiers, subject to the constraint that a decision has to be made after detection of the very first photon that passed through an image-filter, we show that a machine learning system that is permitted to use quantum interference on the photon's state can substantially outperform any machine learning system that can not.  Specifically, we prove that a "classical" MNIST (respectively "Fashion-MNIST") classifier cannot achieve an accuracy of better than $21.28\%$ (respectively $18.28\%$ for "Fashion-MNIST") if it must make a decision after seeing a single photon falling on one of the $28\times 28$ image pixels of a detector array.  We further demonstrate that a classifier that is permitted to employ quantum interference by optically transforming the photon state prior to detection can achieve a classification accuracy of at least $41.27\%$ for MNIST (respectively $36.14\%$ for "Fashion-MNIST"). We show in detail how to train the corresponding quantum state transformation with TensorFlow and also explain how this example can serve as a teaching tool for the measurement process in quantum mechanics.


Under review as a conference paper at ICLR 2021
SINGLE-PHOTON IMAGE CLASSIFICATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Quantum Computing based Machine Learning mainly focuses on quantum comput- ing hardware that is experimentally challenging to realize due to requiring quantum gates that operate at very low temperature. We demonstrate the existence of a "quantum computing toy model" that illustrates key aspects of quantum information processing while being experimentally accessible with room temperature optics. Pondering the question of the theoretical classification accuracy performance limit for MNIST (respectively "Fashion- -MNIST"" classifiers, subject to the constraint that decision has to be made after detection of the very first photon that passed through an image-filter, we show that a machine learning system that is permitted use quantum interference the photon's 's state can substantially outperform any machine learning system that can not. Specifically, we prove that a "classical" MNIST (respectively "Fashion-MNISTT") classifier cannot achieve accuracy of better than 22.96% (respectively 21 21.8% for "Fashion-MNIST") if it must make a decision after seeing single photon falling on one of the 28 x 28 image pixels of a detector array. We further demonstrate that a classifier that is permitted to employ quantum interference by optically transforming the photon state prior to detection can achieve a classification accuracy of at least 41.27% for MNIST (respectively 36.14% for "Fashion-MNISTT -NNIST"). We show detail how to train the corresponding quantum state transformation with TensorFlow and also explain how this example can serve as a teaching tool for the measurement process in quantum mechanics.
INTRODUCTION
Both quantum mechanics and machine learning play a major role in modern technology, and the emerging field of Al applications of quantum computing may well enable major breakthroughs across many scientific disciplines. Yet, as the majority of current machine learning practitioners do not have a thorough understanding of quantum mechanics, while the majority of quantum physicists only have an equally limited understanding of machine learning, it is interesting to look for "Rosetta Stone" problems where simple and widely understood machine learning ideas meet simple and widely understood quantum mechanics ideas. It is the intent of this article to present setting in which textbook quantum mechanics sheds a new light on a textbook machine learning problem, and vice versa, conceptually somewhat along the lines of Google's TensorFlow Playground (Smilkov et al. (2017).) which was introduced as teaching device to illustrate key concepts from Deep Learning to wider audience.
Specifically, we want to consider the question what the maximal achievable accuracy on common one-out-of-many image classification tasks is if one must make decision after the detection of the very first quantum of light (i.e. photon) that passed a filter showing an example image from the test set. In this setting, we do not have one-to-one correspondence between example images from the training (respectively test) set and classification problems. Instead, every example image defines probability distribution for the (x, y) detector pixel location on which the first photon passing an image filter lands, the per-pixel probability being the pixel's brightness relative to the accumulated (across all pixels) image brightness. So, from every (28 x 28 pixels) example image, we can sample arbitrarily many photon-detection-evel classifier examples, where the features are pair of integer pixel coordinates, and the label is the digit class.
On the MNIST handwritten digit dataset (LeCun and Cortes (0100), any machine learning sys- tem that only gets to see a single such "photon detected at coordinates (x,y)" event as its in- put features, of the pixel that flashed up are the only input features, is limited in accuracy

---

Under review as a conference paper at ICLR 2021
by the maximum likelihood estimate, since have: PIIagee class C|Photon detected at (a, EE P(Image class C|Example E) P(Example EJ Photon detected at (z,
On photon detection events generated each by first randomly picking an example image, and then randomly picking a brightness-weige weighted pixel from that, we cannot do any better than predicting the most likely digit class given these input features the two pixel coordinates. As performance is measured on the test set, no classifier could possibly ever outperform one that is built to achieve maximal performance on the test set. This is obtained by determining, for each pixel, what the most likely class is, where examples from the test set are weighted by the fraction of total example-image brightness that comes from the pixel in question. Figure 2b)) shows the most likely image-class per pixel. (For MNIST, some pixels are dark in every test set example.) No classifier can outperform one that simply looks up the coollccood at which a photon was detected in Figure b and returns the corresponding class, and this optimal classifier's accuracy is 22.96% for the the MNIST dataset substantially higher than random guessing (10%). Appendix | a detailed (but mostly straightforward) optimality proof of this accuracy threshold. We cannot, for example, outperform it by redistributing light intensity between pixels, since any such redistribution could only destroy some of the available useful information, not magically create extra useful information.
An entirely different situation arises when we allow quantum mechanics to enter the stage: For a single photon passing through coherently illuminated image filter, with all pixels at the same optical phase on the incoming wave, we can imagine putting some precision optical device between the image filter and the detector array that redistributes not the probabilities (which correspond to light intensity when aggregating over many photons), but the amplitudes that make up the spatial part of the photon wave-function. Illuminating such a set-up with many photons would show a hologram-like interference pattern on the detector array. This transformation of the (single-)photon wave function by linear optical elements then has tuneable parameters which we can adjust to improve classifier accuracy. Quantum mechanics tells us that every (lossless) linear optical device can be represented by a linear unitary transform on the photon state: The action of any complex optical device consisting of (potentially very many) components which transforms a N N-component photon state (in our case, N = 28 amplitudes in the spatial part of the photon wavefunction) can be described by an element of the 2 -d-ieennsio unitary matrix Lie group (NN). Vice versa, Reck et al. (1994) describes constructive al algorithm by which any UNN) transformation matrix can be translated back to a network of optical beam splitters and phase shifters.
1.1 RELATED WORK
Conceptually, exploiting interference to enhance the probability of a quantum experiment producing the sought outcome is the essential idea underlying all quantum computing. The main difference between this problem and modern quantum computing that the latter tries to perform calculations by manipulating quantum states of multiple "entangled" constituents, typically coupled two-state quantum systems called "qubits," via "quantum gates" that are controlled by parts of the total quantum system's 's quantum state. Building many-qubit quantum computer hence requires delicate control over the interactions between constituent qubits. This usually requires climinating thermal noise by going millikelvin temperatures. For the problem studied here, the quantum state can be transformed with conventional optics at room temperature: the energy of a green photon is 2.5 eV, way above the typical room temperature thermal radiation energy of KT ~ 25 meV. The price to pay is that it is challenging to build a device that allows multiple photons to interact in the way needed to build many-qubit quantum computer. Nevertheless, Knill, Laflamme, and Milburn (Knill et al. (2001) devised a protocol to make this feasible in principle, avoiding the need for coherency-preservinn nonlinear optics (which may well be impossible to realize experimentally) by clever exploitation of ancillary photon qubits, boson statistics, and the measurement process. In all such applications, the basic idea is to employ coherent multiphoton quantum states to do computations with multiple qubits. In the problem studied here, there is only single photon, the only relevant information that gets processed is encoded in the spatial part of its wave function (i.e. polarization is irrelevant), so the current work resembles the "optical simulation of quantum logic" proposed by Cerf et al.) (1998) where a N-qubit system is represented by 2 spatial modes of a single photon. Related work studied similar "optical simulations of quantum computing" for implementing various algorithms, in particular (small) integer factorization (Clauser and Dowling (1996); [Summhammer (1997), but to the best of the present authors' knowledge did not consider machine learning problems.
2

---

classifier performance, for reasons explained in Under review as a conference paper at ICLR 2021
This work can be described as belonging the category of machine learning methods quantum non non-scalable architectures. Alternatively, one can regard it as a quantum analogue of recent work that demonstrated digital circuit free MNIST digit classification via classical nonlinear optics, for instance via saturable absorbers Khhoram et al. (2019).) Apart from providing an accessible and commonly understandable toy problem for both quantum and MI. research communities, this simple- quantum/simple-MLL corner also may be of interest for teaching the physics of the measurement process ('the collapse of the wave function") in more accessible setting. Whereas explanations of the measurement process are forced to remain vague where they try to model "the quantum states of the observer" (typically unfathomably many states that one would never hope to be able to model in terms of actual numbers), using machine learning as a sort-of cartoon substitute for high level mental processes actually allows us to come up with fully concrete toy models of the measurement process on low-dimensional (such as: D < 1000) Hilbert spaces that nevertheless capture many of the essential aspects- - to the extent that "ML classifies the measurement as showing the image of a shoe" can be regarded as a crude approximation to "observer sees shoc".
Looking closer at the relation between the present article and Khoram et al. (2019), both articles study the general feasibility of realizing Machine Learning classifiers in the form of an analog optical computer at the theoretical level, using numerical optimization to produce a blueprint of device that can perform inference for specific problem. In both articles, the primary problem under study is MNIST handwritten digit classification, the input is encoded as spatial dependency of (monochromatic) laser beam's 's light intensity, and classification happens by using interference to funnel optical energy onto a detector array. In both cases, stochastic gradient descent is used to shape how this funneling of optical energy happens. Indeed, even the loss function used for training (cross entropy) is essentially equivalent. The key differences are that Khoram et al. (2019) only considers the many -potto limit of classical wave optics, which allows one the luxury of using non-linear optical components, specifically saturable absorbers, to implement non-linearities. This has no analog for the single photon case. Also, having many photons available allows identifying the target class that receives most laser light and calling this the prediction of the model. This is clearly not possible when a decision has to be made after seeing only a single photon. If one sent many photons through an interference device as described in this article and picked the target class with the highest photon count, one would observe classification accuracies of about 90% rather than the claimed about-40% for single photon. This is considerably higher than the accuracies of about 80% presented ir Khoram et al. (2019) as the focus of that article is on manufacturabilityy running gradient backpropagation directly on Finite Difference Frequency Domain PDE simulation of Maxwell's equations and taking materials engineering constraints into account, whereas our work focuses on upper and lower bounds for achievable accuracy, exploiting the one-to-one equivalence between linear optical devices and unitary transforms. Our work directly trains the parameters of the unitary transform, which only afterwards get mapped to a blueprint for an experimental device realization. Speculatively, if device were built experimentally that was designed by the methods in et al. (2019), subject to the extra constraint that no non-linear elements can be used, and then deployed in low-light-intenstc single-photon setting, using a suitable detector such SPAD array, it may manage to realize better- clwssccaall- the current work.
1.2 THE MEASUREMENT PROCESS
How well one can one solve a mental processing task, such as identifying a handwritten digit, if one is permitted to only measure a single quantum? This question leads Hilbert space basis factorization that parallels the factorization needed to study the quantum mechanical measurement process. Let us consider a gedankenexperiment where our quantum system (see |Feynman al.) (Lndd) and Lifshitz (1981) for an introduction to quantum mechanics) is single atom that has two experiment-relevant quantum states, 'spin-up' and 'spin-down',
Amoom) = col Atomm + c1 1 Atomm=))
(I)
This atom undergoes measurement by interacting, over limited time period, with an apparatus. The measurement process may involve for instance an atom emitting a photon that is detected by camera, and it may include human observing the result. We describe quantum state in the potentially enormous Hilbert space of apparatus states with the vector ( AApparatuss) If, in this gedankenexperiment, actually assume that we have maximal information about the (quantum)
3

---

Under review as a conference paper at ICLR 2021
measureen
state of the measurement apparatus (which, however, in practical terms would be unfathomably complicated) at the beginning of the experiment, then the full quantum state of the initial system is the tensor product
|yyttem, initial) = Atom, initial) & |AApaartuss, initial).
This factorization implies that atom and apparatus states are independent before the interaction.
Without interaction between the apparatus and the atom, the time-evolution of the total system factorizes. A measurement requires an interaction between the apparatus and the atom, the solution of the Schrödinger equation is equivalent to the application of a unitary operator U to the state Iysystem, iniuia). This has the effect of combining the state components of the atom and the apparatus and, as consequence, the joint time evolution no longer can be factorized. The overall state is [1system, fimal) = U WSystem,, huitial) and can be always decomposed in the sum:
[1/yystem, ? = az Atom=1t) @ AAApprratss final=t) + (3 Atomm1)) @ | Appparatus, final=4); (3)
where the apparatus states AApparatus, final=t) and AApppraatu, final=1) represent the state of the apparatus after the measurement for the two basis states of the atom. Therefore, the apparatus is in different state for the two cases, which leads to the apparent "collapse" of the wave function. The apparatus in the state |Apparrtus, final=t) perceives the "collapse" because the atom seems to have taken the state (&) Atom=t) The state of the apparatus includes also the representation of the thought aappane
process of a possible human observer, for instance asking herself what instant the atom took well determined state. This thought process disregards the superposed state |tAApparatus, final=1) which represents the alternative reality, where the apparatus observed a different outcome.
Considering that a mental process could be seen as a measurement on the environment, one would naturally be inclined to think that high level mental concepts never would naturally lend themselves to a description in terms of some Hilbert space basis that has tensor product structure |ggeneral coscepp..coccep.. / & [ Machine learning now making the question to what extent this may nevertheless work quantitatively testable for some simple cases, if we consider it as providing reasonably good (for this purpose) models for mental concepts.
Let us consider the spatial part of a single photon's 's quantum state as it traveled through mask that has the shape of a complicated object. For instance, let's s assume that the mask is obtained from a random sample of the MNNIST" dataset, Xiao et al. (2117), where each sample represents an object such as a shirt, trouser, etc. One would generally expect that any sort of transformation that connects a highly regular and mathematically etc. simple description of such a quantum system, such as in terms of per- per-picture-cell ("pixel") amplitudes, with a description in human- -nttaninteepretaable terms, such "the overall intensity pattern resembles shirt," would unavoidably involve very complicated entanglement, and one should not even remotely hope to be able to even only approximately express such photon states in terms of some factorization
| (phtonn) 2
Σ shape classs CE (shirt, .tr..se......) style S
Σ
cas (assl class C) Ø ()styee))
(4)
since one would not expect the existence of a basis of orthonormal quantum states that can be (approxi- mately) labeled (Wshirt), ) |V'soee,, etc. Using machine learning, we can quantitatively demonstrate that, at least for some simple examples, precisely such a factorization does indeed work remarkably well, at least if we content ourselves with the concept of a "shirt shape" being that of a aae-out-of-man machine learning classifier, so not quite that of human. In any case, it is reassuring to see that even near-future few-t quantum computers might be able model high level concepts rather well.
might
2 SINGLE- QUANTUM OBJECT CLASSIFICATION
Our we starts with single photon passing from faraway through a programmable LCD screen, which we here consider to consist of N x N pixels and show an image, where for both the MNIST handwritten digit dataset of LeCun and Cortes (200)) and the "Fashion-MNIST" dataset of Xiao et al. ). we have N = 28.
The size of the screen shall be sufficiently small for the photon's quantum state to be at the same phase as it reaches each individual pixel. This does not mean that the screen has to be small in e
commin comparison the wavelength. Rather, the light source must provide highly collimated illumination. iillmiimaa
4

---

Under review as a conference paper at ICLR 2021
The relevant spatial part of the photon's quantum state is described by element of N x N- dimensional complex vector space. We can choose a basis for this Hilbert space such that the quantum state of a photon that managed to pass through the screen (rather than getting absorbed) has the form
MPPhoton)
Σ
TOW J. column
where the (15x)) basis functions correspond to a photon that went through pixel (j, k). and the coeffi- cients Cjk are real, non-negative, proportional the square roots of the image's brich-brightnnesee. and are normalized according to Σj, (r,&l 1.
As we want to perform rotation on this Hilbert space that maximizes alignment with tensor product Hilbert space where one factor describes an image class, we pad this N²-dimensional Hilbert space into larger Hilbert space with dimensionality M divisible by the number of object classes C, i.e. M C S. This amounts to adding always-dark pixels (that may not form a complete row) to the image. The problem then amounts to engineering, for problem P such as handwritten digit recognition, a single problem woblemm-specffic unitary transform Up of the photon state, [t/Photon) Up (Photoon) such that we can meaningfully claim:
Urly (Tlooooooon) (Photon-)
Σ
Σ
Ces GeslMelassssse is) @ P'style
varinnt vartint is
example class style
Specifically, for each individual example image E,, we would like to have
UplyPhoton.E)
Up B)
R ((CE)) X
Co Calyay variant i (F)),
styles
where C(E) is the ground truth label of the example in supervised learning setting.
Using the method described Reck et al. (1994), this trained matrix then can be translated to an optical network blueprint. The transformed quantum state at the output side of the network of beam splitters and phase shifters then gets measured by a detector array that can discriminate M = S quantum states which are labeled [Vidgii is)) [sstyle variant 1), Wdigit is o) @ variantt 2), ..., Waigit is 33) @ "y 57).
variant
variant Pdigit is 9e @ |1style variant Sma.. ). If we detect the photon in
any of the (ddgitt is 7) @ ... cells, the classifier output a "7", and likewise for the other digits.
From a machine learning perspective, the trainable parameters hence are the complex entries of the matrix Up. which according to quantum mechanics have to satisfy an unitarity constraint, UpU I, and this search space automatically covers all experimentally realizable linear optical devices. For MNIST, where examples have 28 x 28 pixels, the most obvious choice is padding to a M 790-dimensional input vector. While one could implement the unitarity constraint terms of (regularizer) loss-fure function contribution that measures the degree of violation of unitarity, it here makes more sense to instead use a parametrization
Up that automatically guarantees unitarity, using Lie group theory. If Wp is 790 x 790 matrix of trainable (real) weights, then the hermitean matrix Hp = - -i(Wp covers all of the (compact) unitary group (790). This approach slightly over-parametrizes the problem, since, in the censor-produc basis that we are transforming to, we can freely re-define the basis on each of the ten 790/10 10 79-dimensional style subspaces. This means that 10% of the parameters are redundant.
WF) + (Wp + WF) parametrizes the Lie algebra u(790), and Up = p
Overall, with all the trainable weights being provided by the matrix Wp, and the brightness of the pixel at coordinates (y, x) for example E being bggrs we have this formula for the probability of a photon travelling through an optical device that was designed by training weights and landing on detector cell that predicts class c:
E)
expm (Wp WT. + i(Wp + WF)),, j.k.
Σ
SN CN S
Eg bEg
Here, y and x image row- and column-indices (for MNIST, running from 0 to 27), j. k are matrix row- and column-indices (in our example, running from to 789) for the exponentiated unitary matrix Up expm(...). , is style-index (here, running from 5 78), the term under
5

---

Under review as a conference paper at ICLR 2021
c= 0
c=
C= 0
c=
(a)
(b)
Figure 1: (a) The two shapes of the toy example. The four gray pixels correspond to a photon arrival probability of 1/4, i.e. probability amplitude of 1/2. (b) The per-pixel photon arrival probability after the orthogonal transformation is applied. The dark gray pixels correspond to a probability of 1/8 and the light gray pixels to 1/2.
the square root is the relative contribution of the (v, -ppiiel the total brightness of example image E, and the d-factors are used for translating pair of row,column image-indices to linear pixel index, respectively index on the P-rotateed quantum state vector to pair of (class, style)- indices. Technically speaking, from the viewpoint of mapping an optical amplitude that describes light intensity passing through the image-filter the quantum amplitude of particular (class, style)- combination, this is simply linear model (since quantum mechanics is linear), whose linear weights are however specified in a slightly unusual way, underneath a complex matrix exponential (since quantum mechanics is unitary, i.e. probability probability-presevinng preserving). The probability to predict given class c then obtained by summing over the probabilitics associated with the given class (but different style-index).
Model accuracy has to be evaluated with caution: as we need to make prediction after detection of a single photon, accuracy is the quantum probability of the correct label, averaged over all examples. Naturally, we can not determine which the output classes would receive the most photons (= has highest probability) if all we have is single photon detection event. This accuracy, about 40% for the problems considered here, differs substantially from the accuracy that would be obtainable by looking many photons coming from the same example image, which here typically exceeds 90%, roughly in alignment with the expected achievable performance of linear model on MNIST. In other words, probabilities are uncalibrated, and the (non-linear "deep learning") transformation that would be required to calibrate them cannot be expressed as a unitary operator.
Let us consider a radically simplified example that illustrates why this method works. We want to discriminate between only two different shapes (with no further shape variation) on a x 4 pixel screen where each pixel either "on" or "off". using only one photon. Specifically, let us consider the two Tetris "T" shapes represented in figure Ta).
For both shapes, the probability that the single photon arrives on one of the "on" pixels is 1/4; therefore, taking into account that for two pixels the correct shape is identified exactly and for two with 50% probability, we conclude that the baseline accuracy is 1/2 + 1/4 = 75%. Instead, we can apply umitary transformation reshape the probability amplitudes.
Let us now consider the simple but not optimal transformation of the photon amplitude that replaces the pair of amplitudes (a, b) in each 2-pixel column with ((a- ((a-b)/v2 (a + b)/ i.e. creates destructive interference in the top row and constructive interference in the bottom row. This gives the detection probability patterns shown in figure (b). Maximum likelihood estimation here gives an accuracy of 1/2 + 3/8- 87.5%.
Obtaining the maximum achievable accuracy will here require a more complicated all-pixel amplitude transformation, obtained as follows: The quantum amplitude transformation is angle-preserving. and the angle a between the two amplitude quantum states Q1, 92 is given by cos a (q1 (42) 0.5. Hence,
rotate these two states lie in the plane of the first two Cartesian coordinate axes of the Hilbert space, and at the same angle from their bisector. Identifying these coordinate axes with the correct labels, the accuracy is the cosine-squared of the angle between the transformed state and the corresponding axis, i.e. cos2 cos2(T// a/2) (v3- 2)/4 93.30%.

---

Under review as a conference paper at ICLR 2021
Table I: Results for the Fashion-MNIST and MNIST datasets. The "classic" accuracy and information refer to the observation of single photon, while the "quantum" quantities are obtained after applying the quantum transformation.
Dataset
Entropy [bits]
Accuracy Bound (classic)
Information (classic) [bits]
Accuracy (quantum)
Information (quantum) [bits]
Fashion-MNIST
3.32
21.38%
1.10
36.14%
1.85
MNIST
3.32
22.96%
1.20
41.27%
2.04
While the performance measure that we care about here the probability for a correct classification, one observes that model training is nevertheless more effective when one instead minimizes cross- entropy, as one would when training conventional machine learning model. Intuitively, this seems to make sense, as a gradient computed on cross-entropy loss is expected to transport more information about the particular way in which a classification off than gradient that is based only on maximizing the correct classification probability.
Overall, this task is somewhat unusual as a machine learning problem for three reasons: First, it involves complex intermediate quantities, and gradient backpropagation has to correctly handle the transitioning from real to complex derivatives where the loss function the magnitude-squared of a complex quantity. TensorFlow is at the time of this writing the only widely used machine learning framework that can handle this aspect nicely. Appendix [ details on numerical aspects. Second, (as explained above), cannot simply pick the class for which the predicted probability is highest as the predicted class. Rather, the probability for the single-photon measurement to produce the ground truth label sets the accuracy. Third, while most machine learning architectures roughly follow a logistic regression architecture and accumulate per-class evidence which gets mapped to a vector of per-class probabilities, we here have the probabilities as the more readily available data, so the computation of cross-entropy loss will have to infer logits from probabilities. Due to this need to compute logarithms of probabilities, it is very important that the training process does not intermediately see invalid probabilities outside the range (0... 1), and this is ensured by parametrizing unitary transforms as matrix exponentials of anti-hermitean matrices.
TensorFlow code to both train such model and also evaluate its performance is included in the supplementary material.
RESULTS
Figure 2 shows the most probable image class for each pixel, for the "Pashion-MNIST" and MNIST datasets, A classifier that looks up and predicts the most likely class in this table achieves maximal accuracy among all single photon classifiers that do not employ quantum interference. This includes classifiers that have had access the test set during training. This accuracy is reported in the third column of table [ (We note that, as pointed out by /unn et al. (2007), the Faashoo-MMNSST dataset contains many mislabeled instances, which affects both classical and quantum results.) We can compute the amount of information provided by the photon by computing the difference between the class entropy, i.c. logz (0.1) 3.32, since there are 10 classes, and the entropy associated the classification errors, i.c. the accuracy. The mutual information for the classical classifier is given in the fourth column of [
Training a unitary U (790) quantum transformation that gets applied after the photon passed the image filter and before it hits bank of 790 detectors allows boosting accuracy for both the "Fashion- "NISST" and MNIST datasets, as reported in the fifth column of table [ The observation of the photon after the transformation provides a higher amount of mutual information with respect to the classical case. The values of mutual information in this case are given in the last column of table [] Explicit matrices to perform the transformation for the two data sets have been made available with the supplementary material.
The quantum transformation Up allows us to define the pixel-space projection operators:
P C U.². (Itbe) (vhcl 6 Tstyle) Up
7

---

Under review as a conference paper at ICLR 2021
Figure 2: (a) MasHion-MNIST most likely class given detection of a single photon at the corresponding pixel coordinates. Here, the classes are: 0=T-shirt/top. 1=Trouser, 2=Pullover, 3=Dress, 4=Coat, 5=Sandal, 6=Shirt. 7=Sneaker, 8-Bag. 9=Ankle Boot. (b) Most likely digit-class given detection of single photon for MNIST. non-quantum classifier cannot outperform one that looks up its answer on the corresponding table.
Predicted
Predicted
0.060
0.060
True 000-1011BUNN
0.048
0.048
0.036
True
0.036
0.024
0.024
0.000
0.000
012344 0123456788
0
2
3 4 678899
(a)
(b)
Predicted
Predicted
0.060
0.060
0.048
0.048
True
0.036
Tue
0.036
0.024
0.012
0.012
0.000
0.000
0122 56889
6788
(c)
(d)
Figure 3: The confusion matrices for the "Fashion-MNIST" and MNIST datasets when classic and quantum classifiers are used: (a) "Fastion-MNIST"/clasiic, (b) MNIST/classic, (c) "Fashion- MNISTT/quantum, (d) MNIST/quantum.
with which decompose any example into contributions that attributable the different classes. Here, one must keep in mind that such separation done at the level probability amplitudes, so while can compute intensities/probabilties from these components, which are mutually orthogonal as quantum states, summing these per-component per-pixel intensities will reproduce the example's 's per-pixel intensities. This shows most clearly when considering the decomposition of an example "Trouser" from the "Fashion-MNISTT dataset's 's test set with the model we trained for this task, as shown in figure aa). The dark vertical line between the legs in the original image mostly comes from destructive interference between a bright line from the "Trousers" component and matching bright line from the "Dress" component.
Due to the intrinsic quantum nature of this set-up, care must be taken when interpreting confusion matrices. Naturally, we never can claim of any single-photon classifier that would 'classify particular image example correctly', since re-running the experiment on the same example will not see the photon always being counted by the same detector! So, strictly speaking, for any single-photon classifier realized device, the "confusion matrix" could be determined experimentally only in the statistical sense, leaving uncertainty in the entries that decreases with the number of passes over the test set. Confusion matrices are shown in figure
8

---

Under review as a conference paper at ICLR 2021
Sample e= e=1 e=2 e=3
e=4
c=5
c=6 c=7 c=8 e=9
Sample =
c=1 c=2
e=3
e=4
c=s
c=6
c=7 c=8 c=9
Figure 4: Projection of probability amplitudes for some samples of the "Fashion-MNIST"" (a) and the MNIST (b) datasets. The first image shows the original sample probability and the following images show the probability amplitudes for each class. We visualize the complex amplitude by using brightness represent magnitude and hue for phase (the colormap for the phase is shown the right of each row.)
Our factorization ansatz appears to contain hidden constraint: we are forcing each image class use the same number of style-states. One could imagine, for instance, that classifier might achieve even higher accuracy by treating image classes unevenly. Any such model that allows more style- dimesions for some classes can always be embedded into model that allows more style-space dimensions for all classes, so this question can be answered by padding larger Hilbert space. Numerical experiments, e.g. padding to 1000 rather than 790 dimensions, suggest that this has appreciable impact on classification accuracy.
4 DISCUSSION
In summary, demonstrated that, least for the considered datasets, the space of the single observed photon state can be factorized remarkably well by product of the example class space and space collecting the remaining variables, such the style. This factorization by the proposed method and is experimentally realizable with optical elements placed in front of the sensor. The supplementary material contains blueprint for an example circuit outperforming the classical limit (at 36.05% accuracy) on 10 10 downsampled MNIST.
be obtained easily
experimental implementation of the proposed system would be demonstration of a high- temperature and low- effective-qubit quantum ML device. With respect to other experimental approaches to quantum computing, such device would have the limitation that it is built for the specific classification problem and cannot be reconfigured easily. It would be interesting to see whether an advanced quantum protocol along the lines of Knill et al. (200T) might enable the realization of more sophisticated intermediate-sctae high temperature quantum machine learning in way that mostly (like here) bypasses the need for quantum logic built from common quantum gates.
5 TENSORFLOW CODE
TensorFlow2 code reproduce the experiments of this work and all the figures is provided in the ancillary files together with the computed unitary tranformations for MNIST and "Fashion-MNISTT.
REFERENCES
Daniel Smilkov, Shan Carter, D. Sculley, Fernanda B. Viégas, and Martin Wattenberg, Direct-manipulatioo visualization of deep networks. CoRR, hos/1708.03788 2017. URL http:://arxiv..rr//aba.. 1708
03788

---

Under review as a conference paper at ICLR 2021
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. http:://///ann lecun
com/exdb/ hondd//mnsstt mt/st/
Michael Reck, Anton Zeilinger, Herhert Bernstein, and Philip Bertani. Experimental realization of any discrete unitary operator. Phys. Rev. Lett. 73:58-61, Jul 1994, doi: 10.1 N0.100//hysReel.et....... URL.
73.58
E. Knill, Laflamme, R., and G. Milburn. A scheme for efficient quantum computation with linear optics. Nature, 409:46-55 52, 2001.
htttps://dl.... www.oo//.oooo.....ooooooo.../..o 1038/ 05058855555055
N. J. Cerf, C. Adami, and P. G. Kwiat. Optical simulation of quantum logic. Phys. Rev. A, 57:R1477- R1480, Mar 1998. doi: 10.100//hyyRevA.57..1777 URL htttp:://inkk...
(103/PhysRevA.57 R1477
John E. Clauser and Jonathan Dowling. Factoring integers with young's inteffrrnter Phys. Rev. 53:4587 4590, Jun 1996. doi: 10.1 1.1100/ /ttp.///////.........././///////
I103/PhysRev.. 53.4587
Johann Summhammer. Factoring and fourier transformation with innrrhhhchndrrttcccccccmmmmme Phys. Rev. A, 56:4324-43200 Nov 1997. doi: 10. 00.11377hyyRR4455..4444
I103/ 1103/PhysRevA. 56.422
Erfan Khoram, Ang Chen, Dianjing Liu, Lei Ying. Qiqi Wang, Ming Yuan, and Zongfu Yu. Nanophotonic media for artificial neural inference. Photon. Res., 728) 827, Aug 2019. doi: 10.1364/PRJ.7.000 000823.
URL/tttp: wwwww http:////ww.....aaaaaaaaaaa///// httpaa/tee...tttt....s.rrp---/-
Richard Phillips Feynman, Robert Benjamin Leighton, and Matthew Sands. The Feynman lectures on physics;
New millennium ed. Basic Books, New York, NY, 2010. URL https://cds https://cd...cnn.ccl.. www.rn.cccreerraeeeeee
1494701
Originally published 1963-1965.
L. D. Landau and L. M. Lifshitz. Quantum Mechanics Relatvelaiiiitic Theory, Third Edition: Volume
Butterworth Heineermnnneennmaannnn 3 edition, January 1981. ISBN 0750635398. URL http:://wwwt wor ldcat
org sbnn/ 077063533333
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mann mnist: novel image dataset for benchmarking machine learning algorithms. arXiv, 2017.
J. Sun, F. Zhao, C. Wang, and S. Chen. Identifying and correcting mislabeled training instances. In Future Generation Communication and Networking (FGCN 2007), volume 1, pages 244-250 2007.
A APPENDIX
A.1 EXPERIMENT SCHEMATICS
Figure Sjsressshowss the schematics of an exper- imental set-up: The lens (A) stylizes the optical component the monochro- matic. coherent, iinear-pllarieed (i.e. laser) light source that emits photons en- tering from the left and travelling to the right. Light intensity is controlled (e.g. by means absorbing filter, not shown) to be so low that photons travel through the apparatus individually. Any interference effects are hence due to seff-interference of single photon's wave function (just as the double slit gedankenexperiment) The laser photons then hit a coherently il- luminated N x N screen (B) (e.g. LCD screen, in this diagram 10 x 10) which allows light to pass through given pixel with coordinates (y, . with probability that is proportional to the ink density on the example image. The photon arrives each pixel with the optical phase (i.e.
A
B
C
D
Figure 5: Schematics of the experimental set-up. Top: Classical Baseline, Bottom: Quantum Set-Up.
10

---

Under review as a conference paper at ICLR 2021
having travelled the same (fractional) number of wavelengths seen from the laser). The diagram shows an ex- emplary raster image of digit zero with maximal brightness (maximal ink density on the digitized ML example) on 14 pixels (with zero-based row/column coordinates (v.z) x) = (2, 4), (2, 5), (2, 6), (3, 2),... (5, 7)), with 75% brightness on three pixels (coordinates (3, (), (6, 3), (6, 5)), and 50% brightness on one pixel (coordinates (7,6)).
The 'Classical Baseline' set-up does not use interference and would work just well in a world where photons are "Particles of Light' that cannot self ceff-interfere (as envisioned by Newton). The dimensions of the apparatus need to be such that, when the image-filte filter is brightly illuminated, it casts a sharp shadow on the detector-array For the classical case, use of a coherent source of light is not necessary. At very low light levels, photons coming from the light source (A) will keep hitting the screen (B), frequently getting absorhed by dark pixel. At some point, photon will (by chance) manage to hit a non-dark pixel and not get absorbed (the more likely the brighter the specific pixel) and travel on to the single-photon detector array (D) (such as: a SPAD array) and be detected as having passed through a specific pixel. Ignoring experimental imperfections that could in principle be made small such as optical losses, the only possible transform on the photon state one could perform with passive-linear optical device (C) would be equivalent to coupling the photon into array of optical fibers. routing each fiber from one pixel to some other pixel, and coupling out the photon at the other side of the device. This is equivalent to re-shuffling the pixels, which can always be un-done by re-shuffling the addresses of the cells of the detector arryy (D) and so does not affect classification accuracy the diagram hence omits such transfornations that cannot affect performance.
The quantum set-up (bottom) is perhaps easiest analyze via Feyaman's path integral interpretation of Quantum Mechanies: There are different 'histories' which lead from the same initial point (a photon coming from the laser) to the same final result (the photon being detected at a specific pixel, such as: 'at coordinates (4, 5)"), and the prescription is that we have to attribute a complex quantum amplitude to cach such 'history'. summing over all amplitudes that connect the same initial and final state to get the resultant amplitude, and obtaining the associated probability the magnitude-square of the (complex) resultant probability. We can also consider the resultant per-pixel quantum amplitudes for a photon having travelled from the light source (A) not all the way to the detector but to some intermediate point, such as just after passing the screen (B). These are described by vectors with 22 entries. one per pixel, whose any flat, screen pareenlpaaalll surface between and C, k<bc)). has zero entries for all but the 14+3+1=18 non-dark pixels. The entries #ec (25). that correspond the 14 maximally-brighe pixels at (2, 4), (2, 5), ... are identical, as the photon reached cach pixel at the same optical phase. Calling this amplitude eø (B for 'bright'), ) the amplitudes for the three modcrately-brite dim-but-not-dark pixel's amplitude cp, the amplitude magnitude-squares must be proportional to pixel brightness (probability for a photon to pass through the image) and sum I (total probability for the photon that passed through the screen to have passed through a pixel). As the complex quantum amplitude phase matches the optical phasc, these constraints fix eg with 2 being the overall normalization factor that makes the sum magnitude squares of all amplitudes 1, i.c. 2 = 14 1 ++3 3 0.75 + 1- 0.5, and a being some complex number of magnitude 1, the non-obsereee -bseevveve overall quantum phase factor. In the 'Quantum' set-up. we can employ a linear optical device, built from many beam-splitters and phase shifters (= 'delay lines'), ) adjust self-interference of the photon wave function. The diagram shows two example paths out of the total 10 10 different paths that the photon can take before it reaches the detector array. Quantum Mechanics tells us that 'the single photon(!) travels along all these paths simultaneously" this is just Young's 's double slit experiment in a slightly more complicated setting.
wooollut-magiuudee squares sum to 1. In the example, the
photon-state-vectee
pixels cM, and the single
/ 1/Z 0.244, CM u- v 0.75// x 0.212, cD
/ /.55ZZ x 0.173,
There is a 1-to-1 correspondence between physically realizable linear optical devices and crobability-preservve (generalized) rotations of the quantum state vector. The (in this downsampled example: 100 x 100) components of such transformation form a unitary matrix. If we used linear optical device that implemented a random such transformation, and sent many photons through the apparatus, they collectively would produce an image conceptually resembling the interference pattern on photo film that codifies a hologram. Using a basic Machine Learning procedure, stochastic gradient descent, we can train the parameters of the transform such that photons coming from image that shows digit '0" preferentially land on the O-th row of the detector array, photons coming from an image that shows digit '4' preferentially land on the 4-th row, etc. The supplementary material describes a specific sct-up in terms of optical components that reaches > 36% accuracy for MNIST downsampled to 10 x 10: If this were manufactured from ideal-quality optical components, the probability for a photon that passed through the screen (B) to land on the detector row matching its digit-class is better than 36%.
A.2 CLASSICAL BASELINE ACCURACY THRESHOLD: MAXIMALITY PROOF
Elementary statistical considerations allow us to obtain a stringent upper bound for the maximum accuracy that cannot be exceeded by any classifier which satisfics these two propertics:
Pl: The classifier must make prediction using as its input only the index of the one detector in the detector-array that received the first photon. It not use any additional information about the
11

---

Under review as a conference paper at ICLR 2021
example (but may have been trained with arbitaay information about the example xet, even including full knowledge of the training and test set).
P2: There a one-to-one correspondence between image pixels and detector-cells: For each image- pixel, there is exactly one detector-cell such that when the example image E' is presented, then the probability for the first photon to land on the detector cell & is proportional to the brightness of the associated pixel in image E.
This bound is what we call the 'classical accuracy bound'. The 'quantum' classifier violates P2 by employing photon self-interference: The probability for the k-th detector to observe the first photon depends on collective information which the photon 'holographically' transports about the input image once it passed the image- filter, rather than on a single pixel.
The protocol for evaluating classifier accuracy is as follows: We pick random example from the dataset's 's test set, set up the device to present this example as problem, send light towards the filter-screen, and look at the first photon that managed to pass the filter-screen and get counted by the detector array, then map the index of the detector that counted the photon to a predicted class. We register 'successful prediction' if the predicted class matches the example's 's label, otherwise we register an 'unsuccessful prediction' : The accuracy is the probubility of the prediction be successfil.
Somewhat unusually, this means that there is no such thing as 'the predicted class of given image', as one normally would have it in Machine Learning problem. This is due to the inherent randomness of quantum mechanics: Even repeating classification for the same image multiple times, we will see photons land on detectors that correspond to different classifications. If we were to experimentally determine accuracy, this would then suffer from the usual problems of determining a probability via a statistical experiments: make the likelihood to be way off arbitrarily small, but never shrink it to zero. However, the aforementioned protocol makes it possible to directly compute the maximal achievable probability for any classifier, without resorting to a statistical experiment.
The gist of our argument parallels the reasoning behind the claim that we can show with simple statisties that no ML classifier can possibly outperform an accuracy threshold of 29/36 for predicting from the eye total of rolling two dice whether any of the two dice showed six. Here, the reason is that we can get maximal accuracy by looking look at all the possible realizations of any given eye total and make the best possible guess given the situation. For eye totals 2 - 6 (1 + + 4 5 = 15 of 36 cases), we would predict 'No' and always be correct. For eye totals 11 and 12 (2 ++1 additional cases). we would predict 'Yes" and also always be correct. For each other eye total, there are two realizations where one die shows 'six', so we would want to predict 'Yes' for eye totals having at most four realizations, i.c. where we have at least 50% chance of heing correct, and 'No' otherwise. Using this approach, we would incorrectly classify three cases as 'Yes' (5 + 5, 4, + 5,5 + 4), and incorrectly classify four cases as 'No' (6 + 1,1 6, 6 + 2, 2 + 6). Except for these 7/36 cases, we would make a correct prediction, so optimal accuracy is 29/36. Given the perhaps somewhat unfamiliar "quantum ML' setting, and the need to rigorously justify the optimality claim, we prove below. The only material difference to the dice-sum example is that relative weights of realizations are not determined by counting, but by looking at pixel brightnesses.
In analogy to the the dice example. the key observation is that the classifier's 's input is single pixel-index, and its output an image-class. So, we can completely specify any (deterministic not) classifier's action by what the probability is for this classifier to map the given imput pixel-index to each of the possible output classes c. The resulting matrix Kic would, for a deterministic classifier, simply be a matrix with one- one-hot encoded image class, one row per pixcl-index. The classifier's 's accuracy is then given by
tabulating, per-pixe per-pixel-inder
Accuracy = P(Classification is correct) = EE P(E) Ex P(x) ΣE, P(E) P( E)- KR,
P(ye = C(T))) =
(10)
Here, P(E) is the probability to pick example E from the test set (i.c. 1/ftest set size)), P( [)) is the probability to detect the photon in the detector cell with index kt given the example E, and P(ye = C(x)) is
the probability that example E's label ys matches the classificr's 's output on the input "the photon was detected in cell K".
The probability for detecting photon in cell when randomly drawing example image from the test set is P(*)) = ΣE P(E) P(yx)E) The probability for fairly drawn example's ' label to be ye when a photon was detected at cell & is P(yalTa) = -E P(E)- P(ye = ya) P(t (E)) Let us tabulate these P(yelt)) in the (#pixels) } x {#classes} matrix Ake := P(ycla)).
We then have:
Accuracy
P(x)) P(y.TT)) KRo - P(yx) R.C
(11)
Detector cell & Class
In words: We can compute accuracy by looking at each detector cell is and each class determining the probability P(xx) that, when fairly drawing examples from the test set, photon gets detected at cell k, and
12

---

Under review as a conference paper at ICLR 2021
splitting up this probability into contributions from examples where the target class was 0. 1, 2, etc. These contributions are P(yell)), We make a correct classification when the classifier also predicts class e given the input k. The classifier's s behavior when given the input k is specified by row it of the K-matri,, so this probability is Kae-
Here, P(7x) and Rac are determined by the test set. Each admissible matrix KEc that has Σ. Kte 1 specifies different classifier, and the accuracy is function of this matrix K only. The question is now which admissible matrix K maximizes accuracy. Total classification performance (accuracy) is weighted sum over per per-detector- cell performances (the weights being the probabilities to observe photon in cell k when doing detection experiments on samples drawn fairly from the test set). Let K1 be matrix that maximizes accuracy, and Ka be matrix obtained by picking, for each cell-index At a probability row-vector that maximizes 2 Rke R K2,ke- We have Accuracy( K1) 2 Accuracy( (K2) (since K1 optimal), and also Σ. Ree - K1. 2 0 (since K2 maximizes this value on each row k), so, taking a weighted sum with weights P(**) 2 0. we find Σx P(Tx) Ee Re. - Ki.,,) 2 0. i.e. Accuracy (K2) 2 Accuracy (K 1. hence Accuracy ( Accuracy(K2) (22). In words, we achieve maximal accuracy if we individually look at each "photon detected in cell I. case and make the optimal prediction there. Now, for a fixed cell-index k. Σ, K2 is maximal if the matrix-row Kg, has entry for the index e for which RRe is maximal, and is zero otherwise. To see this, let us assume K2,ke" > 0 for some index c for which there is another class index d with Rnd > Ric- Then, incrementing K2,dd by K2, and subsequently setting K2,ke to zero increases E. R2cK2. which is a contradiction. So, optimal choices of Kg, are zero for classes for which REe is not maximal. Also, we always attain the maximum when choosing each row-vector of K to be one-hot and have its l-entry in a place that maximizes Rice i.e. maximal achievable accuracy is obtained by a classifier which, for every cell-index predicts the most likely digit-class subject to the constraint that a randomly drawn example from the test set had its first photon- detection occur at detector cell k. This theoretical upper bound on classifier accuracy hence is given by:
Accuracy Σ P( )max) Rec.
(12)
For the MNIST dataset, this is found to be 22.957% (rounded up 22.96%), while for Fushion-"MNIST get 21 21.375% (rounded up to 21.3%)). Code that implements this calculation available in the supplementary material. We should emphasize that the constructive procedure descrihed here that yields a classifier attaining this stringent upper bound does inspect the test set, and relevant deviations in statistical properties between training and test set would manifest in the form of lowering attainable accuracy for a classifier that is trained on the training set only.
A.3 BACKPROPAGATION WITH COMPLEX INTERMEDIATE QUANTITIES
As explained in the main text, the per-class probabilities defined by Eq. (8). when used input to conventional softmax loss function, make training the real weight- paramm-parrmeeers matrix Wp p terms which the unitary rotation is expressed a straightforward procedure.
Nevertheless, this approach utilizes some capabilitics which at the time of this writing are likely TensorFlow- specific. It hence may make sense to deseribe the training procedure in sufficient detail to allow straightforward re-implementation on top of some other Machine Learning framework, perhaps even directly without of any such library.
The loss function is defined in terms of the magnitude-squee squared of a complex intermediate quantity, which here is the vector of complex quantum amplitudes. one entry per class/style combination. In this appendix, we henceforth consider the simplified 10 10 problem described in detail in appendix
We can perform the calculation entirely in terms of real quantities by replacing every complex number C + iD by a real 2 2 matrix block of the form
C + + C .
(13)
This means in particular that a 100-dimensional (complex) amplitude- vector a, gets replaced by 200 matrix Ama- If we interpret the o-index j as encoding class e and style i.c. i...j j = c. S + s, the total probability for class is p(c) Σ. |ac.st.|2 = (Re s..)+ + (Im s+.)+ and this gets replaced by p(c)= = Σ. (A(c-sto)20 + 2()-5+))2.1) (reading off the real and imaginary part from the 1st column of the 2 xx 2 block that represents az).
As the square root of the relative per-pixel intensity is real, the imput-image amplitudes in this approach likewise get represented by 200 x 2-matrix B. Specifically, if e.g. Q2.5 is the contribution of pixel (y- 2, x 5)'s
13

---

Under review as a conference paper at ICLR 2021
brightness the total image-brightness, this gets represented
B12-1433)2200 B(2-1045):241,0 B12-1045)2.11 B(2-1045)22+111
)
Bao,o
Ba1,0
)= VQ2.5
(14)
B50,1
B51,1
VQ2.5
off-diagonal part, which would correspond to the imaginary part the amplitude, is zero here. The matrix that gets exponentiated is a real 200 x 200 matrix, and its exponential, which also is real 200 x 200 matrix, gets multiplied from the right with the 200 x 2 matrix of input-image amplitudes and gives the real 200 x 2-matrix A from above that contains the real and imaginary parts of class- and style-ampliudes.
The real 200 200 matrix under exponential only depends on the 200 x 200-matrix M, the "2 x 2-blocking" prescription to obtain its entries from Wp is:
100 real parameters Wp. Calling
Ma M,2+1.12 M,241,12411
M,2 3/242
(W,, W (W,, + W (W - W
-(W,, + W5
(15)
Finally, we need backpropagation-frice friendlly preseription for computing a good approximation to the matrix reach every 'generalized' (since complex)
exponential. The theory of compact Lic groups tells that rotation matrix by exponentiating matrices where each entry is from some not too large interval. For matrices with small entries only, we can use a truncated Taylor polynomial to get a good numerical approximation its exponential, using
expm(M) = 1 M M- M M M ....
2
(16)
we can reduce problem of finding the matrix exponential a matrix where this series requires many terms to give a good approximation by repeated halving and squaring, repeatedly using the property expm (M)
expm M /2) = expm(M M/2) (Mpm)(M/2)
For the problem discussed here, the angle-range ranges for rotations that need to be considered are limited, and this makes feasible to in-advance pick both number of squarings (such as: 8) and maximal term in the Taylor expansion (such as: 10th power), and get very good results.
The numerical computation implemented in the supplementary material, being based on TensorFlow, deviates from the procedure described here in two relevant ways. First, while TensorFlow's differentiable matrix exponentiation algorithm employs repeated halving/squaring, it uses a Padé rather than Taylor approximation compute the matrix exponential of matrix with small entries. Second, TensorFlow through complex intermediate quantities, and handles the transition between real and complex gradients in just the way that one also obtains when expanding complex numbers to real 2 x 2 blocks as described above. It can however avoid the inefficiency associated with using actual real 2 x matrix blocks that make every real and every imaginary part show up in memory not once, but twice.
directly backpropagate
B CHANGES
For the sake of transparency and providing better overview over the evolution of this document, added this appendix that tracks changes made during the review period. This appendix will be removed the end of the review period.
B. 2020-1 1 1-13 CORRECTION + 2020- 11-23 UPDATE
Made the changelog a separate appendix.
Fixed an explamation the baseline-accuracy the main text: This still referred to proven-optimal-on- ceamples-eenerate--ffoo-- traininninin--se,, even as the updated baseline now already referred to classifiers that even allow 'cheating', in the form of training the test set.
Slightly adjusted the wording appendix A. to clarify that apparatus dimensions are such that interference effects are negligible when no lincar optical device transforms the photon-state and the image-filter casts its shadow on the detector-array.
Figure Added explanation to caption that these figures actually describe maximal maximal-accuracy accuracy classifier.
Added another appendix on cunplex-bacckpoopagatiio as described in our 2020-11-19 reviewer- response.
B.2 2020-11-13 RESPONSE-+ 2020-1 1-13 CORRECTION
Formula 8 required fix: The sum over style-parameters is, in quantum mechanical language, an 'incoherent summation rather than 'coherent summation': After the unitary transform, the photon hits detector, this gives
14

---

Under review as a conference paper at ICLR 2021
information about the class (for the downsampled 10 10 problem discussed in appendix A. via the of the detector-cell), and also the syleeparameter (via the column of the detector-cell). After this measurement, we know 'class' and style, but decide to ignore 'stylc'. Still, as the 'style" is the outcome of a measurement, there can be no inference between different 'style' results, and so we need to sum the probability-comtibuttoon contributions for all the different style-outcomes, rather than summing amplitudes. We apologize for this rather obvious oversight that happened
somewhat rushed first response in order to give reviewers early answer.
B.3 INITIAL SUBMISSION 2020-11-13 RESPONSE
Major changes
1. Re-structured introduction, introducing 'related work' section and expanding in the main text the explanation of the relation of this work other recent analog all-optical image classification research.
2. Added formula 8 that directly expresses the probability for given example to be classified in particular way terms of image pixel data and model parameters.
Replaced the classical baseline accuracies (which were for any classifier was trained with access to the training set only) with strict proven accuracy bounds that can not even be beaten by classifiers that were trained with knowledge of the test sef. So, these hard thresholds for non-quantum classifiers not be outperformed by any means, including 'cheating' in the form of training on the test set. The impact is that the baseline performance threshold rises from 21. 21.28% 22.96% (MNIST) and from 18. 18.28% 21.38% (F-MNIST), requiring adjustment some plots.
4. Added self-contained appendix with schematics alongside detailed explanation of how image information gets translated to photon quantum
5. Added a self-contained appendix with the detailed proof (including proof idea sketch) of the claimed maximum attainable classical accuracy bound.
15


Reviews:
[{"review_id": "laB_-VYbEUI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goals of this work are ambitious: to clearly define a setting useful to both physicists and machine learning practitioners.\n\nThe current paper is an excellent step in this direction.\n\nHowever, with several semesters of undergrad quantum mechanics courses, I found it difficult to follow the calculations needed to compute predictions. \n\nMoreover, given my graduate-level experience in machine learning, it was even more difficult to clearly understand the data and training algorithm. \n\nIn the current state I do not think ICLR is an appropriate venue for this work, as it may confuse machine learning practitioners. (Though, I think the original goal of the paper is very worthy, and look forward to the final version of this work!)\n\nTo try to give constructive critique:\n\n- it took me a long time to understand the setup of the problem. An illustration of FashionMNIST or MNIST would help, with an incoming photon, an arrow through the cut-out of the image (clearly delineating that the image is binarized, and that white pixels are cut out, allowing the photon to pass through), and a detector (An LCD screen is described — is this the detector as well? Or does the photon bounce back and is then detected?). Such an illustration would go a long way for a machine learning practitioner unfamiliar with the double slit experiment and whatnot.\n\n- an algorithm box. A machine learning audience is used to thinking in terms of training data and algorithms. In this case, an algorithm box would help, specifically as it shows where the additional information is coming from. It seems like the 'training' is in optimizing the parameters of the unitary transform. Clearly delineating input, output, and optimization steps will help clarify the method. \n\n- an equation for computing the predictions of the trained model. Given a photon that passed through a mask, a clear formula for computing the class probability with the trained unitary operator parameters.\n\n- clearly describing the baseline. It was hard to find the details of the classical performance reported. 'maximal classical performance' is confusing wording, and implies that 'maximal' is proven theoretically. Is there a citation for this? I may have missed it. If it is not proven theoretically, then the wording should be changed, and a clear description of the architecture, training data, and training algorithm should be used, and code should be included in the supplement. This will help machine learning folks understand exactly what the comparison is against. From a machine learning standpoint, is the single pixel input to the model randomly sampled every time? \n\n- developing an additional baseline. Finding a unitary transform to find bases corresponding to style and class is unfair to the classical method, which does not have access to this information. Not having a baseline that uses this additional information will further confuse machine learning folks, as it seems obvious that a model that uses additional information will outperform a 'classical' model that does not use this information. For example, a tensor decomposition/SVD that uses information about style and class might be possible.\n\nHope this is helpful; I think with this additional work it could be quite a valuable contribution, as I think the ICLR community could be inspired to develop more methods that require complex-valued numbers such as this one.\n\nEdit:\n\n- the authors meaningfully addressed the above points. I have raised my score accordingly.", "extra_scores": {}, "extra_text": {}}, {"review_id": "j9LcZ_ef5CN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies that  a ML system using quantum interference gives better classification accuracy than a vanilla ML system, under the  constraint that a classification decision has to be made after detection of the very first photon that passed through an image-filter.\n\nThe reader can gather that this work brings together the ML and QC worlds but it is not clear what the real motivation of this work is and primarily why is ‘single-photon’ important. Does single photon equate to a single pixel? Or is this denoting the very first photon that passed the filter? Also is this constraint of detecting the very first photon valid? \n\nIt might be good to know which audience reads the paper. If it is the ML audience, then a section on QC basics/terminology will help. Or at least a graphical abstract to drive home the point home, will be helpful. \n\nWas there a reason to use the Fashion-MNIST in conjunction with MNIST dataset? The authors can also consider to abbreviate Fashion-MNIST to F-MNIST throughout the paper. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cTXzmA_5YvI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper identifies two atomic problems, respectively in fields of ML (MNIST classification) and quantum mechanics (measuring a single photon), and brings them together in a simplified setup that uses a single photon emitted according to the spatial distribution of images to classify MNIST/Fashion-MNIST. The introduction of quantum mechanics into the problem is through a trainable computational model of a beam splitter/phase shifter mechanism, aka a rotation in a high dimensional complex space, that's allowed to alter the photon's state before hitting the measurement device. The paper shows that using this overly simplified (and claimed to be physically feasible) quantum computer, which acts as the representation learning layer, improves classification accuracy over any other representation learning method that doesn't use quantum computing. The major take-away is an accessible demonstration of how an elementary quantum computer might work for ML, and what may be possible with actual qubits.\n\nStrengths:\n* The paper sets out to use two textbook problems in ML and quantum mechanics to introduce a textbook problem at the intersection, and does a fairly good job at analyzing the problem extensively. Given that the overall problem is of broad interest to the representation learning community, the solid execution of the paper is itself a good argument for acceptance.\n* Accessible explanation of quantum states, the measurement process, and the building blocks of quantum computing.\n\nComment:\n* The paper analyzes a single problem where no classical representation learning method can improve accuracy due to the fact that there is a single photon. It would be very beneficial to allude to other setups, perhaps in a related work section, to provide broader context and help the reader understand better the significance of the problem.", "extra_scores": {}, "extra_text": {}}, {"review_id": "G_0WxRu7Egi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the quantum computing based machine learning and proposes a toy model to illustrate the quantum information processing. On the common used handwritten digit dataset MNIST, more than 40% images can be classified accurately. The proposed method looks interesting and the focused problem (combining quantum computing and machine learning) is of certain significance.\n\nStrength:\n+ The topic is interesting, which inspires the following researchers to focus on the combination of quantum computing and machine learning. Both the theoretical analysis and experimental results demonstrate that the proposed classifier works well.\n+ The visualization results in Figure 4 are interesting. With the proposed photon classifier, the semantic information of the input images can be well extracted. The photon classifier tends to produce large amplitudes for the right classes.\n+ The proposed method is analyzed in detail from multiple perspectives, including results on MNIST, confusion matrices and visualization.\n\nWeakness:\n+ In the Results section, only experiment results of the proposed method is shown. There are some previous works related to this paper, such as [r1]. More comparisons and discusses between the proposed method and previous methods are desired. \n+ The experiments are conducted on two simple datasets, MNIST and Fashion-MNIST. Real image data are more complex, such as colored natural images. Could the proposed method be applied on more complex data? If can, how to extend and apply it? Please provide such a discussion.\n[r1] Erfan Khoram, Ang Chen, Dianjing Liu, et.al. Nanophotonic media for artificial neural inference. Photon. Res., 7(8):823–827, Aug 2019.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an unusual, but interesting submission. Can we use a simple "quantum computer" (in fact, physical system) to solve classification problems in ML? A single photon passes through the screen. Its state is described by the complex vector. A quantum computer makes a unitary linear transformation on this state in such a way that it maximizes the overlap with a corresponding class. Such a model can be parametrized by conventional means, and trained and later possibly realized by an quantum system

Pros:  
1.  The area of QC is very important, and such papers shed a new light on the subject. 
2. Inspiration to the ICLR community to work on in this area.
3. Technically correct.


Cons: 
1. The accuracies are far from SOTA and use very toy datasets. It is not clear, how to get to the accuracies needed in practice.
2. The actual computational speed of inference is not clear.
3. Discussion of more complicated models and their possibility is necessary.
4. Quite a few misprints are in the text which need to be fixed in the final version.




Decision: Accept (Poster)