Classical and Quantum Algorithms for Orthogonal Neural Networks

1 INTRODUCTION . In the evolution of neural network structures , adding constraints to the weight matrices has often been an effective path . Recently , orthogonal neural networks ( OrthoNNs ) have been proposed Jia et al . ( 2019 ) ; Wang et al . ( 2020 ) ; Nosarzewski ( 2018 ) ; Bansal et al . ( 2018 ) as a new type of neural networks for which , at each layer , the weight matrix should remain orthogonal . This property is useful to reach higher accuracy performance and avoid vanishing or exploding gradient for deep architectures . Several classical gradient descent methods have been proposed to preserve the orthogonality while updating the weight matrices . However , these techniques suffer from longer running time and sometimes only approximate the orthogonality . In particular , the main method for achieving orthogonality during training is to first perform the usual gradient descent to update the weight matrix ( which is now not going to be orthogonal ) and then perform Singular Value Decomposition to orthogonalize or almost orthogonalize the weight matrix . We can see then why achieving orthogonality hinders a fast training , since at every step an SVD computation needs to be performed ( See Section A.1 ) . In the emergent field of quantum machine learning , several proposals have been made to implement neural networks . Some algorithms rely on long term and perfect quantum computers Kerenidis et al . ( 2020 ) ; Allcock et al . ( 2018 ) , while others try to harness the existing quantum devices using variational circuits Cong et al . ( 2019 ) ; Farhi & Neven ( 2018 ) . As in classical neural networks , they use gradient descent methods to update the quantum parameters of the circuits . Such quantum neural networks have been trained for very small sizes , however there is still a need to understand how such architectures will scale and whether they will provide efficient and accurate training . In this work , we present a new training method for neural networks that preserves perfect orthogonality while having the same running time as usual gradient descent methods without the orthogonality condition , thus achieving the best of both worlds , most efficient training and perfect orthogonality . The main idea comes from the quantum world , where we know that any quantum circuit corresponds to an operation described by a unitary matrix , which if we only use gates with real amplitudes is an orthogonal matrix . In particular , we propose a novel special-architecture quantum circuit , for which there is an efficient way to map the elements of the orthogonal weight matrix to the parameters of the gates of the quantum circuit and vice versa . In other words , while performing a gradient descent on the elements of the weight matrix individually does not preserve orthogonality , performing a gradient descent on the parameters of the quantum circuit preserves orthogonality ( since any quantum circuit with real parameters corresponds to an orthogonal matrix ) and is equivalent to updating the weight matrix . We also prove that performing gradient descent on the parameters of the quantum circuit can be done efficiently classically ( with constant update cost per parameter ) thus concluding that there exists a quantum-inspired , but fully classical way of efficiently training perfectly orthogonal neural networks . Moreover , the special-architecture quantum circuit we defined has many properties that make it a good candidate for NISQ implementation : it uses only one type of quantum gates , requires simple connectivity between the qubits , has depth linear in the input and output node sizes , and benefits from powerful error mitigation techniques that make it resilient to noise . This allows us to also propose an inference method running the quantum circuit on data which might offer a faster running time , given the shallow depth of the quantum circuit . Our main contributions are summarized in Table 1 , where we have considered the time to perform a feedforward pass , or one gradient descent step . A single neural network layer is considered , with input and output of size n . 2 A PARAMETRIZED QUANTUM CIRCUIT FOR ORTHOGONAL NEURAL NETWORKS . In this section , we define a special-architecture parametrized quantum circuit that will be useful for performing training and inference on orthogonal neural networks . As we said , the training will be completely classical in the end , but the intuition of the new method comes from this quantum circuit , while the inference can happen both classically or by applying this quantum circuit . A basic introduction to quantum computing concepts necessary for this work is given in the Appendix ( Section A.3 ) . 2.1 THE RBS GATE The quantum circuit proposed in this work ( see Fig.1 ) , which implements a fully connected neural network layer with an orthogonal weight matrix , uses only one type of quantum gate , the Reconfigurable Beam Splitter ( RBS ) gate . This two-qubit gate is parametrizable with one angle θ ∈ [ 0 , 2π ] . Its matrix representation is given as : RBS ( θ ) = 1 0 0 00 cos θ sin θ 00 − sin θ cos θ 0 0 0 0 1 RBS ( θ ) : { |01〉 7→ cos θ |01〉 − sin θ |10〉|10〉 7→ sin θ |01〉+ cos θ |10〉 ( 1 ) We can think of this gate as a rotation in the two-dimensional subspace spanned by the basis { |01〉 , |10〉 } , while it acts as the identity in the remaining subspace { |00〉 , |11〉 } . Or equivalently , starting with two qubits , one in the |0〉 state and the other one in the state |1〉 , the qubits can be swapped or not in superposition . The qubit |1〉 stays on its wire with amplitude cos θ or switches with the other qubit with amplitude + sin θ if the new wire is below ( |10〉 7→ |01〉 ) or − sin θ if the new wire is above ( |01〉 7→ |10〉 ) . Note that in the two other cases ( |00〉 and |11〉 ) the RBS gate acts as identity . 2.2 QUANTUM PYRAMIDAL CIRCUIT . We now propose a quantum circuit that implements an orthogonal layer of a neural network . The circuit is a pyramidal structure of RBS gates , each with an independent angle , as represented in Fig.2a . In Section 2.3 and 3 , more details are provided concerning respectively the input loading , and the equivalence with a neural network ’ s orthogonal layer . To mimic a given classical layer with a quantum circuit , the number of output qubits should be the size of the classical layer ’ s output . We refer to the square case when the input and output sizes are equal , and to the rectangular case otherwise ( Fig.3a ) . The important property to note is that the number of parameters of the quantum pyramidal circuit corresponding to a neural network layer of size n× d is ( 2n− 1− d ) ∗ d/2 exactly the same as the number of degrees of freedom of an orthogonal matrix of dimension n× d. For simplicity , we pursue our analysis using only the square case but everything can be easily extended to the rectangular case . As we said , the full pyramidal structure of the quantum circuit described above imposes the number of free parameters to be N = n ( n − 1 ) /2 , the exact number of free parameters to specify a n× n orthogonal matrix . In Section 3 we will show how the parameters of the gates of this pyramidal circuit can be easily related to the elements of the orthogonal matrix of size n×n that describes it . We note that alternative architectures can be imagined as long as the number of gate parameters is equal to the parameters of the orthogonal weight matrix and a simple mapping between them and the elements of the weight matrix can be found . Note finally that this circuit has linear depth and is convenient for near term quantum hardware platforms with restricted connectivity . Indeed , the distribution of the RBS gates requires only nearest neighbor connectivity between qubits . 2.3 LOADING THE DATA . Before applying the quantum pyramidal circuit , we will need to upload the classical data into the quantum circuit . We will use one qubit per feature of the data . For this , we use a unary amplitude encoding of the input data . Let ’ s consider an input sample x = ( x0 , · · · , xn−1 ) ∈ Rn , such that ‖x‖2 = 1 . We will encode it in a superposition of unary states : |x〉 = x0 |10 · · · 0〉+ x1 |010 · · · 0〉+ · · ·+ xn−1 |0 · · · 01〉 ( 2 ) We can also rewrite the previous state as |x〉 = ∑n−1 i=0 xi |ei〉 , where |ei〉 represents the ith unary state with a |1〉 in the ith position |0 · · · 010 · · · 0〉 . Recent work Kerenidis ( U.S. Patent Application No . 16/986,553 and 16/987,235 , 2020 ) proposed a logarithmic depth data loader circuit for loading such states . Here we will use a much simpler circuit . It is a linear depth cascade of n−1RBS gates which , due to the particular structure of our quantum pyramidal circuit , only adds 2 extra steps to our circuit . The circuit starts in the all |0〉 state and flips the first qubit using an X gate , in order to obtain the unary state |10 · · · 0〉 as shown in Fig.4 . Then a cascade of RBS gates allow to create the state |x〉 using a set of n − 1 angles α0 , · · · , αn−2 . Using Eq . ( 1 ) , we will choose the angles such that , after the first RBS gate of the loader , the qubits would be in the state x0 |100 · · ·〉 + sin ( α0 ) |010 · · ·〉 and after the second one in the state x0 |100 · · ·〉 + x1 |010 · · ·〉 + sin ( α0 ) sin ( α1 ) |001 · · ·〉 and so on , until obtaining |x〉 as in Eq. ( 2 ) . To this end , we simply perform a classical preprocessing to compute recursively the n − 1 loading angles , in time O ( n ) . We choose α0 = arccos ( x0 ) , α1 = arccos ( x1 sin −1 ( α0 ) ) , α2 = arccos ( x2 sin−1 ( α0 ) sin−1 ( α1 ) ) and so on . The ability of loading data in such a way relies on the assumption that each input vector is normalized , i.e . ‖x‖2 = 1 . This normalization constraint could seem arbitrary and impact the ability to learn from the data . In fact , in the case of an orthogonal neural network , this normalization shouldn ’ t degrade the training because orthogonal weight matrices are in fact orthonormal and thus norm-preserving . Hence , changing the norm of the input vector , by diving each component by ‖x‖2 , in both classical and quantum settings is not a problem . The normalization would impose that each input has the same norm , or the same ” luminosity ” in the context of images , which can be helpful or harmful depending on the case .

Summary: This paper replaces usual weights and orthogonal matrices in orthogonal neural networks with an equivalent pyramidal circuit made of two-dimensional rotations.  By this, the authors proposed a training method for orthogonal neural networks that run in quadratic time, which is a significant improvement from previous methods based on Singular Value Decomposition. The authors also performed some numerical experiments to verify the abilities of their pyramidal circuit, on the standard MNIST dataset, which showed that their methods are efficient for learning a classification task.