Classical and Quantum Algorithms for Orthogonal Neural Networks

1 INTRODUCTION . In the evolution of neural network structures , adding constraints to the weight matrices has often been an effective path . Recently , orthogonal neural networks ( OrthoNNs ) have been proposed Jia et al . ( 2019 ) ; Wang et al . ( 2020 ) ; Nosarzewski ( 2018 ) ; Bansal et al . ( 2018 ) as a new type of neural networks for which , at each layer , the weight matrix should remain orthogonal . This property is useful to reach higher accuracy performance and avoid vanishing or exploding gradient for deep architectures . Several classical gradient descent methods have been proposed to preserve the orthogonality while updating the weight matrices . However , these techniques suffer from longer running time and sometimes only approximate the orthogonality . In particular , the main method for achieving orthogonality during training is to first perform the usual gradient descent to update the weight matrix ( which is now not going to be orthogonal ) and then perform Singular Value Decomposition to orthogonalize or almost orthogonalize the weight matrix . We can see then why achieving orthogonality hinders a fast training , since at every step an SVD computation needs to be performed ( See Section A.1 ) . In the emergent field of quantum machine learning , several proposals have been made to implement neural networks . Some algorithms rely on long term and perfect quantum computers Kerenidis et al . ( 2020 ) ; Allcock et al . ( 2018 ) , while others try to harness the existing quantum devices using variational circuits Cong et al . ( 2019 ) ; Farhi & Neven ( 2018 ) . As in classical neural networks , they use gradient descent methods to update the quantum parameters of the circuits . Such quantum neural networks have been trained for very small sizes , however there is still a need to understand how such architectures will scale and whether they will provide efficient and accurate training . In this work , we present a new training method for neural networks that preserves perfect orthogonality while having the same running time as usual gradient descent methods without the orthogonality condition , thus achieving the best of both worlds , most efficient training and perfect orthogonality . The main idea comes from the quantum world , where we know that any quantum circuit corresponds to an operation described by a unitary matrix , which if we only use gates with real amplitudes is an orthogonal matrix . In particular , we propose a novel special-architecture quantum circuit , for which there is an efficient way to map the elements of the orthogonal weight matrix to the parameters of the gates of the quantum circuit and vice versa . In other words , while performing a gradient descent on the elements of the weight matrix individually does not preserve orthogonality , performing a gradient descent on the parameters of the quantum circuit preserves orthogonality ( since any quantum circuit with real parameters corresponds to an orthogonal matrix ) and is equivalent to updating the weight matrix . We also prove that performing gradient descent on the parameters of the quantum circuit can be done efficiently classically ( with constant update cost per parameter ) thus concluding that there exists a quantum-inspired , but fully classical way of efficiently training perfectly orthogonal neural networks . Moreover , the special-architecture quantum circuit we defined has many properties that make it a good candidate for NISQ implementation : it uses only one type of quantum gates , requires simple connectivity between the qubits , has depth linear in the input and output node sizes , and benefits from powerful error mitigation techniques that make it resilient to noise . This allows us to also propose an inference method running the quantum circuit on data which might offer a faster running time , given the shallow depth of the quantum circuit . Our main contributions are summarized in Table 1 , where we have considered the time to perform a feedforward pass , or one gradient descent step . A single neural network layer is considered , with input and output of size n . 2 A PARAMETRIZED QUANTUM CIRCUIT FOR ORTHOGONAL NEURAL NETWORKS . In this section , we define a special-architecture parametrized quantum circuit that will be useful for performing training and inference on orthogonal neural networks . As we said , the training will be completely classical in the end , but the intuition of the new method comes from this quantum circuit , while the inference can happen both classically or by applying this quantum circuit . A basic introduction to quantum computing concepts necessary for this work is given in the Appendix ( Section A.3 ) . 2.1 THE RBS GATE The quantum circuit proposed in this work ( see Fig.1 ) , which implements a fully connected neural network layer with an orthogonal weight matrix , uses only one type of quantum gate , the Reconfigurable Beam Splitter ( RBS ) gate . This two-qubit gate is parametrizable with one angle θ ∈ [ 0 , 2π ] . Its matrix representation is given as : RBS ( θ ) = 1 0 0 00 cos θ sin θ 00 − sin θ cos θ 0 0 0 0 1 RBS ( θ ) : { |01〉 7→ cos θ |01〉 − sin θ |10〉|10〉 7→ sin θ |01〉+ cos θ |10〉 ( 1 ) We can think of this gate as a rotation in the two-dimensional subspace spanned by the basis { |01〉 , |10〉 } , while it acts as the identity in the remaining subspace { |00〉 , |11〉 } . Or equivalently , starting with two qubits , one in the |0〉 state and the other one in the state |1〉 , the qubits can be swapped or not in superposition . The qubit |1〉 stays on its wire with amplitude cos θ or switches with the other qubit with amplitude + sin θ if the new wire is below ( |10〉 7→ |01〉 ) or − sin θ if the new wire is above ( |01〉 7→ |10〉 ) . Note that in the two other cases ( |00〉 and |11〉 ) the RBS gate acts as identity . 2.2 QUANTUM PYRAMIDAL CIRCUIT . We now propose a quantum circuit that implements an orthogonal layer of a neural network . The circuit is a pyramidal structure of RBS gates , each with an independent angle , as represented in Fig.2a . In Section 2.3 and 3 , more details are provided concerning respectively the input loading , and the equivalence with a neural network ’ s orthogonal layer . To mimic a given classical layer with a quantum circuit , the number of output qubits should be the size of the classical layer ’ s output . We refer to the square case when the input and output sizes are equal , and to the rectangular case otherwise ( Fig.3a ) . The important property to note is that the number of parameters of the quantum pyramidal circuit corresponding to a neural network layer of size n× d is ( 2n− 1− d ) ∗ d/2 exactly the same as the number of degrees of freedom of an orthogonal matrix of dimension n× d. For simplicity , we pursue our analysis using only the square case but everything can be easily extended to the rectangular case . As we said , the full pyramidal structure of the quantum circuit described above imposes the number of free parameters to be N = n ( n − 1 ) /2 , the exact number of free parameters to specify a n× n orthogonal matrix . In Section 3 we will show how the parameters of the gates of this pyramidal circuit can be easily related to the elements of the orthogonal matrix of size n×n that describes it . We note that alternative architectures can be imagined as long as the number of gate parameters is equal to the parameters of the orthogonal weight matrix and a simple mapping between them and the elements of the weight matrix can be found . Note finally that this circuit has linear depth and is convenient for near term quantum hardware platforms with restricted connectivity . Indeed , the distribution of the RBS gates requires only nearest neighbor connectivity between qubits . 2.3 LOADING THE DATA . Before applying the quantum pyramidal circuit , we will need to upload the classical data into the quantum circuit . We will use one qubit per feature of the data . For this , we use a unary amplitude encoding of the input data . Let ’ s consider an input sample x = ( x0 , · · · , xn−1 ) ∈ Rn , such that ‖x‖2 = 1 . We will encode it in a superposition of unary states : |x〉 = x0 |10 · · · 0〉+ x1 |010 · · · 0〉+ · · ·+ xn−1 |0 · · · 01〉 ( 2 ) We can also rewrite the previous state as |x〉 = ∑n−1 i=0 xi |ei〉 , where |ei〉 represents the ith unary state with a |1〉 in the ith position |0 · · · 010 · · · 0〉 . Recent work Kerenidis ( U.S. Patent Application No . 16/986,553 and 16/987,235 , 2020 ) proposed a logarithmic depth data loader circuit for loading such states . Here we will use a much simpler circuit . It is a linear depth cascade of n−1RBS gates which , due to the particular structure of our quantum pyramidal circuit , only adds 2 extra steps to our circuit . The circuit starts in the all |0〉 state and flips the first qubit using an X gate , in order to obtain the unary state |10 · · · 0〉 as shown in Fig.4 . Then a cascade of RBS gates allow to create the state |x〉 using a set of n − 1 angles α0 , · · · , αn−2 . Using Eq . ( 1 ) , we will choose the angles such that , after the first RBS gate of the loader , the qubits would be in the state x0 |100 · · ·〉 + sin ( α0 ) |010 · · ·〉 and after the second one in the state x0 |100 · · ·〉 + x1 |010 · · ·〉 + sin ( α0 ) sin ( α1 ) |001 · · ·〉 and so on , until obtaining |x〉 as in Eq. ( 2 ) . To this end , we simply perform a classical preprocessing to compute recursively the n − 1 loading angles , in time O ( n ) . We choose α0 = arccos ( x0 ) , α1 = arccos ( x1 sin −1 ( α0 ) ) , α2 = arccos ( x2 sin−1 ( α0 ) sin−1 ( α1 ) ) and so on . The ability of loading data in such a way relies on the assumption that each input vector is normalized , i.e . ‖x‖2 = 1 . This normalization constraint could seem arbitrary and impact the ability to learn from the data . In fact , in the case of an orthogonal neural network , this normalization shouldn ’ t degrade the training because orthogonal weight matrices are in fact orthonormal and thus norm-preserving . Hence , changing the norm of the input vector , by diving each component by ‖x‖2 , in both classical and quantum settings is not a problem . The normalization would impose that each input has the same norm , or the same ” luminosity ” in the context of images , which can be helpful or harmful depending on the case .

Summary: The authors introduce a quantum pyramidal circuit to achieve an orthogonal layer of a neural network, which is fast and can maintain orthogonality. The angle gradients are derived. The authors implement the orthogonal NN on simulators and quantum machines to demonstrate the effectiveness.