Learning with Holographic Reduced Representations

1 Introduction . Symbolic and connectionist ( or “ neural ” ) based approaches to Artificial Intelligence ( AI ) and Machine Learning ( ML ) have often been treated as two separate , independent methods of approaching AI/ML . This does not need to be the case , and our paper proposes to study the viability of a hybrid approach to propagation based learning . In particular , we make use of the Holographic Reduced Representation ( HRR ) approach originally proposed by Plate [ 1 ] . Plate proposed using circular convolution as a “ binding ” operator . Given two vectors in a d dimensional feature space , x , y 2 Rd , they can be “ bound ” together using circular convolution , which we denote as s = x ⌦ y . This gives us a new result s 2 Rd . The HRR approach also includes an inversion operation † that maps Rd ! Rd . With this binding and inverse operation , Plate showed that we can assign vectors to have a conceptual symbolic meaning and construct statements ( or “ sentences ” ) from these representations . For example , we can construct S = red ⌦ cat + blue ⌦ dog to represent “ a red cat and a blue dog ” . HRRs can also query the statement representation S. To ask which animal was red , we compose S ⌦ red† ⇡ cat which gives us a numeric output approximately equal to the vector representing “ cat ” . The HRR framework requires a constant amount of memory , relies on well-optimized and scalable operations like the Fast Fourier Transform ( FFT ) , provides symbolic manipulation , and uses only differentiable operations , that would make it seem ideal as a tool for exploring neuro-symbolic modeling . Unfortunately back-propagating through HRR operations does not learn in practice , ⇤Research work completed prior to joining Amazon . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . rendering it seemingly moot for such neuro-symbolic research . The goal of this work is to find sufficient conditions for successful back-propagation based learning with the HRR framework , and to develop evidence of its potential for designing future nero-symbolic architectures . Because the utility of HRRs for cognitive-science tasks are already well established , our application area will focus on a multi-label machine learning task to show how HRRs can provide potential value to research in areas it has not been previously associated with . Backpropagating through HRRs is ineffective if done naively , and our goal is to show how to rectify this issue with a simple projection step and build viable loss functions out of HRR operations . We choose to do this with eXtreme Multi-Label ( XML ) classification tasks as one that is impossible for HRRs to tackle today . XML tasks have an input x , from which we have a large number L of binary prediction problems Y1 , Y2 , . . . , YL . Our approach will be to represent each of the L classes as a HRR concept vector , and construct an alternative to a fully connected output layer that uses the HRR framework for learning . Our experiments and ablations will focus on the impact of replacing a simple output layer with HRRs and other changes to the HRR approach to better understand its behaviors . State-of-the-art XML performance is not a goal . Our contributions and the rest of our paper are organized as follows . In §2 we discuss prior approaches related to our own , and how our work differs from these methods . In §3 we give a brief overview of the HRR operator ’ s details given its niche recognition within the AI/ML community , and introduce a simple improvement that increases the binding stability and effectiveness by 100⇥ . Then in §4 we show how we can leverage HRR ’ s symbolic nature to create a new output layer and loss function combination that provides several benefits to deep extreme multi-label models . With this new method we demonstrate in §5 that compared to a simple fully-connected output layer we can reduce the output size by as much as 99.55 % , resulting in a total reduction in model size by up to 42.09 % , reduce training time by 41.86 % , and obtain similar or improved relative accuracy by up to 30 % . We also show that these results are often competitive with more advanced approaches for XML classification , though do still suffer at the largest output spaces with 670k labels . Finally we conclude in §6 . 2 Related Work . Smolensky [ 2 ] elucidated early arguments for neuro-symbolic approaches with the Tensor Product Representation ( TPR ) that defined the first Vector Symbolic Architecture ( VSA ) , defining binding and unbinding operations that allow symbolic style object manipulation atop some vector field . As argued by Smolensky [ 2 ] ( and echoed by Greff et al . [ 3 ] ) , symbolic logic has greater power for complex tasks , but is brittle in requiring precise specification , where connectionist deep learning is more robustly able to process the raw inputs to a system , its capacity for more powerful “ logic ” is still limited and hotly debated . Combining these two domains in a neuro-symbolic hybrid system may yield positive results , and several recent works have found success in natural language processing tasks by augmenting recurrent neural networks with TPR inspired designs [ 4–6 ] . However , binding n vectors that are each d dimensions requires O ( dn ) space using TPRs . Thus VSAs that require a fixed dimension are computational preferable , but no work to our knowledge has successfully used a VSA of fixed length in gradient based learning . Schlegel et al . [ 7 ] provide a comprehensive summary and comparison of fixed length VSAs , and the details of how we chose to select HRRs based on their potential for gradient based learning and compute/memory efficacy in current frameworks like PyTorch are presented in Appendix D. Though such fixed-length VSAs have not been used in gradient based learning , they have still proven effective low-power embedded computation enviroments due to low-level hardware optimizations possible with many VSAs [ 8 , 9 ] . The HRR operation has found significant use in cognitive science to create biologically plausible models of human cognition [ 10–13 ] . These approaches use the HRR ’ s operations in a primarily symbolic fashion to demonstrate human-like performance on a number of cognitive science tasks [ 14 ] . The vector representation is used as the foundation for biological plausibility after establishing that the operations required to implement HRR are within the plausible sphere of a biological substrate [ 15 ] . When learning has been incorporated in these models it has been primarily through examples of Hebbian updates of spiking models on the outputs of HRR , rather than back-propagation through the HRR operations and preceding inputs/layers [ 16 ] . In our work we advance the understanding of how to learn through the HRR through propagation in an effective manner and how HRR can be leveraged as an integral part of a neural-symbolic approach . Little work exists on gradient based learning with HRR operations explicitly . The only work we are aware that attempts to leverage the symbolic properties of the HRR is by Nickel et al . [ 17 ] , who use the binding operations to connect elements in a knowledge graph as a kind of embedding that combines two vectors of information without expanding the dimension of the representation ( e.g. , concatenation would combine but double the dimension ) . More recently Liao and Yuan [ 18 ] used circular convolution as a direct replacement of normal convolution to reduce model size and inference time , without any leverage of the symbolic properties available within HRRs . While Danihelka et al . [ 19 ] purport to embed the HRR into an LSTM , their approach only augments an LSTM by including complex weights and activations2 , and does not actually use HRR as it lacks circular convolution , and all three works do not leverage the inverse operation † or seek to leverage any symbolic manipulations . By contrast our work explicitly requires the symbolic manipulation properties of HRR to create a vector-symbolic loss function , and we need to leverage the inverse operator † , meaning we use the entire HRR framework . We also demonstrate new insights into learning with HRR operations that make them more effective , where prior works have simply used the ⌦ operator on 2 components without further study . There has been other work in differentiable neuro-symbolic systems outside of TPRs and VSAs , most notably using first-order logic [ 20–22 ] . These works represent another powerful alternative approach , though at this time are often more involved in their design and training procedures . We consider these beyond the scope of our current study , where we seek to obtain similar benefits using the simpler HRR that requires only the careful application of FFTs to leverage . This is an easily satisfiable constraint given their heavy optimization and wide spread use amongst existing deep learning frameworks . Our use of extreme multi-label classification problems is due to it being out-of-reach of current HRR methods , which we found produced random-guessing performance in all cases . There exists a rich literature of XML methods that tackle the large output space from the perspective of decision trees/ensembles [ 23–27 ] , label embedding regression [ 28–31 ] , naive bayes [ 32 ] , and linear classifiers [ 33 , 34 ] . There also exist deep learning XML methods that use either a fully-connected output layer [ 35 ] and others that use a variety of alternative approaches to dealing with the large output space [ 36–43 ] . The variety of approaches is orthogonal to the purpose of this work , which is to investigate HRRs . We will leverage only some of these works to modify their architectures to use a fully-connected output layer ( if they do not already ) and to use our HRR approach , so that we can observe the impact of the HRR specifically on the model . In our experimentation we found that the mechanisms for handling the high cardinality of outputs is highly architecture tuned and specific , making it difficult to do broader comparisons . Our work will explore fully-connected , convolutional , and recurrent architectures to show that our HRR approach can work broadly across many types of architectures that have been used , but it is not our goal to be the “ best ” at XML by any metric . The XML task ’ s value to our study is a problem that current HRR methods could not tackle , and where we can apply the symbolic properties of HRR in developing the solution in a way that demonstrates how one may design HRR based networks for other tasks . 3 Holographic Reduced Representation . The original paper by Plate [ 1 ] introduced the HRR approach and developed significant theory for its use as a symbolic approach to AI on a neural substrate . We will quickly review its foundation and numerical issues that lead to our projection based improvement . The HRR starts with circular convolution as a “ binding ” operator . We use Plate ’ s notation of ⌦ , and a definition of circular convolution using an FFT F and inverse FFT ( F 1 ) is given in eq . ( 1 ) . It is easy to see that the binding operator of circular convolution ⌦ is commutative . What is particularly interesting is the ability to unbind terms that have been bound by this operator . Let F ( a ) i denote the i ’ th complex feature of the FFT of a . By defining an identity function F ( a† ) iF ( a ) i = 1 , we can derive the inverse function eq . ( 2 ) , where we raise each complex coefficient zi of the FFT to the z 1i power before going back to the real domain . a⌦ b = F 1 ( F ( a ) F ( b ) ) ( 1 ) a † = F 1 ✓ 1 F ( a ) ◆ ( 2 ) 2HRRs & circular convolution can naturally exist in the reals . Using this , Plate [ 1 ] showed that one can compose sentences like our example S = red⌦cat+blue⌦ dog by assigning concepts to vectors in a d dimensional space . We can unbind components with the inverse operation , giving S ⌦ cat† ⇡ red , or we can check for the existence of a term by checking that S > cat† ⇡ 1 . A term not present , like “ cow ” , would behave as S > cow† ⇡ 0 . These sentences can be constructed with more complex logic . The binding is also distributive ( and associative ) , so a⌦ ( x+ y ) = a⌦ x+ a⌦ y . This allows us to construct complex symbolic relationships . As an example , we could have : S = first⌦ ( name⌦Jane+age⌦30 ) +second⌦ ( name⌦Joe+age⌦33 ) , and to determine the name of the first person we can perform first† ⌦ name† ⌦ S ⇡ Jane . To make this symbolic manipulations effective , Plate [ 1 ] introduced two points of sufficiency . First , a vector a used for HRR operations should have its values sampled from a normal distribution : a ⇠ N 0 , Id · d 1 . This allows binding and unbinding to work in expectation . Second , is that the inverse † is numerically unstable . The remedy developed is the pseudo inverse ⇤ , given by a⇤ = [ a1 , ad , ad 1 , . . . , a2 ] . This approximation was justified by noting that in polar form one obtains Fj a† = 1rj e i✓j and Fj ( a⇤ ) = rje i✓j , which are off by the reciprocal of the complex magnitude . Plate and others [ 14 ] use the pseudo inverse ⇤ exclusively as the error due to approximation of † was smaller than the errors caused by numerical instability . In Appendix B we provide additional illustration about the potential modeling with HRRs , and worked examples demonstrating why the math works is provided in Appendix B .

Summary: This paper investigates the technique of “Holographic Reduced Representation” (HRR), developed in the early 90’s by Tony Plate as a tool for symbolic reasoning, for use in modern neural network architectures trained using backpropagation. The authors find that Plate’s original formulation leads to low accuracy in this setting and present a simple fix that improves performance. The authors apply the technique to classification problems in which the outcome is a high-dimensional binary vector and show that HRRs can be used effectively in conjunction with gradient based training methods in this setting. The paper is well written, technically sound, and discusses an interesting avenue for bridging classic work on connectionist AI with more modern techniques.