Learning with Holographic Reduced Representations

1 Introduction . Symbolic and connectionist ( or “ neural ” ) based approaches to Artificial Intelligence ( AI ) and Machine Learning ( ML ) have often been treated as two separate , independent methods of approaching AI/ML . This does not need to be the case , and our paper proposes to study the viability of a hybrid approach to propagation based learning . In particular , we make use of the Holographic Reduced Representation ( HRR ) approach originally proposed by Plate [ 1 ] . Plate proposed using circular convolution as a “ binding ” operator . Given two vectors in a d dimensional feature space , x , y 2 Rd , they can be “ bound ” together using circular convolution , which we denote as s = x ⌦ y . This gives us a new result s 2 Rd . The HRR approach also includes an inversion operation † that maps Rd ! Rd . With this binding and inverse operation , Plate showed that we can assign vectors to have a conceptual symbolic meaning and construct statements ( or “ sentences ” ) from these representations . For example , we can construct S = red ⌦ cat + blue ⌦ dog to represent “ a red cat and a blue dog ” . HRRs can also query the statement representation S. To ask which animal was red , we compose S ⌦ red† ⇡ cat which gives us a numeric output approximately equal to the vector representing “ cat ” . The HRR framework requires a constant amount of memory , relies on well-optimized and scalable operations like the Fast Fourier Transform ( FFT ) , provides symbolic manipulation , and uses only differentiable operations , that would make it seem ideal as a tool for exploring neuro-symbolic modeling . Unfortunately back-propagating through HRR operations does not learn in practice , ⇤Research work completed prior to joining Amazon . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . rendering it seemingly moot for such neuro-symbolic research . The goal of this work is to find sufficient conditions for successful back-propagation based learning with the HRR framework , and to develop evidence of its potential for designing future nero-symbolic architectures . Because the utility of HRRs for cognitive-science tasks are already well established , our application area will focus on a multi-label machine learning task to show how HRRs can provide potential value to research in areas it has not been previously associated with . Backpropagating through HRRs is ineffective if done naively , and our goal is to show how to rectify this issue with a simple projection step and build viable loss functions out of HRR operations . We choose to do this with eXtreme Multi-Label ( XML ) classification tasks as one that is impossible for HRRs to tackle today . XML tasks have an input x , from which we have a large number L of binary prediction problems Y1 , Y2 , . . . , YL . Our approach will be to represent each of the L classes as a HRR concept vector , and construct an alternative to a fully connected output layer that uses the HRR framework for learning . Our experiments and ablations will focus on the impact of replacing a simple output layer with HRRs and other changes to the HRR approach to better understand its behaviors . State-of-the-art XML performance is not a goal . Our contributions and the rest of our paper are organized as follows . In §2 we discuss prior approaches related to our own , and how our work differs from these methods . In §3 we give a brief overview of the HRR operator ’ s details given its niche recognition within the AI/ML community , and introduce a simple improvement that increases the binding stability and effectiveness by 100⇥ . Then in §4 we show how we can leverage HRR ’ s symbolic nature to create a new output layer and loss function combination that provides several benefits to deep extreme multi-label models . With this new method we demonstrate in §5 that compared to a simple fully-connected output layer we can reduce the output size by as much as 99.55 % , resulting in a total reduction in model size by up to 42.09 % , reduce training time by 41.86 % , and obtain similar or improved relative accuracy by up to 30 % . We also show that these results are often competitive with more advanced approaches for XML classification , though do still suffer at the largest output spaces with 670k labels . Finally we conclude in §6 . 2 Related Work . Smolensky [ 2 ] elucidated early arguments for neuro-symbolic approaches with the Tensor Product Representation ( TPR ) that defined the first Vector Symbolic Architecture ( VSA ) , defining binding and unbinding operations that allow symbolic style object manipulation atop some vector field . As argued by Smolensky [ 2 ] ( and echoed by Greff et al . [ 3 ] ) , symbolic logic has greater power for complex tasks , but is brittle in requiring precise specification , where connectionist deep learning is more robustly able to process the raw inputs to a system , its capacity for more powerful “ logic ” is still limited and hotly debated . Combining these two domains in a neuro-symbolic hybrid system may yield positive results , and several recent works have found success in natural language processing tasks by augmenting recurrent neural networks with TPR inspired designs [ 4–6 ] . However , binding n vectors that are each d dimensions requires O ( dn ) space using TPRs . Thus VSAs that require a fixed dimension are computational preferable , but no work to our knowledge has successfully used a VSA of fixed length in gradient based learning . Schlegel et al . [ 7 ] provide a comprehensive summary and comparison of fixed length VSAs , and the details of how we chose to select HRRs based on their potential for gradient based learning and compute/memory efficacy in current frameworks like PyTorch are presented in Appendix D. Though such fixed-length VSAs have not been used in gradient based learning , they have still proven effective low-power embedded computation enviroments due to low-level hardware optimizations possible with many VSAs [ 8 , 9 ] . The HRR operation has found significant use in cognitive science to create biologically plausible models of human cognition [ 10–13 ] . These approaches use the HRR ’ s operations in a primarily symbolic fashion to demonstrate human-like performance on a number of cognitive science tasks [ 14 ] . The vector representation is used as the foundation for biological plausibility after establishing that the operations required to implement HRR are within the plausible sphere of a biological substrate [ 15 ] . When learning has been incorporated in these models it has been primarily through examples of Hebbian updates of spiking models on the outputs of HRR , rather than back-propagation through the HRR operations and preceding inputs/layers [ 16 ] . In our work we advance the understanding of how to learn through the HRR through propagation in an effective manner and how HRR can be leveraged as an integral part of a neural-symbolic approach . Little work exists on gradient based learning with HRR operations explicitly . The only work we are aware that attempts to leverage the symbolic properties of the HRR is by Nickel et al . [ 17 ] , who use the binding operations to connect elements in a knowledge graph as a kind of embedding that combines two vectors of information without expanding the dimension of the representation ( e.g. , concatenation would combine but double the dimension ) . More recently Liao and Yuan [ 18 ] used circular convolution as a direct replacement of normal convolution to reduce model size and inference time , without any leverage of the symbolic properties available within HRRs . While Danihelka et al . [ 19 ] purport to embed the HRR into an LSTM , their approach only augments an LSTM by including complex weights and activations2 , and does not actually use HRR as it lacks circular convolution , and all three works do not leverage the inverse operation † or seek to leverage any symbolic manipulations . By contrast our work explicitly requires the symbolic manipulation properties of HRR to create a vector-symbolic loss function , and we need to leverage the inverse operator † , meaning we use the entire HRR framework . We also demonstrate new insights into learning with HRR operations that make them more effective , where prior works have simply used the ⌦ operator on 2 components without further study . There has been other work in differentiable neuro-symbolic systems outside of TPRs and VSAs , most notably using first-order logic [ 20–22 ] . These works represent another powerful alternative approach , though at this time are often more involved in their design and training procedures . We consider these beyond the scope of our current study , where we seek to obtain similar benefits using the simpler HRR that requires only the careful application of FFTs to leverage . This is an easily satisfiable constraint given their heavy optimization and wide spread use amongst existing deep learning frameworks . Our use of extreme multi-label classification problems is due to it being out-of-reach of current HRR methods , which we found produced random-guessing performance in all cases . There exists a rich literature of XML methods that tackle the large output space from the perspective of decision trees/ensembles [ 23–27 ] , label embedding regression [ 28–31 ] , naive bayes [ 32 ] , and linear classifiers [ 33 , 34 ] . There also exist deep learning XML methods that use either a fully-connected output layer [ 35 ] and others that use a variety of alternative approaches to dealing with the large output space [ 36–43 ] . The variety of approaches is orthogonal to the purpose of this work , which is to investigate HRRs . We will leverage only some of these works to modify their architectures to use a fully-connected output layer ( if they do not already ) and to use our HRR approach , so that we can observe the impact of the HRR specifically on the model . In our experimentation we found that the mechanisms for handling the high cardinality of outputs is highly architecture tuned and specific , making it difficult to do broader comparisons . Our work will explore fully-connected , convolutional , and recurrent architectures to show that our HRR approach can work broadly across many types of architectures that have been used , but it is not our goal to be the “ best ” at XML by any metric . The XML task ’ s value to our study is a problem that current HRR methods could not tackle , and where we can apply the symbolic properties of HRR in developing the solution in a way that demonstrates how one may design HRR based networks for other tasks . 3 Holographic Reduced Representation . The original paper by Plate [ 1 ] introduced the HRR approach and developed significant theory for its use as a symbolic approach to AI on a neural substrate . We will quickly review its foundation and numerical issues that lead to our projection based improvement . The HRR starts with circular convolution as a “ binding ” operator . We use Plate ’ s notation of ⌦ , and a definition of circular convolution using an FFT F and inverse FFT ( F 1 ) is given in eq . ( 1 ) . It is easy to see that the binding operator of circular convolution ⌦ is commutative . What is particularly interesting is the ability to unbind terms that have been bound by this operator . Let F ( a ) i denote the i ’ th complex feature of the FFT of a . By defining an identity function F ( a† ) iF ( a ) i = 1 , we can derive the inverse function eq . ( 2 ) , where we raise each complex coefficient zi of the FFT to the z 1i power before going back to the real domain . a⌦ b = F 1 ( F ( a ) F ( b ) ) ( 1 ) a † = F 1 ✓ 1 F ( a ) ◆ ( 2 ) 2HRRs & circular convolution can naturally exist in the reals . Using this , Plate [ 1 ] showed that one can compose sentences like our example S = red⌦cat+blue⌦ dog by assigning concepts to vectors in a d dimensional space . We can unbind components with the inverse operation , giving S ⌦ cat† ⇡ red , or we can check for the existence of a term by checking that S > cat† ⇡ 1 . A term not present , like “ cow ” , would behave as S > cow† ⇡ 0 . These sentences can be constructed with more complex logic . The binding is also distributive ( and associative ) , so a⌦ ( x+ y ) = a⌦ x+ a⌦ y . This allows us to construct complex symbolic relationships . As an example , we could have : S = first⌦ ( name⌦Jane+age⌦30 ) +second⌦ ( name⌦Joe+age⌦33 ) , and to determine the name of the first person we can perform first† ⌦ name† ⌦ S ⇡ Jane . To make this symbolic manipulations effective , Plate [ 1 ] introduced two points of sufficiency . First , a vector a used for HRR operations should have its values sampled from a normal distribution : a ⇠ N 0 , Id · d 1 . This allows binding and unbinding to work in expectation . Second , is that the inverse † is numerically unstable . The remedy developed is the pseudo inverse ⇤ , given by a⇤ = [ a1 , ad , ad 1 , . . . , a2 ] . This approximation was justified by noting that in polar form one obtains Fj a† = 1rj e i✓j and Fj ( a⇤ ) = rje i✓j , which are off by the reciprocal of the complex magnitude . Plate and others [ 14 ] use the pseudo inverse ⇤ exclusively as the error due to approximation of † was smaller than the errors caused by numerical instability . In Appendix B we provide additional illustration about the potential modeling with HRRs , and worked examples demonstrating why the math works is provided in Appendix B .

Summary: This paper introduces a modification to the Holographic Reduced Representation (HRR) method, which are a pair of encoding/decoding operations based on circular convolution; these operations can be used to efficiently approximate symbolic querying, information storage, and retrieval operations in a low-dimensional vector embedding space.  used for approximating symbolic storage and retrieval operations.  The first contribution of this paper is a modification to the original HRR method (introduced by Plate 1995 and primarily used in biologically-plausible computation models) that increases its query accuracy.  The second contribution to this paper is a proof-of-concept loss function that allows the modified HRR encoding to be incorporated into differentiable neural architectures.  The final contribution of this paper is a set of experiments showing that concrete HRR-based models can be used in extreme-multi-label (XML) classification tasks, in some cases exceeding baseline accuracy with standard fully-connected layers used for lookup and retrieval (while using a comparatively smaller and more efficient encoding/decoding scheme.)