Title: A Walsh Hadamard Derived Linear Vector Symbolic Architecture
Venue: NeurIPS 2024
Accepted: yes
Keywords: Vector Symbolic Architectures

Holographic Reduced Representations

Hadamard Transformation

HRR

VTB

MAP

HLB
Abstract: Vector Symbolic Architectures (VSAs) are one approach to developing Neuro-symbolic AI, where two vectors in $\mathbb{R}^d$ are 'bound' together to produce a new vector in the same space. VSAs support the commutativity and associativity of this binding operation, along with an inverse operation, allowing one to construct symbolic-style manipulations over real-valued vectors. Most VSAs were developed before deep learning and automatic differentiation became popular and instead focused on efficacy in hand-designed systems. In this work, we introduce the Hadamard-derived linear Binding (HLB), which is designed to have favorable computational efficiency, and efficacy in classic VSA tasks, and perform well in differentiable systems.

A Walsh Hadamard Derived Linear Vector Symbolic Architecture

Abstract

Vector Symbolic Architectures (VSAs) are one approach to developing Neurosymbolic AI, where two vectors in R d are 'bound' together to produce a new vector in the same space. VSAs support the commutativity and associativity of this binding operation, along with an inverse operation, allowing one to construct symbolicstyle manipulations over real-valued vectors. Most VSAs were developed before deep learning and automatic differentiation became popular and instead focused on efficacy in hand-designed systems. In this work, we introduce the Hadamardderived linear Binding (HLB), which is designed to have favorable computational efficiency, and efficacy in classic VSA tasks, and perform well in differentiable systems. Code is available at https://github.com/FutureComputing4AI/ Hadamard-derived-Linear-Binding .

1 Introduction

Vector Symbolic Architectures (VSAs) are a unique approach to performing symbolic style AI. Such methods use a binding operation B : R d ˆ R d ÝÑ R d , where B p x, y q ' z denotes that two concepts/vectors x and y are connected to each other. In VSA, any arbitrary concept is assigned to vectors in R d (usually randomly). For example, the sentence 'the fat cat and happy dog' would be represented as B p fat , cat q ` B p happy , dog q ' S . One can then ask, 'what was happy' by unbinding the vector for happy, which will return a noisy version of the vector bound to happy. The unbinding operation is denoted B ˚ p x, y q , and so applying B ˚ p S, happy q « dog .

Because VSAs are applied over vectors, they offer an attractive platform for neuro-symbolic methods by having natural symbolic AI-style manipulations via differentiable operations. However, current VSA methods have largely been derived for classical AI tasks or cognitive science-inspired work. Many such VSAs have shown issues in numerical stability, computational complexity, or otherwise lower-than-desired performance in the context of a differentiable system.

As noted in [39], most VSAs can be viewed as a linear operation where B p a, b q ' a J Gb and B ˚ p a, b q ' a J Fb , where G and F are d ˆ d matrices. Hypothetically, these matrices could be learned via gradient descent, but would not necessarily maintain the neuro-symbolic properties of VSAs without additional constraints. Still, the framework is useful as all popular VSAs we are aware fit within this framework. By choosing G and F with specified structure, we can change the computational complexity from O p d 2 q , down to O p d q for a diagonal matrix.

In this work, we derive a new VSA that has multiple desirable properties for both classical VSA tasks, and in deep-learning applications. Our method will have only O p d q complexity for the binding step, is numerically stable, and equals or improves upon previous VSAs on multiple recent deep

learning applications. Our new VSA is derived from the Walsh Hadamard transform, and so we term our method the Hadamard-derived linear Binding (HLB) as it will avoid the O p d log d q normally associated with the Hadamard transform, and has better performance than more expensive VSA alternatives.

Related work to our own will be reviewed in section 2, including our baseline VSAs and their definitions. Our new HLB will be derived in section 3, showing it theoretically desirable properties. section 4 will empirically evaluate HLB in classical VSA benchmark tasks, and in two recent deep learning tasks, showing improved performance in each scenario. We then conclude in section 5.

2 Related Work

Smolensky [38] started the VSA approach with the Tensor Product Representation (TPR), where d dimensional vectors (each representing some concept) were bound by computing an outer product. Showing distributivity ( B p x , y ` z q ' B p x , y q ` B p x , z q ) and associativity, this allowed specifying logical statements/structures [13]. However, for ρ total items to be bound together, it was impractical due to the O p d ρ q complexity. [36, 25, 24] have surveyed many of the VSAs available today, but our work will focus on three specific alternatives, as outlined in Table 1. The Vector-Derived Transformation Binding (VTB) will be a primary comparison because it is one of the most recently developed VSAs, which has shown improvements in what we will call 'classic' tasks, where the VSA's symbolic like properties are used to manually construct a series of binding/unbinding operations that accomplish a desired task. Note, that the VTB is unique in it is non-symmetric ( B p x , y q ‰ B p y , x q ). Ours, and most others, are symmetric.

Next is the Holographic Reduced Representation (HRR) [32], which can be defined via the Fourier transform F p¨q . One derives the inverse operation of the HRR by defining the one vector ⃗ 1 as the identity vector and then solving F p a ˚ q i F p a q i ' 1 . We will use a similar approach to deriving HLB but replacing the Fourier Transform with the Hadamard transform, making the HRR a key baseline. Last, the Multiply Add Permute (MAP) [10] is derived by taking only the diagonal of the tensor product from [38]'s TPR. This results in a surprisingly simple representation of using element-wise multiplication for both binding/unbinding, making it a key baseline. The MAP binding is also notable for its continuous (MAP-C) and binary (MAP-B) forms, which will help elucidate the importance of the difference in our unbinding step compared to the initialization avoiding values near zero. HLB differs in devising for the unbinding step, and we will later show an additional corrective term that HLB employs for ρ different items bound together, that dramatically improve performance.

Our motivation for using the Hadamard Transform comes from its parallels to the Fourier Transform (FT) used to derive the HRR and the HRR's relatively high performance. The Hadamard matrix has a simple recursive structure, making analysis tractable, and its transpose is its own inverse, which simplifies the design of the inverse function B ˚ . Like the FT, WHT can be computed in log-linear time, though in our case, the derivation results in linear complexity as an added benefit. The WHT is already associative and distributive, making less work to obtain the desired properties. Finally, the WHTinvolves only t´ 1 , 1 u values, avoiding numerical instability that can occur with the HRR/FT.

This work shows that these motivations are well founded, as they result in a binding with comparable or improved performance in our testing.

Our interest in VSAs comes from their utility in both classical symbolic tasks and as useful priors in designing deep learning systems. In classic tasks VSAs are popular for designing power-efficient systems from a finite set of operations [14, 23, 17, 30]. HRRs, in particular, have shown biologically plausible models of human cognition [21, 5, 40, 6] and solving cognitive science tasks [8]. In deep learning the TPR has inspired many prior works in natural language processing [34, 16, 35]. To wit, The HRR operation has seen the most use in differentiable systems [43, 41, 42, 27, 29, 33, 1, 2, 28]. To study our method, we select two recent works that make heavy use of the neuro-symbolic capabilities of HRRs. First, an Extreme Multi-Label (XML) task that uses HRRs to represent an output space of tens to hundreds of thousands of classes C in a smaller dimension d ă C [9], and an information privacy task that uses the HRR binding as a kind of 'encrypt/decrypt' mechanism for heuristic security [3]. We will explain these methods in more detail in the experimental section.

3 Methodology

First, we will briefly review the definition of the Hadamard matrix H and its important properties that make it a strong candidate from which to derive a VSA. With these properties established, we will begin by deriving a VSA we term HLB where binding and unbinding are the same operation in the same manner as which the original HRR can be derived [32]. Any VSA must introduce noise when vectors are bound together, and we will derive the form of the noise term as η ˝ . Unsatisfied with the magnitude of this term, we then define a projection step for the Hadamard matrix in a similar spirit to '[9]'s complex-unit magnitude projection to support the HRR and derive an improved operation with a new and smaller noise term η π . This will give us the HLB bind/unbind steps as noted in Table 1.

Hadamard H d is a square matrix of size d ˆ d of orthogonal rows consisting of only ` 1 and ´ 1 s given in Equation 1 where d ' 2 n @ n P N : n ě 0 . Bearing in mind that Hadamard or Walsh-Hadamard Transformation (WHT) can be equivalent to discrete multi-dimensional Fourier Transform (FT) when applied to a d dimensional vector [26], it has additional advantages over Discrete Fourier Transform (DFT). Unlike DFT, which operates on complex C numbers and requires irrational multiplications, WHT only performs calculations on real R numbers with addition and subtraction operators and does not require any irrational multiplication.

Vector symbolic architectures (VSA), for instance, Holographic Reduced Representations (HRR) employs circular convolution to represent compositional structure which is computed using Fast Fourier Transform (FFT) [32]. However, it can be numerically unstable due to irrational multiplications of complex numbers. Prior work [9] devised a projection step to mitigate the numerical instability of the FFT and it's inverse, but we instead ask if re-deriving the binding/unbinding operations may yield favorable results if we use the favorable properties of the Hadamard transform as given in Lemma 3.1.

Lemma 3.1 (Hadamard Properties) . Let H be the Hadamard matrix of size d ˆ d that holds the following properties for x, y P R d . First, H p Hx q ' dx , and second H p x ` y q ' Hx ` Hy .

The bound composition of two vectors into a single vector space is referred to as BINDING. The knowledge retrieval from a bound representation is known as UNBINDING. We define the binding function by replacing the Fourier transform in circular convolution with the Hadamard transform given in Definition 3.1. We will denote the binding function four our specific method by B and the unbinding function by B ˚ .

Definition 3.1 (Binding and Unbinding) . The binding of vectors x, y P R d in Hadamard domain is defined in Equation 2 where d is the elementwise multiplication. The unbinding function is defined in a similar fashion, i.e., B ' B ˚ . In the context of binding, B p x, y q combines the vectors x and y , whereas in the context of unbinding B ˚ p x, y q refers to the retrieval of the vector associated with y from x .

Now, we will discuss the binding of ρ different representations, which will become important later in our analysis but is discussed here for adjacency to the binding definition. Composite representation in vector symbolic architectures is defined by the summation of the bound vectors. We define a parameter ρ P N : ρ ě 1 that denotes the number of vector pairs bundled in a composite representation. Given vectors x i , y i P R d and @ i P N : 1 ď i ď ρ , we can define the composite representation χ as

Next, we require the unbinding operation, which is defined via an inverse function via the following theorem. This will give a symbolic form of our unbinding step that retrieves the original component x being searched for, as well as a necessary noise component η ˝ , which must exist whenever ρ ě 2 items are bound together without expanding the dimension d .

Theorem 3.1 (Inverse Theorem) . Given the identity function Hx ¨ Hx : ' 1 where x : is the inverse of x in Hadamard domain, then B ˚ p B p x 1 , y 1 q ` ¨ ¨ ¨ ` B p x ρ , y ρ q , y : i q ' " x i if ρ ' 1 x i ` η ˝ i else ρ ą 1 where x i , y i P R d and η ˝ i is the noise component.

Proof of Theorem 3.1. We start from the identity function Hx ¨ Hx : ' 1 and thus Hx : ' 1 Hx . Now using Equation 2 we get,

To reduce the noise component and improve the retrieval accuracy, [9, 32] proposes a projection step to the input vectors by normalizing them by the absolute value in the Fourier domain. While such identical normalization is not useful in the Hadamard domain since it will only transform the elements to ` 1 and ´ 1 s, we will define a projection step with only the Hadamard transformation without normalization given in Definition 3.2.

Definition 3.2 (Projection) . The projection function of x is defined by π p x q ' 1 d ¨ Hx .

If we apply the Definition 3.2 to the inputs in Theorem 3.1 then we get

The retrieved value would be projected onto the Hadamard domain as well and to get back the original data we apply the reverse projection. Since the inverse of the Hadamard matrix is the Hadamard matrix itself, in the reverse projection step we just apply the Hadamard transformation again which derives the output to

where η π i is the noise component due to the projection step. In expectation, η π i ă η ˝ i (see Appendix A). Thus, the projection step diminishes the accumulated noise. More interestingly, the retrieved output

term does not contain any Hadamard matrix. Therefore, we can recast the initial binding definition by multiplying the query vector y i to the output of Equation 5 which makes the binding function as the sum of the element-wise product of the vector pairs and the compositional structure a linear time O p n q representation. Thus, the redefinition of the binding function is B 1 p x, y q ' x d y and ρ bundle of the vector pairs is χ 1 ρ ' ř ρ i ' 1 p x i d y i q . Consequently, the unbinding would be a simple element-wise division of the bound representation by the query vector, i.e, B ˚1 p x, y q ' x d 1 y where x and y are the bound and query vector, respectively.

3.1 Initialization of HLB

For the binding and unbinding operations to work, vectors need to have an expected value of zero. However, since we would divide the bound vector with query during unbinding, values close to zero would destabilize the noise component and create numerical instability. Thus, we define a Mixture of N ormal Distribution (MiND) with an expected value of zero but an absolute mean greater than zero given in Equation 6 where U is the Uniform distribution. Considering half of the elements are sampled for a normal distribution of mean ´ µ and the rest of the half with a mean of µ , the resulting vector has a zero mean with an absolute mean of µ . The properties of the vectors sampled from a MiND distribution are given in Properties 3.1.

Properties 3.1 (Initialization Properties) . Let x P R d sampled from Ω p µ, 1 { d q holds the following properties. E r x s ' 0 , E r| x |s ' µ , and } x } 2 ' a µ 2 d

3.2 Similarity Augmentation

In VSAs, it is common to measure the similarity with an extracted embedding ˆ x with some other vector x using the cosine similarity. For our HLB, we devise a correction term when it is known that ρ items have been bound together to extract ˆ x , i.e., B ˚ p χ ρ , z q ' ˆ x . Then if ˆ x is the noisy version of the true bound term x , we want cossim p ˆ x , x q ' 1 , and cossim p ˆ x , y q ' 0 , @ y ‰ x . We achieve this by instead computing cossim p ˆ x , x q ¨ ? ρ , and the derivation of this corrective term is given by Theorem 3.2.

Theorem 3.2 ( ϕ -ρ Relationship) . Given x i , y i ' Ω p µ, 1 { d q @ i P N : 1 ď i ď ρ , the cosine similarity ϕ between the original x i and retrieved vector ˆ x i is approximately equal to the inverse square root of the number of vector pairs in a composite representation ρ given by ϕ « 1 ? ρ .

Proof of Theorem 3.2. We start with the definition of cosine similarity and insert the value of ˆ x i . The step-by-step breakdown is shown in Equation 7.

Employing Properties 3.1 we can derive that } x i } 2 ' ař x i ¨ x i ' a µ 2 d and } x j y j x i } ' a µ 2 d .

Thus, the square of the } x i ` ρ ř j ' 1 , j ‰ i x j y j y i } 2 can be expressed as

Therefore, using Equation 7 and Equation 8 we can write that

The experimental result of the ϕ ´ ρ relationship closely follows the theoretical expectation provided in Appendix C which also indicates that the approximation is valid. Since, we know from Theorem 3.2 that similarity score ϕ drops by the inverse square root of the number of vector pairs in a composite representation ρ , in places where ρ is known or can be estimated from › › χ ρ › › 2 « µ 2 ? ρ ¨ d (proof in Appendix B), it can be used to update the cosine similarity multiplying the scores by ? ρ . Equation 9 shows the updated similarity score where in a positive case p`q , ϕ would be close to 1 { ? ρ and in a negative case p´q , ϕ would be close to zero.

× √

5

10

15

20

25

30

35

40

45

50

No. of bundled pairs

ρ

2

4

6

8

10

n

= log

2

(

d

)

φ

′

(+)

=

φ

→

1

/

√

ρ

ρ

≈

1

5

10

15

20

25

30

35

40

45

50

No. of bundled pairs

ρ

2

4

6

8

10

n

= log

2

(

d

)

φ

′

(

-

)

=

φ

→

0

× √

ρ

≈

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Empirical results of ϕ 1 for varying n and ρ are visualized and verified by a heatmap. In a composite representation χ 1 ρ ' ř ρ i ' 1 p x i d y i q , when unbinding is applied using the query y i , i.e., B ˚1 p χ 1 ρ , y i q ' ˆ x i , a positive case is a similarity between x i and ˆ x i . On the contrary, similarity between ˆ x i and any x j where j P t 1 , 2 , ¨ ¨ ¨ , ρ u and j ‰ i , is a negative case. Mean cosine similarity scores of 100 trials for both positive and negative cases in presented in Figure 1 where the scores for the positive cases are in the red p« 1 q shades and the scores for the negative cases are in the blue p« 0 q shades.

4 Empirical Results

4.1 Classical VSA Tasks

A common VSA task is, given a bundle (addition) of ρ pairs of bound vectors s ' ř ρ i ' 1 B p x i , y i q , given a query x q P s , c can the corresponding vector y q be correctly retrieved from the bundle. To test this, we perform an experiment similar to one in [37]. We first create a pool P of N ' 1000 random vectors, then sample (with replacement) p pairs of vectors for p P t 1 , 2 , ¨ ¨ ¨ , 25 u . The pairs are bound together and added to create a composite representation s . Then, we iterate through all left pairs x q in the composite representation and attempt to retrieve the corresponding y q , @ q P r 1 , p s . A retrieval is considered correct if B ˚ p s , x q q J y q ą B ˚ p s , x q q J y j , @ j ‰ q . The total accuracy score for the bundle is recorded, and the experiment is repeated for 50 trials. Experiments are performed to

compare HRR [32], VTB [12], MAP [10], and our HLB VSAs. For each VSA, at each dimension of the vector, the area under the curve (AUC) of the accuracy vs. the no. of bound terms plot is computed, and the results are shown in Figure 2. In general, HLB has comparable performance to HRR and VTB, and performs better than MAP.

12

2

16

2

20

2

24

2

28

2

32

2

36

2

Dimension

d

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

AUC

(%)

HRR

VTB

MAP

HLB

The scenario we just considered looked at bindings of only two items together, summed of many pairs of bindings. [12] proposed addition evaluations over sequential bindings that we now consider. In the random case we have an initial vector b 0 , and for p rounds, we will modify it by a random vector x t such that b t ` 1 ' B p b t , x t q , after which we unbind each x t to see how well the previous b t is recovered. In the auto binding case, we use a single random vector x for all p rounds.

In this task, we are concerned with the quality of the similarity score in random/auto-binding, as we want B ˚ p b t ` 1 , x t q J b t ' 1 . For VSAs with approximate unbinding procedures, such as HRR, VTB, and MAP-C, the returned value will be 1 if p ' 1 but will decay as p increases. HLBuses an exact unbinding procedure so that the returned value is expected to be 1 @ p . We are also interested in the magnitude of the vectors } B ˚ p b t ` 1 , x t q} 2 , where an ideal VSA has a constant magnitude that does not explode/vanish as p increases.

5

10

15

20

25

0.0

0.2

0.4

0.6

0.8

1.0

Similarity

φ

Random

5

10

15

20

25

0.0

0.2

0.4

0.6

0.8

1.0

Auto

5

10

15

20

25

No. of vectors

p

10

-

3

10

0

Magnitude

5

10

15

20

25

No. of vectors

p

10

3

10

7

HRR

VTB

MAP

HLB

Figure 3 shows that HLB maintains a stable magnitude regardless of the number of bound vectors in both cases. This property arises due to the properties of the distribution shown in Properties 3.1. As all components have an expected absolute value of 1, the product of all components also has an expected absolute value of 1. Thus, the norm of the binding is simply ? d . It also shows HLB maintains the desired similarity score as p increases. Combined with Figure 1 that shows the scores are near-zero when an item is not present, HLB has significant advantages in consistency for designing VSA solutions.

4.2 Deep Learning with Hadamard-derived Linear Binding

Two recent methods that integrate HRR with deep learning are tested to further validate our approach and briefly summarized in the two sub-sections below. In each case, we run all four VSAs and see that HLB either matches or exceeds the performance of other VSAs. In every experiment, the standard method of sampling vectors from each VSA is followed as outlined in Table 1. All the experiments are performed on a single NVIDIA TESLA PH402 GPU with 32 GB memory.

4.2.1 Connectionist Symbolic Pseudo Secrets

When running on low-power computing environments, it is often desirable to offload the computation to a third-party cloud environment to get the answer faster and use fewer local resources. However, this may be problematic if one does not fully trust the available cloud environments. Homomorphic encryption (HE) is the ideal means to alleviate this problem, providing cryptography for computations. HE is currently more expensive to perform than running a neural network itself [11], defeating its own utility in this scenario. Connectionist Symbolic Pseudo Secrets (CSPS) [3] provides a heuristic means of obscuring the nature of the input (content), and output (number of classes/prediction), while also reducing the total local compute required.

CSPS mimics a 'one-time-pad' by taking a random VSA vector s as the secret and binding it to the input x . The value B p s , x q obscures the original x , and the third-party runs the bulk of the network on their platform. A result ˜ y is returned, and a small local network computes the final answer after unbinding with the secret B ˚ p ˜ y , s q . Other than changing the VSA used, we follow the same training, testing, architecture size, and validation procedure of [3].

CSPS experimented with 5 datasets MNIST, SVHN, CIFAR-10 (CR10), CIFAR-100 (CR100), and Mini-ImageNet (MIN). First, we look at the accuracy of each method, which is lower due to the noise of the random vector s added at test time since no secret VSA is ever reused. The results are shown in Table 2, where HLB outperforms all prior methods significantly. Notably, the MAP VSA is second best despite being one of the older VSAs, indicating its similarity to HLB in using a simple binding procedure, and thus simple gradient may be an important factor in this scenario.

However, improved accuracy is not useful in this scenario if more information is leaked. The test in this scenario, as proposed by [3], is to calculate the Adjusted Rand Index (ARI) after attempting to cluster the inputs x and the outputs ˆ y , which are available/visible to the snooping third-party. To be successful, the ARI must be near zero (indicating random label assignment) for both inputs and outputs.

We use K-means, Gaussian Mixture Model (GMM), Birch [45], and HDBSCAN [7] as the clustering algorithms and specify the true number of classes to each method to maximize attacker success (information they would not know). The results can be found in Table 3, where the top rows indicate the clustering of the input B p x , s q , and the bottom rows the clustering of the output ˆ y . All the numbers are percentages p % q , showing all methods do a good job at hiding information from the adversary (except on the MNIST dataset, which is routinely degenerate).

The MNIST result is a good reminder that CSPS security is heuristic, not guaranteed. Nevertheless, we see HLB has consistently close-to-zero scores for SVHN, CIFARs, and Mini-ImageNet, indicating

that its improved accuracy with simultaneously improved security. This also validates the use of the VSA in deep learning architecture design and the efficacy of our approach.

4.2.2 Xtreme Multi-Label Classification

Extreme Multi-label (XML) is the scenario where, given a single input of size d , C ąą d classes are used to predict. This is common in e-commerce applications where new products need to be tagged, and an input on the order of d « 5000 is relatively small compared to C ě 100,000 or more classes. This imposes unique computational constraints due to the output space being larger than the input space and is generally only solvable because the output space is sparse - often less than 100 relevant classes will be positive for any one input. VSAs have been applied to XML by exploiting the low positive class occurrence rate to represent the problem symbolically [9].

While many prior works focus on innovative strategies to cluster/make hierarchies/compress the penultimate layer[19, 20, 31, 18, 44, 22], a neuro-symbolic approach was proposed by [9]. Given K total possible classes, they assigned each class a vector c k to be each class's representation, and the set of all classes a ' ř K k ' 1 c k .

The VSA trick used by [9] was to define an additional 'present' class p and a 'missing' class m . Then, the target output of the network f p¨q is itself a vector composed of two parts added together. First, B p p , ř k c k q represents all present classes, and so the sum is over a finite smaller set. Then the absent classes compute the missing representing B p m , a ´ ř k c k q , which again only needs to compute over the finite set of present classes, yet represents the set of all non-present classes by exploiting the symbolic properties of the VSA.

For XML classification, we have a set of K classes that will be present for a given input, where K « 10 is the norm. Yet, there will be L total possible classes where L ě 100 , 000 is quite common. Forming a normal linear layer to produce L outputs is the majority of computational work and memory use in standard XML models, and thus the target for reduction. A VSA can be used to side-step this cost, as shown by [9], by leveraging the symbolic manipulation of the outputs. First, consider the target label as a vector s P R d such that d ! L . By defining a VSA vector to represent 'present' and 'missing' classes as p and m , where each class is given its own vector c 1 ,...,L , we can shift the computational complexity form O p L q to O p K q by manipulating the 'missing' classes as the compliment of the present classes as shown in Equation 10.

Similarly, the loss to calculate the gradient can be computed based on the network's prediction ˆ s by taking the cosine similarity between each expected class and one cosine similarity for the representation of all missing classes. The excepted response of 1 or 0 for an item being present/absent from the VSA is used to determine if we want the similarity to be 0 (1cos ) or 1 (just cos ), as shown in Equation 11.

The details and network sizes of [9] are followed, except we replace the original VSA with our four candidates. The network is trained on 8 datasets listed in Table 4 from [4] and evaluated using normalized discounted cumulative gain (nDCG) and propensity-scored (PS) based normalized discounted cumulative gain (PSnDCG) as suggested by [19].

The classification result in terms of nDCG and PSnDCG in all the eight datasets is presented in Table 4 where the top four datasets are comparatively easy with maximum no. of features of 5000 and no. of labels of 4000 . The bottom four datasets are comparatively hard with the no. of features and labels on the scale of 100 K . The proposed HLB has attained the best results in all the datasets on both metrics. In contrast to the prior CSPS results, here we see that the performance differences between HRR, VTB, and MAP are more varied, with no clear 'second-place' performer.

5 Conclusion

In this paper, a novel linear vector symbolic architecture named HLB is presented derived from Hadamard transform. Along with an initialization condition named MiND distribution is proposed for which we proved the cosine similarity ϕ is approximately equal to the inverse square root of the no. of bundled vector pairs ρ which matches with the experimental results. The proposed HLB showed superior performance in classical VSA tasks and deep learning compared to other VSAs such as HRR, VTB, and MAP. In learning tasks, HLB is applied to CSPS and XML classification tasks. In both of the tasks, HLB has achieved the best results in terms of respective metrics in all the datasets showing a diverse potential of HLB in Neuro-symbolic AI.

References

Mohammad Mahmudul Alam, Edward Raff, Stella Biderman, Tim Oates, and James Holt. Recasting self-attention with holographic reduced representations. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

Mohammad Mahmudul Alam, Edward Raff, and Tim Oates. Towards generalization in subitizing with neuro-symbolic loss using holographic reduced representations. Neuro-Symbolic Learning and Reasoning in the era of Large Language Models , 2023.

Mohammad Mahmudul Alam, Edward Raff, Tim Oates, and James Holt. Deploying convolutional networks on untrusted platforms using 2D holographic reduced representations. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 367-393. PMLR, 17-23 Jul 2022.

K. Bhatia, K. Dahiya, H. Jain, P. Kar, A. Mittal, Y. Prabhu, and M. Varma. The extreme classification repository: Multi-label datasets and code, 2016.

Peter Blouw and Chris Eliasmith. A neurally plausible encoding of word order information into a semantic vector space. 35th Annual Conference of the Cognitive Science Society , 35:1905-1910, 2013.

Peter Blouw, Eugene Solodkin, Paul Thagard, and Chris Eliasmith. Concepts as semantic pointers: A framework and computational model. Cognitive Science , 40(5):1128-1162, 7 2016.

Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining , pages 160-172. Springer, 2013.

C. Eliasmith, T. C. Stewart, X. Choo, T. Bekolay, T. DeWolf, Y. Tang, and D. Rasmussen. A large-scale model of the functioning brain. Science , 338(6111):1202-1205, 11 2012.

Ashwinkumar Ganesan, Hang Gao, Sunil Gandhi, Edward Raff, Tim Oates, James Holt, and Mark McLean. Learning with holographic reduced representations. Advances in neural information processing systems , 34:25606-25620, 2021.

Ross W. Gayler. Multiplicative binding, representation operators & analogy (workshop poster), 1998.

Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International conference on machine learning , pages 201-210. PMLR, 2016.

Jan Gosmann and Chris Eliasmith. Vector-derived transformation binding: An improved binding operation for deep symbol-like processing in neural networks. Neural Comput. , 31(5):849-869, 2019.

Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv , 2020.

Robert Guirado, Abbas Rahimi, Geethan Karunaratne, Eduard Alarcón, Abu Sebastian, and Sergi Abadal. Whype: A scale-out architecture with wireless over-the-air majority for scalable in-memory hyperdimensional computing. IEEE Journal on Emerging and Selected Topics in Circuits and Systems , 13(1):137-149, 2023.

Mike Heddes, Igor Nunes, Pere VergÃ©s, Denis Kleyko, Danny Abraham, Tony Givargis, Alexandru Nicolau, and Alexander Veidenbaum. Torchhd: An open source python library to support research on hyperdimensional computing and vector symbolic architectures. Journal of Machine Learning Research , 24(255):1-10, 2023.

Qiuyuan Huang, Paul Smolensky, Xiaodong He, Li Deng, and Dapeng Wu. Tensor product generation networks for deep nlp modeling. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1263-1273. Association for Computational Linguistics, jun 2018.

Mohsen Imani, Deqian Kong, Abbas Rahimi, and Tajana Rosing. Voicehd: Hyperdimensional computing for efficient speech recognition. In 2017 IEEE International Conference on Rebooting Computing (ICRC) , pages 1-8. IEEE, nov 2017.

Himanshu Jain, Venkatesh Balasubramanian, Bhanu Chunduri, and Manik Varma. Slice: Scalable linear extreme classifiers trained on 100 million labels for related searches. In Proceedings of the twelfth ACM international conference on web search and data mining , pages 528-536, 2019.

Himanshu Jain, Yashoteja Prabhu, and Manik Varma. Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 935-944, 2016.

Ankit Jalan and Purushottam Kar. Accelerating extreme classification via adaptive feature agglomeration. arXiv preprint arXiv:1905.11769 , 2019.

Michael N. Jones and Douglas J.K. Mewhort. Representing word meaning and order information in a composite holographic lexicon. Psychological Review , 114(1):1-37, 2007.

Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou, et al. Efficient softmax approximation for gpus. In International conference on machine learning , pages 1302-1310. PMLR, 2017.

Denis Kleyko, Mike Davies, Edward Paxon Frady, Pentti Kanerva, Spencer J. Kent, Bruno A. Olshausen, Evgeny Osipov, Jan M. Rabaey, Dmitri A. Rachkovskij, Abbas Rahimi, and Friedrich T. Sommer. Vector symbolic architectures as a computing framework for emerging hardware. Proceedings of the IEEE , 110(10):1538-1571, 2022.

Denis Kleyko, Dmitri Rachkovskij, Evgeny Osipov, and Abbas Rahimi. A survey on hyperdimensional computing aka vector symbolic architectures, part ii: Applications, cognitive models, and challenges. ACM Comput. Surv. , 55(9), jan 2023.

Denis Kleyko, Dmitri A. Rachkovskij, Evgeny Osipov, and Abbas Rahimi. A survey on hyperdimensional computing aka vector symbolic architectures, part i: Models and data transformations. ACM Comput. Surv. , 55(6), dec 2022.

Kunz. On the equivalence between one-dimensional discrete walsh-hadamard and multidimensional discrete fourier transforms. IEEE Transactions on Computers , 100(3):267-268, 1979.

Siyu Liao and Bo Yuan. Circconv: A structured convolution with low complexity. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):4287-4294, Jul. 2019.

Mohammad Mahmudul Alam, Edward Raff, Stella R Biderman, Tim Oates, and James Holt. Holographic global convolutional networks for long-range prediction tasks in malware detection. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Research , pages 4042-4050. PMLR, 02-04 May 2024.

Nicolas Menet, Michael Hersche, Geethan Karunaratne, Luca Benini, Abu Sebastian, and Abbas Rahimi. Mimonets: Multiple-input-multiple-output neural networks exploiting computation in superposition. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 39553-39565. Curran Associates, Inc., 2023.

Peer Neubert, Stefan Schubert, and Peter Protzel. Learning vector symbolic architectures for reactive robot behaviours. In Proc. of Intl. Conf. on Intelligent Robots and Systems (IROS) Workshop on Machine Learning Methods for High-Level Cognitive Capabilities in Robotics , 2016.

Alexandru Niculescu-Mizil and Ehsan Abbasnejad. Label filters for large scale multilabel classification. In Artificial intelligence and statistics , pages 1448-1457. PMLR, 2017.

Tony A Plate. Holographic reduced representations. IEEE Transactions on Neural networks , 6(3):623-641, 1995.

Rebecca Saul, Mohammad Mahmudul Alam, John Hurwitz, Edward Raff, Tim Oates, and James Holt. Lempel-ziv networks. In Proceedings on "I Can't Believe It's Not Better! - Understanding Deep Learning Through Empirical Falsification" at NeurIPS 2022 Workshops , volume 187 of Proceedings of Machine Learning Research , pages 1-11. PMLR, 03 Dec 2023.

Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9355-9366. PMLR, 2021.

Imanol Schlag and Jürgen Schmidhuber. Learning to reason with third order tensor products. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Kenny Schlegel, Peer Neubert, and Peter Protzel. A comparison of vector symbolic architectures. arXiv , 2020.

Kenny Schlegel, Peer Neubert, and Peter Protzel. A comparison of vector symbolic architectures. Artificial Intelligence Review , 55(6):4523-4555, Aug 2022.

Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence , 46(1):159-216, 1990.

Julia Steinberg and Haim Sompolinsky. Associative memory of structured knowledge. Scientific Reports , 12(1), December 2022.

Terrence C. Stewart and Chris Eliasmith. Large-scale synthesis of functional spiking neural circuits. Proceedings of the IEEE , 102(5):881-898, 2014.

Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. Learning to attend via word-aspect associative fusion for aspect-based sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), Apr. 2018.

Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, Nilesh Agrawal, and Partha Talukdar. Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. In Proceedings of the 34th AAAI Conference on Artificial Intelligence , pages 3009-3016. AAAI Press, 2020.

Ryosuke Yamaki, Tadahiro Taniguchi, and Daichi Mochihashi. Holographic CCG parsing. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 262-276, Toronto, Canada, July 2023. Association for Computational Linguistics.

Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. Advances in neural information processing systems , 32, 2019.

Tian Zhang, Raghu Ramakrishnan, and Miron Livny. Birch: an efficient data clustering method for very large databases. ACM sigmod record , 25(2):103-114, 1996.

A Noise Decomposition

When a single vector pair is combined, one of the vector pairs can be exactly retrieved with the help of the other component and the inverse function, recalling the retrieved output does not contain any noise component for a single pair of vectors, i.e., ρ ' 1 . However, when more than one vector pairs are bundled, noise starts to accumulate. In this section, we will uncover the noise components accumulated with and without the projection to the inputs and analyze their impact on expectation. We first start with the noise component without the projection step η ˝ i .

Let, set the value of n to be 1 thus, d ' 2 n ' 2 and the number of vector pairs ρ ' 2 , i.e., χ ρ ' 2 ' B p x 1 , y 1 q ` B p x 2 , y 2 q . We want to retrieve x 1 using the query y 1 , thereby, the expression of η ˝ i is uncovered step by step for ρ ' 2 shown in Equation 13.

Here, φ k @ k P N : 1 ď k ď d are the polynomials comprises of p x 2 , y 2 q , and the query vector y 1 . P is the vector of polynomials consisting of φ k . From the noise expression, we can observe that the numerator is a polynomial and the denominator is the product of all the elements of the Hadamard transformation of the query vector. This is true for any value of n and ρ . Thus, in general, for any query y i we can express η ˝ i as shown in Equation 14.

The noise accumulated after applying the projection to the inputs is quite straightforward as given in Equation 15.

Although the vectors x i , y i @ i P N : 1 ď i ď ρ are sampled from a MiND with an expected value of 0 given in Equation 6, the sample mean of x i or y i would be ˆ µ « 0 but ˆ µ ‰ 0 . Both the numerator of η ˝ i and η π i are the polynomials thus the expected value would be very close to 0 . However, the expected value of the denominator of η ˝ i would be E r ś d k ' 1 p Hy i q k s ' ś d k ' 1 E rp Hy i q k s ' ˆ µ d whereas the expected value of the denominator of η π i is E r y i s ' ˆ µ . Since, ˆ µ d ă ˆ µ , hence, in

expectation η π i ă η ˝ i . This is also verified by an empirical study where n , i.e., the dimension d ' 2 n is varied along with the no. of bound vector pairs ρ and the amount of absolute mean noise in retrieval is estimated.

Figure 4 shows the heatmap visualization of the noise for both η ˝ i and η π i in natural log scale. The amount of noise accumulated without any projection to the inputs is much higher compared to the noise accumulation with the projection. For varying n and ρ , the maximum amount of noise accumulated when projection is applied is 7 . 18 and without any projection, the maximum amount of noise is 19 . 38 . Also, most of the heatmap of η π i remains in the blue region whereas as n and ρ increase, the heatmap of η ˝ i moves towards the red region. Therefore, it is evident that the projection to the inputs diminishes the amount of accumulated noise with the retrieved output.

5

10

15

20

25

30

35

40

45

50

No. of bundled pairs

ρ

2

4

6

8

10

n

= log

2

(

d

)

Noise component without any projection

η

◦

i

5

10

15

20

25

30

35

40

45

50

No. of bundled pairs

ρ

2

4

6

8

10

n

= log

2

(

d

)

Noise component due to the projection

η π

i

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

20

.

0

B Norm Relation

Theorem B.1 ( χ ρ -ρ Relationship) . Given x i , y i ' Ω p µ, 1 { d q P R d @ i P N : 1 ď i ď ρ , the norm of the composite representation χ ρ is proportional to ? ρ and approximately equal to the µ 2 ? ρ ¨ d .

Proof of Theorem B.1. Given χ ρ is the composite representation of the bound vectors, i.e., the summation of ρ no. of individual bound terms. First, let's compute the norm of the single bound term as shown in Equation 16.

Now, let's expand and compute the square norm of the composite representation given in Equation 17.

Thus, given the composite representation and the mean of the MiND distribution, we can estimate the no. of bound terms bundled together by ρ « › › χ ρ › › 2 2 { µ 4 d .

Figure 5 shows the comparison between the theoretical relationship and actual experimental results where the norm of the composite representation is computed for µ ' 0 . 5 and ρ ' t 1 , 2 , ¨ ¨ ¨ , 200 u . The figure indicates that the theoretical relationship aligns with the experimental results. However, as the number of bundled pair increases, the variation in the norm increases. This is because of making the approximation by discarding ξ in Equation 17.

20

40

60

80

100

120

140

160

180

200

No. of bundled pairs

ρ

20

40

60

80

100

120

Norm of composite representation

||

χρ

||

2

Theoretical

||

χρ

||

2

=

µ

2

√

ρ

·

d

Experimental

||

χρ

||

2

≈

µ

2

√

ρ

·

d

C Cosine Relation

Theorem 3.2 shows how the cosine similarity ϕ between the original x i and retrieved vector ˆ x i is approximately equal to the inverse square root of the number of vector pairs in a composite representation ρ . In this section, we will perform an empirical analysis of the theorem and compare it with the theoretical results. For ρ ' t 1 , 2 , ¨ ¨ ¨ , 50 u , similarity score ϕ is calculated for vector dimension d ' 512 . Additionally, the theoretical cosine similarity score is also calculated using the value of ϕ following the theorem. Figure 6 shows the comparison between the two results where the experimental result closely follows the theoretical result. The figure also shows the standard deviation for 100 trials, indicating a minute change from the actual value.

5

10

15

20

25

30

35

40

45

50

No. of bundled pairs

ρ

0.0

0.2

0.4

0.6

0.8

1.0

Cosine Similarity

φ

Theoretical

φ

=

1

√

ρ

Experimental

φ

≈

1

√

ρ

µφ

±

σφ

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper claims to present a new Hadamard-derived linear vector symbolic architecture in the abstract and introduction which accurately reflects the contribution and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are described in the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the theoretical results and proofs are provided in the Methodology section. (See section 3)

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental setup is based on previous papers. All the self-used parameters are discussed in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All the code is provided with the supplemental material. Data is publicly available and instruction is provided on how to get the data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental details and their sources are cited in the Empirical results section. (see section 4)

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Figure 2, Figure 3

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We followed the NeurIPS code of conduct.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The work has no negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No high-risk data or models are used.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Original paper and code are cited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Documentation of the code is provided.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No Crowdsourcing and Research with Human Subjects are used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No participants are used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper introduces a new Vector Symbolic Architecture (VSA) termed Hadamard-derived Linear Binding (HLB), aimed at enhancing computational efficiency and performance in both classical VSA tasks and deep learning applications. VSAs involve binding two vectors to create a new vector in the same space, supporting symbolic operations like binding and unbinding, which is crucial for neuro-symbolic AI.

Traditional VSAs, such as Tensor Product Representation (TPR) and Holographic Reduced Representation (HRR), have limitations like computational complexity or numerical stability issues. In contrast, HLB leverages the Hadamard transform, ensuring linear complexity for binding and addressing these drawbacks.

STRENGTHS:
- The paper is well organized and the concepts are clearly explained.
- The paper details theoretical foundations, derivation processes, and empirical evaluations demonstrating HLB's efficacy across various tasks. It concludes with implications for both classical symbolic AI and modern deep learning systems.

WEAKNESSES:
I do not observe a clear weakness

QUESTIONS:
Since I am not familiar with this VSA domain, I'm eager to understand the problem setting in the experiment sections 4.2.1 and 4.2.2. Could the author elaborate with simple example on these tasks?

LIMITATIONS:
The work has no negative societal impacts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work describes a novel vector-symbolic architecture with linear-complexity binding/unbinding operations which includes the following components:\n- A novel binding/unbinding scheme based on the Walsh-Hadamard transform, which reduces to element-wise multiplication and division due to the self-inverse properties of the Hadamard matrix, the definition of the binding operation, and the proposed projection step.\n- Initialization of hypervector entries following a bimodal Gaussian distribution for purposes of numerical stability.\n- A hypervector projection step which in expectation reduces the amount of noise which is incurred by unbinding a vector from a sum of bindings of pairs of vectors. It also simplifies the final form of the bind/unbind operations.\n- A correction factor that is applied to augment the similarity score when retrieving from a sum of bound vectors, when the number of terms in the sum is known.\n\nThe authors experimentally test the effectiveness of their novel VSA scheme in several tasks including pure synthetic VSA tasks, as well as two tasks in which VSA schemes have previously been combined with deep neural networks.\n\nSTRENGTHS:\n- The paper is written clearly.\n- The experimental results in section 4.2 are convincing.\n- The noise-reducing projection step is an interesting idea. Although the idea is not novel in itself, the definition of the specific projection step is novel as it is well applied in this novel context of Hadamard-derived binding. \n- This VSA is highly efficient, including only element-wise operations.\n- The final HLB bind/unbind operations are trivially simple, but there is a novelty and potentially useful insight in their derivation.\n\nWEAKNESSES:\n1) Perhaps, it is worth mentioning and if possible elaborating/comparing with other parameterizable binding schemes such as \"Steinberg and Sompolinsky, Associative memory of structured knowledge. Sci Rep 12, 21808 (2022)\". This binding scheme is based on a trainable matrix. In fact, this parameterized binding scheme is generic by providing a continuum choice from d (VSA) to d^2 (TPR) for d-dimensional embeddings.\n\n2) Besides the synthetic VSA tasks, out of the studied VSA+deep-learning applications, the XML classification results look very promising and practical (although CSPS looks interesting, it is a pseudo-encryption and hence heuristic). The reviewer was wondering if the proposed HLB binding and unbinding operators could have an edge in other hybrid VSA+deep-learning architectures such as:\n    - [27] where MIMOConv used HRR-based binding and MBAT-based unbinding, and MIMOFormer used MAP-based binding and unbinding\n    - Another neuro-symbolic capability of HRRs was studied in \"M. Hersche, et al. A neuro-vector-symbolic architecture for solving Raven’s progressive matrices. Nat Mach Intell 5, 363–375 (2023)\". Particularly, fractional power encoding was used to describe a set of arithmetic rules with binding and unbinding for abductive reasoning.\n    - Another closely related work to XML classification in which the number of classes C is larger than dimension (d << C) is \"M. Hersche, et al. Factorizers for Distributed Sparse Block Codes. Neurosymbolic Artificial Intelligence, 2024\". Would be interesting to see how HLB can handle sparse (block-wise) codes.\n\n3) The other key issue with the paper is flaws in its presentation. Some of them are:\n    - The sentences starting at lines 39 and 40 are fully out of place.\n    - Line 107 mentions an “unburdening” operation, clearly referring to “unbinding”.\n    - Line 197 \"HLBuses\" needs spacing\n    - In Theorem 3.1, the \\cdot multiplication notation was used in place of the \\odot for element-wise multiplication used otherwise in the paper.\n     - Properties 3.1 is formulated in a weird way: “Let x sampled from omega holds the following properties…”\n     - In properties 3.1, it seems that an expectation operator is missing in the final property concerning the L2-norm of x sampled from MiND.\n\n4) The blue horizontal strip at the top of Figure 4 is a weird anomaly which was not addressed by the authors.\n\nQUESTIONS:\n- It would be great to elaborate on other trainable biding schemes discussed in (1) weaknesses \n- If time and compute resources allow, it would be interesting to have more results based on the VSA neuro-symbolic architectures pointed out in (2) weaknesses, especially since the first two works have code repositories publically available. \n- What is the reason behind the blue horizontal strip at the top of Figure 4?\n- I did not fully understand what the hyperparameter search procedure for experiments in Section 4.2 was. Did the authors re-use the hyperparameters from the experiments in the referenced papers?\n- In which scenarios is the discussion in section 3.2 relevant? The proposed factor only scales the similarity values, it does not modify the rankings of similarity scores, so it is not entirely clear to me in which context this matters.\n\nLIMITATIONS:\nThis work has no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

comments: SUMMARY:
The authors propose a new form of Vector Symbolic Architecture (VSA), which leverages the Walsh Hadamard transform for vector binding. The new binding is named Hadamard-derived linear binding (HLB), and it achieves comparable or better performance than existing VSAs when performing classic VSA tasks and combined with deep learning.

STRENGTHS:
The new binding method is developed with solid mathematical support. With the Hadamard-derived linear binding, the complexity of binding remains at O(d) while also providing some intriguing properties like the stability of similarity scores under the binding of multiple items. In the experiments, the new VSA is able to achieve better or comparable performance than existing VSAs such as MAP and HRR.

WEAKNESSES:
1. The motivation behind using WHT to derive the binding operation is missing.
2. Section 3 (the part before subsection 3.1) is rather hard to follow due to possible typos and poor organization.
3. It is unclear why the proposed method improves the performance in deep learning related tasks (in section 4.2).

QUESTIONS:
1. Is there any connection between the performance improvements in section 4.2 and the numerical stability of HLB?
2. Is there any benefit of having a symmetric binding?
3. In Definition 3.1, what does B=B* mean? Could you provide a more detailed proof of Theorem 3.1?
4. What are the differences between 'the proof of theorem 3.1' and equation (5)?

LIMITATIONS:
The limitations are not provided in the paper. The main limitation is that the merit of the new VSA is unclear in practice as the baselines only involve VSA.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed HLB, a vector symbolic architecture derived from the Hadamard transform, reminiscent of holographic reduced representation, to mitigate the challenges that classical VSAs face in deep learning tasks such as numerical stability. Results show comparable memorization capability from alternative models and improved performance on two practical tasks, CSPS and XML.\n\nSTRENGTHS:\n1.\tThe paper is well-presented and delineates the design principles of HLB in the context of other VSA methods, making the position of the HLB clear. \n2.\tHLB is original in the sense that WH transform has not been leveraged directly to derive a VSA before (to the best of my knowledge); although the main design change from HRR is that the Fourier transform is replaced with the Hadamard transform, subsequent derivations of HDC properties and additional technique (the correction term, and the choice of sampling distribution of the hypervectors) is novel. \n3.\tThe paper presents reasonably rigorous (see weakness) proof for its claim and supports it with empirical results. Experiments cover fundamental analytical properties desired from a VSA model as well as some applications of VSA in DL. \n4.\tThe paper is significant in the field of vector symbolic architecture as a novel VSA model that may address some of the existing challenges in VSA, particularly the numerical stability problem.\n\nWEAKNESSES:\n1.\tSeveral typos in the draft (line 87 it’s, 95 four, 150 dervise, 197 HLBuses, 260)\n2.\tAlthough the key parameter selection is explained in the experimental section, the authors defer the main experimental details to the referenced work. The authors are encouraged to briefly summarize each experiment and how HRR is used. \n3.\tSee questions for additional concerns.\n\nQUESTIONS:\n1.\tIs there a way to compute or estimate $\\rho$ (number of bound items) given the vector? Many VSA can estimate via its norm and subsequent processing does not need to actively track the number of bound items in practice.  \n2.\tHow does HLB compare to other VSAs in computational complexity? \n3.\tIs MAP or MAP-C used in the experiment? They seems to be used interchangeably, but the C (which I assume to stand for “continuous” base vectors as opposed to bipolar) in MAP-C was never explained. \n4.\tAlthough the VSAs in comparisons are selected well, all three approaches leverage approximate unbinding while HLB leverages exact unbinding. How would HLB compare with alternative approaches in general, especially MAP (with bipolar vector initialization) and FHRR (HRR in the Fourier domain)? Both support exact unbinding (due to vectors having exact inverses).\n\nLIMITATIONS:\nNo limitation section is present.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}