Soft Tensor Product Representations for Fully Continuous, Compositional Visual Representations

Authors: bethia  sun
maurice  pagnucco
yang  song
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:20:00

Abstract: Since the inception of the classicalist vs. connectionist debate, it has been argued that the ability to systematically combine  symbol-like entities into compositional representations is crucial for human intelligence. In connectionist systems, the field of disentanglement has emerged to address this need by producing representations with explicitly separated factors of variation (FoV). By treating the overall representation as a *string-like concatenation* of the inferred FoVs, however, disentanglement provides a fundamentally *symbolic* treatment of compositional structure, one inherently at odds with the underlying *continuity* of deep learning vector spaces. We hypothesise that this symbolic-continuous mismatch produces broadly suboptimal performance in deep learning models that learn or use such representations. To fully align compositional representations with continuous vector spaces, we extend Smolensky's Tensor Product Representation (TPR) and propose a new type of inherently *continuous* compositional representation, *Soft TPR*, along with a theoretically-principled architecture, *Soft TPR Autoencoder*, designed specifically for learning Soft TPRs. In the visual representation learning domain, our Soft TPR confers broad benefits over symbolic compositional representations: state-of-the-art disentanglement and improved representation learner convergence, along with enhanced sample efficiency and superior low-sample regime performance for downstream models, empirically affirming the value of our inherently *continuous* compositional representation learning framework.

Main paper: 
 introduction: 
 Compositional structure, capturing the property of being decomposable into a set of constituent parts, is ubiquitous in our surroundings -from the recursive application of syntax in language, to the parsing of richly complex visual scenes into their constituent parts. Given the central role such structure plays in our understanding of the world, it is highly intuitive that deep learning representations also embody compositional structure. Indeed, empirical evidence highlights the usefulness of explicitly compositional representations, showcasing a multitude of benefits, including increased interpretability [10, 12] , reduced sample complexity [30, 33] , increased fairness [20, 25, 41] , and improved performance in out-of-distribution generalisation [33, 48, 50] .We consider the following, intuitive notion of compositional representations. A representation of compositionally-structured data is a compositional representation if it has a structure that faithfully reflects the compositional structure of the represented data [49] . In the visual representation learning domain, data is clearly compositionally-structured, as images can be decomposed into a set of constituent factors of variation (FoVs), e.g., {magenta floor,orange wall, aqua object colour, oblong object shape} for the image in Figure 1 .A widely explored representation learning framework producing explicitly compositional representations is that of disentanglement. We adopt the conventional [5, 25, 33, 43] , intuitive definition of a disentangled representation, which states that a representation, ψ(x), is disentangled if each of the underlying FoVs can be cleanly separated into a distinct dimension (or contiguous subset of dimensions) of ψ(x), or, in other words, if each FoV has a 1-1 correspondence with a distinct part of the representation [43] . Framed in this way, it is apparent that disentangled representations are explicitly compositional by nature. The majority of state-of-the-art disentanglement approaches use a variational autoencoder backbone, and rely on weak supervision [13, 22, 31, 33, 35] , or a penalisation of the aggregate posterior q(z|x)p(x)dx [10, 14, 17, 24, 26, 32] to promote disentanglement. More recent approaches depart from the restrictive assumptions of a variational framework, and instead use standard autoencoding [47] , or energy-function based optimisation [36] , with additional inductive biases to encourage disentanglement. Despite the diversity of methods characterising existing work, we make a crucial observation which unifies them together: by enforcing the 1-1 correspondence between FoVs and distinct parts of the representation, existing approaches [10, 13, 14, 17, 22, 24, 26, 31, 32, 33, 35, 36, 47] essentially produce compositional representations corresponding to a concatenation of scalar-valued or vector-valued FoV tokens, as illustrated in Figure 1a . This concatenative approach enforces a rigid, slot-based representational structure that constraints how information can be represented and combined, and mirrors symbolic systems, where distinct symbols occupy discrete slots within a representation. We argue that this fundamentally symbolic approach creates a deep incompatibility with the inherent continuity of the vector spaces underlying deep learning for the following reasons:1. Misalignment with Gradient-Based Learning: The symbolic approach's introduction of discrete representational slots for each FoV introduces non-differentiable boundaries between FoV slots, challenging the efficacy of gradient-based learning, which thrives on smooth and continuous transformations. For example, when modifying a given FoV, this symbolic structure restricts gradient propagation to the dimensions associated solely to the corresponding representational slot, inhibiting the smooth permeation of gradients across the entire vector space. Managing these discrete, slot-based boundaries thus fragments gradient flow across the vector space, producing abrupt, discontinuous transitions that may potentially complicate learning. 2. Restrictive / Incompatible Structure: By allocating distinct representational slots for each FoV, the symbolic approach imposes a rigid representational structure that prevents the representation from exploiting the expressivity inherent in continuous vector spaces. More concretely, this slot-based framework prevents the encoding of FoVs as flexible combinations of basis vectors that span the entire representational space -an approach that is not only more intuitive, but also critical for capturing rich interactions and complex dependencies among FoVs. By failing to allow for this flexibility, the symbolic approach prevents the representation from leveraging the full expressivity of its underlying vector space.Critically, we hypothesise that the fundamental incompatibility between the symbolic treatment of compositional structure provided by disentanglement, and the continuous vector spaces of deep learning produces suboptimal behaviour in the models that learn or use these representations. This hypothesis prompts the following question: can we instead represent compositional structure in an inherently continuous manner? A continuous compositional representation would yield the representation, ψ(x), by continuously combining the FoVs, rather than maintaining a discrete, slotbased separation, as in the symbolic approach. The continuous approach to representing compositional structure is thus, a more mathematically intuitive framework in the context of deep learning.Pioneered by Smolensky, the Tensor Product Representation [3] is a specific representational form that encodes compositional structure in an inherently continuous manner. At the crux of it, TPRs are formed by continuously blending the FoVs together into the overall representation, in a manner analogous to superimposing multiple waves together to produce a complex waveform, as illustrated in Figure 1b . For a representation to qualify as a TPR, it must adhere to a highly specific mathematical form, which confers upon the TPR valuable theoretical properties (elaborated on in Section 3.2), but also imposes two major limitations (see B.1 for further details). First, as depicted by the stars in Figure 1c , only a discrete subset of points in the underlying representational space, Ṽ , satisfies the stringent mathematical criteria to qualify as TPRs. Consequently, to learn TPRs, representation learners must map from the data manifold onto this discrete subset, which constitutes a highly constrained and inherently challenging learning task. Second, the TPR specification enforces a strict, algebraic definition of compositional structure, limiting the TPR's ability to faithfully represent real-world data which is often quasi-compositional, only approximately adhering to a rigid, formal definition of compositionality. Historically, these limitations have confined TPR learning to formal domains characterised by explicit, algebraic structure -as evidenced by the near exclusive deployment of TPRs in language [19, 23, 28, 34, 52] -and, to contexts where strong supervision from highly structured downstream tasks is available to steer the representation learning process [23, 28, 38, 51] . To negate these drawbacks and extend continuous compositional representations to weakly supervised, non-formal domains, we propose Soft TPR, a new, inherently continuous compositional representation that can be thought of as a continuous relaxation of the traditional TPR, as illlustrated by the translucent circular regions in Figure 1c . At its core, the Soft TPR is designed to promote representational flexibility and ease of learning while simultaneously preserving the structural and mathematical integrity of the traditional TPR. We additionally introduce Soft TPR Autoencoder, a theoretically-principled weakly-supervised architecture for learning Soft TPRs, which we use to operationalise the Soft TPR framework in the visual representation learning domain.Our main contributions are threefold: i) We propose a novel compositional representation learning framework, introducing the inherently continuous Soft TPR compositional form, alongside a dedicated, weakly-supervised architecture, Soft TPR Autoencoder, for learning this form. ii) Our framework is the first to learn continuous compositional representations in the non-formal domain of vision. iii) We empirically affirm the far-reaching benefits of enhanced vector space alignment produced by the Soft TPR framework, demonstrating that Soft TPRs achieve state-of-the-art disentanglement, accelerate representation learner convergence, and provide downstream models with enhanced sample efficiency and superior low-sample regime performance. 
related work: 
 Disentanglement: In aiming to produce explicitly compositional representations without strong supervision, our work shares the same objective as disentangled representation learning. Prior to the highly influential work of [25] , which proved the impossibility of learning disentangled representations without supervision or other inductive biases, disentangled representations were learnt in a completely unsupervised fashion [8, 10, 14, 17, 24, 26, 37] . Our use of weak supervision is inspired by the work [13, 22, 31, 33, 35] relating to this highly influential impossibility result.In particular, we leverage the type of weak supervision termed 'match pairing' [35] , where pairs, (x, x ′ ), differing in values for a subset of known FoVs are presented to the model, to incentivise disentanglement. Our work, however, fundamentally diverges from all disentanglement work we are aware of, by adopting an inherently continuous representation of compositional structure, which contrasts with the symbolic representations of compositional structure characterising existing work.TPR-based Work: Existing TPR-based approaches generate continuous representations of compositional structure by producing an element with the explicit mathematical form of a TPR. To learn this highly specific form, these approaches rely on the algebraic characterisation of compositionality present in formal domains, such as mathematics [38] , or language [19, 23, 28, 34, 52] in addition to strong supervision signals from highly structured downstream tasks, such as part-of-speech tagging [23] , and answering structured language [28, 51] or mathematics questions [38] . In contrast, our Soft TPR eases these stringent constraints by offering a relaxed specification of inherently continuous compositional structure. This allows our approach to extend continuous representations of compositional structure to an orthogonal and less structured domain, that of visual representation learning, while also reducing reliance on annotated data by instead using weak supervision to learn this relaxed representational form. 
preliminaries 3.1 a formal framework for compositional representations: 
 We adopt a generalised, non-generative version of the definition of compositional representations from [49] . Data x ∈ X is compositionally-structured if there exists a decomposition function β : X → A 1 × . . . × A n decomposing x into constituent parts, i.e. β(x) = {a 1 , . . . , a n }, where a i ∈ A i .A map ψ : X → V F produces a compositional representation if ψ(x) = C(ψ 1 (a 1 ), . . . , ψ n (a n )) where ψ i : A i → V i denote component functions that independently embed the parts of x into vector spaces, and C : V 1 × . . . × V n → V F denotes a composition function that combines the embedded parts of x together to form the overall representation. Intuitively, this definition enforces a faithful structural correspondence between the constituency structure of the data, x (i.e., the parts, {a 1 , . . . , a n }) and the constituency structure of the representation, ψ(x) (i.e., the embedded parts, {ψ 1 (a 1 ), . . . , ψ n (a n )}) (assuming C is invertible).We formalise a symbolic compositional representation, ψ s (x), as a compositional representation where C is a concatenation operation. Thus, ψ s (x) = ψ 1 (a 1 ) T , . . . , ψ n (a n ) T T for any symbolic compositional representation, ψ s (x). Clearly, the aforementioned disentanglement methods [8, 10, 13, 14, 17, 22, 24, 26, 31, 32, 33, 35, 36, 37, 47] all fit this framework. While we observe that the fundamentally symbolic definition of C as concatenation produces an inherent misalignment with the continuous vector spaces of deep learning, it has one notable benefit: the embedded FoVs, {ψ i (a i )}, can be easily recovered from the representation, ψ s (x), by simply partitioning ψ s (x).It is highly intuitive that for any compositional representation, ψ(x), to be broadly useful, the FoVs, {a i }, should be easily recoverable from ψ(x) (here we assume the ψ i 's are invertible, and so, that this property corresponds to being able to recover the representational components, {ψ i (a i )} from ψ(x)).We thus explore whether an alternative C exists that simultaneously 1) combines the embedded FoVs into the overall representation in an inherently continuous manner and 2) preserves the direct recoverability of the embedded FoVs.TPR [3] is a specific type of representation that is compositional, continuous, and under certain conditions, ensures the direct recoverability of the embedded parts {ψ i (a i )} from the overall representation. We briefly introduce essential aspects of the framework, deferring further details and formal proofs to Appendix A. The TPR framework views compositionally-structured objects as possessing a number of (potentially infinite) roles 1 , where each role is bound to a corresponding filler. It thus defines the constituent parts {a i } of any compositionally-structured object as a set of role-filler bindings. This role-filler binding formalism has predominantly been applied in the natural language domain [19, 28, 34, 52] , with fillers often corresponding to words and roles to grammatical categories (e.g., the word cat as a filler, and the the category noun as a role). We translate this formalism into the domain of visual representation learning by informally equating roles as FoV types, and fillers as FoV values, e.g., {floor colour, wall colour, object colour, object size, object shape, orientation} and {blue, magenta, orange, green, small, medium, large, oblong, cube, . . .} respectively for the Shapes3D domain of Figure 1 . The binding of a filler, f , from a set F of N F fillers, to a role, r, from a set R of N R roles, such as the filler magenta to the role object colour conveys a sort of filler-specific, role-modulated semantic content, and is denoted by f /r. The compositional structure of the image, x, in Figure 1 would thus correspond to the following set of role-filler bindings: β(x)= {magenta/floor colour, orange/wall colour, aqua/object colour, large/object size, oblong/object shape}.To produce the TPR, the roles and fillers for each binding in x are independently embedded using role and filler embedding functions, ξ R : R → V R , ξ F : F → V F respectively. To produce an embedding of the binding f /r, a tensor product, which we denote by ⊗, is then taken over the embedded role, ξ R (r), and embedded filler, ξ F (f ), comprising the binding. Finally, a summation is performed over all embedded bindings in x to produce the overall representation. More concretely, the TPR, ψ tpr (x), is defined as:EQUATIONwhere m : {1, . . . , N R } → {1, . . . , N F } is a matching function associating each of the roles to the unique filler it binds to in the decomposition of x2 .We now place the TPR in the formal framework of Section 3.1, by observing that the TPR defines the component functions asψ i (f m(i) , r i ) := ξ F (f m(i) ) ⊗ ξ R (r i ), and the composition function, C, as ordinary vector space addition. As the TPR does not concatenate, but rather, additively superimposes all representational components {ψ i (f m(i) , r i )} together to produce the overall representation, it instantiates an inherently continuous representation of compositional structure. Defining C as ordinary addition, however, prompts the question of whether the summed up representational components {ψ i (f m(i) , r i )} can be recovered from the representation, ψ tpr (x). Remarkably, due to the special form of the TPR, each representational component ψ i (f m(i) , r i ) corresponding to an embedded role-filler binding, ξ F (f m(i) ) ⊗ ξ R (r i ) can be faithfully recovered from the TPR through a process referred to as unbinding (see A.2 for more details). More concretely, provided that all role embedding vectors {ξ R (r i )} are linearly independent, the embedded filler bound to the i-th role can be unbound from the representation, ψ tpr (x), by taking a (tensor) inner product between ψ tpr (x) and the i-th unbinding vector, u i :EQUATIONwhere u i is a vector corresponding to the i-th column of U T , the (left) inverse of the matrix formed by taking all (linearly independent) role embeddings as columns. By repeating the unbinding procedure using each of the N R unbinding vectors, the embedded fillers bound to each role, and hence, the embeddings (ξ F (f m(i) ), ξ R (r i )) comprising each binding embedding, ξ F (f m(i) ) ⊗ ξ R (r i ), can be recovered from the overall representation, ψ tpr (x). Thus, provided that the linear independence condition is satisfied, the TPR represents a continuous compositional representation that retains the key benefit of symbolic compositional representations: the direct recoverability of the representational parts, {ψ i (f m(i) , r i )}.Building explicit TPRs requires that the embedded fillers and roles comprising a binding (i.e. (ξ F (f m(i) ), ξ R (r i ))) are first tokened before the entire compositional representation, ψ tpr (x) can be produced [19, 23, 28, 34, 38, 51, 52] . This sort of sequential approach, where constituents must be tokened before the compositional representation can be formed, is a key characteristic of the symbolic representation of compositional structure [45] . In contrast, the Soft TPR allows the Encoder to produce any arbitrary element of V F ⊗ V R (in this case ∼ = R 6 ), provided that the sufficient closeness requirement holds. Thus, once the Encoder is trained, it is theoretically possible for the Soft TPR Autoencoder to exploit vector space continuity to generate approximately compositional representations by mapping directly from the data to a Soft TPR, without needing to token any representational constituents. 
soft tpr: an extension to the tpr framework: 
 The TPR's highly specific representational form, ψ tpr (x) := i ξ F (f m(i) ) ⊗ ξ R (r i ), can only be satisfied by a discrete subset of points in the underlying representational space, V F ⊗ V R . This imposes an arduous learning task on representation learners: to parameterise the highly constrained map from the data manifold to a discrete subset of points. This representational form additionally assumes a strict algebraic definition of compositionality that corresponds to a set of bindings, where each binding comprises a single role and a single filler, precluding the TPR from representing quasicompositional objects that only approximately satisfy this strict, algebraic definition of compositional structure (e.g., French liaison consonants, where a weighted sum of multiple fillers, rather than a single filler, bind to a role [9] ). Our primary insight is that both these drawbacks can be mitigated by continuously relaxing the TPR specification (see B.1). This relaxation allows for a wider variety of mappings within a 'cloud' around each TPR (represented by the translucent circular regions in Figure 1c ), which eases the difficulty of representation learning. Furthermore, it relaxes the rigid role-filler based specification of compositional structure into a softer, less rigid notion, enabling the representation of nuanced, quasi-compositional data. We thus introduce the Soft TPR, a continuously relaxed, less stringently defined variant of the explicit TPR that simultaneously retains the TPR's key properties of 1) continuous compositional structure, and 2) direct recoverability of representational parts { ξ F (f m(i) ), ξ R (r i ) } from the overall representation.Consider an element z in V F ⊗ V R , the vector space underlying TPRs produced by an arbitrary role embedding function ξ R : R → V R and an arbitrary filler embedding function ξ F : F → V F . If z is sufficiently 'close' to some TPR, ψ tpr3 , quantified by ||z -ψ tpr || F < ϵ, where ||A|| F denotes the Frobenius norm of the rank-2 tensor, A, and ϵ is some small, scalar valued quantity, then this distance metric induces the approximation z ≈ ψ tpr . By performing unbinding of the i-th filler to both sides of the approximation, we have:EQUATIONEQUATIONThus, performing unbinding on an element z in V F ⊗ V R where the sufficient closeness condition holds recovers a soft filler embedding, fi , that approximates the true filler embedding, ξ F (f m(i) ), of the filler bound to role r i for ψ tpr with approximation error ϵ i . We define such elements as Soft TPRs, noting that these elements both 1) softly approximate the continuous compositional structure captured by some explicit TPR, ψ tpr , as z is sufficiently 'close' to ψ tpr based on the chosen distance metric, and 2) approximately preserve the recoverability of the representational components{(ξ F (f m(i) ), ξ R (r i ))} of the explicit TPR, ψ tpr , they approximate, with the only difference being that soft filler embeddings, fi are returned in place of the actual filler embeddings, ξ F (f m(i) ). Defining Soft TPRs in this way 1) provides a less restrictive representational specification that can be satisfied by any arbitrary element from V F ⊗ V R provided that, for some explicit TPR, ψ tpr , the sufficient closeness requirement, ||z -ψ tpr || F < ϵ, holds, and 2) allows learned representations to embody a more flexible, relaxed notion of compositional structure.We define our vector spaces of interest over the reals as V F := R D F and V R := R D R where D F , D R denote the dimensionality of the filler and role embedding spaces. The main insight underlying our method is that, as the Soft TPR is effectively any arbitrary element from a vector space4 R D F •D R that is sufficiently close to some explicit TPR, any (D F • D R )-dimensional vector produced by an encoder in a standard autoencoding framework can be treated as a Soft TPR candidate. This suggests that a simple autoencoding framework only needs to be slightly modified to produce Soft TPRs. Briefly speaking, our Soft TPR Autoencoder contains a standard encoder, E, the TPR decoder, and a standard decoder, D, where the encoder output, z, corresponds to the Soft TPR. At a high level, our framework aims to ensure two properties: 1) representational form, and 2) representational content. Representational form requires that the encoder output, z, has the desired Soft TPR form (i.e. that ||z -ψ tpr || F < ϵ for some TPR, ψ tpr ). However, having a Soft TPR form alone is insufficient; the representation produced by the autoencoder must also reflect the true role-filler content of the data to be a good representation, as required by the aim of representational content. These 2 properties are (mostly) respectively achieved using the unsupervised and weakly supervised components of our method.Representational Form: to encourage the autoencoder to produce elements that are Soft TPRs, we penalise the Euclidean distance ||z -ψ * tpr || 2 between the encoder output, z, and the explicit TPR, ψ * tpr , that z best approximates. To obtain ψ * tpr , needed to penalise the above distance, we derive an explicit analytical form for ψ * tpr and construct elements satisfying this analytical form using a role embedding matrix, M ξ R containing N R D R -dimensional role embedding vectors, {ξ R (r i )}, and a filler embedding matrix, M ξ F containing N F D F -dimensional filler embedding vectors, {ξ F (f i )}.To define an explicit analytical expression for ψ * tpr , we are guided by the intuition that the TPR that z best approximates should have an explicit dependency on the soft filler bindings of z, and so, elect to use the following, greedily optimal definition:ψ * tpr := i ξ F (f m(i) ) ⊗ ξ R (r i ), where m(i) := arg min j || fk -ξ F (f j )|| 2, and fk := zu i . ( 5)That is, we define ψ * tpr as the TPR constructed from explicit filler embeddings ξ F (f j ) with the smallest Euclidean distance to the soft filler embeddings fk of z. To construct elements satisfying (5), we use a 3-step process carried out by the novel TPR decoder we introduce, visible in Figure 2 : 1) Unbinding: The unbinding module consists of a fixed semi-orthogonal role embedding matrix M ξ R and recovers the soft filler embeddings { fi } associated with each encoder output, z, by performing the TPR unbinding operation. We elaborate on our theoretically-informed reason for this choice of role embedding matrix, how the unbinding vectors are obtained, and for not backpropagating gradient to M ξ R in B.3.1. 2) Quantisation: The quantisation module, containing a learnable filler embedding matrix M ξ F , employs the VQ-VAE vector quantisation algorithm [11] to both 1) learn the explicit filler embeddings, and 2) quantise the soft filler embeddings { f1 , . . . , fN R } produced by the unbinding module into the explicit filler embeddings {ξ F (f 1 ), . . . , ξ F (f N R )} with the smallest Euclidean distances. 3) TPR Construction: The TPR construction module recovers ψ * tpr by simply performing the TPR operation i ξ F (f m(i) )⊗ξ R (r i ) over the fixed, semi-orthogonal role embedding vectors with the corresponding explicit filler embeddings produced by the quantisation module. To ensure the quantised filler embeddings {ξ F (f m(i) )} depend explicitly on the reconstructed image, we pass the output of the TPR decoder, ψ * tpr to the image decoder, D, for reconstruction. The overall unsupervised loss, L u , is thus given by the following, where s denotes the stop-gradient operator, β a hyperparameter, and L r any suitable image-based reconstruction loss (we use L 2 ):L u := ||z -ψ * tpr || 2 2Soft TPR penaltyEQUATIONRepresentational Content: While the unsupervised loss L u encourages the autoencoder to produce encodings z with the desired Soft TPR form, it may not ensure that the content of z accurately reflects the true role-filler semantics of the represented data. To address this, we introduce a weakly supervised loss to ensure that the explicit TPR, z * tpr , which z best approximates, reflects the groundtruth semantics of the image. We employ a match-pairing context similar to [13, 22, 33, 35] , where image pairs (x, x ′ ) share the same role-filler bindings for all but one of the roles, r i , and the identity of r i is known, but not any of the fillers, or role-filler bindings. Our intuition is that, for the role-filler binding embeddings of z to reflect the semantics of the represented image, the Euclidean distance between the quantised fillers of x and x ′ bound to role r i should be maximal, relative to the distances between the pairs of filler embeddings for all other roles r j , j ̸ = i. To encourage this, we apply the cross entropy loss corresponding to the 3rd term in Eq 7, where ∆q denotes the N R -dimensional vector with each dimension (∆q) k populated by the Euclidean distance between the quantised fillers of x and x ′ for role r k , and l denotes the one-hot vector of dimension N R with the index for r i set to 1. Additionally, we apply a reconstruction loss using the TPRs, ψ s tpr (x) and ψ s tpr (x ′ ), which are constructed by swapping the quantised filler embeddings of x and x ′ bound to role r i , to reconstruct x ′ and x respectively.Our final loss, L, is a weighted sum over the unsupervised and weakly supervised loss components, where λ 1 and λ 2 are hyperparameters:EQUATION 
results: 
 To assess the compositional representations produced by our Soft TPR framework, we perform evaluation along three dimensions: 1) Compositional Structure / Disentanglement: What is the degree to which Soft TPR representations achieve explicitly compositional structure? 2) Representation Learner Convergence Rate: Can representation learners learn the inherently continuous compositional structure embodied in the Soft TPR faster than symbolic alternatives? 3) Downstream Models: Does the enhanced vector space alignment produced by the Soft TPR facilitate benefits for downstream models using compositional representations?We benchmark against a suite of weakly supervised disentanglement baselines: Ada-GVAE [33] , GVAE [22] , ML-VAE [13] , SlowVAE [39] , and the GAN-based model of [35] , which we henceforth refer to as 'Shu'. These models all produce symbolic compositional representations corresponding to a concatenation of scalar-valued FoV tokens. Like our model, Ada-GVAE, GVAE, ML-VAE, and Shu are trained with paired samples (x, x ′ ) sharing values for all but a subset of FoVs types (roles), I. ML-VAE, GVAE, and Shu assume access to I, the FoV types (roles) that differ between x and x ′ , matching our model's level of supervision, while Ada-GVAE does not. We thus modify Ada-GVAE (method detailed in Appendix C.2.2) for more direct comparability, denoting our modification by Ada-GVAE-k. In contrast, SlowVAE is trained with pairs of samples where all underlying FoV values change, and also assumes that this change can be characterised as a sample from a Laplacian. We additionally benchmark against 2 baselines producing vector-tokened compositional representations: COMET [36] , and Visual Concept Tokeniser (VCT) [47] . These vector-tokened models cannot be directly compared to our model as they are fully unsupervised, however, we include them for completeness. We train 5 instances of each representation learning model using 5 random seeds for 200,000 iterations across all datasets, and report results averaged over the 5 random runs.To evaluate the degree to which Soft TPRs achieve explicitly compositional structure, we quantify representational disentanglement using standard disentanglement datasets Cars3D [4] , MPI3D [21] , and Shapes3D [24] (see C.4.1 for further details). As can be seen in Table 1 , our model achieves state-of-the-art disentanglement for all datasets, with notable DCI metric increases of 29% and 74% on the 2 more challenging datasets of Cars3D and MPI3D respectively. To rule out the possibility that the improvement in disentanglement produced by our model is due to a slight increase of 13,568 (Cars3D), 1,824 (Shapes3D), 1,600 (MPI3D) learnable parameters produced by the addition of the filler embedding matrix, M ξ F , to a standard (variational) encoder-decoder framework, we modify models with fewer parameters to have an identical number of parameters as ours. We note that the modifications are applicable only for the scalar-tokened baselines, as our model has tens of millions less parameters than COMET and VCT. In line with [40] , we observe that the performance of scalar-tokened disentanglement models remains fairly consistent, or even deteriorates, when the number of learnable parameters increases, so we defer control experiment results, and our associated method to Appendix C.2.4.The Soft TPR's superior level of explicit compositionality (as quantified by the disentanglement metrics) in a controlled, parameter-count environment, suggests that its fundamentally continuous representational form is potentially easier for deep learning models to learn compared to symbolic representational forms characterising existing disentanglement work.To evaluate whether the inherently continuous compositional form of the Soft TPR can be learned more quickly than symbolic alternatives, we consider representations produced at 100, 1,000, 10,000, 100,000 and 200,000 iterations of training, and evaluate 1) their disentanglement, and 2) their utility, as quantified by the performance of downstream models using these representations. For downstream model performance, we consider two commonly used [30, 46, 33] tasks: the classification-based abstract visual reasoning task of [30] and a regression task involving the prediction of continuous FoV values for the disentanglement datasets. Downstream models are evaluated on a fixed, held-out test set for both tasks. While our framework does not promote faster disentanglement convergence (results in Appendix C.4.1), it interestingly, promotes accelerated learning of useful representations for both downstream tasks compared to baselines. The downstream performance improvements are particularly pronounced in the low iteration regime of 100 iterations of representation learner training, as demonstrated by the improvements of 10%, 10%, and 31% in Table 2 and the 27% improvement in Table 3 . To ensure fair comparison, we embed baseline representations of both higher, and lower dimensionality into the same space as our model. For each baseline model, we take the best result from either the original, or the modified model (denoted by †), and present the full suite of results, and details of our embedding method and downstream model setup in Appendix C.Key Implication: The representation learning convergence results suggests that while our model may not learn the explicit compositional structure captured by disentanglement metrics more quickly than baselines (though in the limit, the greatest possible disentanglement is higher), it learns useful information for downstream tasks more quickly than baselines, where that useful information is encoded in the relaxed compositional structure of the Soft TPR. To evaluate whether the enhanced alignment between compositional representations and continuous vector spaces produced by our Soft TPR benefits downstream models, we examine downstream 1) sample efficiency, and 2) raw performance in the low sample regime. We use the previously mentioned tasks of abstract visual reasoning and FoV regression. To quantify sample efficiency, in line with [25] , we use a ratio-based metric obtained by dividing the performance of the downstream model when trained using a restricted number of samples (100, 250, 500, 1,000 and 10,000 samples), by its performance when trained using all samples (between 19,104-1,036,800 samples depending on the task). As illustrated in Table 4 , our model has superior sample efficiencies compared to baselines, especially in the most restrictive case where downstream models have access to only 100 samples produced by representation learners, achieving a 93% improvement. The Soft TPR representations produced by our model additionally produce substantial raw performance increases in the low sample regime, as evidenced in Table 4 , where its performance in the low-sample regimes of 100 and 200 samples constitutes a respective 138% and 168% improvement, and in Table 5 , a 30% improvement. Key Implication: The consistent performance improvement we observe in generic downstream models with no a priori knowledge of the Soft TPR's representational form, suggests the relaxed, inherently continuous representation of compositional structure embodied by our Soft TPR can be more efficiently leveraged by downstream models compared to symbolic compositional representations, benefiting both sample efficiency, and raw performance in the low sample regime.We additionally repeat our full suite of experiments using the explicit TPR, ψ * tpr , produced by the TPR decoder, in place of our Soft TPR, z. These experiments, a subset of which is presented in Table 6 , empirically demonstrate that the Soft TPR's continuously relaxed specification of compositional structure confers exclusive benefits for both the representation learner and the downstream models not captured by the traditional TPR (see Appendix C.6.1). Note for MPI3D, the explicit TPR has a lower raw R 2 score when fully trained (0.785 ± 0.019 vs 0.882 ± 0.016), contributing to its higher sample efficiency in Table 6 . We additionally examine the importance of the following properties of our model in producing explicitly compositional Soft TPR representations: 1) the presence of weak supervision, by setting λ 1 = λ 2 = 0 in Equation 7, 2) the explicit dependency between the quantised filler embeddings and the decoder output, by instead using the Soft TPR to reconstruct the input image, and 3) the semi-orthogonality of the role embedding matrix, M ξ R by removing this constraint in the random initialisation of M ξ R , with the results of these ablations illustrated in Table 7 . 
conclusion: 
 In this work, we address a longstanding issue in the connectionist approach to compositionality: the fundamental mismatch between disentangled representations and the inherently continuous nature of deep learning vector spaces. To overcome this, we introduce Soft TPR, a novel, inherently continuous compositional representational form that extends Smolensky's Tensor Product Representation, together with the Soft TPR Autocoder, a theoretically-principled architecture designed for learning Soft TPRs. Our flexible, continuous framework yields substantial improvements in the visual domain, enhancing compositional structure, accelerating convergence in representation learners, and boosting efficiency in downstream models. These wide-ranging empirical benefits underscore the importance of rethinking compositional representations to honour deep learning's continuous foundations. Future work will extend our continuous framework to hierarchical forms of compositionality, enabling bound fillers themselves to decompose into role-filler bindings for enhanced representational expressivity.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our work poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: In the main paper, as well as the Appendix, we cite the papers associated with each dataset we use. We additionally make explicit references in the Appendix to all externally created code that we use in our experiments. Furthermore, if the paper is to be accepted, we will clearly provide licenses and attribution to the original authors where applicable in the code files. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The code we submit in our submission corresponds with our Soft TPR Autoencoder architecture, and is thus reasonably straightforward to understand, especially when supplemented by this paper. If this paper is accepted, however, we will include more extensive written documentation for the official, more extensive codebase we release. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used.  
 
 Appendix: In this section, we provide additional details regarding Smolensky's TPR framework [3] , as well as formal proofs of presented results. We refer interested readers to [3] for a more comprehensive dive into this fascinating framework.By defining the constituent components of any compositional object as a set of role-filler bindings, the TPR defines the decomposition function, β, of Section 3.1 that maps from a set of compositional objects, X, to a set of parts, more explicitly as follows [3] :EQUATIONwhere F denotes a set of fillers, and R denotes a set of roles. Note that in contrast to the formal definition of β we use in Section 3.1, which assumes each x ∈ X is decomposable into a set of n parts, the above decomposition allows objects to be decomposed into a variably-sized set of role-filler bindings, with this set corresponding to an element in the powerset of F × R. For the visual representation learning domain we consider, all considered disentanglement datasets clearly have the property of being decomposable into a fixed size set of role-filler bindings, as all images in these datasets contain the same number of FoV types and each FoV type is bound to a FoV token.Due to this property, we can take β as a special subcase of the generalised definition in Equation 8:EQUATIONwhere m : {1, . . . , N R } → {1, . . . , N F } denotes a matching function that associates each role r i with the filler it binds to in β(x) (we again drop the dependence of m on x for ease of notation), and A denotes the set of all possible bindings produced by binding a filler to each of the N R roles, with sizeN F +N R -1 N R(we assume the same filler can bind to multiple roles).We now formally prove that the TPR with β defined in 9 has formψ tpr (x) = C(ψ 1 (a 1 ), . . . , ψ n (a n ))and thus corresponds to the definition of a compositional representation in Section 3.1.Proof. We denote the role embedding and filler embedding functions asξ R : R → V R , ξ F : F → V F respectively.By definition of the TPR in Eq 1, we have that:ψ tpr (x) := i ξ F (f m(i) ) ⊗ ξ R (r i ).and hence,ψ tpr (x) = i ψ i (f m(i) , r i ),wherea i := (f m(i) , r i ) ∈ β(x), ψ i : F × R → V F ⊗ V R ; (f m(i) , r i ) → ξ F (f m(i) ) ⊗ ξ R (r i ), and C is ordinary vector space addition. Hence, almost trivially, ψ tpr (x) clearly has the required form to be a compositional representation. Now, we prove the recoverability of the embedded components {ψ i (f m(i) , r i )} from the TPR, ψ tpr (x), provided that the set of all role embedding vectors, {ξ R (r i )}, are linearly independent. Similar variants of this proof can be found in [3] , [19] .Proof. Assume the set of all role embedding vectors {ξ R (r i )} are linearly independent. Then, the role embedding matrix, M R := (ξ R (r 1 ) . . . ξ R (r N R )) formed by taking the role embedding vectors as columns, has a left inverse, U , such that:U M ξ R = I N R ×N R .Hence, we have that(U M ξ R ) ij = U i: M ξ R :j = I ij .For ease of notation, let u i denote the i-th column of U T , and note that ξ R (r j ) clearly corresponds toM ξ R :j . So, U i: M ξ R :j = (U T :i ) T M ξ R :j = u T i ξ R (r j ) = I ij .Hence, we have that:u T i ξ R (r j ) = δ ij = 1 i = j 0 otherwise.Using the definition of ψ tpr (x),ψ tpr (x) = i ξ F (f m(i) ) ⊗ ξ R (r i ),we apply the (tensor) inner product of ψ tpr (x) with u i :ψ tpr (x) = i ξ F (f m(i) ) ⊗ ξ R (r i ) u i = i ξ F (f m(i) )ξ R (r i ) T u i = i ξ F (f m(i) )δ ji = ξ F (f m(i) ).Thus, the embedding of the filler, f m(i) , bound to each role, r i , can be recovered through use of a tensor inner product with the unbinding vector, u i , corresponding to the i-th column of U T . Note that the representational components of ψ tpr (x), i.e., the embedded bindings, ψ i (f m(i) , r i ) are fully determined by the embedding of the role, ξ R (r i ), and filler, ξ F (f m(i) ), comprising the binding, as ξ F (f m(i) , r i ) simply corresponds to their tensor product. Thus, recovering (ξ(f m(i) ), ξ R (r i )) for each binding in β(x) corresponds to recovering the representational component, ψ i (f m(i) , r i ). So, provided the set of role embeddings are linearly independent, and they can be obtained (e.g. through a look-up table of role embeddings), all representational components, ψ i (f m(i) , r i ) can be directly recovered from the overall TPR representation, ψ tpr (x).In this section, we provide additional information regarding our Soft TPR framework, more extensively detailing our theoretically-informed rationale behind certain design decisions, as well as providing all necessary model architecture information for our Soft TPR Autoencoder to replicate our results (note that code is available at https://github.com/gomb0c/soft_tpr/).In this subsection, we provide a more detailed explanation regarding the fundamental limitations produced by the TPR's stringent specification (Eq 1). For concreteness, we use an example. We consider the following set of roles and fillers, R = {shape, colour}, F = {red, blue, square}.Additionally, we define the role ξ R : R → R 2 and filler ξ F : F → R 3 embedding functions as follows:ξ R (shape) = 1 0 , ξ R (colour) = 1 1 , ξ F (red) = 1 2 3 , ξ F (blue) = 2 2 3 , ξ F (square) = 0 0 1 , 1.Discrete Mapping: Consider the set of possible TPRs, T , that can be produced by R, F, ξ R , ξ F as defined above. We have:ψ tpr (x red_square ) = ξ F (red) ⊗ ξ R (colour) + ξ F (square) ⊗ ξ R (shape) = 1 2 3 ⊗ 1 1 + 0 0 1 ⊗ 1 0 ∼ =        1 2 3 1 2 3        +        0 0 1 0 0 0        =        1 2 4 1 2 3        ψ tpr (x blue_square ) = ξ F (blue) ⊗ ξ R (colour) + ξ F (square) ⊗ ξ R (shape) = 2 2 3 ⊗ 1 1 + 0 0 1 ⊗ 1 0 ∼ =        2 2 3 2 2 3        +        0 0 1 0 0 0        =        2 2 4 2 2 3       These 2 possible TPRs form a discrete, 2 element subset of the underlying representational space, R 6 , T = {(2 2 4 2 2 3) T , (1 2 4 1 2 3) T }. Relaxing the TPR specification to the Soft TPR allows any point, z, in the underlying representational space, R 6 to qualify as a Soft TPR provided that for some explicit TPR, ψ tpr ∈ T , the sufficient closeness requirement holds: ||z -ψ tpr || 2 < ϵ. So, if we consider the set of possible Soft TPRs, T S , we have = T S := {(2 2 4 2 2 3) T + α, (1 2 4 1 2 3) T + α : |α| < ϵ}. Hence, T S clearly has strictly more points than T , so there should be more functions parameterising the map from the observed data to T S compared to T . Furthermore, in contrast to T , which contains discrete (singular) points scattered around in R 6 , T S contains continuous cloud-like regions centered at each ψ tpr ∈ T . Both these factors should make the Soft TPR representation potentially easier to learn and extract information from, compared to the TPR. This is reflected in our empirical results in Section C.6.1, where, compared to the traditional TPR, the Soft TPR demonstrates 1) greater representation learner convergence, 2) superior sample efficiency for downstream tasks, and 3) superior raw downstream performance in the low sample regime. 2. Quasi-Compositional Structure: The traditional TPR enforces a strict algebraic definition of compositionality (i.e. i ξF (f m(i) ) ⊗ ξ R (r i )).Even explicitly algebraically-structured domains such as natural language, do not always adhere to this rigid specification of compositional structure (e.g. French liaison consonants, which consist of a weighted sum of fillers bound to a single role [9] ). The Soft TPR's continuous relaxation of this constraint allows it to represent structures that only approximately satisfy the TPR's strict definition of compositionality. A more relevant example could be the construction of compositional representations of coloured squares where the colour is a weighted combination of fillers blue and red (i.e., varying shades of purple). In this case, even if purple has not been seen by the TPR Autoencoder previously (i.e., there is no quantised filler representing ξ F (purple)), it may be possible for the TPR Autoencoder to produce a Soft TPR corresponding to purple square if there is a suitable ϵ-level relaxation of any of the explicit TPRs (i.e., red square, blue square) that represents a purple square.In contrast to the greedily optimal analytic form of ψ * tpr that we derive in Equation 5, it is possible to instead derive a globally optimal TPR that the (D R • D F )-dimensional vector, z, produced by the encoder, E, best approximates with reference to the Frobenius norm metric of ||z -ψ tpr || F . We fix the set of fillers, F , and roles, R, as well as the embedding functions, ξ F : F → R D F and ξ R : R → R D R to be arbitrary sets and embedding functions respectively. Note that with F , R, ξ F , and ξ R fixed, the only degree of freedom in defining the space of possible TPRs is given by defining M * , the set of role-filler matching functions that define the permissible role-filler binding decompositions of x ∈ X. The globally optimal TPR that z best approximates, ψ opt tpr , is given by:EQUATIONwhereT := { i ξ F (f m(i) ) ⊗ ξ R (r i )} m∈M * denotesthe set of all possible TPRs given the choices of F , R, ξ F , ξ R and M * . Again, we drop the dependency of both ψ tpr and m on x for ease of notation.If M * is a subset of the entire set of matching functions, M, i.e., if there are some constraints to what fillers can be bound to what roles, then, solving for 10 is NP-complete. If M * however corresponds to the entire set of possible matching functions, i.e., M * = M, then, 10 can be solved by simply using any method that solves the (one-to-many) assignment problem. Noting the poor worst case time complexity of O(n 3 ) for such solutions, and guided by our intuition that there should be a more explicit dependency between ψ * tpr and the structure of z, we thus propose the greedily optimal definition of ψ * tpr in 5, which creates an explicit dependency between the unbound soft fillers of z, { fi }, and ψ * tpr .Note that defining m as m(i) := arg min j || fk -ξ F (f j )|| 2 in Equation 5 effectively permits any filler, f j , to bind to a role, r i . This is somewhat inevitable given the weakly supervised framework we are situated in, as without knowledge of the ground-truth FoV type-token bindings, it is impossible to define M * as a subset of all possible matching functions, M, that produces the ground-truth set of role-filler binding decompositions, i.e. we cannot know m ∈ M|{(ξF (f m(i) ), ξ R (r i ))} = β(x) x∈X . We observe, however, that the size of M is combinatorially large, i.e., |M| = N F +N R -1 N R, and contains role-filler binding decompositions that are clearly misaligned with the ground-truth semantics of images, e.g., {cube/object colour, purple/object shape, green/orientation, blue/wall colour, 2π/floor colour} for the image in Figure 2 . We thus, decide to design our Soft TPR Autoencoder with 2 orthogonal and separately achieved aims: 1) representational form, to ensure the produced representation has the form of a Soft TPR, and 2) representational content, to ensure that m produces sensible role-filler bindings, and hence, that the Soft TPR, z, which best approximates ψ * T P R reflects the ground-truth FoV type-token bindings of the image.We now provide additional details on how our method achieves the representational form property. Recall from Section 4.2, that we penalise the Euclidean distance between the encoder, E's output, z, and the explicit TPR ψ * tpr that z best approximates with the analytic form of Equation 5. Such a penalisation should ideally encourage the Soft TPR Autoencoder to produce encodings of inputted images that have the form of a Soft TPR, as all of E's outputs are directly penalised to satisfy the sufficient closeness property of ||z -ψ tpr || 2 5 for the TPR, ψ tpr = ψ * tpr . To recover ψ * tpr , we use our TPR decoder. As specified in Section 4.2, the TPR decoder consists of 3 main modules, 1) Unbinding, where the soft fillers of z, { fi } are unbound from z, 2) Quantisation, where the explicit TPR filler embeddings {ξ F (f m(i) )} with the closest Euclidean distances to each soft filler of z are obtained, and 3) TPR Construction, where the required operations are applied to the embedding vectors of the role-filler bindings produced from previous modules, to construct the explicit TPR,i ξ F (f m(i) ) ⊗ ξ R (r i )with the analytic form of 5. We now provide additional details on these 3 modules.1) Unbinding: While FoV types (roles) can be bound to the same FoV value (fillers), e.g. object colour/magenta, floor colour/magenta, each FoV type (role) clearly represents an independent visual concept type, and so, it is reasonable to use linearly independent embedding vectors for the FoV types. Thus, within our considered domain of visual representation learning, we can reasonably satisfy the linear independence requirement that ensures recoverability of the embedded representational components (ξ F (f m(i) ), ξ R (r i )) from the TPR representation. Note that any randomly initialised role embedding matrix M ξ R ∈ R D R ×N R is likely (though not guaranteed) to consist of N R linearly independent role embedding vectors provided D R >> N R , and furthermore, that we can encourage that the linear independence property to be preserved during all stages of training by adding orthogonality regularisation,||M T ξ R M ξ R -I N R ×N R || F .There is one pertinent problem, however, if we simply let the role embedding matrix, M ξ R , consist of any set of arbitrary, linearly independent role embedding vectors, {ξ R (r i )}. That is, for any choices of N R , D R , where these values are large, it is computationally expensive to obtain U , the (left) inverse of M ξ R required to produce the unbinding vectors, {u i }, needed for unbinding. We thus, decide to leverage the properties of (left-invertible) semi-orthogonal matrices, A ∈ R d×n , for which A T A = I n×n . In this case, U T = M ξ R , and so, the i-th unbinding vector, u i simply corresponds to the i-the role embedding vector, ξ R (r i ). Thus, we can simply unbind the soft filler embedding from z 'bound' to role r i by taking the (tensor) inner product between z and ξ R (r i ), the i-th column of M ξ R .To accomplish this in practice, our unbinding module randomly initialises the role embedding matrix, M ξ R as a semi-orthogonal matrix, through use of torch.nn.utils.parameterizations.orthogonal, and fixes M ξ R during all stages of training (i.e., gradient is not backpropagated to M ξ R ), to ensure that the semi-orthogonal property is preserved during all stages of training. Fixing M ξ R may prompt concerns that that role embedding vectors do not capture semantically informative content. However, for the visual representation learning domain we consider, it is intuitive that role embeddings themselves do not need to convey much semantic content, as the majority of an image's semantic content can be conveyed through filler embeddings and the bindings of fillers to roles. More concretely, filler embeddings, being learnable, can convey semantic information by being close (in some norm) to one another, e.g., the filler embeddings for {green, magenta, blue} may be close in Euclidean distance to one another, and thus convey the semantic information that they correspond to values for the same (colour) role. Additionally, as the decoder, D, receives information from all embedded bindings in the form of the TPR, ψ * tpr , to perform image reconstruction, D should intuitively learn through the reconstruction loss term of Equation 6, the semantics associated with a role. For example, given the embedding for binding (oblong/object shape), D should learn from the reconstruction loss, the mapping between the randomly initialised role embedding vector for object colour and the semantics of the corresponding FoV type (i.e., that this role changes the colour of the object in the image). Now, we proceed to formally prove the aforementioned semi-orthogonality properties.We first prove that for a (left-invertible) semi-orthogonal matrix, A ∈ R d×n , A T A = I n×n .Proof. Let A denote a (left-invertible) semi-orthogonal matrix of dimension R d×n . Writing A out using its columns, {a 1 , . . . , a n }, we have:A T A =    a T 1 . . . a T n    (a 1 • • • a n )As the columns {a 1 , . . . , a n } are orthonormal, we have that: a i • a j = δ ij , where • denotes the dot product, andδ ij =1 i = j 0 otherwise, from which the result clearly follows.For completeness, we also prove that u i corresponds to the i-th role embedding vector, though this follows rather trivially from the properties of semi-orthogonal matrices, and the definition of u i .Proof. Let M ξ R be a (left-invertible) semi-orthogonal matrix. Then, from the properties of semiorthogonal matrices proved earlier, we have that M T ξ R M ξ R = I, and so,U T = M T ξ R T = M ξ R .Hence, the i-th column of U T , which corresponds to the unbinding vector, u i , by definition, is simply(M ξ R ) :i = ξ R (r i ), the i-th role embedding vector.The quantisation module relies on the VQ-VAE vector quantisation algorithm [11] to learn the filler embedding vectors of the filler embedding matrix, M ξ F , and to quantise the soft filler embeddings { fi } into the explicit filler embeddings {ξ F (f i )} with the closest Euclidean distances. Briefly speaking, to perform vector quantisation, the VQ-VAE algorithm simply quantises each soft filler fi into the embedding vector from M ξ F with the smallest Euclidean distance [11] . This clearly corresponds with the definition of m in Equation 5 (i.e. for each soft filler embedding, fi , m matches an explicit filler embedding with the smallest Euclidean distance to fi to produce ψ * tpr ). As this quantisation operation corresponds to an argmax and is thus non-differentiable, the VQ-VAE algorithm uses a simple L 2 loss, the codebook loss, to move the embedding vectors ξ R (r i ) towards the soft filler embeddings, as captured by the first term of the VQ-VAE quantisation loss term in Equation 6. To prevent the embedding space from growing arbitrarily, a commitment loss, corresponding to the final term of the VQ-VAE quantisation loss in 6 is added, to ensure the encoder commits to an embedding.We refer interested readers to [11] for more details on how the quantisation algorithm works.The TPR construction module contains no learnable parameters, and simply deterministically applies the operations required to produce a TPR from the quantised filler embeddings, {ξ F (f m(i) )}, with the corresponding role embeddings, {ξ R (r i )}, from the role embedding matrix, M ξ R .In this section, we provide an explicit form for the 'swapped' TPRs, ψ s tpr (x) and ψ s tpr (x ′ ), used in our weakly supervised reconstruction loss corresponding to the second term of 7.Given a pair of images, (x, x ′ ) where x and x ′ share the same role-filler bindings for all roles but r i , and the identity of r i , but not any of the role-filler bindings are known, we construct the 'swapped' TPRs for x and x ′ as follows:EQUATIONEQUATIONwhere m and m ′ denote the matching functions for x and x ′ respectively. That is, we construct 'swapped' TPRs by simply swapping the quantised filler embedding associated with role r i in constructing new TPR representations for x and x ′ . Note that in contrast to a similar operation that might be applied to scalar-tokened or vector-tokened compositional representations, swapping the quantised filler between the representations for x and x ′ in this case produces a global (not local) effect on the resulting representation.We now provide concrete details of our model architecture. Our model consists of a standard Convbased encoder, E, (Table 8 ) our TPR decoder, (Table 9 ) and a standard Conv-transpose based decoder, D (Table 10 ). The only learnable component of the TPR decoder corresponds to the filler embedding matrix, M ξ F , and so, our Soft TPR Autoencoder only adds an additional D F • N F parameters to a standard (variational) autoencoding framework consisting of the encoder and decoder, where D F and N F correspond to the dimensionality of the filler embedding space, and number of filler embedding vectors respectively.For all our experiments, we use the same general architecture for the encoder, E, the TPR decoder, and the decoder, D. The tunable hyperparameters of the Soft TPR Autoencoder are the following following: 1) architectural hyperparameters N R , N F , D R , and D F corresponding to the number of role (respectively filler) embedding vectors and the dimensionalities of their respective embedding spaces, and 2) loss function hyperparameters β (Equation 6), λ 1 (Equation 7) and λ 2 (Equation 7).In line with VQ-VAE [11] , we set β, the coefficient for the VQ-VAE commitment loss to 0.5. As N R corresponds to the number of FoV types, to ensure fair comparisons with scalar-tokened generative baselines, and COMET, which all assume that the number of FoVs equals 10, (VCT assumes 20), we fix N R to be 10. We tune remaining hyperparameters by running hyperparameter optimisation using the open-source hyperparameter sweep framework of Weights and Biases (WandB). We set the search method as Bayesian search, and the optimisation criterion to be the fully unsupervised MSE reconstruction loss corresponding to the second term in Equation 6. During hyperparameter optimisation, we train all models for between 50,000-100,000 iterations. The obtained hyperparameter values for the Soft TPR Autoencoder are listed in Table 11 . See Section C.6.2 for ablation experiments demonstrating our model's robustness to hyperparameter configurations. In practice, we find that the Soft TPR penalty of Equation 6 is negligible, most likely because the codebook loss of VQ-VAE, which pushes the quantised fillers {ξ F (f m(i) )} and the soft filler embeddings { fi } of z together, is already sufficient to push z to ψ * tpr . To prevent redundantly calculating this term, we remove it from the overall loss in the implementation of our model.Our model is implemented in Pytorch [27] and trained using the Adam optimiser on the loss corresponding to 7. We use a learning rate of 1e-4, and the default setting of (β 1 , β 2 ) = (0.9, 0.999) across all instances of model training.In this section, we provide additional details about our experiments, and present additional results.For disentanglement, we consider the standard disentanglement datasets: Cars3D [4] , Shapes3D [24] , and MPI3D [21] , where for MPI3D, we consider the 'real' variant of the dataset, (i.e., not the 'realistic', 'toy', or 'complex' variants of the dataset). These datasets contain 3, 6, and 7 ground-truth FoVs respectively, and are procedurally generated by taking the Cartesian product of all possible FoV values for each FoV type. We provide metadata in Tables 12, 13 and 14 and denote all continuousvalued FoV types by the symbol ‡ . As the colours in the Shapes3D dataset correspond to 10 linearly spaced values with a natural ordering (e.g. red, with value 0, is perceptually more similar to orange, with value 1, compared to aqua, with value 5), we treat all FoV colour types in the Shapes3D dataset as continuous-valued variables, in line with [42] . For the downstream task of regression, we simply use the above disentanglement datasets, regressing on each dataset's continuous-valued FoVs, as indicated by the ‡ symbol in Tables 12, 13 and 14. For the downstream task of abstract visual reasoning [30] , we use the dataset from [30] . Each sample from this dataset consists of a Raven's Progressive Matrix (RPM) style question (samples of the dataset can be found in [30] ). Each RPM-style question has a set of 8 context panels, and 6 answer panels, where each panel corresponds to an image from the Shapes3D dataset. For the model to select the correct answer out of a selection of 6 possible answer panels, it must correctly infer the abstract visual relation that each row of the context panels share.For the generative weakly supervised, scalar-valued baselines (i.e., AdaGVAE-k, GVAE, MLVAE, SlowVAE, the Shu model), and our model, we perform model training on a single Nvidia RTX4090 GPU. We also perform all associated experiments for these models (i.e., downstream model training, downstream model evaluation, disentanglement evaluation, etc.) on this single Nvidia RTX4090 GPU. Our model takes approximately 1.5 hours to fully train (i.e., run 200,000 iterations) on the Cars3D and Shapes dataset, and approximately 4.0 hours to fully train on the MPI3D dataset.For the vector-valued baselines of COMET and VCT, we perform model training, and all associated experiments on a single Nvidia V100 GPU.For Ada-GVAE, GVAE, MLVAE and SlowVAE, use the Pytorch-based, open-source implementation of [42] , which was verified by authors to reproduce official reported results for each model. For COMET, and VCT, we use official open-source implementations published by the authors of the corresponding models [36, 47] . For the GAN-based model of Shu, we convert the official Tensorflowbased implementation of [35] to Pytorch and verify that our implementation reproduces official results. For all baseline models, we use suggested hyperparameters where dataset-specific hyperparameters are given, otherwise, we perform hyperparameter tuning using an identical WandB hyperparameter sweep setup as our model.As mentioned in the first paragraph of Section 5, for the weakly supervised baselines of Ada-GVAE, GVAE, MLVAE, SlowVAE and Shu, all models with exception to Ada-GVAE assume access to I, the FoV that differ between each pair in an observed sample (x, x ′ ), and thus have identical levels of weak supervision as our model. Ada-GVAE, however, does not assume access to I, so we make the following modification to make Ada-GVAE more comparable with our model. Ada-GVAE adaptively estimates I using a method that relies on the estimation of k := |I|, i.e., the number of FoVs that have changed in each observed sample. Thus, we simply amend Ada-GVAE by providing it access to the ground-truth value for k (note that if we instead provide Ada-GVAE with knowledge of the FoVs comprising I itself, it becomes identical to GVAE). We empirically verify that our modification, which we denote by Ada-GVAE-k, produces superior or in worst case, comparable results to the original Ada-GVAE model. We train all models, with exception to SlowVAE, which assumes all FoV values in observed pairs (x, x ′ ) change, with k = 1 (i.e., only 1 FoV changes between x and x ′ ). This ensures all models are trained in an identical setting as our framework.For the selected hyperparameter configuration associated with each model, including our own, we obtain 5 trained models using 5 random seeds, and collate results over these 5 seeds. All models are trained for 200,000 iterations on all datasets.For the task of FoV regression, in line with [42] , we use a simple, generic MLP model, with the architecture listed in Table 15 . For each disentanglement dataset, the MLP regression model receives representations of images produced by a representation learner (i.e., a disentanglement model), and is trained to predict the corresponding ground-truth FoV values in a supervised fashion. We evaluate regression performance by computing the R 2 score on a held out, randomly selected test set of 1,000 samples.For each of the 5 instances of a representation learning model produced by the 5 random seeds, we obtain 2 MLPs, resulting in a total of 10 MLP models for each representation learner. We obtain each MLP by uniformly sampling the number of output nodes in the first, second, and fifth layers of the MLP, as indicated in 15. All reported results are averaged over these 10 MLP models. Note that all MLP regression models have no a priori knowledge of the representational form (i.e. scalar-tokened symbolic compositional representation, vector-tokened symbolic compositional representation, fully continuous compositional representation) that they will receive during training as we do not provide any inductive biases to the MLP models or optimise them in any representation-learner specific way (apart from through using the supervised, MSE-based training loss).For each MLP model, we use the Adam optimiser on the MSE loss between predicted and ground-truth FoV values, with a learning rate of 1e-4, and the default setting of (β 1 , β 2 ) = (0.9, 0.999). For the abstract visual reasoning task, in line with [30] , we use the Wild Relation Network (WReN) model [18] on representations obtained by all representation learners to predict the ground-truth answer for each RPM matrix. For each of the 5 instances of a model produced by the 5 random seeds, we randomly sample 2 possible configurations of the WReN model, producing 10 WReN models for each representation learning model. In line with [30] , for the edge MLP, g, in the WReN model, we uniformly sample either 256 or 512 hidden units. Similarly, we uniformly sample either 128 or 256 hidden units for the graph MLP, f . We however, fix the number of hidden layers in g to 2, and f in 1, representing the smallest possible number of hidden layers, to constrain the capacity of the WReN model. We refer readers to [18] for further details on the WReN architecture. All WReN models are trained using Adam optimisation on the BCE loss between the predicted logits and ground-truth labels, with a learning rate of 1e-5 and the default setting of (β 1 , β 2 ) = (0.9, 0.999).Recall that our main hypothesis is that neural networks can both learn explicit compositional structure, and leverage it more easily when that structure is instantiated in a fully continuous way, e.g., when embodied by our Soft TPRs. To ensure that the experimental results indeed provide empiricial support for this hypothesis, we apply a series of controls to rule out the contribution of any confounding variables to the empirical results. We detail these controls below.To ensure that any performance boosts on the disentanglement metrics (shown in Tables 16 and 17 ) are not attributable to the slight increase in learnable parameters of our model (note that our model only adds one learnable component, i.e., the filler embedding matrix, M ξ F on top of a standard (variational) autoencoding framework, with this corresponding to 13,568, 1,824, and 1,600 additional parameters for the Cars3D, Shapes3D, and MPI3D datasets respectively), we perform a control to fix the number of learnable parameters in baseline models with fewer learnable parameters compared to our model. The baseline models with fewer learnable parameters than our model are SlowVAE, Ada-GVAE-k, GVAE, ML-VAE, and the Shu model. VCT and COMET are substantially more parameter hungry (10+ million) than our model, so we do not perform the controls on these models. We increase the number of parameters by increasing the number of filters in the convolution (and transpose-convolution) layers of the generative baseline models, and/or increasing the number of convolution (and transpose-convolution) layers until the modified model has at least as many parameters as our model, repeating this process for each of the disentanglement datasets. We denote these parameter-controlled models in Tables 16 and 17 with the symbol * . In line with [40] and as shown in Tables 16 and 17 , we do not observe any increase in disentanglement performance conferred by increasing the number of learnable parameters in these generative, scalar-tokened baselines, so we apply representation learning convergence experiments, and downstream model experiments using the original models, and not their parameter-controlled variants.To ensure that any performance boosts on the downstream tasks of FoV regression and abstract visual reasoning are not attributable to our representation's increased dimensionality, we postprocess representations produced by all models (including models producing higher-dimensional representations) to match the dimensionality of our representation. Note that all generative, weakly supervised models producing scalar-tokened representations produce representations with a dimension of 10 for all datasets, which contrasts with the representational dimensions of 1536 (Cars3D), 512 (Shapes3D), and 320 (MPI3D) of our Soft TPRs, highlighting the necessity of such a control.To perform this control, we apply separate methods for the scalar-tokened and vector-tokened models. For scalar-tokened models, we multiply each dimension k of the latent vector, l k with an random embedding vector e k ∈ R d from an fixed, randomly initialised, semi-orthogonal embedding matrix E ∈ R d×10 , and subsequently concatenate all multiplied random embedding vectors together to form the dimensionality-controlled representation, (l 1 e k , . . . , l 10 e 10 ) ∈ R 10d . d is a dataset specific integer that ensures that the size of the dimensionality-controlled representation is at least as small as the dimensionality of the Soft TPR representation for the given dataset, i.e., d := ⌈dim(z)⌉/10, where z is the Soft TPR representation produced by a given dataset. Note that we choose a semiorthogonal embedding matrix with the intuition that this ensures the maximal distinguishability of each continguous subset of dimensions corresponding to l i e i .For COMET and VCT, which both produce vector-tokened representations, we consider alternative methods of performing dimensionality-control. COMET's representations have a dimensionality of 640 for all datasets, and so, the model produces lower-dimensional representations than ours for Shapes3D and Cars3D, but not MPI3D. VCT, on the other hand, produces representations with dimension 5120, and so, produces higher-dimensional representations than ours for all datasets. For any vector-tokened representation with higher dimensionality than our representation, we apply PCAbased postprocessing to the model representation, reducing the dimensionality of each vector-tokened value in the representation to the required dimensionality, d, where d := ⌈dim(z)⌉/dim(z baseline ), before concatenating all PCA-reduced vectors together to produce the modified representation. For any vector-tokened representation produced by COMET with lower dimensionality than ours, we apply a simple matrix multiplication between the representation, and a randomly initialised matrix of required dimensionality to embed the representation into the same-dimensional space as ours.We denote the results produced by all such postprocessing by † in relevant tables, and indicate such postprocessing in the captions of relevant plots.In reporting the disentanglement metric results for baseline models, we use published results where applicable, i.e., we use the results published in [47] for VCT and COMET and the published results for SlowVAE [39] . For GVAE, MLVAE, and Ada-GVAE-K, we evaluate disentanglement using the Pytorch-based implementation of disentanglement metrics, [42] , which corresponds to a Pytorchbased implementation of the official disentanglement lib of [25] . We also use this implementation to evaluate the disentanglement of representations produced by all models.In line with [47] , we consider the following 4 disentanglement metrics: the FactorVAE score [24] , the DCI Disentanglement score [16] (we refer to DCI Disentanglement as DCI), the BetaVAE score [10] , and the MIG score [15] . We provide a brief overview of all 4 disentanglement metrics, and refer interested readers to the papers these metrics are introduced in for further details, as well as the Appendix of [25] for more details on how the metrics are implemented in [42] and [25] .FactorVAE Metric: A randomly selected FoV of the dataset is fixed, and a mini-batch of observations is subsequently randomly sampled. The representation learner produces representations for the samples. Disentanglement is quantified using the accuracy of a majority vote classifier that predicts the index of the ground-truth fixed FoV based on the index of the representation vector with the smallest variance. DCI Metric: A Gradient Boosted Tree (GBT) is trained to predict the ground-truth FoV values from representations produced by a representation learner. The predictive importance of the dimensions of a representation is obtained using the model's feature importances. For each sample, a score is computed that corresponds to one minus the entropy of the probability that a dimension of the learned representation is useful for FoV prediction, weighted by the relative entropy of the corresponding dimension. An average of these scores over the mini-batch of samples is taken to produce the final score.BetaVAE Metric: This metric quantifies disentanglement by predicting the index of a fixed FoV from representations produced by a representation learner, similar to the FactorVAE metric. In contrast to the FactorVAE metric, however, the BetaVAE metric uses a linear classifier on difference vectors to predict the index of the fixed FoV. Each difference vector is produced by taking the difference between representations produced for a pair of samples (x, x ′ ) with one underlying fixed FoV.MIG Metric: For each FoV, the MIG metric computes the mutual information between each dimension in the representation, and the corresponding FoV. The score is obtained by computing the average, normalised difference between the highest and second highest mutual information of each FoV with the dimensions of the representation.Since disentanglement metrics are typically computed under the assumption that the representation corresponds to a concatenation of scalar-valued FoV tokens, we now detail how we compute the above disentanglement metrics on the Soft TPR, a continuous compositional representation. Similar amendments have been made in COMET and VCT, so that the disentanglement of their representations, corresponding to a concatenation of vector-valued FoV tokens, can be computed.To compute the FactorVAE score of our Soft TPR, we produce a N R -dimensional vector, v, for each Soft TPR, where N R corresponds to the number of roles, i.e. the FoV types. We produce v by simply populating each dimension, v i , with the index of the quantised filler that role r i is bound to, i.e. we set v i to m(i). We use the resulting v's to compute the variances required by the FactorVAE metric, noting that if the FoV type corresponding to role i is fixed, and this is reflected in our representation, dimension i of the v vectors produced in a mini-batch should clearly have the smallest variance, as all fillers, f m(i) , that role r i binds to, will have the same identity across the mini-batch.As the DCI relies on computing the ground-truth FoV values from N R -dimensional representations produced by the representation learner, we again, follow a similar procedure as the above, in converting our (D F • D R )-dimensional Soft TPR representation to a N R -dimensional representation. That is, we simply consider a N R -dimensional vector, v, where each dimension, v i , is populated by the index of the quantised filler, m(i) that role r i is bound to, and use this vector to compute the corresponding DCI result.BetaVAE metric: For the BetaVAE metric, as each sample used to train the linear classifier consists of a N R -dimensional difference vector obtained by computing the difference between the N Rdimensional scalar-tokened compositional representations, we obtain the following difference vector, d, for our Soft TPR representations. For each role i ∈ {1, . . . , N R }, we obtain the corresponding quantised filler embeddings for each sample x, x ′ in the pair, i.e., we obtain ξF (f m(i) ), ξ F (f m ′ (i) ).For each pair of quantised filler embeddings, we obtain a scalar distance measure corresponding to the cosine similarity between the the pair. The i-th dimension of d is populated using this value. We use difference vectors obtained in this way to compute the BetaVAE metric.MIG metric: For the MIG metric, which relies on a discretisation of the values in each dimension i of the N R -dimensional scalar-tokened compositional representations to compute the discrete mutual information, we apply the same postprocessing technique as in the FactorVAE, and DCI metric, to evaluate MIG on our Soft TPRs. That is, we produce the same N R -dimensional vector, v, noting that this choice of postprocessing also performs the discretisation required by the MIG computation.We now present unabridged results for all considered disentanglement metrics, denoting the learnable parameter control modification of each relevant baseline with the symbol * . As can be seen in Tables 16 and 17 , our Soft TPR representations are explicitly more compositional (as quantified by the disentanglement metric scores) compared to all considered baselines, with especially notable performance increases on the more challenging datasets of Cars3D and MPI3D. We additionally examine representation learning convergence by evaluating the representations produced at 100, 1,000, 10,000, 100,000, and 200,000 iterations of model training, where the latter stage of 200,000 iterations corresponds to fully trained models. To quantify representation learning convergence, we evaluate both 1) the explicit compositionality of representations produced at these stages of training (as quantified by disentanglement metric performance), and 2) the usefulness of these representations for the downstream tasks of FoV regression and abstract visual reasoning.As mentioned in Section 5.1, the representation learning convergence of our model as measured by disentanglement performance is comparable with baselines, however, our model consistently converges faster than baselines in producing representations that can be effectively leveraged for both downstream tasks, as indicated by higher downstream model performance across the majority of representation learner training stages. We now present the full suite of unabridged results, first presenting disentanglement results, and subsequently downstream results.In all line plots, we plot the mean, and indicate the standard deviation by the shaded regions. We use the same legend for all plots, where 0 (grey) denotes SlowVAE, 1 (orange) denotes AdaGVAE-k, 2 (green) denotes GVAE, 3 (red) denotes MLVAE, 4 (purple) denotes Shu, 5 (pink) denotes VCT, 6 (brown) denotes COMET, and 7 (blue) denotes our model, Soft TPR Autoencoder.We first present line plots of representation learner convergence for each of the four considered disentanglement metrics (i.e., the FactorVAE, DCI, BetaVAE and MIG scores) for all three disentanglement datasets (Cars3D, Shapes3D, MPI3D). A series of tables containing the values associated with these line plots is presented following the plots. As the disentanglement results produced by our learnable parameter controls for the models of Ada-GVAE-k, GVAE, ML-VAE and Shu, do not achieve superior disentanglement results compared to the original models, we only present disentanglement convergence results for the original variants of all baseline models. Slow-VAE 0.937 ± 0.025 0.999 ± 0.002 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) Ada-GVAE-k 0.971 ± 0.011 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) GVAE 0.958 ± 0.027 0.999 ± 0.001 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) MLVAE 0.964 ± 0.011 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) Shu 0.534 ± 0.164 0.742 ± 0.072 0.728 ± 0.105 0.433 ± 0.016 0.449 ± 0.099 Symbolic vector-tokened compositional representations VCT 0.889 ± 0.044 1.000 ± 0.001 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) COMET 0.592 ± 0.027 0.592 ± 0.028 0.592 ± 0.028 0.592 ± 0.028 0.592 ± 0.028Fully continuous compositional representations Ours 0.953 ± 0.042 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 VCT 0.889 ± 0.044 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) COMET 0.395 ± 0.000 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=)Fully continuous compositional representations Ours 0.638 ± 0.015 0.710 ± 0.035 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) Slow-VAE 0.597 ± 0.048 0.927 ± 0.007 0.960 ± 0.032 0.999 ± 0.000 (=) 1.000 ± 0.000 (=) Slow-VAE † 0.564 ± 0.060 0.920 ± 0.011 0.955 ± 0.037 0.999 ± 0.000 (=) 0.999 ± 0.000 Ada-GVAE-k 0.684 ± 0.068 0.974 ± 0.007 0.997 ± 0.000 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) Ada-GVAE-k † 0.698 ± 0.063 0.973 ± 0.007 0.997 ± 0.000 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) GVAE 0.565 ± 0.034 0.951 ± 0.015 0.997 ± 0.000 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) GVAE † 0.565 ± 0.022 0.951 ± 0.014 0.998 ± 0.001 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) MLVAE 0.551 ± 0.059 0.959 ± 0.012 0.997 ± 0.000 0.999 ± 0.000 (=) 1.000 ± 0.000 (=) MLVAE † 0.556 ± 0.055 0.960 ± 0.010 0.996 ± 0.000 0.999 ± 0.000 (=) 1.000 ± 0.000 (=) Shu 0.327 ± 0. 0.474 ± 0.062 0.809 ± 0.013 0.970 ± 0.011 1.000 ± 0.000 (=) 1.000 ± 0.000 (=)Fully continuous compositional representations Ours 0.981 ± 0.003 0.989 ± 0.007 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) 1.000 ± 0.000 (=) We present now present our full suite of results for the downstream task of abstract visual reasoning.As demonstrated in Table 33 and the corresponding Figures 21 and 22 , representations produced by our model at only 10 2 iterations of training are able to be leveraged by downstream models to achieve a 80.04% accuracy for the challenging abstract visual reasoning task, in contrast to the value of 63.1% obtained by the best performing baseline, representing a 26.78% performance increase. This again provides strong empirical support for our hypothesis that the approximate, continuouslyinstantiated compositional structure embodied by our Soft TPR can be learnt more quickly by representation learners than alternative, symbolic representations of compositional structure, thereby allowing downstream models to effectively leverage these representations despite a small number of representation learner training iterations. We present our full suite of experimental results that empirically demonstrate the utility of our Soft TPR representation from the perspective of downstream models, by considering the sample efficiency, and low-sample regime performance of downstream models on the tasks of FoV regression, and abstract visual reasoning.For sample efficiency, as mentioned in Section 5.2, and in line with [25] , we compute a ratio-based metric obtained by dividing the performance of the downstream model when trained using a restricted number of 100, 250, 500, 1,000 and 10,000 samples, by its performance when trained using all samples. The total number of samples corresponds to 19,104, 480,000, 1,036,800 and 100,000 for the tasks of regression on the Cars3D, Shapes3D, MPI3D datasets, and the abstract visual reasoning task respectively. As this metric is dependent on the performance of downstream models when trained using all samples, we do not compute this metric for representation learners where the corresponding downstream models achieve an R 2 score of less than 0.5 for regression, as this may produce sample efficiency scores with very little semantic meaning (e.g. a model that achieves a sample efficiency score of 0.9 when its final R 2 score is 0.1). This corresponds to removing the Shu model from Shapes3D sample efficiency calculations, and COMET and Shu from the Cars3D sample efficiency calculations.As many models for the abstract visual reasoning task have low classification accuracies on the held-out test set following training with the maximal number of 100,000 samples, we do not compute sample efficiencies for this task, and instead refer readers to results in Section C.5.2 for the raw classification accuracies associated with each model.Note that for all box plots, we follow standard convention, and display the median in each box with a solid line, where the box shows the quartiles of the corresponding values, and the whiskers extend to 1.5 times the interquartile range. We again, use the same legend, where grey denotes SlowVAE, orange denotes AdaGVAE-k, green denotes GVAE, red denotes MLVAE, purple denotes Shu, pink denotes VCT, brown denotes COMET, and blue denotes our model, Soft TPR Autoencoder. To verify that our Soft TPR confers exclusive benefits compared to TPRs, we repeat all our experiments using the explicit TPR that our TPR decoder produces, which represents the Soft TPR's greedily optimal explicit TPR counterpart, ψ * tpr . In all plots, we use the same legend, denoting the explicit TPR as yellow (0) and the Soft TPR as blue (1) . Across all considered cases: i.e., 1) convergence rate of representation learning (as measured by the downstream model's ability to effectively leverage representations produced at different stages of training), 2) sample efficiency of downstream models, and 3) raw performance of downstream models in the low sample regime, the Soft TPR confers differential performance boosts compared to the explicit TPR. This offers strong empirical evidence for our hypothesis that embodying a more approximate form of explicitly compositional structure helps representation learners by alleviating the strigent requirement of having to produce explicit TPRs, and additionally provides representation learners with more knowledge of the entire manifold underlying compositional structure, which TPRs cannot fully capture either due to representation learner deficits, or the inherent quasi-compositional structure of represented objects. We perform an additional experiment to empiricially verify that our model is robust to different hyperparameter choices. For the MPI3D dataset, which disentanglement models experience the greatest difficulty in producing disentangled representations for (see Table 1 ), we randomly pick another set of hyperparameters, shown in Table 41 , from the top 5 models based on the MSE loss criterion. For this randomly chosen hyperparameter configuration, we evaluate the disentanglement of the representations on the MPI3D dataset produced by the resulting model. Applying the Soft TPR to the TPR's typical domain of language is an intriguing future direction, especially as language can deviate from strict algebraic compositionality -for instance, idiomatic expressions such as 'spill the beans' cannot be understood as a function of their constituents alone. Soft TPR's more flexible specification allows it to capture approximate forms of compositionality precluded from the TPR's strict algebraic definition (Eq 1), thereby potentially providing the Soft TPR the ability to better handle the nuance and complexity of language.To adapt our framework to language, we replace our Conv/Deconv encoder/decoders with simple RNNs, retrain our TPR decoder, and remove the semi-supervised loss, using Eq 6 as the full loss.Preliminary results in Table 43 on the BaBI [7] dataset are compared with TPR baselines from AID [51] . Our Soft TPR Autoencoder does not presently surpass AID, but notable points include:1. Our use of simpler RRN-based encoders and an MLP-based downstream network, unlike the more sophisticated architectures of [51] . 2. Soft TPR retains its performance improvement above the corresponding explicit TPR it can be quantised into. 3. The smaller gap between systematic vs non-systematic dataset splits in our model compared to TPR-RNN (+AID) and FWM. 4. We train our representation learner using self-supervision (reconstruction loss) alone, only employing supervision on the downstream prediction network, while the baselines employ strong supervision and end-to-end training to produce representations. To produce a compositional representation, ψ(x) = C(ψ 1 (a 1 ), . . . , ψ n (a n )) (as in Section 3.1), each representational constituent, ψ i (a i ), must map 1-1 to a data constituent, a i . Without supervision (i.e., access only to observational data, {x i }), this is challenging, because the data constituents {a i } underlying each object, x, are unknown and cannot be identified.We can frame the above intuition in a more mathematically rigorous way, using the generative framework. Essentially, as formally proved in [25] , it is impossible to identify the true distribution for the data constituents (generative factors), p(a), using observational data, {x i }, alone as there are infinitely many bijective functions f : supp(a) → a such that: 1) a and f (a) are completely entangled (i.e. non-diagonal Jacobian) and 2) the marginal distributions of a and f (a) are identical (meaning the marginal distributions of the observations are also identical, i.e., p(x|a)p(a)da = p(x|f (a))p(f (a))da). Thus, without inductive biases, it is impossible to infer the data constituents {a i } of any observation, x, from observational data {x i } alone.To combat this non-identifiability result, in line with [13, 22, 31, 33, 35] , we use weak supervision, presenting the model with data pairs (x, x ′ ) where x and x ′ differ in a subset of FoVs, e.g. β(x) = {shape/cube, colour/purple, size/large}, β(x ′ ) = {shape/cube, colour/cyan, size/large}, where the FoV types corresponding to the different FoV values are known to the model. Note that this weak supervision is minimal, only providing the model to access to the differing FoV types in an index set, I, (i.e., I := {colour}) and not any of the FoV values (i.e., {cube, purple, cyan, large} are all not known by the model). Some possible future extensions to reduce this level of weak supervision, or alternative forms of weak supervision include:1. Embodied Learning: In the visual domain, some roles, e.g. object position correspond to affordances. Embodied agents may be able to reduce the need for explicit supervision by collecting (x, x ′ ) and I through interaction with their environment. 2. Pretrained Filler Embeddings: Initialising the filler embedding matrix, M ξ F with embeddings learnt by a pre-trained vision model could impart knowledge of domain-agnostic fillers (e.g., colours, shapes), reducing the need to explicitly provide (x, x ′ ) and I to the model. 3. Segmentation Masks: Segmentation masks for each object may potentially reduce the need to explicitly provide (x, x ′ ) and I to the model.Our investigation of downstream utility centers on two selected tasks -FoV regression/classification and abstract visual reasoning, which aligns with the standard framework for assessing the quality and downstream utility of compositional representations [26, 30, 33, 40, 44, 46] . While existing work [30, 33] demonstrates that explicitly compositional representations enhance downstream sample efficiency compared to non-compositional representations, a result we improve upon (C.5.1), the broader utility of compositional representations remains a topic of ongoing exploration [21, 40, 42, 44] .Theoretical perspectives [1, 2] argue that explicitly compositional representations are fundamental in the production of productive, systematic, and inferentially coherent thought -3 key properties characterising human cognition. Investigating how explicitly compositional representations can yield empirical benefits across these dimensions represents an essential avenue for future research.Although preliminary studies [40, 42, 44] do not find strong evidence that explicitly compositional representations improve compositional generalisation (a key aspect of systematicity), [44] suggests that this finding is because compositional representations are necessary, but not sufficient to induce systematicity; an explicitly compositional processing approach [6] is also required.Future work could extend our theoretical framework with the hope of producing empirical results consistent with the theoretical arguments of [1, 2] . In particular, as our unbinding module is designed to provably and efficiently recover structured role-filler constituents from the Soft TPR, it may be possible to exploit this module to systematically reconfigure roles and fillers from existing representations to create representations of novel combinations of role-filler bindings (i.e., novel compositional data). This type of ability could prove extremely beneficial in areas such as concept learning and compositional generalisation.In this subsection, we denote the dimensions of the role and filler embedding spaces as D F and D R respectively. We also denote the number of fillers as N F and the number of roles as N R .The Soft TPR belongs to V F ⊗ V R , which is a D F × D R dimensional space, which grows multiplicatively in D F , D R . Several factors, however, mitigate scalability concerns in light of this fact:1. Independence of Embedding Space Dimensionality: We note that the dimensionality of the role and filler embedding spaces (D R , D F respectively) are properties of the corresponding embedding functions (ξ R : R → V R and ξ F : F → V F ) and thus, can be fixed independently of the number of roles, N R , the number of fillers, N F , or the number of total role-filler bindings (which we denote by n) within a domain. Thus, it is possible to fix the Soft TPR's dimensionality (D F × D R ) to be smaller than N F × N R (the number of roles/FoV types multipled by the number of fillers/FoV tokens) or n (the number of bindings), which all may be large in complex visual domains. As illustrated in Table 44 , the dimensionality of the TPR is smaller than N F × N R in both the Shapes3D and MPI3D domains. 2. Relaxing Orthogonality: While D F can be set a priori with no regard to N R , N F or n, we require D R ≥ N R for semi-orthogonality of the role-embedding matrix M ξ R , which guarantees faithful (see proof 2 of A.2) and computationally efficient (see 'Unbinding' heading of B.3.1) recoverability of constituents. It is, however, possible to relax this constraint (i.e., to have D R < N R ) to further reduce dimensionality. In this case, semiorthogonality of M ξ R is impossible and hence the recoverability of constituents cannot be guaranteed, however, there are some less stringent guarantees on the outcome of unbinding that can still be derived (see p291 of [3] for more details).We also more explicitly compare the dimensionality of the Soft TPR with baselines in Table 45 . Scalar-tokened symbolic representations have a low dimensionality of 10 (N R ) at the expense of representational expressivity (each representational constituent ψ i (a i ) is scalar-valued). In contrast, Soft TPR has vector-valued representational constituents (i.e. ≈ ξ F (f m(i) ) ⊗ ξ R (r i )), similar to VCT and COMET. When compared to these models, the Soft TPR has significantly lower dimensionality compared to VCT and is comparable with COMET. In the Soft TPR Autoencoder, the expensive tensor product operation is employed to generate ψ * tpr . Given the computational cost of the tensor product, we more concretely examine the computational cost of training the Soft TPR Autoencoder by computing the FLOPs for a single forward pass on a batch size of 16 using the open-source implementation of fvcore https://github.com/ facebookresearch/fvcore/tree/main/docs, visible in Table 46 . This data demonstrates that, despite the tensor product's computational cost, the mathematically-informed derivation of our model allows it to obtain compositional representations with vector-valued representational constituents at a significantly lower cost compared to relevant, vector-tokened baselines (2 orders of magnitude less than VCT, and 4 orders of magnitude less than COMET).Future work could explore the use of tensor contraction techniques to reduce computational expense. For instance [29] uses a Hadamard product based tensor product compression technique. This reduces computational cost from n 2 (tensor product of 2 vectors) to n (Hadamard product), but comprises the theoretical guarantees on constituent recoverability. We believe developing tensor contraction techniques within the TPR framework is an important direction for future research, to ensure efficient TPR-based representations with provable recoverability of constituents. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: As the central aim of our work relates to the representational form of compositional representations, our research is theoretical in nature and has no immediate societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We assume a finite set of NR roles, an intuitive assumption to make in the context of visual representation learning.We omit the dependence of m on x for notational clarity.We occasionally omit the dependence on x for notational clarity.Due to isomorphism of vector spacesR D F ⊗ R D R ∼ = R D F •D R , we henceforth use vectors from R D F •D R in place of rank-2 tensors from R D F ⊗ R D R ,and the Euclidean norm instead of the Frobenius norm to align the Soft TPR framework more seamlessly with the autoencoding framework.We again assume we are working with vector spaces over the reals, and use the vector space isomorphism property ofR D F ⊗ R D R ∼ = R D F •D Rto align the TPR framework more seamlessly with the autoencoding framework. 


Decision: This paper proposes Soft Tensor Product Representations (Soft TPR) and the Soft TPR Autoencoder, a novel continuous compositional representation framework designed to bridge the gap between discrete symbolic representations and continuous vector spaces.

It tackles a crucial problem addressing a major limitation of current deep learning. Reviewers found the paper strong due to its thorough theoretical analysis, novelty of approach, clear explanation, and comprehensive experiments. Initial concerns, such as some experimental setting details, were also well addressed during the rebuttal. Therefore, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel framework for representation learning known as Soft Tensor Product Representations (Soft TPR), aimed at capturing the compositional structure of data more effectively. The authors propose a continuously-valued compositional representation that contrasts with traditional symbolic methods. The paper makes several key contributions in the realm of representation learning, including the conceptualization of Soft TPR, the Soft TPR Autoencoder, and the demonstration of the benefits of this representation for both representation learning and downstream models.
strength_weakness: strength_weakness: 1. No MPI scores for disentanglement (in Table 1) are reported. 
2. The construction of the model's loss function involves a multitude of hyperparameters, suggesting that the model might require a complex and intricate tuning process. Compared to other models like VCT, the process of adjusting parameters to achieve optimal results could involve additional complexity and such a requirement might pose challenges in terms of computational resources and time.
3. The reason why *Soft TPR is better than TPR is not fully explored.
questions: questions: 1. what is the dissentanglment performance of TPR?  The paper only report the downstream results of TPR in Table 6, is it possible to report the disentangment performance of TPR in Table 1?
2. how many pairs are provided for the weakly supervised? For example, in Table 4, the numbers are the total samples for training, or the labeled sample pairs?
3. VCT uses attention-based operations (cross-attention and self-attention) to extract concept vectors, and reconstruct images from the concept tokens, what is the difference between those attention-based operation and the unbinding module  or TPR constructor in this paper?
limitations: limitations: I don't see any negative societal impact.
paper_summary: paper_summary: This work explores compositional representations -- considered a crucial capability underlying intelligent human behaviour in deep learning systems. It argues that there is an incompatibility between discrete symbolic compositional representations—e.g. as obtained through traditional disentanglement approaches—and the continuous vector spaces underlying deep learning systems. To address this, the authors introduce a novel continuous compositional representation that builds on the Tensor Product Representation (TPR) approach, akin to a soft approximation to the TPR. They introduce a method for learning soft TPRs with weakly supervised data called the Soft TPR Autoencoder, and apply it to visual representation learning. This model demonstrates state of the art disentanglement for representation learning and improved sample efficiency for downstream models using those representations.
strength_weakness: strength_weakness: - motivation for the approach & model could be unpacked a bit more (see questions)
- other domains: the authors focus on applying the work to the visual data, outside the typical domain of TPR models. In general, a strong feature, enabling them to tackle the messier world of complex visual data (and weaker super vision) and compare the model to the many disentanglement approaches that have previously been applied in the visual domain. However, it could be interesting to see how the _Soft_ TPR approach compares to traditional TPR in its typical domain of language.
 - no high impact is shown for downstream utility. the improvements seen for downstream tasks are certainly interesting, particularly in the low data regime, however only two tasks are explored. are there any more good downstream tasks to evaluate the utility of their learned representations? could there even be more speculation as to where future work might really leverage what can be learned with this approach?
 - need for supervision (albeit weakly): could the work be extended to use different forms of supervision or less/no supervision. noting that their ablation showed the importance of the supervision, perhaps the authors could explore more variants on this ablation, and speculate on extensions of the model that could reduce the need for supervision?
questions: questions: A couple of questions, both on the motivations of the approach in the paper and how it could be improved. In both cases, these are more questions focused on the conceptual motivations and understanding than what the results show:
 - In terms of motivating the approach, can you be more specific about how representations obtained with traditional disentanglement approaches (typically continuously-valued themselves) could create an incompatibility with the continuous vector space of deep learning systems, and lead to suboptimal performance for the representation learning and downstream models? Any hypothetical or concrete examples (and/or references discussing this point)? 
- Relative to TPR, what is the relaxation in Soft TPR supposed to be helping with, and why? The manifold example is given, e.g. in Figure 1c, but seems quite vague. Also discussed is that the data being explained maybe only approximately compositional, but why would TPR fail with this and why would we specifically expect soft TPR to help? Could this be unpacked more, perhaps with some concrete examples.
limitations: limitations: N/A
paper_summary: paper_summary: The authors propose the use of the tensor product to model the interactions between object properties and their values, in contrast to the usual concatenation-based fusion for compositional representations. Extensive experiments on a large variety of image datasets are performed, where performance gains are often significantly higher than those of the baselines considered.
strength_weakness: strength_weakness: # [W1] Multiplicatively large dimensionality of the TPR representations & issues scaling to larger settings

My main criticism of the paper is the resultant TPR's dimensionality. In particular, the soft TPR representations live in a $D_f\cdot D_r$ dimensional space. Due to the use of the Kronecker product, the dimension of the TPR representations grows multiplicatively with the two terms. 

Whilst the datasets studied in the paper are relatively simple and only 10 factors of variation are modeled, it seems prudent to acknowledge that the TPR size could grow prohibitively large for increased values of $D_f, D_r$ for more complex datasets outside of controlled settings, where significantly larger number of FoVs exist. In particular, the NeurIPS checklist states: `The authors should discuss the computational efficiency of the proposed algorithms`.

For example, I see from [L1008] that $N_R:=10$, and $N_F$ is as high as $106$ for Cars3D. Even in this regime with a very small number of roles, the TPR representations are (presumably?) larger than the concatenation-based representations of the baselines. A comparison of FLOPs/multiply-adds needed for the compared methods would be appreciated to better understand the methods’ drawbacks through the use of the (often computationally expensive) tensor product.
questions: questions: **[Q1]** As expanded upon in [L933], the “role” embedding matrix $M_{\xi_R}$ is initialized with $n$ orthogonal columns but not trained. We are told that it is “intuitive” that role embeddings do not need to convey semantic information, but I think it would be interesting to see an ablation study on this.

In particular: if it is the case that the embedding matrix does not need to learn semantic representations, why not make the simpler choice of initializing $M_{\xi_R}$ as a (truncated) identity matrix (this shares the same property of semi-orthogonality)? The decision not to make this parameter learnable is peculiar, and I would be interested in seeing its impact explored experimentally (for example, through the DCI score as in Table. 7).
limitations: limitations: Limitations are discussed well throughout (and in further detail throughout the appendix), but an extra discussion of the potential drawbacks of increased computational costs should be made.

Comments: comment: Dear Reviewer 93av, 

We are glad our response addressed many of your concerns. Thank you again for your time and insightful review. We really appreciate your thoughtful feedback, 

Submission14814 Authors.
Thanks for the detailed and clear follow-up, many of concerns have been addressed and I have raised my score.
Dear Reviewer 6VtP, 

We are glad that our response addressed most of your concerns. Thank you again for your time and thoughtful feedback regarding our approach.

Best,

Submission14814 Authors.
Thanks for the response. Most of my concerns are addressed. I have raised my score.
Dear Reviewer xKnt, 

We are glad you found our additional experiments and ablations insightful. Your feedback on our approach, especially regarding the dimensionality, is greatly valued. Thank you again for your time and thoughtful review. 

Best,

Submission14814 Authors.
Thanks to the authors for their thorough response.

The rebuttal was a pleasure to read; the extra experiments and ablations are insightful, and the reviewers make a good point that—despite the dimensionality of the space growing multiplicatively—the independent dimensionality of the two factors means this is not always too problematic.

The “weaknesses” raised in my initial review were only very minor in the first place—the authors have acknowledged the multiplicative growth of the space, and I see no critical reasons why the paper should not be accepted.