Title: Fully Distributed, Flexible Compositional Visual Representations via Soft Tensor Products
Venue: NeurIPS 2024
Accepted: yes
Keywords: representation learning

compositional representations

disentanglement

tensor product representations

compositionality
Abstract: Since the inception of the classicalist vs. connectionist debate, it has been argued that the ability to systematically combine  symbol-like entities into compositional representations is crucial for human intelligence. In connectionist systems, the field of disentanglement has gained prominence for its ability to produce explicitly compositional representations; however, it relies on a fundamentally *symbolic, concatenative* representation of compositional structure that clashes with the *continuous, distributed* foundations of deep learning. To resolve this tension, we extend Smolensky's Tensor Product Representation (TPR) and introduce *Soft TPR*, a representational form that encodes compositional structure in an inherently *distributed, flexible* manner, along with *Soft TPR Autoencoder*, a theoretically-principled architecture designed specifically to learn Soft TPRs.  Comprehensive evaluations in the visual representation learning domain demonstrate that the Soft TPR framework consistently outperforms conventional disentanglement alternatives -- achieving state-of-the-art disentanglement, boosting representation learner convergence, and delivering superior sample efficiency and low-sample regime performance in downstream tasks. These findings highlight the promise of a *distributed* and *flexible* approach to representing compositional structure by potentially enhancing alignment with the core principles of deep learning over the conventional symbolic approach.

Fully Distributed, Flexible Compositional Visual Representations via Soft Tensor Products

Abstract

Since the inception of the classicalist vs. connectionist debate, it has been argued that the ability to systematically combine symbol-like entities into compositional representations is crucial for human intelligence. In connectionist systems, the field of disentanglement has gained prominence for its ability to produce explicitly compositional representations; however, it relies on a fundamentally symbolic, concatenative representation of compositional structure that clashes with the continuous, distributed foundations of deep learning. To resolve this tension, we extend Smolensky's Tensor Product Representation (TPR) and introduce Soft TPR , a representational form that encodes compositional structure in an inherently distributed, flexible manner, along with Soft TPR Autoencoder , a theoretically-principled architecture designed specifically to learn Soft TPRs. Comprehensive evaluations in the visual representation learning domain demonstrate that the Soft TPR framework consistently outperforms conventional disentanglement alternatives - achieving state-of-the-art disentanglement, boosting representation learner convergence, and delivering superior sample efficiency and low-sample regime performance in downstream tasks. These findings highlight the promise of a distributed and flexible approach to representing compositional structure by potentially enhancing alignment with the core principles of deep learning over the conventional symbolic approach.

1 Introduction

Compositional structure, capturing the property of being decomposable into a set of constituent parts, permeates numerous aspects of our world - from the recursive application of syntax in language, to the parsing of richly complex visual scenes into their constituent parts. Given this ubiquity, it is natural to seek deep learning representations that also embody compositional structure. Indeed, empirical evidence demonstrates a multitude of benefits conferred by explicitly compositional representations, including increased interpretability [ 10 , 12 ], reduced sample complexity [ 30 , 33 ], increased fairness [ 20 , 25 , 41 ], and improved performance in out-of-distribution generalisation [ 33 , 48 , 50 ].

We consider the following, intuitive notion of compositional representations. A representation of compositionally-structured data is a compositional representation if its structure explicitly reflects the constituency structure of the represented data [ 49 ]. In the visual representation learning domain, data is clearly compositionally-structured , as images can be decomposed into a set of constituent factors of variation (FoVs), e.g., { magenta floor , orange wall , aqua object colour , oblong object shape } for the image in Figure 1 .

Gradient Flow and Learning : Symbolic compositional representations utilise a localist encoding approach, assigning each FoV to a unique, non-overlapping subset of the dimensions of the representation. This modular separation may introduce practical misalignments with gradient-based optimisation. By restricting each FoV to its own slot, gradients for one FoV are confined to a set of designated dimensions, limiting the smooth propagation of gradients globally across all dimensions of the representation. Consequently, updates to a single FoV provide minimal feedback to other FoVs, hindering the model's ability to jointly refine interdependent components. Furthermore, the localist encoding - where each FoV occupies a small, disjoint subset of the total dimensions - can induce abrupt, discontinuous shifts in the representation when transitioning between FoV updates, potentially complicating convergence.

Representational Expressivity : The localist encoding inherent in symbolic compositional structures, where each FoV is assigned a distinct subset of the overall dimensions, may constrain practical expressiveness. Specifically, confining each of the n FoVs to a subset of size n/d within a d -dimensional representation causes each FoV to underutilise the full capacity of the d -dimensional space, potentially limiting the richness and flexibility of the learned representations.

Robustness to Noise : By confining each FoV to a small, non-overlapping subset of dimensions, symbolic compositional representations become highly vulnerable to dimension-wise noise. Even slight perturbations in a single dimension can significantly impair the representation of the corresponding FoV, as there is no overlapping or redundant encoding to mitigate such disruptions.

Critically, we hypothesise that this potential incompatibility between symbolic, localist representations and the distributed, continuous nature of deep learning results in suboptimal behaviour in models that learn or use these representations. To overcome these limitations, we are motivated to pursue an inherently distributed approach to representing compositional structure. Instead of concatenating discrete slots ( FoV tokens ) dimension-wise to form the compositional representation, an inherently distributed approach continuously combines densely encoded FoVs within a unified vector space. This design allows information from each FoV to be smoothly interwoven throughout all dimensions of the representation, as illustrated in Figure 1 b, potentially resulting in smoother gradient flow, enhanced expressivity, and heightened robustness to noise. By reconciling the demand for representations that are explicitly compositional with the continuous, distributed nature of deep learning, distributed compositional representations offer a compelling alternative to traditional symbolic, slot-based paradigms.

Floor colour

Object colour

Wall colour

Factors of variation

Compositional representations

a)

b)

Object size

Orientation

Object shape

c)

Pioneered by Smolensky, the Tensor Product Representation [ 3 ] is a specific representational form that encodes compositional structure in an inherently distributed manner. At the crux of it, TPRs are formed by continuously blending representational components together into the overall representation, in a manner analogous to superposing multiple waves together to produce a complex waveform, as illustrated in Figure 1 b. For a representation to qualify as a TPR, it must adhere to a highly specific mathematical form, which confers upon the TPR valuable structural properties (elaborated on in Section 3.2 ), but also imposes two limitations (see B.1 for further details). First, as depicted by the stars in Figure 1 c, only a discrete subset of points in the underlying representational space, ˜ V , satisfies the stringent mathematical criteria to qualify as TPRs. Consequently, to learn TPRs, representation learners must map from the data manifold onto this discrete subset , which constitutes a highly constrained and inherently challenging learning task. Second, the TPR specification enforces a strict, algebraic definition of compositional structure, limiting the TPR's ability to flexibly represent ambiguous, real-world data which is often quasi -compositional, only approximately adhering to some rigid, formal definition. Historically, these limitations have confined TPR learning to formal domains characterised by explicit, algebraic structure - as evidenced by the near exclusive deployment of TPRs in language [ 19 , 23 , 28 , 34 , 52 ] - and, to contexts where strong supervision from highly structured downstream tasks is available to steer the representation learning process [ 23 , 28 , 38 , 51 ]. To negate these drawbacks and extend distributed compositional representations to weakly supervised, non-formal domains, we propose Soft TPR , which can be thought of as a continuous relaxation of the traditional TPR, as illlustrated by the translucent circular regions in Figure 1 c. At its core, the Soft TPR is designed to promote representational flexibility and ease of learning while simultaneously preserving the structural integrity of the traditional TPR. We additionally introduce Soft TPR Autoencoder , a theoretically-principled weakly-supervised architecture for learning Soft TPRs, used to operationalise the Soft TPR framework in the visual representation learning domain.

Our main contributions are threefold: i) We propose a novel compositional representation learning framework, introducing the distributed, flexible Soft TPR compositional form, alongside a dedicated, weakly-supervised architecture, Soft TPR Autoencoder , for learning this form. ii) Our framework is the first to learn distributed compositional representations in the non-formal, less explicitly algebraic domain of vision . iii) We empirically affirm the far-reaching benefits produced by the Soft TPR framework, demonstrating that Soft TPRs achieve state-of-the-art disentanglement, accelerate representation learner convergence, and provide downstream models with enhanced sample efficiency and superior low-sample regime performance.

2 Related Work

Disentanglement : In aiming to produce explicitly compositional representations without strong supervision, our work shares the same objective as disentangled representation learning. Prior to the highly influential work of [ 25 ] which proved the impossibility of learning disentangled representations without supervision or other inductive biases, disentangled representations were learnt in a completely unsupervised fashion [ 8 , 10 , 14 , 17 , 24 , 26 , 37 ]. Our use of weak supervision is inspired by the work [ 13 , 22 , 31 , 33 , 35 ] relating to this highly influential impossibility result. In particular, we leverage the type of weak supervision termed 'match pairing' [ 35 ], where pairs, ( x, x ′ ) , differing in values for a subset of known FoVs are presented to the model, to incentivise disentanglement. Our work,

however, fundamentally diverges from all disentanglement work we are aware of, by adopting an inherently distributed representation of compositional structure, which contrasts with the inherently symbolic , localist representations of compositional structure characterising existing work.

TPR-based Work : Existing TPR-based approaches generate distributed representations of compositional structure by producing an element with the explicit mathematical form of a TPR. To learn this highly specific form, these approaches rely on the algebraic characterisation of compositionality present in formal domains, such as mathematics [ 38 ], or language [ 19 , 23 , 28 , 34 , 52 ] in addition to strong supervision signals from highly structured downstream tasks, such as part-of-speech tagging [ 23 ], and answering structured language [ 28 , 51 ] or mathematics questions [ 38 ]. In contrast, Soft TPR eases these stringent constraints by offering a continuously relaxed specification of compositional structure. This allows our approach to extend distributed representations of compositional structure to the orthogonal and less algebraically structured domain of visual data, while also reducing reliance on annotated data by instead using weak supervision to learn this relaxed representational form.

3 Preliminaries

3.1 A Formal Framework for Compositional Representations

We adopt a generalised, non-generative version of the definition of compositional representations from [ 49 ]. Let data x ∈ X be compositionally-structured if there exists a decomposition function β : X → A 1 × . . . × A n decomposing x into constituent parts (FoVs), i.e. β ( x ) = { a 1 , . . . , a n } , where a i ∈ A i . A mapping ψ : X → V F then produces a compositional representation if ψ ( x ) = C ( ψ 1 ( a 1 ) , . . . , ψ n ( a n )) , where each ψ i : A i → V i is a component function that independently embeds a part a i (i.e., a particular FoV) into a vector space, and C : V 1 × . . . × V n → V F is a composition function that combines these embedded parts to form the overall representation. This construction enforces a faithful correspondence between the constituency structure of the data { a i } and that of the representation, { ψ i ( a i ) } (provided C is invertible).

We formalise a symbolic compositional representation, ψ s ( x ) , as one in which C is given by a concatenation operation. Concretely, ψ s ( x ) = ( ψ 1 ( a 1 ) T , . . . , ψ n ( a n ) T ) T . Indeed, the disentanglement approaches of [ 8 , 10 , 13 , 14 , 17 , 22 , 24 , 26 , 31 , 32 , 33 , 35 , 36 , 37 , 47 ] all fit this framework, concatenating scalar or vector-valued FoV tokens together to produce the representation. While this design may be fundamentally at odds with the continuous, distributed nature of deep learning (as detailed in Section 1 ), it provides one clear benefit: the embedded FoVs, { ψ i ( a i ) } are trivially recoverable by partitioning ψ s ( x ) .

In many scenarios, readily recovering the data constituents (i.e., FoVs), { a i } , from the compositional representation ψ ( x ) is essential for practical utility. Here, we assume each ψ i is invertible, so direct recoverability of the data constituents, { a i } , from the representation ψ ( x ) is guaranteed if the embedded FoVs, { ψ i ( a i ) } , remain structurally separable in the representation, ψ ( x ) . Motivated by this requirement, we aim to construct compositional representations that: 1) achieve an inherently distributed encoding of compositional structure by smoothly interweaving FoVs throughout all dimensions of ψ ( x ) , and 2) preserve the direct recoverability of the embedded FoVs.

3.2 The TPR Framework

Smolensky's Tensor Product Representations (TPR) [ 3 ] provides one compelling realisation of an inherently distributed encoding of compositional structure that, under certain conditions, preserves the direct recoverability of the embedded FoVs. We briefly review key aspects here, with additional details and formal proofs deferred to Appendix A . The TPR framework conceptualises a compositionallystructured object, x , as comprising a number, N R , of roles, where each role r ∈ R is bound to a corresponding filler f ∈ F , with this binding being denoted by f/r . Thus, compositionally-structured objects are decomposeable into a set of role-filler bindings, β ( x ) = { f i /r i } N R i =1 . In natural language, where the TPR is most commonly deployed [ 19 , 28 , 34 , 52 ], roles may correspond to grammatical categories (e.g., noun ) and fillers specific words (e.g., cat ). Translating this role-filler formalism to the visual domain, we reinterpret roles as FoV types (e.g., floor colour ), and fillers as FoV values (e.g., blue ). Using this role-filler formalism to decompose the Shapes3D image in Figure 1 , we have β ( x ) = { magenta / floor colour , orange / wall colour , aqua / object colour , large / object size , oblong / object shape } .

To construct a TPR, the roles and fillers for each binding in x are independently embedded via role and filler embedding functions, ξ R : R → V R , ξ F : F → V F respectively. The binding f/r is then encoded by taking a tensor product (denoted by ⊗ ) over the embedded role, ξ R ( r ) , and embedded filler, ξ F ( f ) . Summing over the embeddings for all role-filler bindings in x yields the TPR, ψ tpr ( x ) :

where m : { 1 , . . . , N R } → { 1 , . . . , N F } is a matching function 2 associating each of the N R roles to the filler it binds to in the decomposition of x (i.e., β ( x ) = { f m ( i ) /r i } ).

We now situate the TPR within the formal framework of Section 3.1, by observing that it defines each representational component, ψ i ( a i ) as an embedded role-filler binding, ξ F ( f m ( i ) ) ⊗ ξ R ( r i ) , and adopts ordinary vector space addition as its composition function, C . Unlike the concatenation in symbolic, localist schemes, the TPR additively superposes all embedded FoVs in the same underlying vector space, producing an inherently distributed representation of compositional structure in which information from multiple role-filler bindings can be smoothly interwoven into the same dimensions. 3

A natural question arises from defining C as ordinary vector space addition: can we recover each representational component - the embedded role-filler binding, ξ i ( f m ( i ) ) ⊗ ξ R ( r i ) - from the TPR, which is their sum? Remarkably, despite the non-injectivity of the summation operation, the TPR's specific algebraic structure enables faithful recoverability of all representational components from the TPR, through a process referred to as unbinding (see A.2 for more details). More concretely, provided that the role embedding vectors { ξ R ( r i ) } are linearly independent and known, the embedded filler bound to the i -th role can be unbound from the representation, ψ tpr ( x ) , by taking a (tensor) inner product between ψ tpr ( x ) and the i -th unbinding vector, u i :

where u i is a vector corresponding to the i -th column of the (left) inverse of the matrix obtained by taking all (linearly independent) role embeddings as columns. Repeating this unbinding procedure using each of the N R unbinding vectors recovers each of the embedded fillers bound to the N R roles, and thus, the entire set of embedded role filler bindings, { ξ F ( f m ( i ) ) ⊗ ξ R ( r i ) } . Hence, provided that the linear independence of the roles holds, the TPR offers an inherently distributed representation of compositional structure that nonetheless retains the chief benefit of symbolic approaches - direct recoverability of each representational component.

4 Methods

4.1 Soft TPR: An Extension to the TPR Framework

While Tensor Product Representations offer a robust framework for encoding compositional structure in a distributed manner, their rigid representational form imposes practical limitations. The core limitation lies in TPR's strict algebraic specification of compositional structure as a set of discrete bindings, each of which comprise a single role and a single filler. This specification precludes the representation of more ambiguous, quasi-compositional structures that cannot be neatly decomposed in this way. For instance, even in the highly compositional domain of natural language, there exist phenomena such as idiomatic expressions that resist straightforward decomposition into role-filler pairs because their meanings cannot be directly inferred from their composite parts. Such examples illustrate that real-world compositional structures may require more flexible representation than what is offered by traditional TPRs. Consequently, TPRs may struggle to model more nuanced forms of compositional structure that characterise less structured, non-formal data domains, such as vision. Additionally, TPRs present a challenging learning task. Their representational form ψ tpr ( x ) := ∑ i ξ F ( f m ( i ) ) ⊗ ξ R ( r i ) is only satisfied by a discrete subset of points within the underlying representational space, V F ⊗ V R , as illustrated in Figure 1 c. This constraint forces representation learners to map data to a highly restricted, discrete set of points, making learning inherently arduous.

To overcome these shortcomings, we introduce Soft TPR , a continuous relaxation of the traditional TPR specification. Soft TPRs allow any point in the underlying representational space V F ⊗ V R that approximately conforms to the TPR specification. This removes the need for representations to perfectly adhere to the rigid structural specification of the TPR, instead permitting them to flexibly approximate this rigid form. This flexibility provides two main advantages: 1. Enhanced Representational Flexibility : Soft TPR facilitates the representation of quasi -compositional structures that do not perfectly adhere to the traditional TPR's structural specification. This includes scenarios where bindings are approximate rather than exact, or where multiple fillers influence a single role. 2. Improved Ease of Learning : The relaxed representational specification allows representation learners to map data to broader regions of the representational space (as illustrated by the translucent circular regions in Figure 1 c). This may facilitate more efficient learning, as there are more possible mappings from the data to the required representational form.

Formally, to define the Soft TPR, we consider the vector space underlying TPRs, V F ⊗ V F , produced by an arbitrary role embedding function ξ R : R → V R and an arbitrary filler embedding function ξ F : F → V F . A Soft TPR is defined as any element z ∈ V F ⊗ V R in this underlying representational space that is sufficiently close to a traditional TPR, ψ tpr 4 , as measured by a chosen distance metric (we take the Frobenius norm). Denoting the Frobenius norm by ||·|| F , Soft TPR satisfies || z -ψ tpr || F < ϵ , where ϵ is some small, scalar-valued positive quantity. This condition ensures that the Soft TPR approximately encodes the TPR-based compositional structure of ψ tpr , while allowing for slight deviations which capture a more flexible, relaxed notion of compositionality that cannot be reduced to a set of role-filler bindings (see B.2 ).

Despite this relaxation, Soft TPRs retain the critical advantage of traditional TPRs: the ability to recover constituent role-filler bindings through the unbinding operation, albeit approximately. Specifically, when unbinding a Soft TPR z , the operation yields soft filler embeddings ˜ f i that closely approximate the true filler embeddings ξ F ( f m ( i ) ) . Formally, for a Soft TPR z , performing the unbinding operation for the i -th role yields:

where ϵ i = zu i -ξ F ( f m ( i ) ) represents the approximation error. This approximate recoverability ensures that Soft TPRs implicitly encode a precise, algebraically-expressible form of compositional structure, even when their representational form deviates from the exact TPR specification.

4.2 Soft TPR Autoencoder: Learning Distributed and Flexible Compositional Representations

We define our vector spaces of interest over the reals as V F := R D F and V R := R D R where D F , D R denote the dimensionality of the filler and role embedding spaces. The pivotal insight underlying our method is that a Soft TPR can be effectively realised by ensuring that the encoder produces representations that are amenable to quantisation into explicit TPRs. Specifically, since a Soft TPR is any arbitrary element from the vector space R D F · D R 5 sufficiently close to an explicit TPR, any ( D F · D R ) -dimensional vector produced by an encoder in a standard autoencoding framework can be treated as a Soft TPR candidate . This realisation suggests that a conventional autoencoding framework only requires modest modifications to generate Soft TPRs.

Our Soft TPR Autoencoder comprises three main components: a standard encoder, E , the TPR decoder, and a standard decoder, D . The encoder output, z , serves as the Soft TPR. The overarching intuition guiding our architecture is to ensure that z can be effectively quantised or decoded into an explicit TPR, thereby enforcing the Soft TPR property, || z -ψ tpr || 2 < ϵ .

Representational Form : To ensure that the encoder produces elements with the form of a Soft TPRs, we introduce a mechanism that penalises the Euclidean distance || z -ψ ∗ tpr || 2 between the encoder

Encoder

(E)

Quantisation

Module

Unbound 'soft' fillers

Unbinding

Module

TPR Decoder

TPR

Constructor

Vector

Quantisation

TPR

operator

Decoder

(D)

Quantised fillers

Unbinding

operator

Filler embedding matrix

Role embedding matrix

output, z , and some explicit TPR, ψ ∗ tpr , that z best approximates. To obtain ψ ∗ tpr needed to compute the loss, we derive an explicit analytical form for ψ ∗ tpr and construct elements satisfying this analytical form using a role embedding matrix, M ξ R containing N R D R -dimensional role embedding vectors, { ξ R ( r i ) } , and a filler embedding matrix, M ξ F containing N F D F -dimensional filler embedding vectors, { ξ F ( f i ) } . To define an explicit analytical form for ψ ∗ tpr , we use a greedily optimal selection of filler embeddings based on their proximity to the soft filler embeddings extracted from z :

That is, we define ψ ∗ tpr as the TPR constructed from explicit filler embeddings ξ F ( f j ) with the smallest Euclidean distance to the soft filler embeddings ˜ f k of z . To construct elements satisfying (5), we use a three-step process carried out by the novel TPR decoder we introduce, visible in Figure 2 : 1) Unbinding : Utilising a randomly-initialised dense role embedding matrix M ξ R , the unbinding module extracts soft filler embeddings { ˜ f i } by performing the TPR unbinding operation on z . We elaborate on our theoretically-informed reason for this design, how the unbinding vectors are obtained, and for not backpropagating gradient to M ξ R in B.4.1 . 2) Quantisation : The quantisation module, containing a learnable filler embedding matrix M ξ F , employs the VQ-VAE vector quantisation algorithm [ 11 ] to both 1) learn the explicit filler embeddings, and 2) quantise the soft filler embeddings { ˜ f 1 , . . . , ˜ f N R } produced by the unbinding module into the explicit filler embeddings { ξ F ( f 1 ) , . . . , ξ F ( f N R ) } with the smallest Euclidean distances. 3) TPR Construction : The quantised filler embeddings { ξ F ( f m ( i ) ) } are then bound to their respective roles using the tensor product, and all embedded bindings are summed together to produce the explicit TPR ψ ∗ tpr with the form of Eq 5 . ψ ∗ tpr is subsequently used by the decoder, D , to reconstruct the input image. The overall unsupervised loss, L u , comprises three components, where s denotes the stop-gradient operator, β is a hyperparameter controlling the commitment loss in VQ-VAE, and L r is a suitable image-based reconstruction loss (we use L 2 ):

̸

Representational Content : While the unsupervised loss L u ensures that z maintains the Soft TPR form by being close to an explicit TPR, it does not guarantee that the content of z accurately reflects the true role-filler semantics of the data. To address this, we introduce a weakly supervised loss that aligns z with the ground-truth semantics of the image. We employ a match-pairing context similar to [ 13 , 22 , 33 , 35 ], where image pairs ( x, x ′ ) share the same role-filler bindings for all but one role, r i . The identity of r i is known, but not any of the specific fillers or bindings. Our intuition is that, for z to accurately reflect the semantics of the image, the Euclidean distance between the quantised fillers of x and x ′ bound to role r i should be maximal, relative to the distances between the filler embeddings pairs for all other roles r j , j = i . To encourage this, we apply the cross entropy loss corresponding to the 3rd term in Eq 7 , where ∆ q denotes the N R -dimensional vector with each dimension (∆ q ) k

populated by the Euclidean distance between the quantised fillers of x and x ′ for role r k , and l denotes the one-hot vector of dimension N R with the index for r i set to 1. Additionally, like [ 47 ], we incorporate a reconstruction loss (the 2nd term of Eq 7 ) to enforce consistency when swapping the quantised filler embeddings for role r i . Specifically, we construct new TPRs by swapping the quantised fillers for r i between x and x ′ to generate ψ s tpr ( x ) and ψ s tpr ( x ′ ) , and ensure that the decoder can accurately reconstruct the corresponding swapped images from these swapped TPRs.

Combining these components, our final loss, L , is a weighted sum over the unsupervised and weakly supervised loss components, where λ 1 and λ 2 are hyperparameters:

5 Results

To evaluate our Soft TPR framework, we perform evaluation along three dimensions that are standard in the compositional representation learning literature [ 25 , 30 , 33 , 42 ]: 1) Compositional Structure / Disentanglement : To what extent do Soft TPR representations capture explicitly compositional structure ? 2) Representation Learner Convergence : How efficiently can representation learners acquire Soft TPR's distributed, flexible compositional structure? 3) Downstream Model Performance : Does Soft TPR's distributed, flexible compositional form offer tangible benefits for downstream models utilising compositional representations?

We benchmark against a suite of weakly supervised disentanglement baselines: Ada-GVAE [ 33 ], GVAE [ 22 ], ML-VAE [ 13 ], SlowVAE [ 39 ], and the GAN-based model of [ 35 ], which we henceforth refer to as 'Shu'. These models produce symbolic compositional representations corresponding to a concatenation of scalar-valued FoV tokens. Similar to our approach, Ada-GVAE, GVAE, ML-VAE, SlowVAE, and Shu are trained with paired samples ( x, x ′ ) sharing values for all but a subset of FoVs types (roles), I , with ML-VAE, GVAE, SlowVAE, and Shu assuming access to I , matching our model's level of supervision. To ensure a fair comparison, we modify Ada-GVAE (method detailed in Appendix C.2.2 ) for more direct comparability, denoting our modification by Ada-GVAE-k. Additionally, we benchmark against 2 baselines producing symbolic vector-tokened compositional representations: COMET [ 36 ], and Visual Concept Tokeniser (VCT) [ 47 ], although these are fully unsupervised and not directly comparable to our method. We train five instances of each representation learning model using five random seeds for 200,000 iterations across all datasets, and report averaged results (an unabridged suite of all results is contained in the Appendix).

5.1 Compositional Structure / Disentanglement

To demonstrate that Soft TPRs inherently capture the algebraic compositional structure required for effective disentanglement, we first quantise each Soft TPR, z , into its corresponding explicit TPR, ψ ∗ tpr using the TPR decoder (note that once the model is trained, quantisation is fully deterministic). This quantisation process transforms the implicit compositional information encoded within Soft TPRs into an explicit algebraic form suitable for evaluation. We then apply standard disentanglement metrics across benchmark datasets including Cars3D [ 4 ], MPI3D [ 21 ], and Shapes3D [ 24 ]. Table 1 illustrates that Soft TPR consistently outperforms symbolic compositional baselines across all datasets, achieving state-of-the-art results with notable DCI metric improvements of 29% on Cars3D and 74% on the more challenging MPI3D dataset. To ensure these gains are attributable to Soft TPR framework rather than merely a marginal increase in model capacity (Soft TPR Autoencoder adds between 1,600-13,568 model parameters to a standard (variational) autoencoder), we conduct control experiments by equalising parameter counts in applicable baselines. Consistent with findings from prior work [ 40 ], we observe no performance improvements in relevant symbolic baselines when parameter counts are increased (see C.2.4 ).

5.2 Representation Learner Convergence Rate

To explore how the Soft TPR framework influences the convergence rate of representation learners, we analyse representations produced at varying stages of representation learner training (100, 1,000, 10,000, 100,000 and 200,000 iterations), and evaluate both their 1) Disentanglement , and 2) Downstream Utility . We quantify downstream utility of learned representations by examining the

performance of downstream models on two tasks common in the disentanglement literature [ 25 , 30 , 33 , 42 ]: a classification-based task assessing the model's ability to perform abstract visual reasoning [ 30 ] and a regression task involving the prediction of continuous FoV values for the disentanglement datasets [ 42 ]. Our findings indicate that the Soft TPR framework generally achieves faster disentanglement convergence, particularly on Cars3D and MPI3D datasets (see Appendix C.4.1 ). Moreover, Soft TPR consistently accelerates the learning of useful representations for both downstream tasks. The downstream performance improvements are particularly pronounced in the low iteration regime for instance, at only 100 iterations of representation learner training, as shown in Tables 2 and 3 . For fair comparison, we embed baseline representations of both higher and lower dimensionality into the same vector space as Soft TPR, which we denote by † , and select the best-performing variant (original or dimensionality-matched) for each baseline (details in Appendix C ).

Ours

0.999

±

0.001

0.863

±

0.027

0.984

±

0.012

0.926

±

0.028

0.949

±

0.032

0.828

±

0.015

Ours

0.531

±

0.054

0.981

5.3 Downstream Models

To evaluate whether the distributed and flexible encoding of compositional structure offered by the Soft TPR benefits downstream models, we examine both: 1) Sample Efficiency and 2) Raw Performance in Low Sample Regimes , using the previously mentioned abstract visual reasoning and FoV regression tasks. In line with [ 25 ], we quantify sample efficiency with a ratio-based metric comparing downstream model performance using restricted sample sizes (100, 250, 500, 1,000, and 10,000) against performance when trained using all samples. As illustrated in Table 4 , Soft TPR substantially outperforms baseline models in sample efficiency, especially in the most restrictive case involving only 100 samples, achieving a 93% improvement. Additionally, Soft TPR produces substantial raw performance increases in the low sample regime, as evidenced by the 138% and 168% improvements in Table 4 for the low-sample regimes of 100 and 200 samples respectively, and the 30% improvement in Table 5 .

5.4 Ablation Studies

To disentangle the contributions of Soft TPR's foundational components, we conduct ablation studies focusing on two critical properties that are hypothesised to influence the Soft TPR's learning of compositional representations: 1) Relaxed Representational Constraints and 2) Distributed Encoding , denoted RRC and DE respectively in Table 7 . For Relaxed Representational Constraints , we significantly increase the weighting of the form penalty || z -ψ ∗ tpr || 2 in the Soft TPR loss to enforce rigid representational constraints on the Soft TPR Autoencoder, effectively forcing it to

±

0.003

0.732

±

0.012

Ours

0.490

±

0.068

0.556

±

0.078

0.594

±

0.056

0.665

±

0.067

Ours

0.360

±

0.033

produce representations that precisely match the TPR-specified compositional form. For Distributed Encoding , we modify the model to encode compositional structure in an inherently less distributed manner, by modifying the sparsity of ψ ∗ tpr (see C.2.4 ). Results in Table 7 demonstrate that both the Soft TPR's distributed encoding and relaxed constraints are essential for learning representations with high compositional structure, as quantified by the disentanglement metrics. Additionally, we examine the impact of Soft TPR's flexible, quasi-compositional form on downstream performance by replacing each Soft TPR, z , with its quantised explicit TPR, ψ ∗ tpr in downstream tasks. Table 6 and Appendix C.6.3 indicates that Soft TPRs provide downstream models with unique performance improvements that explicit TPRs (which are produced under the same conditions) cannot account for. Please see Appendix C.6 for further details and full ablation results, including additional ablations.

5.5 Key Insights

The empirical results of the Soft TPR framework reveal two primary insights. Firstly, deep learning models may more effectively learn precise compositional structures when these structures are 1) implicitly acquired through relaxed representational constraints , allowing the model to acquire precise compositional forms (i.e., ψ ∗ tpr ) through quantisation of flexible Soft TPRs, z , and 2) when the compositional structure is assumed to be encoded in a distributed format. Secondly, the flexible representational form of Soft TPR offers downstream models unique advantages by enabling more efficient utilisation of implicitly encoded compositional information compared to their rigid counterparts, ψ ∗ tpr . Together, these comprehensive model benefits empiricially demonstrate the value of the framework's 1) relaxed representational constraints, 2) distributed encoding of compositional structure, and 3) flexible representational form.

6 Conclusion

In this work, we tackle a challenge tracing its roots to the conception of the connectionist vs. classicalist debate: the fundamental mismatch between compositional structure and the inherently distributed nature of deep learning. To bridge this gap, we introduce the Soft TPR , a new, inherently distributed , flexible compositional representational form extending Smolensky's Tensor Product Representation, together with the Soft TPR Autocoder , a theoretically-principled architecture designed for learning Soft TPRs. Our flexible , distributed framework demonstrates substantial improvements in the visual representation learning domain - enhancing the representation of compositional structure, accelerating convergence in representation learners, and boosting efficiency in downstream models. Future work will extend this framework to hierarchical compositional structures, enabling bound fillers to recursively decompose into role-filler bindings for enhanced representational expressivity.

References

Noam Chomsky. Syntactic Structures . The Hague: Mouton, 1957.

Jerry A. Fodor. The Language of Thought: A Theory of Mental Representation . Cambridge, MA: Harvard University Press, 1975.

Paul Smolensky. 'Tensor product variable binding and the representation of symbolic structures in connectionist systems'. In: Artificial Intelligence 46.1 (1990), pp. 159-216.

Sanja Fidler, Sven Dickinson, and Raquel Urtasun. '3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model'. In: Advances in Neural Information Processing Systems . 2012.

Yoshua Bengio. Deep Learning of Representations: Looking Forward . 2013. arXiv: 1305 . 0445 [cs.LG] .

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 'Neural Module Networks'. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015), pp. 39-48. URL: https://api.semanticscholar.org/CorpusID:5276660 .

Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks . 2015. arXiv: 1502.05698 [cs.AI] . URL: https://arxiv.org/abs/1502.05698 .

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 'InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets'. In: Advances in Neural Information Processing Systems . Ed. by D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett. Vol. 29. Curran Associates, Inc., 2016.

Paul Smolensky and Matthew A. Goldrick. 'Gradient Symbolic Representations in Grammar: The case of French Liaison'. In: 2016. URL: https://api.semanticscholar.org/CorpusID:36953611 .

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 'beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework'. In: International Conference on Learning Representations . 2017.

Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu koray. 'Neural Discrete Representation Learning'. In: Advances in Neural Information Processing Systems . Ed. by I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. Vol. 30. Curran Associates, Inc., 2017.

Tameem Adel, Zoubin Ghahramani, and Adrian Weller. 'Discovering Interpretable Representations for Both Deep Generative and Discriminative Models'. In: Proceedings of the 35th International Conference on Machine Learning . Vol. 80. Proceedings of Machine Learning Research. PMLR, 2018, pp. 50-59.

Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. 'Multi-Level Variational Autoencoder: Learning Disentangled Representations From Grouped Observations'. In: Proceedings of the AAAI Conference on Artificial Intelligence 32.1 (2018).

Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in β -VAE . 2018. arXiv: 1804.03599 [stat.ML] .

Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud. 'Isolating Sources of Disentanglement in Variational Autoencoders'. In: Advances in Neural Information Processing Systems . 2018.

Cian Eastwood and Christopher K. I. Williams. 'A Framework for the Quantitative Evaluation of Disentangled Representations'. In: International Conference on Learning Representations . 2018.

Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. 'Variational Inference of Disentangled Latent Concepts from Unlabeled Observations'. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . 2018.

Adam Santoro, Felix Hill, David Barrett, Ari Morcos, and Timothy Lillicrap. 'Measuring abstract reasoning in neural networks'. In: International conference on machine learning . 2018, pp. 4477-4486.

Kezhen Chen, Qiuyuan Huang, Hamid Palangi, Paul Smolensky, Kenneth D. Forbus, and Jianfeng Gao. 'Natural- to formal-language generation using Tensor Product Representations'. In: CoRR abs/1910.02339 (2019). arXiv: 1910.02339 . URL: http://arxiv.org/abs/1910.02339 .

Elliot Creager, David Madras, Joern-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. 'Flexibly Fair Representation Learning by Disentanglement'. In: Proceedings of the 36th International Conference on Machine Learning . 2019.

Muhammad Waleed Gondal, Manuel Wüthrich, undefinedor ¯ de Miladinovi´ c, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. 'On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset'. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems . 2019.

Haruo Hosoya. 'Group-based learning of disentangled representations with generalizability for novel contents'. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence . 2019, pp. 2506-2513.

Qiuyuan Huang, Li Deng, Dapeng Wu, Chang Liu, and Xiaodong He. 'Attentive Tensor Product Learning'. In: Proceedings of the AAAI Conference on Artificial Intelligence 33.01 (2019), pp. 13441351.

Hyunjik Kim and Andriy Mnih. Disentangling by Factorising . 2019. arXiv: 1802.05983 [stat.ML] .

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. 'Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations'. In: Proceedings of the 36th International Conference on Machine Learning . Vol. 97. Proceedings of Machine Learning Research. PMLR, 2019, pp. 4114-4124.

Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. 'Disentangling Disentanglement in Variational Autoencoders'. In: Proceedings of the 36th International Conference on Machine Learning . 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 'PyTorch: An Imperative Style, High-Performance Deep Learning Library'. In: Advances in Neural Information Processing Systems 32 . Curran Associates, Inc., 2019, pp. 80248035. URL: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-highperformance-deep-learning-library.pdf .

Imanol Schlag and Jürgen Schmidhuber. 'Learning to Reason with Third-Order Tensor Products'. In: Advances in Neural Processing Information Systems . 2019.

Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao. 'Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving'. In: ArXiv abs/1910.06611 (2019). URL: https://api.semanticscholar.org/CorpusID:204575948 .

Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. 'Are disentangled representations helpful for abstract visual reasoning?' In: Proceedings of the 33rd International Conference on Neural Information Processing Systems . 2019.

Junxiang Chen and Kayhan Batmanghelich. 'Weakly Supervised Disentanglement by Pairwise Similarities'. In: Proceedings of the AAAI Conference on Artificial Intelligence 34.04 (2020), pp. 34953502.

Zheng Ding, Yifan Xu, Weijian Xu, Gaurav Parmar, Yang Yang, Max Welling, and Zhuowen Tu. 'Guided Variational Autoencoder for Disentanglement Learning'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2020.

F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. 'Weakly-Supervised Disentanglement Without Compromises'. In: Proceedings of the 37th International Conference on Machine Learning (ICML) . V ol. 119. Proceedings of Machine Learning Research. PMLR, 2020, pp. 63486359.

R. Thomas McCoy, Tal Linzen, Ewan Dunbar, and Paul Smolensky. 'Tensor Product Decomposition Networks: Uncovering Representations of Structure Learned by Neural Networks'. In: Proceedings of the Society for Computation in Linguistics 2020 . Association for Computational Linguistics, 2020, pp. 277-278. URL: https://aclanthology.org/2020.scil-1.34 .

Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. 'Weakly Supervised Disentanglement with Guarantees'. In: International Conference on Learning Representations . 2020.

Yilun Du, Shuang Li, Yash Sharma, B. Joshua Tenenbaum, and Igor Mordatch. 'Unsupervised Learning of Compositional Energy Concepts'. In: Advances in Neural Information Processing Systems . 2021.

Insu Jeon, Wonkwang Lee, Myeongjang Pyeon, and Gunhee Kim. 'IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks'. In: Proceedings of the AAAI Conference on Artificial Intelligence 35.9 (May 2021), pp. 7926-7934. DOI: 10.1609/aaai. v35i9.16967 . URL: https://ojs.aaai.org/index.php/AAAI/article/view/16967 .

Yichen Jiang, Asli Celikyilmaz, Paul Smolensky, Paul Soulos, Sudha Rao, Hamid Palangi, Roland Fernandez, Caitlin Smith, Mohit Bansal, and Jianfeng Gao. 'Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization'. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Online: Association for Computational Linguistics, 2021, pp. 4780-4793. DOI: 10.18653/ v1/2021.naacl-main.381 . URL: https://aclanthology.org/2021.naacl-main.381 .

David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. 'Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding'. In: International Conference on Learning Representations . 2021. URL: https://openreview.net/ forum?id=EbIDjBynYJ8 .

Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. 'The role of Disentanglement in Generalisation'. In: International Conference on Learning Representations . 2021. URL: https://openreview.net/forum?id=qbH974jKUVy .

Sungho Park, Sunhee Hwang, Dohyung Kim, and Hyeran Byun. 'Learning Disentangled Representation for Fair Facial Attribute Classification via Fairness-aware Information Alignment'. In: Proceedings of the AAAI Conference on Artificial Intelligence 35 (2021), pp. 2403-2411. DOI: 10.1609/aaai.v35i3. 16341 . URL: https://ojs.aaai.org/index.php/AAAI/article/view/16341 .

L. Schott, J. von Kügelgen, F. Träuble, P. Gehler, C. Russell, M. Bethge, B. Schölkopf, F. Locatello, and W. Brendel. 'Visual Representation Learning Does Not Generalize Strongly Within the Same Domain'. In: ICLR 2021 - Workshop on Generalization beyond the training distribution in brains and machines . 2021.

Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. 'On Disentangled Representations Learned from Correlated Data'. In: Proceedings of the 38th International Conference on Machine Learning . Vol. 139. Proceedings of Machine Learning Research. 2021, pp. 10401-10412.

Milton Montero, Jeffrey Bowers, Rui Ponte Costa, Casimir Ludwig, and Gaurav Malhotra. 'Lost in Latent Space: Examining failures of disentangled models at combinatorial generalisation'. In: Advances in Neural Information Processing Systems . Ed. by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh. Vol. 35. Curran Associates, Inc., 2022, pp. 10136-10149.

Zoltán Gendler Szabó. 'Compositionality'. In: The Stanford Encyclopedia of Philosophy . Ed. by Edward N. Zalta and Uri Nodelman. Fall 2022. Metaphysics Research Lab, Stanford University, 2022.

Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Nanning Zheng. 'Towards Building A Group-based Unsupervised Representation Disentanglement Framework'. In: International Conference on Learning Representations . 2022. URL: https://openreview.net/forum?id=YgPqNctmyd .

Tao Yang, Yuwang Wang, Yan Lu, and Nanning Zheng. 'Visual Concepts Tokenization'. In: Advances in Neural Information Processing Systems . 2022.

H. Zhang, Y.-F. Zhang, W. Liu, A. Weller, B. Schölkopf, and E. Xing. 'Towards Principled Disentanglement for Domain Generalization'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2022, pp. 8024-8034. URL: https://openaccess.thecvf.com/ content/CVPR2022/papers/Zhang_Towards_Principled_Disentanglement_for_Domain_ Generalization_CVPR_2022_paper.pdf .

Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. 'Compositional Generalization from First Principles'. In: Thirty-seventh Conference on Neural Information Processing Systems . 2023. URL: https://openreview.net/forum?id=LqOQ1uJmSx .

Haoyang Li, Xin Wang, Zeyang Zhang, Haibo Chen, Ziwei Zhang, and Wenwu Zhu. 'Disentangled Graph Self-supervised Learning for Out-of-Distribution Generalization'. In: Forty-first International Conference on Machine Learning . 2024. URL: https://openreview.net/forum?id=OS0szhkPmF .

Taewon Park, Inchul Choi, and Minho Lee. 'Attention-based Iterative Decomposition for Tensor Product Representation'. In: The Twelfth International Conference on Learning Representations . 2024. URL: https://openreview.net/forum?id=FDb2JQZsFH .

Qiuyuan Huang, Paul Smolensky, Xiaodong He, Li Deng, and Dapeng Wu. 'Tensor Product Generation Networks for Deep NLP Modeling'. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) . Association for Computational Linguistics.

Appendix

A TPR Framework

In this section, we provide additional details regarding Smolensky's TPR framework [ 3 ], as well as formal proofs of presented results. We refer interested readers to [ 3 ] for a more comprehensive dive into the TPR framework.

A.1 Additional Details

By defining the constituent components of any compositional object as a set of role-filler bindings, the TPR defines the decomposition function, β , of Section 3.1 that maps from a set of compositional objects, X , to a set of parts, more explicitly as follows [ 3 ]:

where F denotes a set of fillers, R denotes a set of roles, and f/r denotes the binding of filler f ∈ F to role r ∈ R . Note that in contrast to the formal definition of β we use in Section 3.1 , which assumes each x ∈ X is decomposable into a set of n parts, the above decomposition allows objects to be decomposed into a variably-sized set of role-filler bindings, with this set corresponding to an element in the powerset of F × R . For the considered visual representation learning domain, all datasets clearly have the property of being decomposable into a fixed size set of role-filler bindings, as all images in these datasets contain the same number of FoV types and each FoV type is bound to a FoV token . Due to this property, we can take β as a special subcase of the generalised definition in Equation 8 :

where m : { 1 , . . . , N R } → { 1 , . . . , N F } denotes a matching function that associates each role r i with the filler it binds to in β ( x ) (we again drop the dependence of m on x for ease of notation), and A denotes the set of all possible bindings produced by binding a filler to each of the N R roles, with size N N R F (we assume the same filler can bind to multiple roles).

A.2 Formal Proofs

We now formally prove that the TPR with β defined in 9 has form ψ tpr ( x ) = C ( ψ 1 ( a 1 ) , . . . , ψ n ( a n )) and thus corresponds to the definition of a compositional representation in Section 3.1 .

Proof. We denote the role embedding and filler embedding functions as ξ R : R → V R , ξ F : F → V F respectively.

By definition of the TPR in Eq 1 , we have that:

where a i := ( f m ( i ) , r i ) ∈ β ( x ) , ψ i : F × R → V F ⊗ V R ; ( f m ( i ) , r i ) → ξ F ( f m ( i ) ) ⊗ ξ R ( r i ) , and C is ordinary vector space addition. Hence, almost trivially, ψ tpr ( x ) clearly has the required form to be a compositional representation .

Now, we prove the recoverability of the embedded components { ψ i ( f m ( i ) , r i ) } from the TPR, ψ tpr ( x ) , provided that the set of all role embedding vectors, { ξ R ( r i ) } , are linearly independent. Similar variants of this proof can be found in [ 3 ], [ 19 ].

Proof. Assume the set of all role embedding vectors { ξ R ( r i ) } are linearly independent. Then, the role embedding matrix, M ξ R := ( ξ R ( r 1 ) . . . ξ R ( r N R )) formed by taking the role embedding vectors as columns, has a left inverse, U , such that:

Hence, we have that ( UM ξ R ) ij = U i : M ξ R : j = I ij .

For ease of notation, let u i denote the i -th column of U T , and note that ξ R ( r j ) clearly corresponds to M ξ R : j . So, U i : M ξ R : j = ( U T : i ) T M ξ R : j = u T i ξ R ( r j ) = I ij .

Hence, we have that:

and hence,

Using the definition of ψ tpr ( x ) , ψ tpr ( x ) = ∑ i ξ F ( f m ( i ) ) ⊗ ξ R ( r i ) , we apply the (tensor) inner product of ψ tpr ( x ) with u i :

Thus, the embedding of the filler, f m ( i ) , bound to each role, r i , can be recovered through use of a tensor inner product with the unbinding vector, u i , corresponding to the i -th column of U T . Note that the representational components of ψ tpr ( x ) , i.e., the embedded bindings, ψ i ( f m ( i ) , r i ) are fully determined by the embedding of the role, ξ R ( r i ) , and filler, ξ F ( f m ( i ) ) , comprising the binding, as ξ F ( f m ( i ) , r i ) simply corresponds to their tensor product. Thus, recovering ( ξ ( f m ( i ) ) , ξ R ( r i )) for each binding in β ( x ) corresponds to recovering the representational component, ψ i ( f m ( i ) , r i ) . So, provided the set of role embeddings are linearly independent, and they can be obtained (e.g. through a look-up table of role embeddings), all representational components, ψ i ( f m ( i ) , r i ) can be directly recovered from the overall TPR representation, ψ tpr ( x ) .

A.3 Shortcomings of Symbolic Compositional Representations and How TPR Helps

Here, we provide a more detailed elaboration on the shortcomings of symbolic compositional representations as outlined in 1 . We additionally illustrate how TPR-based continuous compositional representations circumvent such limitations through use of a concrete example.

We employ the formal definition of a symbolic compositional representation from Section 3.1 , which states a representation, ψ s ( x ) , is a symbolic compositional representation if ψ s ( x ) = ( ψ 1 ( a 1 ) T , ..., ψ n ( a n ) T ) T , i.e., if ψ s ( x ) corresponds to a concatenation of representational components, { ψ i ( a i ) } .

For more direct comparison between the symbolic compositional representation, ψ s ( x ) and the TPR-based compositional representation, ψ tpr ( x ) , we rewrite the TPR-based representation as ψ tpr ( x ) = ˜ ψ (˜ a 1 ) + . . . + ˜ ψ n (˜ a n ) . Thus, the TPR can be viewed as an ordinary vector sum of components, { ˜ ψ (˜ a i ) } , where each component, ˜ ψ i corresponds to an embedded role-filler binding.

We consider the following example: suppose there are 2 FoV types (i.e., roles), colour , shape and 2 FoV values (i.e., fillers), purple , square . For the symbolic approach, suppose the representational components , { ψ i ( a i ) } are defined as follows: ψ colour (purple) = [ 1 0 ] and ψ shape (square) = [ 2 3 ] . For the distributed approach, suppose the (linearly independent) role embeddings are: ξ R (colour) = [ 1 2 ] , ξ R (shape) = [ 1 1 ] , and the filler embeddings are: ξ F (purple) = [ 1 1 ] and ξ F (square) = [ 2 3 ] .

Then, a symbolic compositional representation is given by (denoting the concatenation operation by ⊕ ):

On the other hand, the distributed, TPR-based compositional representation is given by:

Note that both the symbolic and distributed, TPR-based representations share the same underlying representational space, i.e., R 4 .

Gradient Flow and Learning 6 : The symbolic compositional representation allocates the FoVs to 2 distinct, independent slots of ψ d ( x ) . This creates discrete boundaries between the first two and last two dimensions of the representation, which may potentially complicate gradient-based optimisation for the following reasons:

(a) Discrete Boundaries and Restricted Gradient Flow : In the symbolic compositional representation, each FoV is confined to specific, non-overlapping dimensions of the representation. This type of assignment means that updates to a particular FoV requires restricting gradient flow exclusively to the dimensions associated with its designated slot. For example, updating the FoV for square / shape requires modifying the last two dimensions (i.e., permitting gradient flow through these two dimensions), while keeping the first two dimensions unchanged (i.e., completely restricting gradient flow through these two dimensions). This constrains gradient flow across all dimensions of representation. Mathematically, the gradient update for the above example (modifying square / shape ) can be expressed in the symbolic case as:

which categorically restricts gradient flow dimensionwise across the underlying representation space, R 4 .

(b) Abrupt, Discontinuous Shifts in the Representational Space : When multiple FoVs are updated consecutively, the symbolic, slot-based representational form may induce sudden and discontinuous changes in the representational space. For instance, updating purple / colour at time t and then shape / square at t +1 produces:

These abrupt shifts in the representation space may contribute to unstable and fragmented learning dynamics, potentially hindering the convergence of optimisation algorithms.

In contrast, the continuous, distributed , TPR-based compositional representation integrates the FoVs continuously into all dimensions of the representation (i.e., ˜ ψ (purple / colour) = (1 1 2 2) T and ˜ ψ (square / shape) = (2 3 2 3) T are summed together in R 4 ), eliminating the necessary existence of dimension-wise slots with non-differentiable boundaries for each FoV. This alleviates the previously mentioned problems:

(a) Smoother Gradient Flow : The inherently distributed structure of the TPR allows gradients to flow freely across all dimensions of the representational space, R 4 . Updating the FoV for square / shape involves modifying all relevant dimensions simultaneously, avoiding categorical dimension-wise restrictions and potentially allowing for more flexible and effective gradient-based optimisation as FoVs can be updated in a coordinated manner (note that backpropagating through the dimensions corresponding to square / shape can also affect purple / colour , as the same dimensions are occupied):

(b) Smoother Learning Dynamics : By forgoing the slot-based structure, a distributed encoding of compositional structure ensures that update transitions between FoVs do not cause sudden shifts in the representational space. For example:

potentially promoting more stable and reliable convergence during training.

Representational Expressivity : In the example above, both ψ s ( x ) and ψ tpr ( x ) reside in R 4 . However, the symbolic, slot-based compositional representation, ψ s ( x ) restricts each FoV to a specific, 2-dimensional subset (i.e., ψ colour (purple) , ψ shape (square) each belong to R 2 ). This rigid allocation limits expressivity as each FoV is confined to a subset of available dimensions, preventing the full utilisation of R 4 . In contrast, an inherently distributed approach allows each FoV - such as ˜ ψ (purple / colour) , ˜ ψ (square / shape) - to exist within the same R 4 space as the overall representation, ψ tpr ( x ) . This allows the representation learner to encode FoVs as combinations of the entire set of 4-dimensional basis vectors spanning the representational space, offering enhanced expressivity.

Robustness to Noise : In symbolic compositional representations, the impact of dimensionwise noise has a localised sensitivity: since each FoV is isolated within its designated slot, noise directly degrades the representation of the corresponding FoV. For example, a small perturbation in the third dimension (assigned to square / shape ) has a localised, direct effect only on this FoV. In contrast, for the inherently distributed TPR-based structure, noise introduced to any dimension of the representation is distributed across multiple FoVs. For example, for ψ tpr ( x ) , the same perturbation in the third dimension will have its impact distributed across both square / shape and purple / colour , as both these FoVs contribute to that dimension.

A.4 TPR's Distributed Nature and the Equivalence between Degenerate TPRs and Symbolic Representations

The TPR offers an inherently distributed representation of compositional structure, allowing information from multiple FoVs to intermingle across the same dimensions of the representational space.

Here, we elaborate on 2 critical properties of the TPR in relation to this notion of a distributed embedding of compositional structure.

Equivalence of Degenerate TPRs to Concatenated Fillers : While the TPR naturally supports a distributed encoding of compositional structure, it is worth noting that when the role embedding vectors are aligned with the canonical basis vectors, the TPR reduces to concatenated filler vectors, thereby reducing to a symbolic compositional representation where the representational components ψ i ( a i ) correspond to filler embeddings. In contrast to the TPR, however, it is impossible for a symbolic compositional representations to encode compositional structure in a distributed fashion.

Distributed Representation through Dense Role Vectors : In the general case where the role vectors are linearly independent (but do not correspond to the canonical basis), provided that the role vectors are dense , then TPRs facilitate a truly distributed representation by allowing each dimension of the representation to encapsulate a summation of information from multiple fillers. Such a representation is not reducible to a dimension-wise concatenation of fillers, as in the degenerate TPR case.

We now prove both statements:

Equivalence of Degenerate TPRs to Concatenated Fillers:

Let { e 1 , e 2 , . . . , e D R } denote the canonical basis vectors in R D R , where each e k has a 1 in the k -th position and 0s elsewhere.

Assume that each role vector r i = e i for i = 1 , 2 , . . . , N R .

The tensor product f m ( i ) ⊗ r i results in a matrix where the i -th column is f m ( i ) and all other columns are zero vectors:

Summing over all N R role-filler bindings in β ( x ) gives the required result:

This shows that in the case where the role vectors correspond to the canonical basis vectors, the the TPR effectively concatenates the filler vectors f i along distinct, non-overlapping subsets of representational dimensions. Consequently, the TPR ψ tpr ( x ) behaves equivalently to a symbolic, slot-based compositional structure where each filler occupies a unique representational slot without interacting with other fillers, hence validating the equivalence of degenerate TPRs to concatenated fillers.

Distributed Representation through Dense Role Vectors:

Assume that the role vectors { r 1 , r 2 , . . . , r N } in R D R are dense and linearly independent but are not aligned with the canonical basis vectors.

Each tensor product f m ( i ) ⊗ r i yields a matrix where each element ( j, k ) is f j m ( i ) r k i . Summing across all role-filler bindings yields:

Given that role vectors are dense, each element ψ j,k tpr incorporates contributions from multiple fillers f j m ( i ) , each weighted by the respective component r k i of their role vectors. This intermingling ensures

that information from different filler vectors are distributed across the same dimensions of ψ tpr ( x ) , rather than being confined to distinct subsets of ψ tpr ( x ) 's dimensions, as in the degenerate TPR case.

B Soft TPR Framework

In this section, we provide additional information regarding our Soft TPR framework.

B.1 Shortcomings of the TPR and How Soft TPR Helps

In this subsection, we discuss potential practical limitations of traditional TPR due to their rigid specification (Eq 1 ) and demonstrate how Soft TPR addresses these issues using a concrete example.

Consider the following set of roles R = { shape , colour } , and fillers F = { red , blue , square } , with embedding functions ξ R : R → R 2 and ξ F : F → R 3 defined as:

Discrete Mapping : The set of possible TPRs T generated by these roles and fillers is limited to discrete points. For instance:

Thus, T = { (2 2 4 2 2 3) T , (1 2 4 1 2 3) T } , a discrete subset of R 6 . By relaxing the TPR specification to the Soft TPR , we allow any z ∈ R 6 within an ϵ -neighbourhood of some ψ tpr in T , thereby forming continuous regions defined as: T S := { (2 2 4 2 2 3) T + α, (1 2 4 1 2 3) T + α : | α | < ϵ } . This relaxation significantly expands the representational space, as T S has strictly more points than T . Consequently, learning theoretically becomes easier because there are more functions parameterising the mapping from the observed data to T S compared to T . Furthermore, in contrast to T , which contains discrete, singular points scattered throughout R 6 , T S comprises continuous spherical regions centered around each ψ tpr ∈ T . These factors collectively make the Soft TPR representation potentially easier to learn and extract information from compared to the TPR. This advantage is reflected in our empirical results in Section C.6.3 , where the Soft TPR demonstrates 1) greater representation learner convergence, 2) superior sample efficiency for downstream tasks, and 3) superior raw downstream performance in the low sample regime compared to the traditional TPR. Additionally, our ablation results in Table 43 indicate that imposing stricter constraints on the encoder to produce representations in the form of explicit TPRs substantially impairs the disentanglement of the resulting representations.

Quasi-Compositional Structure : The traditional TPR enforces a strict algebraic definition of compositionality, specifically of the form ∑ i ξ F ( f m ( i ) ) ⊗ ξ R ( r i ) . However, even in algebraically-structured domains such as natural language, compositional structures do not always adhere to this rigid specification. For instance, French liaison consonants consist of a weighted sum of fillers bound to a single role [ 9 ], which deviates from the strict TPR formulation. The Soft TPR's continuous relaxation of this constraint permits the representation of structures that only approximately satisfy the TPR's algebraic definition of compositional structure as a set of discrete role-filler bindings. Consequently, Soft TPRs can encode forms of quasi-compositionality that are not strictly algebraically precise or directly amenable to TPR-based symbolic expression (see B.2 for a proof).

Duality : Although Soft TPRs employ a relaxed structural specification to encode quasi -compositional structures, they inherently preserve the exact algebraic form of compositionality through their corresponding explicit TPRs, ψ ∗ tpr , via quantisation using the TPR Decoder. In other words, while Soft TPRs facilitate approximate and flexible compositional representations, they retain the capability to revert to the precise algebraic structures defined by traditional TPRs when necessary. This dual functionality ensures that Soft TPRs strike a balance between the flexibility required to represent complex, approximately compositional structures-which benefits learning through relaxed representational constraints-and the preservation of exact algebraic forms essential for symbolic manipulation and interpretation.

Serial Construction : Building explicit TPRs requires that the embedded fillers and roles comprising a binding (i.e. ( ξ F ( f m ( i ) ) , ξ R ( r i )) ) are first tokened before the entire compositional representation, ψ tpr ( x ) can be produced [ 19 , 23 , 28 , 34 , 38 , 51 , 52 ]. This sort of sequential approach, where constituents must be tokened before the compositional representation can be formed, is a key characteristic of the symbolic representation of compositional structure [ 45 ]. In contrast, the Soft TPR allows the Encoder to produce any arbitrary element of V F ⊗ V R (in this case ∼ = R 6 ), provided that the sufficient closeness requirement holds. Thus, once the Encoder is trained, it is theoretically possible for the Soft TPR Autoencoder

to generate approximately compositional representations by mapping directly from the data to a Soft TPR in a single step, without needing to token any representational constituents.

B.2 Proof that Soft TPRs Do Not Necessarily Have the Explicit Form of a TPR

It is intuitive that Soft TPRs, due to their continuous relaxation, do not necessarily conform to the explicit algebraic form ∑ i ξ F ( f m ( i ) ) ⊗ ξ R ( r i ) required by traditional TPRs. We provide a formal proof for completeness.

Proof. Let ξ R : R → V R and ξ F : F → V F be fixed role and filler embedding functions, respectively. A traditional TPR is defined as:

where m ( i ) is a matching function assigning each role r i to a filler f m ( i ) .

A Soft TPR z is any element within an ϵ -neighborhood of some traditional TPR in V F ⊗ V R :

where ϵ > 0 .

Suppose, for contradiction, that every Soft TPR z ∈ B ϵ ( ψ tpr ) can be expressed as another traditional TPR with a different matching function m ′ :

Given that ξ R and ξ F are fixed, varying m ′ only changes the assignment of fillers to roles without altering the underlying embeddings.

Let δ = z -ψ tpr , where ∥ δ ∥ < ϵ . Substituting, we obtain:

Since δ = 0 , this equality implies:

̸

However, because ξ R and ξ F are fixed, the only way to satisfy this equality is if the matching function m ′ compensates exactly for δ . This is generally impossible due to the following constraints:

Fixed Embeddings : m ′ can only reassign fillers to roles without altering ξ R ( r i ) or ξ F ( f m ( i ) ) .The fixed ξ F ( f m ( i ) ) restrict the possible variations, making it infeasible to account for arbitrary δ through mere reassignment.

Dimensionality and Span : In high-dimensional spaces V F ⊗ V R , δ may lie outside the span of all traditional TPRs generated by varying m ′ , especially as ϵ increases.

Consequently, the perturbation δ cannot be exactly compensated by any reassignment m ′ , leading to a contradiction. Therefore, Soft TPRs with non-zero ϵ -distances from their corresponding quantised TPRs, ψ ∗ tpr , are also not necessarily expressible as other TPRs in the same underlying space.

The introduction of a continuous relaxation in Soft TPRs ensures that they do not strictly adhere to the traditional TPR's explicit algebraic form. This allows Soft TPRs to represent quasi-compositional structures that deviate from strict algebraic compositionality, which extends distributed compositional representations beyond the formal, explicitly algebraic domains required by traditional TPRs.

̸

B.3 Alternative Formulations

In contrast to the greedily optimal analytic form of ψ ∗ tpr derived in Equation 5 , a globally optimal TPR with respect to || z -ψ tpr || F can also be formulated. By fixing the sets of fillers F and roles R , along with the embedding functions ξ F : F → R D F and ξ R : R → R D R , the only degree of freedom lies in defining M ∗ , the set of role-filler matching functions that specify permissible role-filler binding decompositions for x ∈ X . The globally optimal TPR that best approximates z , denoted ψ opt tpr , is given by:

where T := {∑ i ξ F ( f m ( i ) ) ⊗ ξ R ( r i ) } m ∈M ∗ denotes the set of all possible traditional TPRs defined by the choices of F , R , ξ F , ξ R , and M ∗ . For clarity, we omit the dependency of both ψ tpr and m on x .

If M ∗ is a restricted subset of the entire set of matching functions M , meaning there are constraints on which fillers can be bound to which roles, then solving Equation 10 is NP-complete. Conversely, if M ∗ = M , allowing any filler to be bound to any role with possible reuse, the optimisation simplifies significantly. In this unrestricted scenario, the problem reduces to independently assigning each role to the filler that minimises the local contribution to || z -ψ tpr || F , which can be solved efficiently with a time complexity of O ( N R × N F ) . Acknowledging the impracticality of exact global optimisation, and guided by the intuition that ψ ∗ tpr should have an explicit dependency on z , we propose the greedily optimal definition of ψ ∗ tpr in Equation 5 , which links the unbound soft fillers of z , { ˜ f i } , to ψ ∗ tpr .

B.4 Soft TPR Autoencoder

Defining m ( i ) := arg min j ∥ ˜ f k -ξ F ( f j ) ∥ 2 in Equation 5 allows any filler f j to bind to a role r i , which may not preserve the ground-truth semantics of the role-filler bindings in the image. To address this, the Soft TPR Autoencoder targets two objectives jointly: 1) Representational Form , ensuring the output resembles a Soft TPR, and 2) Representational Content , ensuring sensible role-filler bindings reflecting ground-truth semantics.

B.4.1 Representational Form

We now describe how our method satisfies the representational form property. As detailed in Section 4.2, we penalize the Euclidean distance between the encoder E 's output z and the explicit TPR ψ ∗ tpr defined in Equation 5 . This penalty encourages the Soft TPR Autoencoder to produce encodings that resemble a Soft TPR by enforcing ∥ z -ψ tpr ∥ 2 < ϵ 7 . To obtain ψ ∗ tpr , we employ our TPR decoder, which consists of three modules: 1) Unbinding , 2) Quantization , and 3) TPR Construction .

1) Unbinding : Although multiple roles can bind to the same filler (e.g., object colour / magenta and floor colour / magenta ), each role represents an independent concept type. Therefore, we use linearly independent embedding vectors for roles to ensure the recoverability of embedded components ( ξ F ( f m ( i ) ) , ξ R ( r i )) from the TPR. Typically, a randomly initialised role embedding matrix M ξ R ∈ R D R × N R with D R ≫ N R will have linearly independent columns. To maintain this property during training, it is possible to apply orthogonality regularisation: ∥ M T ξ R M ξ R -I ∥ F .

However, using arbitrary linearly independent role embeddings poses a computational challenge in obtaining U , the (left) inverse of M ξ R , necessary for unbinding vectors { u i } . To address this, we utilise semi-orthogonal matrices A ∈ R d × n where A T A = I . In this case, U T = M ξ R , and the unbinding vector u i is simply the role embedding ξ R ( r i ) . Thus, unbinding a soft filler from z bound to role r i involves taking the tensor inner product between z and ξ R ( r i ) .

To implement this, our unbinding module initialises M ξ R as a dense, semi-orthogonal matrix using torch . nn . utils . parameterizations . orthogonal and keeps M ξ R fixed during training to preserve its semi-orthogonal property 8 .

Now, we proceed to formally prove the aforementioned semi-orthogonality properties.

We first prove that for a (left-invertible) semi-orthogonal matrix, A ∈ R d × n , A T A = I n × n .

Proof. Let A denote a (left-invertible) semi-orthogonal matrix of dimension R d × n . Writing A out using its columns, { a 1 , . . . , a n } , we have:

As the columns { a 1 , . . . , a n } are orthonormal, we have that: a i · a j = δ ij , where · denotes the dot product, and

from which the result clearly follows.

For completeness, we also prove that u i corresponds to the i -th role embedding vector, though this follows rather trivially from the properties of semi-orthogonal matrices, and the definition of u i .

Proof. Let M ξ R be a (left-invertible) semi-orthogonal matrix. Then, from the properties of semiorthogonal matrices proved earlier, we have that M T ξ R M ξ R = I , and so, U T = ( M T ξ R ) T = M ξ R . Hence, the i -th column of U T , which corresponds to the unbinding vector, u i , by definition, is simply ( M ξ R ) : i = ξ R ( r i ) , the i -th role embedding vector.

2) Quantisation

The quantisation module utilises the VQ-VAE algorithm [ 11 ] to learn the filler embedding matrix M ξ F and to quantise the soft filler embeddings { ˜ f i } into explicit embeddings { ξ F ( f i ) } based on the closest Euclidean distances. This process aligns with the matching function m in Equation 5 , where each soft filler ˜ f i is assigned to the nearest explicit filler in M ξ F . Since quantisation involves an argmax operation and is non-differentiable, VQ-VAE employs an L 2 codebook loss to adjust the embedding vectors towards the soft fillers and a commitment loss to ensure the encoder commits to specific embeddings, preventing the embedding space from expanding indefinitely.

For further details on the quantisation process, refer to [ 11 ].

3) TPR Construction

The TPR construction module is parameter-free and deterministically binds the quantised filler embeddings { ξ F ( f m ( i ) ) } to their corresponding role embeddings { ξ R ( r i ) } from M ξ R , forming the explicit TPR:

B.4.2 Representational Content

Here, we explicitly define the 'swapped' TPRs, ψ s tpr ( x ) and ψ s tpr ( x ′ ) , used in our weakly supervised reconstruction loss (second term of Equation 7 ). For image pairs ( x, x ′ ) sharing all role-filler bindings

except for role r i , we construct the swapped TPRs by exchanging the quantised filler embeddings for r i :

̸

̸

where m and m ′ are the matching functions for x and x ′ , respectively.

B.4.3 Model Architecture

We now provide concrete details of our model architecture in Table 8 , 9 and Table 10 . It is worth noting that Soft TPR Autoencoder only adds an additional D F · N F parameters to a standard (variational) autoencoding framework consisting of the encoder and decoder as the only learnable parameters we introduce are in the filler embedding matrix, M ξ F .

For all our experiments, we use the same general architecture for the encoder, E , the TPR decoder, and the decoder, D .

B.5 Model Hyperparameters and Hyperparameter Tuning

The Soft TPR Autoencoder has the following tunable hyperparameters:

Architectural Hyperparameters :

These correspond to the number of role and filler embedding vectors and the dimensionalities of their respective embedding spaces.

Loss Function Hyperparameters :

β ( which weights the VQ-VAE commitment loss as per Equation 6 and )

λ 1 , λ 2 ( which weigh the unsupervised and supervised losses in Equation 7 )

Following the VQ-VAE framework [ 11 ], we set β , the coefficient for the commitment loss, to 0.5. To ensure fair comparisons with scalar-tokened generative baselines and COMET, which assume 10 FoV types (and VCT assumes 20), we fix N R = 10 .

We optimise the remaining hyperparameters using the Weights and Biases (WandB) hyperparameter sweep framework. The optimisation employs Bayesian search with the unsupervised MSE reconstruction loss from Equation 6 as the criterion. During this optimization, models are trained for between 50,000 and 100,000 iterations.

The final hyperparameter values for the Soft TPR Autoencoder are listed in Table 11 . For ablation experiments demonstrating the model's robustness to different hyperparameter settings, refer to Section C.6.4 .

Our model is implemented in Pytorch [ 27 ] and trained using the Adam optimiser on the loss corresponding to 7 . We use a learning rate of 1e -4 , and the default setting of ( β 1 , β 2 ) = (0 . 9 , 0 . 999) across all instances of model training.

C Results

In this section, we provide additional details about our experiments, and present additional results.

C.1 Datasets

For disentanglement, we utilise standard disentanglement datasets: Cars3D [ 4 ], Shapes3D [ 24 ], and the 'real' variant of MPI3D [ 21 ], containing 3 , 6 , and 7 ground-truth FoVs respectively. These datasets are procedurally generated via taking the Cartesian product of all possible FoV values per FoV type. Metadata is provided in Tables 12 , 13 and 14 , with continuous-valued FoV types marked by ‡ . We treat colour FoVs as continuous variables due to their 10 linearly spaced, naturally ordered values, in line with [ 42 ].

For the downstream regression task, we use these disentanglement datasets and train downstream models to regress on each dataset's continuous-valued FoVs, as indicated by the ‡ symbol in Tables 12 , 13 and 14 . For abstract visual reasoning [ 30 ], we employ the Raven's Progressive Matrix (RPM) style dataset from [ 30 ], where each sample consists of 8 context panels and 6 answer panels derived from Shapes3D images. The model must infer abstract visual relations from the context panels to select the correct answer among six options.

C.2 Baseline Implementations and Experimental Settings

C.2.1 Experiment Compute Resources

For the generative weakly supervised, scalar-valued baselines (i.e., AdaGVAE-k, GVAE, MLVAE, SlowVAE, the Shu model), and our model, we perform model training on a single Nvidia RTX4090 GPU. We also perform all associated experiments for these models (i.e., downstream model training, downstream model evaluation, disentanglement evaluation, etc.) on this single Nvidia RTX4090 GPU. Our model takes approximately 1.5 hours to fully train (i.e., run 200,000 iterations) on the Cars3D and Shapes dataset, and approximately 4.0 hours to fully train on the MPI3D dataset.

For the vector-valued baselines of COMET and VCT, we perform model training, and all associated experiments on a single Nvidia V100 GPU.

C.2.2 Disentanglement Models

For the generative weakly supervised scalar-valued baselines (AdaGVAE-k, GVAE, MLVAE, SlowVAE, Shu model) and our model, training and all related experiments (downstream training, evaluation, disentanglement) were conducted on a single Nvidia RTX4090 GPU. Our model requires approximately 1.5 hours to train (200,000 iterations) on the Cars3D and Shapes datasets, and about 4.0 hours on the MPI3D dataset.

Vector-valued baselines COMET and VCT were trained and evaluated on a single Nvidia V100 GPU. We utilised the PyTorch-based open-source implementations from [ 42 ] for Ada-GVAE, GVAE, MLVAE, and SlowVAE, verified to reproduce reported results. Official implementations from the authors were used for COMET and VCT [ 36 , 47 ]. For the GAN-based Shu model, we converted the TensorFlow implementation from [ 35 ] to PyTorch and ensured it reproduced official results. Baseline models employed recommended hyperparameters; otherwise, hyperparameter tuning was conducted using the same Weights and Biases (WandB) hyperparameter sweep as our model.

As discussed in Section 5, all weakly supervised baselines except Ada-GV AE assume access to I , the differing FoV between each pair ( x, x ′ ) , matching our model's supervision level. To make Ada-GVAE

comparable, we modified it to receive the ground-truth k = | I | , the number of changing FoVs per sample, termed Ada-GVAE-k. This adjustment resulted in superior or comparable performance to the original Ada-GVAE. All models, except SlowVAE (which assumes all FoVs change), were trained with k = 1 , ensuring consistent training conditions across models.

For each model's selected hyperparameters, including our own, we trained five models using five random seeds and aggregated the results. All models were trained for 200,000 iterations on each dataset.

C.2.3 Downstream Models

For FoV regression, following [ 42 ], we use a simple, generic MLP model, with the architecture listed in Table 15 . For each disentanglement dataset, the MLP regression model receives representations of images produced by a representation learner, and is trained to predict the ground-truth FoV values in a supervised fashion. Performance is measured using the R 2 score on a held out, randomly selected test set of 1,000 samples.

For each representation learning model instantiated with a random seed, we train 2 MLPs by uniformly sampling the number of output nodes in the first, second, and fifth layers as specified in Table 15 , resulting in a total of 10 MLPs per learner (we use 5 random seeds). Reported results are averaged over these models. Notably, the MLPs have no prior knowledge of the representation type (e.g., scalar-tokened symbolic, vector-tokened symbolic, or fully continuous compositional) since no inductive biases or representation-specific optimisations are applied, aside from the supervised MSE loss.

Each MLP is trained using the Adam optimiser with an MSE loss, a learning rate of 1 × 10 -4 , and default ( β 1 , β 2 ) = (0 . 9 , 0 . 999) settings.

For the abstract visual reasoning task, following [ 30 ], we use the Wild Relation Network (WReN) model [ 18 ] on representations produced by representation learners to predict ground-truth answers for each RPM matrix. For each of model instances produced by a random seeds, we randomly sample 2 possible configurations of the WReN model by uniformly sampling either 256 or 512 for the edge MLP, g , and 128 or 256 hidden units for the graph MLP, f , in line with [ 30 ]. We however, fix the number of hidden layers in g to 2, and f in 1, representing the smallest possible number of hidden layers, to constrain the capacity of the WReN model. As we use 5 random seeds, and sample 2 configurations per seed, we produce 10 WReN models for each representation learner. All WReN models are trained using Adam optimisation on the BCE loss between the predicted logits and ground-truth labels, with a learning rate of 1e -5 and the default setting of ( β 1 , β 2 ) = (0 . 9 , 0 . 999) .

C.2.4 Experimental Controls

To ensure that the performance improvements of our model are attributable to the distributed, flexible nature of the Soft TPR framework, we implement controls to eliminate confounding variables.

Disentanglement Controls

To ensure performance gains in disentanglement metrics (Tables 16 and 17 ) are not attributable to our model's additional parameters (with our model's the filler embedding matrix M ξ F , adding

13,568, 1,824, and 1,600 parameters for Cars3D, Shapes3D, and MPI3D respectively to a standard autoencoding framework), we adjust baseline models with fewer parameters. Specifically, for SlowVAE, Ada-GVAE-k, GVAE, ML-VAE, and the Shu model, we increase their parameters by adding filters or layers until they match or exceed our model's parameter count, denoted with ∗ in Tables 16 and 17 . VCT and COMET are substantially more parameter hungry (10+ million) than our model, so parameter controls are not applicable to these vector-valued symbolic compositional baselines. Consistent with [ 40 ], we observe that increasing parameters in these generative, scalartokened baselines does not enhance disentanglement performance. Therefore, we proceed with representation learning convergence experiments and downstream model tests using the original models, not the parameter-adjusted variants.

Downstream Task Controls

To ensure that any performance boosts on the downstream tasks of FoV regression and abstract visual reasoning are not attributable to our representation's increased dimensionality, we postprocess representations produced by all models (including models producing higher-dimensional representations) to match the dimensionality of our representation. Specifically, generative, weakly supervised models with scalar-tokened representations produce representations with a dimension of 10 for all datasets, whereas our Soft TPRs have dimensions of 1536 (Cars3D), 512 (Shapes3D), and 320 (MPI3D) of our Soft TPRs, highlighting the necessity of such a control.

To perform this control, we apply separate methods for the scalar-tokened and vector-tokened models. For scalar-tokened models, we multiply each dimension k of the latent vector, l k with an random embedding vector e k ∈ R d from an fixed, randomly initialised, semi-orthogonal embedding matrix E ∈ R d × 10 , and subsequently concatenate all multiplied random embedding vectors together to form the dimensionality-controlled representation, ( l 1 e k , . . . , l 10 e 10 ) ∈ R 10 d . d is a dataset specific integer that ensures that the size of the dimensionality-controlled representation is at least as small as the dimensionality of the Soft TPR representation for the given dataset, i.e., d := ⌈ dim( z ) ⌉ / 10 , where z is the Soft TPR representation produced by a given dataset. Note that we choose a semiorthogonal embedding matrix with the intuition that this ensures the maximal distinguishability of each continguous subset of dimensions corresponding to l i e i .

For COMET and VCT, which both produce vector-tokened representations, we consider alternative methods of performing dimensionality-control. COMET's representations have a dimensionality of 640 for all datasets, and so, the model produces lower-dimensional representations than ours for Shapes3D and Cars3D, but not MPI3D. VCT, on the other hand, produces representations with dimension 5120, and so, produces higher-dimensional representations than ours for all datasets. For any vector-tokened representation with higher dimensionality than our representation, we apply PCAbased postprocessing to the model representation, reducing the dimensionality of each vector-tokened value in the representation to the required dimensionality, d , where d := ⌈ dim( z ) ⌉ / dim( z baseline ) , before concatenating all PCA-reduced vectors together to produce the modified representation. For any vector-tokened representation produced by COMET with lower dimensionality than ours, we apply a simple matrix multiplication between the representation, and a randomly initialised matrix of required dimensionality to embed the representation into the same-dimensional space as ours.

We denote the results produced by all such postprocessing by † in relevant tables, and indicate such postprocessing in the captions of relevant plots.

C.3 Disentanglement

In reporting the disentanglement metric results for baseline models, we use published results where applicable, i.e., we use the results published in [ 47 ] for VCT and COMET and the published results for SlowVAE [ 39 ]. For GVAE, MLVAE, and Ada-GVAE-K, we evaluate disentanglement using the Pytorch-based implementation of disentanglement metrics, [ 42 ], which corresponds to a Pytorchbased implementation of the official disentanglement lib of [ 25 ]. We also use this implementation to evaluate the disentanglement of representations produced by all models.

C.3.1 Disentanglement Metrics

In line with [ 47 ], we consider the following 4 disentanglement metrics: the FactorVAE score [ 24 ], the DCI Disentanglement score [ 16 ] (we refer to DCI Disentanglement as DCI), the BetaVAE score [ 10 ], and the MIG score [ 15 ]. We provide a brief overview of all 4 disentanglement metrics, and

refer interested readers to the papers these metrics are introduced in for further details, as well as the Appendix of [ 25 ] for more details on how the metrics are implemented in [ 42 ] and [ 25 ].

FactorVAE Metric : Arandomly selected FoV of the dataset is fixed, and a mini-batch of observations is subsequently randomly sampled. The representation learner produces representations for the samples. Disentanglement is quantified using the accuracy of a majority vote classifier that predicts the index of the ground-truth fixed FoV based on the index of the representation vector with the smallest variance.

DCI Metric : A Gradient Boosted Tree (GBT) is trained to predict the ground-truth FoV values from representations produced by a representation learner. The predictive importance of the dimensions of a representation is obtained using the model's feature importances. For each sample, a score is computed that corresponds to one minus the entropy of the probability that a dimension of the learned representation is useful for FoV prediction, weighted by the relative entropy of the corresponding dimension. An average of these scores over the mini-batch of samples is taken to produce the final score.

BetaVAE Metric : This metric quantifies disentanglement by predicting the index of a fixed FoV from representations produced by a representation learner, similar to the FactorV AE metric. In contrast to the FactorVAE metric, however, the BetaVAE metric uses a linear classifier on difference vectors to predict the index of the fixed FoV. Each difference vector is produced by taking the difference between representations produced for a pair of samples ( x, x ′ ) with one underlying fixed FoV.

MIG Metric : For each FoV, the MIG metric computes the mutual information between each dimension in the representation, and the corresponding FoV. The score is obtained by computing the average, normalised difference between the highest and second highest mutual information of each FoV with the dimensions of the representation.

C.3.2 Evaluating the Disentanglement of Soft TPRs

Since disentanglement metrics are typically computed under the assumption that the representation corresponds to a concatenation of scalar-valued FoV tokens, we now detail how we compute the above disentanglement metrics on the Soft TPR, a continuous compositional representation. Similar amendments have been made in COMET and VCT, so that the disentanglement of their representations, corresponding to a concatenation of vector-valued FoV tokens, can be computed.

FactorVAE metric : To compute the FactorVAE score of our Soft TPR, we produce a N R -dimensional vector, v , for each Soft TPR, where N R corresponds to the number of roles, i.e. the FoV types . We produce v by simply populating each dimension, v i , with the index of the quantised filler that role r i is bound to, i.e. we set v i to m ( i ) . We use the resulting v 's to compute the variances required by the FactorVAE metric, noting that if the FoV type corresponding to role i is fixed, and this is reflected in our representation, dimension i of the v vectors produced in a mini-batch should clearly have the smallest variance, as all fillers, f m ( i ) , that role r i binds to, will have the same identity across the mini-batch.

DCI metric : As the DCI relies on computing the ground-truth FoV values from N R -dimensional representations produced by the representation learner, we again, follow a similar procedure as the above, in converting our ( D F · D R ) -dimensional Soft TPR representation to a N R -dimensional representation. That is, we simply consider a N R -dimensional vector, v , where each dimension, v i , is populated by the index of the quantised filler, m ( i ) that role r i is bound to, and use this vector to compute the corresponding DCI result.

BetaVAE metric : For the BetaVAE metric, as each sample used to train the linear classifier consists of a N R -dimensional difference vector obtained by computing the difference between the N R -dimensional scalar-tokened compositional representations, we obtain the following difference vector, d , for our Soft TPR representations. For each role i ∈ { 1 , . . . , N R } , we obtain the corresponding quantised filler embeddings for each sample x, x ′ in the pair, i.e., we obtain ξ F ( f m ( i ) ) , ξ F ( f m ′ ( i ) ) . For each pair of quantised filler embeddings, we obtain a scalar distance measure corresponding to the cosine similarity between the the pair. The i -th dimension of d is populated using this value. We use difference vectors obtained in this way to compute the BetaV AE metric.

MIG metric : For the MIG metric, which relies on a discretisation of the values in each dimension i of the N R -dimensional scalar-tokened compositional representations to compute the discrete mutual

information, we apply the same postprocessing technique as in the FactorVAE, and DCI metric, to evaluate MIG on our Soft TPRs. That is, we produce the same N R -dimensional vector, v , noting that this choice of postprocessing also performs the discretisation required by the MIG computation.

C.3.3 Full Results

We now present unabridged results for all considered disentanglement metrics, denoting the learnable parameter control modification of each relevant baseline with the symbol ∗ . As can be seen in Tables 16 and 17 , our Soft TPR representations are explicitly more compositional (as quantified by the disentanglement metric scores) compared to all considered baselines, with especially notable performance increases on the more challenging datasets of Cars3D and MPI3D.

0.863

0.027

0.984

0.012

Ours

0.999

0.001

0.949

0.032

0.828

0.015

Models

Cars3D

Shapes3D

MPI3D

BetaVAE score

MIG

BetaVAE score

MIG

BetaVAE score

MIG

Symbolic scalar-tokened compositional representations

Slow-VAE

1.000

±

0.000

(=)

0.104

±

0.018

1.000

±

0.000

(=)

0.615

±

0.045

0.666

±

0.069

0.329

±

0.026

Slow-VAE

∗

1.000

±

0.000

(=)

0.071

±

0.013

1.000

±

0.000

(=)

0.655

±

0.067

0.629

±

0.079

0.287

±

0.045

Ada-GVAE-k

1.000

±

0.000

(=)

0.395

±

0.095

1.000

±

0.000

(=)

0.556

±

0.064

0.750

±

0.053

0.213

±

0.064

Ada-GVAE-k

∗

1.000

±

0.000

(=)

0.321

±

0.102

1.000

±

0.000

(=)

0.498

±

0.073

0.783

±

0.061

0.241

±

0.079

GVAE

1.000

±

0.000

(=)

0.096

±

0.036

0.998

±

0.004

0.251

±

0.072

0.704

±

0.072

0.145

±

0.074

GVAE

∗

1.000

±

0.000

(=)

0.100

±

0.052

1.000

±

0.000

0.203

±

0.089

0.681

±

0.061

0.109

±

0.087

MLVAE

1.000

±

0.000

(=)

0.088

±

0.020

0.976

±

0.038

0.354

±

0.165

0.703

±

0.039

0.142

±

0.062

MLVAE

∗

1.000

±

0.000

(=)

0.050

±

0.058

0.921

±

0.052

0.298

±

0.091

0.689

±

0.041

0.096

±

0.071

0.912

±

0.022

0.025

±

0.012

0.498

±

0.064

0.005

±

0.003

0.353

±

0.040

0.013

±

0.007

∗

0.923

±

0.031

0.015

±

0.009

0.512

±

0.071

0.006

±

0.002

0.327

±

0.059

0.009

±

0.011

Symbolic vector-tokened compositional representations

VCT

1.000

±

0.000

(=)

0.117

±

0.045

0.999

±

0.0004

0.525

±

0.028

0.844

±

0.038

0.227

±

0.048

COMET

0.343

±

0.006

0.000

±

0.000

0.166

±

0.004

0.0002

±

0.000

0.144

±

0.005

0.000

±

0.0001

Fully continuous compositional representations

Ours

1.000

±

0.000

(=)

0.348

±

0.0124

1.000

±

0.000

(=)

0.471

±

0.088

1.000

±

0.000

0.620

±

0.067

C.4 Representation Learning Convergence

We additionally examine representation learning convergence by evaluating the representations produced at 100, 1,000, 10,000, 100,000, and 200,000 iterations of model training, where the latter stage of 200,000 iterations corresponds to fully trained models. To quantify representation learning convergence, we evaluate both 1) the explicit compositionality of representations produced at these stages of training (as quantified by disentanglement metric performance), and 2) the usefulness of these representations for the downstream tasks of FoV regression and abstract visual reasoning.

In all line plots, we plot the mean, and indicate the standard deviation by the shaded regions. We use the same legend for all plots, where 0 (grey) denotes SlowVAE, 1 (orange) denotes AdaGVAE-k, 2 (green) denotes GVAE, 3 (red) denotes MLVAE, 4 (purple) denotes Shu, 5 (pink) denotes VCT, 6 (brown) denotes COMET, and 7 (blue) denotes our model, Soft TPR Autoencoder.

C.4.1 Disentanglement

We first present line plots of representation learner convergence for each of the four considered disentanglement metrics (i.e., the FactorV AE, DCI, BetaV AE and MIG scores) for all three disentanglement datasets (Cars3D, Shapes3D, MPI3D). A series of tables containing the values associated with these line plots is presented following the plots. As the disentanglement results produced by our learnable parameter controls for the models of Ada-GVAE-k, GVAE, ML-VAE and Shu, do not achieve superior disentanglement results compared to the original models, we only present disentanglement convergence results for the original variants of all baseline models.

10 2

10 3

10 4

10 5

0.0

0.2

0.4

0.6

0.8

1.0

Factor score

Cars3D

0

1

2

3

4

5

6

7

Iteration count

10 2

10 3

10 4

10 5

Iteration count

0.0

0.2

0.4

0.6

0.8

DCI score

Cars3D

0

1

2

3

4

5

6

7

10 2

10 3

10 4

10 5

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Beta score

Cars3D

0

1

2

3

4

5

6

7

Iteration count

10 2

10 3

10 4

10 5

Iteration count

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

MIG score

Cars3D

0

1

2

3

4

5

6

7

10 2

10 3

10 4

10 5

0.0

0.2

0.4

0.6

0.8

1.0

Factor score

Shapes3D

0

1

2

3

4

5

6

7

Iteration count

10 2

10 3

10 4

10 5

Iteration count

0.0

0.2

0.4

0.6

0.8

1.0

DCI score

Shapes3D

0

1

2

3

4

5

6

7

10 2

10 3

10 4

10 5

0.2

0.4

0.6

0.8

1.0

Beta score

Shapes3D

0

1

2

3

4

5

6

7

Iteration count

10 2

10 3

10 4

10 5

Iteration count

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

MIG score

Shapes3D

0

1

2

3

4

5

6

7

10 2

10 3

10 4

10 5

0.0

0.2

0.4

0.6

0.8

1.0

Factor score

MPI3D

0

1

2

3

4

5

6

7

Iteration count

10 2

10 3

10 4

10 5

Iteration count

0.0

0.2

0.4

0.6

0.8

DCI score

0

1

2

3

4

5

6

7

10 2

10 3

10 4

10 5

0.2

0.4

0.6

0.8

1.0

Beta score

MPI3D

0

1

2

3

4

5

6

7

Iteration count

10 2

10 3

10 4

10 5

Iteration count

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

MIG score

0

1

2

3

4

5

6

7

Ours

0.379

±

0.117

0.381

±

0.026

0.812

±

0.033

0.843

±

0.024

0.832

±

0.048

Models

BetaVAE score

10

2

iterations

10

3

iterations

10

4

iterations

10

5

iterations

2

×

10

5

iterations

Symbolic scalar-tokened compositional representations

Slow-VAE

0.937

±

0.025

0.999

±

0.002

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

Ada-GVAE-k

0.971

±

0.011

1.000

±

0.000

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

GVAE

0.958

±

0.027

0.999

±

0.001

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

MLVAE

0.964

±

0.011

1.000

±

0.000

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

0.534

±

0.164

0.742

±

0.072

0.728

±

0.105

0.433

±

0.016

0.449

±

0.099

Symbolic vector-tokened compositional representations

VCT

0.889

±

0.044

1.000

±

0.001

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

COMET

0.592

±

0.027

0.592

±

0.028

0.592

±

0.028

0.592

±

0.028

0.592

±

0.028

Fully continuous compositional representations

Ours

0.953

±

0.042

1.000

±

0.000

(=)

1.000

±

0.000

(=)

1.000

±

0.000

(=)

1.000

±

0.000

Ours

0.057

0.018

0.183

0.077

0.887

0.068

Ours

0.334

±

0.079

0.350

±

0.049

0.750

±

0.065

0.937

±

0.034

0.951

±

0.083

C.4.2 Downstream Performance

We additionally evaluate the representation learning convergence by examining the usefulness of representations produced at different stages of training (i.e., 100, 250, 500, 1,000, 10,000, 100,000

and 200,000 iterations of training). To quantify 'usefulness', we consider performance of downstream models on the tasks of FoV regression, and abstract visual reasoning when trained using representations produced by each stage of training. For each task, we present line plots, and additionally tables with values corresponding to each of the plots. We use the same legend as in the previous section, where 0 (grey) denotes SlowVAE, 1 (orange) denotes AdaGVAE-k, 2 (green) denotes GVAE, 3 (red) denotes MLVAE, 4 (purple) denotes Shu, 5 (pink) denotes VCT, 6 (brown) denotes COMET, and 7 (blue) denotes our model, Soft TPR Autoencoder. We additionally provide all results for the dimensionality-control setting, where the dimensionality of representations produced by all representation learners is held constant following the approach detailed in C.2.4 , denoting this clearly in plot captions, and by the symbol † in the tables.

FoV Regression

As clearly visible in the tables and plots, generic regression models are able to more effectively use representations produced by our Soft TPR Autoencoder produced across almost all stages of training for all disentanglement datasets. Improvements are most notable in the low-iteration regime of 10 2 iterations, and across most stages of training for the more challenging task of FoV regression on the MPI3D dataset (Figures 19 and 20 ).

10 2

10 3

10 4

10 5

0.0

0.2

0.4

0.6

0.8

1.0

RSq

Cars3D

0

1

2

3

4

5

6

7

Iteration count

10 2

10 3

10 4

10 5

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

RSq

Cars3D

0

1

2

3

4

5

6

7

Iteration count

10 2

10 3

10 4

10 5

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RSq

Shapes3D

0

1

2

3

4

5

6

7

Iteration count

10 2

10 3

10 4

10 5

Iteration count

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RSq

Shapes3D

0

1

2

3

4

5

6

7

10 2

10 3

10 4

10 5

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RSq

MPI3D

0

1

2

3

4

5

6

7

Iteration count

10 2

10 3

10 4

10 5

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RSq

MPI3D

0

1

2

3

4

5

6

7

Iteration count

Abstract Visual Reasoning

We present now present our full suite of results for the downstream task of abstract visual reasoning. As demonstrated in Table 33 and the corresponding Figures 21 and 22 , representations produced by our model at only 10 2 iterations of training are able to be leveraged by downstream models to achieve a 80.04 % accuracy for the challenging abstract visual reasoning task, in contrast to the value of 63.1 % obtained by the best performing baseline, representing a 26.78 % performance increase.

Abstract visualreasoning dataset

1.0

2

3

4

0.8

5

6

0.6

0.4

0.2

102

103

104

105

Iterationcount

Abstract visualreasoning dataset

1.0

3

4

5

0.8

0.6

0.4

0.2

102

103

104

105

Iterationcount

C.5 Downstream Performance

We present our full suite of experimental results that empirically demonstrate the utility of our Soft TPR representation from the perspective of downstream models, by considering the sample efficiency, and low-sample regime performance of downstream models on the tasks of FoV regression, and abstract visual reasoning.

C.5.1 Sample Efficiency Results

For sample efficiency, as mentioned in Section 5.2, and in line with [ 25 ], we compute a ratio-based metric obtained by dividing the performance of the downstream model when trained using a restricted number of 100, 250, 500, 1,000 and 10,000 samples, by its performance when trained using all samples (19,104, 480,000, 1,036,800 and 100,000 samples for the tasks of regression on the Cars3D, Shapes3D, MPI3D datasets, and the abstract visual reasoning task respectively). As this metric is dependent on the performance of downstream models when trained using all samples, we do not compute this metric for representation learners where the corresponding downstream models achieve an R 2 score of less than 0 . 5 for regression, as this may produce sample efficiency scores with very little semantic meaning (e.g. a model that achieves a sample efficiency score of 0.9 when its final R 2 score is 0.1). As a result, we remove Shu from Shapes3D sample efficiency calculations, and COMET and Shu from the Cars3D sample efficiency calculations.

As many models for the abstract visual reasoning task have low classification accuracies on the held-out test set following training with the maximal number of 100,000 samples, we do not compute sample efficiencies for this task, and instead refer readers to results in Section C.5.2 for the raw classification accuracies associated with each model.

Note that for all box plots, we follow standard convention, and display the median in each box with a solid line, where the box shows the quartiles of the corresponding values, and the whiskers extend to 1 . 5 times the interquartile range. We again, use the same legend, where grey denotes SlowVAE, orange denotes AdaGVAE-k, green denotes GVAE, red denotes MLVAE, purple denotes Shu, pink denotes VCT, brown denotes COMET, and blue denotes our model, Soft TPR Autoencoder.

0

1

2

3

4

5

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (100/all)

0

1

2

3

4

5

0.0

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (250/all)

0

1

2

3

4

5

0

1

2

3

4

5

0.0

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (500/all)

0

1

2

3

4

5

0.0

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (1,000/all)

0

1

2

3

4

5

0.0

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (10,000/all)

0

1

2

3

4

5

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (100/all)

0

1

2

3

4

5

6

7

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (250/all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.5

0.6

0.7

0.8

0.9

1.0

Rsq ratio

Sample efficiency (500/all)

0

1

2

3

4

5

6

7

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Rsq ratio

Sample efficiency (1,000/all)

0

1

2

3

4

5

6

7

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (10,000/all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Rsq ratio

Sample efficiency (100/all)

0

1

2

3

4

5

6

0.0

0.2

0.4

0.6

0.8

Rsq ratio

Sample efficiency (250/all)

0

1

2

3

4

5

6

0

1

2

3

4

5

6

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (500/all)

0

1

2

3

4

5

6

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (1,000/all)

0

1

2

3

4

5

6

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Rsq ratio

Sample efficiency (10,000/all)

0

1

2

3

4

5

6

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Rsq ratio

Sample efficiency (100/all)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

Rsq ratio

Sample efficiency (250/all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (500/all)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (1,000/all)

0

1

2

3

4

5

6

7

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Rsq ratio

Sample efficiency (10,000/all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Rsq ratio

Sample efficiency (100/all)

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Rsq ratio

Sample efficiency (250/all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

Rsq ratio

Sample efficiency (500/all)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

Rsq ratio

Sample efficiency (1,000/all)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

Rsq ratio

Sample efficiency (10,000/all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Rsq ratio

Sample efficiency (100/all)

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Rsq ratio

Sample efficiency (250/all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

Rsq ratio

Sample efficiency (500/all)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

Rsq ratio

Sample efficiency (1,000/all)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

Rsq ratio

Sample efficiency (10,000/all)

0

1

2

3

4

5

6

7

C.5.2 Low Sample Regime Results

To evaluate the utility of our Soft TPR representation from the perspective of downstream models, we additionally evaluate the raw performance of downstream models as a function of the number of samples they have been trained on (again considering 100, 250, 500, 1,000, 10,000 and all samples). We find that our Soft TPR representation contributes to a substantial performance boost in the downstream model's performance in a low-sample regime where the downstream model has been trained with 100, 250, 500, and 1,000 samples. We present our full suite of experimental results, and highlight the particular performance differentials conferred by our representational form in the low-sample regime.

FoV Regression

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

RSq

Downstream performance (100)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

RSq

Downstream performance (250)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

RSq

Downstream performance (500)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

RSq

Downstream performance (1,000)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

RSq

Downstream performance (10,000)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

RSq

Downstream performance (all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

RSq

Downstream performance (100)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

RSq

Downstream performance (250)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

RSq

Downstream performance (500)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

RSq

Downstream performance (1,000)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

RSq

Downstream performance (10,000)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

RSq

Downstream performance (all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

RSq

Downstream performance (100)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

RSq

Downstream performance (250)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

RSq

Downstream performance (500)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

RSq

Downstream performance (1,000)

0

1

2

3

4

5

6

7

0.2

0.4

0.6

0.8

1.0

RSq

Downstream performance (10,000)

0

1

2

3

4

5

6

7

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RSq

Downstream performance (all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

RSq

Downstream performance (100)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

RSq

Downstream performance (250)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

RSq

Downstream performance (500)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

RSq

Downstream performance (1,000)

0

1

2

3

4

5

6

7

0.2

0.4

0.6

0.8

1.0

RSq

Downstream performance (10,000)

0

1

2

3

4

5

6

7

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RSq

Downstream performance (all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

RSq

Downstream performance (100)

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

RSq

Downstream performance (250)

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

RSq

Downstream performance (500)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

RSq

Downstream performance (1,000)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

RSq

Downstream performance (10,000)

0

1

2

3

4

5

6

7

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RSq

Downstream performance (all)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

RSq

Downstream performance (100)

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

RSq

Downstream performance (250)

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

RSq

Downstream performance (500)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

RSq

Downstream performance (1,000)

0

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

RSq

Downstream performance (10,000)

0

1

2

3

4

5

6

7

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

RSq

Downstream performance (all)

0

1

2

3

4

5

6

7

0.858

Ours

0.019

0.917

0.013

0.910

0.010

b) Abstract Visual Reasoning

0

1

2

3

4

5

6

7

0.15

0.20

0.25

0.30

0.35

Downstream performance (100)

0

1

2

3

4

5

6

7

0.15

0.20

0.25

0.30

0.35

Downstream performance (250)

0

1

2

3

4

5

6

7

0.15

0.20

0.25

0.30

0.35

0.40

Downstream performance (500)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Downstream performance (1,000)

0

1

2

3

4

5

6

7

0.2

0.3

0.4

0.5

0.6

Downstream performance (10,000)

0

1

2

3

4

5

6

7

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Downstream performance (100,000)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Downstream performance (100)

0

1

2

3

4

5

6

7

0.10

0.15

0.20

0.25

0.30

0.35

Downstream performance (250)

0

1

2

3

4

5

6

7

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Downstream performance (500)

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Downstream performance (1,000)

0

1

2

3

4

5

6

7

0.2

0.3

0.4

0.5

0.6

0.7

Downstream performance (10,000)

0

1

2

3

4

5

6

7

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Downstream performance (100,000)

0

1

2

3

4

5

6

7

C.6 Ablation Experiments

We perform the following ablation studies:

Foundational Model Components : We evaluate the impact of Relaxed Representational Constraints (RRC) and Distributed Encoding (DE) on disentanglement performance, as presented in Table 43 . Previous work [ 30 , 33 ] has established a positive correlation between disentanglement and both downstream sample efficiency and overall performance. Therefore, we do not separately assess the effects of these foundational components on downstream tasks.

Auxiliary Model Components : We investigate the influence of additional model components, as detailed in Appendix C.6.2 .

Effect of Soft TPR on Downstream Tasks : Separately, we examine whether Soft TPR offers advantages over its quantised, explicit counterpart, ψ ∗ tpr , in terms of downstream model performance.

C.6.1 Distributed Encoding Ablation

To perform an ablation study on distributed encoding, we set the role embedding matrix M ξ R to the identity matrix. This modification results in maximally sparse role-filler bindings of the form ξ F ( f m ( i ) ) ⊗ e i , specifically:

In this setup, the embedded filler ξ F ( f m ( i ) ) occupies the i -th column of the resulting matrix. Consequently, the TPRs generated by the TPR decoder become degenerate, effectively forming a concatenation of filler embeddings:

This configuration confines each filler's information to specific dimensions within the representation, thereby limiting the model's ability to leverage distributed information across multiple dimensions.

Our experiments reveal moderate reductions in disentanglement performance and increases in loss compared to the distributed encoding scenario. Considering the rotational isomorphism between an identity matrix M ξ R and a (semi)-orthogonal M ξ R , it is somewhat surprising to observe these differences under identical conditions (e.g., initialisation and random seeds). While future work

should aim to learn linearly independent (but not necessarily orthogonal) M ξ R , which would more convincingly account for such results by embedding each filler in overlapping role subspaces and eliminating the rotational isomorphism with the degenerate TPR case, this behaviour warrants further analysis and investigation. One potential avenue of investigation is examining whether the sparsity of binding embeddings ξ F ( f m ( i ) ) ⊗ ξ R ( r i ) affects 'neuron' specialisation (i.e., the tendency of each row of the weight matrix to attend only to certain filler-specific information).

C.6.2 Additional Ablations

We additionally examine the importance of the following properties of our model in producing explicitly compositional Soft TPR representations: 1) the presence of weak supervision, by setting λ 1 = λ 2 = 0 in Equation 7 , 2) the explicit dependency between the quantised filler embeddings and the decoder output, by instead using the Soft TPR to reconstruct the input image, and 3) the semi-orthogonality of the role embedding matrix, M ξ R by removing this constraint in the random initialisation of M ξ R , with the results of these ablations illustrated in Table 42 .

C.6.3 Soft TPR vs TPR

To examine the effect of the Soft TPR on downstream tasks, for each fully trained Soft TPR representation learner, we extract both the Soft TPR, z , and the Soft TPR's corresponding quantised TPR, ψ ∗ tpr , and investigate downstream performance using both types of representation. In all plots, we use the same legend, denoting the explicit TPR as yellow (0) and the Soft TPR as blue (1). Across all considered cases: i.e., 1) convergence rate of representation learning (as measured by the downstream model's ability to effectively leverage representations produced at different stages of training), 2) sample efficiency of downstream models, and 3) raw performance of downstream models in the low sample regime, the Soft TPR confers differential performance boosts compared to the explicit TPR.

Convergence Rate of Representation Learning

10 2

10 3

10 4

10 5

0.0

0.2

0.4

0.6

0.8

RSq

Cars3D

0

1

Iteration count

10 2

10 3

10 4

10 5

Iteration count

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

RSq

Shapes3D

0

1

10 2

10 3

10 4

10 5

0.3

0.4

0.5

0.6

0.7

0.8

0.9

RSq

MPI3D

0

1

Iteration count

10 2

10 3

10 4

10 5

n epochs

0.70

0.75

0.80

0.85

0.90

0.95

Abstract visual reasoning dataset

0

1

Sample Efficiency of Downstream Models

0

1

0.4

0.5

0.6

0.7

Rsq ratio

Sample efficiency (100/all)

0

1

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

Rsq ratio

Sample efficiency (250/all)

0

1

0

1

0.84

0.86

0.88

0.90

0.92

0.94

0.96

0.98

1.00

Rsq ratio

Sample efficiency (500/all)

0

1

0.88

0.90

0.92

0.94

0.96

0.98

1.00

Rsq ratio

Sample efficiency (1,000/all)

0

1

0.96

0.98

1.00

1.02

1.04

Rsq ratio

Sample efficiency (10,000/all)

0

1

0

1

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

Rsq ratio

Sample efficiency (100/all)

0

1

0.4

0.5

0.6

0.7

0.8

Rsq ratio

Sample efficiency (250/all)

0

1

0

1

0.60

0.65

0.70

0.75

0.80

0.85

0.90

Rsq ratio

Sample efficiency (500/all)

0

1

0.75

0.80

0.85

0.90

0.95

Rsq ratio

Sample efficiency (1,000/all)

0

1

0.94

0.96

0.98

1.00

1.02

1.04

Rsq ratio

Sample efficiency (10,000/all)

0

1

0

1

0.45

0.50

0.55

0.60

0.65

0.70

Rsq ratio

Sample efficiency (100/all)

0

1

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

Rsq ratio

Sample efficiency (250/all)

0

1

0

1

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

Rsq ratio

Sample efficiency (500/all)

0

1

0.65

0.70

0.75

0.80

0.85

0.90

0.95

Rsq ratio

Sample efficiency (1,000/all)

0

1

0.80

0.85

0.90

0.95

1.00

Rsq ratio

Sample efficiency (10,000/all)

0

1

0

1

0.200

0.225

0.250

0.275

0.300

0.325

0.350

0.375

0.400

Accuracy ratio

Sample efficiency (100/100,000)

0

1

0.25

0.30

0.35

0.40

0.45

Accuracy ratio

Sample efficiency (250/100,000)

0

1

0

1

0.25

0.30

0.35

0.40

0.45

Accuracy ratio

Sample efficiency (500/100,000)

0

1

0.30

0.35

0.40

0.45

0.50

0.55

0.60

Accuracy ratio

Sample efficiency (1,000/100,000)

0

1

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

Accuracy ratio

Sample efficiency (10,000/100,000)

0

1

Low Sample-Regime Performance of Downstream Models

0

1

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

RSq

Downstream performance (100)

0

1

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

RSq

Downstream performance (250)

0

1

Downstreamperformance(1,0oo)Downstreamperformance(10,0oo)Downstreamperformance(all)

0.92

0.96

0.96

0.90

0.94

0.94

0.88

0.92

0.92

80.86

bs

RS

0.90

0.90

0.84

0.82

0.88

0.88

0.80

0.86

0.86

0.78

0.84

0.84

0

1

0

1

0

1

0

1

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

RSq

Downstream performance (100)

0

1

0.4

0.5

0.6

0.7

0.8

RSq

Downstream performance (250)

0

1

0

1

0.75

0.80

0.85

0.90

0.95

RSq

Downstream performance (1,000)

0

1

0.94

0.96

0.98

1.00

1.02

1.04

RSq

Downstream performance (10,000)

0

1

0.96

0.98

1.00

1.02

1.04

RSq

Downstream performance (all)

0

1

0

1

0.35

0.40

0.45

0.50

0.55

0.60

RSq

Downstream performance (100)

0

1

0.45

0.50

0.55

0.60

0.65

0.70

RSq

Downstream performance (250)

0

1

0

1

0.55

0.60

0.65

0.70

0.75

0.80

RSq

Downstream performance (1,000)

0

1

0.65

0.70

0.75

0.80

0.85

RSq

Downstream performance (10,000)

0

1

0.75

0.80

0.85

0.90

RSq

Downstream performance (all)

0

1

0

1

0.175

0.200

0.225

0.250

0.275

0.300

0.325

0.350

Downstream performance (100)

0

1

0.200

0.225

0.250

0.275

0.300

0.325

0.350

0.375

Downstream performance (250)

0

1

0

1

0.25

0.30

0.35

0.40

0.45

0.50

Downstream performance (1,000)

0

1

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

Downstream performance (10,000)

0

1

0.775

0.800

0.825

0.850

0.875

0.900

0.925

0.950

Downstream performance (100,000)

0

1

C.6.4 Robustness to Hyperparameter Choices

We perform an additional experiment to empiricially verify that our model is robust to different hyperparameter choices. For the MPI3D dataset, which disentanglement models experience the greatest difficulty in producing disentangled representations for (see Table 1 ), we randomly pick another set of hyperparameters, shown in Table 43 , from the top 5 models based on the MSE loss criterion. For this randomly chosen hyperparameter configuration, we evaluate the disentanglement of the representations on the MPI3D dataset produced by the resulting model.

D Limitations and Future Work

D.1 Extension to Linguistic Domains

Applying the Soft TPR to the TPR's typical domain of language is an intriguing future direction, especially as language can deviate from strict algebraic compositionality - for instance, idiomatic expressions such as 'spill the beans' cannot be understood as a function of their constituents alone. Soft TPR's more flexible specification allows it to capture approximate forms of compositionality precluded from the TPR's strict algebraic definition (Eq 1 ), thereby potentially providing the Soft TPR the ability to better handle the nuance and complexity of language.

To adapt our framework to language, we replace our Conv/Deconv encoder/decoders with simple RNNs, retrain our TPR decoder, and remove the semi-supervised loss, using Eq 6 as the full loss. Preliminary results in Table 45 on the BaBI [ 7 ] dataset are compared with TPR baselines from AID [ 51 ]. Our Soft TPR Autoencoder does not presently surpass AID, but notable points include:

Our use of simpler RRN-based encoders and an MLP-based downstream network, unlike the more sophisticated architectures of [ 51 ].

Soft TPR retains its performance improvement above the corresponding explicit TPR it can be quantised into.

The smaller gap between systematic vs non-systematic dataset splits in our model compared to TPR-RNN (+AID) and FWM.

We train our representation learner using self-supervision (reconstruction loss) alone, only employing supervision on the downstream prediction network, while the baselines employ strong supervision and end-to-end training to produce representations.

D.2 Need for Weak Supervision

To produce a compositional representation, ψ ( x ) = C ( ψ 1 ( a 1 ) , . . . , ψ n ( a n )) (as in Section 3.1 ), each representational constituent, ψ i ( a i ) , must map 1-1 to a data constituent, a i . Without supervision (i.e., access only to observational data, { x i } ), this is challenging, because the data constituents { a i } underlying each object, x , are unknown and cannot be identified.

We can frame the above intuition in a more mathematically rigorous way, using the generative framework. Essentially, as formally proved in [ 25 ], it is impossible to identify the true distribution for the data constituents (generative factors), p ( a ) , using observational data, { x i } , alone as there are infinitely many bijective functions f : supp( a ) → a such that: 1) a and f ( a ) are completely entangled (i.e. non-diagonal Jacobian) and 2) the marginal distributions of a and f ( a ) are identical (meaning the marginal distributions of the observations are also identical, i.e., ∫ p ( x | a ) p ( a ) da = ∫ p ( x | f ( a )) p ( f ( a )) da ). Thus, without inductive biases, it is impossible to infer the data constituents { a i } of any observation, x , from observational data { x i } alone.

To combat this non-identifiability result, in line with [ 13 , 22 , 31 , 33 , 35 ], we use weak supervision, presenting the model with data pairs ( x, x ′ ) where x and x ′ differ in a subset of FoVs, e.g. β ( x ) = { shape / cube , colour /purple, size / large } , β ( x ′ ) = { shape / cube , colour /cyan, size / large } , where the FoV types corresponding to the different FoV values are known to the model. Note that this weak supervision is minimal, only providing the model to access to the differing FoV types in an index set, I , (i.e., I := { colour } ) and not any of the FoV values (i.e., { cube , purple , cyan , large } are all not known by the model).

Some possible future extensions to reduce this level of weak supervision, or alternative forms of weak supervision include:

Embodied Learning : In the visual domain, some roles, e.g. object position correspond to affordances. Embodied agents may be able to reduce the need for explicit supervision by collecting ( x, x ′ ) and I through interaction with their environment.

Pretrained Filler Embeddings : Initialising the filler embedding matrix, M ξ F with embeddings learnt by a pre-trained vision model could impart knowledge of domain-agnostic fillers (e.g., colours, shapes), reducing the need to explicitly provide ( x, x ′ ) and I to the model.

Segmentation Masks : Segmentation masks for each object may potentially reduce the need to explicitly provide ( x, x ′ ) and I to the model.

D.3 Downstream Utility

Our investigation of downstream utility centers on two selected tasks - FoV regression/classification and abstract visual reasoning, which aligns with the standard framework for assessing the quality and downstream utility of compositional representations [ 26 , 30 , 33 , 40 , 44 , 46 ]. While existing work [ 30 , 33 ] demonstrates that explicitly compositional representations enhance downstream sample efficiency compared to non-compositional representations, a result we improve upon ( C.5.1 ), the broader utility of compositional representations remains a topic of ongoing exploration [ 21 , 40 , 42 , 44 ].

Theoretical perspectives [ 1 , 2 ] argue that explicitly compositional representations are fundamental in the production of productive, systematic, and inferentially coherent thought - 3 key properties characterising human cognition. Investigating how explicitly compositional representations can yield empirical benefits across these dimensions represents an essential avenue for future research. Although preliminary studies [ 40 , 42 , 44 ] do not find strong evidence that explicitly compositional representations improve compositional generalisation (a key aspect of systematicity), [ 44 ] suggests that this finding is because compositional representations are necessary, but not sufficient to induce systematicity; an explicitly compositional processing approach [ 6 ] is also required.

Future work could extend our theoretical framework with the hope of producing empirical results consistent with the theoretical arguments of [ 1 , 2 ]. In particular, as our unbinding module is designed to provably and efficiently recover structured role-filler constituents from the Soft TPR, it may be possible to exploit this module to systematically reconfigure roles and fillers from existing representations to create representations of novel combinations of role-filler bindings (i.e., novel

compositional data). This type of ability could prove extremely beneficial in areas such as concept learning and compositional generalisation.

D.4 Dimensionality

In this subsection, we denote the dimensions of the role and filler embedding spaces as D F and D R respectively. We also denote the number of fillers as N F and the number of roles as N R .

The Soft TPR belongs to V F ⊗ V R , which is a D F × D R dimensional space, which grows multiplicatively in D F , D R . Several factors, however, mitigate scalability concerns in light of this fact:

Independence of Embedding Space Dimensionality : We note that the dimensionality of the role and filler embedding spaces ( D R , D F respectively) are properties of the corresponding embedding functions ( ξ R : R → V R and ξ F : F → V F ) and thus, can be fixed independently of the number of roles, N R , the number of fillers, N F , or the number of total role-filler bindings (which we denote by n ) within a domain. Thus, it is possible to fix the Soft TPR's dimensionality ( D F × D R ) to be smaller than N F × N R (the number of roles/FoV types multipled by the number of fillers/FoV tokens) or n (the number of bindings), which all may be large in complex visual domains. As illustrated in Table 46 , the dimensionality of the TPR is smaller than N F × N R in both the Shapes3D and MPI3D domains.

Relaxing Orthogonality : While D F can be set a priori with no regard to N R , N F or n , we require D R ≥ N R for semi-orthogonality of the role-embedding matrix M ξ R , which guarantees faithful (see proof 2 of A.2 ) and computationally efficient (see 'Unbinding' heading of B.4.1 ) recoverability of constituents. It is, however, possible to relax this constraint (i.e., to have D R < N R ) to further reduce dimensionality. In this case, semiorthogonality of M ξ R is impossible and hence the recoverability of constituents cannot be guaranteed, however, there are some less stringent guarantees on the outcome of unbinding that can still be derived (see p291 of [ 3 ] for more details).

We also more explicitly compare the dimensionality of the Soft TPR with baselines in Table 47 . Scalar-tokened symbolic representations have a low dimensionality of 10 ( N R ) at the expense of representational expressivity (each representational constituent ψ i ( a i ) is scalar-valued). In contrast, Soft TPR has vector-valued representational constituents (i.e. ≈ ξ F ( f m ( i ) ) ⊗ ξ R ( r i ) ), similar to VCT and COMET. When compared to these models, the Soft TPR has significantly lower dimensionality compared to VCT and is comparable with COMET.

D.5 Computational Cost

In the Soft TPR Autoencoder, the expensive tensor product operation is employed to generate ψ ∗ tpr . Given the computational cost of the tensor product, we more concretely examine the computational

cost of training the Soft TPR Autoencoder by computing the FLOPs for a single forward pass on a batch size of 16 using the open-source implementation of fvcore https://github.com/ facebookresearch/fvcore/tree/main/docs , visible in Table 48 . This data demonstrates that, despite the tensor product's computational cost, the mathematically-informed derivation of our model allows it to obtain compositional representations with vector-valued representational constituents at a significantly lower cost compared to relevant, vector-tokened baselines (2 orders of magnitude less than VCT, and 4 orders of magnitude less than COMET).

Future work could explore the use of tensor contraction techniques to reduce computational expense. For instance [ 29 ] uses a Hadamard product based tensor product compression technique. This reduces computational cost from n 2 (tensor product of 2 vectors) to n (Hadamard product), but comprises the theoretical guarantees on constituent recoverability. We believe developing tensor contraction techniques within the TPR framework is an important direction for future research, to ensure efficient TPR-based representations with provable recoverability of constituents.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims accurately reflect the paper's contributions and scope. We provide theoretical proofs in the Appendix, as well as our complete suite of experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We explicitly state in the last paragraph of the paper that future work is to be done in developing hierarchical Soft TRP representations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We include complete proofs for all theoretical results in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all information (specification of model architecture, computing resources, hyperparameters) to replicate experimental results in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All datasets are open-access. We provide sufficient instructions in our Appendix to reproduce experimental results. The code base is also public.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide these details in the Appendix, and additionally, in the code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We present results with standard deviations, as well as the IQR and whiskers extending to 1.5 times the IQR for all box plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialisation, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specify the specific GPUs we use for all experiments in the Appendix, as well as the average amount of time it takes to train the Soft TPR Autoencoder for each dataset.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: As the central aim of our work relates to the representational form of compositional representations, our research is theoretical in nature and has no immediate societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In the main paper, as well as the Appendix, we cite the papers associated with each dataset we use. We additionally make explicit references in the Appendix to all externally created code that we use in our experiments. Furthermore, if the paper is to be accepted, we will clearly provide licenses and attribution to the original authors where applicable in the code files.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our open access code is reasonably straightforward to understand, especially when supplemented by the Appendix of this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our research does not involve crowdsourcing nor human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose the use of the tensor product to model the interactions between object properties and their values, in contrast to the usual concatenation-based fusion for compositional representations. Extensive experiments on a large variety of image datasets are performed, where performance gains are often significantly higher than those of the baselines considered.\n\nSTRENGTHS:\n- Modeling “multiplicative interactions” has proven to be a powerful mechanism for deep learning [1], but is largely neglected in modern times. This paper proposes an interesting way of fusing compositional representations of object properties and their values through the tensor product (and subsequent summation). Beyond the TPR’s success in this paper, I imagine the insights here through the proposed model form could help the design of more expressive deep learning architectures in future work more generally.\n- The experiments are exhaustive, the technical exposition clear and crisp, and for the most part, every architectural design and decision is justified in great detail. Overall, the paper is remarkably digestible given the technical sophistication, yet offers many insights.\n\n---\n\n- [1]: Jayakumar, Siddhant M. et al. “Multiplicative Interactions and Where to Find Them.” *International Conference on Learning Representations* (2020).\n\nWEAKNESSES:\n# [W1] Multiplicatively large dimensionality of the TPR representations & issues scaling to larger settings\n\nMy main criticism of the paper is the resultant TPR's dimensionality. In particular, the soft TPR representations live in a $D_f\\cdot D_r$ dimensional space. Due to the use of the Kronecker product, the dimension of the TPR representations grows multiplicatively with the two terms. \n\nWhilst the datasets studied in the paper are relatively simple and only 10 factors of variation are modeled, it seems prudent to acknowledge that the TPR size could grow prohibitively large for increased values of $D_f, D_r$ for more complex datasets outside of controlled settings, where significantly larger number of FoVs exist. In particular, the NeurIPS checklist states: `The authors should discuss the computational efficiency of the proposed algorithms`.\n\nFor example, I see from [L1008] that $N_R:=10$, and $N_F$ is as high as $106$ for Cars3D. Even in this regime with a very small number of roles, the TPR representations are (presumably?) larger than the concatenation-based representations of the baselines. A comparison of FLOPs/multiply-adds needed for the compared methods would be appreciated to better understand the methods’ drawbacks through the use of the (often computationally expensive) tensor product.\n\nQUESTIONS:\n**[Q1]** As expanded upon in [L933], the “role” embedding matrix $M_{\\xi_R}$ is initialized with $n$ orthogonal columns but not trained. We are told that it is “intuitive” that role embeddings do not need to convey semantic information, but I think it would be interesting to see an ablation study on this.\n\nIn particular: if it is the case that the embedding matrix does not need to learn semantic representations, why not make the simpler choice of initializing $M_{\\xi_R}$ as a (truncated) identity matrix (this shares the same property of semi-orthogonality)? The decision not to make this parameter learnable is peculiar, and I would be interested in seeing its impact explored experimentally (for example, through the DCI score as in Table. 7).\n\nLIMITATIONS:\nLimitations are discussed well throughout (and in further detail throughout the appendix), but an extra discussion of the potential drawbacks of increased computational costs should be made.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work explores compositional representations -- considered a crucial capability underlying intelligent human behaviour in deep learning systems. It argues that there is an incompatibility between discrete symbolic compositional representations—e.g. as obtained through traditional disentanglement approaches—and the continuous vector spaces underlying deep learning systems. To address this, the authors introduce a novel continuous compositional representation that builds on the Tensor Product Representation (TPR) approach, akin to a soft approximation to the TPR. They introduce a method for learning soft TPRs with weakly supervised data called the Soft TPR Autoencoder, and apply it to visual representation learning. This model demonstrates state of the art disentanglement for representation learning and improved sample efficiency for downstream models using those representations.\n\nSTRENGTHS:\n- importance: compositional representation learning, and compositional generalisation and sample efficiency is an important and relatively under-explored area in deep learning research. By focusing on an approach for representation that aims to be more compatible with deep learning this work makes important contributions to this area.\n- novelty: the method presented is interesting & new but builds on the well established TPR framework.\n- clarity: the method is explained well and in enough detail to understand and reproduce it.\n- evaluation: the paper provides a thorough evaluation of the model, including comparisons with the relevant baselines, convergence rates, and performance in low sample regimes. The better performance on downstream tasks w.r.t. number of training examples is particularly interesting and supports some of the motivations for compositional representation.\n - interested to see where this work might develop in future work exploring hierarchical compositional representations.\n\nWEAKNESSES:\n- motivation for the approach & model could be unpacked a bit more (see questions)\n- other domains: the authors focus on applying the work to the visual data, outside the typical domain of TPR models. In general, a strong feature, enabling them to tackle the messier world of complex visual data (and weaker super vision) and compare the model to the many disentanglement approaches that have previously been applied in the visual domain. However, it could be interesting to see how the _Soft_ TPR approach compares to traditional TPR in its typical domain of language.\n - no high impact is shown for downstream utility. the improvements seen for downstream tasks are certainly interesting, particularly in the low data regime, however only two tasks are explored. are there any more good downstream tasks to evaluate the utility of their learned representations? could there even be more speculation as to where future work might really leverage what can be learned with this approach?\n - need for supervision (albeit weakly): could the work be extended to use different forms of supervision or less/no supervision. noting that their ablation showed the importance of the supervision, perhaps the authors could explore more variants on this ablation, and speculate on extensions of the model that could reduce the need for supervision?\n\nQUESTIONS:\nA couple of questions, both on the motivations of the approach in the paper and how it could be improved. In both cases, these are more questions focused on the conceptual motivations and understanding than what the results show:\n - In terms of motivating the approach, can you be more specific about how representations obtained with traditional disentanglement approaches (typically continuously-valued themselves) could create an incompatibility with the continuous vector space of deep learning systems, and lead to suboptimal performance for the representation learning and downstream models? Any hypothetical or concrete examples (and/or references discussing this point)? \n- Relative to TPR, what is the relaxation in Soft TPR supposed to be helping with, and why? The manifold example is given, e.g. in Figure 1c, but seems quite vague. Also discussed is that the data being explained maybe only approximately compositional, but why would TPR fail with this and why would we specifically expect soft TPR to help? Could this be unpacked more, perhaps with some concrete examples.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for representation learning known as Soft Tensor Product Representations (Soft TPR), aimed at capturing the compositional structure of data more effectively. The authors propose a continuously-valued compositional representation that contrasts with traditional symbolic methods. The paper makes several key contributions in the realm of representation learning, including the conceptualization of Soft TPR, the Soft TPR Autoencoder, and the demonstration of the benefits of this representation for both representation learning and downstream models.\n\nSTRENGTHS:\nThe idea of Soft TPR brings a fresh perspective to the field of representation learning, offering a new way to represent compositional structures in a continuous manner. It provides a thorough theoretical exploration of Soft TPR, complete with detailed mathematical proofs and framework extensions. The paper clearly articulates the differences between Soft TPR and existing methods, holding significant potential for enhancing model interpretability and improving robustness to covariate shift.\n\nWEAKNESSES:\n1. No MPI scores for disentanglement (in Table 1) are reported. \n2. The construction of the model's loss function involves a multitude of hyperparameters, suggesting that the model might require a complex and intricate tuning process. Compared to other models like VCT, the process of adjusting parameters to achieve optimal results could involve additional complexity and such a requirement might pose challenges in terms of computational resources and time.\n3. The reason why *Soft TPR is better than TPR is not fully explored.\n\nQUESTIONS:\n1. what is the dissentanglment performance of TPR?  The paper only report the downstream results of TPR in Table 6, is it possible to report the disentangment performance of TPR in Table 1?\n2. how many pairs are provided for the weakly supervised? For example, in Table 4, the numbers are the total samples for training, or the labeled sample pairs?\n3. VCT uses attention-based operations (cross-attention and self-attention) to extract concept vectors, and reconstruct images from the concept tokens, what is the difference between those attention-based operation and the unbinding module  or TPR constructor in this paper?\n\nLIMITATIONS:\nI don't see any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}