Fully Distributed, Flexible Compositional Visual Representations via Soft Tensor Products
Authors: Bethia Sun
Maurice Pagnucco
Yang Song
Venue: NeurIPS 2024 poster (2024)
Keywords: representation learning
compositional representations
disentanglement
tensor product representations
compositionality
Primary area: machine_vision

Abstract: Since the inception of the classicalist vs. connectionist debate, it has been argued that the ability to systematically combine  symbol-like entities into compositional representations is crucial for human intelligence. In connectionist systems, the field of disentanglement has gained prominence for its ability to produce explicitly compositional representations; however, it relies on a fundamentally *symbolic, concatenative* representation of compositional structure that clashes with the *continuous, distributed* foundations of deep learning. To resolve this tension, we extend Smolensky's Tensor Product Representation (TPR) and introduce *Soft TPR*, a representational form that encodes compositional structure in an inherently *distributed, flexible* manner, along with *Soft TPR Autoencoder*, a theoretically-principled architecture designed specifically to learn Soft TPRs.  Comprehensive evaluations in the visual representation learning domain demonstrate that the Soft TPR framework consistently outperforms conventional disentanglement alternatives -- achieving state-of-the-art disentanglement, boosting representation learner convergence, and delivering superior sample efficiency and low-sample regime performance in downstream tasks. These findings highlight the promise of a *distributed* and *flexible* approach to representing compositional structure by potentially enhancing alignment with the core principles of deep learning over the conventional symbolic approach.

Fully Distributed, Flexible Compositional Visual Representations via Soft Tensor Products
Bethia Sun"
Maurice Pagnucco
Yang Song
School of Computer Science and Engineering UNSW Sydney
Abstract
Since the inception of the classicalist vs. connectionist debate, has been argued that the ability to systematically combine symbol-like entities into compositional representations is crucial for human intelligence. In connectionist systems, the field of disentanglement has gained prominence for its ability to produce explic- itly compositional representations: however, it relies on a fundamentally symbolic, concatenative representation of compositional structure that clashes with the contin- wous, distributed foundations of deep learning. To resolve this tension, we extend Smolensky's Tensor Product Representation (TPR) and introduce Soft TPR, rep- resentational form that encodes compositional structure in an inherently distributed, flexible manner, along with Soft TPR Auloencoder, theoretically-prin prrinciple archi- tecture designed specifically to learn Soft TPRs. Comprehensive evaluations in the visual representation learning domain demonstrate that the Soft TPR framework consistently outperforms conventional disentanglement alternatives achieving state-of-the-ee disentanglement, boosting representation learner convergence, and delivering superior sample efficiency and low-sample regime performance in down- stream tasks. These findings highlight the promise of distributed and flexible approach to representing compositional structure by potentially enhancing align- ment with the core principles of deep learning over the conventional symbolic approach.
Introduction
Compositional structure, capturing the property of being decomposable into a of constituent parts, permeates numerous aspects of our world - from the recursive application of syntax in language, to the parsing of richly complex visual scenes into their constituent parts. Given this ubiquity, is natural to seek deep learning representations that also embody compositional structure. Indeed, empirical evidence demonstrates a multitude of benefits conferred by explicitly compositional representations, including increased interpretability [10, 12], reduced sample complexity [30, 33], increased fairness [20, 25, 41], and improved performance in distribbutoon generalisation [33, 48, 50).
We consider the following, intuitive notion of compositional representations. A representation of compositionally- structurrd data is compositional representation if its structure explicitly reflects the constituency structure of the represented data [49]. In the visual representation learning domain, data clearly composiiionallyssraauuee
images can be decomposed into of constituent factors of variation (FoVs), e.g., (magenta floor, orange wall, aqua object colour, oblong object shape} for the image Figure 1.
"Correspondence to: bethia. sun@unsw... edu. Code available at: hatps://githhn com/ htt.u/b..coo//ooo////oo/// got.oo//ootttttpegeeeeee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A widely explored framework for learning explicitly compositional representations is that of dis- entanglement. We adopt the conventional [5. 25, 33, 43], intuitive definition of a disentangled representation, which states a representation, 1z(z), is disentangled if each data constituent (FoV) can be mapped onto a distinct dimension or contiguous group of dimensions in (xx)), effectively establishing a 1-1 correspondence between Fo and distinct representational parts [43]. Framed in this way, it is clear that disentangled representations explicitly encode the data's constituency structure and are thus, compositional representations by nature. The majority of state-of-the-art disentanglement approaches use a variational autoencoder backbone, and rely on weak supervision [13, 22, 31, 33, 35], or a penalisation of the aggregate posterior f g(2[2; (p dz [10, 14, 17, 24, 26, 32] to promote disentanglement. More recent approaches depart from the restrictive assumptions of variational framework, and instead use standard autoencoding [47]. or soergy-functinn based optimisation [36], with additional inductive biases to encourage disentanglement. Despite their methodological diversity, disentanglement methods share a unifying principle: by enforcing the 1-1 correspondence between Vs and distinct "slots" in the representation, they produce compositional representations dimension-wise concatenation of scalar-valued or vector-valued FoV tokens [10, 13, 14, 17, 22, 24, 26, 31, 32, 33, 35, 36, 47], as illustrated in Figure la. This design utilises a fundamentally localist encoding scheme, where discrete parts of the representation are exclusively dedicated to encoding single Vs,s, paralleling symbolic representations, in which discrete repre- sentational slots are occupied by individual symbols - e.g., the symbols f'c'. 'a'' 't'} in the word 'cat'. We theorise that this inherently symbolic method, offering a localist encoding of compositional structure, may be misaligned with the continuous, distributed nature of deep learning models for the following reasons (see A.3 for further details):
1. Gradient Flow and Learning: Symbolic compositional representations utilise localist en- coding approach, assigning each FoV to unique, non-overlapping subset of the dimensions of the representation. This modular separation may introduce practical misalignments with gradient-based optimisation. By restricting each FoV to its own slot, gradients for one FoV are confined to a set of designated dimensions, limiting the smooth propagation of gradients globally across all dimensions of the representation. Consequently, updates to a single VoV provide minimal feedback to other FoVs, hindering the model's ability to jointly refine interdependent components, Furthermore, the localist encoding where each FoV occupies a small, disjoint subset of the total dimensions can induce abrupt, discontinuous shifts in the representation when transitioning between FoV updates, potentially complicating convergence.
2. Representational Expressivity: The localist encoding inherent in symbolic compositional structures, where each FoV is assigned a distinct subset of the overall dimensions, may constrain practical expressiveness. Specifically, confining each of the n FoVs to subset of size n/d within a d-dimensional representation causes each FoV to underutilise the full capacity of the d-dimensional space, potentially limiting the richness and flexibility of the learned representations.
3. Robusiness to Noise:
By confining each FoV to a small, non-overlapping subset of dimen- sions, symbolic compositional representations become highly vulnerable to dimension- wise noise. Even slight perturbations in a single dimension can significantly impair the repre- sentation of the corresponding Fo there is no overlapping redundant encoding to mitigate such disruptions.
Critically, we hypothesise that this potential incompatibility between symbolic, localist representations and the distributed, continuous nature of deep learning results in suboptimal behaviour in models that learn or use these representations. To overcome these limitations, we are motivated to pursue an inherently distributed approach to representing compositional structure. Instead of concatenating discrete slots (FoV tokens) dimension wise to form the compositional representation, an inherently distributed approach continuously combines densely encoded FoVs within a unified vector space. This design allows information from each FoV to be smoothly interwoven throughout all dimensions of the representation, as illustrated in Figure Ib, potentially resulting in smoother gradient flow, enhanced expressivity, and heightened robustness to noise. By reconciling the demand for representations that are explicitly compositional with the continuous, distributed nature of deep learning, distributed compositional representations offer a compelling alternative to traditional symbolic, slot-based paradigms.
2

---

Factors of variation
Compositional representations
TPR vs Soft TPR
a)
b)
C)
Floor colour
Object size
Wall colour
Object shape
Object colour
Orientation
Figure I: (a) Disentangled representations can he conceptualised as a concatenation of FoV tokens (coloured blocks), enforcing a symbolic, string-like compositional structure, where each FoV is allocated to a discrete slot in the representation. We instead, consider distributed representation of compositional structure, (b), where information from densely encoded FoV (first waves) continuously combined together form the representation, vi((:) (in red), effectively distributing the information from multiple FoVs into a single dimension of (xx).. (c) Only a subset of points (stars) in the underlying representational space (rainbow manifold) satisfy the TPR specification. The Soft TPR relaxes this, capturing larger, continuous regions of the underlying representational space (the translucent circles), while approximately preserving the TPR's key properties.
Pioneered by Smolensky, the Tensor Product Representation [3] a specific representational form that encodes compositional structure in an inherently distributed manner. At the crux of it, TPRs are formed by continuously blending representational components together into the overall representation, in manner analogous to superposing multiple waves together to produce a complex waveform, as illustrated in Figure 1b. For a representation to qualify as a TPR, it must adhere to a highly specific mathematical form, which confers upon the TPR valuable structural properties (elaborated on in Section 3.2), but also imposes two limitations (see B. for further details). First, as depicted by the stars in Figure Ic, only discrete subset of points in the underlying representational space, V. satisfies the stringent mathematical criteria to qualify as TPRs. Consequently, to learn TPRs, representation learners must map from the data manifold onto this discrete subset, which constitutes highly constrained and inherently challenging learning task. Second, the TPR specification enforces a strict, algebraic definition of compositional structure, limiting the TPR's s ability to flexibly represent ambiguous, real-world data which is often quasi-compositional only approximalely adhering some rigid, formal definition. Historically, these limitations have confined TPR learning to formal domains characterised by explicit, algebraic structure as evidenced by the exclusive deployment of TPRs in language [19, 23, 28, 34, 52] - and, to contexts where strong supervision from highly structured downstream tasks is available steer the representation learning process [23, 28, 38, 51]. To negate these draw bawks and extend distributed compositional representations to weakly supervised, non-formal domains, we propose Soft TPR, which can be thought of as a continuous relaxation of the traditional TPR, as illlustrated by the translucent circular regions in Figure lc. At its core, the Soft TPR is designed to promote representational flexibility and ease of learning while simultaneously preserving the structural integrity of the traditional TPR. We udditionally introduce architecture for learning Soft TPRs, used to operationalise the Soft TPR framework in the visual representation learning domain.
Soft TPR Autoencoder, a theoretically lyyreiccally-pricppleed
Our main contributions are threefold: i) We propose a novel compositional representation learning framework, introducing the distributed, flexible Soft TPR compositional form, alongside a dedicated, weakly-supervised architecture, Soft TPR Autoencoder, for learning this form. ii) Our framework is the first to learn distributed compositional representations in the non-formal, less explicitly algebraic domain of vision. iii) We empirically affirm the far-reaching benefits produced by the Soft TPR framework, demonstrating that Soft TPRs achieve state-of oh-the-art disentanglement, accelerate representation learner convergence, and provide downstream models with enhanced sample efficiency and superior low-sample regime performance.
Related Work
Disentanglement: In aiming produce explicitly compositional representations without strong supervision, our work shares the same objective as disentangled representation learning. Prior to the highly influential work of [25] which proved the impossibility of learning disentangled representations without supervision or other inductive biases, disentangled representations were learnt in a completely unsupervised fashion [8, 10, 14, 17, 24, 26, 37]. Our use of weak supervision inspired by the work [13, 22, 31, 33, 35] relating to this highly influential impossibility result. In particular, we leverage the type of weak supervision termed 'match pairing' [35], where pairs, x'), differing in values for a subset of known Fs are presented to the model, to incentivise disentanglement. Our work.
3

---

however, fundamentally diverges from all disentanglement work we are aware of, by adopting an inherently distributed representation of compositional structure, which contrasts with the inherently symbolic, localist representations of compositional structure characterising existing work.
TPR-based Work: Existing TPR-based approaches generate distributed representations of compo- sitional structure by producing an element with the explicit mathematical form of TPR. To learn this highly specific form, these approaches rely on the algebraic characterisation of compositionality present in formal domains, such as mathematics [38], or language [19, 23, 28, 34, 52] in addition to strong supervision signals from highly structured downstream tasks, such as part-of-speech tagging [23], and answering structured language [28, 51] or mathematics questions [38]. In contrast, Soft TPR eases these stringent constraints by offering a continuously relaxed specification of compositional structure. This allows our approach to extend distributed representations of compositional structure to the orthogonal and less algebraically structured domain of visual data, while also reducing reliance on annotated data by instead using weak supervision to learn this relaxed representational form.
3 Preliminaries
3.1 A Formal Framework for Compositional Representations
We adopt a generalised, non-generative version of the definition of compositional representations from [49]. Let data X be compositionally-struuua strrucuurdd there exists a decomposition function 3 : X + Ai X x An decomposing into constituent parts (FoVs), i.c. B(x) = {a1..... ) where a; E A.. mapping & X + VF then produces a compositional representation if (x) C(bb (a.):..., Vn((a,) (),,),, where each th : A, + V; is a component function that independently embeds a part a; (i.e., a particular FoV) into a vector space, and C Vi X ... x V, - Vi is a composition function that combines these embedded parts to form the overall representation. This construction enforces a faithful correspondence between the constituency structure of the data {a;} and that of the representation, {+b;(a)) (a,)} (provided C' is invertible).
We formalise a symbolic compositional representation, vhs (ar), as one in which C' is given by
concatenation operation. Concretely, 1 = (1h1 (a,) ..... the (an))) 17) Indeed, the disentangle- ment approaches of [8, 10, 13, 14, 17, 22, 24, 26, 31, 32. 33, 35, 36, 37, 47] all fit this framework, concatenating scalar or vector-valued FoV tokens together to produce the representation. While this design may be fundamentally at odds with the continuous, distributed nature of deep learning (as detailed in Section 1), it provides one clear benefit: the embedded FoVs, (w)((a)) are trivially recoverable by partitioning (x))
In many scenarios, readily recovering the data constituents (i.e., FoVs), { from the compositional representation (x()) is essential for practical utility. Here, we assume each v is invertible, so direct recoverability of the data constituents, {a:}. from the representation (xx) is guaranteed if the embedded Vs,s, (a;)), remain structurally separable in the representation, (xx).. Motivated by this requirement, we aim to construct compositional representations that: 1) achieve an inherently distributed encoding of compositional structure by smoothly interweaving FoVs throughout all dimensions of (xx), and 2) preserve the direct recoverability of the embedded FoVs.
3.2 The TPR Framework
Smolensky's Tensor Product Representations (TPR) [3] provides compelling realisation of an inherently distributed encoding of compositional structure that, under certain conditions, preserves the direct recoverability of the embedded FoVs. We briefly review key aspects here, with additional details and formal proofs deferred to Appendix A. The TPR framework conceptualises a compositionally- structured object, x, comprising a number, Nn. of roles, where each role R is bound to corresponding filler f E F, with this binding being denoted by f/r. Thus, compositionalll-struuuuue
objects are decomposeable into a set of role-filler bindings, ((x) = (fs/rr1." N In natural language, where the TPR is most commonly deployed [19, 28, 34, 52], roles may correspond to grammatical cat- egories (e.g., noun) and fillers specific words (e.g., cat). Translating this role-filler formalism to the visual domain, we reinterpret roles as FoV types (e.g., floor colour), and fillers as FoV values (e.g., blue). Using this role-filler formalism to decompose the Shapes3I image in Figure I, we have ((x) = (magenta/ floor colour, orange/ wall colour, aqua/object colour, large/ob object size, oblong/object shape)

---

To construct a TPR, the roles and fillers for each binding in a are independently embedded via role and filler embedding functions, εR: : R - &F F - Vp respectively. The binding f /r is then encoded by taking a tensor product (denoted by ()) over the embedded role, En(r), and embedded filler, (r(f). Summing over the embeddings for all role-filler bindings yields the TPR, Vtpr (x):
Vtpr (x) = Σ EF @ Ex(T.).
(1)
where m : fI....,NNp + {1,...,,,, Nr) is matching function? associating each of the Nn roles to the filler it binds to in the decomposition of x (i.e., (xx) = (fm(s)/ra)).
We now situate the TPR within the formal framework of Section 3.1, by observing that it defines each representational component, (a,), as an embedded role-filler binding, {F(s((s) & (R(Ti), and adopts ordinary vector space addition as its composition function, C. Unlike the concatenation in symbolic, localist schemes, the TPR additively superposes all embedded Fo in the same underlying vector space, producing an inherently distributed representation of compositional structure in which information from multiple role-fill filler bindings can be smoothly interwoven into the same dimensions."
A natural question arises from defining C' as ordinary vector space addition: can we recover each representational component - the embedded role-filler binding, &, {i((())) & En((r)) - from the TPR, which is their sum? Remarkably, despite the non-injectivity of the summation operation, the TPR's specific algebraic structure enables faithful recoverability of all representational components from the TPR, through a process referred to as unbinding (see A.2 for more details). More concretely, provided that the role embedding vectors {{R(r.)} are linearly independent and known, the embedded filler bound to the i-th role can be unbound from the representation, thppe (I), product between (igr(() and the i-th unbinding vector, u,:
(xr), by taking a (tensor) inner
Σ Er (F((m(()) @ En(ri)) 24 = =FF(fm()))
Vipr (z))4=
where is vector corresponding to the i-th column of the (left) inverse of the matrix obtained by taking all (linearly independent) role embeddings as columns. Repeating this unbinding procedure using each of the NR unbinding vectors recovers each of the embedded fillers bound to the Nn roles, and thus, the entire set of embedded role filler bindings, {fF(fm @ [R(r:)} Hence, provided that the linear independence of the roles holds, the TPR offers an inherently distributed representation of compositional structure that nonetheless retains the chief benefit of symbolic approaches direct recoverability of each representational component.
4 Methods
4.1
Soft TPR: An Extension to the TPR Framework
While Tensor Product Representations offer a robust framework for encoding compositional structure in distributed manner, their rigid representational form imposes practical limitations. The core limitation lies in TPR's strict algebraic specification of compositional structure as a set of discrete bindings, each of which comprise single role and single filler. This specification precludes the representation of more ambiguous, quasi-compositional structures that cannot be neatly decomposed in this way. For instance, even in the highly compositional domain of natural language, there exist. phenomena such as idiomatic expressions that resist straightforward decomposition into role-fille pairs because their meanings cannot be directly inferred from their composite parts. Such examples illustrate that real-world compositional structures may require more flexible representation than what is offered by traditional TPRs. Consequently, TPRs may struggle to model more nuanced forms of compositional structure that characterise less structured, non-formal data domains, such as vision. Additionally, TPRs present challenging learning task. Their representational form Wh (v) == Σ, E (Jm(t) ) @ (n(r;) is only satisfied by a discrete subsel of points within the underly- ing representational space, Vr 6 Vn, as illustrated Figure 1c. This constraint forces representation learners to map data to a highly restricted, discrete set of points, making learning inherently arduous.
We omit the dependence of m on r for notational clarity.
"See Appendix A.4 for further discussion the distributed nature of the TPR.
5

---

To overcome these shortcomings, we introduce Soft TPR. continuous relaxation of the traditional TPR specification. Soft TPRs allow any point in the underlying representational space Vr & VR that approximately conforms to the TPR specification. This removes the need for representations to perfectly adhere to the rigid structural specification of the TPR, instead permitting them to flexibly approximate this rigid form. This flexibility provides two main advantages: 1. Enhanced Repre- sentational Flexibility: Soft TPR facilitates the representation of quasi-compositioaal structures that do not perfectly adhere to the traditional TPR's ss structural specification. This includes scenarios where bindings are approximate rather than exact, or where multiple fillers influence a single role. 2. Improved Ease of Learning: The relaxed representational specification allows representation learners to map data to broader regions of the representational space (as illustrated by the translucent circular regions in Figure lc). This may facilitate more efficient learning, there more possible mappings from the data to the required representational form.
Formally, to define the Soft TPR, we consider the vector space underlying TPRs, Vr @ produced by an arbitrary role embedding function En R + Vn and an arbitrary filler embedding function &F F + Vp. A Soft TPR is defined any element E VF & VR in this underlying representational space that sufficiently close to a traditional TPR, 2 4, as measured by a chosen distance metric (we take the Frobenius norm). Denoting the Frobenius norm by ||-||p, Soft TPR satisfies =|= || e, where c is some small, ecalar-valued positive quantity. This condition ensures that the Soft TPR approximately encodes the IPR-based compositional structure of Wipr. while allowing for slight deviations which capture a more flexible, relaxed notion of compositionality that cannot be reduced to set of role-filler bindings (see B.2).
Despite this relaxation, Soft TPRs retain the critical advantage of traditional TPRs: the ability to recover constituent role-filler bindings through the unbinding operation, albeit approximately. Specifically, when unbinding a Soft TPR z, the operation yields soft filler embeddings f; that closely approximate the true filler embeddings {r(sm(s). Formally, for a Soft TPR z, performing the unbinding operation for the i-th role yields:
EEe(fmo) @ En(rs) u, = =F((f(()))
204 Utr
(3)
= =FF(Jm(O) + Jú,
where E; = zu,-- - {F(fm(t) represents the approximation error. This approximate recoverability ensures that Soft TPRs implicitly encode precise, beabbraccally-xpreesstiiei form of compositional structure, even when their representational form deviates from the exact TPR specification.
4.2
Soft TPR Autoencoder: Learning Distributed and Flexible Compositional Representations
We define our vector spaces of interest over the reals as Vi := Rr and VR := RR where Dp, Dr denote the dimensionality of the filler and role embedding spaces. The pivotal insight underlying our method is that a Soft TPR can be effectively realised by ensuring that the encoder produces representations that are amenable to quantisation into explicit TPRs. Specifically, since a Soft TPR any arbitrary element from the vector space Dy-Da sufficiently close to an explicit TPR, any (Dp DDR (0))ddmmesional vector produced by encoder in standard autoencoding framework can be treated as a Soft TPR candidate. This realisation suggests that a conventional autoencoding framework only requires modest modifications to generate Soft TPRs.
Our Soft TPR Autoencoder comprises three main components: a standard encoder, E, the TPR decoder, and a standard decoder, D. The encoder output, 2, serves as the Soft TPR. The overarching intuition guiding our architecture is to ensure that 2 can be effectively quantised or decoded into an explicit TPR, thereby enforcing the Soft TPR property, ||= vltpr ll2 <.
Representational Form: To ensure that the encoder produces elements with the form of Soft TPRs, |
we introduce a mechanism that penalises the Euclidean distance ||2 - / ll2 between the encoder
"We occasionally omit the dependence $ 'Due to isomorphism of vector spaces RD @ RD = I DD we henceforth use vectors from IR Pr in place of rank-2 tensors from Pr @ RRR and the Euclidean norm instead of the Frobenius norm to align the Soft TPR framework more seamlessly with the autoencoding framework.
for notational clarity.

---

Role embedding matrix
Filler embedding matrix
Max
R Dax
Mar E RDxxN
TPR Decoder
Unbinding Unbound 'soft filles Module
Quantisation Quantised filers Module
TPR Constructor
Encoder (E)
Vector
+$ TPR operator
Decoder (D)
Cuanfisation
Unbinding operator
R Dr
Dr
Vige
Figure 2: Diagram illustrating the Soft TPR Autoencoder. We encourage the encoder E's output, z, to have the form of Soft TPR by penalising its distance with the greedily defined, explicit TPR, Vip- of Equation that best approximates. Viyn is recovered using a step process performed by our TPR decoder (center rectangle); 1) unbinding, 2) quantisation, and 3) TPR construction. The decoder, D, reconstructs the input image using dépr-
output, and some explicit TPR, Virr. that z best approximates. To obtain v needed to compute the loss, we derive an explicit analytical form for tpr and construct elements satisfying this analytical form using a role embedding matrix, MşR containing Nn Diz-dimensional role embedding vectors, (En(r:)): and filler embedding matrix, Mg₃ containing Np Dy-dimensional filler embedding vectors, {fr(fi). To define an explicit analytical form for epr of filler embeddings based on their proximity to the soft filler embeddings extracted from z:
greedily optimal selection
Ppr Σ Ex (f(((()) & En(rs), where m(t) arg min fl Er (ff)))12, and fr 214-
(5)
That is, we define dypp as the TPR constructed from explicit filler embeddings {x(ft) with the smallest Euclidean distance to the soft filler embeddings fa of 2. To construct elements satisfying (5), we use three-step process carried out by the novel TPR decoder we introduce, visible in Figure 2: 1) Unbinding: Utilising
dense role embedding matrix Mcn> the unbinding module extracts soft filler embeddings (fi) by performing the TPR unbinding operation on z. We elaborate on our tors obtained, and for not backpropagating gradient to Mgn in B.4.1. 2) Quantisation: The quantisation module, containing learnable filler embedding matrix Mε employs the VQ-VAE vector quantisation algorithm [11] to both 1) learn the explicit filler embeddings, and 2) quantise the soft filler embeddings (iy..., -.. (NN) produced by the unbinding module into the explicit filler embeddings (E=(fi)....,&E (F } with the smallest Euclidean distances. 3) TPR Construction: The quantised filler embeddings {EF( (fm(()()))] are then bound their respective roles using the tensor product, and all embedded bindings are summed together to produce the explicit TPR Vir with the form of Eq 5. dhppr subsequently used by the decoder, D, to reconstruct the input image. The overall unsupervised loss, L. comprises three components, where s denotes the stop-gr -raddint operator, 33 is hyperparameter controlling the commiment loss in VQ-VAE, and C, is a suitable image-based reconstruction loss (we use T.2):
cereeiicalll-iffrmen reason for this design, how the unbinding vec-
Cu = ||2 -/rrvippll C,, D(Uip))) Σ
NR (Is(EF(fm())) fall =l1EE((m(m) - [
Soft TPR form ponalty
reconstraction loss
VQ-VAE quantisation loss
(6)
While the unsupervised loss C. ensures that maintains the Soft TPR form by being close to an explicit TPR, does not guarantee that the content of accurately reflects the true filler semantics of the data. To address this, introduce weakly supervised loss that aligns with the ground -tuth semantics of the image. We employ a Match-ppiring context similar to [13, 22, 33, 35], where image pairs (ar x²) share the same role-filler bindings for all but one role, ri- The identity of r; is known, but not any of the specific fillers or bindings. Our intuition is that, for to accurately reflect the semanties of the image, the Euclidean distance between the quantised fillers of and x' bound to role r; should be maximal, relative to the distances between the filler embeddings pairs for all other roles Tz. j # To encourage this, we apply the cross entropy loss corresponding to the 3rd term in Eq where Aq denotes the -pddimensional vector with each dimension (Aq)x
Representational Content:
7

---

populated by the Euclidean distance between the quantised fillers of x and x' for role and / denotes the one-hot vector dimension Nn with the index for r; set to Additionally, like [47]. we incorporate reconstruction loss (the 2nd term of Eq 7) to enforce consistency when swapping the quantised filler emheddings for role rı. Specifically, we construct new TPRs by swapping the quantised fillers for between z: and x' to generate ( (2) and v (z'), (z'), and ensure that the decoder can accurately reconstruct the corresponding swapped images from these swapped TPRs.
Combining these components, our final loss, L. is a weighted sum over the unsupervised and weakly supervised loss components, where A1 and 2 are hyperparameters:
1 (2 D(Vip (x))) 2 ((Vip(()))) + 12CE((q,.).
L: Lu +
Results
To evaluate Soft TPR framework, perform evaluation along three dimensions that are standard in the compositional representation learning literature [25, 30, 33, 42]: 1) Compositional Structure / Disentanglement: To what extent do Soft TPR representations capture explicitly compositional struc- ture?? 2) Representation Learner Convergence: How efficiently can representation learners acquire Soft TPR's distributed, flexible compositional structure? 3) Downstream Model Performance: Does Soft TPR's is distributed, flexible compositional form offer tangible benefits for downstream models utilising compositional representations?
We benchmark against a suite of weakly supervised disentanglement baselines: Ada-GVAH [33], GVAE [22]. ML-VAE [13], Slow VAE [39], and the GAN-based model of [35], which henceforth refer to as 'Shu'. These models produce symbolic compositional representations corresponding to a concatenation of scalar-valued FoV tokens. Similar to our approach, Ada-GVAE, GVAE, ML-VAE. VAE, VAE, and Shu are trained with paired samples (r, x') sharing values for all but subset of FoVs types (roles), with ML-VAE, GVAE, Slow and Shu assuming access to matching our model's level of supervision. To ensure fair comparison, we modify Ada-GVAE (method detailed in Appendix C.2.2) for more direct comparability, denoting our modification by Ada-GVAE-k. Additionally, we benchmark against 2 baselines producing symbolic vector-tokened compositional representations: COMET [36]. and Visual Concept Tokeniser (VCT) [47], although these are fully unsupervised and not directly comparable to our method. We train five instances of each representation learning model using five random seeds for 200,000 iterations across all datasets, and report averaged results (an unabridged suite of all results is contained in the Appendix).
5.1 Compositional Structure/ / Disentanglement
To demonstrate that Soft TPRs inherently capture the algebraic compositional structure required for effective disentanglement, we first quantise each Soft TPR, z, into its corresponding explicit TPR, fhr using the TPR decoder (note that once the model trained, quantisation fully deterministic). This quantisation process transforms the implicit compositional information encoded within Soft TPRs into an explicit algebraic form suitable for evaluation. We then apply standard disentanglement metrics across benchmark datasets including Cars3D [4], MPI3D [21]. and Shapes3D [24]. Table illustrates that Soft TPR consistently outperforms symbolic compositional baselines across all datasets, achieving state-of-the-arr results with notable DCI metric improvements of 29% on Cars3D and 74% on the more challenging MPI3D dataset. To ensure these gains are attributable to Soft TPR framework rather than merely a marginal increase in model capacity (Soft TPR Autoencoder adds between 1,600- 13,568 model parameters to a standard (variational) autoencoder). we conduct control experiments by equalising parameter counts in applicable baselines. Consistent with findings from prior work [40], we observe no performance improvements relevant symbolic baselines when parameter counts are increased (see C.2.4).
5.2 Representation Learner Convergence Rate
To explore how the Soft TPR framework influences the convergence rate representation learn- ers, we analyse representations produced at varying stages of representation learner training (100, 1,000, 10,000, 100,000 and 200,000 iterations), and evaluate both their 1) Disentanglement, and 2) Downstream Utility. We quantify downstream utility of learned representations by examining the
8

---

performance of downstream models on two tasks common in the disentanglement literature [25, 30, 33, 42]: a classification-basee based task assessing the model's s ability to perform abstract visual reasoning [30] and regression task involving the prediction of continuous FoV values for the disentanglement datasets [42]. Our findings indicate that the Soft TPR framework generally achieves faster disentan- glement convergence, particularly on Cars3l and MPI3D datasets (see Appendix C.4. 1). Moreover, Soft TPR consistenly accelerates the learning of useful representations for both downstream tasks. The downstream performance improvements are particularly pronounced in the low iteration regime- for instance, at only 100 iterations of representation learner training, as shown in Tables 2 and 3. For fair comparison, we embed baseline representations of both higher and lower dimensionality into the same vector space as Soft TPR, which we denote by f. and select the best tersfrrefonning variant (original or dimensionalliyy matched) for each baseline (details in Appendix C).
Table I: Vac and DCI scores. Additional results in Section C.3.3
Car33D
Shage<3D
MPR3D
Models
FactorVAB
DCI score
VAc score
DCI
VAc score
DCI
Symbolic scalar tokenr- compositional representations
Slow- VAF
0.902 0.035
0.509 0.027
0,950
0.032
0.850 + 0.047
0.455
0.027
Ada-GVAE-k
0,947 = 0.064
0.664 = 0.167
0.973 0.096
0.963 = 0.077
0.496 :: 0.095
0.343 = 0.040
GVAE
0.877 + 0.081
0.262 0.095
0,921
0.075
0.842 + 0.040
0.378 0.024
0.245 0.074
ML-
0,870 + 0.052
0.26 + 0.063
0.835 0.111
0.739 : 0.1
0.390
:: 0.026
0.251
+: 0.029
Shu
0,573 1 0.062
0.032 0,014
0,265
0.043
0.017 1 0.006
0,287 0.034
0.033 0.008
Symbolic vector-tokened compositional representations
VCT
0.966 1 0.029
0,382
0.957 0.043
0.884 :: 0.013
0.689
0.475 + 0.005
COMET
0,339 = 0.008
0.024 = 0.026
0.168 = 0.005
0.002 = 0.000
0,145 + 0.024
0.005 = 0.001
Fully continuous compositional representations
Ours
0.999 1 0.001
0,863 1 0.027
0.984 0.012
0.926 1 0.028
0.949 1 0.032
0.828 11 0.015
Table 2: FoV regression 172 scores (100 iterations of repre- Table 3: Abstract visual reasoning accuracy (100
sentation learner training).
iterations of representation learner training).
Car33D
Shapes31)
MPBD
Absiract visual reasoning dataset
Models
Symbolic scalar tokened
Models
Symbulic scalarttoknned
Slow-VAE
0.233 0.048
0.600
0,048
0.557 +: 0.012
Slow-VAE!
0.552 0.035
Ad=GVVE-k
0.307 1 0.084
0.684
0.059
0.519 1: 0.023
Ada-GVAF-k!
0.637 0.037
GVAE
0.319 1: 0.073
0.565
0.034
0.511 +: 0.037
GVAE
0.554 0,031
ML-VAB
0,317 1317:000 0.058
0.551
0.059
0.504
1: 0.016
ML-VAE!
0.550 0.025
Shu!
0.012 + 0.007
0.461
0.075
0.299 + 0,040
Shu
0.208 = 0.052
Symbolic vector-tokeued
Symbolic voctor-tokened
VCT
0.080
0.001
0.886 0.033
0.316 0,016
VCT COMET
0.440 0.033
COMETT
0.484 0.477
0.474 0.062
0,240 + 0.010
0.348 0.069
Fully comtinnous
Fully continaous
Ours
0,531 1 0.054
0,981 1 0.003 0.732 1: 0.012
Ours
0.804 0.016
5.3 Downstream Models
To evaluate whether the distributed and flexible encoding of compositional structure offered by the Soft TPR benefits downstream models, we examine both: 1) Sample Efficiency and 2) Raw Performance in Low Sample Regimes, using the previously mentioned abstract visual reasoning and FoV regression tasks. In line with [25]. we quantify sample efficiency with a ratio-based metric comparing downstream model performance using restricted sample sizes (100, 250, 500, 1,000, and 10,000) against performance when trained using all samples. As illustrated in Table 4, Soft TPR substantially outperforms baseline models in sample efficiency, especially in the most restrictive case involving only 100 samples, achieving 93% improvement. Additionally, Soft TPR produces substantial raw performance increases in the low sample regime, evidenced by the 138% and 168% improvements in Table 4 for the low-sample regimes of 100 and 200 samples respectively, and the 30% improvement in Table 5.
5.4 Ablation Studies
To disentangle the contributions of Soft TPR's foundational components, we conduct ablation studies focusing on two critical properties that are hypothesised to influence the Soft TPR's 's learning of compositional representations: 1) Relaxed Representational Constraints and 2) Distributed Encoding, denoted RRC and DE respectively in Table 7. For Relaxed Representational Constraints, we significantly increase the weighting of the form penalty ||> - wipzlll in the Soft TPR loss to enforce rigid representational constraints on the Soft TPR Autoencoder, effectively forcing it to
9

---

Table 4: Downstream FoV It scores (odd columns) and sample Table 5: Abstract visual reasoning accuracy in the low-sample regime of 500 samples.
efficiencies (even columns) on the MPI3D dataset.
100 samples
100 samples/all
250 sumples
250 samples/all
Models
Symbolic scalar-tokened
Models
Symbolic scalartokened compesitiona representations
Slow Slow-VAF VAE
0.196 + 0,028
Slow-VAR
0.127 a. 0.050
0.130 2: 0.051
0.152 0.011
0.155 1: 0.011
Ada G
0.203 0.007
Ada-GVAFF
0,200 + 0.221
0.270 + 0.037
0.213 0.023
0.279 + 0.026
GVAR
0.182 + 0,013
GVAE
0.181 0.030
0.234 + 0.035
0.217
0.025
0.282 0.027
ML-VAE
0.193 0.012
ML.YAE
0182 +: 0.113
0.236 + 0.019
0.222
0.024
0.288 0,030
Shu
0.200 + 0.010
Sbu
0.151 1 0.016
0.343 4: 0.024
0.211 2 0.026
0.452 1: 0.075
Symbolic vector-tokened
Symbolic vector tokened compesitional representations
VCT COMET
0.277 = 0.039
VCT
D006
0.981
0.189 0.107
0.119
0.000
0.346 0.137
COMET
-0.05 = 0.015
0.000 = 0.000
-0.042 +: 0,018
0.000 1: 0,000
0.259 = 0.016
Fully continuoes compositional representations
Fully continuous
Ours
0.490 1: 0.068
0.556 = 0.078
0.594
0.056
0.665 0.067
Ours
0.360 1: 0.033
Table 6: Downstream FoV At scores for TPR and Soft TPR (num- ber of samples in brackets)
Tahle 7: Effect of model properties on disen- tanglement (MPI3D dataset).
Representational form
Cars3D
Shapes3D
MPI3D
Property
DCI Score
TPR (100 samples)
0.361 + 0.031
0.447 +: 0.108
0.415 + 0.050
RRC & DE
0,225 + 0.034
Soft TPR (100 samples)
0.638 + 0.010
0.481 + 0.081
0.531 0.069
RRC
0,598 1 0.093
TPR (250 samples)
0.537 + 0.053
0.669 + 0.108
0.598 0.081
DE
0,704 1 0.135
Soft TPR (250 samples)
0.832 + 0.027
0.728 + 0.021
0.621 + 0.072
+ RRC & DE (Full)
0.828 + 0.015
produce representations that precisely match the TPR-specified compositional form. For Distributed Encoding, we modify the model to encode compositional structure in an inherently less distributed manner. by modifying the sparsity of Uire (see C.2.4). Results in Table 7 demonstrate that both the Soft TPR's distributed encoding and relaxed constraints are essential for learning representations with high compositional structure, as quantified by the disentanglement metrics. Additionally, we examine the impact of Soft TPR's 's flexible, by replacing each Soft TPR, z. with its quantised explicit TPR, Uip in downstream tasks. Table and Appendix C.6.3 indicates that Soft TPRs provide downstream models with unique performance improvements that explicit TPRs (which are produced under the same conditions) cannot account for. Please see Appendix C.6 for further details and full ablation results, including additional ablations.
coasssiioopiiii form on downstream performance
5.5 Key Insights
The empirical results of the Soft TPR framework reveal two primary insights. Firstly, deep learning models may more effectively learn precise compositional structures when these structures are 1) implicitly acquired through relaxed representational constraints, allowing the model to acquire precise compositional forms (i.e., Vip..)) through quantisation of flexible Soft TPRs, and 2) when the compositional structure is assumed to be encoded in a distributed format. Secondly, the flexible representational form of Soft TPR offers downstream models unique advantages by enabling more efficient utilisation of implicitly encoded compositional information compared to their rigid coun- terparts, d/pp Together, these comprehensive model benefits empiricially demonstrate the value of the framework's 's 1) relaxed representational constraints, 2) distributed encoding of compositional structure, and 3) flexible representational form.
6 Conclusion
In this work, we tackle a challenge tracing its roots to the conception of the connectionist vs. classicalist debate: the fundamental mismatch between compositional structure and the inherently distributed nature of deep learning. To bridge this gap, we introduce the Soft TPR, a new, inherently distributed, flexible compositional representational form extending Smolensky's 's Tensor Product Representation, together with the Soft TPR Autocoder, a theoretically-princiii for learning Soft TPRs. Our flexible, distributed framework demonstrates substantial improvements in the visual representation learning domain - eenanncing the representation of compositional structure, accelerating convergence in representation learners, and boosting efficiency in downstream models. Future work will extend this framework to hierarchical compositional structures, enabling bound fillers to recursively decompose into role-filler bindings for enhanced representational expressivity.
architecture designed
10

---

Acknowledgements
This research has been supported by an UNSW University Postgraduate Award. We thank the reviewers for their valuable comments, J. Hershey for a healthy dose of skepticism which greatly improved this work, and B. Spehar for insightful discussions.
References
[1] Noam Chomsky. Syntactic Structures. The Hague: Mouton, 1957.
[2] Jerry A. Fodor. The Language of Thought: A Theory of Mental Representation. Cambridge, MA: Harvard University Press, 1975.
[3] Paul Smolensky. "Tensor product variable binding and the representation of symbolic structures in connectionist systems". In: Artificial Intelligence 46.1 (1990), pp. 159-216.
[4] Sanja Fidler, Sven Dickinson, and Raquel Urtasun. "3D Object Detection and Viewpoint Estimation with Deformable 3D Cuboid Model". In: Advances Neural Information Processing Systems. 2012.
[5] Yoshua Bengio. Deep Learning of Representations: Looking Forward. 2013. arXiv: 1305. 0445 [cs. LG].
[6] Jacob Andreas, Marcus Rohrbach, Trevor Darelll and Dan Klein. "Neural Module Networks". In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015), pp. 39-48. URl: ht /tttps:/app.... www.etiieccooooe....... .org org/ http//rrpusDD 5276660.
[7] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI- Al-Collee Question Answering: A Set of Prerequisite Toy Tasks. 2015. arXiv: 1502. 05698 [cs. AI]. URL: https: //arxiv org/ abs/1502. 05698.
[8] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbecl. "InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets". In: Advances Neural Information Processing Systems. Ed. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett. Vol. 29, Curran Associates, 2016.
[9] Paul Smolensky and Matthew Goldrick. "Gradient Symbolic Representations in Grammar: The case of French Liaison". In: 2016. URL: aattps://app http:://.a.....aaaiaaeeemmaa sww.attiiiis... commanttsscchlarr woggaaaooorroooooo Orgg/orrpusI] ID: 36953611.
[10] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess. Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. "beta- VAE: Learning Basic Visual Concepts with a Constrained Variational Framework". In: International Conference on Learning Representations. 2017.
[11] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu koray. "Neural Discrete Representation Learning". In: Advances in Neural Information Processing Systems. Ed. by 1. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. Vol. 30. Curran Associates, Inc.. 2017.
[12] Tameem Adel, Zoubin Ghahramani, and Adrian Weller. "Discovering Interpretable Representations for Both Deep Generative and Discriminative Models".. In: Proceedings of the 35th International Conference on Machine Learning. Vol. 80. Proceedings of Machine Learning Research. PMLR, 2018, pp. 50-59.
[13] Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. "Multi-Level Variational Autoencoder: Learning Disentangled Representations From Grouped Observations". In: Proceedings of the AAAI Conference Artificial Intelligence 32.1 (2018).
[14] Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in B-VAE. 2018. arXiv: 1804. 03599 [stat.ML]. [15] Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud. "Isolating Sources of Disentangle- ment in Variational Autoencoders" In: Advances in Neural Information Processing Systems. 2018.
[16] Cian Fastwood and Christopher K. I. Williams. "A Framework for the Quantitative Evaluation of Disentangled Representations" In: International Conference on Learning Representations. 2018.
[17] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. "Variational Inference of Disentangled Latent Concepts from Unlabeled Observations" In: 6th International Conference on Learning Represen- tations, JCLR 2018, Vancouver, BC, Canada, April May 3, 2018, Conference Track Proceedings. 2018.
[18] Adam Santoro, Felix Hill, David Barrett, Ari Morcos, and Timothy Lillicrap. "Measuring abstract reasoning in neural networks". In: International conference on machine learning 2018, pp. 4477 4486.
[19] Kezhen Chen, Qiuyuan Huang, Hamid Palangi, Paul Smolensky, Kenneth D. Forbus, and Jianseng Gao. "Natural- to
generation using Tensor Product Representations", . In: CoRR Abs/1910.00239
(2019). arXiv: 1910. 02339. URL: http: arrxiv.vvv .rrg/i.or//////eo absaa//s//991 19900. 02339.
[20] Elliot Creager, David Madras, Henrik Jacobsen. Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. "Flexibly Fair Representation Learning by Disentanglement" In: Proceedings of the 36th International Conference on Machine Learning, 2019.
11

---

[21] Muhammad Waleed Gondal, Manuel Wüthrich, undefinedorde Miladinovié, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf. and Stefan Bauer. "On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset". In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019.
[22] Haruo Hosoya. "Group-basee learning disentangled representations with generalizability for novel contents". In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019, pp. 2506-2513.
[23]
Qiuyuan Huang, Li Deng, Dapeng Wu, Chang Liu, and Xiaodong He. "Attentive Tensor Product Learning". In: Proceedings of the AAAI Conference on Artificial Intelligence 33.01 (2019), pp. 1344- 1351.
[24]
Hyunjik Kim and Andriy Mnih. Disentangling by Factorising. 2019. arXív: 1802. 05983 [stat MLJ] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf. and Olivier Bachem. "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations".. In: Proceedings of the 36th International Conference on Machine Learning. Vol. 97. Proceedings of Machine Learning Research. PMLR, 2019, pp. 4114-4124.
[25]
[26]
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. "Disentangling Disentanglement in Variational Autoencoders". In: Proceedings of the 36th International Conference on Machine Learning. 2019.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. "PyTorch: An Imperative Style, High- Perforaaren Deep Learning Library". In: Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 2019, pp. 8024- 8035. URL: http: / /apprssppessaaa// neurips. cc/ ptppp/pppppppppppp performance- dee-- wwwrninnnnnn--ee.e -iirrain 1wwbrrrrrryy... .dff pdf.
ch- an- an--iperrative-styee stwreresssssssssseee
[28]
Imanol Schlag and Jürgen Schmidhuber. "Learning to Reason with Third-Order Tensor Products". In: Advances in Neural Processing Information Systems. 2019.
[29] Imanol Schlag, Paul Smolensky, Roland Femaandez, Nehojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao. "Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving", In: ArXiv abs/1910.06611 1 (2019). URL: https: h //api. wtw.pp...aaaaat.ee.....ee.....ae semanti escholar org/CorpusID. 204575948.
[30] Sjoerd van Steenkiste, Francesco I Locatello, Jürgen Schmidhuher, and Olivier Bachem. "Are disentan- gled representations helpful for ahstract visual reasoning?" In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019.
[31] Junxiang Chen and Kayhan Batmanghelich. "Weakly Supervised Disentanglement by Pairwise Sim- ilaritics". In: Proceedings of the AAAI Conference on Artificial Intelligence 34.04 (2020), pp. 3495- 3502.
[32] Zheng Ding, Yifan Xu, Weijian Xu, Gaurav Parmar, Yang Yang. Max Welling, and Zhuowen Tu. "Guided Variational Autoencoder for Disentanglement Learning". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
[33] F. Locatello, B. Poole, G. Rätsch, B. Schölkopf. O. Bachem, and M. Tschannen. Supervied Disentanglement Without Compromises" " In: Proceedings of the 37th International Conference on Machine Learning (ICML). Vol. 119. Proceedings of Machine Learning Research. PMLR, 2020, pp. 6348- 6359.
[34] R. Thomas McCoy, Tal Linzen, Ewan Dunbar, and Paul Smolensky. "Tensor Product Decomposition Networks: Uncovering Representations of Structure Learned by Neural Networks".. In: Proceedings of the Society for Computation in Linguistics 2020. Association for Computational Linguistics, 2020,
pp. 277-278. URL: https: / aallattoloooyyy..... .wrggy.orgg.ooo org/ scil-1. 1.34.
[35] Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. "Weakly Supervised Disentan- glement with Guarantees". In: International Conference on Learning Representationss 2020.
[36] Yilun Du, Shuang Li, Yash Sharma, B. Joshua Tenenbaum, and Igor Mordatch. "Unsupervised Learning of Compositional Energy Concepts". In: Advances Neural Information Processing Systems. 2021.
[37] Insu Jeon. Wonkwang Lee, Myeongjang Pyeon, and Gunhee Kim. "BB-GAN: Disentangled Representa- tion Learning with Information Bottleneck Generative Adversarial Networks". In: Proceedings of the AAAI Conference on Artificial Intelligence 35.9 (May 2021), pp- 7926-7934. DOT: 10. 1609 / aaai.
v3519.16967. URL.: nottps://o]. aaai baai..rr///. org/ /n//inddxxx// wnndee-pp//aee
[38] Yichen Jiang, Asli Celikyilmaz, Paul Smolensky, Paul Soulos, Sudha Rao, Hamid Palangi. Roland Fernandez, Caitlin Smith, Mohit Bansal, and Jianteng Gao. "Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization". In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Online: Association for Computational Linguistics, 2021, 1p. 4780-4793. DOt: 10. 18653/ v1/2021. naacl ma main. 381. URL:: https: http://aclllaaa http:///aaaaaallllaaaa/ wwwpollggyooggg///// org/2021 www.clmain....... 381.
12

---

[39] David A. Klindt, Lukas Schoit, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge. and Dylan Paiton. "Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding". In: International Conference on Learning Reprexentations. 2021. URL: https: /ttpss////pp/eeeeeeeeeee .ntt// forum?id=EbID)
nnt//
[40] Milton Llera Montero, Casimir Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. "The role of Disentanglement in Generalisation.. In: International Conference on Learning Representations. 2021. URL: https: / wpwpr/vepeeeeee...eeee//. net/forumt stte/ffrmmidddgbff/// 1d=qbH974,KKVV
[41] Sungho Park, Sunhee Hwang, Dohyung Kim, and Hyeran Byun. "Laarnnng Disentangled Representation for Fair Facial Attribute Classification via Fairness- aware Information Alignment", In: Proceedings of the AAAI Conference Artificial Intelligence 35 (2021), pp. 2403-2411. DOI: 10. 1609/ /aaii. vv513.
16341. URL: https: / //ojs. aaai aeaai.orgg.. orgg ///inndx...aa// php/ AAAI AAA//rrre aatiicleeeiaeaeeeeeeaaaa wttle//veeee63////
[42] L. Schott, J. von Kügelgen, F. Träuble, P. Gehler, C. Russell, M. Bethge, B. Schölkopf, F. Lacatello, and W. Brendel. "Visual Representation Learning Does Not Generalize Strongly Within the Same Domain". In: ICIR 2021 - Workshop on Generalization beyond the training distribution in brains and machines. 2021.
[43]
Frederik Träuble, Elliot Creager, Niki Kilhertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. "On Disentangled Representations Learned from Correlated Data". In: Proceedings of the 38th International Conference on Machine Learning. Vol. 139, Proceedings of Machine Learning Research. 2021. pp. 10401-10412.
[44] Milton Montero, Jeffrey Bowers, Rui Ponte Costa, Casimir Ludwig, Gaurav Malhotra. "Lost in Latent Space: Examining failures of disentangled models at combinatorial generalisation" In: Advances in Neural Information Processing Systems. Fd. by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave,
K. Cho, and A. Oh. Vol. 35. Curran Associates, Inc., 2022, pp. 10136-10149.
[45] Zoltán Gendler Szabó. "Compositionality" In: The Stanford Encyclopedia of Philosophy. Ed. by Edward N. Zalta and Uri Nodelman. Fall 2022. Metaphysics Research Lab, Stanford University, 2022.
[46] Tao Yang. Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Namning Zheng. "Towards Building A Stouupbased Unsupervised Representation Disentanglement Framework". In: International Conference on Learning Representations. 2022. URL: https: http:///ppeeeieeeeeeee wet.eeewweeeeeeeeeeeeeeeeeeeeeee net/ orrm iid-gggaictmydddd
[47] Tao Yang, Yuwang Wang, Yan Lu, and Nanning Zheng. "Visual Concepts Tokenization". In: Advances in Neural Information Processing Systems. 2022.
[48] H. Zhang, Y.-F. Zhang, W. Liu, A. Weller, B. Schölkopf, and F. Xing. "Towards Principled Disentangle- ment for Domain Generalization", In: Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022, pp. 8024-8034. URL: https: / /pwpa//cee/cccecee//me thecvf .orf..oo com/ content/ CVtp://2/22222222// httpa/22222paprrr/e/eeee httppppppps///pppeeee Gemeralization. wttp:e__eC___22.P_eeee_eeee
Towards Prricipped Disentangl _for___ _Domain___
[49] Thaddaus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. "Compositional Generalization from First Principles". In: Thirty-seventh Conference on Neural Information Processing
Systems. 2023. URL: httpp:://opeerevieee http:///peweveveeeeee/or//o//a/a
[50] Haoyang Li, Xin Wang, Zeyang Zhang, Haibo Chen, Ziwei Zhang. and Wenwu Zhu. "Disentangled Graph Self-supervised Learning for Out-of--istributiio Generalization". In: Forty-first International Conference on Machine Learning. 2024. URI:: https: /oppreeeeeeeeeeeeeeeeeeeeeeeeeee net/ oormm i OS0szhkPP..
[51] Taewon Park, Inchul Choi, and Minho Lee. "Attention-base Iterative Decomposition for Tensor Product Representation"" In: The Twelfth International Conference on Learning Representations. 2024. URL:
https://openreviie
[52] Qiuyuan Huang, Paul Smolensky, Xiaodong He. Li Deng, and Dapeng Wu. "Tensor Product Generation Networks for Deep NLP Modeling". In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers). Association for Computational Linguistics.
13

---

Appendix
TPR Framework
14
A.1
Additional Details
14
A.2
Formal Proofs
15
A.3
Shortcomings of Symbolic Compositional Representations and How TPR Helps
16
A.4 TPR's Distributed Nature and the Equivalence between Degenerate TPRs and Sym- bolic Representations
18
Soft TPR Framework
20
B.1
Shortcomings of the TPR and How Soft TPR Helps
20
B.2 Proof that Soft TPRs Do Not Necessarily Have the Explicit Form of TPR
22
B.3
Alternative Formulations
23
B.4 Soft TPR Autoencoder
23
B.5 Model Hyperparameters and Hyperparameter Tuning
26
Results
27
C.1 Datasets
27
C.2 Baseline Implementations and Experimental Settings
27
C,3 Disentanglement
29
C,4 Representation Learning Convergence
31
C.5 Downstrcam Performance
50
C.6 Ablation Experiments
69
D Limitations and Future Work
81
D.1 Extension Linguistic Domains
81
D.2 Need for Weak Supervision
82
D.3 Downstream Utility
82
D.4 Dimensionality
83
D.5 Computational Cost
83
A TPR Framework
In this section, we provide additional details regarding Smolensky's TPR framework [3], as well as formal proofs of presented results. We refer interested readers to [3] for a more comprehensive dive into the TPR framework.
A.1 Additional Details
By defining the constituent components of any compositional object as a of role-filler bindings, the TPR defines the decomposition function, B, of Section 3.1 that maps from a set of compositional objects, X, to set of parts, more explicitly follows [3]:
3:X + FXR. I ((fr))f/r),
(8)
14

---

where F denotes set of fillers, R denotes a set of roles, and f/r denotes the binding of filler f E F role r E R. Note that contrast the formal definition each E X is decomposable into set of n. parts. the above decomposition allows objects to be decomposed into variably-sized set of role-filler bindings, with this set corresponding to an element in the powerset of F x R. For the considered visual representation learning domain, all datasets clearly have the property images in these datasets contain the same number of FoV types and each FoV type is bound to FoV token. Due to this property, we can take as special subcase of the generalised definition in Equation 8:
we use Section 3. which assumes
being decomposable into a fixed size set of role-filler bindings, as all
B: X + A: {(fm()) 14)lfm))m3)
where m (l,..., NR) + (1....,N, NF} denotes matching function that associates each role ri with the filler it binds to in ((z) (we again drop the dependence of for case of notation), and A denotes the set of all possible bindings produced by binding a filler to each the NR roles, with size N
(we assume the same filler can bind to multiple roles).
A.2 Formal Proofs
We now formally prove that the TPR with 3 defined in 9 has form (x) C(vi (a1)..... si (an)) and thus corresponds the definition of compositional representation in Section 3.1.
Proof.
We denote the role embedding and filler embedding functions as En: R + VR,EF: : F - Ve
respectively.
By definition the TPR in Eq 1, we have that:
Vtpo (2) = Σ (F(sm(e) 8 En(rs).
and hence,
t/ro (x) = Σ Sh (i((mtoyrr)))
where a; == (fm(a) r;) E B(x). 14 F R Vp @ VRI (fm(e), Ts) - (ff((()) En(r:), and C ordinary vector space addition. Hence, almost trivially, 1/tpr (x) clearly has the required form be a compositional representation.
Now, we prove the recoverability of the embedded components (vi(Sm(()) r() from the TPR, ()rrr(r), (I)), provided that the set of all role embedding vectors, {Ex(r:)), are linearly independent. Similar variants of this proof can be found in [3], [19].
Proof. Assume the set of all role embedding vectors (En(r:)) are linearly independent. Then, the role embedding matrix, Mgn. == (En(ri).. (.(NN))))) formed by taking the role embedding vectors columns, has left inverse, U, such that:
UER INRxNR
Hence, we have that (UMe))as U, = lij- For ease of notation, let u; denote the i-th column of U., and note that En(r,) clearly corresponds to Mgn:j- So, U. Mg (U.) Me (T[(r))) Iij. Hence, we have that:
i=j otherwise.
of ER(;)) = 8,g =
15

---

Using the definition of (x)r ()pr(z)) = Σi EF (fm(t)) @ En(rs), we apply the (tensor) inner product of Vepr (z) with uy:
U (x) =
Σ (F(fm(t) @
Σ Ef(fm())
(f((m()))
Thus, the embedding of the filler, fm(s)) bound to each role, Ti, can be recovered through use of tensor inner product with the unbinding vector, Mg corresponding to the i-th column of U Note that the representational components of Ptpr (z), i.e. the embedded bindings, (f((())) T:) are fully determined by the embedding of the role, En(r:), and filler, (f(fm()) comprising the binding, as for each binding in ((x)) corresponds recovering the representational component, thx (4((m(()))) r.). So, provided the set of role embeddings are linearly independent, and they can be obtained (e.g. through
(r(J((s):) :y)) simply corresponds their tensor product. Thus, recovering (E(Im()))&
look-up table of role embeddings), all representational components, (fe(fm(t)) Tx) can be directly
recovered from the overall TPR representation, t'tpr (z).
A.3 Shortcomings of Symbolic Compositional Representations and How TPR Helps
Here, we provide more detailed elaboration on the shortcomings of symbolic compositional representations as outlined in 1. We additionally illustrate how TPR-based continuous compositional representations eircumvent such limitations through use of concrete example.
We employ the formal definition which states representation, vis (x)), is a symbolic compositional representation if (3)
symbolic compositional representation from Section 3.1,
(ai)) ...., vi (an)) i.e., if (1) corresponds to a concatenation of representational compo- nents, (as)
For more direct comparison between the symbolic compositional representation, (h)( and the TaR-based compositional representation, 1/tpr (x), we rewrite the TPR-based representation as yh (x) = (ā))) +... + (ān). Thus, the TPR can be viewed as an ordinary vector sum of components, (T))).), ), where each component, e corresponds to an embedded fole-filler binding.
We consider the following example: suppose there are FoV types (i.e., roles), colour, shape and FoV values (i.e., fillers), purple, square. For the symbolic approach, suppose the representational
components, (vi (a:)) are defined as follows: Vectour (purple) (purpee 0 and Vishaee
For the distributed approach, suppose the (linearly independent) role embeddings are: {R(colour) [
2
2 (shapee)) and the filler embeddings are: (purple
[ and &ur (square)
Then, a symbolic compositional representation is given by (denoting the concatenation operation by
@):
(x) = ccolour (purple) @ vishapo (square)
0
=
2
3
On the other hand, the distributed, TPR-based compositional representation is given by:
16

---

Vitor (x) (purple/colour) +
(eteoooooooottttuuueooeereeeeeee ((square//shape)
(ppurlee) @ (coooour)) + &a Esquare)) @ (a(shape)
22
3
3
2
2
3
[4]
Note that both the symbolic and distributed, TPR-baser representations share the same underlying representational space, i.e., R.
Gradient Flow and Learning The symbolic compositional representation allocates the FoVs to 2 distinct, independent slots of (z(x). This creates discrete boundaries between the first two and last two dimensions of the representation, which may potentially complicate gradient-basse optimisation for the following reasons:
(a) Discrete Boundaries and Restricted Gradient Flow: In the symbolic compositional representation, each FoV is confined to specific, non-overlapping dimensions of the representation. This type of assignment means that updates to particular FoV requires restricting gradient flow exclusively to the dimensions associated with its designated slot. For example, updating the FoV for square/ two dimensions (i.e., permitting gradient flow through these two dimensions), while keeping the first two dimensions unchanged (i.e., completely restricting gradient flow through these two dimensions). This constrains gradient flow across all dimensions of representation. Mathematically, the gradient update for the above example (modifying square/shape) can be expressed in the symbolic case as:
requires modifying the last
D
Vo, 1
where a, b E R
b
which categorically restricts gradient flow dimensionwise across the underlying repre- sentation space, R24
(b) Abrupt, Discontinuous Shifts in the Representational Space: are updated consecutively, the symbolic, slot-based representational form may induce sudden and discontinuous changes in the representational space. For instance, updating perple/colout time and then shape/square t produces:
When multiple FoVs
0
0
(.(x); (2))--
7
a
b
(x(2)11 = (x(x))- -
These abrupt shifts in the representation space may contribute to unstable and frag- mented learning dynamics, potentially hindering the convergence of optimisation algorithms.
'Please note there is a small error in the main Author Rebuttal www.r/vdsstoooedd-iiiPPi
in L16 under section '1) Incompatibility between disentangled repre- sentations and deep learning's continuous
spaces', where the tensor product mistakenly taken
over vent (purpee) & (square))) to produce the representation, instead computing Er (purple) & (R(cOl) + (squuar)) & (R(s)), but the conclusion remains exactly the same. (Note that teat (purple) { (x(puplee) & Ex(col) and (aa(square) = E (square) & {s(sh) should both be 4 dimensiional, not 2- dimensional.
17

---

In contrast, the continuous, distributed, TPR- EPR-based compositional representation integrates the Vs continuously into all dimensions of the representation (i.e., (ppurple/colou (1 1 2 2) and ((square//happe necessary existence of dimension-wise slots with non-differentiable boundaries for each FoV. This alleviates the previously mentioned problems:
(232 3) are summed together in IR), eliminating the
(a) Smoother Gradient Flow: The inherently distributed structure of the TPR allows gra- dients to flow freely across all dimensions of the representational space, R-. Updating the FoV for square/shape involves modifying all relevant dimensions simultaneously, avoiding categorical dimension-wise restrictions and potentially allowing for more flexible and effective gradient- optimisation as FoVs can be updated in a coor- dinated manner (note that backpropagating through the dimensions corresponding to square/ shape can also affect purple/ coloou, as the same dimensions are occupied);
b
Vo. = -1
where a, a,b,c,d
é,
d
(b) Smoother Learning Dynamies: By forgoing the slot-based structure, distributed encoding of compositional structure ensures that update transitions between Fs do cause sudden shifts in the representational space. For example:
b
expr (xr) = 3/ppr (x)
c-
[+
by
(iro = V/prr (z))
C
de
potentially promoting more stable and reliable convergence during training.
2. Representational Expressivity: In the example above, both th,s (x) and thr reside in R However, the symbolic, slot-based compositional representation, vis (z) restricts each FoV to a specific, 2-dimensional subset (i.e., Vectour (purple), Vshape (square) each belong to 3R This rigid allocation limits expressivity as each FoV is confined to subset of available dimensions, preventing the full utilisation of RRA, In contrast, an inherently distributed (ppurlleccooour) colour), iasssu/e//saappp
to exist within the same R4 space as the overall representation, (x)) This allows the representation learner to encode Fo as combinations of the entire set of 4-dimensional basis vectors spanning the representational space, offering enhanced expressivity.
approach allows each FoV - such (purple//
3. Robustness to Noise: In symbolic compositional representations, the impact of dimension- wise noise has a localised sensitivity: since each FoV is isolated within its designated slot, noise directly degrades the representation of the corresponding Fo For example, small perturbation in the third dimension (assigned to square/shape) has a localised, direct effect only this FoV. In contrast, for the inherently distributed TPR-based structure, noise introduced to any dimension of the representation is distributed across multiple FoVs. For example, for (x)) the same perturbation in the third dimension will have its impact distributed across both square /saape and purple/ /colour as both these FoVs contribute to that dimension.
A.4 TPR's Distributed Nature and the Equivalence between Degenerate TPRs and Symbolic Representations
The TPR offers an inherently distributed representation of compositional structure, allowing in- formation from multiple FoVs to intermingle across the same dimensions of the representational space.
18

---

Here, we elaborate on 2 critical properties of the TPR in relation to this notion of distributed embedding of compositional structure.
1. Equivalence of Degenerate TPRs to Concatenated Fillers:
While the TPR naturally supports distributed encoding of compositional structure, it is worth noting that when the role embedding vectors are aligned with the canonical basis vectors, the TPR reduces to concatenated filler vectors, thereby reducing to symbolic compositional representation where the representational components eh (a;) correspond to filler embeddings. In contrast the TPR, however, it is impossible for a symbolic compositional representations encode compositional structure in distributed fashion.
2. Distributed Representation through Dense Role Vectors: In the general case where the role vectors are linearly independent (but do not correspond to the canonical basis), provided that the role vectors are dense, then TPRs facilitate a truly distributed representation by al- lowing each dimension of the representation to encapsulate summation of information from multiple fillers. Such a representation not reducible to a dimension-wer wise concatenation of fillers, in the degenerate TPR case.
We now prove both statements:
Equivalence of Degenerate TPRs to Concatenated Fillers:
Let {e1, . position and Os elsewhere.
CDn } denote the canonical basis vectors in Dx, where each eg has a 1 in the k-th
Assume that each role vector = for i = 1,2, .n-
The tensor product mmi) r; results in matrix where the i-th column is fm(s) and all other columns are zero vectors:
0
5m(t) & = fm(j)e
Summing over all NR role-filler bindings in ((x) gives the required result:
f
DR
f
f2
DD
Exrr (z) = Smih @ 3e.=
JPr
$2 Dr
Drr
This shows that in the case where the role vectors correspond to the canonical basis vectors, the the TPR effectively concatenates the filler vectors fa along distinct, non-overlapping subsets of representational dimensions. Consequently, the TPR t/ppr (x) behaves equivalently to a symbolic, slot-based compositional structure where each filler occupies unique representational slot without interacting with other fillers, hence validating the equivalence of degenerate TPRs to concatenated fillers.
Distributed Representation through Dense Role Vectors:
Assume that the role vectors {r1, r2,..., NN aligned with the canonical basis vectors.
} in Px are dense and linearly independent but are not
Each tensor product fmis) &r yields matrix where each element (j.k) is Pact r. Summing across all role-filler bindings yields:
E Smmcen
Σ -
fl
m
ft
DR (i) r m()) 1
Wtpr (x)
Σ (1)
Σ p2
Σi=1
N
m(t)
EN Seityr
N MN (igor?
Σ
DR fm
Given that role vectors are dense, each element virlh incorporates contributions from multiple fillers Prntey each weighted by the respective component & of their role vectors. This intermingling ensures
19

---

that information from different filler vectors are distributed across the same dimensions of Vipr (z), rather than being confined distinct subsets of ()rpr(a)) dimensions, in the degenerate TPR case.
B Soft TPR Framework
In this section, we provide additional information regarding our Soft TPR framework.
B.1 Shortcomings of the TPR and How Soft TPR Helps
In this subsection, we discuss potential practical limitations of traditional TPR due to their rigid specification (Eq 1) and demonstrate how Soft TPR addresses these issues using concrete example.
Consider the following of roles R (shape, colour), and fillers F {red, blue, square). with embedding functions εn R - IR² and &F + IR3 defined as:
ER(shape)
En(colour)
(red)
2
3
2
{F(blue)
2
[0'
{F(square)
1. Discrete Mapping: The set of possible TPRs T generated by these roles and fillers is limited to discrete points. For instance:
t'prr "Jrer (quare)) (x(red) @ Ex(colour) (squuare)) (n(shpee)
2
0
3
1
2
IR
+
I
0
3
2
0
3
0
2
3
20

---

Veepr PPblue_squande = (x(blu)) @ Ex(colour) + (squuare)
@ (s(sppe))
2
0
2
0
2
0
3
1
2
83
0
+
2
0
3
2
0
3
0
2
2
2
2
3
Thus, T = [(222222)) (1 241222 241)) discrete subset of IR.. By relaxing the TPR specification to the Soft TPR, we allow any E TR within an e-neighbourhood of some 1/pr in T, thereby forming continuous regions defined as: Ts == (
+ a: |a| < c). This relaxation significantly expands the representational space, as T's has strictly more points than T. Consequently, learning theoretically becomes easier because there are more functions parameterising the mapping from the observed data to T's compared to T. Furthermore, in contrast to T. which contains discrete, singular points scattered throughout R Ts comprises continuous spherical regions centered around each Utpr E T. These factors collectively make the Soft TPR representation potentially casier to learn and extract information from compared to the TPR. This advantage is reflected in our empirical results in Scction C.6.3, where the Soft TPR demonstrates 1) greater representation learner convergence, 2) superior sample efficiency for downstream tasks, and 3) superior raw downstream performance in the low sample regime compared to the traditional TPR. Additionally, our ablation results in Table 43 indicate that imposing stricter constraints on the encoder to produce representations in the form of explicit TPRs substantially impairs the disentanglement of the resulting representations.
(1 2 (124123)
2.
Structure: The traditional TPR enforces a strict algebraic definition of compositionality. specifically of the form Σ, (f(((()) @ (Tt)). However, even in algebraically-e strrucuure domains such as natural language, compositional structures do not always adhere to this rigid specification. For instance. French liaison consonants consist of weighted sum of fillers bound to single role [9]. which deviates from the strict TPR formulation. The Soft TPR's continuous relaxation of this constraint permits the representation of structures that only approximately satisfy the TPR's algebraic definition of compositional structure as a set of discrete role-filler bindings. Consequently, Soft TPRs can encode forms of quasi-compositionalliy that are not strictly algebraically precise or directly amenable to TPR-based symbolic expression (see B.2 for a proof).
3. Duality: Although Soft TPRs employ a relaxed structural specification to encode quasi- compositional structures, they inherently preserve the exact algebraic form of composi- tionality through their corresponding explicit TPRs, vhppr via quantisation using the TPR Decoder. In other words, while Soft TPRs facilitate approximate and flexible compositional representations, they retain the capability to revert to the precise algebraic structures defined by traditional TPRs when necessary. This dual functionality ensures that Soft TPRs strike balance between the flexibility required to represent complex, approximately compositional structures- which benefits learning through relaxed representational constraints- and the preservation of exact algebraic forms essential for symbolic manipulation and interpretation.
4. Serial Construction:
Building explicit TPRs requires that the embedded fillers and roles comprising binding (i.e. (Er(fm(t)) ((=(f(())))(((()))) are first tokened before the entire compo- sitional representation, titprr (x:) can be produced [19, 23, 28, 34, 38, 51, 52]. This sort of sequential approach, where constituents must be tokened before the compositional represen- tation can be formed, is key characteristic of the symbolic representation of compositional structure [45]. In contrast, the Soft TPR allows the Encoder to produce any arbitrary element of VF & VR (in this case Sx RR), provided that the sufficient closeness requirement holds. Thus, once the Encoder is trained, it is theoretically possible for the Soft TPR Autoencoder
21

---

to generate approximately compositional representations by mapping directly from the data a Soft TPR single step, without needing to token any representational constituents.
B.2 Proof that Soft TPRs Do Not Necessarily Have the Explicit Form of TPR
It is intuitive that Soft TPRs, due to their continuous relaxation, do not necessarily conform to the explicit algebraic form 2, (F @ ER(ri) required by traditional TPRs. We provide a formal proof for completeness.
Proof. Let En + Vn and Er F Vr be fixed role and filler embedding functions, respectively, A traditional TPR is defined as:
tpr (z)
Σ f (fm(t) @ En(r.),
where m(i) is a matching function assigning each role r; to filler fm(i)-
A Soft TPR is any element within e-neighborhood of some traditional TPR in Vp @ VR:
E Be (e(tp)) = {2 E Vp @ VR || Viprll < <e),
where c >> 0.
Suppose, for contradiction, that every Soft TPR E Be (1 can be expressed as another traditional TPR with different matching function m/:
((((f))))) @ En(rs).
Given that En and čr are fixed, varying m' only changes the assignment of fillers to roles without altering the underlying embeddings.
Let d d'tpr, where ||ö|| < Substituting, we obtain:
EF((frrr) @ En(r:) = Vipr + 5.
Since 8 + 0, this equality implies:
EF((mm)) @ Cn(rs) + Mtpr-
However, because Ên and er are fixed, the only way to satisfy this equality if the matching function mn' compensates exactly for 5. This is generally impossible due to the following constraints:
1. Fixed Embeddings: m' can only reassign fillers roles without altering (n(r)) or & (f(f((o). The fixed (f(fm(s) restrict the possible variations, making it infeasible to ac- count for arbitrary 8 through mere reassignment.
2. Dimensionality and Span: In high-dimensional spaces Vp @ Vn. may lie outside the span of all traditional TPRs generated by varying m', especially as increases.
Consequently, the perturbation å cannot be exactly compensated by any reassignment m', leading to a contradiction. Therefore. Soft TPRs with non zero e-distances from their corresponding quantised TPRs, vi Vppe are also not necessarily expressible as other TPRs in the same underlying space.
The introduction of a continuous relaxation in Soft TPRs ensures that they do not strictly adhere to the traditional TPR's explicit algebraic form. This allows Soft TPRs represent quasi-compositional structures that deviate from strict algebraic compositionallty. which extends distributed compositional representations beyond the formal, explicitly algebraic domains required by traditional TPRs.
22

---

B.3 Alternative Formulations
In contrast to the greedily optimal analytic form of Vir derived in Equation 5, a globally optimal TPR with respect to ||a - U || can also be formulated. By fixing the sets of fillers F and roles R. along with the embedding functions &F F + Dr and &R Rt RD, the only degree of freedom lies in defining M the set of role-filler matching functions that specify permissible role-filler binding decompositions for E X. The globally optimal TPR that best approximates z, denoted Vippr is given by:
oott == arg min ||2 1 Vxpr
(10)
mE denotes the set of all possible traditional TPRs defined by the choices of F, R. Er. En, and M For clarity, we omit the dependency of both d'ter and m T.
where T = (E; (x(fm()) @ ER(r.))
If MM is restricted subset of the entire set of matching functions M. meaning there are constraints on which fillers can be bound to which roles, then solving Equation 10 is NP-complete. Conversely, M M, allowing any filler to be bound to any role with possible reuse, the optimisation simplifies significantly. In this unrestricted scenario, the problem reduces to independently assigning each role to the filler that minimises the local contribution to || 1/rrr lI which can be solved efficiently with a time complexity of O(NR x NF). Acknowledging the impracticality of exact global optimisation, and guided by the intuition that Vir should have explicit dependency
we propose the greedily optimal definition of di in Equation 5, which links the unbound soft fillers of 2, {fi), to v Vipr
B.4 Soft TPR Autoencoder
in Equation 5 allows any filler ʃj to bind to a role rs, which may not preserve the ground -ouuh semantics of the role-filler bindings in the image. To address this, the Soft TPR Autoencoder targets two objectives jointly: 1) Representational Form, ensuring the output resembles Soft TPR, and 2) Representational Contens, ensuring sensible role-filler bindings reflecting ground-truth semantics.
Defining mi(1) == arg min, I1 - {r(si)ll
B.4.1 Representational Form
We now describe how our method satisties the representational form property. As detailed in Section 4.2, we penalize the Euclidean distance between the encoder E's output 2 and the explicit TPR v defined in Equation 5. This penalty encourages the Soft TPR Autoencoder to produce encodings that resemble a Soft TPR by enforcing ||2 et la < é². To obtain uhp we employ our TPR decoder, which consists of three modules: 1) Unbinding. Quantization, and 3) TPR Construction.
1) Unhinding: Although multiple roles can bind to the same filler (e.g., object colour colour/magenta magenta and floor colour/magente
each role represents independent concept type. Therefore, use lincarly independent embedding vectors for roles to ensure the recoverability of embedded components (EF (f=(fm(()) (R(r)) from the TPR. Typically, randomly initialised role embedding matrix Mep E RaDNN with D >> NR will have linearly independent columns. To maintain this property during training, it is possible to apply orthogonality regularisation: MM, MER T|
However, using arbitrary linearly independent role embeddings poses a computational challenge in obtaining U, the (left) inverse of Mşn- necessary for unbinding vectors )u,). To address this, we utilise semi-orthogonal matrices E dx where AT A In this case, U Men. and the unbinding vector u, is simply the role embedding & (r:). Thus, unbinding a soft filler from 2 bound role 7; involves taking the tensor inner product between z and (n(r.).
We assume vector spaces over the reals and utilise the isomorphism Pr & RR PE to integrate the TPR framework with the autoencoding framework.
23

---

To implement this, our unbinding module initialises Mcn as dense, semi-orthogonal matrix using torch. nn. utils. www.etrrrraatiossooooooooooooooi and keeps Men fixed during training preserve its semi-orthogonal property 8.
Now, we proceed to formally prove the aforementioned semi- oriooonthyg properties. We first prove that for a (left-invertible) semi-orthogonal matrix, A RRdx" A A = Inxa
Proof. Let denote (left-invertiblee semi-orthogonal matrix of dimension Rdx=, Writing A out using its columns, {ai..... an.. we have:
A A
a1
ax)
As the columns fai,...,az] On,) are orthonormal, we have that: Oj 8ij, where denotes the dot product, and
6 = {
1 i i=
otherwise,
from which the result clearly follows.
For completeness, we also prove that u, corresponds to the i-th role embedding vector, though this follows rather trivially from the properties of semi-orthogonal matrices, and the definition of u,.
Proof. Let MER be a (left-invertible) semi-orthogonal matrix. Then, from the properties of semi-
orthogonal matrices proved earlier, have that M MER and so, U (ME) MER
Hence, the i-th column of U which corresponds the unbinding vector, us by definition, is simply (Men). = (x(rs). the i-th role embedding vector.
2) Quantisation
The quantisation module utilises the VQ-VAE algorithm [11] to learn the filler embedding matrix Mer and to quantise the soft filler embeddings (ff} into explicit embeddings {Er(fi)} based on the closest Euclidean distances. This process aligns with the matching function in Equation where each soft filler is assigned to the nearest explicit filler in Męg. Since quantisation involves an argmax operation and is non-differentiable. VQ-VAE employs an L2 codebook loss to adjust the embedding vectors towards the soft fillers and commitment loss to ensure the encoder commits to specific embeddings, preventing the embedding space from expanding indefinitely.
For further details on the quantisation process, refer to [11].
3) TPR Construction
The TPR construction module is parameter-fie free and deterministically binds the quantised filler embeddings (EF(sm(9)) to their corresponding role embeddings {ER(r.)) from Meg. forming the explicit TPR:
Σ (r(Sm(o) @ @ En(re)
B.4.2 Representational Content
Here, we explicitly define the 'swapped" TPRs, vir ( (x) and (I'), used in our weakly supervised reconstruction loss (second term of Equation 7). For image pairs (x, x²) sharing all role-filler bindings
Astute readers may notice that the explicit TPRs representation leamer rotationally isomorphic the degenerate TPR (i.c., concatenation of fillers). See C.6.1 for further discussion.
24

---

except for role rs, we construct the swapped TPRs by exchanging the quantised filler embeddings for ry:
(pp((x)) = - (fm()) @ En(ry) + &F (frr()) @ En(ri).
(11)
342
Vin
(x') (f((m))) @ En(ry) + Cr (r(m(()) @ En(ri).
(12)
jti
where m and m' are the matching functions for a and x', respectively.
B.4.3 Model Architecture
We provide concrete details of our model architecture in Table 8, 9 and Table 10. It is worth noting that Soft TPR Autoencoder only adds an additional DF NN parameters to a standard (variational) autoencoding framework consisting the encoder and decoder the only learnable parameters we introduce are in the filler embedding matrix, Mgr.
For all our experiments, and the decoder, D.
the same general architecture for the encoder, E, the TPR decoder,
Table 8; Encoder (E)) architecture.
Encoder
Input 64 64 c
Cоnv 32, 4 x 4, stride = 2. padding = 1
BatchNorm 32
ReLU 32
Conv 64, 4 x 4, stride 2, padding =
BatchNorm 64
ReLU 64
Conv 256, 4 4, stride padding
BatchNorm 256
ReLU 256
Conv 512, 4 x 4, stride 2, padding =
BatchNorm 512
ReLU 512
Flatten 512 x 4 x 4
Linear 1024
ReLU 1024
Linear 512
ReLU 512
Linear 512
ReLU 512
Linear Dp DR
Table 9: TPR Decoder architecture.
TPR Decoder
Role embedding matrix (fixed) DR x NR Filler embedding matrix Dr x Nr
25

---

Table 10: Decoder (D) architecture.
Decoder
Input DF DR
ConyTranspose 512, x 4, stride 2, padding
BatchNorm 512
ReLU 512
Conv TonnTrosspos 256, x 4, stride = 2, padding = 1
BatchNorm 256
ReLU 256
ConvTransposs 64, 4 x 4, stride = 2, padding
BatchNorm 64
ReLU 64
ConvTrranspose 32, 4 x 4, stride 2, padding
BatchNorm 32
ReLU 32
ConvTranspose 3, 4 stride padding
B.5 Model Hyperparameters and Hyperparameter Tuning
The Soft TPR Autoencoder has the following tunable hyperparameters:
Architectural Hyperparameters:
NR.
NF, DR. DF
These correspond to the number of role and filler embedding vectors and the dimensionalities of their respective embedding spaces.
Loss Function Hyperparameters:
3 (which weights the VQ- IQEV commitment loss as per Equation and)
X1 X2
(which weigh the unsupervised and supervised losses in Equation 7)
Following the VQ-VAE framework [11], To ensure fair comparisons with scalar-tokened generative baselines and COMET, which assume 10 FoV types (and VCT assumes 20), fix NR 10.
3, the coefficient for the commitment loss, to 0.5.
We optimise the remaining hyperparameters using the Weights and Biases (WandB) hyperparameter sweep framework. The optimisation employs Bayesian search with the unsupervised MSE reconstruc- tion loss from Equation as the criterion. During this optimization, models are trained for between 50,000 and 100,000 iterations.
The final hyperparameter values for the Soft TPR Autoencoder are listed Table 11. For ablation experiments demonstrating the model's robustness to different hyperparameter settings, refer to Section C.6.4.
Table 11: Hyperparameter values.
Cars3D
Shapes3D
MPI3D
Hyperparameter
Architectural hyperparameters
Dr
12
16
12
NR (fixed)
10
10
10
Dy
128
32
32
Nr
106
50
Loss function hyperparameters
0.00024
0.00091
0.0000
d 8 (fixed)
0.02200
0.00228
1.16050
0.5
0.5
0.5
Our model is implemented in Pytorch [27] and trained using the Adam optimiser on the loss corresponding to We use learning rate of 1e-4, and the default setting of (81, B2) (0.9, 0.999) across all instances of model training.
26

---

C
Results
In this section, we provide additional details about our experiments, and present additional results.
C.1 Datasets
For disentanglement, we utilise standard disentanglement datasets: Cars3D [4], Shapes3D [24], and the 'real' variant of MPI3D [21], containing 3. 6, and 7 ground-truth Fo respectively. These datasets are procedurally generated via taking the Cartesian product of all possible FoV values per FoV type. Metadata is provided in Tables 12, 13 and 14, with continuous-valued FoV types marked by 4. We treat colour Fo as continuous variables due to their 10 linearly spaced, naturally ordered values, in line with [42].
Table 12: Cars3D dataset
Table 13: Shapes3D dataset
Table 14: MPI3D dataset
FoV types
FoV values
FoV types
FoV values
FoV types
FoV values
Car type
199 types
Floor colour!
10 colours
Object colour
6 colours
Object shape
6 shapes
Rotation +
24 angles
Wall colour
10 colours
Object sizet
sizes
Camera e
4 types
Object colourt
10 colours
Camera height
3 heights
Object size
sizes
Background colour
3 colours
Object type shapes
Horizontal axist
values
Azimuth
1.5 angles
Vertical uxis* 40 values
For the downstream regression task, use these disentanglement datasets and train downstream models to regress on each dataset's s continuous-valuee FoVs, as indicated by the $ symbol in Tables 12, 13 and 14. For abstract visual reasoning [30], we employ the Raven's 's Progressive Matrix (RPM) style dataset from [30], where each sample consists of context panels and answer panels derived from Shapes3D images. The model must infer abstract visual relations from the context panels to select the correct answer among six options.
C.2 Baseline Implementations and Experimental Settings
C.2.1 Experiment Compute Resources
For the generative weakly supervised, vvllarvvlued baselines (i.e.. AdaGVAE-K, GVAE. MLVAE, VAE, the Shu model), and our model, we perform model training on a single Nvidia RTX4090 GPU. We also perform all associated experiments for these models (i.e., downstream model training. downstream model evaluation, disentanglement evaluation, etc.) on this single Nvidia RTXX090 GPU. Our model takes approximately 1.5 hours to fully train (i.e., run 200,000 iterations) on the Cars3D and Shapes dataset, and approximately 4.0 hours to fully train on the MPI3D dataset.
For the vector-valued baselines of COMET and VCT, we perform model training, and all associated experiments on single Nvidia V100 GPU.
C.2.2 Disentanglement Models
For the generative weakly supervised valued baselines (AdaGVAE-k, GVAE, MLVAE, Slow- VAE, Shu model) and our model, training and all related experiments (downstream training, eval- uation, disentanglement) were conducted on single Nvidia RTX4090 GPU. Our model requires approximately 1.5 hours to train (200,000 iterations) the Cars3D and Shapes datasets, and about 4.0 hours on the MPI3D dataset.
Vector-valued baselines COMET and VCT were trained and evaluated on single Nvidia V100 GPU, We utilised the PyTorch-baae based open- implementations from [42] for Ada-GVAE, GVAE, MLVAE, and VAE, verified to reproduce reported results. Official implementations from the authors were used for COMET and VCT [36, 47]. For the GAN-based Shu model, we converted the TensorFlow implementation from [35] to PyTorch and ensured it reproduced official results. Baseline models employed recommended hyperparameters; otherwise, hyperparameter tuning was conducted using the same Weights and Biases (WandB) hyperparameter sweep as our model.
As discussed in Section 5, all weakly supervised baselines except Ada-GVAB assume access to J, the differing FoV between each pair (z, x²), matching our model's 's supervision level. To make Ada-GVAE
27

---

comparable, we modified it to receive the ground-truthy k = |4], the number of changing FoVs per sample, termed Ada-GVAE-k. This adjustment resulted superior or comparable performance the original AVA-GVAF. All models, except SlowVAE (which assumes all FoVs change), were trained with : = 1, ensuring consistent training conditions across models.
For each model's selected hyperparameters, including our own, we trained five models using five random seeds and aggregated the results. All models were trained for 200,000 iterations on each dataset.
C.2.3 Downstream Models
For FoV regression, following [42], we use simple, generic MLP model, with the architecture listed in Table 15. For each disenttnglement dataset, the MLP regression model receives representations of images produced by a representation learner, and is trained predict the ground-truth FoV values in supervised fashion. Performance is measured using the R² score on held out, randomly selected test of 1,000 samples.
For each representation learning model instantiated with a random seed, we train 2 MLPs by uniformly sampling the number of output nodes in the first, second, and fifth layers specified in Table 15, resulting in a total of 10 MLPs per learner (we use 5 random seeds). Reported results are averaged over these models. Notably, the MLPs have no prior knowledge of the representation type (e.g.. scalar-tokened symbolic, vector-tokened symbolic, or fully continuous compositional) since no inductive biases or representatioon-seciii optimisations are applied, aside from the supervised MSE loss.
Each MLP is trained using the Adam optimiser with MSE loss, learning rate of 1 10-4, and default (3i, (2) = (0.9, 0.999) settings.
Table 15: MLP architecture
MI.P
Linear di,di E [256, 512] ReLU
Linear d2.d3 d2 E [256, 512]
ReLU Linear dim(x)
ReLU Linear dim(z)
ReLU
Linear dg. E [128, 256]
ReLU Linear
For the abstract visual reasoning task, following [30], we use the Wild Relation Network (WReN) model [18] representations produced by representation learners to predict ground-truth answers for each RPM matrix. For each of model instances produced by a random seeds, we randomly sample 2 possible configurations of the WReN model by uniformly sampling either 256 or 512 for the edge MLP, g. and 128 or 256 hidden units for the graph MLP, f, in line with [30]. We however, fix the number of hidden layers in g to and in representing the smallest possible number of hidden layers, to constrain the capacity of the WReN model. As we use 5 random seeds, and sample 2 configurations per seed, we produce 10 WReN models for each representation learner. All WReN models are trained using Adam optimisation on the BCE loss between the predicted logits and ground-truth labels, with a learning rate of 1e-5 and the default setting of (81,32) (0.9, 0.999).
C.2.4 Experimental Controls
To ensure that the performance improvements of our model are attributable to the distributed, flexible nature of the Soft TPR framework, we implement controls eliminate confounding variables.
Disentanglement Controls
To ensure performance gains in disentanglement metrics (Tables 16 and 17) are not attributable to our model's additional parameters (with our model's the filler embedding matrix Megs adding
28

---

13.568, 1,824, and 1,600 parameters for Cars3D, Shapes3D, and MPI3D respectively to standard autoencoding framework), we adjust baseline models with fewer parameters. Specifically. for SAowVAF, Ada-GVAE-k, GVAE, MI,-VAF, and the Shu model, we increase their parameters by adding filters or layers until they match or exceed our model's parameter count, denoted with * in Tables 16 and 17. VCT and COMET are substantially more parameter hungry (10+ million) than our model, so parameter controls are not applicable to these vector-valued symbolic compositional baselines. Consistent with [40], we observe that increasing parameters in these generative, scalar- tokened baselines does not enhance disentanglement performance. Therefore, we proceed with representation learning convergence experiments and downstream model tests using the original models, not the
variants.
Downstream Task Controls
To ensure that any performance boosts on the downstream tasks of FoV regression and abstract visual reasoning are not attributable to our representation's increased dimensionality, we post- process representations produced by all models (including models producing higher-dimennional representations) to match the dimensionality of our representation. Specifically, generative, weakly supervised models with scalar-tokened representations produce representations with a dimension of 10 for all datasets, whereas our Soft TPRs have dimensions of 1536 (Cars3D), 512 (Shapes3D), and 320 (MP13D) of our Soft TPRs, highlighting the necessity of such a control.
To perform this control, we apply separate methods for the For scalar-tokened models, we multiply each dimension A: of the latent vector, & with random emhedding vector 6 E IR from an fixed, randomly initialised, semi-orthogonal embedding matrix E Rdx and subsequently concatenate all multiplied random embedding vectors together to form the aimneniionalitycoorroooo
and vector-tokened models.
representation, (4,e),..., (10e10)) E E 10d d is a dataset specific representation is at least as small
integer that ensures that the size of the dimensionnlity-contooli as the dimensionality of the Soft TPR representation for the given dataset, i.e., d == [dim(z)]/ / where: z is the Soft TPR representation produced by given dataset. Note that choose a semi- orthogonal embedding matrix with the intuition that this ensures the maximal distinguishability of each continguous subset of dimensions corresponding to 4e;,
For COMET and VCT, which both produce vector-tokened representations, we consider alternative methods of performing dimensionality-contte
COMET's representations have dimensionality of 640 for all datasets, and so, the model produces lower-dimensional representations than ours for Shapes3D and Cars3D, but not MPI3D. VCT, on the other hand, produces representations with dimension 5120, and so, produces higher-dimensional representations than ours for all datasets. For any vector-tokened representation with higher dimensionality than our representation, we apply PCA- based postprocessing to the model representation, reducing the dimensionality of each vector-tokened value in the representation to the required dimensionality, d, where d' [dim( 2)] /dim( (bbseeline). before concatenating all PCA-reduces vectors together to produce the modified representation. For any seckretokened representation produced by COMET with lower dimensionality than ours, we apply a simple matrix multiplication between the representation, and a randomly initialised matrix of required dimensionality to embed the representation into the came-dimensional space as ours.
We denote the results produced by all such postprocessing by f in relevant tables, and indicate such postprocessing the captions of relevant plots.
C.3 Disentanglement
In reporting the disentanglement metric results for baseline models, we use published results where applicable, i.c., the results published in [47] for VCT and COMET and the published results for Slow [39]. For GVAE, MLVAE, and Ada-GVAE-K, we evaluate disentanglement using the Pytorch-based based implementation of disentanglement metrics, [42], which corresponds to a Pytorch- based implementation of the official disentanglement lib [25]. We also use this implementation to evaluate the disentanglement of representations produced by all models.
C.3.1 Disentanglement Metrics
In line with [47]. we consider the following 4 disentanglement metrics: the Factor score [24], the DCI Disentanglement score [16] (we refer DCI Disentanglement DCI), the Beta VAE score [10], and the MIG score [15]. We provide a brief overview of all 4 disentanglement metrics, and
29

---

refer interested readers to the papers these metrics are introduced in for further details, as well as the Appendix of [25] for more details on how the metrics are implemented [42] and [25].
FactorVAF Metric: A randomly selected FoV of the dataset is fixed, and a mini-batch of observations is subsequently randomly sampled. The representation learner produces representations for the samples. Disentanglement is quantified using the accuracy of a majority vote classifier that predicts the index of the ground-truth fixed FoV based on the index of the representation vector with the smallest variance.
DCI Metric: A Gradient Boosted Tree (GBT) is trained to predict the ground-truth FoV values from representations produced by a representation learner. The predictive importance of the dimensions of representation is obtained using the model's feature importances. For each sample, a score is computed that corresponds to one minus the entropy of the probability that a dimension of the learned representation is useful for FoV prediction, weighted by the relative entropy of the corresponding dimension. An average of these scores over the mini-batch of samples is taken to produce the final score,
This metric quantifies disentanglement by predicting the index of a fixed FoV from representations produced by a representation learner, similar to the FactorVAE metric. In contrast to the Factor VAE metric, however, the VAE metric uses a linear classifier on difference vectors to predict the index of the fixed FoV. Each difference vector is produced by taking the difference between representations produced for a pair of samples (z, x') with one underlying fixed FoV.
BetaVAE Metrie:
MIG Metric: For each FoV, the MIG metric computes the mutual information between each dimension in the representation, and the corresponding FoV. The score obtained by computing the average, normalised difference between the highest and second highest mutual information of each VoV with the dimensions of the representation.
C.3.2 Evaluating the Disentanglement of Soft TPRs
Since disentanglement metrics are typically computed under the assumption that the representation corresponds to a concatenation of scalar-valued FoV tokens, we now detail how we compute the above disentanglement metrics the Soft TPR, continuous compositional representation. Similar amendments have been made in COMET and VCT, so that the disentanglement of their representations, corresponding to concatenation of vector-valued FoV tokens, can be computed.
FactorVAE metric: To compute the FactorVAE score of our Soft TPR, we produce a dimeennnioaa vector, v, for each Soft TPR, where Nn corresponds to the number of roles, i.e. the FoV types. We produce by simply populating each dimension, 0; with the index of the quantised filler that role r; is bound to, i.e. we set to m()). We use the resulting v to compute the variances required by the Factor metric, noting that if the FoV type corresponding to role is fixed, and this is reflected in our representation, dimension i of the u vectors produced in mini-batch should clearly have the smallest variance, as all fillers, fm(()). that role r; binds to, will have the same identity across the mini-batch.
DCI metric: As the DCI relies on computing the ground truth FoV values from NR- da-dimensional representations produced by the representation learner, we again, follow a similar procedure as the above, in converting our (Dr- DR) dimensional Soft TPR representation to a Ni-ddimenionnal representation. That is, we simply consider N Nn-dimensional vector, v, where each dimension, U; is populated by the index of the quantised filler, m(i) that role r; is bound to, and use this vector to compute the corresponding DCI result.
For the VAE metric, as each sample used to train the linear classifier consists of a Ng-dimensional difference vector obtained by computing the difference between the NR- dimensional scalar-tokened compositional representations. we obtain the following difference vector, d, for our Soft TPR representations, For each role i E {I.... NR). we obtain the corresponding quantised filler embeddings for each sample x, x' in the pair. i.c., we obtain & (fm())) ) E (t)) For each pair of quantised filler embeddings, we obtain scalar distance measure corresponding to the cosine similarity between the the pair. The i-th dimension of is populated using this value. We use difference vectors obtained in this way to compute the VAE metric.
Beta VAE metric:
MIG metric: For the MIG metric, which relies discretisation of the values in each dimension i of the Nn-dimensional scalar-tokened compositional representations to compute the discrete mutual
30

---

information, we apply the same postprocessing technique as in the FactorVAE and DCI metric, to evaluate MIG on our Soft TPRs. That is, we produce the same -n-dimensional vector, v, noting that this choice of postprocessing also performs the discretisation required by the MIG computation.
C.3.3 Full Results
We now present unabridged results for all considered disentanglement metrics, denoting the learnable parameter control modification of each relevant baseline with the symbol * As can be seen in Tables 16 and 17, our Soft TPR representations are explicitly more compositional (as quantified by the disentanglement metric scores) compared to all considered baselines, with especially notable performance increases on the more challenging datasets Cars3D and MPI3D.
Table 16: Factor VAE and DCI scores
Cars3D
Shapes3D
MP13D
Models
Vac score
DCI
VaC score
DCI
Factor score
DCI
Symbolic scalar-tokened compositional representations
Slow VAE
0,902 0.035
0,509 0.027
0.950 0.032
0.047
0,455 0.083
0.355
0.027
Slow-VAE VAE
0.872 + 0.038
0,518 0.039
0.961 + 0.028
0.867 + 0.028
0.421 + 0.091
0.317
0.039
Ada -VAE-
0,947 0.064
0.664 = 0.167
0.973 : 0.006
0,963 0.077
0.496 + 0.095
0.343 0.040
Ada-GVAE-K"
0 0.051
0,641 = 0.179
0.932 + 0.012
0,923 1:23:000
0,451 0.129
0.319 0,056
GVAE
0.877 + 0.081
0,262 0.095
0.921 + 0.075
0.842 + 0.040
0.378 + 0.024
0.245 0.074
GVAR"
0.841 + 0.123
0.219 0.012
0.881 + 0.129
08800 0.102
0.341 + 0.067
0.216 0.109
ML-VAE
0.870 0.052
0,216 0.063
0,835 0.111
0..115
0,390 0.026
0.251 0.029
ML-VAE
0,881 + 0.041
0,220 0.051
0.823 0.091
0.714 0.091
0,401 = 0.019
0.240 0.043
Shu
0,573 0.062
0.032 0.014
0.265 + 0.043
0.017 0.006
112887 + 0.034
0.033 0.008
Shu
0.551 0.062
0.019 0.021
0.297 0.031
0.020 0.006
0.057
0.029 0.010
Symbolic vector-tokened compositiona representations
VCT
0.966 0.029
0.382 0.080
0.957 0.043
0,884 0.013
0.689 + 0.035
0.475
0.005
COMET
0.339 + 0.008
0.024 0.026
0.168 + 0.005
0.002 + 0.000
0,145 + 0.024
0.005 0.001
Fully continuous compositional representations
Ours
0.999 + 0.001
0.863 0.027
0.984 + 0.012
0.926 +: 0.028
0.949 = 0.032
0.828 = 0.015
Table 17: BetaVAE and MIG scores
Car33D
Shapes3D
MPB3D
Models
BetaVAE score
MIG
BetaVAE score
BetaVAE score
MIG
Symbolic scalar-tokened compositional representations
Slow-VAE
1,000 = 0.000 (=)
0.104 0.018
1.000 =0,000 (=)
0.615
0.045
0.666 = 0.069
0.329 = 0.026
Slow-VAE"
1.000 = 0.000
(=)
0,071 0.013
1.000 10.000 (=)
0.655 0.067
0.629 : 0.079
0.287 : 0.045
Ada GVAE-K
1,000 0.000
0.395 0.095
1,000
1 0.000
0.556 0.064
0.750 1 0.053
0.213 1 0.064
Ada-GVAE-k
1,000 0.000
0,321 + 0.100
1JM00 + 0,000
0.498 0.073
0.783 + 0.061
0.241 + 0.079
GVAE
1,000 0.000
0.0% 0.036
0.998 0.004
0,251 =0.072
0.704 0.072
0.145 = 0.074
GVAE-
1.000 = 0.000
0,100
1.000 1 0.000
0,203 20.089
0.681 = 0.061
0.109 = 0.087
MLVAE
1.000
(=)
0.088 0.020
0.165
0.039
0.142 0.062
MLVAE'
1,000 = 0.000
(=)
0.050 0.058
0,921 0.052
0.298 0.091
0.689 = 0.041
0.096 = 0.071
Shu
0,912 0.022
0.025 0.012
0.498 0.064
0.005
0.003
0.353 1 0.040
0.013 1 0.007
Shu"
0,923 0.031
0.015 0.009
0,512 0.071
0.006 0.002
0.327 0.059
0.009 + 0.011
Symbolic vector tokened compusitional representations
VCT
L,00 + 0,000 (=)
0.117 + 0.045
0.999 + 0.0004
0.525
0.028
0.844 + 0.038
0.227 + 0.048
COMET
0.343 0.006
0.000 0.000
0.166 0.004
0.0002 4 0.000
0.144 0.005
0.000 : 0.0001
Fully contieuous compositional representations
Ours
1.000 = 0.000 (=)
0,348 1 0.0124
1.000 + 0.000 (=)
0.471 = 0.088
1.000 :: 0.000
0.620 :: 0.067
C.4 Representation Learning Convergence
We additionally examine representation learning convergence by evaluating the representations produced at 100, 1,000, 10,000, 100,000, and 200,000 iterations of model training, where the latter stage of 200,000 iterations corresponds to fully trained models. To quantify representation learning convergence, we evaluate both 1) the explicit compositionality of representations produced at these stages of training (as quantified by disentanglement metric performance), and 2) the usefulness of these representations for the downstream tasks of FoV regression and abstract visual reasoning.
31

---

In all line plots, we plot the mean, and indicate the standard deviation by the shaded regions. We use the same legend for all plots, where (grey) denotes VAE, (orange) denotes AdaGVAE-k: (green) denotes GVAE, 3 (red) denotes MI.AF,, 4 (purple) denotes Shu, 5 (pink) denotes VCT, 6 (brown) denotes COMET, and 7 (blue) denotes our model, Soft TPR Autoencoder.
C.4.1 Disentanglement
We first present line plots of representation learner convergence for each of the four considered disentanglement metrics (i.e., the FactorVAE, DCI, BetaVAE and MIG scores) for all three disentan- glement datasets (Cars3D, Shapes3D, MPI3D). A series of tables containing the values associated with these line plots is presented following the plots. As the disentanglement results produced by our learnable parameter controls for the models of Ada-GVAE-k, GVAE, ML-VAE and Shu, do not achieve superior disentanglement results compared to the original models, we only present disentanglement convergence results for the original variants of all baseline models.
32

---

Cars3D
1.0
Factor Score
0.6
0.2
0.0
10a
10a
10
103
Iteration count
Figure 3: Factor score convergence on the Cars3D dataset
Cars3 D
0.8
DO score
0.4
0.2
10a
10
10a
Iteration count
Figure 4: DCI score convergence on the Cars3D dataset
33

---

Cars3D
1.0
0.9
0.8
Score
0.6
0.5
0.4
10a
10a
10
103
Iteration count
Figure BetaVAE score convergence on the Cars3D dataset
Cars3
0.40
0.35
0.30
0.25
0.15
0.05
10a
10c
10a
Iteration count
Figure 6: MIG score convergence on the Cars3D dataset
34

---

Shapes3D
0.8
Factor Score
0.6
0.4
0.2
0.0
10a
10a
10
103
Iteration count
Figure 7: Factor score convergence on the ShapesåD datasct
Shapes3D
1.0
0.8
DO score
0.6
0.4
0.2
0.0
10a
10
10a
Iteration count
Figure 8: DCI score convergence on the Shapes3D dataset
35

---

Shapes3D
0.8
Bela Score
0.6
0.2
10a
10a
10
103
Iteration count
Figure 9: VAE score convergence on the Shapes3D datasct
Shapes3D
0.7
0.6
0.5
MIG score
0.3
0.2
0.1
10a
10
10a
Iteration count
Figure 10: MIG score convergence the Shapes3D dataset
36

---

MPI3D
0.8
Factor Score
0.6
0.4
0.0
10a
10a
10
103
Iteration count
Figure 11: Factor score convergence on the MPI3D dataset
MPI3D
0.6
score
10a
10'
10
10a
Iteration count
Figure 12: DCT score convergence on the MPI3D dataset
37

---

MPI3D
1.0
0.8
Bela score
0.4
0.2
10a
10a
10
103
Iteration count
Figure 13: Beta VAE score convergence on the MP13D dataset
MPI3D
0.7
0.6
MIG Score
0.3
0.2
0.1
0.0
10a
10'
10
103
Iteration count
Figure 14: MIG score convergence on the MPI3D dataset
38

---

Table 18: Representation learner convergence the Cars3D dataset (Factor score)
Factor score
Models
10a iterations
10 iterations
10 iterations
10 iterations
10 iterations
Symbolic scalar-tokened compositional representations
Slow-VAL
0,455 + 0.020
0,546 0.043
0.624 + 0.050
0.713 + 0.054
0.743 :: 0.046
Ada-GVAR -V--GVEE-k
0,460 = 0.024
0.642 + 0.031
0.755 = 0.042
0.873 = 0,077
0.904 = 0.082
GVAE
0,478 = 0.026
0.560 + 0.036
0.641 + 0.052
0.701 + 0.044
0.722 = 0.035
MLVAE
0,481 = 0.022
0.575 0.059
0.652 = 0.041
0.721 = 0.035
0.719 = 0.031
Shu
0.000 = 0.000
0.294 0.162
0.229 0.194
0.000 = 0.000
0.069 0.138
Symbolic vector-tokened compositional representations
VCT
0.618 + 0.046
0.887 0.074
0.907 + 0.036
0.918 + 0.037 0.531 + 0.024
0,915 - 0.028
COMET
0.531 + 0.024
0.531 + 0.024
0.531 + 0.024
0.531 + 0.024
Fully continuous compositional representations
Ours
0.639 + 0.045
0.898 0.038
0.999 0.001
0.999 + 0.001
0.999 0.001
Table 19: Representation leamer convergence on the Cars3D dataset (DCI score)
DCI score
Models
10a iterations
10 iterations
10 iterations
10 iterations
2 x 10 iterations
Symbolic scalar- scalar-tokened compositional representations
SAO
0.047 + 0.012
0.115 0.021
0.154 + 0.031
0.261 + 0.041
0.308 0.073
Ada-GVAE-k
0.054 0.011
0.165 0.012
0.338 0.042
0.558 + 0.167
0.694 0.140
GVAE
0.071
0.024
0.098
0.021
0.147 0.025
0.216 + 0.053
0.238 0.061
MIVAR
0.062 + 0.022
0.100 0.018
0.175 + 0.050
0.197 + 0.062
0.238 + 0.077
Shu
0.007 = 0.005
0.013 + 0.011
0.018 + 0.012
0.038 = 0.036
0.026 = 0.015
Symbolic vector-tokened compositional representations
VCT COMET
0.055 0.044
0.240 0.043
0.227 0.024
0.226 + 0.020
0.230 0.044
0.020 0.013
0.015 0.011
0.011 + 0.006
0.008 + 0.003
0.009 0.003
Fully continuous compositional representations
Ours
0.379 = 0.117
0.381 + 0.026
0.812 + 0.033
0.843 + 0.024
0.832 = 0.048
Table Representation learner convergence on Cars3D dataset (Beta VAE score)
BetaVAE score
Models
10a iterations
10 iterations
10 iterations
10a iterations
2 x 10 iterations
Symbolic scalar-tokened compositional representations
Slow-VAF
0.937 + 0.025
0.999 + 0.002
1.000 + 0.000 (=)
1.000
0.000 (=)
1.000 + 0.000
Ada-GVAE-k
0.971 = 0.011
1.000 = 0,000
1.000 + 0.000
1.000 = 0.000
1.000 = 0.000
GVAE
0.958 + 0.027
0.999 = 0.007
1.000 + 0.000
1.000
0.000
1.000 = 0.000
MLVAE
0,964 1 0.011
1.000 : 0.000
1.000 1 0.000
1.000
0.000
1.000 1 0.000
Shu
0.534 = 0.164
0.742 0.072
0.728 0.105
0.433 0.016
0.449 = 0.099
Symbolic vector-tokened compositional representations
VCT
0.889 = 0.044
1.000 =: 0,001 (-)
1.000 + 0.000 (=)
1.000 = 0.000 (=)
1.000 + 0.000 (=)
COMET
0.592 = 0.027
0.592 + 0.028 0.592 + 0.028
0.592 + 0.028
0.592 = 0.028
Fully continuous compositional representations
Ours
0.953 = 0.042
1.000 0.000 (=) 1.000 + 0.000 (=) 1.000 0.000 (=)
1.000 0.000
39

---

Table 21: Representation learner convergence the Cars3D dataset (MIG score)
MIG score
Models
10a iterations
10 iterations
10a iterations
10 iterations
10 iterations
Symbolic collee-tokened compositional representations
Slow- VAE
0.008 + 0.004
0.042 0.017
0.061 + 0.024
0.071 + 0.028
0,081 = 0.031
ula-GVAE-E
0,011 = 0.005
0.068 = 0.013
0.092 = 0.035
0.241 = 0.095
0.312 0.085
GVAE
0.014 = 0.006
0.031 0.015
0.056 = 0.008
0.084 = 0.031
0.095
0.032
MLVAE
0.012 0.007
0.037 0.024
0.057 + 0.018
0.085 + 0.021
0.087 0.017
Shu
0.007 = 0.004
0.014 0.005
0.019 0.010
0.002 = 0.001
0.001 0.001
Symbolic vector-tokened compositional representations ceapppptioaalepppeesetttttonnn
VCT
0.001 = 0.001
0.034 0.016
0.033 + 0.021
0.033 + 0.022
0.032 = 0.015
COMET
0.000 + 0.000
0.000 + 0.000
0.000 1 0.000
0.000 + 0.000
0.000 0.000
Fully continuous compositional representations
Ours
0.016 0.010
0.136 0.009 0.344 0.011
0.349 = 0.010
0.349 0.010
Table 22: Representation learner convergence on the Shapes3D dataset (Factor score)
Factor score
Models
10a iterations
10 iterations
10 iterations
10 iterations
2 x 10th iterations
Symbolic scalar-tokened compositional representations
SAO
0.270 + 0.025
0.392 0.095
0.547 + 0.138
0.762 + 0.156
0.923 0.091
Ada-GVAE-k
0.296 0.047
0.577 0.023
0.815 + 0.057
0.960 0.057
0.961 0.070
GVAE
0,263 = 0.036
0.469
0.063
0.674 + 0.086
0.890 : 0.058
0.879 + 0.077
MIVAE
0.166 + 0.084
0.457 0.045
0.621 + 0.078
0.746 + 0.080
0.765 + 0.084
Shu
0,000 = 0.000
0.000 + 0.000
0.242 + 0.072
0.194 1 0.021
0.227 + 0.016
Symbolic vector-tokened compositional representations
VCT
0.263 0.000
0.984 0.031
1.000 + 0.000
0.976 + 0.033
0.967 0.044
COMET
0.268 0.022
0.265 0.039
0.334 1 0.028
0.384 + 0.040
0.440 0.071
Fully continuous compositional representations
Ours
0.356 = 0.011
0.435 + 0.048
0.975 + 0.040
0.995 + 0.005
0.960 + 0.053
Table 23: Representation learner convergence on the Shapes3D dataset (DCI score)
DCI score
Models
10a iterations
10 iterations
10 iterations
10 iterations
x 10 iterations
Symbolic
compositional representations
Slow-VAE
0.056 + 0.015
0.192 0.060
0.353 + 0.111
0.616 + 0.147
0.749 = 0.118
Ada-GVAF-k
0,053 + 0.018
0.331 + 0.022
0.662 + 0.042
0.944 + 0.063
0.964
0.067
GVAE
0.042 = 0.013
0.198 0.046
0.438 + 0.064
0.749 = 0.053
0.835
0.038
MIVAE
0.026 0.003
0.183 0.036
0.407 0.051
0.668 0.084
0.739
0.103
Shu
0.030 + 0.007
0.028 0.012
0.032 + 0.006
0.019 + 0.010
0.018
0.006
Symbolic vector-tokened compositional representations
VCT
0.044 = 0.000
0.817 + 0.114
0.909 + 0.010
0.887 = 0.026
0,880 0.022
COMET
0.017 = 0.006
0.031 0.005
0.062 = 0.017
0.173 = 0.064
0.233 0.066
Fully continuous compositional representations
Ours
0.057 0.018
0.183 0.077
0.860 0.052
0.887 + 0.068
0.924 0.027
40

---

Table 24: Representation learner convergence the Shapes3D dataset (BetaVAE score)
Beta
Models
10a iterations
10a iterations
10 iterations
10a iterations
2 x 10 iterations
Symbolic scalar-tokened compositional representations
Slow-VAE
0,539 = 0.061
0.694 1 0.012
0.949 1 0.037
0.989 1: 0.020
1.000 + 0,000 (-)
Ada-GVAE-k
0,553 0.050
0.699 + 0.016
0.999 0.001
1,000
0.000 (=)
1.000 + 0,0M0) (=)
GVAE
0.505
0.031
0.683 + 0.019
0,974 0.025
1,000
0.000 (=)
0,998 0.004
MLVAR
0.484
0.015
0.684 + 0.013
0,970 + 0.029
0.949 = 0.041
0.974 = 0.036
Shu
0.204 0,311
0.357 + 0.050
0.495 + 0.040
0.379 + 0.056
0.403 + 0.058
Symbolic vector-tokened compositional representations
VCT
0.889
0.044
1.000
0.000
1.000 + 0.000 (=)
1.000
0.000
1,000 + 0.000
COMET
0.395
0.000
1.000 0.000
1.000
0.000 (-)
1,000 1 0.000 (-)
1.000 1: 0.000
Fully continuous compositional representations
Ours
0.638 0.015
0.710 1 0.035
1.000 1 0.000 (-) 1.000 0.000 (-) 1.000 1: 0.000 (-)
Table 25: Representation leamer convergence on the S datasct (MIG score)
MIG score
Models
10a iterations
10 itcrations
10 iterations
10 iterations
x 10 iterations
Symbolic scalar-tokened compositional representations
Slow-VAL VAE
0,018 + 0.009
0.080 + 0.053
0.163 + 0.107
0.297 + 0.156
0.386 0.169
Ada-GVAE-K
0.023 + 0.012
0.189
0.071
0.288 1 0.138
0.520 1 0.149
0.555
0.152
GVAE
0.019 0.012
0.074
0.053
0.153 0.039
0.225 = 0.052
0.243
0.064
MLVAE
0.010 0.004
0.082
0.042
0.179 0.076
0.305 + 0.126
0.354
0.148
Shu
0.025 0.007
0.008 0.005
0.013 + 0.005
0.010 + 0.003
0.008 0.004
Symbolic vector-tokened compositional repeessttoonii
VCT
0.000 = 0.000
0.405 0.103
0.466 + 0.023
0.469 + 0.025
0.448 0.065
COMET
0.000 0.000
0.004 0.003
0.013 0.010
0.037 + 0.022
0.044 0.025
Fully continuous compositional representations
Ours
0.015 = 0.005
0.043 0.018
0.343
0.412 = 0.066
0,402 = 0.091
Table 26: Representation learner convergence the MPI3D dataset (Factor score)
Factor score
Models
102 iterations
10s iterations
10a iterations
10s iterations
x 10 iterations
Symbolic scalar-tokened compositional representations
Slow- VAE
0.240 = 0.008
0.300
0.015
0.325 + 0.014
0.458 = 0.095
0.471 0.109
Ada-GVAE-k
0.246 + 0.012
0.275 0.011
0.321 + 0.011
0.314 + 0.050
0.318 + 0.050
GVAE
0.236 + 0.020
0.253 + 0.017
0.277 + 0.023
0.237 + 0.020
0.232 + 0.023
MLVAE
0.235 = 0.008
0.257
0.014
0.261 = 0.018
0.229 = 0.021
0.240 0.019
Shu
0.000 0.000
0.061 0.075
0.173 + 0.089
0.093 + 0.077
0.124 0.056
Symbolic vector-tokened happreuuoooooiiiooooeeeeeettnno representations
VCT
0,220 + 0.024
0.360 0.017
0.411
0.678 + 0.125
0.678 1 0.045
COMET
0.227 = 0.019
0.233 + 0.016
0.233 0.016 0.243 = 0.005
0,233 = 0.015
Fully continuous compositional representations
Ours
0.334
0.079
0.350 0.049
0.750 + 0.065
0.937 + 0.034
0.951
0.083
41

---

Table 27: Representation learner convergence the MPI3D dataset (DCI score)
DCI score
Models
10a iterations
10 iterations
10 iterations
10 iterations
10 iterations
Symbolic scalar-tokened compositional representations
Slow-VAL
0,090 +: 0.013
0.125 0.014
0.200 1 0.014
0.341 + 0.095
0,381 +: 0.081
Ada-GVa -e--GVEE-k
0.097 = 0.008
0.117 = 0.009
0.174 = 0,005
0.319 = 0,038
0.338 = 0.038
GVAE
0.101 + 0.019
0.111 + 0.011
0.142 + 0.022
0.247 = 0.062
0.258 = 0.063
MLVAE
0.097 + 0.015
0.103 0.010
0.137 + 0.013
0.252 + 0.031
0.266 0.026
Shu
0.083 = 0.000
0.065 0.030
0.063 = 0.018
0.036 = 0.012
0.050 0.012
Symbolic vector-tokened compositional representations
VCT
0.123 + 0.021
0.220 0.029
0.535 1 0.048
0.580 1 0.046 0.029 + 0.003
0,611 - 0.035
COMET
0.039 1 0.016
0.036 0.011
0.033 1 0.008
0.032 1 0.007
Fully continuous compositional representations
Ours
0.172 + 0.051
0.174 0.037
0.670 + 0.060
0.813 + 0.010
0.799 + 0.078
Table 28: Representation learner convergence on the MPI3D dataset (BetaVAE score)
BetaVAE
Models
10a iterations
10a iterations
10 iterations
10 iterations
2 x 10 iterations
Symbolic scalar tokened compositional representations
Slow- VAE
0.359 0.004
0.484 0.019
0.629 1 0.038
0.833 + 0.035
0.851 0.045
Ada-GVAE-k
0.362 + 0.006
0.397 0.028
0.587 0.040
0.741 + 0.047
0.748 = 0.041
GVAE
0,352 - 0.012
0.378 0.015
0.506 1 0.041
0.683 1 0.071
0.703 - 0.062
MLVAE
0.343 + 0.021
0.394 1 0.022
0.507 1 0.015
0.672 1 0.038
0.697 1 0.035
Shu
0.170 = 0.057
0.294 0.026
0.325 = 0.038
0.269 = 0.023
0.278 0.021
Symbolic vector-tokened compositional representations
VCT
0.889
0.044
1.000 0.001
1.000 + 0.000
1.000 + 0.000
1.000 0.000
COMET
0,277 = 0.011
0.277 0.011
0.277 + 0.011
0.270 - 0.007
0.278 = 0.010
Fully continuous compositional representations
Ours
0.302 = 0.011
0.577 + 0.021
0.973 = 0.048
0.980 + 0.022
0.976 + 0.024
Table 29; Representation learner convergence on the MPI3D dataset (MIG score)
MIG score
Models
10 iterations
1 iterations
10 iterations
10a iterations
2 x 10 iterations
Symbolic scalar-tokened compositional representations
Slow- VAE
0.014 0.004
0.059 0.042
0.084 +: 0.061
0.206 0.120
0.214 0.125
Ada-GVAE-k
0.016 +: 0.005
0.039 1 0.011
0.113 + 0.051
0,213 1 0.062
0.214 1 0.057
GVAE
0.014 + 0.004
0.022 + 0.003
0.058 + 0.035
0.140 + 0.061
0.147 + 0.066
MIVAE
0.010 0.004
0.032 + 0.022
0.044 + 0.015
0.136 + 0.051
0.144 + 0.054
Shu
0.005 0.000
0.010 0.006
0.024 = 0.019
0.009 1 0.005
0.013 = 0.005
Symbolic vector-tokened composttinnl representations
VCT
0.000 + 0.000
0.013 + 0.003
0.216 + 0.055
0.240 + 0.050
0.248 + 0.057
COMET
0.000 0.000
0.000 0.000
0.000 + 0.000
0.000 0.000
0.000 + 0.000
Fully continuous compositional representations
Ours
0.037 + 0.020
0.046 + 0.018
0.393 + 0.063 0.612 + 0.068
0.514 + 0.202
C.4.2 Downstream Performance
We additionally evaluate the representation learning convergence by examining the usefulness of representations produced at different stages of training (i.c., 100, 250, 500, 1,000, 10,000, 100,000
42

---

and 200,000 iterations of training). To quantify 'usefulness', we consider performance of downstream models on the tasks of FoV regression, and abstract visual reasoning when trained using represen- tations produced by each stage of training. For each task, we present line plots, and additionally tables with values corresponding to each the plots. We use the same legend in the previous section, where 0 (grey) denotes Slow (orange) denotes AdaGVAE-k, 2 (green) denotes GVAE, 3 (red) denotes MLVAE, (purple) denotes Shu, 5 (pink) denotes VCT, (brown) denotes COMET, and 7 (blue) denotes our model, Soft TPR Autoencoder. We additionally provide all results for the dimensionality-conie
setting, where the dimensionality of representations produced by representation learners is held constant following the approach detailed in C.2.4, denoting this clearly in plot captions, and by the symbol f in the tables.
FoV Regression
As clearly visible in the tables and plots, generic regression models are able representations produced by our Soft TPR Autoencoder produced across almost all stages of training for all disentanglement datasets. Improvements are most notable in the low-iteration regime of 10a iterations, and across most stages of training for the more challenging task of FoV regression on the MPI3D dataset (Figures 19 and 20).
effectively
43

---

Cars3D
1.0
0.8
0.6
RSq
0.4
0.2
0.0
10a
10a
10
103
Iteration count
Figure 15: Convergence representation learners as measured by FoV regression on the Cars3D dataset (original sctting)
Cars3D
1.00
0.75
0.50
0.25
RSq
-0.55
0.50
0.75
1.00
10a
10a
10a
Iteration count
Figure 16: Convergence of representation learners measured by FoV regression on the Cars3D dataset cilmensionalityycomtooooo setting)
44

---

Shapes3D
1.0
0.9
0.8
0.7
0.5
0.4
0.2
10a
10a
10
103
Iteration count
Figure 17: Convergence of representation learners as measured by FoV regression on the Shapes3D dataset (original setting)
Shapes3D
1.0
0.9
0.8
0.7
0.6
0.4
0.2
10a
10
Iteration count
Figure 18: Convergence of representation learners measured by FoV regression the Shapes3D dataset (dimensionality-conee) controllld setting)
45

---

MPI3D
1.0
0.7
0.6
0.3
0.2
10a
10a
10
203
Iteration count
Figure 19: Convergence representation learners as measured by FoV regression on the MP13D dataset (original sctting)
MPI3D
0.9
0.8
0.7
R59
0.6
0.4
0.3
10a
10
Iteration count
Figure 20: Convergence of representation learners measured by FoV regression the MPI3D dataset peteeniioonaityyooorlllll setting)
46

---

Table 30: Convergence of representation learners measured by FoV regression the Cars3D dataset
Rt score
Models
102 iterations
10 iterations
10 iterations
10 iterations
10 iterations
Symbolic scalar-tokenede compositiona representations
Slow- VAE
0.233 +: 0.048
0.816 +: 0.017
0.892 0.007
0.925 0.005
0.935 + 0.004
Slow-VEE
0.203 + 0.074
0.797 + 0.009
0.891 + 0.006
0.917 = 0.006
0,937 = 0.001
Ada-GVAE-k
0.307 = 0.084
0.821 + 0.016
0.896 + 0.009
0.944 + 0.007
0.957 = 0.005
Ada-GVAE-kl
0.264 = 0.088
0.805 = 0.015
0.888 0.009
0,940 0.007
0.951 = 0.008
GVAE
0.319 0.073
0.831 = 0.012
0.901 0.009
0.942 0.004
0.947 = 0.005
GVAET
0.282 + 0.056
0.821 + 0.008
0.895 + 0.009
0.936 0.007
0.943 + 0.004
MLVAE
0.317 = 0.058
0,820 + 0.007
0.904 + 0.007
0.945 0.004
0,948 = 0.007
MLVAET
0.26 0.061
0.808 + 0.005
0.900 0.009
0.939 0.004
0.948 0.007
Shu
-0.016 0.016
0.003 + 0.019
0.343 0.037
0.076
0.075 = 0.051
Shu
0.012 = 0.007
+0.009 + 0.02
0.343 + 0.042
0.074 0.070
0.094 - 0.055
Symbolic vector-tokened compositional representations
VCT
0,080 + 0.001
0.829 +: 0.015
0.877 + 0.007
0.832 0.010
0.813 = 0.014
VCTI
0.038 + 0.014
0.243 0.03
0.655 0.011
0.631 0.029
0.623 = 0.036
COMET
-0.029 0.011
-0.035 0.023
-0.035 0.023
-0.035 = 0.023
-0.035 0.023
COMETT
-0.484 0.477
-0.495 + 0.512
-0.495 + 0.512
-0.515 + 0.47
-0.715 = 0.295
Fully continuous compositional representations
Ours
0.531 + 0.054
0.855 + 0.012
0.920 + 0.009
0.914 + 0.008
0,910 + 0.010
Table 31: Convergence representation learners as measured by FoV regression on the Shapes3D dataset
R score
Models
102 iterations
10 iterations
10 iterations
10 iterations
2 x 10 iterations
Symbolic scalar- tokened compositional representations
Slow-VAE
0.597 + 0.048
0.927 + 0.007
0.960 + 0.032
0,999 + 0.000 (=)
1.000 0.000 (=)
Slow- VAEf
0.564 +:20 0.060
0.920 +:: 0.011
0.955 = 0.037
0.999 + 0.000
0.999 + 0. 0.000
Ada-GVAE-k
0.684 + 0.068
0.974 + 0.007
0.997 + 0.000
1.000 + 0.000
(=)
1.000 0.000
(=)
Ada-GVAE-k
0.698 = 0.063
0.973 + 0.007
0.997 = 0.000
1.000 = 0.000
1.000 0.000
GVAE
0.565 1 0.034
0.951 i 0.015
0.997 1 0.000
1.000 1 0.000
1.000 0.000
GVAE!
0.565 1 0.022
0.951 0.014
0.998 0.001
1.000 1 0.000
1.000 0.000
MIVAE
0.551 + 0.059
0.959 + 0.012
0.997 = 0.000
0.999 + 0.000
1.000 0.000
MLVAEt
0.556 + 0.055
0.960 0.010
0.996 0.000
0.999 - 0.000
1.000 0.000
Shu
0.327 0.13
0.497 006
0.631 0.087
0.471 0.027
0.427 + 0.082
Shul
0.461 + 0.075
0.525 0.107
0.632 0.088
0.471 0.008
0.419 + 0.082
Symbolic vector-tokened compositional ruuposstiooalrppeeeenttttiioo representations
VCT
0,886 + 0.033
0,997 +: 0.000
0.995 = 0.001
0.97 + 0.013
0.954 + 0.008
VCTI
0.255 + 0.049
0.819 0.077
0.896 : 0.021
0.888 0.028
0.843 0.04
COMET
0.395 1 0.063
0.772 0.016
0.968 1 0.011
1.000 : 0.000
1.000 0.000
COMETT
0.474 + 0.062
0.809 0.013
0.970 0.011
1.000 0.000
1.000 0.000
Fully continuous compositional representations
Ours
0.981 1: 0.003
0.989 1: 0.007
1.000 1: 0.000 (=) 1.000 : 0.000 (=) 1.000 0.000 (=)
47

---

Table 32: Convergence of representation learners measured FoV regression the MPI3D dataset
R score
Models
10s iterations
10 iterations
10 iterations
10 iterations
x 10 iterations
Symbolic scalar-tokened compositional representations reprpeettooon
Jlow-VEE
0.537 + 0.012
0.634 + 0.015
0.818 + 0.016
0.960 = 0.013
0.980 + 0.005
Slow- VAE
0.512 0.009
0.612 + 0.023
0.809 0.025
0.964 + 0.008
0.981 0.004
Ada-GVAE-k
0,519 0.023
0.611 + 0.028
0.694 +: 0.007
0.750 + 0.008
0.762 0.014
Ada-GVAE-ki
0.524 -: 0.015
0.595 0.015
0.704 1:04:0000 0.025
0.753 = 0.012
0.774 0.003
GVAE
0.511 0.037
0.602 0.021
0.724 0.012
0.748 = 0.013
0.771 0.018
GVAEt
0.508 + 0.028
+.0200
0.717 0.013
0.773 + 0.024
+
MLVAE
0.504 + 0.016
0.605 0.001
0.717 0.020
0.780 = 0.016
0.773
MLVAE
0.497 0.019
0.602 0.008
1
0.766 0.010
0.790 0.027
Shu
0.270 0.041
0.498 1 0.095
0.592 0.082
0.470 + 0.036
0.441 0.041
Shu
0.299 0.040
0.519 + 0.062
0.568 0.079
0.474 + 0.036
0.438 0.037
Symbolic vector-tokened compositional representations
VCT
0,316 0.016
0.316 = 0.011
0.516 = 0.02
0.543 = 0.000
0.456 0.061
VCTi
0.251 = 0.04
0.396 = 0.041
0.546 = 0.027
0.508 = 0.016
0.561 0.051
COMET
0.266 0.004
0.266 1 0.004
0.266 0.004
0.266 + 0.004
0.266 0.004
COMET
0.240 0.010
0.236 1 0.008
0.236 0.008
0.236 1 0.008
0.236 0.008
Fully continuous composstionl representations
Ours
0.732 0.073
0.741 + 0.058
0.914 0.012 0.891 + 0.011
0.882 0.016
Abstract Visual Reasoning
present now present our full suite of results for the downstream task abstract visual reasoning. As demonstrated in Table 33 and the corresponding Figures 21 and 22, representations produced by our model at only 102 iterations of training are able to leveraged by downstream models to achieve 80.04% accuracy for the challenging abstract visual reasoning task, in contrast to the value of 33.1% obtained by the best performing baseline, representing 26.78 76% performance increase.
48

---

Abstract visual reasoning dataset
1.0
0.8
0.4
10a
10a
10
Iteration count
Figure 21: Convergence of representation learners as measured by classification performance on the abstract visual reasoning datasct (original setting)
Abstract visual reasoning dataset
--- 0
- 1
+ 2
1.0
- 3
5
0.8
7
Accuracy
0.6
0.4
0.2
10a
10
Iteration count
Figure 22: Convergence of representation learners measured by classification performance the abstract visual reasoning dataset (dimensionality-contooo setting)
49

---

Table 33:
Convergence of representation learners as measured by classification performance the abstract
visual reasoning dataset
Classification accuracy
Models
10 itcrations
10 iterations
10 itcrations
10 iterations
x 10 iterations
Symbolic scalar-tokened compositional representations
Slow-VAE
0.332 0.022
0.450 = 0.023
0.460 + 0.031
0.429 = 0.038
0.396 + 0.022
Slow- VAB
0,552 0.035
0.791 + 0.011
0.904 0.020
0.949 + 0.012
0,959 0.009
Ada-GVAE-k
0.397 0.021
0.375 + 0.038
0.409 0.029
0.455 1 0.047
0.472 1 0.037
Ada-GVAE-K
0.631 1.037
0.845 1 0.010
0.892 0.015
0,943 1 0.017
0.954 0.009
GVAE
0.346 0.029
0.424 0.034
0.431 0.020
0.444 0.057
0.478 0.032
GVAE!
0.554 + 0.031
0.815 0.008
189440000
+
0.936 + 0.008
MLVAE
0.336 0.016
0.404 0.013
0.431 0.038
0.432 = 0.023
MLVAE
0.550 0.025
0.824 1 0.021
0.878 1 0.014
1
0.925 0.024
Shu
0.208 0.052
0.273 1 0.012
0.331 1 0.007
0.317 1 0.025
0.310 1 0.038
Shu
0.207 0.048
0.399 = 0.072
0.462 + 0.024
0.433 + 0.023
0.444 + 0.013
Symbolic vector-tokened compositional representations
VCT
0.440 0.033
0.932 = 0.062
0.769 0.087
0.695 + 0.049
0.641 0.039
VCTI
0,432 0.043
0.723 = 0.065
0.731
0.017
0.479 = 0.023
0.458 0.040
COMET
0.369 0.077
0.683 + 0.037
0.848 0.049
0.501 + 0.251
0.984 0.006
COMETT
0.348 0.069
0.545 + 0.197
0.829 0.051
0.856 + 0.256
0.532 0.348
Fully continuous compositional representations
Ours
0,804 0.016
0.864 + 0.011
0.952 0.008
0.884 + 0.012
0.869 + 0.024
C.5
Downstream Performance
We present our full suite of experimental results that empirically demonstrate the utility of our Soft TPR representation from the perspective of downstream models, by considering the sample efficiency, and low-sample regime performance of downstream models on the tasks of FoV regression, and abstract visual reasoning.
C.5.1 Sample Efficiency Results
For sample efficiency, as mentioned in Section 5.2, and in line with [25], we compute ratio-based metric obtained by dividing the performance of the downstream model when trained using a restricted number of 100, 250, 500, 1,000 and 10,000 samples, by its performance when trained using all samples (19, 104,, 480,000, 1,036,800 and 100,000 samples for the tasks of regression on the Cars3D, Shapes3D, MPI3D datasets, and the abstract visual reasoning task respectively). As this metric is dependent on the performance of downstream models when trained using all samples, we do not compute this metric for representation learners where the corresponding downstream models achieve an R score of less than 0.5 for regression, as this may produce sample efficiency scores with very little semantic meaning (e.g. model that achieves a sample efficiency score of 0.9 when its final R² score is 0).). As result, we remove Shu from Shapes31) sample efficiency calculations, and COMET and Shu from the Cars3D sample efficiency calculations.
As many models for the abstract visual reasoning task have low classification accuracies on the held-out test set following training with the maximal number of 100,000 samples, we do not compute sample efficiencies for this task, and instead refer readers results in Section C.5.2 for the raw classification accuracies associated with each model.
Note that for all box plots, follow standard convention, and display the median in each box with solid line, where the box shows the quartiles of the corresponding values, and the whiskers extend to 1.5 times the interquartile range. We again, the same legend, where grey denotes VAE, orange denotes AdaGVAE-k, green denotes GVAE, red denotes MLVAE, purple denotes Shu, pink denotes VCT, brown denotes COMET, and blue denotes our model, Soft TPR Autoencoder.
50

---

Sample efficiency (100/all)
Sample efficiency (250/all)
1.0
0.8
Rsq ratio
0.6
Rsq
0.2
0.2
0.0
5
Figure 23: Downstream regression model sample efficiency the Cars3D dataset (original setting).
Sample efficiency (500/all)
Sample efficiency (1,000/all)
Sample efficiency (10,000/al)
1.0
0.8
0.8
R5q ratio
0.6
ratio
0.6
Rsq ratio
0.6
0.2
0.2
0.2
Figure 24: Downstream regression model sample efficiency on the Cars3D dataset (original setting).
51

---

Sample efficiency (100/all)
Sample efficiency (250/all)
1.0
0.8
0.8
R5q ratio
0.6
Rsq ratio
0.6
0.4
0.2
0.0
3
7
0
3
7
Figure 25; Downstream regression model sample efficiency on the Cars3D dataset (Cimensiionallyyycottooll sctting).
Sample efficiency (500/all)
Sample efficiency (1,000/all)
Sample efficiency (10,000/all)
1.0
1.0
0.9
0.9
0.8
Rsg ratio
Rsq ratio
0.6
0.7
0.6
0.4
0.5
0.3
02
Figure 26: Downstream regression model sample efficiency on the Cars3D dataset (dimensionality-com) controlled contrllld setting).
52

---

Sample efficiency (100/all)
Sample efficiency (250/all)
0.8
0.7
0.6
0.6
Rsq ratio
0.4
0.4
0.2
0.2
0.1
Figure 27: Downstream regression model sample efficiency on the Shapes3D dataset (original setting).
Sample efficiency (500/all)
Sample efficiency (1,000/all)
Sample efficiency (10,000/all)
1.0
1.0
1.0
0.8
0.8
R5q ratio
0.6
Rsq rato
0.6
Rsq ratio
0.6
0.4
0.2
Figure 28: Downstream regression model sample efficiency on the Shapes3D dataset (original setting).
53

---

Sample efficiency (100/all)
Sample efficiency (250/all)
0.8
0.8
0.6
0.6
0.5
Rsg ratio
0.3
0.1
0.0
2
3
7
0
3
4
7
Figure 29: Downstream regression model sample efficiency on the Shapes3D dataset (dimensionnaltyycootollll sctting).
Sample efficiency (500/all)
Sample efficiency (1,000/all)
Sample efficiency (10,000/all)
1.0
10
0.9
0.8
0.8
R5q ratio
0.6
ratio
0.6
Rsq ratio
0.4
02
0.3
0.0
0.2
Figure 30: Downstream regression model sample efficiency on the Shapes3D dataset ddimensionality-cootoolll setting).
54

---

Sample efficiency (100/all)
Sample efficiency (250/all)
0.7
0.6
0.6
Rsq ratio
0.3
0.3
0.2
0.0
7
Figure 31: Downstream regression model sample efficiency the MPI3D dataset (original setting).
Sample efficiency (500/all)
Sample efficiency (1,000/all)
Şample efficiency (10,000/all)
0.8
0.6
0.6
Rsq ratio
Rsq ratio
Rsq ratio
0.4
0.2
02
0.0
0.0
Figure 32: Downstream regression model sample efficiency on the MP13D dataset (original setting).
55

---

Sample efficiency (100/all)
Sample efficiency (250/all)
0.7
0.6
0.5
Rsq ratio
0.4
0.3
0.2
0.2
0.0
7
0
3
7
Figure 33: Downstream regression model sample efficiency the MPI3D dataset (dimensioonallyycoottolle sctting).
Sample efficiency (500/all)
Sample efficiency (1,000/all)
Sample efficiency (10,000/all)
0.6
Rsq ratio
Rsq ratio
Rsq ratio
0.4
0.2
02
0.0
0.0
Figure 34: Downstream regression model sample efficiency on the MP13D dataset wwwpaniionnniiyycoooootti setting).
56

---

Table 34: Downstream regression model sample efficiency the Cars3D dataset
ratio
Models
all samples
2.5 x 10% samples
5 10" samples
Vall samples
10ll samples
Symbolic scalar-tokened compositional representations
Slow-VAE
0.273 + 0.052
0.537 = 0.012
0.833 + 0.025
0.882 = 0.018
0.994 0.004
Slow-VARt
0.166 + 0.055
0.292 + 0.014
0,866 + 0.005
0.895 0.010
0.993 0.002
Ada-Gi GVAE-k
0.610 + 0.131
0.834 0.040
0.881 0.026
0.922 0.019
0.989 0.005
Ada-GVAE-k
0.452 1 0.156
0.821 1 0.028
0.879 0.029
0.920 0.021
0.992 0.007
GVAE
0,412 1
0.855 0.012
0,900 0.008
0.986 0.006
0.314 0.066
0.799 0,010
0.849 ::
0,895 0.014
0.987 0.004
MLVAE
0.477 + 0.069
0.698
0.844
0.893 0.016
0.984 0.010
MLVAE
0.305 + 0.071
0.648 0.072
0.854 0.013
0.896 0.012
0.989 0.008
Symbolic tocorrt compositional representations
VCT
0.558 1 0.028
0.694 0.034
0.808 0.013
0.837 - 0.023
0.973 0.024
VCT
0.197 + 0.151
0.640 0.113
0.640 0.113
0.725 0.038
1.000 0.000 (=)
Fully continuous compositional representations
Ours
0.705 1 0.023
0.889 1 0.042
0.926 1. 0.009
0.942
0.018
1,000 1 0.000 (-)
Table 35: Downstream regression model sample efficiency the Shapes33D dataset R² ratio
Models
a0ll samples
2.5 x 10 samples
5 Vall samples
1all samples
10 samples
Symbolic sokearl compositional representations
Slow-VAE
0.021 + 0.016
0.085 = 0.035
0,183 0.051
0.563 = 0.070
0.994 0.002
Slow-VARt
0.003 + 0.008
0,021 + 0.025
0.066 + 0.036
0.419 - 0.107
0.995 0.002
Ada-GVAIE GVAE-k
0.480 1: 0.155
0,510 1 0.160
0.717 0.206
0.855 0.103
0.997 0.001
Ada-GVAE-k
0.133 1 0.088
0.199 1 0.058
0.643 0.231
0.809 0.164
0.998
0.000 (-)
0.417 1 0.056
0.594 0.079
0.688 0.052
0.893 0.024
0.997 0.000
GVAE
0.13 0.047
0.234 0.038
0.479 0.076
0.871 = 0.029
0.998 + 0.000 (=)
MLVAE
0.371 = 0.041
0.515
0,590 0.080
0,855 0.059
0.997 0.000
MLVAE
0.123 + 0.048
0.235 0.069
0.413 0.035
0.808 0.141
0.999 0.000
Symbolic tokenr- compositional
representations
VCT
0.020 + 0.025
0.216 0.139
0.377 0.100
0.591 = 0.079
0.884 0.008
VCT
0.000 + 0.000
0.000 0.000
0.000 0.000
0.009 = 0.018
0.470 0.071
COMET
0.352 0.036
0.699 0.038
0.772 0.028
0.812 0.025
0.997 0.001
COмETT
0.752 0.039
0.810 0.034
0.870 0.029
0.916 0.031
0.996 0.001
Fully continuous compositional representations
Ours
0.464 1: 0.071
0,730 1 0.038
0.804 0.075
0.889 :: 0.035
0.996 0.001
57

---

Table 36: Downstream regression model sample efficiency the MPI3D dataset
R ratio
Models
10la samples
2.3 x 10l samples
5 x 10l/ samples
10 samples
10% samples
Symbolic scalar-tokened compositional representations
Slow-VAE
0.130 = 0.051
0.155 0.011
0.608 0.082
0,692 0.081
0,881 0.022
Slow- VAE!
0.107 1
0.095 0.011
0.541
0.117
Ada-GVAE-k
0.270 + 0.037
0.279 0.026
0.500 0.016
0.541 0.020
0.703 0.017
Ada-GVAF-kt
0.053 0.053
0.120 0.023
0.442 0.018
0.504 0.015
0.680 0.012
GVAE
0.234 + 0.035
0.282 0.027
0.481 0.032
0.524 0.035
0.686 0.033
GVAE!
0.077 0.073
0.137 0.037
0.415
0.443 0.034
0.648 0.025
MLVAE
0.236 1 0.019
0.288 0.030
0.462 0.051
0.497 1.017
0.663 0.012
MLVAEt
0.065 + 0.042
0.114 0.049
0.387 0.024
0.469 0.034
0.659 0.030
Shu
0.343 + 0.024
0.482 0.075
0.549 0.091
0.601 0.047
0.750 0.058
Shu
0.143 0.103
0.427 0.035
0.493 0.073
0.547 0.070
0.714 0.067
Symbolic vector-tokened compositional representations
VCT
0.189 + 0.107
0.246 0.137
0.294 + 0.145
0.312 + 0.14
0.418 + 0.180
VCTt
0.039 + 0.088
0.168 0.082
0.230 = 0.110
0.279 0.127
0.502 0.052
COMET
0.000 # 0.000
0.000 = 0.000
0.000 = 0.000
0.000 = 0.000
0.823 0.139
COMET
0,000 1 0.000
0.000 1 0.000
0.000 1 0.000
0,000 1 0.000
0.187 0.374
Fully continuous compositional representations
Ours
0,556 0.078
0.665 0.067
0.721 0.089
0.787
0.078
0.858 0.040
C.5.2 Low Sample Regime Results
To evaluate the utility of our Soft TPR representation from the perspective of downstream models. we additionally evaluate the raw performance of downstream models as a function of the number of samples they have been trained on (again considering 100, 250, 500, 1, 1,000, 10,000 and all samples). We find that our Soft TPR representation contributes to substantial performance boost in the downstream model's performance in a low-sample regime where the downstream model has been trained with 100, 250, 500, and 1,000 samples. We present our full suite of experimental results, and highlight the particular performance differentials conferred by our representational form low-sample regime.
FoV Regression
58

---

Downstream performance (100)
Downstream performance (250)
Downstream performance (500)
0.7
0.5
0.6
0.6
R59
0.3
0.2
0.1
0.0
0
1
2
3
4
5
1
2
Figure 35: Downstream regression model R scores the CarsãD dataset (original setting).
Downstream performance (1,000) Downstream performance (10,000)
Downstream performance (all)
1.0
1.0
0.8
0.6
0.6
0.6
RSq
RSq
RSq
0.4
0.4
0.4
0.2
0.0
3
Figure 36: Downstream regression model R scores on the Cars3D dataset (original setting).
59

---

Downstream performance (100)
Downstream performance (250)
Downstream performance (500)
0.6
0.8
0.5
0.6
0.4
0.3
0.2
0.2
0.1
0.0
0
1
2
3
4
5
1 2
1
Figure 37: Downstream regression model R² scores on the Cars3D dataset (dimensionnlltyycoottollee setting).
Downstream performance (1,000) Downstream performance (10,000)
Downstream performance (all)
1.0
0.6
RSq
RSq
0.4
0.4
0.2
0.0
0.0
1
3
1
Figure 38: Downstrcam regression model R² scores on the Cars3D datasct (dimensionalltyycoottolldd sctting).
60

---

Downstream performance (100)
Downstream performance (250)
Downstream performance (500)
0.8
1.0
0.8
0.8
0.6
0.5
0.6
0.4
0.4
85q
0.4
0.2
0.2
0.1
0.0
0.0
0.0
0
1
2
3
4
7
0
1
2
3
4
0
6
7
Figure 39: Downstream regression model R scores on the Shapes3D dataset (original setting).
Downstream performance (1,000) Downstream performance (10,000)
Downstream performance (all)
1.0
1.0
0.9
0.6
0.7
RSq
RSq
RSq
0.6
0.4
0.4
0.3
1
2
3
5
6
7
1
1
2
7
Figure 40: Downstream regression model R² scores on the Shapes3D dataset (original setting).
61

---

Downstream performance (100)
Downstream performance (250)
Downstream performance (500)
0.8
1.0
0.8
0.8
0.6
0.5
0.4
85q
859
0.4
0.2
0.1
0.0
0.0
0
1
2
3
1
4
5
6
0
1
2
4
5
6
7
Figure 41: Downstream regression model R2 scores the Shapes3D dataset (dimensiionallty-coorrllll setting).
Downstream performance (1,000) Downstream performance (10,000)
Downstream performance (all)
1.0
1.0
0.9
0.6
0.7
RS4
RSq
0.6
0.4
0.2
0.4
0.3
0.0
1
2
3
7
1
5
1
2
3
5
7
Figure 42: Downstream regression model R² scores on the Shapes3D datasct (dimenssionallyyyconoollcc setting).
62

---

Downstream performance (100)
Downstream performance (250)
Downstream performance (500)
0.6
0.7
0.6
0.5
0.5
0.4
0.5
0.4
0.4
R59
0.3
0.3
0.2
0.2
0.1
0.0
0.0
0
1
2
3
4
5
6
0
1
2
3
4
7
0
1
2
Figure 43: Downstream regression model R² scores the MPI3D dataset (original setting).
Downstream performance (1,000) Downstream performance (10,000)
Downstream performance (all)
0.8
0.8
0.9
0.6
0.5
0.6
0.7
R59
RSq
0.3
0.5
0.2
0.2
0.3
0.0
0.0
1
2
3
5
1
1
Figure 44: Downstream regression model R scores on the MP[3D dataset (original setting).
63

---

Downstream performance (100)
Downstream performance (250)
Downstream performance (500)
0.6
0.7
0.6
0.5
0.5
0.4
0.5
0.4
0.4
R59
0.3
0.3
0.2
0.2
0.1
0.0
0.0
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
0
1
2
Figure 45: Downstream regression model R scores on the MPB3 dataset (dimensiinnaliyy-ootttllll setting).
Downstream performance (1,000) Downstream performance (10,000)
Downstream performance (all)
0.8
1.0
0.7
0.8
0.5
0.6
0.7
0.4
RSq
0.6
0.4
0.3
0.5
0.2
0.1
0.3
0.0
0.2
1
2
3
1
2
3
5
1
2
3
Figure 46: Downstream regression model R² scores on the MPI3D dataset (dimensionalltyyccotoool setting).
64

---

Table 37: Downstream regression model performance the Cars3D dataset
R score
Models
10 samples
2.5 x 10 samples
5 10" samples
10 samples
10" samples
all samples
Symbolic scalar-tokened compositional representations
Slow-VAE
0.255 0.05
0.502 0.009
0.779 0.023
0.825 0.016
0.929 0.003
0.935 0.004
VAE!
0.274
Ada-GVAFE
0.584 + 0.125
0,798 + 0,040
0.843 + 0.028
0.883 0.020
0.947
0.009
0.957 + 0.005
Ada-GVAE-k
0.43 0.144
0.784 + 0.033
0.836 :: 0.028
0.873 0.018
0.944 + 0.010
0.957 - 0.005
GVAE
0,390 0.087
0.750 0.011
0.810 0.009
0.852 0.006
0,934 0.006
0.947 0.005
GVAE!
0.296 + 0.066
0.754 + 0.009
0.801 + 0.016
0.845 = 0.015
0.911 0.007
0,947 + 0.005
MLVAE
0.452 0.063
0,662 + 0.029
0.800 0.018
0.847 0.010
0.933
0.948 0.007
MLVAE
0,289
0.614 + 0.067
0,938 0.006
0.007
Shu
0.044 1 0.045
0.055 1 0.044
0.061 1: 0.047
0.067 0.050
0.085 0.056
0.075 0.051
Shu
0.010 0.042
0.065 0.022
0.079 0.031
0.079 0.030
0.101 0.046
0.075 0.051
Symbolic vector-tokened compositional representations
VCT
0.454 0.021
0.565 0.033
0.657 0.009
0.681 0,019
0.792 0.018
0.813 0.014
VCT1
0.117 + 0.087
0.340 0.073
0,390 + 0.059
0.444 = 0.019
0.643 - 0.034
0813 0.014
COMET
-0.058 + 0.027
-0.098
: 0.054
-0.074 :: 0.036
-0.147 : 0.101
-0.036 + 0.028
-0.035 1 0.023
COMET
-0.379 0.303
0,258 0.085
-0.966 0.470
0.103
-0.39 0.751
0.023
Fally contimueus compositional representations
Ours
0.642 + 0.023
0.8019 + 0.037
0.843 + 0.010
0,858 0.019
0.917 + 0.013
0.910
Table 38: Downstream regression model performance the Shapes33 dataset
Ht score
Models
10a samples
2.5 x 10" samples 5 x 10 samples
10a samples
10 samples
all samples
Symbolic scalar-tokened compositional representations
Slow-VAE
0.019 + 0.017
0.085 : 0.035
0.183 : 0.051
0.562 1 0.070
0.993 0.002
1,000 0.000 (=)
SloR-VAE!
-0.030 + 0.034
0.018 = 0.028
0.066 = 0.036
0.418 0 0.10 107
0.994 = 0.003
0.999 = 0.000
Ada-GVAB-K
0,480 0.155
0.160
0.666 0.170
0.103
Adn-GVAE-R
0.133 = 0.088
0.199 = 0.058
0.563 = 0.209
0.809 + 0.164
0,997 + 0.007 (=)
1,000 = 0.000
GVAE
0,417 = 0.056
0.593 = 0.079
0,687 = 0.052
0.893 - 0.024
0.997 0.000 (=)
1,000 0.000
GVAE!
0.130 = 0.047
0.234 0.038
0.478 0.076
0.029
0,998 0.000
1,000 0.000
MIXAF
0.370 + 0.041
0.515 0.093
0.590 0.080
0,854 + 0.059
0.996 0.000
L,000 0.000
MLVAE
0.234 0.069
0,997
(=)
1,000
Shu
0,062 1: 0.086
0.117 1 0.075
0.146 1 0.073
0,157 1 0.078
0.245 0.096
0.427 0.082
Shu
-0.029 0.076
0.093 = 0.08
0.118 = 0.077
0.131 + 0.079
0.228 0.085
0.419 0.082
Symbolic voctor-tokened compositional repeoscntations
VCT
-0.201 + 0.289
0.202 = 0.136
0.359 = 0.093
0.564 + 0.075
0.844 = 0.012
0.954 0.008
VCT
.4.183 + 1.679
-1.464 0.492
.0.552 + 0.307
-0.205 + 0.134
0.396 0.063
0.843 0.040
COMET
0.352 1 0.036
0.699 0.038
0.772 1 0.029
0.812 0.025
0.996 0.001
1,000 0.000
COMET
0.751 = 0.039
0,870 = 0.029
0 = 0.001
0.915 : 0.031
0.996 = 0.001
1.000 : 0.000 (=)
Fully continuous compositional represcotations
Ours
0.464 + 0.071
0.730 0.038
0.804 = 0.075
0.889 0.035
0.995 0.002
1,000 0.000
65

---

Table 39: Downstream regression model performance the MPI3D dataset
Rf
Models
10a samples
2.5 10 samples
10 samples
10 sumples
10a samples
all samples
Symbolic scalar-tokened composttoonl representations
Slow-VAL
0,127 0.059
0.152 + 0.011
0.59 0,081
0.678
0,082
0.863 0.023
0.980 0,005
Slow- VAE!
0,105 + 0.026
0.093 + 0.011
0.531 0.077
0.655
0.113
0.860 + 0.016
0.981
0.004
Ada-GVAE-k
0,206 :: 0.031
0.213
0.381 = 0.014
0.412 0.015
0.536 0.013
0.762
0.014
Ada-GVAE-K
0,037 0.045
0.093
0.342 0.014
0.390 0.012
0.527 0.010
0.774 0.003
GVAE
0.181 0.030
0.02
0.371 0.026
0.404
0.530
0.771 0.018
GVAET
0,056 + 0.057
0.120
0.316 0.027
0.338 0.019
0.495 0.020
0.764 0.019
MLVAE
0,182 0.013
0.222 + 0.024
0.357 = 0.042
0.384
0
0.773 0.031
MLVAE
0.051 +: 0.032
0.089 + 0.037
0.305 0.118
0.370 - 0.023
0.520 + 0.033
0. 0,790 0.027
Shu
Shu
0,057 0.048
0.186 + 0.012
0.214 0.017
0,237 0.013
0.311 0.020
0.438 0.037
Symbolic
tokened compositional representations
VCT
0.047 = 0.139
0.110 + 0.087
0.124 = 0.104
0.105 0.175
0.164 + 0.183
0.455 0.071
VCT
-0.005 + 0.058
0,084 + 0.062
0.106 0.092
0.153 0.072
0.276 + 0.036
0.550 0.046
COMFT
-0,151 + 0.015
0.042 0.018
40.037 0.012
-0.053 + 0.024
0.218 + 0.036
0.266 0.004
COMET
-0.118 = 0.045
-0.089 0.050
-0.074 0.039
-0.179 0.098
0.027 0.099
0.236 0.008
Fully continuous compositional representations
Ours
0.068
0.594 +:
0.635
0.693 0.060
+
0.882 = 0.016
b) Abstract Visual Reasoning
66

---

Downstream performance (100)
Downstream performance (250)
Downstream performance (500)
0.35
0.35
0.40
0.30
0.30
Accuracy
0.25
Accuracy
Accuracy
0.25
0.25
0.20
0.20
0.15
0.15
2
0 1
1
Figure 47: Downstream WReN model classification accuracy on the abstract visual reasoning dataset (original setting).
Downstream performance (1,000)
Downstream performance (10,000) Downstream performance (100,000)
0.50
0.6
0.9
0.40
0.5
Accuracy
Accuracy
0.7
0.6
0.30
0.3
0.4
0.20
0.2
Figure Downstream WReN model classification accuracy on the abstract visual reasoning dataset (original setting).
67

---

Downstream performance (100)
Downstream performance (250)
Downstream performance (500)
0.35
0.30
0.30
0.25
Accuracy
Accurago
0.25
Accuracy
0.20
0.10
0.15
0.15
0.10
0.10
1
2
3
3
1
2
Figure 49: Downstream WReN model classification accuracy on the abstract visual reasoning dataset (dinensionalliy--ootrrlll setting).
Downstream performance (1,000)
Downstream performance (10,000) Downstream performance (100,000)
0.50
0.7
0.9
0.45
0.6
0.8
Accuracy
Accuracy
0.5
Accuracy
0.6
0.30
0.4
0.25
0.4
0.20
0.3
0.2
0.15
1
Figure 50: Downstream WReN model classification accuracy on the abstract visual reasoning dataset (dinensionallty-cottrollll setting).
68

---

Table 40: Downstream WReN model performance the abstract visual reasoning dataset
Classification accuricy
Models
10a samples
2.5 102 sumples
5 102 samples
10a samples
10 samples
10 samples
Symbolic scalar-tokened compositional representations
Slow-VAE
0.178 0.008
0.196 + 0.024
0.196 0,028
0.228 0.030
0.361 + 0.028
0.396 +:200 0.022
Slow-VAR?
0.146 0.084
0.149 +: 0.034
0.163 + 0.039
0.237 +: 0.023
0.345 +: 0.021
0,959 +: 0,009
Ada-GVAB4
0.188 +: 0.023
0.198 + 0.018
0,203 = 0.007
0.194 + 0.008
0.295 : 0.028
0.472 + 0.037
Ada-GVAE-k
0.182 + 0.007
0.234 + 0.024
0.285 = 0.025
0.319 + 0.004
0.662 : 0.023
0.954 + 0.009
GVAB
0.171 0.009
+: 0.008
0.182
0.478 + 0.032
GVAE!
0.180 0.022
0.237 + 0.018
0.287 0.014
0.306 0.020
0.684 0.020
0,936 +: 0.008
MLVAE
0.171
0.188 + 0.009
0.193 0.012
0.194
0.293 0.012
0,432 1 0.023
MLVAE
0
0.221 + 0.023
0.277 - 0.023
0.325 4
0.644 + 0.026
0,925 4 0.024
Shu
0.175 0.009
0,184 + 0.018
0,200 - 0.010
0.202 + 0.014
0.288 + 0.038
0,310 + 0.038
Shut
0.177 0.014
0.230 + 0.045
0,285 0.028
0.302 0.034
0.444 + 0.013
0.444 + 0.013
Symbolic vector-tokenn compositional representations
VCT
0.228 0.036
0.229 + 0.049
0.277 0.039
0.326 0,042
0.371 + 0.040
0.641 + 0.039
VCTI
0,191 0.017
0.226 + 0.031
0.259 0.009
0.282
0.319 + 0.018
0.458 : 0.040
COMET
0.174
0,241 0.010
0.259 +: 0.016
0.283 0.009
0.221 + 0.012
0,983 + 0.006
COMETT
0.190 0.012
0.214 + 0.004
0.236 0.013
0.213 0.023
0.532 + 0.348
0.532 + 0.348
Fully continuous compositional representations
Ours
0.273 0.033
0.312 + 0.027
0.360 0.033
0.412 0.066 0.560 +: 0.103
0.869 + 0.024
C.6 Ablation Experiments
We perform the following ablation studies:
Foundational Model Components: We evaluate the impact of Relaxed Representational Constraints (RRC) and Distributed Encoding (DE) on disentanglement performance, as presented in Table 43. Previous work [30, 33] has established a positive correlation between disentanglement and both downstream sample efficiency and overall performance. Therefore, we do not separately assess the effects of these foundational components on downstream tasks.
Auxiliary Model Components: We investigate the influence of additional model compo- nents, as detailed in Appendix C.6.2.
Effect of Soft TPR on Downstream Tasks: Separately, we examine whether Soft TPR offers advantages over its quantised, explicit counterpart, Epr in terms of downstream model performance.
C.6.1 Distributed Encoding Ablation
To perform ablation study distributed encoding, we set the role embedding matrix MęR to the identity matrix. This modification results in maximally sparse role-filler bindings of the form (x(m(() @ e,, specifically:
{F(Sm(o) & (n(ri) = [0 O
EF (F)(())
0
0]
In this setup, the embedded filler (F(mm()) occupies the i-th column of the resulting matrix. Conse- quently, the TPRs generated by the TPR decoder become degenerate, effectively forming concate- nation of filler embeddings:
Vipr (x) =
This configuration confines each filler's information to specific dimensions within the representation, thereby limiting the model's ability to leverage distributed information across multiple dimensions.
Our experiments reveal moderate reductions in disentanglement performance and increases in loss compared to the distributed encoding scenario. Considering the rotational isomorphism between an identity matrix Mşp and (semi)-orthogonal M it is somewhat surprising to observe these differences under identical conditions (c.g., initialisation and random seeds). While future work
69

---

should aim to learn linearly independent (but not necessarily orthogonal) Men which would more convincingly account for such results by embedding each filler in overlapping role subspaces and eliminating the rotational isomorphism with the degenerate TPR case, this behaviour warrants further analysis and investigation. One potential avenue of investigation examining whether the sparsity of binding embeddings & (f(m()) & (r(r,) affects 'neuron' specialisation (i.e., the tendency of each row the weight matrix to attend only to certain filler-specific information).
Table 41: Effect DE on disentanglement (MPI3D dataset).
Property
Factor
score
score
VAta score
0.901 0.102
0.704 + 0.135
0.530 0.087
0.991 0.021
+ DE (Full)
0.949 = 0.032
0.828 + 0.015
0.620 + 0.067
1.000 : 0.000
C.6.2 Additional Ablations
We additionally examine the importance of the following properties of our model in producing explicitly compositional Soft TPR representations: 1) the presence of weak supervision, by setting A1 22 0 in Equation 7, 2) the explicit dependency between the quantised filler embeddings and the decoder output, by instead using the Soft TPR to reconstruct the input image, and 3) the semi-orthogonality of the role embedding matrix, McR by removing this constraint in the random initialisation of Mşn. with the results of these ablations illustrated in Table 42.
Table Effect of model properties on disentanglement performance (MPI3D dataset).
Property
DCI Score
Weak supervision
0.225 + 0.034
Explicit filler dependency
0.718 + 0.051
Semi orthogonality
0.756 1 0.039
Full
0.828 + 0.015
C.6.3 Soft TPR TPR
To examine the effect of the Soft TPR downstream tasks, for each fully trained Soft TPR representation learner, we extract both the Soft TPR, =, and the Soft TPR's corresponding quantised all plots,
TPR, dip and investigate downstream performance using both types of representation. we use the same legend, denoting the explicit TPR as yellow (0) and the Soft TPR as blue (I). Across all considered cases: i.e., 1) convergence rate of representation learning (as measured by the downstream model's 's ability to effectively leverage representations produced at different stages of training), 2) sample efficiency of downstream models, and 3) raw performance of downstream models in the low sample regime, the Soft TPR confers differential performance boosts compared to the explicit TPR.
Convergence Rate of Representation Learning
70

---

Cars3D
0.8
0.4
0.2
0.0
10a
10a
10
10a
Iteration count
Figure 51: Convergence Soft TPR (0) vs TPR (1) as measured by FoV regression on the Cars3D datasct
Shapes3D
1.00
0.95
0.90
0.85
0.80
0.75
0.65
10a
10a
10a
Iteration count
Figure 52: Convergence of Soft TPR (0) vs TPR (1) measured by FoV regression the S dataset
71

---

MPI3D
0.9
0.8
0.7
RSa
0.6
0.5
0.3
102
10a
10
10a
Iteration count
Figure 53: Convergence of Soft TPR (0) vs TPR (1) as measured by FoV regression on the MPI3D dataset
Abstract visual reasoning dataset
0.90
0.80
0.75
0.70
10a
10a
10a
10a
epochs
Figure 54: Convergence of Soft TPR (0) TPR (1) as measured by classification performance on the abstract visual reasoning dataset
Sample Efficiency of Downstream Models
72

---

Sample efficiency (100/all)
Sample efficiency (250/all)
0.85
Rsg ratio
0.80
0.75
0.5
0.70
0.4
Figure 5S: Downstream regression model sample efficiency the Cars3D dataset using TPR representations (0)
vs Soft TPR representations (1)
Sample efficiency (500/all)
Sample efficiency (1,000/all)
Sample efficiency (10,000/all)
1.00
1.04
1.00
0.96
0.94
0.96
Rsq ratio
1.00
RSq
Rsq
0.94
0.90
0.90
0.86
0.96
0.88
Figure 56: Downstream regression model sample efficiency on the Cars3D dataset using TPR representations (0) Soft TPR representations (1)
73

---

Sample efficiency (100/all)
Sample efficiency (250/all)
0.60
0.8
0.7
RSq ratio
0.45
ratio
0.40
0.6
0.35
0.5
0.30
0.25
0.20
Figure 57: Downstream regression model sample efficiency the Shapes3D dataset using TPR representations
(0) vs Soft TPR representations (1)
Sample efficiency (500/all)
Sample efficiency (1,000/all)
Sample efficiency (10,000/all)
0.90
1.04
0.80
RSq ratio
Req rañio
0.90
Rsq rablo
1.00
0.85
0.65
0.80
0.96
Figure 58: Downstream regression model sample efficiency on the Shapes3D dataset using TPR representations (0) Soft TPR representations (1)
74

---

Sample efficiency (100/all)
Sample efficiency (250/all)
0.70
0.85
0.65
0.80
Rsg ratio
R5q
0.55
0.70
0.50
0.60
0.45
Figure 59: Downstream regression model sample efficiency the MPI3D dataset using TPR representations (0) vs Soft TPR representations (1)
Sample efficiency (500/all)
Sample efficiency (1,000/all)
Sample efficiency (10,000/all)
0.95
0.90
0.95
0.85
RSq ratio
0.75
Rsq ratio
Rsq
0.80
0.70
0.75
0.65
0.70
0.80
0.63
Figure 60: Downstream regression model sample efficiency on the MP13D dataset using TPR representations (0) Soft TPR representations (1)
75

---

Sample efficiency (100/100,000)
Sample efficiency (250/100,000)
0.45
0.400
0.375
0,350
Accuracy ralio
0.325
Accuracy ratlo
0.35
0.300
0.30
0.200
Figure 61: Downstream WReN model sample efficiency the abstract visual reasoning dataset using TPR representations (0) vs Soft TPR representations (1)
Sample efficiency (500/100,00)
Sample efficiency (1.000/100,000)
Sample efficiency 00,0000/1000000
0.60
0.45
0.65
Accurscy ratio
0.40
Accuracy rAtio
0.30
Accuracy ratlo
0.60
0.45
0.55
0.40
0.50
0.35
0.25
Figure 62: Downstream WReN model sample efficiency on the abstract visual reasoning dataset using TPR representations (0) vs Soft TPR representations (1)
Low Sample-Regine Performance of Downstream Models
76

---

Downstream performance (100)
Downstream performance (250)
0.70
0.85
0.65
0.80
0.60
0.75
0.55
RSq
RSa
0.70
0.50
0.45
0.60
0.40
0.55
0.50
0.30
Figure 63: Downstream regression model R² scores on the Cars3D dataset using TPR representations (0) vs
Soft TPR representations (1)
Downstream performance (1,000) Downstream performance (10,000)
Downstream performance (all)
0.92
0.90
0.94
0.94
0.88
0.92
0.92
R5g
0.86
R5q
R5g
0.90
0.84
0.88
0.80
0.86
0.78
0.84
0.84
Figure 64: Downstream regression model R² scores on the Cars3D dataset using TPR representations (0) vs Soft TPR representations (1)
77

---

Downstream performance (100)
Downstream performance (250)
0.60
0.8
0.55
0.50
0.7
0.45
RSq
RSq
0.35
0.5
0.25
Figure 65: Downstrcam regression model R scores on the Shapes3D dataset using TPR representations (0) vs
Soft TPR representations (1)
Downstream performance J.000rDDwwstteam performance (10,000)
Downstream performance (all)
1.04
0.95
1.02
1.02
ASg
RSq
R5g
1.00
0.85
0.98
0.80
0.75
0.94
Figure 66: Downstream regression model R² scores on the Shapes3D dataset using TPR representations (0) vs Soft TPR representations (1)
78

---

Downstream performance (100)
Downstream performance (250)
0.70
0.60
0.65
0.55
0.60
0.50
RSq
R5q
0.55
0.45
0.50
0.45
Figure 67: Downstream regression model R scores on the MPI3D dataset using TPR representations (0) vs Soft TPR representations (1)
Downstream performance Downsttram performance (10,000)
Downstream performance (all)
0,80
0.90
0.80
0.70
A5q
ASq
0.75
R5q
0.65
0.75
0.55
Figure 68: Downstream regression model R²² scores on the MPI3D dataset using TPR representations (0) vs Soft TPR representations (1)
79

---

Downstream performance (100)
Downstream performance (250)
0.350
0.375
0.350
0.300
0.325
Accuracy
0.275
Accuracy
0.300
0.250
0.225
0.250
0.200
0.225
0.175
0.200
Figure 69: Downstream WReN model classification accuracy on the abstract visual reasoning dataset using TPR representations (0) vs Soft TPR representations (1)
Downstream performance (1,000)
Downstream performance (10,000)
Downstream performance (100,000)
0.50
0.65
0. 925
0.60
0.45
0.55
0.900
Accurac
0.40
Accuracy
Accuracy
0.875
0.45
0.40
0.25
800
0.30
0.775
Figure 70: Downstream WReN model classification accuracy the abstract visual reasoning dataset using TPR representations (0) vs TPR representations (1)
C.6.4 Robustness to Hyperparameter Choices
We perform an additional experiment to empiricially verify that our model is robust to different hyperparameter choices. For the MPI3D dataset, which disentanglement models experience the greatest difficulty in producing disentangled representations for (see Table 1), we randomly pick another set of hyperparameters, shown in Table 43, from the top models based the MSE loss criterion. For this randomly chosen hyperparameter configuration, we evaluate the disentanglement of the representations the MPI3D dataset produced by the resulting model.
80

---

Table 43: Hyperparameter values of ablation setting
Hyperparameter
MPI3D
Architectural hyperparameterss
Du Na (fixed) Dy Nr
16
64
50
Loss function hyperparamelerrs
A Ag B (fxed)
0.25378
0.5
Table 44: Disentanglement metric the MPT3D dataset
Hyperparameter configuration
Factor score
DCI score
BetaVAE score
MIG score
Original Ablation
0.949 1 0.032 0.911 1 0.029
0.828 0.015
1 1.000 1L 0.000
0.620 0.067
0.798 0.031
1,000 1 0.000
0.590 1 0.047
D Limitations and Future Work
D.1 Extension to Linguistic Domains
Applying the Soft TPR to the TPR's typical domain of language is an intriguing future direction, especially language can deviate from strict algebraic compositionality for instance, idiomatic expressions such as 'spill the beans' cannot be understood as a function of their constituents alone. Soft TPR's more flexible specification allows capture approximate forms of compositionaliyy precluded from the TPR's 's strict algebraic definition (Eq 1), thereby potentially providing the Soft TPR the ability to better handle the nuance and complexity of language.
To adapt our framework to language, we replace our Conv/Deconv encoder/decoders with simple RNNs, retrain our TPR decoder, and remove the semi-supervised loss, using Eq the full loss. Preliminary results in Table 45 on the BaBI [7] dataset are compared with TPR baselines from AID [51]. Our Soft TPR Autoencoder does presently surpass AID, but notable points include:
1. Our use of simpler RRN-based encoders and an MLP-based downstream network, unlike the more sophisticated architectures [51].
2. Soft TPR retains performance improvement above the corresponding explicit TPR can be quantised into.
3. The smaller gap between systematic vs non-systematic dataset splits in our model compared TPR-RNN (+AID) and FWM.
4. We train representation learner using self-supervision (reconstruction loss) alone, only employing supervision on the downstream prediction network, while the baselines employ strong supervision and end-to-end training produce representations.
Table 45; Mean word error rate [%] on the sys- sys-bAbl task [51]
Model
w/o sys diff'
w/ sys difi'
Gap
TPR-RNN
0.79 0.16
8.74
3.74
7.95
TPR-RNN AID
0.69 0.08
5.61
1.78
4.92
FWM
0.79 : 0.14
2.85 + 1.61
2.06
FWM AID
0.45 0.16
1.21 1 0.66
0.76
Ours (TPR)
4.54 0.61
6.51
1.23
1.97
Ours (Soft TPR)
2.79 + 0.24
3.64 + 0.69
0.85
81

---

D.2 Need for Weak Supervision
To produce a compositional representation, xxx)) (11(
(a,)) (as in Section 3.1), each representational constituent, (a,)) must map 1-1 to data constituent, aN- Without supervision (i.e., access only to observational data, {z,)) )) this is challenging, because the data constituents {a:} underlying each object, are unknown and cannot be identified.
We can frame the above intuition in a more mathematically rigorous way, using the generative framework. Essentially, as formally proved in [25], it is impossible to identify the true distribution for the data constituents (generative factors), p(a), using observational data, {r,). alone there
are infinitely many bijective funcctions f supp(a) + a such that: 1) and f(a) are completely entangled (i.e. non-diagonal Jacobian) and 2) the marginal distributions of and f(a are identical (meaning the marginal distributions of the observations are also identical, i.e., p(=]aa)p(aaee = (f(==]((a)))) (a) )da). Thus, without inductive biases, it is impossible to infer the data constituents {az) of any observation, I. from observational data (x:) alone.
To combat this non-identifiabilityy result, in line with [13, 22, 33, 35], we use weak supervision, presenting the model with data pairs (x,x') x²) where and x' differ in subset of Fo e.g. {(z) (shape/cube, colour/ / purple, size/ blzgge/r 3(x') = (shape/cube, colour beoour/cyon /cyan, aize//lrre]; where the FoV types corresponding to the different FoV values are known the model. Note that this weak supervision is minimal, only providing the model to access to the differing FoV rypes in an index set. I, (i.e., I == fcolour)) }) and not any of the FoV values (i.e., {cube, purple, cyan, large) are all not known by the model).
Some possible future extensions to reduce this level of weak supervision, or alternative forms of weak supervision include:
1. Embodied Learning: the visual domain, some roles, e.g. object position correspond to affordances. Embodied agents may be able to reduce the need for explicit supervision by collecting (x, x²) and / through interaction with their environment.
2. Pretrained Filler Embeddings: Initialising the filler embedding matrix, Mep with em- beddings learnt by pre-trained vision model could impart knowledge of domain-agnostic fillers (e.g., colours, shapes), reducing the need to explicitly provide (x,x²) and I to the model.
3. Segmentation Masks: Segmentation masks for each object may potentially reduce the need explicitly provide (2r, x²) and to the model.
D.3 Downstream Utility
Our investigation of downstream utility centers on two selected tasks - FoV regressionnlassificatiinn and abstract visual reasoning, which aligns with the standard framework for assessing the quality and downstream utility of compositional representations [26, 30, 33, 40, 44, 46). While existing work [30, 33] demonstrates that explicitly compositional representations enhance downstream sample efficiency compared to con-compositional representations, a result we improve upon (C.5.1). the broader utility of compositional representations remains a topic of ongoing exploration [21, 40, 42, 44].
Theoretical perspectives [1, 2] argue that explicitly compositional representations are fundamental in the production of productive, systematic, and inferentially coherent thought - 3 key properties characterising human cognition. Investigating how explicitly compositional representations can yield empirical benefits across these dimensions represents essential avenue for future research. Although preliminary studies [40, 42, 44] do not find strong evidence that explicitly compositional representations improve compositional generalisation (a key aspect of systematicity), [44] suggests that this finding is because compositional representations are necessary, but not sufficient to induce systematicity; explicitly compositional processing approach [6] is also required.
Future work could extend our theoretical framework with the hope of producing empirical results consistent with the theoretical arguments of [1, 2]. In particular, as our unbinding module is designed to provably and efficiently recover structured role-filler constituents from the Soft TPR, it may be possible to exploit this module to systematically reconfigure roles and fillers from existing representations to create representations of novel combinations of role-filler bindings (i.e., novel
82

---

compositional data). This type of ability could prove extremely beneficial in areas such as concept learning and compositional generalisation.
D.4 Dimensionality
In this subsection, we denote the dimensions of the role and filler embedding spaces as Dr and Dr respectively. We also denote the number of fillers NE and the number of roles Nn-
The Soft TPR belongs to VF @ VR. which is a Dr x DR dimensional space, which grows mul- tiplicatively in Dr, Dn- Several factors, however, mitigate scalability concerns in light of this fact:
1. Independence of Embedding Space Dimensionality: We note that the dimensionality of the role and filler embedding spaces (Dn, Dr respectively) are properties of the corre- sponding embedding functions (En= R -> Vn and {r F - Vr) and thus, can be fixed independently of the number of roles, Nn. the number of fillers, Nr, or the number of total role-filler bindings (which we denote by n) within domain. Thus, it is possible to fix the Soft TPR's ss dimensionality (Dp x DR) to be smaller than N₂ x NR (the number of roles/FoV types multipled by the number of fillers/FoV tokens) or (the number of bindings), which all may be large in complex visual domains. As illustrated in Table 46, the dimensionality of the TPR smaller than Np NR in both the Shapes3D and MPI3D domains.
2. Relaxing Orthogonality: While Dr can be set a priori with no regard to NR, NN or n, require DR 2 NR for semi-otthogonallty of the role role-embedding matrix Mgn- which guarantees faithful (see proof of A.2) and computationally efficient (see 'Unbinding' heading of B.4.1) recoverability of constituents. It is, however, possible to relax this constraint (i.e., to have Dr < Nn) to further reduce dimensionality. In this case, semi- orthogonality of MşR is impossible and hence the recoverability of constituents cannot be guaranteed, however, there are some less stringent guarantees on the outcome of unbinding that can still be derived (see p291 of [3] for more details).
We also more explicitly compare the dimensionality of the Soft TPR with baselines in Table 47. Scalar-tokened symbolic representations have low dimensionality of 10 (Nx) at the expense of representational expressivity (each representational constituent ();(a;) is scalar-valued) In contrast, Soft TPR has vector-valued representational constituents (i.e. {f(((() @ (r(r:)), similar to VCT and COMET. When compared to these models, the Soft TPR has significantly lower dimensionality compared to VCT and is comparable with COMET.
Table 47: Comparison of dimensionality representa- tions
Ca
Shapes3D
MP13D
Model
Representational dimension
Table 46: Comparison of multiplicative dimensionality
Symbolic scalar-tokened compositional representations
Dataset
Parameter
Cars3D
Shipes3D
MPI3D
Slow VAF
10
10
10
Ada-GVAE-k
10
10
10
DR
12
16
12
GVAE
10
10
10
NR.
10
10
ML-VAR
10
D3
128
32
32
Shu
Ny
106
57
50
Dp Dg (TPR dimension)
384
Symbolic vector-tnkened compositional representations
Ng.
NR
1060
570
500
VCT
5120
5120
5120
COMET
640
640
640
Fully continuous compositional representations
Ours (Soft TPR)
1536
512
384
D.5
Computational Cost
In the Soft TPR Autoencoder, the expensive tensor product operation is employed to generate d Given the computational cost of the tensor product, more concretely examine the computational
83

---

cost of training the Soft TPR Autoencoder by computing the FLOPs for single forward pass on batch size of 16 using the open-source implementation of fvcore https://githut com/ wttee//eeeeeeeeeeeeneeeeeeeeee
visible in Table 48. This data demonstrates that, despite the tensor product's computational cost, the mathematicalll-inforre allows it to obtain compositional representations with vector-valued representational constituents at significantly lower cost compared to relevant, vector-tokened baselines (2 orders of magnitude less than VCT, and 4 orders of magnitude less than COMET).
facebookreseare
derivation of our model
Future work could explore the use of tensor contraction techniques reduce computational expense. For instance [29] uses Hadamard product based tensor product compression technique. This reduces computational cost from n² (tensor product of vectors) to n (Hadamard product), but comprises the theoretical guarantees on constituent recoverability. We believe developing tensor contraction techniques within the TPR framework an important direction for future research, to ensure efficient TPR-based representations with provable recoverability of constituents.
Table 48: Comparison of FLOPs required for forward pass of batch size 16
Cars3D
Shapes3D
MPI3
Model
Representational dimension
Symbolic scalar- compositional representations
Slow VAE
1.47 x 10a
1.47 10
1.47 x 10a
Ada-GVAE-k
1.47 x 10
1.47 10
1.47 x
GVAE
1.47 x 10
1.47 10
1.47 x 10a
ML-VAE
1.47 x 10a
1.47 x 10
1.47 x 10a
Shu
1.45 x 108
1.45 x 10
1.45 x 10a
Symbolic vector-tokened compositional representations
VCT
2.53 x 10t1
2.53 x 10s1
2.53 x 10a1
COMET
5.12 10
5.12 x 10
5.12 x 10
Fully continuous compositional representations
Ours (Soft TPR) 3.21 x 109 2.93 10
2.89 x 10
84

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims accurately reflect the paper's contributions and scope. We provide theoretical proofs the Appendix, as well as our complete suite of experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We explicitly state in the last paragraph of the paper that future work is to be done in developing hierarchical Soft TRP representations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate 'Limitations' section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
85

---

Answer:
[Yes]
Justification:
We include complete proofs for all theoretical results in the Appendix.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all information (specification of model architecture, computing resources, hyperparameters) to replicate experimental results in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
86

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All datasets are open-access. We provide sufficient instructions in our Ap- pendix to reproduce experimental results. The code base is also public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
onPolicy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ttggguiee///oduennnnnn / //nips wwtppccccppppppetccccccccccccccc g/guidgs//////eeeeoo CodeSubmi ssi aasonnooiccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide these details in the Appendix, and additionally, in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We present results with standard deviations, as well as the IQR and whiskers extending to 1.5 times the IQR for all box plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialisation, random drawing of some parameter, or overall with given experimental conditions).
87

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the specific GPUs we use for all experiments in the Appendix, well the average amount of time it takes train the Soft TPR Autoencoder for each dataset.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///eurriieee iis.ripppo.. cc Ctwpblic/ttiisuuiieee. ines?
Answer: [Yes]
Justification:
Our research conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: As the central aim our work relates to the representational form of composi- tional representations, our research is theoretical in nature and has no immediate societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
88

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: In the main paper. as well as the Appendix, we cite the papers associated with each dataset we use. We additionally make explicit references the Appendix to all externally created code that we use in our experiments. Furthermore, if the paper is to be accepted, we will clearly provide licenses and attribution to the original authors where applicable in the code files.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI.
89

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our open access code is reasonably straightforward to understand, especially when supplemented by the Appendix of this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing nor human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
90

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
91


Reviews:
[{"review_id": "n4S8CgACIX", "reviewer": "Reviewer_6VtP", "summary": "This paper introduces a novel framework for representation learning known as Soft Tensor Product Representations (Soft TPR), aimed at capturing the compositional structure of data more effectively. The authors propose a continuously-valued compositional representation that contrasts with traditional symbolic methods. The paper makes several key contributions in the realm of representation learning, including the conceptualization of Soft TPR, the Soft TPR Autoencoder, and the demonstration of the benefits of this representation for both representation learning and downstream models.", "questions": "1. what is the dissentanglment performance of TPR?  The paper only report the downstream results of TPR in Table 6, is it possible to report the disentangment performance of TPR in Table 1?\n2. how many pairs are provided for the weakly supervised? For example, in Table 4, the numbers are the total samples for training, or the labeled sample pairs?\n3. VCT uses attention-based operations (cross-attention and self-attention) to extract concept vectors, and reconstruct images from the concept tokens, what is the difference between those attention-based operation and the unbinding module  or TPR constructor in this paper?", "limitations": "I don't see any negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of Soft TPR brings a fresh perspective to the field of representation learning, offering a new way to represent compositional structures in a continuous manner. It provides a thorough theoretical exploration of Soft TPR, complete with detailed mathematical proofs and framework extensions. The paper clearly articulates the differences between Soft TPR and existing methods, holding significant potential for enhancing model interpretability and improving robustness to covariate shift.", "weaknesses": "1. No MPI scores for disentanglement (in Table 1) are reported. \n2. The construction of the model's loss function involves a multitude of hyperparameters, suggesting that the model might require a complex and intricate tuning process. Compared to other models like VCT, the process of adjusting parameters to achieve optimal results could involve additional complexity and such a requirement might pose challenges in terms of computational resources and time.\n3. The reason why *Soft TPR is better than TPR is not fully explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and thoughtful review. \n\n**W1**: To our best knowledge, MPI score is not established in disentanglement learning literature [19,23-26,31,32,34,38]. \n\nConsistent with VCT [34], we evaluate disentanglement using 3 datasets (Cars3D, Shapes3D, MPI3D) and 4 metrics (FactorVAE, DCI, MIG, BetaVAE) commonly used in disentanglement studies [19,23-26,29,31-34]. Table 1 in our paper compares our method with SoTA approaches using FactorVAE and DCI scores. For MIG and BetaVAE results, please see Table 17 of C.3.3. \n\nTo avoid confusion, we will clarify in Table 1’s caption that additional results are in the supplementary. \n\n**W2**: Our loss function in Eq 7 has 2 tuneable hyperparameters, $\\lambda_{1}$ and $\\lambda_{2}$ in the weakly supervised loss (we set $\\beta$ in $\\mathcal{L}_{u}$ of Eq 6 to 0.5 (L1006)). \n    \nThe use of hyperparameters for loss functions is common in compositional representation learning. Canonical VAE disentanglement methods [4,8,9,11,18] and our baselines [31,26,29] each have 1-2 loss hyperparameters. \n\nVCT’s lack of tuneable loss hyperparameters is a strength but VCT is far more computationally expensive to train compared to our model.\n\nTo illustrate this point in this rebuttal, Table* 7 is added for comparing FLOPs. Table* 1 is added to show the total time for tuning our model on an RTX 4090 including 2 loss hyperparameters and 4 architectural hyperparameters (details in B.3). Table* 2 is added for the cost of tuning our model relative to the total time required for training a single VCT model on a V100 GPU for 200,000 iterations, excluding the 48-144 hours for training VCT’s VQ-VAE image tokeniser. \n\nFurthermore, ablation experiments (C.6.2) demonstrate our model’s robustness to both architectural and loss hyperparameters indicating multiple viable hyperparameter points that ease the tuning process.\n\n\n**W3**: Please refer to our detailed explanation of the differences between Soft TPR and TPR in Section 2 of our General Response. \n\n**Q1**: The disentanglement performance of the Soft TPR and TPR are identical. To establish why, we outline our method of extending existing disentanglement evaluation procedures to the continuous instantiation of compositional structure of the Soft TPR/TPR (please refer to C.3.2 for additional details). \n\nExisting disentanglement metrics assume a symbolic representation of compositional structure, requiring a vector $v$ with a dimension $N_{R}$ matching the number of roles (FoV types). Each dimension of $v$ is populated with a scalar-valued filler (FoV token). To produce $v$ from the Soft TPR we follow the procedure in C.3.2: \n\n1) Quantised fillers $\\{\\psi_{F}(f_{m(i)})\\}$ are extracted from $z$ via Unbinding and Quantisation modules.\n2) Each dimension of $v$ is populated with the index of the quantised filler for role $i$ (i.e. $v_{i} = m(i)$). \n\nAs the explicit TPR, $\\psi_{tpr}^{*}$ is built from the Soft TPR’s quantised fillers (Eq 5), the disentanglement scores for both are identical. \n\nWe use the quantised fillers of the Soft TPR to construct $v$, not its soft fillers $\\{\\tilde{f}_{m(i)}\\}$ for 2 reasons: \n\n1) There is no natural way of quantising soft fillers into scalars other than using PCA, which incurs significant information loss. \n\n2) Quantised fillers align with the intent of disentanglement metrics, which assess perfect (not approximate) compositionality. \n\nThe Soft TPR, $z$, does not enhance disentanglement compared to the explicit TPR, $\\psi_{tpr}^{*}$. However, it is important to note our motivation for continuously relaxing the TPR into the Soft TPR is to not to more explicitly represent compositional structure, but rather, to further align TPR-based structure with continuous vector spaces (details in Section 2 of General Response). This enhances downstream performance and representation learner convergence compared to the TPR (C.6.1). Our model is also able to always recover precise, algebraic compositional structure (with form $\\sum_{i} \\psi_{F}(f_{m(i)}) \\otimes \\psi_{R}(r_{i})$) from the Soft TPR, $z$, by quantising it using the TPR decoder. Our model can thus leverage the duality of the Soft TPR’s (approximately) compositional structure.\n\n**Q2**: The term refers to the number of individual data points, not pairs (i.e., 100 samples=50 pairs). We will add a footnote for clarification. \n\n**Q3**: The key differences are as follows: \n\n1) Motivation: VCT applies attention to generate concept/image tokens, whereas our framework uses the 3 modules of the TPR decoder to generate a highly specific element $\\psi_{tpr}^{\\*}$ as per Eq 5. VCT uses these concept tokens to produce a symbolic form of compositional structure, whereas we use $\\psi_{tpr}^{\\*}$ to minimise $||z - \\psi_{tpr}^{\\*}||^{2}_{2}$ (term 1 in Eq 6), which encourages the Encoder, $E$, to produce a Soft TPR, a continuous representation of (approximately) compositional structure.  \n\n2) Continuous vs Symbolic Representation: VCT represents compositional structure symbolically, through direct sums, with each constituent $c_{i}$ embedded in a separate subspace. Our method embeds all constituents in the same vector space and additively superimposes them. This continuous form of compositional structure produces empirical benefits (C.3.3, C.4.2, C.5). \n\n3) Constituent Structure: VCT’s unstructured constituents (concept tokens) do not guarantee systematic relations between constituents with the same filler/role. Soft TPR, however, with its structured (soft) role-filler bindings ensures systematic relations between constituents with the same filler/role, resulting in a disentanglement improvement. \n\n4) Mathematical Basis: Our modules are mathematically designed to recover soft fillers, quantised fillers, and explicit TPRs (Eqs 3-5). Attention, as employed by VCT, cannot theoretically guarantee the production of these structures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M7qlpZxUCQ", "reviewer": "Reviewer_93av", "summary": "This work explores compositional representations -- considered a crucial capability underlying intelligent human behaviour in deep learning systems. It argues that there is an incompatibility between discrete symbolic compositional representations—e.g. as obtained through traditional disentanglement approaches—and the continuous vector spaces underlying deep learning systems. To address this, the authors introduce a novel continuous compositional representation that builds on the Tensor Product Representation (TPR) approach, akin to a soft approximation to the TPR. They introduce a method for learning soft TPRs with weakly supervised data called the Soft TPR Autoencoder, and apply it to visual representation learning. This model demonstrates state of the art disentanglement for representation learning and improved sample efficiency for downstream models using those representations.", "questions": "A couple of questions, both on the motivations of the approach in the paper and how it could be improved. In both cases, these are more questions focused on the conceptual motivations and understanding than what the results show:\n - In terms of motivating the approach, can you be more specific about how representations obtained with traditional disentanglement approaches (typically continuously-valued themselves) could create an incompatibility with the continuous vector space of deep learning systems, and lead to suboptimal performance for the representation learning and downstream models? Any hypothetical or concrete examples (and/or references discussing this point)? \n- Relative to TPR, what is the relaxation in Soft TPR supposed to be helping with, and why? The manifold example is given, e.g. in Figure 1c, but seems quite vague. Also discussed is that the data being explained maybe only approximately compositional, but why would TPR fail with this and why would we specifically expect soft TPR to help? Could this be unpacked more, perhaps with some concrete examples.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- importance: compositional representation learning, and compositional generalisation and sample efficiency is an important and relatively under-explored area in deep learning research. By focusing on an approach for representation that aims to be more compatible with deep learning this work makes important contributions to this area.\n- novelty: the method presented is interesting & new but builds on the well established TPR framework.\n- clarity: the method is explained well and in enough detail to understand and reproduce it.\n- evaluation: the paper provides a thorough evaluation of the model, including comparisons with the relevant baselines, convergence rates, and performance in low sample regimes. The better performance on downstream tasks w.r.t. number of training examples is particularly interesting and supports some of the motivations for compositional representation.\n - interested to see where this work might develop in future work exploring hierarchical compositional representations.", "weaknesses": "- motivation for the approach & model could be unpacked a bit more (see questions)\n- other domains: the authors focus on applying the work to the visual data, outside the typical domain of TPR models. In general, a strong feature, enabling them to tackle the messier world of complex visual data (and weaker super vision) and compare the model to the many disentanglement approaches that have previously been applied in the visual domain. However, it could be interesting to see how the _Soft_ TPR approach compares to traditional TPR in its typical domain of language.\n - no high impact is shown for downstream utility. the improvements seen for downstream tasks are certainly interesting, particularly in the low data regime, however only two tasks are explored. are there any more good downstream tasks to evaluate the utility of their learned representations? could there even be more speculation as to where future work might really leverage what can be learned with this approach?\n - need for supervision (albeit weakly): could the work be extended to use different forms of supervision or less/no supervision. noting that their ablation showed the importance of the supervision, perhaps the authors could explore more variants on this ablation, and speculate on extensions of the model that could reduce the need for supervision?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s detailed and thoughtful feedback. \n\n**W1, Q1, Q2: Motivation for approach and model**\n\nThank you for the detailed questions on the motivation. We clarify these points in the General Response. \n    \n**W2: Limited Domains**\n\nApplying Soft TPR to language is an intriguing future direction, especially as language can deviate from strict algebraic compositionality. E.g., idiomatic expressions like ‘spill the beans’ cannot be understood as a function of their constituents alone. Traditional TPR, relying on strict algebraic compositionality (Eq 1) may be less effective in such cases. Soft TPR's more flexible, approximately compositional structure might better handle such complexities.\n\nTo adapt our framework to language, we replace our Conv/Deconv encoder/decoders with simple RNNs, retain our TPR decoder and remove the semi-supervised loss, using Eq 6 as the full loss. Preliminary results (Table* 4 in this rebuttal) on the BaBI dataset [A] are compared with TPR baselines from AID [36]. Our Soft TPR Autoencoder does not yet surpass AID, but notable points include: \n    \n  1) Our use of simpler RRN-based encoders and an MLP-based downstream network, unlike the more sophisticated architectures of [36]\n  \n  2) Soft TPR retains its performance improvement above the corresponding explicit TPR it can be quantised into\n  \n  3) The smaller gap between systematic vs non-systematic dataset splits in our model compared to TPR-RNN (+AID) and FWM \n  \n  4) We train our representation learner using self-supervision (reconstruction loss) alone, only employing supervision on the downstream prediction network, while the baselines employ strong supervision and end-to-end training to produce representations\n\n\n**W3: Downstream Utility**\n    \nOur focus on the 2 selected tasks (FoV regression/classification and abstract visual reasoning) aligns with the standard framework for assessing the quality and downstream utility of compositional representations [19,23,26,32,B,C]. While explicitly compositional representations enhance downstream sample efficiency over non-compositional representations [23,26], which we improve upon (C.5.1), their broader utility remains an open question [15,32,33,C]. \n\nTheoretical arguments [D,E] posit that explicitly compositional representations are crucial for productive, systematic, and inferentially coherent thought/processing – 3 key properties of human cognition. Using compositional representations to produce empirical benefits along these dimensions is a crucial direction of future research. While preliminary results [32,33,C] do not find strong evidence that compositional representations enhance compositional generalisation (a form of systematicity), [C] suggests that compositional representations are necessary, but not sufficient to promote systematicity; an explicitly compositional form of processing is also required. \n\nWe believe our theoretical formalism offers a unified framework for future research to achieve empirical results aligned with [D,E]. In particular, as our unbinding module provably recovers structured role-filler constituents from the Soft TPR (L189-197, L841-858), our model enables the systematic rearrangement of a representation’s roles/fillers to form novel composite representations. This potentially advances the development of compositional processing methods as suggested by [C] and holds potential in the domains such as concept learning and compositional generalisation. \n\nWe will add some discussion in the paper regarding this point.\n        \n**W4: Need for Weak Supervision**\n\nTo produce a compositional representation $\\psi(x) = C(\\psi_{1}(a_{1}), \\ldots, \\psi_{n}(a_{n}))$ (L116), each representational constituent $\\psi_{i}(a_{i})$ must map directly to a data constituent, $a_{i}$. Without supervision, this is challenging because the data constituents are unknown. \n    \nIn generative models, [19] showed that unsupervised disentanglement is theoretically impossible for essentially this reason: without supervision, there is no mechanism to match representational constituents with ground truth generative factors (role/filler bindings). Although our non-generative model avoids this impossibility result, the proof’s intuition is compelling. Thus, we use weak supervision [7,16,24,26,28] by presenting the model with data pairs $(x, x')$ differing in a subset of FoVs, indexed by set $I$. This weak supervision is minimal, only providing knowledge of the changing FoV types (e.g. ‘floor colour’) and not the FoV tokens (i.e., the specific floor colours in $x, x’$).\n    \nAs suggested by the reviewer, Table* 3 is added in the rebuttal to present additional variants on the supervision ablation from the perspective of disentanglement performance (please refer to Eq 7 for definitions of $\\lambda_{1}, \\lambda_{2}$).\n\nWe outline some possible extensions to reduce weak supervision:\n    \n 1. In the visual domain, some roles e.g. ‘object position' correspond to affordances.  Embodied agents may be able to reduce the need for explicit supervision by collecting $(x, x'), I$ through interaction. \n    \n 2. Initialising the filler embedding matrix, $M_{\\psi_{F}}$ with embeddings learnt by a pre-trained vision model could impart knowledge of domain-agnostic fillers (e.g. colours, shapes), reducing the need for weak supervision. \n    \n 3.  Segmentation masks for each constituent may potentially reduce the need for weak supervision.\n\n\n[A] Weston et al. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. \n\n[B] Yang et al. 2022. Towards Building a Group-based Unsupervised Representation Disentanglement Framework. ICLR.\n\n[C] Montero et al. 2022. Lost in Latent Space: Examining Failures of Disentangled Models at Combinatorial Generalisation. NeurIPS. \n\n[D] Fodor. 1975. The Language of Thought: A Theory of Mental Representation.\n\n[E] Chomsky. 1957. Syntactic Structures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bILZ2UvO7T", "reviewer": "Reviewer_xKnt", "summary": "The authors propose the use of the tensor product to model the interactions between object properties and their values, in contrast to the usual concatenation-based fusion for compositional representations. Extensive experiments on a large variety of image datasets are performed, where performance gains are often significantly higher than those of the baselines considered.", "questions": "**[Q1]** As expanded upon in [L933], the “role” embedding matrix $M_{\\xi_R}$ is initialized with $n$ orthogonal columns but not trained. We are told that it is “intuitive” that role embeddings do not need to convey semantic information, but I think it would be interesting to see an ablation study on this.\n\nIn particular: if it is the case that the embedding matrix does not need to learn semantic representations, why not make the simpler choice of initializing $M_{\\xi_R}$ as a (truncated) identity matrix (this shares the same property of semi-orthogonality)? The decision not to make this parameter learnable is peculiar, and I would be interested in seeing its impact explored experimentally (for example, through the DCI score as in Table. 7).", "limitations": "Limitations are discussed well throughout (and in further detail throughout the appendix), but an extra discussion of the potential drawbacks of increased computational costs should be made.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Modeling “multiplicative interactions” has proven to be a powerful mechanism for deep learning [1], but is largely neglected in modern times. This paper proposes an interesting way of fusing compositional representations of object properties and their values through the tensor product (and subsequent summation). Beyond the TPR’s success in this paper, I imagine the insights here through the proposed model form could help the design of more expressive deep learning architectures in future work more generally.\n- The experiments are exhaustive, the technical exposition clear and crisp, and for the most part, every architectural design and decision is justified in great detail. Overall, the paper is remarkably digestible given the technical sophistication, yet offers many insights.\n\n---\n\n- [1]: Jayakumar, Siddhant M. et al. “Multiplicative Interactions and Where to Find Them.” *International Conference on Learning Representations* (2020).", "weaknesses": "# [W1] Multiplicatively large dimensionality of the TPR representations & issues scaling to larger settings\n\nMy main criticism of the paper is the resultant TPR's dimensionality. In particular, the soft TPR representations live in a $D_f\\cdot D_r$ dimensional space. Due to the use of the Kronecker product, the dimension of the TPR representations grows multiplicatively with the two terms. \n\nWhilst the datasets studied in the paper are relatively simple and only 10 factors of variation are modeled, it seems prudent to acknowledge that the TPR size could grow prohibitively large for increased values of $D_f, D_r$ for more complex datasets outside of controlled settings, where significantly larger number of FoVs exist. In particular, the NeurIPS checklist states: `The authors should discuss the computational efficiency of the proposed algorithms`.\n\nFor example, I see from [L1008] that $N_R:=10$, and $N_F$ is as high as $106$ for Cars3D. Even in this regime with a very small number of roles, the TPR representations are (presumably?) larger than the concatenation-based representations of the baselines. A comparison of FLOPs/multiply-adds needed for the compared methods would be appreciated to better understand the methods’ drawbacks through the use of the (often computationally expensive) tensor product.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comprehensive and insightful review. \n\n**W1: Scalability Issues - Multiplicative Dimensionality of (Soft) TPR**\n\nAs noted, the Soft TPR lives in a $D_{F} \\cdot D_{R}$ dimensional space, which grows multiplicatively. However, several factors mitigate scalability concerns: \n\n1.\tIndependent Dimensionality: The dimensions of the role and filler embedding spaces ($D_{R}$ and $D_{F}$) can be fixed independently of the number of roles, $N_{R}$, and the number of fillers, $N_{F}$. Thus, Soft TPR’s dimensionality ($D_{F} \\cdot D_{R}$) can be smaller than $N_{F} \\cdot N_{R}$, the number of roles (FoV types) multiplied by the number of fillers (FoV tokens), which may be large in complex visual domains. In this rebuttal, Table* 5 is added to illustrate this, with the dimensionality of the TPR being smaller than $N_{F} \\cdot N_{R}$ in both the Shapes3D and MPI3D datasets.\n    \n2.\tRelaxing Orthogonality: While $D_{R} \\geq N_{R}$ is needed for semi-orthogonality of the role embedding matrix, $M_{\\psi_{R}}$, which guarantees faithful (L841-858) and computationally efficient (L917-929) recoverability of constituents, it is possible to relax this constraint to further reduce dimensionality.  When $D_{R} < N_{R}$, semi-orthogonality of $M_{\\psi_{R}}$, and thus faithful recoverability of constituents cannot be guaranteed, but some less stringent guarantees on unbinding outcomes can still be provided (see 291 of [1] for details).\n    \nTable* 6 is added in this rebuttal to compare Soft TPR’s dimensionality with other baselines. Scalar-tokened symbolic representations have a low dimensionality of 10 ($N_{R}$) at the expense of expressivity (each representational constituent $\\psi_{i}(a_{i})$ is scalar-valued). In contrast, Soft TPR has vector-valued constituents (i.e. approx $\\psi_{F}(f_{m(i)}) \\otimes \\psi_{R}(r_{i})$), like VCT and COMET. When compared to these models, our Soft TPR has significantly lower dimensionality than VCT and is comparable to COMET. \n\nDespite these mitigations, we acknowledge that the tensor product is computationally expensive. To more concretely address this concern, Table* 7 is added to indicate the FLOPs for a single forward pass on a batch size 16 using  fvcore [F]. This data demonstrates that, despite the tensor product's computational cost, the mathematically-informed derivation of our model allows it to obtain compositional representations with vector-valued constituents at a significantly lower cost than relevant baselines (2 orders magnitude less than VCT and 4 orders magnitude less than COMET). \n\nFuture work could explore the use of tensor contraction techniques [G,H] to reduce computational expense. For instance, [I] uses a Hadamard product based tensor product compression technique. This reduces computational cost from $nm$ (tensor product of 2 vectors) to $n$ (Hadamard product), but compromises the theoretical guarantees of constituent recoverability. We believe developing tensor contraction techniques within the TPR framework is an important direction for future research, to ensure efficient TPR-based representations with provable recoverability of constituents. \n\n\n**Q1: Non-learnability of role embedding matrix**\n\nSemi-orthogonality of the role embedding matrix, $M_{\\psi_{R}}$ ensures that the unbinding vector, $u_{i}$ (defined as the $i$-th column of $M_{\\psi_{R}}$’s left inverse (L841-850) corresponds to the $i$-th column of $M_{\\psi_{R}}$ (L925-927). The Unbinding Module leverages this to efficiently unbind soft fillers from $z$, using columns of $M_{\\psi_{R}}$ as unbinding vectors instead of performing costly matrix inversion. If $M_{\\psi_{R}}$ is not semi-orthogonal, the Unbinding Module cannot reliably produce soft fillers from $z$, as its columns are not true unbinding vectors. \n\nExtracting the soft fillers from the Soft TPR via unbinding ensures that the TPR decoder produces $\\psi_{tpr}^{\\*}$ as per Eq 5. Minimising $||z-\\psi_{tpr}^{\\*}||_{2}^{2}$ (term 1 in Eq 6) encourages the encoder, $E$, to produce outputs that are continuous relaxations of explicit TPRs. \n\nIn response to the reviewer's suggestion, we add Table* 8 in this rebuttal with results of the ablation where $M_{\\psi_{R}}$ is learnable. This retains a competitive edge over disentanglement baselines, but substantially lower FactorVAE and DCI scores empirically demonstrate the importance of fixing $M_{\\psi_{R}}$ for effective Soft TPR Autoencoder performance. This fixation leverages the unique mathematical properties of the Soft TPR/TPR framework, which are essential for effectively learning the representation. \n\nWe thank the reviewer for the suggestion to use a truncated identity matrix. Results in Table*8 show that a truncated identity matrix retains an edge over learnable $M_{\\psi_{R}}$, again highlighting the need for semi-orthogonality. However, the lower disentanglement results suggest potential benefits of random initialisation. Unbinding with columns of a truncated identity will force soft fillers to be axis-aligned ($\\tilde{f}\\_{m(i)}:= zu_{i}$), potentially limiting the flexibility and richness of learned representations. Additionally, a more varied set of weights in $M_{\\psi_{R}}$ produced by random initialisation helps to break symmetry in subsequent layers, and injects noise, which can improve overall learning dynamics [J, K]. \n\n[F]: Fvcore https://github.com/facebookresearch/fvcore/tree/main/docs \n\n[G]: Kossaifi et al. \"Tensor Contraction Layers for Parsimonious Deep Nets.\" CVPR 2017.\n\n[H]: Sharir et al. \"Neural Tensor Contractions and the Expressive Power of Deep Neural Quantum States.\" Physical Review 2022.\n\n[I]: Schlag et al. \"Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving.\" 2019.\n\n[J]: Glorot and Bengio. “Understanding the difficulty of training deep forward neural networks”. AISTATS 2010. \n\n[K] Noh et al. “Regularising Deep Neural Networks by Noise: Its Interpretation and Optimization.” Neurips 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
(Please note all rebuttal tables are included in the 1-page PDF)

**1) Incompatibility between disentangled representations and deep learning’s continuous vector spaces (Reviewer 93av Q1)**

Traditional disentanglement methods, although producing continuously-valued representations $\psi_{d}(x)$, use a symbolic, direct-sum approach to represent compositional structure (L37-40,122-124). Specifically, each FoV/data constituent $a_{i}$ is represented by a single dimension/contiguous subset of dimensions, $\psi_{i}(a_{i})$, which are concatenated together to form $\psi_{d}(x) = \psi_{1}(a_{1}) \oplus \ldots \oplus \psi_{n}(a_{n})$ 

Here, each representational component $\psi_{i}(a_{i})$ representing a FoV is embedded into a separate, independent subspace $V_{i}$ within the overall vector space. For example, consider 2 features, colour and shape, with embeddings $\psi_{col}(purple) = (1\\;0)^{T}, \psi_{sh}(square) = (1\\;0)^{T}$.

A disentangled representation is: $\psi_{d}(purple\\;square) = \psi_{col}(purple) \oplus \psi_{sh}(square) = \begin{bmatrix} 1 \\\ 0 \end{bmatrix} \oplus \begin{bmatrix} 1 \\\ 0 \end{bmatrix} = \begin{bmatrix} 1 \\\ 0 \\\ 1 \\\ 0 \end{bmatrix}$.

Here, colour and shape are allocated to distinct subspaces of the representational space $\mathbb{R}^{4}$ (i.e. dims 1&2 for colour,  dims 3&4 for shape). This discrete allocation mirrors symbolic systems where discrete symbols occupy separate spaces in the representation, although $\psi_{d}(x)$ is continuously-valued. 

Deep learning systems rely on gradient-based optimisation. Disentanglement’s symbolic method of representing constituency structure (i.e., allocating constituents to discrete subspaces of $\mathbb{R}^{4}$) complicates this process, as discrete subspace boundaries (i.e. between dims 1&2, dims 3&4) must be managed for each constituent. This complicates gradient-based processes, as changes must navigate abrupt transitions between subspaces rather than smooth, continuous alterations. 

Additionally, this symbolic method of representing constituency structure requires subspace alignment to ensure the overall representation in the larger vector space, $V_{F}$ is semantically meaningful (e.g. the colour subspace having significantly larger magnitudes than others). Guaranteeing this may be challenging. 

In contrast, TPR-based representations combine features in a unified vector space (e.g. $\psi_{col}(purple) \otimes \psi_{sh}(square) = (1\\;0)^{T} \otimes (1\\;0)^{T} \cong (1\\;0\\;0\\;0)^{T}$), meaning that constituents (e.g., colour, shape) are integrated into the same vector space and cannot be separated into discrete parts of the representational space, $\mathbb{R}^{4}$. This continuous approach of representing constituency structure avoids the issues of discrete subspace boundaries and subspace alignment, facilitating smooth gradient updates and more effective learning to enhance representation learner (C.3.3, C.4.2) and downstream (C.5) performance. 

**2) Why the relaxation of the TPR into the Soft TPR (Reviewer 6VtP W3, 93aV Q2)**

 For concreteness, consider the set of roles and fillers $R = \\{object\\;colour, object \\; shape\\}, F = \\{purple, green, square\\}$. We define role embedding $\psi_{R}: R \rightarrow \mathbb{R}^{2}$ and filler embedding $\psi_{F}: F \rightarrow \mathbb{R}^{3}$ functions as follows: $\psi_{R}(object\\;shape) = (1\\;0)^{T}, \psi_{R}(object\\;colour) = (0\\;1)^{T}$ and $\psi_{F}(purple) = (1\\;2\\;3)^{T}$,  $\psi_{F}(green) =  (2\\;2\\;3)^{T}$, $\psi_{F}(square) = (0\\;0\\;1)^{T}$

2.1. Discrete Mapping: The possible TPRs that can be produced are: 

1.	$\psi_{tpr}(purple/object\\;colour, square/object\\;shape) = \psi_{F}(purple) \otimes \psi_{R}(object\\;colour) + \psi_{F}(square) \otimes \psi_{R}(object\\;shape)  \cong (0\\;0\\;1\\;1\\;2\\;3)^{T} $  
2.	$\psi_{tpr}(green/object\\;colour, square/object\\;shape) = \psi_{F}(green) \otimes \psi_{R}(object\\;colour) + \psi_{F}(square) \otimes \psi_{R}(object\\;shape)  \cong (0\\;0\\;1\\;2\\;2\\;3)^{T}$  
	
The possible TPRs form a 2 element subset, $T$, of the underlying vector space $ \mathbb{R}^{6}$. Relaxing TPRs to Soft TPRs (L191) gives a larger set of points, $T_{s} = \\{(0\\; 0\\;1\\;2\\;2\\;3)^{T} + \alpha, (0\\;0\\;1\\;1\\;2\\;3)^{T} + \alpha : |\alpha| < \epsilon\\}$, which includes points like $(-0.0001\\;0.0002\\;1\\;1\\;2.004\\;3.009)^{T}$ and others. As $T_{s}$ has strictly more points than $T$, there should be more functions e.g., parameterising the map from the observed data to $T_{s}$, the set of Soft TPRs, than $T$, the set of TPRs. This should make the Soft TPR representation potentially easier to learn/extract information from than the TPR, reflected in our empirical results in C.6.1.

2.2. Quasi-Compositional Structure: The TPR enforces a strict algebraic definition of compositionality (i.e. $\sum_{i}\psi_{F}(f_{m(i)}) \otimes \psi_{R}(r_{i})$). The relaxation of this constraint in Soft TPR (L191) enables it to represent structures that only approximately satisfy the TPR’s strict algebraic definition of compositionality (e.g., in French liaison consonants, where a weighted sum of multiple fillers, rather than a single filler, bind to a role [L]) 

2.3. Serial Construction: Building explicit TPRs requires tokening constituents (role-filler binding embeddings) before the compositional representation can be produced [13,17,22,27,30,36,37]. Soft TPRs allow the encoder $E$ produce any arbitrary element of $V_{F} \otimes V_{R}$ (in this case $\cong \mathbb{R}^{6}$) provided it is sufficiently close to a TPR. Thus, once the Soft TPR Autoencoder is trained, it is theoretically possible to remove the TPR Decoder, and exploit vector space continuity to create approximately compositional representations directly from the Encoder, $E$, without needing to token the constituents. 

[L]: Smolensky and Goldrick. Gradient Symbolic Representations in Grammar: The case of French Liaison. 2016.

Meta-review: This paper proposes Soft Tensor Product Representations (Soft TPR) and the Soft TPR Autoencoder, a novel continuous compositional representation framework designed to bridge the gap between discrete symbolic representations and continuous vector spaces.

It tackles a crucial problem addressing a major limitation of current deep learning. Reviewers found the paper strong due to its thorough theoretical analysis, novelty of approach, clear explanation, and comprehensive experiments. Initial concerns, such as some experimental setting details, were also well addressed during the rebuttal. Therefore, I recommend acceptance.
Decision: Accept (poster)