Enriching Disentanglement: From Logical Definitions to Quantitative Metrics
Authors: Yivan Zhang
Masashi Sugiyama
Venue: NeurIPS 2024 poster (2024)
Keywords: Disentanglement
Representation Learning
Logic
Metric
Algebra
Category Theory
Topos Theory
Primary area: learning_theory

Abstract: Disentangling the explanatory factors in complex data is a promising approach for generalizable and data-efficient representation learning. While a variety of quantitative metrics for learning and evaluating disentangled representations have been proposed, it remains unclear what properties these metrics truly quantify. In this work, we establish algebraic relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metrics. Concretely, we introduce a compositional approach for converting a higher-order predicate into a real-valued quantity by replacing (i) equality with a strict premetric, (ii) the Heyting algebra of binary truth values with a quantale of continuous values, and (iii) quantifiers with aggregators. The metrics induced by logical definitions have strong theoretical guarantees, and some of them are easily differentiable and can be used as learning objectives directly. Finally, we empirically demonstrate the effectiveness of the proposed metrics by isolating different aspects of disentangled representations.

Enriching Disentanglement: From Logical Definitions to Quantitative Metrics
Yivan Zhang
Masashi Sugiyama
The University of Tokyo, RIKEN AIP Tokyo, Japan
RIKEN AIP, The University of Tokyo
Tokyo, Japan
yivanzhangōms.. .. u-t.kyo..oaoooo ac.
Abstract
Disentangling the explanatory factors in complex data is a promising approach for generalizable and data-efficient representation learning. While variety of quantitative metrics for learning and evaluating disentangled representations have been proposed, it remains unclear what properties these metrics truly quantify. In this work, we establish algebraic relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metries. Concretely, we introduce a compositional approach for converting a higher-order predicate into real-valued quantity by replacing (i) equality with strict premetric, (ii) the Heyting algebra of binary truth values with a quantale of continuous values, and (iii) quantifiers with aggregators. The metrics induced by logical definitions have strong theoretical guarantees, and some of them are easily differentiable and can be used as learning objectives directly. Finally, we empirically demonstrate the effectiveness of the proposed metries by isolating different aspects of disentangled representations.
Introduction
In supervised learning, we usually use real-valued cost function f: Y Y - R2o to measure how close an output ((x) function f X Y to target label y, i.e., {(f()),)) quantify the cost of inaccurate prediction. Then, we can use the total cost collection of input-output pairs to measure the performance of this function. From a functional perspective, this construction induces a quantitative metric L [X, Y] x [X, Y] - Rzo between functions:"
L(S.g) == E.
where X + Y is a "ground-truth function" that maps each input x to its target label y. This metric be used both learning objective and evaluation metric for the learning model f: X + Y. What does L(J, g) quantify? It quantifies the extent to which two functions f and g are equal:
(f --xx g) := Vr X. f(x) =y g(x).
(2)
Considering the equality as a predicate, we can observe a parallel between
(binary-valued) equality =y: Y x Y + [T, 113 and (real-valued) cost L. Y universal quantifier ("for all") Va E X and summation E and
R
R
function equality =| [X Y| x [X, + [T,L} and total cost L [X, YJ x X, R2o
We would like to ask: Is it possible measure and optimize other properties in the same way?
[X, Y] denotes the of all functions from X to set Y. In this paper, the domains of equality predicates explicitly subscripted.
(T, 1] denotes the set of binary truth values: true T and false 1.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In representation learning [Bengio et al., 2013], measuring and optimizing the performance of learning model becomes a non-trivial task. The quality of a model cannot always be measured by how close it is to fixed ground truth. Instead, we often need to consider the properties of the model architecture or learned representation itself, such as convexity [Amos et al., 2017], uniformity [Wang and Isola, 2020], invariance [Kvinge et al., 2022], and equivariance [Lee et al., 2019, Brehmer et al., 2023]. A proper comprehension of what constitutes good representations and how to assess their quality is important for designing suitable models, learning objectives, and evaluation metrics.
Disentangled representation learning: definitions, metrics, and methods Disentanglement is an important property in representation learning, which intuitively means that different explanatory factors in data should be encoded separately [Bengio et al., 2013J. However, disentanglement has no universally agreed-upon formal definition [Higgins et al., 2018. Suter et al., 2019, Shu et al., 2020, Fumero et al., 2021], and it is typically viewed not as a single property but rather as a combination of several requirements [Ridgeway and Mozer, 2018, Eastwood and Williams, 2018, Do and Tran, 2020, Tokui and Sato, 2022]. While many metrics for measuring disentanglement have been proposed [Carbonneau et al., 2022], it remains unclear what properties these metrics truly quantify and how they can be optimized directly. Often, a new evaluation metric is introduced along with new learning method, but is usually unproven that the method can optimize the new metric [Higgins et al., 2017, Kim and Mnih, 2018, Chen et al., 2018, Li et al., 2020]. This lack of theoretical understanding makes it difficult to design learning models that can effectively learn disentangled representations.
A logical and algebraic approach to defining and measuring disentangled representations Recently, Zhang and Sugiyama [2023] proposed a general and abstract definition of disentanglement, shedding light the common structures underlying the algebraic, statistical, and topological definitions of disentanglement. It was shown that the abstract concept of product [Mac Lane, 1978]
underlies essential property of disentanglement called modularity [Ridgeway and Mozer, 2018], and other properties of learning models, such as informativeness [Eastwood and Williams, 2018],
can also be defined abstractly using only the composition and identity of morphisms. Following this algebraic approach, we aim to derive theoretically grounded quantitative metrics of disentanglement from the logical definitions of the desired properties, extending the parallel between Eqs. (1) and (2).
Contributions In this paper, we focus on logically defined properties of disentangled representation learning, such as modularity and informativeness (Section 2). We introduce a compositional approach to converting higher-order equational predicate into real-valued quantity (Table 1), which serves as a quantitative metric of the extent to which a function satisfies the predicate (Section 3). Our analysis the relationship between the logical definitions and the induced quantitative metrics provides theoretical guarantee on the properties of the optimal functions (Theorem 1). Then, we demonstrate the usefulness of this conversion method by deriving quantitative metrics for measuring properties of disentangled representations, and we analyze these metrics in terms of computation, optimization, and differentiability (Section 4). Lastly,
compare the derived metrics with several existing ones in a fully controlled experiment and demonstrate that the proposed metrics are able to isolate different aspects of disentangled representations (Section 5).
2 Logical definitions of disentangled representations
In this section, let us first take closer look at the logical definitions of two properties of disentangled representation learning informativeness [Eastwood and Williams, 2018] and modularity [Ridgeway and Mozer, 2018], which are arguably more important than other properties [Carbonneau al., 2022]. We limit our discussion to sets and functions, but the generalization to other morphisms, such as equivariant, stochastic, or continuous functions, is straightforward.
2.1 Informativeness: injectivity or refractability of learning model
Being informative, expressive, faithful. or useful is a basic requirement for learned representations [Bengio et al., 2013]. We want a representation learning model to preserve explanatory factors in data that are informative to the downstream tasks. For functions, this criterion could be formulated as follows: If two factors y and y are different, then their representations m(y) and m(y') extracted by function Y + Z should be different too. This means that the function m should be injective:

---

Definition (Injective function). A function m Y - Z is injective if Punjectve m Y Z) == vy Vy' Y. (m(y) =2 m y/)
(3)
Alternatively, because injective functions are precisely functiins with retractions (left inverses) [Lawvere and Rosebrugh, 2003, Chapter 2], we can measure the retractabilityy instead:
Definition (Retractable function). A function m + Z has a retraction h - Y
Pretractable (m Y - Z) == 3h Z + Y. h [YYY] idy.
(4)
Note that these properties are predicates Punjectvee Pretractable | [T, on the set [Y, Z] of all functions from Y to Z. Analogous to using the total cost in Eq. (1) to measure function equality in Eq. (2), if we want to measure the injectivity in Eq. (3) or retractability in Eq. (4), we need to find quantitative counterparts of the implication ->, universal quantifier V. and existential quantifier E used in their logical definitions. Generally, it is desirable extend the parallel between Eqs. (I) and (2) to other predicates by finding quantitative operations corresponding to logical connectives and quantifiers. This correspondence allows to construct and analyze quantitative metrics for machine learning models in a compositional manner [Boole, 1854].
2.2 Modularity: product structure preserved by a learning model
Modularity [Ridgeway and Mozer, 2018] is an essential property of disentangled representation learning, which means that the explanatory factors in data, such the color and shape of an object, are separated into independent components in the learned representation [Bengio et al., 2013].
As shown in Fig. I, modularity can be defined as follows. We assume that data with multiple explanatory factors (e.g., color and shape) is generated via function g Y > X from a product Y == Y1 x Y2 of factors. An encoder product 7 == Z x 7-p of codes. Then, an encoder is said be modular if it can reconstruct the product structure, such that the composition
bluc
circke
red circle
blue circle
(0,0)
(1.0)
triangle
md
blue triangle
(0.
(L.1)
f: X Z is a function
factors
observations
codes
Y Color x Y shape
Zoolor x shaps
fog: Y + Z of the generator and the encoder is product function.*
m:= Mog Maas
Figure 1: Disentangled representation learning
Formally, being product function is also property that can be represented predicate: Definition 3 (Product function). Let Y == Yi x Y2 and Z = Z, x Zz be products sets. A function m Y + Z is product function if
Prrodusa(1
+ 2) == 1m1,1 : Y1 + Z1. 3mz YY + Zy. m FYY22 11.1 x m2
(5)
Example 1. Let us compare the following functions from Y (0, to 2 == R²:
(0,0) - (1.2)
(0,0) + (a,c)
(0,1) (3.4)
(0, + (a.d)
(0+
m :=
(1, - 6)
(6)
m ==
(1,0) (b
1
x
(1,1) (7,8)
(7,8)
(1, (b,d)
m1
m22
where a, b, c, and are arbitrary real numbers. According to Definition 3, only m = m1,1 x 2222 is
product function, whose first/second output depends only the first/second input.
In Example although m is not product function, we want to address the following questions:
(Metric) Can we quantify the extent to which it resembles a product function? (Approximation) Can find product function that is closest to it? (Differentiabilitt)) Can we make it slightly closer to a product function?
Answers to these questions will be given in the following sections.
'For two functions f A + C' and g B + D. their product f g: Ax + C' x D applies these two functions "in parallel" by mapping pair (a, b) in A x B pair (f(a) in C D.
3

---

3 Enrichment: from logic to metric
In Appendices A and B, we describe in detail the theory of converting higher-order predicate into a real-valued quantity. In this section, we only introduce the conversion procedure using concrete examples and present the theoretical results. A summary of the conversion is given in Table 1.
First of all, let us clarify the terms predicate and quantity. In the realm of classical logic, a predicate p : A + {T, 1 on a set A function from the set A to the set { of binary truth values. For example, the predicates Pinjective> Pretractable. and Pproduct in Definitions I to 3 are functions from the set [Y, 2| of functions to the set [T, They are logical definitions of some properties of functions. On the other hand, in this work, a quantity q A + [0, oo] on set A is defined [0, oo] of extended non- nog-tegae real numbers. The quantities associated with predicate will serve as quantitative metrics for the property defined by the predicate.
Table 1: From logic to metric
Logic
Metric
truth values
(T,1) real values
[0, oo]
equality conjunction disjunction implication
strict premetric d'
A
addition
+
V
minimum
min
subtraction"
universal quantifier
A
aggregator
V
existential quantifier F
infimum
inf
function to the set
truncated subtraction: == max (b a, 0}
c.g., maximum, sum, mean, and mean square
3.1 From equality predicate to strict premetric
In this work, predicate of central importance is the equality predicate =x= A x A + {T, +} [Mazur, 2008]. A quantity associated with the equality predicate should be a strict premetric:
Definition& 4 (Strict premetric). A strict premetric on set A is function dA A A + [0, oo] that
Va E A. Va' E A. (da(a, a') 0) ++ (a - aa)
(8)
3.2 From logical operation to quantitative operation
Next, let us have a look at the logical connectives and quantifiers used in the definitions of properties. The product sets and functions plays a significant role in this work. Two functions f,g : - Ax B to product are equal if and only if all their component functions are equal:
([=AA 9) == (fi FIC.A] 91) A (f2 [C. 92).
(9)
Note that the conjunction A [] x [T
(T, logical connective, is used in Eq. (9).
To obtain a corresponding quantity, we replace with the addition + [0, oo) x [0) - [0, oo]:
dic.Axe] 9) == (fc. + (f2.
(10)
The universal quantifier on set a specific (second-order) predicate VA {T, -T.
1T
on the set LJ of predicates. We can replace it with the supremum sup : [0, oo] - [0, oo]. We can also choose function from the (i) maximum, (ii) sum, (iii) mean, and (iv) mean square when the set A is finite. More generally, we can replace with quantity VA [0, so]* - co) on the set [0, oo) of quantities that satisfies some conditions, which we refer to as a (universal) aggregator. Intuitively, universal aggregator should output 0 if and only if all inputs are 0. Therefore, the median, mode, and range are non-examples. Different choices of aggregators yield metrics with different characteristics in computation and optimization. For example, the function equality predicate
[|AAB]] g) := Va E A. f(a) =B g(a)
(11)
converts to a quantity whose aggregator V is not limited to the sum (cf. Eqs. (1) and (2)):
(f g) == aел dn(f(a). g()))
(12)
Dually, we also need to consider the disjunction V and the existential quantifier 3. We replace them with the minimum and the infimum, respectively. Lastly, we replace the implication a > b with the (truncated) subtraction b - a == max(b a, 0}. These operations are illustrated in Fig. 2.
For a function f : C + A x B to a product A x B, its component functions fi : C + A : P1 o f
and f2 C + A = P2 o f are denoted by numeric subscripts, where A B + A = (a. - and (a, b) + b are projection functions.
P2: A B
4

---

a
Ael neji Apele) 950 deld a0te)
M60 sald aclo) 1060 oels!
(a) predicate and quantity
(b) conjunction
(c) disjunction
(d) implication
Figure 2: From predicates and logical operations to quantities and quantitative operations
3.3 From compound predicate compound quantity
Following Table 1, we can convert any compound predicate defined using equational predicates and logical operations into corresponding compound quantity defined using strict premetries and quantitative operations. Our main result on their relationship is as follows:
set A, and let q A 00, oo) be a quantity converted from p according to Table I. Then, for any E A. g(a) O implies p(a) = T. Conversely, implies q(a) if and only i does contain the implication.
Theorem 1. Let p : - (T, 1}} be predicate
for any A. p(a)
The implication is special because we must sacrifice logical equivalence for the sake of continuity, which is necessary for gradient- hased optimization. We will explore this through concrete example regarding injectivity Section 4.3 and discuss in detail Appendices and
3.4
hommooomnn from metric to logic
Finally, for readers interested in the theoretical background, we briefly introduce the following algebraic concepts and proof sketch underlying Table 1 and Theorem 1.
Definition 5 (Zero predicate). The zero predicate Ç : [0, oo] + {T, I (a = 0) is function that maps 0 to true T and any positive value to false 1.
Definition ((SubJhomomorphisn from quantity a predicate). Let A be a set. quantity
q: A + 00, oc] on the set A is homomorphic to predicate A [T. via the zero predicate [0] LT if o q = p. and is subhomomorphic p if ( o
Definition 7 (SSum)homooophhis from a quantitative operation a logical operation). Let be natural number. An n-ary quantitative operation : [0, oo]" |0, is homomorphic logical operation 3 [T, + [T, via the zero predicate Ç [0, ooj + [f
and
is subhomomorphic to 3 if o + <".
Definition 8 ((Sub) (Sooooohmmoophhisn from aggregator quantifier). Let A be set. An aggregator
a [0, oc] - [0, oo] on the set is homomorphic to a quantifier 3 [T, {T, 1} via the zero predicate Ç [0, oo] + [T,-} ( (o = 33 o ÇA and is subhomomorphic to B < Çoaa 9A A
Homomorphic quantities, quantitative operations, and aggregators can be illustrated as follows:
A A A
[0, 00)
11
[0 00
{T, 1]
(13)
[0, 00]
5 (T,
00, 00)
[T,
00|
[T
1}
"We use the infix notation, so o
means that Va E A. (g(a) = 0) + p(a). and y means that
Va A. (g(a) 0) p(a).
'S" [0, xo]" + [T,11" = (g....., 9n) + (4)
On 0) is the n-fold product of the zero
predicate <: [0, oo] - [T,L).
is the postcomposition with the zero predicate Ç: 00, oo) + [T,1} that maps quantity g: :: 10 20 the predicate ( o A {1,1).
(x [0, oo] + [T, = (-
5

---

Based on these algebraic concepts. we can say that strict premetrics are homomorphic to equality predicates, addition is homomorphic conjunction (since the sum is zero and only if both addends are zero), minimum is homomorphic to disjunction, truncated subtraction is subhomomorphic to implication, and universal aggregators are homomorphic to the universal quantifier.
Theorem means that any compound quantity is hsmmooohio to the corresponding compound predicate if each component (quantities, quantitative operations, aggregators) is (sub)homomorphic to the corresponding component (predicates, logical operations, quantifiers). For implication, we use the truncated subtraction, which is only subhomomorphic, since there no continuous operation that is homomorphic to implication (see also Appendix D).
More abstractly and concisely, can say that we replace the Heyting algebra of truth values {T, }
with quantale of extended non-negative real numbers [0, oo]. and we replace the quantifiers and derive quantitative metries for any logically defined properties of learning models compositionally.
with aggregators and inf (see also Appendix B). In this way,
4 Quantitative metrics of disentangled representations
In this section, we demonstrate how to apply the conversion method introduced above to derive quantitative metries for measuring the modularity (Definition 3) and informativeness (Definitions and 2) of disentangled representations.
In Sections 4.1 and 4.2, introduce modularity metrics based on two approaches and discuss their differences in terms of computation and optimization. We point out that the main obstacle lies in the optimization step, resulting from the existential quantifiers in the definition. Then, show that we can derive easily computable and differentiable metrics from a logically equivalent definition. In Section 4.3, introduce informativeness metries and present a result of Theorem
4.1 Modularity metrics via product approximation
We begin with modularity, which is an essential property of disentangled representation learning. Recall that modularity can be defined using the product function (Definition 3). For casier reference, we provide the following diagram, which shows the domains and codomains of the functions involved the upcoming discussion:
Y
P1
Y1 X Yg (41 V/2
Y2
V1
1/2
7
mg.2
(14)
71
Z1 Z22 m V2)
7z2
mi (91, y2)
m2 m2 U22
Pa
11,1
m222((2)))
From Definition 3, can derive the following metric:
Definition 9 (Product approximation). Let Y + Z be function from a product Y := Y1 x Y2 of sets to another product == Z, x Zn of sets. The extent to which m resembles a functtinn can be measured by distance between and its best product function approximation:
function
Aprochuct (m Y Z) 11,1 [[V
inf [V,,E(V
inf
d1v.2 (m, M1,1 m222)
(15)
The derivation of Gproduct from Pproduct follows the conversion described in Table replacing the equality =| [Y, 2) [Y, 2] + [T, with a strict premetric d(y,2] [Y, 2] [Y, Z] - [0, oo] and the existential quantifiers 3 with the infimum operators inf.
This modularity metric can be interpreted as distance from a point E [Y, 2] to subset {m1,1 X M2,2 | m1,1 E [Y,, 2,), m2,2 E [Y,, Zz]} C [Y, Z] of product functions (cf. the Hausdorff distance [Lawvere, 1986, Tuzhilin, 2016]). Following from Theorem 9produet
(m) (m) 0 if and only if Ppexuuc (m) T. This means that the minimizers of this metric are precisely product functions.

---

However, we still face two obstacles: the product operation and the minimization problem. For the product operation, we can employ Eqs. (10) and (12) to rewrite Aproduct into more computable form: Proposition 2.
The quantity 9prodcr (m: + Z) equals
V dz, (m))()11)))) V dz, (m, (21, mi, (v1)) W2EV2 ViEY
(16)
VIEV V2EY,
where the functions mi,1- : Vi= + 7 and m2,2 Y/2 702 are given by
(21, Mi,1 11 + 2 == 21 + arg inf V dz, (m 12).
(17)
11Z UsEY
22 : 222 + Z2 = y/2 1 ang dz, (m2(11 V2): 22).
(18)
2g BYE
A detailed derivation can be found in Appendix C. Note that we can obtain the optimal product function approximation mi.1 m2.2 explicitly via Eqs. (17) and (18). Intuitively, we need find an approximation of (multi)set of codes with one factor fixed and other factors varying, and then we use the aggregation of all the approximation errors as modularity metric.
The second obstacle - the minimization problem - still needs to be addressed. Since the code spaces Z1 and Zz can be infinite sets, the minimization problem may not have closed-form minimizer or even an exact solver. Even if an exact solver exists, the solution may not be differentiable with respect the inputs. Let us examine some concrete examples of 9produc by choosing different aggregators in Eqs. (17) and (18). In the following three examples, we assume that the code spaces Z and Z2 are Euclidean spaces equipped with the usual Euclidean distances.
Example 2. If the aggregator V is the supremum, the best approximation is the center of the smallest bounding sphere [Megiddo, 1983], and the approximation error is the radius.
This metric has the advantage of being definable even when the factor spaces Y, and Y2 are infinite sets, and can be computed using either randomized [Welzl, 1991] or exact [Fischer et al., 2003] algorithms. However, it is not easy to calculate its gradient. Thus, we cannot use it as a learning objective and directly optimize it using gradient-based optimization.
Example 3. If the aggregator v is the mean, the best approximation is the (geometric) median [Weiszfeld, 1937], and the approximation error is the mean absolute deviation around the median.
is known that there is no exact algorithm for obtaining the geometric median [Cockayne and Melzak, 1969], but can be effectively approximated using convex optimization [Cohen et al., 2016]. The geometric median has found applications in robust estimation in the fields of statistics and machine learning [Meer et al., 1991, Minsker, 2015, Pillutla et al., 2022, Guerraoui et al., 2023].
Example 4. If the aggregator V is the mean square, the best approximation is the mean, and the approximation error is the variance. In this case, Opoduua can be simplified to
mean var m2(11, U2) + mean var m2(11 32)- EY WsEY,
(19)
MrEY, 91E%,
The variance is easier to compute and differentiate than the radius of the smallest bounding sphere and the mean absolute deviation around the median, but it is also more susceptible to outliers and noise. Further work could explore the theoretical implications these metrics, especially in cases where only partial combinations of factors or noisy annotations are available.
Then, let us revisit our motivating example Example
Example 5. Let us consider the function m {0, 13 - TR. Eq. (6). its best product function approximation is
(0,0) + (2,4)
(0,1) + (2,6)
s0+ 2
0 4
m" 1) + +2²
x
(20)
(1,0) -6,0)+6,,4)
11+6 6
1
(1,1)+(6,6)
mia
m2,2
because mi,1 = 2 is the center/mediaa//mean of the set {m, (0,0) 1, (0, 1) = 3). and so on.
The modularity metric is distance between and m
7

---

4.2 Modularity metrics via constancy
Upon analyzing the metrics above, becomes evident that what we need is not the best approximation itself (e.g,, the mean) but rather the approximation error (e.g., the variance) a measure of the constancy of a set of codes. Following this insight, our next objective is to formulate modularity metric that eliminates the need for optimization step. Zhang and Sugiyama [2023] have proved that function is product function if and only if the curried functionn of its component functions are constant, as shown in the following example:
Example 6. Consider the functions m, m'
{0, 1)² + R in Eqs. (6) and (7) and the curried functions of their second component functions m2, m2 {0, 1) + R:
(0,0)
2 0
(0,0) +C
0 0 c 1
0
(0,1)
1
(0,1)
m2
(1,0) -
0 6
(21)
m2
(1,0)
0 H c
(22)
(1,1)+8
1 8
(1,1)+
1 d
The curried function m2 {0, 13 + [[o, 1}. IR] not constant, while m2 constant with value {0 + c, + d] [[0, 1]] (and is m)), indicating that m' is a product function.
Based on this fact, we propose an alternative approach for measuring modularity:
Definition 10 (Constancy of curried function). Let my and my be the component functions of a function m : Y + Z from a product Y := Y2 x Y/2 of sets another product == Z1 Zz of sets. The extent to which m resembles a product function can be measured by the constancy of the curried functions of m and mz:
Gconst curry (m: Y - Z) == Oconst (mij Tconst (m2),
(23)
where Iconst is a quantity for constant functions.
To complete this construction, we adopt the following definition and metric of the constant function:
Definition 11 (Constant function). A function f A + B is constant if
Poonst (f A + B) == Va E A. Va' E A. f(a) B B f(a'),
(24)
which can be measured by
Aconst (f : A + B) == V v da(f(c),f(d)),
(25)
e'A
This constancy metric Gonns only needs to compute pairwise distances between the outputs, requiring | times distance computation but no optimization. Incorporating Iconst into Iconst-cury- we can get the following metric:
Proposition 3. The quantity Qconar-cury (m Y + Z) equals
V V dz, (m, (V1. v2), mn (11. 3/2)
MY EY VEEY, V WEY. MieY,
(26)
V dz (m2(11. V2), m2 (2/1 U2))
Here are two examples of Aconst and Gconst-curry using different aggregators V in Eqs. (25) and (26).
Example 7. If the aggregator is the maximum, Icona the diameter (the maximum pairwise distance) of the outputs. In this case, Jconst-cury (m) can be simplified to
max diam my (V1, U2) + max diam m,(11. V,EY, eY
(27)
in eY My
Example 8. If the aggregator V is the mean square, Iconst is the mean pairwise squared distance, which equals the variance. In this case, leumss-vurr (m) coincides with Eq. (19).
In summary, Eqs. (19) and (27) are easily computable and difffcrentiable metrics, and their minimizers are precisely product functions. They do not contain any hyperparameters or stochastic components and thus can serve as both learning objectives and evaluation metrics.
For a binary function f A B C. its curried function f= A [B, C] is unary function such that for all a A and b E B. f(a b) (b) [Curr, 1980].
8

---

4.3 Informativeness metrics
If an encoder f + Z is constant, mapping everything to the same value, according to Definition it is perfectly modular. However, a constant encoder is also completely useless. In this subsection, we shift our focus the property of informativeness
measurement of usefulness.
Informativeness is not a unique requirement for disentangled representations. Other representation learning paradigms, such contrastive learning [Jaiswal et al., 2020, Wang and Isola, 2020] and metric learning [Musgrave et al., 2020], also emphasize the importance of mapping dissimilar data far-apart locations in the representation space. While one could integrate this requirement into a single disentanglement score (e.g.. [Higgins et al., 2017, Kim and Mnih, 2018J), we argue that it better to evaluate the usefulness of representations separately for more fine-grained assessment [Carbonneau et al., 2022].
One straightforward way to measure informativeness is to measure how much encoding process:
invert the
Definition 12 (Retraction approximation). Let m : Y be a function. The extent to which m is retractable can be measured by distance between the composition of m and its best retraction approximation and the identity function:
Gretractble (m Y 2)
dr:r m, idy) kE(ZM)
inf
inf v dy (h((((())) ne(ZY) VEY
(28)
This metric Geetretable is derived from Definition 2 following the conversion procedure in Table 1. This informativeness metric also involves an optimization step similar to the modularity metric Spoduct: potentially introducing randomness or higher computation costs. Note that we may use parameterized subset of the set [Z, Y] of all functions from codes Z to factors Y, such as the set of linear functions. Then, the problem becomes regression/classifccttio the performance of the predictor. A number of existing works adopted this approach and used the accuracy, the area under the ROC curve (AUC-ROC), or the mean squared error (MSE) measure the informativeness [Ridgeway and Mozer, 2018, Eastwood and Williams, 2018, Eastwood et al., 2023]. However, such metrics necessitate additional hyperparameter tuning and are more likely to exhibit varying behavior across different implementations [Carbonneau et al., 2022].
problem, and the metric is
raises the question of whether we can measure the informativeness of an encoder without approximating its retraction. We propose to measure informativeness by directly measuring the injectivity of the encoding process:
Definition 13 (Contraction). Let Y + Z be function. The extent to which m is injective can be measured by how much m contracts pairs of inputs:
lnjesctive (m Z) = yEY s'eY (y( - m2(ml))
(29)
This metric Iinjective is derived from Definition I following the conversion procedure in Table 1. According to Theorem 1, we know that Ietraetable (m) if and only if m is retractable. However, Unjeective (m) = 0 implies the injectivity of m but not the other way around:
(Premactable (m) = T)
(Pijecttive (m) T)
logically equivalent
(30)
Theorem
Theorem
(Iretractable (m) 0)
(Tinjective (m) = 0)
In other words, a minimizer of Uuneective is required to be non-contractive, which is a stronger condition than being injective. For example, let us consider the function m : [0, 1] - RR == y + 0.01 x y. Although it is injective, its outputs are less distinguishable from each other in terms of the Euclidean distance. Therefore, Onjecttvve still assign a non-zero value to this function.
Although not all injective functions necessarily minimize Injective according to Theorem 1, we can guarantee that minimizing Qnjjettive will not lead to non-injective functions. Moreover, Qnjecuive does not require training regressors or classifiers to approximate the retraction. Consequently, it does not need any time-consuming hyperparameter tuning or vals-vvaliotionn like existing informativeness metrics [Eastwood and Williams, 2018, Ridgeway and Mozer, 2018].

---

Table Supervised disentanglement metrics
Modulardy
Infonmativeness
Existing metnes
Product uppros.
Comstancy
Retraction appros.
Contraction
Pair
Info.
Regressur
Rod.
MAD
Diam.
MPD
ME
MAE
MSE
Max
Mecnn
Beta"
Fucter"
MIG
Dis."
Com." f
Info."
entangloment
0.44
0.75
01.96
0.19
0..8
/
1020
11:30
11:11
0.44
0.78
0..89
0.83
0.18
0.25
0.28
1.00
rotation
0.22
0.51
0.80
0.05
0.64
/
1:00
1.00
1.00
1:00
1:30
0.96
0.34
0.17
0.40
0.40
1.00
duplicate
0.24
0.43
0.67
0.06
0.56
/
1.00
1.00
1.00
1.00
1.30
1.00
1.00
1.00
1.00
0.50
1.00
complement
0.12
0.25
0.55
0.01
0.42
/
1.00
1.00
1.00
1.00
1.00
1.00
0.00
1.00
1.00
0.63
misalignmeut
0.74
0.58
1.00
1.000
1.000
1.00
1.00
1.001
1.00
redundancy
1,00
1.000
1,000
1.00
1.000
1.00
contraction
1.00
1.00
1.00
1.00
1,00
1.00
0.16
0.40
1.00
1.00
1.00
1.00
monlincar
1.00
1.00
1.00
1.00
1,00
11:0
11:00
0.65
0.95
1.00
1.00
0.58
1.00
1.00
constani
1.00
1.00
1.00
1.00
1,00
X
0:42
0.76
0.90
0.18
0.68
0.35
0:33
0.00
0.00
0.00
0.00
random
0.22
0.48
0.78
006
0.81
X
0.42
0.76
0.30
0.22
0.85
0.33
0.30
0.00
0.00
0.00
0.01
[Higgins al., 2017]
[Kim and Mnih, 2018]
[Chen et al., 2018]
[Eastwood and Williams, 2018]
5 Experiments
In this section, we cmpirically demonstrate the effectiveness of the proposed metrics. Following Carbonneau et al. [2022], we did not learn representations datasets but directly defined functions m Y - Z from factors to codes, which allows us to capture typical failure patterns.
We evaluated modularity metrics based on (i) the radius of the smallest bounding sphere, (ii) mean absolute deviation (MAD) around the median, (iii) variance, (iv) diameter, and (v) mean pairwise distance (MPD) introduced in Sections 4.1 and 4.2. We evaluated informativeness metrics based on retraction approximation using the maximum error (ME), mean absolute error (MAE), and mean squared error (MSE), and calculated the contraction discussed in Section 4.3. To compare with existing metrics [Higgins et al., 2017, Kim and Mnih, 2018, Chen et al., 2018, Eastwood and Williams, 2018], we transformed the results isomorphically using e- : [0, ooj - [O, 1]. meaning that 1 is the perfect score. The results are shown in Table 2, and our observations
follows.
If representation is given a perfect score by proposed metric, it must satisfy the property that the metric quantifies, which confirms our theoretical result. In Table 2, the light cells show that the proposed metrics can assign a perfect score when the function truly satisfies the properties, which is indicated by / and X. The dark cells are supposed to be perfect scores, but they fall short due to the limited expressiveness of the linear models used for the approximation. Meanwhile, some existing metrics that only provide a single score may entangle modularity and informativeness..
The metrics derived from equivalent definitions may differ in terms of computation cost and differentiability. Concretely, the radius, MAD, ME, MAE, and MSE are not differentiable due to the inner optimization problem, while the variance, diameter, MPD, and max/mean contraction are differentiable. In terms of computation, they are much faster than metrics requiring hyperparameter tuning, such DCI [Fastwood and Williams, 20. Further comparisons are provided in Appendix E.
Different metries may rank imperfect representations differently, even though they have exactly the same minimizers. This difference can lead to differences in risk preferences, sensitivity to outliers, and learning dynamics when these metrics are used as learning objectives. Illustrations and further diseussion can be found in Appendix D.
Further, the proposed metrics can be used in weakly supervised or mrained evaluation. See Appendix E for detailed data configuration and further experimental results.
6 Conclusion
In this work, we developed a systematic and rigorous method for converting logical definitions of properties of representation learning models into quantitative metrics (Table 1). We applied this method to assess two important and distinct properties of disentangled representations: modularity and informativeness. We derived two families of metries for each property based on their logically equivalent definitions. We theoretically analyzed the minimizers of these metrics (Theorem 1) and compared their differences in terms of computation cost and differentiability. Future research could compare metrics derived from different aggregators and design appropriate models to optimize these metrics with minimal supervision.
10

---

Acknowledgments
We thank Paolo Perrone for generously sharing insights on Markov categories enriched in divergence spaces. We thank Ken Sakayori for reviewing an earlier version of Appendices A and B and providing constructive suggestions. We thank Zhiyuan Zhan for insightful discussions on metrics, topology. measures, and optimization of function properties, and for reviewing and proofreading some parts of Appendix We thank Masahiro Negishi for valuable discussions on disentanglement metries and weakly supervised disentanglement. We thank Jingwen Fu for checking the algebraic concepts in Section 3. We also thank Johannes Ackermann, Xin-Qiang Cai, and Tongtong Fang for their valuable feedback on the manuscript.
YZ was supported by JSPS KAKENHI Grant Number 22KJ0880. MS was supported by JST CREST Grant Number JPMJCR 18A2 and grant from Apple, Inc. Any views, opinions, findings, and conclusions recommendations expressed in this material are those of the authors and should be interpreted as reflecting the views, policies position, either expressed or implied, of Apple Inc.
References
Jiří Adámek, Horst Herrlich, and George Strecker. Abstract and Concrete Categories: The Joy of Cats. John Wiley and Sons, 1990. URL http:///www www.cmm.aa..m. wttpaa//taaaaaeeeeeeee httpttaaaaepprintaaaaaaaaaa ricles/a7/tr17abs. html. html. A.1
K Amer. Equationally complete classes of commutative monoids with monus. Algebra Universalis, 18:129-131, 1984. URL https: htttt:::://ooooooo /tt::://oooooooooo../// 10t1///100..00//8111811222.. 18
Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference on Machine Learning, 2017. URL htt.htpp//prroceedingg
mlr. prt..press/77//mmos117
html. 1, D.1
Steve Awodey. Category Theory. Oxford University Press, 2010. URL htw.:o::ooo0000100000 05556/99889988881222. 001 0001. A.I
2093/acprod
Giorgio Bacci, Radu Mardare, Prakash Panangaden, and Gordon Plotkin. Propositional logics for the Lawvere quantale. Electronic Notes in Theoretical Informatics and Computer Science, 2023.
URL hts / /doi. ora.iorg//oo0 10. 46298 entiestti...e...eee 12292. 2302 01224. B.9, D.2
Giorgio Bacci, Radu Mardare, Prakash Panangaden, and Gordon Plotkin. Polynomial Lawvere logic. arXiv preprint, 2024. URL https: wttp:rgg///b...a///////////// 03543. D.2
Samy Badreddine, Artur d' Avila Garcez, Luciano Serafini, and Michael Spranger. Logic tensor networks. Artificial Intelligence. 303:103649, 2022. URL https: //oii. htgg/10.1011 artint. 2021. 103649. ht httpp:::o /ttp:////axivv//// htng/abs/201110 13635. D.2
Nikita Balabin, Daria Voronkova, Ilya Trofimov, Evgeny Burnaev, and Serguei Barannikov. Disentanglement learning via topology. In International Conference on Machine Learning, 2024.
D.1
Han Bao and Masashi Sugiyama. Calibrated surrogate maximization of linear-fracte fractional utility in binary classification. In International Conference on Artificial Intelligence and Statistics, pages 2337-2347, 2020. URL httt::://rooeeeeee http::/procceeingg... mlr wttp://ssswm.r..rs/erereeee /waress/voo//// bwtp:/aaooaaaaaaa html. D.2
Han Bao, Clay Scott, and Masashi Sugiyama. Calibrated surrogate losses for adversarially robust classification. In Conference on Learning Theory, pages 408-451, 2020. URL http: / /ppooeeeiings.... mlr. press/v125. prress/va25/bao2aa bao2ea. html. D.2
Michael Barr and Charles Wells. Category Theory for Computing Science, volume 1. Prentice Hall New York, 1990. URL https: //wp:::/www.... wawhww/wwtthhhha...ooo..mm.mmm S. sdp.f. A.1
Welss-ce
Sugato Basu, Arindam Banerjee, and Raymond J Mooney. Semi-supervised clustering by seeding. In International Conference on Machine Learning, 2002. URI. https: //dl. arm. doi1
o. 00.5555/64555 656012. E.2

---

Jens Behrmann, Will Grathwohl, Ricky TQ Chen. David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference on Machine Learning, 2019. URL httpss://proceednngg mwr. wil..ppess//ve pttppp/s////eeeeraaananneeeeeeee
html. D.1
Itai Ben Yaacov. On the expressive power quantifiers continuous logic. arXiv preprint, 2022.
URL httph::t///// attps://arxivv www.oorgg//////eee abs/ 01863. D.2
Itai Ben Yaacov and Alexander Usvyatsov. Continuous first order logic and local stability. Transactions of the American Mathematical Sociery, 362(10):5213355255, 2010. URL https: //doi. wtw..o..././///........ 01.4303 D.2
9947 10- 04837-3, https://axxi.oorg/abss.
Itai Ben Yaacov, Alexander Berenstein, C. Ward Henson, and Alexander Usvyatsov. Model Theory for Metric Structures, page 315-427. London Mathematical Society Lecture Note Series. Cambridge University Press, 2008. URL https:://don org/ 10. 101177/BB09/88511772299
D.2
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and perspectives. IEEE Transactions Pattern Analysis and Machine Intelligence, 35(8): 1798-1828, 2013. URL https: /oii orgg 10. 1109/TPAMM 201AAMI.200.... 50. https: //arxiv.org org./vvvorr//e htt.r//gaaa///o/oae.e/// 5538. 1, 2.1, 2.2, D.1
Merrie Bergmann. An Introduction to Many-
Many-Valued Valued and Fuzzy Logic: Semantics, Algebras, and Derivation Systems. Cambridge University Press, 2008. URL httpt: http::://doi.orr///// 1017/
C809780511801122 B.1, D.2
Mikhail Bilenko, Sugato Basu, and Raymond J Mooney. Integrating constraints and metric learning in semi-supervised clustering. In International Conference on Machine Learning. 2004. URL
attp:::///lllac.ooorgo.o... org/ /wtgg//oo/10// 10. 1tt.:///....111/111111111
1015360. E.2
George Boole, An Investigation of the Laws of Thought: On Which Are Founded the Mathematical Theories of Logic and Probabilities. Cambridge University Press, 1854. URL https: 2.1, D.2
ordoii.or/// 10. 1017 1017/CB09 (019999788511199900
Tai-Danae Bradley, John Terilla, and Yiannis Vlassopoulos, An enriched category theory of language: From syntax to semantics. La Matematica, ( 1(2):551-580, 2022. URL https: : //ooi. org/10 10
1.007/544007 022 00021- 2. https: /tts:///arxiii www.rxiwi.orggb...e. wtrg/bbs/2166 07890. A.3
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal representation learning. In Neural Information Processing Systems, 2022. URL. https: //openreview. net:/orrmmidddz79giie
D.2
Johann Brehmer, Pim De Haan, Sönke Behrends, and Taco Cohen. Geometric algebra transformer.
In Neural Information Processing Systems, 2023. URL
? rid=M7r2C04tJe D.1
Leo Breiman, Jerome Friedman, Richard A. Olshen, and Charles J. Stone. Classification and Regression Trees. Routledge, 1984. URL https:///of: 1ttp://....0011///1111199977777 D.l
10.
Chris Burgess and Hyunjik Kim. 3D shapes dataset, 2018. https://github.com comp:/ttthub.momm///i.m..... d/3d-shaper (Apache License 2.0). 6, 5, E.3, 7, 10
Matteo Capucci. On quantifiers for quantitative reasoning, arXiv preprint, 2024. URL https:
//arxiv. wwwp:ivv.rrgab///eeee/ee 2406 04936. D.2
Marc-André Carbonneau, Julian Zaidi, Jonathan Boilard, and Ghyslain Gagnon. Mcasuring disentanglement: A review of metrics. IEEE Transactions on Neural Networks and Learning Systems, 2022. URL https: //op::::/ddi..o org/ 10. 1109/ TTNNNS 2022. 32182222..... 1, 2, 4.3, 4.3, 5, D.1
Hugo Aaselles-uupr's Michael Garcia Ortiz, and David Filliat. Symmetry-based disentangled representation learning requires interaction with environments. In Neural Information Processing Systems, 2019. URL https: htttt::::/pproceeddnneee newrips...... cc
h.mm. D.I
12

---

Chen Chung Chang. Algebraic analysis of many valued logics. Transactions of the American Mathematical society, 88(2):467-490, 1958. URL httpp:://oott org. 1993277 D.2
Chen Chung Chang and H Jerome Keisler. Continuous Model Theory, volume 58 of Annals of Mathematics Studies. Princeton University Press, 1966. URL https://doi.orge 15115/155/1 9781400882052. D.2
Ricky TQ Chen, Xuechen Li, Roger В Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Neural Information Processing Systems, 2018. http://oooeenn.nn....iue.rueeaee www.nneuriiiccmm http/papeee paper/ ptaaper/2188. 2018/ hawsh/ahhhh//e wwwaaaahshhheeeeeeeeeeeeeeeeeeee 1ee3df cd8a064 5a25a35977997223d22 Abstract. URL http://9w222d222aaasttt.....ct html. 1, 2, 5, D.1, 5, E.3, 6,
Shuxiao Chen, Edgar Dobriban, and Jane H Lee. A group-theoretic framework for data augmentation. The Journal of Machine Learning Research, 21(245):1-71, 2020. URL httptt/p/jjm...o
apers/v21/201 163. html. D.1
Yiting Chen, Zhanpeng Zhou, and Junchi Yan. Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. In International Conference on Learning Representations, 2024. URL //tp:e///ppeevvieww www.eewwweeffrruum ?id=4bSQ3lst lsEV.. A.1
Kenta Cho and Bart Jacobs. Disintegration and Bayesian inversion via string diagrams. Mathematical Structures in Computer Science, 29(7):938-771 2019. URL h http:://doooooo////./// 1017/ 10117
50960129518000488 https: http::/aax.....r/////////////// 00322. A.1
Simon Cho. Categorical semantics metric spaces and continuous logic. The Journal of Symbolic Logic, 853))11044-1078, 2020. URL
org/ 10. 1017/jsl. 2020 44. D.2
Ernest Cockayne and Zdzislaw Melzak. Euclidean constructibility in graph-minimizatioo problems. Mathematics Magazine, 42(4):206-200 1969. URL htthtts:://doi http://oo.rr///.... 108 0/002557XX 196.. 11975961. 4.1
Michael B Cohen, Yin Tat Lee, Gary Miller, Jakub Pachocki, and Aaron Sidford. Geometric median in nearly linear time. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 9-21, 2016. URL 4.1
Taco Cohen. Equivariant convolutional networss. PhD thesis, University of Amsterdam, 2021. URL http:///t...aaaeeeeeee//a//////. 0e6. D.1
Taco Cohen and Max Welling. Learning the irreducible representations of commutative Lie groups. In International Conference Machine Learning, 2014. URL httpt:t/p/occeediiiiii ml r. T.press/v32/ cohen14 cotp////....///......////. html. D.1
Taco Cohen and Max Welling. Transformation properties of learned visual representations, In International Conference on Learning Representations, 2015. URL http::/aaxxv..orgg..////// //1412.7659 D.1
Taco Cohen and Max Welling. Group equivariant convolutionl networks. International Conference Machine Learning, 2016. URL http::/ppooceeeinggeee .mw.. wwwpp/eesssseeee cohencc
html. D.I
Geoffrey SH Cruttwell, Bruno Gavranović, Neil Ghani, Paul Wilson, and Fabio Zanasi. Categorical foundations gradient- learning. Programming Languages and Systems, pages 1-28. Springer International Publishing, 2022. URL https:/ doi.. org/10. 10. 0007/9788 300 -99336-8 1. http::://aaw...o//aaaaeoae 2tt.a///2211...11...... 01931. A.l
Haskell B. Curry. Some philosophical aspects of combinatory logic. In Studies in Logic and the Foundations of Mathematics, volume 101, pages 85-101. Elsevier, 1980. URL https:
//doi. 10. 1011 1011///000011100 (0t
13

---

Francesco Dagnino and Fabio Pasquali. Logical foundations of quantitative equality. In Logic in Computer Science, 2022. URL https://doi. org/ 10. 100111131111110000 3533337. D.2
Hennie Daniels and Marina Velikova. Monotone and partially monotone neural networks. IEEE Transactions on Neural Networks, 21(6):906 2010. URL httpt:::///iiorr/e/ 110
9/TNN. 2010. 2044803. D.1
Artur d' Avila Garcez, Krysia B. Broda, and Dov M. Gabbay. Neural-Symbolic Learning Systems. wwg/1o...... 1 1-77---------111----- D.2
Springer, 2002. URL
Pim de Haan, Taco Cohen, and Max Welling, Natural graph networks. In Neural Information Processing Systems, 2020. URL httph: http:::/oorroecedggg..o neurips. cc/ wttpripppccppppee///cppppp pappp/ppppaa// 2020/
www.::aatt..-ttttttttaaa.aaa.eee html. A.I, D.1
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wuthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. In International Conference on Learning Representations, 2021. URL https: /ttps::/opennreview net/forum?id=sVXy 1QRR11, D.1
Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representations. In International Conference Learning Representations, 2020. URL wwwpt:ttt//:/teeeeeee
D.1
Andrew Dudzik. Quantales and Hyperstructures: Monads, Mo'Problems. PhD thesis, University of California, Berkeley, 2017. URL. https//////ax
B.9
Andrew Joseph Dudzik and Petar Veličković. Graph neural networks are dynamic programmers. In Neural Information Processing Systems, 2022. URL http:ttt////ooeeeeiiew.... www.wriiiiiiee/orummm d=wu1ZagdY1GY. A.1
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018. URL ht tpst / miee/forrumid=BBB 7 AZ. 1, 4.3, 4.3, D.1, 5, 6,7, 7, E.4, E.5
Cian Eastwood, Andrei Liviu Nicolicioiu, Julius Von Kügelgen, Armin Kekić, Frederik Träuble, Andrea Dittadi, and Bernhard Schölkopf. DCI-ES: An extended disentanglement framework with connections identifiability. In International Conference Learning Representations, 2023.
URL wwwph:tpss///mm wwwphtttt///pereeeeeeee http://pppeeeeeeeeeeeeeeeee
(ttprrumiiii422---ae gLgSht. 4.3, D.1
Ronald Fagin, Ryan Riegel, and Alexander Gray. Foundations of reasoning with uncertainty via real-valued logics. Proceedings of the National Academy of Sciences, 121(21), 2024. URL
https://dol: org/ 10. 1073 pnas 2309905121, D.2
Daniel Figueroa and Benno van den Berg. A topos for continuous logic. Theory and Applications of Categories, 38(28), 2022. URL httpt::/// //www...//w///// maa.mmta.ca aa/aa///aaa //cc/ wowouuuu///eeooooooooeeo /tteee////////// 8-28abs. html. D.2
Kaspar Fischer, Bernd Gärtner, and Martin Kutz. Fast smallest-enclosing-bar computation in high dimensions. In European Symposium on Algorithms,, pages 630-641. Springer, 2003. URL htt////ii..rg//..10......./
540- 1110039658-115777 1_57. 4.1
Brendan Fong David Spivak. An Invitation to Applied Category Theory: Seven Sketches in Compositionality Cambridge University Press, 2019. URL https: / /ww./oooooooooooooo org/ orrg/o... 1017/ 9781108668804, https: :ttps://artie /ttp://aaxiv.a/aaa org/ / abs/ 05316. A.I, B.1, B.6, B.8
Tobias Fritz. A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statisties. Advances Mathematics, 370:107239, 2020. URI. https:://oo.i...o org/
10. 1016/
(a.e0166.... a.am.. 2020 107239.
wwwpxiiv.orgaaaaa harg/abs/sag www.aa//....////..//..////eeee 07921. A.I
Soichiro Fujii. Ordered semirings and subadditive morphisms. arXiv preprint, 2023. URL http://wwwiivi.rr/////.1....
https://arxiin wrg. httpg///b///....p.p// 03862. B.9
14

---

Marco Fumero, Luca Cosmo, Simone Melzi, and Emanuele Rodolà. Learning disentangled representations via product manifold projection. In International Conference on Machine Learning.
2021. URL http: http://poooeeen...////////////.p mlr. wtr..rreesvv1/ fumero21a press/v11g/fumero211a html. 1, D.1
Bruno Gavranović, Paul Lessard, Andrew Joseph Dudzik, Tamara
Glehn, João Guilherme Madeira Araújo, and Petar Veličković. Position: Categorical deep learning is an algebraic theory of all architectures. International Conference on Machine Learning, 2024.
URL https::t///o /ttps://pppeevvieeeee www.ee.weeee/teee SewiumiidIIIIITTTSSS A.1
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation the real world: new disentanglement dataset. In Neural Information Processing Systems, 2019. URL httpt::t//prroceeeinnggg neurips. cc -wwe:e44400bbttrtaaaoo Abstract html. https: www.ttt..co/rrrrteeriinnnnnneee. Learr--/arniinnngdeee disggg/diieentte www.entttnnttnnnnnneeeeeeeeeeeee dataeet (Creative Commons Attribution 4.0 International License). 6, 5, E.3, E.6, 11
Ian Goodfellow, Honglak Lee, Quoe Le, Andrew Saxe, and Andrew Ng. Measuring invariances in deep networks. In Neural Information Processing Systems, 2009. URL https::////cceee dings. neurips..... CC/
e e-Abstract html. D.1
Rachid Guerraoui, Nirupam Gupta, and Rafael Pinot. Byzantine machine learning: primer. ACM Computing Surveys, 2023. URL pttpps://ooil. orroorg///// 11t.//...1111/.1111
4.1
Petr Hájek. Metamathhematics of Fuzzy Logic, volume 4 of Trends in Logic. Springer, 1998. URL
https: /tts:///ooiii org. 11.11011110110 100000077999 978 90-8-94--00 0144--1--3350 5300- 3. D.2
Charles R. Harris, K. Jarrod Millman, Stéfan J. der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Marchan,, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):357-362, 2020. URL https::////ooo org 10. 1000/0411/1 /ttp:p////umpaa/ org. D.6
86 000-220-264--
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAL VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017. URL hww.h:tttts///ctm http:///ppeeeeeeeeeeeeee 5, F.3, E.4, 6, 7, E.6
1, 4.3, 2, D.1, D.2,
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loie Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint, 2018. URL /ttt:s://arxivv org. tabs/1812. 02230. 1, D.1
Aapo Hyvärinen and Erkki Oja. Independent component analysis: Algorithms and applications. Neural networks, 13(4):411-43)) 13000 2000. URL https://dooo...... org/ 10. 1011/S0893--1 608 o 00026-5. D.1
Isao Ishikawa, Takeshi Teshima, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Universal approximation property of invertible neural networks. Journal of Machine Learning Research, 24(287):1-68, 2023. URL https: htthttts::/ :/wp::///www. jmlr. org/ ptpppsppppaas///e /t/p//2222222222222 html. D.I
22 0384.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey contrastive self-supervised learning. Technologies, 9(1):2, 2020. URL 10. https://doi.. wroooro///00 10. 3390/ htt/://..33//tcecoooooogog// www.eooogieeeooeeoooooooo S/2011. 00362. 4.3
https://arxiv.org.
Peter Johnstone. Sketches of an Elephant: A Topos Theory Compendium. Oxford University Press, 2002. A.2
15

---

Max Kelly. Basic concepts of enriched category theory. In London Mathematical Society Lecture Note Series, volume 64, Cambridge University Press, 1982. URL.ttp://www. mta. itta.. html. A.3
Maurice G. Kendall. A new measure rank correlation. Biometrika, 30(1/2):81-93, 1938. URL
hetps://doi..rg// 10. 2307 2307/2332226, E.4
Hamza Keurti, Hsiao-Ru Pan, Michel Besserve, Benjamin F Grewe, and Berharr Schölkopf. Homomorphism AutoEncoder learning group structured representations from observed transitions. In International Conference on Machine Learning, 2023. URL https://pr
oceedings mlr. wttSss//222/keurrizg html. D.1
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, 2018. URL http: http::/prooceedings www.mmlrrr..ss.. prwss/ Taress/v8//kae v80/kim180 html. 1, 4.3, 2, 5, D.1, D.2, 5, E.3, E.4, 6,
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014. URL / www.eeeeeoooummdeeeeeeeeee fd2 9F2d. http://arxiv orwoorgg////// /tt///b///1122 6114. E.3
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In International Conference on Machine Learning, 2020. URL
httpp::/prooeeddnnss www.smmr..r.ess/.e.. porees//v1g//ooleeeeeeeeee
html. D.1
Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. MIT press, 2009. URL https::/mittrrssstssst mit. edu addu//7862220111122// D.1
Solomon Kullback and Richard A. Leibler. On information and sufficiency. The Annals of
Mathematical Statistics, 779-),79-86, 1951. URL https://doi. org 1000 1214/211/22 aoms /1177729694. https: www. jwwor.ssor..... wtrg//tabble22366g5 D.2
Henry Kvinge, Tegan Emerson, Grayson Jorgenson, Scott Vasquez, Timothy Doster, and Jesse Lew. In what ways are deep neural networks invariant and how should we measure this? In Neural Information Processing Systems, 2022. URL Dohn3kMHw. 1, D.2
net/forum?id=$
William Lawvere. Adjointness in foundations. Dialectica, pages 281-296, 1969. URL 10. h https: : wtt.//ii..//// /tt/:/.0000....1.......6.6......
1746-3361 1969. tbo1194......... x. http:
tac www. /ttp://ww....cccccc/o/ittttttttt
B.10
F. William Lawvere. Metric spaces, generaazzege logic, and closed categories. Rendiconti
Rendicoinn del seminario
matématico e fisico di Milano, 43:135-166, 1973. URL Utthttpp::/oo...or//ooo.0 BF
924844. http:// /www. tac www.mtaaammmm http:///ww....ttc/////t/tttttttt html.
A.3, A.3, B.1, B.9, D.2
F. William Lawvere. Taking categories seriously. Revista colombiana muntemttca 2003-44)ccc 178, 1986. http:://www..ttttttaaaaaaaaaaaap http://www..ccc..c.crrrrrrrttttt html. 4.1
William Lawvere and Robert Rosebrugh. Sets for Mathematics. Cambridge University Press, 2003.
URL https:://oi.ono org./.o.oo//ooooooooooooooooo 10.0 /ttg://...000/00077/311155555550 2.1, A.2
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee whye Teh. Set transformer: A framework for attention-based permutation--ivvariatt neural networks. In International Conference on Machine Learning, 2019. URL URL.http:/prroceeeings.m ess/ ess/v97/lee1gg D.1
Tom Leinster. An informal introduction to topos theory. arXiv preprint, 2010. URL https:
// //arxiv.ooro org. / 11t... 5647. A.2
Tom Leinster. Basic Category Theory. Cambridge University Press, 2014. URL https://d/// org/ 10. 1017 111111/C0998111070000000000
http://aavvvvvrraa////////// 09375. A.1
16

---

Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progressive learning and disentanglement of hierarchical representations. In International Conference on Learning Representations, 2020. URL http::////ppeeeveeeeeeeefrrrorse rYPS. 1, D.1
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairesss of disentangled representations. In Neural Information Processing Systems, 2019a. URL https:h/poocceeediiiii 3667755188ebe8888444446666o0b--3 Abb Att. html. D.1
newrips. cc/ wtt.urpps.cpppppe//oo.ao//// pppppr//tgghhahh//aaaap
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference Machine Learning, 2019b. URL mlr https://proceediin wwt.ssswwr..r..reeeeeeeeemeeeeee gttess/vv7/Locatte www.aateelllllaaaeeeee. html. D.I, 14, E.3
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervise disentanglement without compromises. In International Conference on Machine Learning, 2020. URL http::// /tttt:://rrre http::////cceeeoesseeeee mlr. press/vv19/Locatel lo20a html. D.2
Jan Eukasiewicz. O logice trójwartošciowej (On three-valued logic). In Selected Works, volume 11 of Studies Logic, pages 87-88. North-Holland Publishing Company, 1920. D.2
Jan Eukasiewicz and Alfred Tarski. Untersuchungen über den Aussagenkalkül (Investigations on the propositional calculus). Comptes Rendus des Séances de la Société des Sciences et des Lettres de Varsovie, Class III, 23, 1930. D.2
Saunders Mac Lane. Categories for the Working Mathematician, Springer, 1978. URL https: //doi. org/ 10. 1007 997 1-77----4777-72111
1, A.1, A.1
Saunders Mac Lane and leke Moerdijk. Sheaves in Geometry and Logic: A First Introduction to Topos Theory. Springer, 1994. URL htthttt////dooorg//// 10. 1007 9 4412-662--000 0927 A.2
Louis Mahon, Lei Shah, and Thomas Lukasiewicz. Correcting flaws in common disentanglement metrics. arXiv preprint, 2023. URL
http///axiv.orras/////////////// 02335. D.7
Gregorz Malinowski. Many-valued logic and its philosophy. In Handbook of the History Logic, volume 8, pages 13-94. Elsevier, 2007. URL wwtthttt///hi...o org/ www.ooog..o..ooooooooooooo 1016/ S187/-8884555 585 707 ((o7)8000) 88004--0--- D.2
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. DeepProbL Neural probabilistic logic programming. In Neural Information Processing Systems, 2018. URL https: :/tp:p:::////ee http:////ooeeeeeeeeeeeeee neurips httppps.ccpppppppp hctp//aper/oo////aaa/eeeeteeeeee hccpaaahh//ccdddcceeee 1862233eed7006540bsbaaaaa
html. D.2
Radu Mardare, Prakash Panangaden, and Gordon Plotkin. Quantitative algebraic reasoning. In Logic in Computer Science, 2016. URL httht::/dd/o.. orwoorgg///oo. 10. /135//143//32222222 2934518. D.2
Radu Mardare, Prakash Panangaden, and Gordon Plotkin. Fixed-points for quantitative equational logics. In Logic in Computer Science, 2021. URL 264. 2021. 94 70662. D.2
1109/LICS52
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference Learning Representations, 2019. URL htlf/forumiddyy/e
D.I
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dSprites: Disentanglement testing sprites dataset, 2017. https: //githttbbb..... htt.hhu..ooomeaccoa wttp://ccc//eeeeeeeeeeeeeee (Apache License 2.0). 6, 5, E.3, 7,9
dataset
Barry Mazur. When is one thing equal to some other thing? Proof and Other Dilemmas:
Mathematics and Philosophy, page 221 221-242. Mathematical Association of America, 2008. URL http::///o..orr//...ggu//////// ott.:///............////........ 015. https: attps://peopla htwppppppppeeeeeeeeeeeoo math. servarddedd/-mzuu//rrerrrrr Baavarr..eu//mnuuuur//eeenntnttn is pdf. 3.1
17

---

Peter Meer, Doron Mintz, Azriel Rosenfeld, and Dong Yoon Kim. Robust regression methods for computer vision: A review. inteemaaiionalluuue of computer vision, 6:59-70, 1991. URL
https://doi.org orgoorgg////o 10.. 107/FFF022200. http55///0oo..org///..111//61211 4.1
Nimrod Megiddo. The weighted Euclidean 1-center problem. Mathematics of Operations Research, ((4):498-504, 1983. URL http:::://o..ooo//////. /t..://.////...////////./ ..arr....4 498..
Stanislav Minsker. Geometric median and robust estimatiin in Banach spaces. Bernouli,, 21(4): 2308-2335, 2015. URL
14666665. 4.1
Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. In Neural Information Processing Systems, 2022. URL https:
wwweenreview.ne//ooo www.u//dddbiiikuu////
D.1
Milton L. Montero, Jeffrey Bowers, Rui Ponte Costa, Casimir JH Ludwig, and Gaurav Malhotra. Lost in latent space: Examining failures of disentangled models at combinatorial generalisation. In Neural Information Processing Systems, 2022. URL https: /tttttt//ppeereiiieee /ttp////eeeeeeeeeeeeeeeeeeeeeeee wwt/
d=7yUxTNWyQGf. D.7
Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2021. URL https: httt::::/opprreviieee http://wwweeet/oooooddddiiieeeoe D.I, D.7
Christopher Mulvey, &. Supplemento ai Rendiconti del Circolo Matemàtico di Palermo. Serie II, 12:99-104, 1986. URL
www.oogg///.. 46065. B.9
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In European Conference on Computer Vision, pages 681-699, 2020. URL https: http://ooo..oggo..o 1007/ 978- --000-030--0 -00-0330-55555-5 58595-25444 4.3
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference Machine Learning, 2023. URL https: //pp://////cccengg/// mlr, prwsss/vee/nnavon// html. D.1
Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On the calibration of multiclass classification with rejection. In Neural Information Processing Systems, 2019. URL https: http:///poooeeenn..eeeee wwwninuuipp... pss. CC wwtppps.cccc.p..p.peeep http:///wooo///aas///aaaaaaaaaa f65b643d0003bf4- www.eeoooooob--bbaaaaaaaaaa html. D.2
Matthew Painter, Adam Prugel-Bennee and Jonathon Hare. Linear disentangled representations and unsupervised action estimation. In Neural Information Processing Systems, 2020. URL neuri ps CC/
https://proceedings wwweeeiiiinnnnuuurrreeee acdb925892f68fb-22
www.rrttbbbtttttt...m html. html. D.1
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high- performance deep learning library. In Neural Information Processing Systems, 2019. URL https://proceeding/ nwwrips...... CC httpc/pppp/////// http:///19/ghhhhh/a// wwwccaa8feeeeeeeeeeeeeeeeee 0tw.77/122277740o00brrrree html. httpt:m//tttttsssstttt/oeeee
org. D.6, E.3
Edward Pearce-Crump. Categorification of group equivariant neural networks. arXiv preprint, 2023.
URL https::/arxiv..tttt /br0g// 14144. A.1
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 201 URL httph: httpps://mmlrr wwrgppapers/vv.... egosalia. html. t
wtt.//iiiiteeeanorrrrr.. org. E.4, E.5
18

---

Paolo Perrone. Markov categories and entropy. IEEE Transactions on Information Theory, 2023. URL https::tp//. pectts://doee //doi. org/ 10/g///.///0 1109/ TIT 2023 3328825.. https://arxiv.ort org/ abs/2212. 11719. A.1, D.2
David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière. Disentangling by subspace diffusion. In Neural Information Processing Systems, 2020. URL /a/////aaa/////a//
neurips. wwspradddddbbbtttt....
cC/ 2020/ hash/ mash/c9fool ml. D.I
Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70:1142-1154, 2022. URL https: //doi. orgiiorg/100 10.
3153135. 4.1, 13
Jean-Eric Pin. Tropical semirings. In Idempotency, pages 50-69, Cambridge University Press, 1998. URL htlhhtt::///e //tpii://diiio orgo 10. 1011/ 10..10177/30996655112222
004. B.7
Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. In Neural Information Processing Systems, 2020. URL htwp://ppppoeednnnn.......sssss nww.ips..pp... CC/ pwppr//ppppeppe http/////aaa/eeeggg111111
20codd5ad60222255449 awwpaedsssss--bbaaattt.t.....a.. Abstract. html. D.1
Scott E Reed, Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. In Neural Information Processing Systems, 2015, URL https::/ppooceednnneeee
couurpp..c/pppr hat.://77777770---aaaaattt...... Abwtract. 6, 5,
/r///o/5//hh/////233333353221223 7,8
Mark D. Reid and Robert C. Williamson. Composite binary losses. Journal of Machine Learning Research, 1108)):2377-2422, 2010. URL http::/jmlr..... hrrpp//r///rrr////////////// http:/pppr/////eiiiteeeee ml. D.2
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. International Conference on Machine Learning, 2015. URL https://procceddng.. .wr..ng.sm.................... pt...p.eses.//////
rezende15. html. D.1
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddinngs with the f-statistic loss. In Neural Information Processing Systems, 2018. URL htts neur
html. 2, 2.2, 4.3, 4.3, D.1, D.2, E.2
Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. In International Conference on Learning Representations, 2023. URL httt::///peenrveeeeeene/rrroroo hpQiGiX. D.7
Bernhard Schölkopf and Julius von Kügelgen. From statistical to causal learning. In International Congress of Mathematicians. EMS Press, 2022. URL https w/tp///..//oo orw.orgg///oo /0rg/1..r 4171/icma 022/173 173. https::////m httt::://arxi..orrt http:/ab//22...022/
D.1
Prithviraj Sen, Breno WSR de Carvalho, Ryan Riegel, and Alexander Gray. Neuro-symbolic inductive logic programming with logical neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022. URL htt.t:t/t/ooii.... orwoorgg////ooo 10. 1609 aaai. v3618. 23618.20795. 20795. D.2
Dan Shiebler, Bruno Gavranović, and Paul Wilson. Category theory in machine learning. arXiv preprint, 2021. URL https://arxis wwt.l/.ogggab/////// hts//bb/220066 07032. A.I
Daniel Shiebler. Compositionality and Functorial Invariants in Machine Learning. PhD thesis, University of Oxford, 2023. URL http:://do.org::
/2w0//oddeiia//nn DE1aDx4ZZ
A.I
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. International Conference on Learning Representations, 2020. URL httpts/://pppeeeeeeeeeee/rrrrrre
1, D.2, E.2
19

---

Joseph Sill. Monotonic networks. In Neural Information Processing Systems, 1997. URL
https://proceedingg www.neiinnnnnnnnnnnneeeeum peeurips.cc/papeee.eeeeeeee..epe 467777ce-2558fe55557--tt1t httpr//s.../pppee///////////aaa/
html. (imml.D. D.1
Ingo Steinwart. How to compare different loss functions and their risks. Constructive Approximation, -000000/00655-000 00003366-0000066-----00-- D.2
26:225-285 287, 2007. URL
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning, 2019, URL http://proceedingg .os.smmm.. www.mm...ess//p..
suter19a. 1tml.
Seiya Tokui and Issei Sato. Disentanglement analysis with partial information decomposition. In International Conference on Learning Representations, 2022. URL https://oppereeiiww et/forum? -dnedrddd-pdddeeee HVvGtt. 1, D.1
Loek Tonnaer, Luis A Pérez Rey, Vlado Menkovski, Mike Holenderski, and Jacobus W Portegies. Quantifying and learning linear symmetry-based disentanglement. In International Conference on Machine Learning, 2022. URL httphh::////peeeeeeeeee er22a. html. D.1
mlr. wwwps/eessvvaeeee verees/vv62// 1it///on222/oa tonna
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International Conference Machine Learning, 2021. URL https: //proceedings. .ml..mr....... www.mmmlr..r.....mmm. wwwpprrsssvvveee hatps///vv///truubeeeeee ttttttrrullleeeee html. D.7, D.7
Todd Trimble. An elementary approach to elementary topos theory, 2019. URL https: httpddtimmmmeepuisseeeeeel aet.eeeedd/eeeeeeeeettta------
//ncatlab. www.or///ggooiiii elementary theor. A.2
Alexey A. Tuzhilin. Who invented the Hiromov-Hausdorf distance? arXiv preprint, 2016. URL
https: /aasss///axvv... wrw.orgg////oo abs//bb///// 1612 00728. 4.1
Elise van der Pol, Herke van Hoof, Frans Olichock, and Max Welling. Multi-agent MDP homomorphic networks. In International Conference Learning Representations, 2022. URL
https://opeereviie net
D.I
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser. Jonathan Bright, Stéfan J. der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Îlhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, António Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature methods, 27(3)2261-272, 2020. URL https: org 10. 1008/ 011-59992-19---6 019-068 org. E.4 Kiri Wagstaff. Claire Cardie, Seth Rogers, and Stefan Schrödl. Constrained k-means clustering with background knowledge. In International Conference on Machine Learning, 2001. URL https: htt.///.aaaaa / acm. org /tt///ooooo//// 5.....55566666 50555555555500 6555699,.56 E.2
6-2.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, 2020. URL https::/proceeei/iee
mlr.
html. 1, 4.3
Endre Weiszfeld. Sur le point pour lequel la somme des distances de points donnés est minimum (on the point for which the sum of the distances to given points is minimum). Tohoku Mathematical Journal, First Series. 43:355-386, 1937. URL https:: :tttp:://onf //doi. org/ 10 1007 000007/511477 510479-041911 08 0352-2.2222
Emo Welzl. Smallest enclosing disks (balls and ellipsoids). In New Results and New Trendy in Computer Science, pages 359-370. Springer, 1991. URL. htt.ht:ss://oo/ org 10. 1007 /BFF b0038202. 4.1, 12
20

---

Zhenlin Xu, Mare Niethammer, and Colin A Raffel. Compositional generalization in unsupervised compositional representation learning: study on disentanglement and emergent language. In Neural Information Processing Systems, 2022. URL htt.httts:://pereviiiii net mattt/orumm orum?i d=ZEQ5Gf8DiD. D.1
Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Nanning Zheng. Towards building a group-based unsupervised representation disentanglement framework. In International Conference on Learning Representations, 2022. URL https://openrevieet http://wwwwwwneftteeeeeeeeeee www.et/forummio-yggp Nctmyd. D.1
Yang Yuan. On the power of foundation models. In International Conference on Machine Learning, 2023. URL https: wwwpt:tt/tttceeeeeeeeeeeeee ppwpeeeiiiggg.... www..ppressvvveee /aa.2//yyun2b..... html. A.1
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Neural Information Processing Systems, 2017. URL newrips...pp.. CC/ httppp/e//oot//aaaa///2222277//0 wea.odoofffaeee-ebtaaaa...c html. D.I
Sharon Zhang, Amit Moscovich, and Amit Singer. Product manifold learning. International Conference Artificial Intelligence and Statistics, 2021. URL htthtt://pproccdinngg mllr.
press/v130/ zhang21j. html. D.1
Yivan Zhang and Masashi Sugiyama. A category-theoretical meta-analysis of definitions of disentanglement. In International Conference Machine Learning, 2023. URL https: //proceedings. mlr. ar.r parrpprss// wwwps//ess... http///222//aaaa2/2/ zhang2aak html. 1, 4.2, A.1, D.1
Sharon Zhou, Eric Zelikman, Fred Lu, Andrew Y Ng, Gunnar Carlsson, and Stefano Ermon. Evaluating the disentanglement of deep generative models through manifold topology. In International Conference on Learning Representations, 2020. URL ew www.netfforum id= dd=djwSom4 D.1
21

---

Contents
1 Introduction
1
2
Logical definitions of disentangled representations
2
2.1
Informativeness: injectivity or retractability of a learning model
2.2
Modularity; product structure preserved by learning model
Enrichment: from logic to metric
3.1
From equality predicate strict premetric
3.2
From logical operation to quantitative operation
3.3
From compound predicate compound quantity
3,4
hSmmooommoo from metric to logic
4
Quantitative metrics of disentangled representations
4.1
Modularity metries via product approximation
4.2
Modularity metries via constancy
4.3
Informativeness metrics
5
Experiments
10
6
Conclusion
10
Bibliography
11
Preliminaries
24
A.I
Basic category theory
24
A.2
Elementary topos theory
25
A.3
Enriched category theory
25
B
Theory
26
B.1
Subobject quantifier and quantizer
B.2
Equality and premetric
................................
B.3
Preorder
28
B.4
Operation: algebra over the product endofunctor
29
B.5
Negation...
31
B.6
Conjunction
32
B.7
Disjunction
................................
34
B.8
Implication
35
B.9
Heyting algebra, quantale, and ordered semiring
36
B.10
Quantifier: algebra over the exponentiation endofunctor
37
B.11
Enrichment
B.12
Summary
39
Proofs
41
C.1 Product function
41
C.2 Constant curried function
Discussions
42
D.I Background
42
D.2
Related work
43
D.3
Implication and equivalence
45
D.4
Constant function
46
D.5 Rank of imperfect retrrstttoo
47
D.6 Implementation
48
D.7 Limitations
48
D.8
Broader impact
48
Experiments
49
E.I
Synthetic data
49
E.2
Weakly supervised modularity metrics
50
E.3
Evaluation of existing models image datasets
51
E,4
Kendall tau distance between metries
51
E.5
Computation time of metrics
52
E,6
Factor-wise modularity metrics
53
22

---

List of Figures
1
Disentangled representation learning
2
From predicates and logical operations to quantities and quantitative operations
3
Negation
31
4
Conjunction
32
5
Disjunction
32
6
Implication
32
7
Equivalence
32
8
Quantitative operations for implication
45
9
Quantitative operations for equivalence
45
10
Constancy metries...
46
11
Radius and variance
47
12
Rank of imperfect representations
47
13
Illustration of synthetic data
49
14
Entanglement of distribution
49
List of Tables
1
From logic to metric Supervised disentanglement metrics
2
10
3
Supervised modularity metrics
50
4
Weakly supervised modularity metrics
50
5
Supervised disentanglement metrics on image datasets
51
6
Average Kendall tau rank distances bewteen disentanglement metrics
52
7
Computation time (seconds) of supervised disentanglement metrics on image datasets
52
8
pactor-wise modularity metrics on 3D Cars [Reed et al., 2015]
53
9
Factor-wise modularity metries on dSprites [Matthey et al., 2017]
54
10
Factor-wise modularity metrics 3D Shapes [Burgess and Kim, 2018]
55
11
Factor-wise modularity metrics on MPI3D [Gondal et al., 2019]
56
23

---

A Preliminaries
In this paper, used abstract mathematical tools such category theory and topos theory develop a theory of the relationship between logical definitions and quantitative metrics. However, this level of abstraction may be unfamiliar even intimidating to some readers, and sometimes unnecessary for machine learning practitioners. Therefore, we have used only the most basic algebraic concepts, such homomorphism, the main text. For readers interested in the mathematical background, we provide a brief introduction to the basic categorical concepts in this section.
A.1 Basic category theory
Category theory is a branch of mathematics that studies mathematical structures in an abstract way. which is suitable for identifying and organizing common patterns across various fields of mathematics [Mac Lane, 1978, Adámek et al., 1990, Awodey, 2010]. It has found applications in many fields, including computer science [Barr and Wells, 1990], probability theory [Cho and Jacobs, 2019, Fritz, 2020, Perrone, 2023], and machine learning [de Haan et al., 2020, Shiebler et al., 2021, Cruttwell et al., 2022, Dudzik and Veličković, 2022, Shiebler, 2023, Yuan, 2023, Pearce-Crump, 2023, Chen et al., 2024, Gavranović et al., 2024].
The most fundamental concept is that of a category:
Definition 14. A category C = (Obj, Hom, o, id) consists of
collection Obj of objects, set Hom (A, B) of morphisms between objects, 10
a composition function o Hom(B, C) x Hom(A, (A, B) -+ Hom(A, C) for each triple of objects, and
an identity morphism i E Hom(A, A) for each object,
subject to
associativity: ( o h (go and identity: idg of = f id
A crucial example is the category Set of sets and functions. However, what is of particular interest not the category itself but its relationships with other categories. Building on the concepts of the functor and natural transformation, whose definitions are omitted here, we can develop tools to better understand the properties of a category.
Moreover, we can define objects in terms of their relations with other objects, employing what is known as universal construction. For example, a terminal object in a category is an object such that for any object A, there exists unique morphism e A - 1 to it, which we call a terminal morphism. In Sct, any set {+] with only one element a terminal object. Based on the concept of the terminal object, global element of an object B is defined to be a morphism b 1 + B from terminal object.
We write ba: A + B as an abbreviation for the constant morphism b co A Ed, 1 & with value b: 1 + B. These concepts will be used to develop our theory in Appendix B. Other important universal constructions include the product, pullback, and exponential. The concept of the product is of great importance in disentangled representation learning [Zhang and Sugiyama, 2023].
Regarding the pullback, we need to mention the following useful lemma:
Lemma 4 (Pullback lemma). Suppose that in the following commutative diagram, the right square is a pullback.
(31)
Then, the left square a pullback if and only if the outer rectangle is a pullback.
This lemma is usually left as an exercise in textbooks [c.g., Mac Lane, 1978, p. 72, Exercise 8, Leinster, 2014, Exercise 5.1.35 A proof can be found in Fong and Spivak [2019, Proposition 7.3]. We need to use this lemma to (de)compose pullbacks. As side note, we use the asterisk f"g to denote the pullback of g along f.
10 be precise, category whose morphisms sets is called locally small category.
24

---

A.2 Elementary topos theory
Topos theory studies categories that, in some sense, exhibit behavior akin the category of sets and functions [Lawvere and Rosebrugh, 2003]. Topos theory has found applications in geometry, topology, and logic [Mac Lane and Moerdijk, 1994, Johnstone, 2002, Leinster, 2010, Trimble, 2019]. In this work, we only explore its relation to logic.
To formally define topos, two essential concepts are those of the subobject and subobject classifier. A subobject of an object C' is simply a monomorphism b B - C' to the object C. The subobject classifier is defined follows:
Definition 15. In finitely complete category, subobject classifier is universal subobject T:1 - Ω such that for every subobject : B = C, there exists a unique morphism Xs, C + Q such that is pullback of T along Xt- The morphism Xs is called the classifying morphism of b.
B
1
(32)
T
X
Alteraatvell, we can state that
Proposition 5. A subobject classifier is precisely a terminal object in the category of monomorphisms and pullbacks.
Then, we can study the morphisms to the object Ω::
Definition 16. In a category with subobject classifier T > Q predicate on an object C is morphism p : C $2.
Based on Definition 15, we can state that subobjects of an object C are classified by predicates on C.
For example, in Set, subobjects subsets, function from singleton {*} two-element is a subobject classifier, which is usually denoted by T {*} + [T.L}, a predicate on a set C' is function p
[T, and subset precisely corresponds its indicator function.
Among various equivalent definitions of a topos, concise one is as follows:
Definition An elementary topos finitely complete and cartesian closed category with subobject classifier.
Despite its concise definition, a great number of logical structures can be derived from it, which will be explored in Appendix B.
A.3 Enriched category theory
Enriched category theory generalizes the concept of the category by replacing the sets of morphisms with objects in suitable category [Kelly, 1982]. It has been used better understand wide range of domains, from metric spaces [Lawvere, 1973] to language [Bradley et al., 2022].
Let us dive into the definition of an enriched category:
Definition 18. A category C (Obj, Hom, id) enriched in monoidal category (V, (,I)) consists of
collection Obj of objects. hom-object Hom (A, B) Objy between objects,
composition morphism Hom B, C') @ Hom(A. (A, B) + Hom(A, C", for each triple of objects, and identity element , / + (Amm(A, A) for each object,
subject associativity and identity.
Comparing Definitions 14 and 18, we can say that (locally small) category is a category enriched in the category Set of sets and functions. Enrichment way to describe the additional structures of morphisms and the properties that need to be respected by composition.
25

---

An example is a preorder, which can be seen as category enriched in the category of boolean values. Another example is Lawvere metric space [Lawvere, 1973], which is a with premetric that satisfies the triangle inequality. Note that the transitivity of preorder and the triangle inequality of Lawvere metric are descrihed by their composition morphisms, respectively.
In this work, we use enrichment to describe the association of set of morphisms with additional operations, such strict premetric and aggregators.
B
Theory
In this section, we detail the theory of converting logical definitions into their corresponding quantitative metries based on elementary topos theory and enriched category theory.
B.1 Subobject quantifier and quantizer
Since our main goal develop multi-valued (possibly continuous and differentiable) quantification of properties defined by certain type of logic, we begin with category E that has sufficient structures to allow the desired logical operations and build the quantification upon these structures.
Firstly, recall that subobject classifier Q, exists, is the representing object of the subobject functor Sube, such that subobject b B - C corresponds to a unique classifying morphism X C + Ω and an external operation on the Subg(C) of subobjects that natural in the object C corresponns to an internal operation.
For example, the intersection
Nc SCb x Subg(C) + Subg(C) (C)
(33)
corresponds a natural transformation
Homg(-,52) x Homp(=,90 = Homg(-,0))
(34)
which, because the hom-functor Home preserves limits, is isomorphic natural transformation
between hom-functors
Homg( S x ?)) + Homg(-,9)).
(35)
which, by the Yoneda lemma, isomorphic to an internal operation on the subobject classifier Q:
A : Ω. x Ω + Ω.
(36)
Note that the subobject classifier, the classifying morphisms, and those internal operations are determined uniquely up isomorphism. However, in order obtain multi-valued quantification, the requirement for uniqueness might be too restrictive. Thus, we propose to study weakened concept instead:
Definition 19. In finitely complete category, a subobject quantifier is subobject o: that for every subobject b: B - C, there exists at least one morphism @h, C -> V/ such that b is a pullback of along Φb.- The morphism d, called a quantifying morphism of b. the category has subobject classifier T - Ω the quantizer & W -> St of the subobject quantifier is the classifying morphism
such
B
(37)
V
Ω
X
More succinctly, we can state that (cf. Proposition 5)
Proposition 6. subobject quantifier a weakly terminal object the category of monomorphisms and pullbacks.
26

---

Thus, there is a unique morphism Xs C + I2 classifying subobject b B - C of an object C, but there could be multiple morphisms 0b C + v quantifying this subobject.
It is provable that the domain of a terminal object T in the category of monomorphisms and pullbacks (Proposition 5) must be terminal object in the category E, but not all weakly terminal objects in the category of monomorphisms and pullbacks (Proposition 6) are monomorphisms out of terminal object. We choose Definition 19 because we want only one global element o - V be designated to the truth value T - f2
Here, give two examples motivate this definition of subobject quantifier. One is related to three-valued logic [Bergmann, 2008, Fong and Spivak, 2019, Exercise 2.34]:
Example 9. In Set, the function
yes {+} + {no, maybe, yes),
(38)
which maps the element in singleton set {*} (a terminal object in Sct) to an element yes in a three-element set (no, maybe, yes} is a subobject quantifier. For any subset B of a set C, a quantifying morphism O; C - v is function that maps all elements in the subset B to yes and all other elements to either maybe or no.
The other is related to metric spaces [Lawvere, 1973] and will be our running example in the following subsections:
Example 10. In Set, the function 0: {*) + [0, oa] selecting the number 0 out of the set [0, oo] of extended non-negative real numbers is subobject quantifier. The quantizer is function
{
T
n= 0, m>> 0,
: [0, xc] + =
(39)
which maps to T and any non-zero number I..
Intuitively, with a subobject quantifier, there only one way be true, but there may be many ways to be false. In Set, quantizer maps multiple "degrees of fruth" from potentially large, even infinite set V to a smaller set of truth values, hence the name.
Next, we define the counterpart of the concept of predicate (Definition 16): Definition 20. In category with subobject quantifier o: V quantity object C' is morphism q : C - V.
Since we weakened the requirement for uniqueness, there is no one-to-one correspondence between subobjects and quantities. However, they are still related as follows:
Lemma 7. In category with subobject classifier T + Ω, subobject quantifier o: - V and a quantizer k : V + Q. a quantity q : C > on an object C is a quantifying morphism of a subobject of the object C, which is isomorphic to subobject (N o o) q)"T.
Proof. q" o and (N o T)"T are both subobjects of because pullbacks preserve subobjects. Their isomorphism follows from the pullback lemma.
O
Lemma 8. a category with subobject classifier T Ω2 a subobject quantifier o: o:1 V
and a quantizer k. " + Ω, a quantity q C V object C' is quantifying morphism of subobject b: B - C if and only if nog = Xb, where Xs. is the classifying morphism of the subobject b.
Proof. Necessity follows from the pullback lemma and the uniqueness of the classifying morphism: sufficiency follows from Lemma 7.
The following relationship between a quantity and the quantizer also useful:
Lemma 9. In category with subobject classifier T 1 - Q. a subobject quantifier o: - M and a quantizer
v + Qt for any quantity q : C- V on an object C, q Oc if and only if K o = K o OC = Tc. where og is the constant morphism o o ec C "c, 2 V with value o: V.
Proof. This is due to the universal property of pullback o of aalong K.
We can see that the hom-functor on the quantizer Homg(-,6) -Hompl v) = Homg( (-, - is a natural transformation which maps the quantities Homp v) on an object to the predicates Homg (C, ?) on C, which are precisely subobjects of C.
27

---

B.2 Equality and premetric
Next, we take closer look a conerete and important predicate - equality and its corresponding quantities.
Q2 the equality predicate -c: C C + Ω on an object C is the classifying morphism of the diagonal morphism
Definition 21. In category with subobject classifier :
De: : C C
C = (idg; idg).
By Lemma 8, quantity de C + v is quantifying morphism of A if and only o de c, depicted in the following diagram:
1
(40)
C
C
V
dc
In Set, we have the following definitions:
Definition 22. premetric on set C' is binary function de C C [0, oo) such that
Wc C. d)(c,c) =
(41)
Or equivalently, de a Ac = Oc, depicted in the following diagram:
C
Ae C
(42)
C
[0]
de
Definition 23. A strict premetric on a set C is premetric de C C - [0, such that
Ve E. Ve2 E C. (do(e1, = 0) - (c1 =o 62).
(43)
Or equivalently, Ag a pullback of 0 along de:
C
A0
(44)
CXC
|0,
de
In other words, strict premetrics precisely quantifying morphisms of the diagonal morphism Ac in the category Set with 0 {*} + jo, oo) as a subobject quantifier.
Note that the symmetry
Yc₁ E C. Ves E C. delc,, = dc(c2.
(45)
and the triangle inequality
Ve, E C. Ve, E C. Veg E E. dele, + dc((2, 2 dc(c),0));
(46)
which make de a metric, are not required. The addition + and the order 2 on the set [0, oz) are not needed to define strict premetric. However, they necessary for defining other operations and properties, which will be discussed the next subsection.
B.3 Preorder
There is a preorder Sc of inclusion defined on the set (C) of subobjects of an object C: for two subobjects a A C and b B C. So b if and only there exists a morphism - B such that bo f:
B
(47)
C
This preorder on the subobjects Subg (C) induces a preorder on the predicates Homp Q) via the isomorphism. We can generalize this construction and define preorder on other hom-sets:
28

---

Definition 24. In category E with pullbacks, the inclusion preorder So on the set (C') of subobjects of an object C' and a subobject m S T induce a preorder -/ on the hom-set
t Homg T) via pullback of m: for any two morphisms 12 C + T. h 2C J2 if and only if f.m m Sc m...
S
S
(48)
S m
v
C
>
Based on this definition, we can explore the preorders on any hom-sets. From now, we assume that E category with necessary structures that need.
Then, the preorder of predicates is 20 Homg(C, Rt,, and the preorder of quantities is 36 Homp (C, ). By Lemma 7, we know that for two quantities 91 92 C' + M 91 Ie 42 and only if (K 41) zc (x 92), which means that Homg x) is an order-preserving function from (Homg(C, V) = to (Homp (), ())
Next, we will explore the structures of Homg(C,s)) and Homg (C, v. To begin with, To is a top in Home Boom@(C, (C, 52), and oc is top in Hom (C, V) because idc is top in Subg
C
(49)
idg
o
V
Ω
OC
The inclusion preorder on Subg(C) also has a bottom - the initial morphism &c 0 - C from an initial object ( in the category E to the object C. Then, its classifying morphism 1c C is a bottom in Home (C, It can be proven that Le is a constant morphism with value 1: - Ω which is the classifying morphism of the initial/terminal morphism 0 1. Any quantifying morphism We : C + Vr of ig is bottom in Homp(C. (C, V). but it is not necessarily a constant morphism.
(50)
30
Ω
de
Le
The preorder the global elements plays a special role:
Example 11. In Sct, =I. also denoted by is a preorder on the set (T, with only one non- identity relation L.
Example 12. In Set, =i is preorder on the set [0, oo] where =i 0 for any number and =1
and =i m for any positive numbers and
By definition, (IT, and ((,, 20), =i) equivalent. However, ([0, xo), = e.g., the usual "greater than or equal to" 2 total order, further differentiate positive numbers. Note that 0 remains the top in this suborder ([0, oc),
consider suborder of
B.4 Operation: algebra over the product endofunctor
In elementary topos E, the subobjects Sube(C) of an object C' not only forms preorder by inclusion but also are equipped with certain set operations (e.g.. intersection and disjoint union), which are defined in terms of the universal properties of their corresponding order operations (e.g.. meet and join). Further, these operations are reflected in the structures of the subobject classifier (e.g.. conjunction and disjunction).
Here, we establish a link between the structures the subobject classifier and those of subobject quantifier. Our primary result is follows:
29

---

Theorem 10. Consider category with subobject classifier T - I2 subobject quantifier
o: D, and a quantizer
Q. Let n be a natural number. Let B : Ω2 -y Ω be an n-ary logical operation on Q and let a Np" - be n-ary quantitative operation Vp
For i {I....,TU n. let P; : C' St be a predicate on an object C, and let qi C - be a quantity on the object C such that P; = 9;, Let p (Pi.....Pr) be the tupling of the predicates, and let q = (gi..... qnl be the tupling of the quantities. Let b B + C. (3 T)"T be the subobject classified by B P. and let A C == (a q) o be the subobject quantified by q.
a
B
(51)
TT
V 0
idg
C
52
Then, we have
(i) R" oq p
(ii) (3oR K ca 0= = T 1+ (33 0a 0a TA)
x
(iii) (3ok"
-000
[Bok" 00
(iv) (Bok" = -oo - (xo = 80p)
(v) (3ok" -ooa (laog)ob= on)
For convenience, call n-ary operation Va - V (as algebra over the product endofunctor
(-)") homomorphic to 3 Ω + Ω via a morphism k: y + if
8 KM
(52)
and subhomomorphic Ω 52 if satisfies the condition
3 K" o a' oao== T
(53)
Proof. (i) follows from the property of tupling and product: x (k (............................... .....,,k (P.... ,p....,P.?? p.
gn)
(ii) states that if subhomomorphic to 3, then 3 p the classifying morphism of the subobject quantified by a q. 8opoa
(54)
=Bok" oqoa
(i) (55)
=Bon" x a 00
(pullback) (56)
=T,'1 (a o)'²
(subhomomorphism) (57)
= =TA
(composition) (58)
(iii) means that a being homomorphic to 3 stronger condition than being merely subhomomorphic to 3.
Bok" oao
(59)
=oaaaao =00000aa
(homomorphism) (60)
(pullback) (61)
(composition) (62)
(iv) shows the relationship between the predicate 3 oop and the quantity o q when a is homomorphic
3.
koaog
(63)
=Bok" og
(homomorphism)
(64)
= B op
ii) (65)
30

---

Figure 3: Negation
(v) means that a is homomorphic to 3, then a o g is quantifying morphism of b.
koaoqob
(66)
bpopob
(iv) (67)
=TB
(pullback) (68)
o o = 0g follows from Lemma 9.
In summary, if is subhomomorphic to 3, then is included in b ((i)); if is homomorphic 8, then a and b are isomorphic ((ii) and (v)). We consider this weaker condition because subhomomorphic but non- non-homomen non-homomorphic operations may exhibit favorable properties in other aspects, such continuity. We will discuss several concrete examples in the following subsections.
B.5 Negation
First, let take closer look at unary logical operation negation - Ω - Ω. which is defined as the classifying morphism of 1. Q. Recall that is the classifying morphism -
Let us consider a unary quantitative operation ~ - V. If ~ is homomorphic to - via the quantizer K, means that x ~,, the following diagram commutes:
(69)
Let us consider the set [0, oo] in Set. A quantitative operation ~ [0, oc] + [0, oo] homomorphic to
the negation -: - [ {T, 1T is function
no n 0. 0
(n) == 0 nop
(70)
n > 0.
which maps non-zero number n4 and any non-zero number to 0. However, this function is
discontinuous at O. On the other hand, if we consider quantitative operation subhomomorphic 0, or, by
the negation -, then the only requirement that for all n. ~(n) 0 implies
contraposition, ~() 0. Continuous choices include the hinge function >> 1 max (1-n1, 0}. reciprocal function J. and exponential decay function
(see Fig, 3). Note that the latter
"H] (T. oo) 1.
31

---

muxio, (cartesion)
(socantesin))
Jople" + e1
logle (erlly
Figure 4: Conjunction
Figure 5: Disjunction
[s x (rdj to max)
ja
[ad). sum)
mind Mal minis 6. m (1 60)
mini 0,
Figure Implication
Figure 7: Equivalence
two are actually homomorphic to the constant false 1 because their outputs are always non-zero. The hinge function 1 as discussed later, can be seen as derived from the implication p + 1.
In this way, if we have quantity for predicate of the predicate as well. If the quantitative operation ~ is subhomomorphic but not homomorphic to the logical operation vice versa.
obtain quantity <-g for the negation -p
guarantee that for any x, (α(x) 0 implies -p(x) T, but not
B.6 Conjunction
Next, let move on to important binary logical operation conjunction A Ω Ω - Ω which defined as the classifying morphism of (T, ~ Ω2 x Q. Similarly, we consider binary quantitative operation V
x v + T homomorphic to the conjunction A via the quantizer
(71)
RXK
32

---

By abuse of notation, the conjunction A also denotes binary operation on the set Homg (C, Q2) of predicates. such that for any two predicates P1 P2 Homp(C,S2).
P1 A P2 == A o (2))
(72)
The same goes for the quantitative operation ( For the conjunction, we can prove stronger result:
Theorem 11. Consider category with subobject classifier T : - Q2 subobject quantifier 0:1- - M, and a quantizer & : + Ω.
Let the conjunction A Q x Ω - Ω be the classifying morphism of T): : - Q x Q and let : + v be binary operation on homomorphic the conjunction A via the quantizer
Let P1.P3 C Q be two predicates on an object C, and let 91 42 +C v be two quantities on the object C such that P1 o 91 and P2 92. Let (P1, (p)) be the pairing of the predicates, and let = (41 92) be the pairing of the quantities. Let b B ~ C == (P1 A p2)" be the subobject classified by P1 A Pz-
B
1
1
b
(0.0)
C
V x W
V
(73)
An)
ide
AXN
8
C
Ω 52
(p))Pa)
Then, 83 is a quantifying morphism of (o, o), and is a pullback of (o, o) along (2,42))
Proof. If q1 &2 = oc, then (k= ( Tc. which leads to (KE x xO = (T, T) oec = (Tc, Tc) due to the universal property of pullback. Then, we have K 91
92 TC due
to the universal property of pairing, and consequently 91 = 4 = oe according to Lemma 9, i.c. (q1.%)) (2) (o, ec. Therefore, (0,0) o) is pullback of o along 8. According to Theorem 10, b is a pullback of o along qı @ Q2. Then, following the pullback lemma, b is a pullback of (o, o) along (4, 92).
In other words, the pullback square symbols in Eq. (73) are unambiguous the top row, the left column, the right column, the top-left square, and the top-right square are all pullbacks.
Note that for any quantities a, b, : C V, we have
((a b) 0 c) = K (a O)),
(74)
R (a 2 b) = K (b 2 a),
(75)
Ko (oc X a) K a,
(76)
due to the associativity, commutativity, and unitality the conjunction, but the quantitative operation & is not required to satisfy these properties, i.e., it is possible that
(a 2 b) + a c),
(77)
a 2 b + 6 a,
(78)
Og X +
(79)
However, in the following examples, we mainly consider quantitative operations & such that these properties are satisfied. In such cases, (Homæ ) 8, oc) forms a commutative monoid, and consequently Homp x) is a monoid homomorphism from it to (Homp(C,Q), A, Tc).
In Appendix B.3, we introduced preorder structures on the predicates Homg (C.2) and the quantities Homg (C,, Vr The conjunction A is a commutative monoidal structure compatible with the preorder (Homg (C, .2). =t) because it the meet operation. For the set Homg v of quantities, we can choose the quantitative operation @ to be the meet operation as well. Alternatively, we can only require it to compatible with the preorder, in the sense that the monoid product is order-preserving: for any quantities 41, Q1 92, E Hommg V. if 4
<e di and 3C U2 then oj @ 92 Sc gi & 42- In other words, we require (Homg(C, V), _c 68, Oc) to symmetric monoidal preorder [Fong and Spivak, 2019, Definition 2.2].
33

---

Example 13. Let us consider two commutative monoidal structures on the preorder ((O, o0], 2). The max operation max: [0, oo] x [0, so] [0, oo] is the meet, i.e., cartesian product, while the addition +: [0, oo] x [0, oo] [0, 2o] is monoidal product. They are both semicartesian because the top 0 the unit.
Note that (00, oo], +. +,0), (I0, 1], x, 1), and ([1, o0]. x, 1) are isomorphic to each other with the following isomorphisms:
(10, oo]. +.
-ogg(s)
xx)(s)
(80)
expl
([0, 11, x,
([1, o0]; x,
We can also induce monoidal structure on set if it is isomorphic to monoid: Lemma 12. Let f A = B: g be a pair of bijections between sets and B. (B, @n, In) is monoid, then (A, 0 := 2n o (S x f), == g o IB) is also a monoid.
Proof. Associativity:
@ b) A
(81)
=[((f(a) @n f(b)) @B f(c))
(82)
= =9(((a) 8n (f(b) 8n f(c)))
(83)
@ cb)
(84)
Left unitality:
IA A
(85)
(I[nn) 0
(86)
= =g(f(o(La)) 0B f(a))
(87)
9 @B f(a)) g({(a))
(88)
(89)
(90)
Right unitality can be proven similarly.
In this way, can obtain richer choice of monoidal structures the set [0, oo] beyond the addition (see Fig. 4): Example 14 (Semicartesian monoidal product).
1 ((0, 00), 8, 0) ((0, oo), +) log(1 + x)
@ log(e" " +e-
(91)
2 : (10, 00) 0) = (0), +, v/
@ Va² +b² @
(92)
Ve 00) 8, 0) (0), ()) =2
2vab
(93)
+ ((0, 2o), 8, 0) ((1, oo), x, :
2 +
(94)
In summary, the max operation on [0, oo] can be regarded as a continuous conjunction, whereas semicartesian monoidal product, such the addition, be viewed soft
B.7 Disjunction
Dually, the disjunction Similarly, we want to find a quantitative operation t : & x V + T homomorphic to the disjunction via the quantizer k:
Ω + Ω reflects the join of the inclusion preorder of subobjects.
(95)
xXX
Ω
Using the same technique as Lemma 12, we can obtain several monoidal structures on the set [0, oo] homomorphic to the disjunction (see Fig. 5):
34

---

Example 15 (Semicocartesiaa monoidal product).
ob
((0, oo., aB, oo) ب (10, oo], +. +)
① 6: =
(96)
a a+ b
ab
2 ((0, oo], 8 00) = (O0, oo), +.
a b
(97)
Va +
1- : (00, 00 8 x0) = (00, 11, x, loog 1 x)
a log "M + -e-
(98)
tanh: (10, 00), 3 x)) = ((0, 11. x, arrtanh
+ b = arctanh(tanh(a) tanh(b))
(99)
However, usually choose the quantitative operation @ to be the join operation min of the preorder (00, xo, ). In this way, & distributes over {B, i.e., for any quantities a, c, C - AD, we have
(b + c) = (a & b) (a 8 c).
(100)
typical example is the plus semiring ((0, oo), min, +) [Pin, 1998].
B.8 Implication
Finally, let us construct quantitative counterpart of the implication + x Ω2 - Q, which right adjoint the conjunction A. For global elements a, b, 22 this means that
c a - b if and only if c b.
(101)
There are two ways to construct quantitative operation - v V + V corresponding to the implication -. One way is to find a quantitative operation homomorphic the implication + via the quantizer K:
V
(102)
x K
Ω
Ω
Example 16. For the set 00, oo]. a quantitative operation [0, o0) (0, oc] - [0, oo) homomorphic + {T, is function
to the implication :T [T.1)
((b) = (0 [S))
a a= and O, otherwise,
(a. + aa
(103)
where f: (0, oo] + (0, oo] is an arbitrary function non-zero numbers. Note that this function is and > 0 (cf. Appendix B.5).
discontinuous at the line
The other way is to find quantitative operation right adjoint to operation & homomorphic the conjunction A, i.e., the internal hom of the monoidal closed preorder [Fong and Spivak, 2019, Definition 2.79].
Example 17. For the meet-semilattice (10, oc]. 2, max, 0), the function
0 a 2 b b a < b.
(a, + la < b) x
(104)
is right adjoint to the max because
maxfc, a} 2 and only if 2 b a
(105)
While this function is subhomomorphic to the implication, is still discontinuous at the line a = b. Example 18. For the monoidal preorder ([0, oo], 2, +, 0), the truncated subtraction (a.k.a. monus [Amer, 1984])
0 2 b. 6 a a b
- max - a,
(106)
35

---

is right adjoint to the addition because
c if and only if 2 20b
(107)
The truncated subtraction is continuous and subhomomorphic to the implication. Note that the hinge function ++ max 0} = 1 = for the negation can be interpreted as the quantitative operation derived from -n = n 1, where is homomorphic to the constant false L.
Similarly Lemma 12, if symmetric monoidal preorders are isomorphic and one of them is closed, we can induce that the other is also closed:
be pair of isomorphisms between symmetric monoidal preorders (A, A. @A. IA) and (B. 3n. &n In). (B. 3n. &n, In. Ia...nn.pp -o ) closed, then (A, 3A
Lemma 13.
Let f A = B
A. IA. = 9o = (f f)) is also closed.
Proof. For all a, c, A, we have
(108)
=c te>gs(f(a) -B f(b))
(109)
=f(c) 3B f(a) 8 f(b)
(110)
=((c) 0n f(a) in f(b)
(111)
=c A b
(112)
This means that A is right adjoint to A.
In this way, we can find the internal homs corresponding to the monoidal products discussed in Appendix B.6 (see Fig. 6).
As side note, we can use the quantitative operations discussed above define quantitative operations for other logical connectives. For example, the logical equivalence <> b can be represented as (a + 6) A (b - a) (bi-implication), (-u V b) (-b a) (conjunctive normal form (CNF)), or (a A b) V (~a A -b) (disjunctive normal form (DNF)), and its quantitative operations can be defined accordingly. Some examples are shown in Fig.
B.9 Heyting algebra, quantale, and ordered semiring
Now, having constructed the logical operations and their corresponding quantitative operations, we compare the structures of the subobject classifier with those of subobject quantifier.
known that the global elements of the subobject classifier with the logical operations form Heyting algebra (S,,, T, L,,, +)):
Definition 25. A Heyting algebra is a cartesian closed bounded lattice.
In categorical terms, T is the terminal object, 1 is the initial object, A is the product, V is the coproduct, and + the exponential in the preorder (2, (as a thin category).
We weakened the requirements to construct the algebraic structures the subobject quantifier, which usually forms what is called a quantale 11 1, 0, @, B -o) [Mulvey, 1986, Dudzik, 2017]:
Definition 26. A (unital) quantale is monoidal closed suplattice.
This means that we can consider preorder (V= on the subobject quantifier as thin category, where the terminal object, is the initial object, @ is a monoidal product and not necessarily the product, is still the coproduct, and -o is the internal hom right adjoint to the monoidal product 8. An example the Lawvere quantale (10, oo], 2, O, oo, +, min, -) [Lawvere, 1973, Bacci et al., 2023]. Then, the quantizer V - St is homomorphism preserving some all the structures.
Note that due to the adjoint functor theorem and the fact that left adjoints preserve colimits, the product distributes over the coproduct in Heyting algebra, and the monoidal product distributes over the coproduct in a quantale. We can further relax the requirement for @ to be the coproduct and instead consider monoidal product (Appendix B.7). If we still require the distributivity for the two monoidal structures and (), the algebraic structure an ordered semiring (Fujii, 2023]. Further investigation is left for future work.
36

---

B.10 Quantifier: algebra over the exponentiation endofunctor
Up this point, our focus has been on n-ary operations propositional logic (Appendix B.4). Next, we introduce the universal quantification A and existential quantification E used in predicate logic and their quantitative counterparts.
Externally, the universal quantification and the existential quantification are right and left adjoint to the puilback of projection, respectively [Lawwere, 1969]. Internally, the universal quantifier VD .2 + f and existential quantifier än Ω2 92 are given by morphisms from the power object s2? of object D the subobject classifier $2.
Recall that the power object s2" is also an exponential object into the subobject classifier Ω2 so the universal quantifier VD and existential quantifier 3p can also be viewed as algebras over the exponentiation endofunctor (-) of exponentiation on the subobject classifier Ω. Then, we can consider algebras on the subobject quantifier homomorphic to them, which serve as quantitative counterparts of these quantifiers.
Our main result as follows (cf. Theorem 10):
Theorem 14, Consider an elementary topos with a subobject classifter - Ω2 a subobject quantifier 1 > V, and quantizer Vp f2.
Let p C D + Qt be predicate on a product, and let q C C - V be a quantity such that p p= nO q. Let p: C + Ω and q: C u be their exponential transposes.
Let BD :ΩP Ω2 be predicate, and let ap D - T be quantity homomorphic to BD via the quantizer K, i.e. 3n o K = K koap a.p
Letb: B = pp T be the subobject classified by D and let a A - C' = (a be the subobject quantified by ap o q.
A
B
(113)
6
T
C
D
q
0
C
Ω
0
3p
Then, we have
(i)
K o =
D
(ii) K0 (ap Q 3n T
(ii) (pp
oa T
(iv) (a o ob = OB
Proof. (i) follows from the property of exponential.
(ii) shows the relationship between the predicate Bn P and the quantity on when op is
homomorphic to DD-
koap q
(114)
= PP q
(homomorphism)
(115)
=ppor o
(i) (116)
(iii) means that 8n f a classifying morphism of a.
3D opopa
(117)
=KOOp o a
(ii) (118)
=KOOA
(pullback) (119)
TA
(composition) (120)
37

---

(iv) means that ap @ is quantifying morphism of b.
koapoqob
(121)
=
(ii)
(122)
=TB
(pullback) (123)
op o b 0g follows from Lemma 9.
Definition 27 (Universal aggregator). universal aggregator V vp +
a quantity that is
homomorphic to the universal quantifier Vp 22" + Ω:
P
V
(124)
Ω
D
Ω
Definition 28 (Existential aggregator). An existential aggregator 4 DD V is quantity that is homomorphic the existential quantifier 3D
_
Ω:
V
(125)
Ω
Ω
3D
Example 19. For the set [0, oc]. the canonical choices of universal aggregator Vp and existential aggregator DD are sup and inf. If the set D finite, we can also use sum and mean as the universal aggregator. Non-examples of universal aggregator include median and mode, which are not homomorphic to the universal quantifier.
Note that the universal quantifier and existential quantifier are commutative up isomorphism:
Ax = VB v
V v YxxA
(126)
A = 3 o
R ~ 3 3
(127)
However, we are free to choose different aggregators for different objects that are not necessarily commutative. For example, the sum of max is usually not equal to the max of sum.
B.11 Enrichment
Lastly, we describe the conversion based on enrichment.
First, let us define the enriching category:
Definition 29. Let (V, 3, @,0.-1 an internal quantale object in Set. We define V-Set to be category whose objects are tuples consisting of a set C, a u-valued strict premetric de on C, and universal and existential aggregators C:
(C, C V.Vc, C + 1).
(128)
and morphisms from (A, A 4 to (B, d.. Ag) are functions f A B. Definition 30. A bifunctor X on S-t is given by the Cartesian product of sets and functions, together with the following products strict premetrics and aggregators:
d A dp (AA
(A
B) = (Ax A) B de Raxdg
x
0 V.
(129)
V V A AxB
A B
D SB v, B
(130)
A X An A AxB ~ AB B 62
4n V.
(131)
Proposition 15. The product d A dd of strict premetrics given Definition 30 is again a strict premetric.
38

---

Proof. This is a result of Theorem 11. Consider the following diagram:
A
A B
A B
1
AAXB
AxxDB
(0,0)
(132)
(Ax B)
1 x B² d xdp
Vp 1
A x AB is a pullback of o along @ (d x dp) because (o, o) is a pullback of o along & according
to Theorem 11, A A Ag is pullback of (o, o) along d d.. and
apply the pullback
lemma. A x Ag is isomorphic to AXBB which means that @ o (d x dg) is a strict premetric. D
Based on this definition,
show that
(d K Bde (dgBdc)) 8dc).
(133)
because x and @ are associative up to isomorphism.
Further, we have
V A X X
V
V
(134)
( X B
X
AC
Ac).
(135)
because composition is associative.
The singleton (f+), [[+]x(=); id(,); (a)) the unit of the bifunctor R. this way. (V-Set, K (*))
forms monoidal category.
However, note that the bifunctor X is symmetrie because V X V and A X A are necessarily symmetric, and A X d ~ d &d if and only if the monoidal product @ of the quantale is commutative.
Since V-Set is monoidal category, we can consider a strict monoidal functor Fy Set + V-Set, which equips products of sets with product strict premetrics and product aggregators:
dAxB == d X dg.
(136)
V AxB V V
(137)
AAx 4 X An
(138)
Such a monoidal functor induces a functor from (Set-enriched) category to U-Set-enriched category, called the base change of enriching category. This means that we have systematic way to equip a set [A, B) morphisms with a strict premetric
dA.B : [A, BJ
[A, B V
(139)
universal aggregator
[A.B] : (AA.B) + V
(140)
and an existential aggregator
|| v/4.81 + L
(141)
which is compatible with the product.
Then, Theorem 1 is special of this enrichment. The relationship between predicates and quantities follows from Theorems 10 and 14.
B.12 Summary
Finally, to accommodate readers without background in category theory, present the instantiated definitions and theoretical results free categorical terminology. The non-categorical proofs are omitted. We will be using the following functions:
the zero predicate Ç: [0, co] + [T, x (x= 0).
the product (" [0, oo "
+ (T,1)" (g)..... an) - (a) = -0...,4, = 0), and
the postcomposition ca [0, 0o) - (o of the zero predicate.
39

---

Definition 31 (Quantity). A quantity A - [0, oo] is homomorphic to predicate p: p:A-1 + {T,1} if p = Çog:
Va E A. (q(a) = 0) < p(a).
(142)
A quantity is subhomomorphic predicate pf o -
Ya E A. (q(a) = 0) p(a)) Example 20. A strict premetric da X A - [0, x0] quantity on the product set x A
(143)
homomorphic to the equality predicate = A x A + IT.
Definition 32 (Quantitative operation). Let E N be natural number. A quantitative operation
a 00 oo]" + [0, oo) is homomorphic to a logical operation 3 [T. + [T, via the zero predicate ( a 3O
Van. Un) E [0, oo]" (aly)..... ) 0) ++ B(n 0,... ................................ Un 0).
(144)
quantitative operation a is subhomomorphic to a logical operation 3 via the zero predicate ςoa 3 3 V(g....... (n) E [0, col"
0) - B(qi ...... 9n = 0).
(145)
The relationship between quantitative operations and logical operations is follows (Theorem 10): Proposition 16.
Let n E N be a natural number. For E {l....,.7 n). let p. A + {T, be a predicate, and let q; : A -> [0, oo] be quantity. Let p A {T, 1)" == (P1y---, Pad) be the tupling of the predicates, and let q - [0, oo) " := (f....., qn) be the tupling of the quantities. Let co]* [0, oo] be quantitative operation, and let B ; 1) + LT, be logical operation. Assume that for all E (I...., q is homomorphic to ps. Then,
a a is homomorphic to B, a ooq q homomorphic to 3 o p; and a is subhomonorphic to 3, a o subhomomorphic to B o.p.
In fact, we can also show that if for all i E {1.....n), ), is subhomomorphic to ps, and a is subhomomorphic to 3, then a q is subhomomorphic to / p.
Definition 33 (Aggregator). An aggregator a' 1a:10 o0] -
oo] is homomorphic quantifier 3A [T, - [T via the zero predicate Ç a = BB 5
Vg E [0, x0)
((a DEA
((a)) = (()))=))))))))))))))))))))))({) 0) ++ B (g(a) = =))
(146)
aaEA
Example 21. A universal aggregator V 0, x0] + [0, oo] is function such that
1y E [0, 00) (VV (())) 0) + (Va E A. g(a) = 0)- 4EA
(147)
Example 22. An existential aggregator A oo] + 10, oo] is function such that ((a)) =0 + (3a E A. g(aa) 0).
Vq E [0, 0o)
( "
(148)
The relationship between aggregators and quantifiers is follows (Theorem 14):
Proposition 17. Let A B + [T. be a predicate, and let q : A B - 10, oo] be a quantity. Let p: B - 1J and q: B - [0, ox]* be the exponential transposes of and q. Let a : [0, co] + [0, oo) be aggregator, and let BA 1} {T, be a quantifier. Then, if q is homomorphic to p. and a is homomorphic to B, then a o q is homomorphic to 3 p.
We have a compositional way to assign strict premetric and an aggregator to a product of sets: Proposition 18. Let d, A A [0, oo] and dB B B J0. oo) be strict premetrics. Then, dAxB (A B) (Ax x B) + [0, == ((a, (a' b')) + (a)aa) + d(b,b') is strict premetric on the product sei A x B. Proposition 19. Let V
(149)
[0, 00) A [0, xo] and V (0, [0) 10, oo] be universal aggregators.
Then,
V 00, 00)
AXB + |0, o0 :
BEB ((a
7
(150)
Ax
is universal aggregator on the product set A B.
40

---

C Proofs
C.1 Proposition 2
Proof.
9pmducr( + Z)
(151)
inf inf d dy:z (m, x m222)
(152)
(,, 22 inf
inf (d (d1x22, (m, m1,1 o P1) + diy.zz)(m2 22,2 o P2))
(153)
EI,,, 2
inf d dv.2, (m, 111 o p1) + 2,
inf
d 4y.2,/ (m2 222 p2)
(154)
m1,1E(M, inf v dz, (m) m1,1 (vI)) mi.1EMM,,21e
inf 22 yEY
& dz, (m2 m2
(155)
inf Dien v2tt
v dz (m, (11, 12),
inf
4 dzg
v2) (1
(156)
m2,2 2, EYY Mv
V. EY BEE,
dz, (m (1)
dz, (m2(V1, mi
(157)
where
mi,
==
arg arginf inf
y v dz, (m, U2). m1,1
(158)
m MY wEy,
31 + ing v dz, (0 82). 21),
(159)
2, BEE,
22,2 = ing inf V V dzp (m2 12). m2 (
(160)
m2 2.] BsEY, viey
= y/ ト arg inf V dz, (1)(3.)2)),)))
(161)
ZZ Mey,
C.2
Proposition 3
Proof.
Cuurst- (m: Y + 7)
(162)
Soonst (7g + Goonst m2g
(163)
Va ViEV, dr,, (mi (322. (bb)) + & V.
V.
mY vien, ( (( (z)1)
(164)
7
4 d2 (m, (122 (16)
M2 V EY,
7 dz, (m)
(165)
Me ViEY E
V V
v dz, (m, (V)
V2EY, v
eY
v
V d2 (m
(166)
MYY, ViE, EYEY
V 7 dz (m U2), mn (2), 2/21)
YY E, E
V
v d2, (m U2) my (V/, 02))
m2
(167)
E 1 viEy
41

---

D Discussions
D.1 Background
Defining and measuring the properties of learning models is core topic in machine learning, especially representation learning [Bengio et al., 2013]. A proper comprehension of what constitutes good representations and how to assess their quality is important for developing suitable learning objectives and evaluation metrics. To define these properties, many important concepts are given by equational predicates, such as independence of random variables, extensively used in statistical learning and causal learning [Hyvärinen and Oja, 2000, Koller and Friedman, 2009, Schölkopf and von Kügelgen, 2022], and equivariance of learning models, reflecting the symmetries and structures of the data [Cohen and Welling, 2016, Zaheer et al., 2017, Higgins et al., 2018, Maron et al., 2019, de Haan et al., 2020, Cohen, 2021, van der Pol et al., 2022, Navon et al., 2023].
Considerable efforts have been put into designing model architectures that perfectly satisfy specific properties, such as monotonicity [Sill, 1997, Daniels and Velikova, 2010], invertibility [Rezende and Mohamed, 2015, Behrmann et al., 2019, Ishikawa et al., 2023], convexity [Amos et al., 2017], and equivariance [Lee et al.. 2019, Brehmer et al., 2023]. However, hard-coding multiple properties into a model by design could be challenging [Köhler et al., 2020J. Hence, it is desirable to devise quantitative metrics to directly measure these properties, even if the models do not have the properties built-in [Goodfellow et al., 2009, Chen et al., 2020, K Kvinge et al., 2022]. Ideally, these metrics should be easily computable or even differentiable, allowing us directly optimize the properties.
Disentangled representation learning [Bengio et al., 2013], our main focus of this paper, is such a field where defining and measuring the desired properties not straightforwatt tasks [Carbonneau et al., 2022, Zhang and Sugiyama, 2023]. It has been suggested that disentangling the underlying explanatory factors in complex data is promising approach for reliable, interpretable, generalizable, and data-efficient representation learning [Locatello et al., 2019a,b, Montero et al., 2021, Dittadi et al., 2021, Xu et al., 2022]. However, in contrast the wealth of results regarding invariant and equivariant layers. the exploration of designing a "disentangled layer" has been relatively limited. One reason that disentanglement was not considered a singular property but rather a combination of several requirements. The absence of a clear definition and appropriate metries for disentanglement has created a gap between the learning objectives and evaluation metrics. new evaluation metric is often introduced along with a new representation learning method [Carbonneau et al., 2022], but it is usually unproven that the method can optimize the new metric, and the metric truly quantifies the alleged property [Higgins et al., 2017, Kim and Mnih, 2018, Chen et al., 2018, Li et al., 2020].
To formally define disentanglement, line of research utilized group theory and representation theory [Cohen and Welling, 2014, 2015, Higgins et al., 2018], with a focus on the direct producl of groups. Thanks to the rich algebraic structure, it becomes possible to derive various model architectures and learning objectives from the equational requirements of the product and equivariance [Caselles-Doe Duuré et al., 2019. Pfau et al., 2020. Quessard et al., 2020, Painter et al., 2020, Miyato et al., 2022, Yang et al., 2022, Tonnaer et al., 2022, Keurti et al., 2023]. Another approach adopted a topological perspective, using concepts such as the product manifold to define disentanglement [Zhou et al., 2020, Fumero et al., 2021, Zhang et al., 2021, Balabin et al., 2024]. However, theoretically comparing different approaches has been a challenging task.
To quantitatively measure disentanglement, Ridgeway and Mozer [2018] proposed three concepts called modularity, compactness, and explicimess, which were defined verbally but not mathematically. Eastwood and Williams [2018] proposed similar three criteria called disentanglemen, completeness, and informativeness and corresponding evaluation metrics. However, it was unclear what properties these metrics truly quantify. Additionally, due to the necessity for additional training of classifiers along with hyperparameter tuning and the involvement of non-differentiable regressors such as the random forest [Breiman et al., 1984], it is impossible to directly optimize these metrics using gradient- based optimization. Recently, Eastwood al. [2023] extended this framework with two new metrics called explicinesss/easeofff of-use and size based on the functional capacity. Do and Tran [2020] introduced metrics for informativeness, separubility, independencce and interpretability from an cnformation-thooetii perspective, while Tokui and Sato [2022] introduced a new metric in terms of uniqueness, redundancy, and synergy based on partial information decomposition. These metries have been mainly used during the evaluation stage, after model is trained with other learning objectives.
42

---

D.2 Related work
Equivariance The work by Kvinge al. [2022] might be the closest to our approach in spirit. They directly converted equivariance, an equational predicate, to quantitative metric and analyzed their relationship (Proposition 3.2). In contrast, based on our proposed conversion method, we can use the following definition and metric:
Definition 34 (Equivariant function). Let A, B, and C' be sets. A function f A - B is equivariant to actions (any binary functions) 'A : C' x A + A and 'B" C B B if
Pequivariant (f : B) = Yc C. fo (c -) [ (c. -) Vc C. Va E A. f(c = C f(a),
(168)
(169)
which can be measured by
Ioquivariant (f A + v V CEC'EA
(170)
Calibration More broadly, the study of the relationship between different metrics in statistical learning is called calibration analysis [Steinwart, 2007, Reid and Williamson, 2010, Ni et al., 2019, Bao and Sugiyama, 2020, Bao et al., 2020]. Our work can be seen as an extension of the concept of the calibration to a wider range of properties defined by equational predicates.
Disentanglement metric In disentangled representation learning, metrics similar to Eq. (26) have been proposed by Higgins et al. [2017], Kim and Mnih [2018]. Their metrics also fix one factor and vary all others and calculate some constancy metrics (the mean pairwise distance in Higgins et al. [2017] and the variance in Kim and Mnih [2018]). However, both studies took an indirect approach, involving the training of classifier to predict the fixed factor. Consequently, the resulting metrics are not differentiable anymore and entangle modularity and informativeness. In this work, we argue that it is better to measure these two properties separately.
Weakly supervised disentanglement Ridgeway and Mozer [2018] proposed and investigated the similarity supervision and argued that such supervision is easy to obtain via crowdsourcing. Shu et al. [2020] further studied this type of supervision based on distribution matching and referred as match pairing. Other weaker forms of supervision were also investigated, such as the number of changed factors [Locatello al., 2020] or paired data with unknown intervention [Brehmer et al., 2022]. Given that our theory can establish connections between logical definitions and quantitative metrics, it holds promise for deriving disentanglement metrics for various types of weak supervision based on logical inference.
Multi-valued logic Aristotelian logic assumes that every proposition is either true or false, adhering to the principle of bivalence. The law of Aristotelian logic can be algebraically represented on the set {0, 1} of binary truth values [Boole, 1854], known as the two-element Boolean algebra. The exploration of non-Aristotelian logic involves investigating logical systems that relax modify this strict binary valuation, allowing for a broader range of truth values and accommodating various forms of uncertainty, vagueness, context-dependence in reasoning [Hájek, 1998, Malinowski, 2007, Bergmann, 2008].
The mathematical study of multi-valued logic can date back to the seminal work by Lukasiewicz in 1920, who introduced a third truth value interpreted as "possibility" and symbolized by J. Lukasiewicz [1920] examined several principles in this three-valued logic such as the principles of identity, implication, syllogism, and contradiction, and discussed its theoretical and practical importance in indeterministic philosophy and deductive sciences. Later, Lukasiewicz and Tarski [1930] proposed propositional calculus, a theory of propositions with values from the real interval [o, 1]. which is now also commonly known Lukasiewicz logic. Eukasiewicz logic involves new continuous logical connectives such as strong/weak conjunction and disjunction.
Furthermore, Chang [1958] studied the algebraic systems for many-valued logic, called MV-algebras. Chang and Keisler [1966] then proposed continuous model theory, also referred to as compuct-valued
logic [Ben Yaacov, 2022]. where the truth values can be in arbitrary compact Hausdorff spaces and a wide variety of quantifiers was studied. Later, Ben Yaacov et al. [2008] studied model theory for metric structures and proposed (real-valued) continuous first-order logic [Ben Yaacov and Usvyatsov,
43

---

2010], where the space of truth values is closed, bounded interval of real numbers with the order topology (e.g.. 1)), and suggested that we only need two canonical quantifiers sup and inf. From a categorical perspective, Cho [2020] developed categorical semantics of metric spaces and continuous logic by introducing the notion of continuous subobject classifier, and Figueroa and van den Berg [2022] studied topos of continuous logic using the notion of hyperdoctrine.
On the other hand, from a categorical perspective, Lawvere [1973] showed that a generalized metric space, also known as Lawvere metric space, is a category enriched over what is now commonly called the Lawvere quantale (10, oo]. 2.+ 0), i.e., the set [0. oo] of extended non-negative real numbers equipped with addition + as (semicartesian) monoidal product and truncated subtraction as the internal hom. In other words, a Lawvere metric space is set A equipped with a function d: : A x A + [0, oo] such that for all a E A, we have 0 2 d(a, a) or d(a, a) = 0 (identity), which makes d a premetric, and for all a, b, c E A, we have d(b, c) + d(a,b) b) 2 d(a, e) (composition), which means that d satisfies the triangle inequality.
Recently, Mardare et al. [2016] took an equational approach to quantitative algebraic reasoning, which was later also referred to as quantitative equational logic [Mardare et al., 20211, by introducing approximate equality predicates = indexed by rational numbers & (i.e., a = bf a and b are at most & aaprrt,, and suggested that this approach essentially involves working with enriched Lawvere theory, Dagnino and Pasquali [2022] provided a logical ground quantitative reasoning in the categorical language of Lawvere's 's doctrines by viewing distances as equality predicates in linear logic. Bacci al. [2023] further studied the natural deduction systems of propositional logics for the Lawvere quantale and introduced what was later called affine Lawvere logic, including Eukasiewicz logic and Ben Yaacov's continuous propositional logic. Bacci et al. [2024] extended affine Lawvere logic to polynomial Lawvere logic by allowing multiplication as an extra logical connective. These studies are on propositional logic and do not involve predicates and quantifiers. Recently, Capucci [2024] studied spectrum of quantifiers in [0, vol-valued quantitative predicate logic.
In the context of machine learning, these relatively recently developed logies have yet prove their practical importance. While these innovative approaches often hold theoretical promise, they need to demonstrate tangible benefits in real- weal-world applications. Key areas where these new logics might eventually make an impact include neuro-symbolic reasoning and logic/probabllistic programming [d' Avila Garcez et al., 2002, Manhaeve et al., 2018, Sen et al., 2022, Badreddine et al., 2022, Fagin et al., 2024], which hold promise for integrating low-level perception with high-level reasoning, improving model interpretability, enhancing training efficiency, and enabling more robust decision- making processes. However, widespread adoption and validation through practical use cases are necessary to establish their true value and effectiveness in the machine learning landscape.
Under this background, let us contextualize our proposed methodology for deriving j0, oc] volluel quantitative metrics from logical definitions. We highlight three characteristies of our framework:
We allowed not only metrics, but strict premetrics, such as the relative entropy Skullback-eiible divergence) [Kullback and Leibler, 1951, Perrone, 2023] widely used in machine learning, as the real-valued counterparts for the equality predicates;
We focused on whether the metrics zero not and the (differentiable) optimization of the derived metrics, because our main goal is to guarantee that the minimizers of the derived metrics satisfy the predicate:
We included a wide range of real-valued quantifiers (or aggregators in our terms) beyond sup and inf, such as mean and mean square, as long as they are homomorphic the two-valued quantifiers, because the derived metrics may have nicer properties or even analytical solutions.
44

---

in seld) macisl
16 pofo Harll) 062 (udlo eclll andi
palo scldl
p9ll pufli pell) o16 goted i gule!
Ki0 (ala) f lelo sste
660 tn a462-00 ando
(a) homomorphic
(b) adjoint to max
(c) less coverage
(d) more coverage
pull pefel (h) fraction
(c) negation, disjunction
min, hinge
gg) min, reciprocal
Figure 8: Quantitative operations for implication
Hid ratst espele artllveestt eetdiaii
snle)) whoeee wild wril))
edell wi
(a) bi-implication
(b) CNF
(c) DNF
(d) fraction
Figure 9: Quantitative operations for equivalence
D.3 Implication and equivalence
In Section 3, we only used the truncated subtraction as a quantitative operation for the implication - (Table 1). In Theorem 1, noted that the implication is special because if predicate involves the implication, then not all elements satisfying the predicate minimize the corresponding quantities (Fig. 2). In Appendix B.8, we discussed other possible quantitative operations corresponding to the implication >.
In Fig. 8, showed eight alternative quantitative operations for the implication. In the first row, the first one is homomorphic to the implication, which means that its minimizers are exactly those that satisfy the predicate (Example 16); the second one is right adjoint to the max (Example 17); and the other two are variants the truncated subtraction (Example 18), which have more or less coverage. In the second row, used the logical equivalence between operations for the implication using quantitative operations for the negation and disjunction. For example,
b and -a b define quantitative
the hinge function - and min, which lead min a, b), or the reciprocal
function
1 and ab a+t. which lead to I I+ab
16 b
Similarly, we can use logically equivalent expressions of the logical equivalence ++ such as (a b) A (b a) (bi-implication), (-a v 6) A (-b V a) (conjunctive normal form (CNF)), and (a A b) V (-a A -b) (disjunctive normal form (DNF)), derive quantitative operations for the equivalence, shown in Fig.
45

---

diameter
median
center
mean
(a) central point
(b) pairwise distance
Two approaches for measuring the constancy of set in R²: (a) finding a central point, such the center of the smallest bounding sphere, the geometric median, or the mean, and then measuring the dispersion around this point; and (b) aggregating pairwise distances between points.
Figure 10:
Note that a quantitative operation homomorphic to the implication cannot be continuous everywhere, which is undesirable for gradient-basee based optimization. For example, the following quantity also measures the injectivity of a function m Z: 0] (dy 3') 0] dy (v.8')
V (d)) WEY VEYY"
(171)
V v [m(v) =z m(y') i Fv v1 dy (v. v/).
(172)
vEY y v'EY
This quantity aggregates distances between pairs of different inputs mapped to the same outputs. However, unlike lnjeetive introduced in Section 4.3, is not differentiable with respect to the function m Y + Z. Thus, we cannot use it to improve the injectivity of a function by gradient descent.
D.4
Constant function
Note that there two logically equivalent definitions of constant function (to non-empty set). One is based on the equality between all pairs, as in Definition 11. The other is based on the constant output value (see Fig. 10):
Definition 35 (Constant function with value). A function f A B is constant function with value E B if
Peonst-v (f :
3b B. Va A. (f(a) = b).
(173)
which can be measured by
Aconst (s:A + B) := inf V BEB
dn(s(a), aEA
(174)
This quantity 4cons-v finds a central point in the codomain that best approximates all the outputs of function, which similar the approach we discussed in Section 4.1. In fact, we can prove that we use Gconst-v in lcoms-cunyy we will end up with the same quantity Ipoduct
46

---

(a) small radius, large variance
(b) large radius, small variance
Figure 11: Metrics may rank imperfect representations differently.
Figure 12: For pair of constancy metries (each column), find two sets of points in IR² ranked differently by these metrics, except for the radius and diameter, because for subset Ao in a set A, we have inf E SuPaEA, d a) < inf in,, SUPaeAo d (ag: a) SuPaoEA, SuPaeAg d (ao: a).
D.5 Rank of imperfect representations
is worth noting that Theorem only guarantees that the minimizers of different quantitative metrics derived from the same logical definition are the same, but imperfect representations, whose evaluation results are non-zero, may be ranked differently by different metrics.
For example, Fig. 11 illustrates two constancy metrics, the radius of the smallest bounding sphere and the variance, on two sets of points in IR. where one set has small radius but large variance, while the other has large radius but a small variance. More examples are presented in Fig. 12, and such results can also be observed in Table 2. This difference can lead to differences in risk preferences, sensitivity to outliers, and learning dynamics when these metrics are used as learning objectives. Further investigation of the characteristies of these metrics for imperfect representations is left for future work.
47

---

D.6 Implementation
Thanks to advanced indexing (e.g., NumPy [Harris al., 2020] and PyTorch [Paszke et al., 2019]) and analytical solutions to some optimization problems (e.g., Eq. (19)), some of the proposed metrics can be easily implemented, even as Python one-liners.
For example, the following function implements a family of modularity metrics:
def q_product(y; np-ndarray, 2: ap. ndarray, aggregate, deviation): return пр ap..sum[aagg aalgggrggate(deeiittiiontn zip(y, 2)1)
=- yvl) for yv in np. (p.uuqque(yy))) for yi, zi in
Here, y and z are NumPy arrays of shape (factor, index); aggregate can be max, mean, or sum; deviation can be function calculating the radius of the smallest bounding sphere, 12 mean absolute deviation around the geometric median, variance, diameter, or mean pairwise distance. Please note, however, that the deviation function can be computationally expensive, depending the dimension of the codes.
D.7 Limitations
Lastly, we discuss several aspects that are not covered in this work and potential directions for future research.
Function equality A collection of input-out pairs {x 91) Hi) E (X Y)" may not define function g XX + Y for two reasons: First, the set of all inputs X, == { is unlikely to enumerate all possible inputs (i.e., Xo Ç X), especially when the cardinality of the domain X is infinite (e.g., IR), so the data may only define a purtial function g : X
or a function from smaller domain 9o : .x + Y. Second, the inputs may not be distinct, e.g., when input is given multiple labels by different annotators, so the data may define a multi-valiee function. The extension from functions relations stochastic maps is important future direction of our work.
Partial combinations A more general issue learning and evaluating disentangled representations given only subset of all combinations of factors, which is common when dealing with a large number factors [Träuble et al., 2021, Montero et al., 2021, 2022, Roth et al., 2023]. It crucial to evaluate and justify whether metric computed partial combinations of factors reliable proxy for the performance of the model on unseen combinations.
Unknown projections Another common scenario is when the extracted representation is not properly aligned with the underlying factors. For example, a model may extract a diree-iimensional representation z E R for two factors y E [o, 1]2, and it can project to ((21, 22), 2g) or (21, (22, 23)). How can we determine which better, without enumerating all possible projections? Finding the optimal assignment [Mahon et al., 2023] and correcting a pre-trained model post hoc [Träuble et al., 2021] based the proposed metries interesting future directions.
D.8 Broader impact
This paper focuses on the theoretical aspects of disentangled representation learning, and we do not foresee any immediate negative societal consequences. However, would acknowledge that disentanglement is closely related to data-efficiency and fairness, potentially sparking discussions on ethical considerations. Besides, the application of category theory may facilitate the transfer and integration of knowledge across disciplines, fostering closer connections between various fields of study, even beyond the machine learning community.
(MIT License) [Welzl, 1991]
Phttps://githut httpu///ow..riiiaapp//ggooooonno (GNU General Public License, Version 3 (GPLv3)) [Pillutla al., 2022]
48

---

deviation
aporoximation
contraction
(a) true factors
(b) entangled codes Z (c) product approximation (d) linear approximation of
of an encoder m + Z its retraction h : + Y
Figure 13: (a) a set factors Y represented by the RGB color model; (b) a set of entangled codes Z extracted by encoder Y - Z; (C) product function approximation; and (d) linear approximation the retraction - Y the encoder.
(at ori & button
15) Meggo tranfom
invers Spundfom
1ST arthegoral tranufrrascoo
1s] stagral braubmm
mvene tranaform
Figure 14: Entangling a multivariate distribution via probability integral transform, inverse transform, and orthogonal transformation of standard normal distribution [Locatello et al., 2019b, Theorem 1].
E Experiments
In this section, provide the detailed data configuration used in Section 5 and further experimental results.
E.1 Synthetic data
used a simple synthetic setup to simulate entanglement factors and common failures patterns. Concretely, we used Cartesian product == {00, 0.1, 1 1}3 (Fig. 13a). We used random rotation matrix R to entangle factors and componentwise exponential as non-linear transformation. We composited this procedure twice and used an affine transformation to normalize the outputs (Fig. 13b). That is, used the following function the data generating process:
three sets as the underlying factors
g g:Y
+ a exp(R. exp(R y)) +
(175)
Note that this function is injective but not product or linear.
We used the following functions as the function Y + Z:
entanglement
(U1: U2- Us
g(V1 V22 Us))
rotation
(1/1 V2. Vs)
R (31, 22 My3
duplicate
(31 1/2 Vs)
((11 W2 V3). (111 M2, Us)s Us)
complement
(31 3/2 V/)
(V2, Na); (31 Ma), (31, 92))
misalignment
(31 V/z Vs)
(U2, Us: V1
redundancy
(111 V/2 Vs)
+
(()1 91.. V2. Va)
contraction
(U11 3/2 Va)
0.01 x (y) V2. V3
nonlinear
(U1 V/z Vs)
(v? U2: v3)
2 2
constant
(U1 V/ Vs)
+
(0, (0,0,0)
The rotation operation entangles factors but can (linearly) inverted. The duplicate encoder has modular decoder (projections), but itself is not modular. The redundancy encoder is both modular and informative, but not all codes can decoded. The constant encoder is perfectly modular not informative.
49

---

Table 3: Supervised modularity metrics
Product approx.
Constancy
Rad.
MAD
Var.
Diam.
MPD
entanglement
0.44
0.75
0.96
0.19
0.82
rotation
0.22
0.51
0.80
0.05
0.64
duplicate
0.24
0.43
0.67
0.06
0.56
complement
0.12
0.28
0.55
0.01
0,42
misalignment
0.22
0.44
0.74
0.05
0.58
random
0.22
0.48
0.78
0.05
0.61
Table 4: Weakly supervised modularity metrics
Product approx.
Constancy
Rad.
MAD
Var.
Diam.
MPD
entanglement
0.50
0.77
0.96
0.26
0.84
rotation
0.24
0.54
0.83
0.06
0.68
duplicate
0.28
0.46
0.71
0.07
0.60
complement
0.14
0.32
0.59
0.02
0.47
misalignment
0.22
0.48
0.77
0.05
0.62
random
0.24
0.52
0.81
0.06
0.64
E.2 Weakly supervised modularity metrics
We briefly comment on the possibility of employing weak supervision for measuring disentangled
representations.
For supervised disentanglement metrics we discussed in Section 4, the necessary data consists of observation-facee factor pairs (z, y), representing a generator g Y - X. In order to evaluate encoder f X + Z, we compose it with a generator and study the properties of the composition Z fog.
worth noting that Pproduct and Pinjective are equational predicates, which means that they are invariant to bijections. Similarly, Opoduct and Linjective are invariant to isometries. This implies that the exact values of the factors are not important; we only need to know two factors are equal or not. Hence, only need weak supervision of the form (z, z', y/ =y, y that we can construct some cquivalence classes of factors, and we can still calculate or approximate Eqs. (19) and (27). Note that this type of supervision has been partially investigated by Ridgeway and Mozer [2018], Shu et al. [2020].
For example, suppose we have an object A (e.g., red circle) and an object B (e.g., red triangle), and all know is that objects A and have the same color. Based on such weak information, we can still construct an equivalence class containing objects with the same color object A. Then, can regularize an encoder f > Z by minimizing the variance of the color representations over this equivalence class (Eq. (19)). In this way, the modularity of the encoder can be improved. The challenge arises when there is noise or only partial combinations, which is an interesting future work direction. In such cases, we may need to semi-supervised clustering to group the data [Wagstaff' et al., 2001, Basu et al., 2002, Bilenko et al., 2004].
To validate this idea, we conducted experiments where we only used a random sample of pairs and their similarities. Table an excerpt of Table 2, showing only the proposed modularity metries, and Table shows these metrics calculated using only similarity supervision. We reported the mean values 10 random samples of pairs, and the variances are negligible. Comparing Tables and 4, we can observe that weakly supervised metrics may overestimate imperfect representations, but they can still maintain the ranks. This observation suggests the potential utility of employing weak supervision for both learning and evaluating disentangled representations using the proposed metrics.
50

---

Table 5: Supervised disentanglement metries on image datasets
Modularity
Informaiveness
Existing metrics
Product approx.
Constancy
Retraction approx.
Contraction
Pair
Info.
Regressor
MPD
MB
Mean
Factor"
MIG
Com." 2 Info."
AD Cars (Reed et al., 2015]
VAE
0.27
0.76
0.95
0.07
0.83
0.44
0.82
0.94
0.21
0.75
0.500
0.22
0.07
0.05
0.54
2-VAE
0:26
0.76
0.95
0.07
0.82
0.42
0.82
0.94
0.20
0.74
0.50
0.21
0.01
0.13
0.10
0.54
FactorVAB
0.24
0.75
0.95
0.06
0.82
0.35
0.82
0.94
0.21
0.74
0.89
0.20
0.03
0.11
0.06
0.54
3TCVAE
0.29
0.77
0.95
0.08
0.83
0.40
0.82
0.94
0.21
0.74
0.90
0.21
0.14
0.59
dSprites [Matthey ot al., 2017]
VAE
0:21
0.64
0.94
0.06
0.74
0.37
00.82
0.941
0.18
0.68
0.54
0.25
0.009
0.16
0.15
0.40
2-VAE
0:14
0.64
0..83
VAc
0.18
0.63
0.93
0.03
0.73
0.38
0.82
0.94
0.17
0.67
0.18
0.25
0.13
0.20
0.23
0.36
3-TCVAE
0.22
0.64
0.93
0.05
0.73
0.42
0.80
0.94
0.21
0.70
0.56
0.27
0.18
0.22
0.31
0.59
AD Shupes [Burgess
2018]
VAE
0:25
0.76
0..96
0.06
0.82
0.39
0.888
0.96
0.20
0.78
0.599
0.94
0.39
0.33
0.22
0.76
8-VAE
0.20
0.72
0.95
0.01
0.79
0.39
0.81
0.94
0.19
0.69
0.86
0.77
0.26
0.69
0.62
0.99
VactorVAE
0.24
0.34
0.82
0.93
0.64
0.83
0.57
0.84
2-TCVAE
0:255
0.69
0.94
0.06
0.77
0.32
0.82
0.93
0.18
0.63
0.76
0.50
0.31
0.68
0.67
0.98
MPL3D (Gooddll
2019)
VAE
0.01
0.56
0.91
0,00
0.66
0.30
0.76
0.89
0.12
0.46
0.48
0.11
0.12
0.29
0.33
0.64
2-VAE
0.02
0.84
0.97
0.00
0.87
0.28
0.775
0.50
0.10
0.41
0.30
0.11
0.07
0.15
0.18
0.47
VactorVAl
0:00
0.75
0.47
0.00
9-TCVAE
0.07
0.65
0.95
0.01
0.74
0.28
0.76
0.89
0.12
0.10
0.45
0.07
0.12
0.24
0.31
0.57
[Higgins ct al., 2017]
[Kim and Mnih, 2018]
[Chen et al., 2018] 4 [Eastwood and Williams, 2018]
E.3
Evaluation of existing models on image datasets
We also report the results of several widely used unsupervised disentangled representation learning methods (VAE [Kingma and Welling, 2014], 33VAE (Higgins et al., 2017], FactorVAE, [Kim and Mnih, 2018 8), and 3-TCVAE [Chen et al., 2018) 8)) evaluated on four image datasets (3D Cars [Reed al., 2015], dSprites [Matthey et al., 2017], 3D Shapes [Burgess and Kim, 201 8], and MPI3D [Gondal et al., 2019]) in Table 5.
We used public PyTorch implementation [Paszke et al., 2019] of these methods and used the same encoder/decoder architecture with the default hyperparameters described in Locatello et al. [2019b] for all methods for fair comparison. We used linear projection find the most informative representations for each factor. The experiments were conducted on a NVIDIA Tesla V100 GPU.
Before analyzing these results, it is important to note that the evaluation of these learning models is not meant to be a proof of the correctness of the proposed metrics, since we cannot tell whether a bad result is due to the insufficiency of learning method, the quality of the datasets, to the problem of the evaluation, if we have no theoretical guarantee for the metrics. We can trust the results of the proposed metrics because the properties of their minimizers are guaranteed by Theorem 1.
From Table we can observe that the considered learning methods do not exhibit significant difference in terms of modularity and informativeness. This result supports the theoretical finding of Locatello et al. [2019b] that unsupervised learning of disentangled representations by matching the distributions of observations is fundamentally impossible (see also Fig. 14) as well as their empirical finding that there is no evidence that learning disentangled representations in an unsupervised manner is reliable.
E,4 Kendall tau distance between metrics
To analyze the relationship between these metrics, we report the Kendall tau distance [Kendall, 1938, Virtanen et al., 2020] averaged over experimental settings in Table 6. The Kendall tau distance a correlation measure for ordinal data valued in [-1, 1] which counts the number of pairwise disagreements between two ranking lists. Values close to 1 indicate strong agreement, and values close to -1 indicate strong disagreement.
From Table 6 we can observe that even though different metrics derived from the same logical definition may rank imperfect representations differently (see also Fig. 12), they still have positive comelations with each other, indicating that they measure the same property. The metrics proposed by Higgins al. [2017] and Kim and Mnih [2018] have the highest correlations with each other (except for themselves), and we hypothesize that this is because they both based on the pairwise distance
51

---

Table 6: Average Kendall tau rank distances bewteen disentanglement metrics
Modularity
Informativeness
Exising metrics
Product approx.
Constancy
Retraction approx.
Conrmction
Info.
Regressor
Rad.
MAD
Vir.
Diam.
MPD
ME
MAE
MSE
Max
Mean
Beta"
Factor
MIG"
Dis."
Com."
Infoo"
Rad.
1.00
01008
0..25
1.001
0.33
-0.08
D.08
0.17
0.08
0.17
-0.25
0.11
0.001
0.000
0.17
- -0.08
MAD
0.08
1.00
0.50
0.08
0.75
0.33
0.67
0.58
0.00
0.42
-0.50
-0.58
-0.25
0.25
0.06
-0.00
0.50
0.25
-0.98
0.42
Diam.
1.00
0.08
0.25
1.00
0.33
-0.08
0.08
0.17
0.08
0.17
-0.25
0.17
0.00
0.00
0.17
-0.08
MPD
0:33
0.75
0.75
0.33
1.00
0.25
0.42
0.50
-0.08
0.33
-0.25
-0.33
-0.17
0.33
0.17
0.08
ME
-0 0.08
0.33
0.33
-0.08
0.25
1.00
0.50
0.38
0.33
0.42
-0.17
-0.25
-0.42
0.08
-0.06
0.00
MAE
0.08
0.87
0.53
0.08
0.42
0.50
1,00
0.92
0.17
0.75
-0.67
-0.58
-0.25
0.08
-0.05
-0.17
MSE
0.58
0.17
0.52
1.00
0.25
0.58
0..00
-0.33
0.37
0.25
0.00
-0.17
0.08
-0.08
0.33
-0.33
-0.42
-0.25
Mean
0.17
0.42
0.25
0.17
0.33
0.42
0.75
0.83
0.42
1.00
-0.75
-0.50
-0.33
-0.17
-0.33
-0.12
Beta
-1.25
-0.50
-0.00
-0.25
-0.25
-0.17
-0.67
-0.55
-0.33
-0.75
1.00
0.58
0.25
0.25
0.25
0.50
Factor
0.17
-0.58
-0.25
0.17
-0.35
-0.25
-0.88
0..00
0.08
0.50
0.58
1.00
-0.17
-0.17
-0.17
0.08
MIG
-0.25
-0.08
0.00
0..17
-0.42
0.33
00.42
0.25
00.17
1.00
0.33
Dis.
0.22
0.00
6.08
00.25
-0.17
0.25
-0.17
0.53
Com.
0.17
0.08
0.42
0.17
0.17
-0.08
-0.08
-0.17
-0.42
-0.33
0.25
-0.17
0.33
0.83
1.00
0.75
Info.
0.08
0..00
0.33
0.08
0.08
0.00
0.17
0.25
0.33
0.42
0.50
0.08
0.08
0.75
0.75
1.00
[Higgins ct al., 2017]
[Kim and Mnih, 2018] [Chen et al., 2018]
4 [Fastwood and Williams, 2018]
Table 7: Computation time (seconds) of supervised disentanglement metrics on image datasets
Modularity
Informativeness
Existing metrics
Product approx.
Coustancy
Retraction appros.
Contrattonn
Pair
Info.
Regressor
Rad.
MAD
Var.
Diam.
MPD
ME
MAE
MSE
Mean
Heta"
Factor"
MIG
DCT
Cars el al. 2015]
0:35
2.22
0.03
2.11
dSprites [Matthey al., 2017]
0:47
1.04
0.00
0.03
0.03
0.51
3.86
0.00
1.36
1.37
7.00
6.37
11.51
353
3D Shapes [Burgess and Kim, 2018]
0.00
0.02
0.00
0.03
0.03
0.06
4.77
0.00
1.07
1.52
4.85
4.65
10.16
109:11
MPED [Gondal et al., 2019]
0.86
9.30
0.00
0.09
0.09
1.61
5.08
0.00
1.89
1.94
9.54
8.60
21.70
310.38
[Higgins et al., 2017]
[Kim and Mnih, 2018] [Chen et al., 2018]
[Eastwood and Williams, 2018]
approach. The DCI disentanglement metric [Eastwood and Williams, 2018] weakly agrees with the modularity metrics. However, the DCI informativeness metric [Eastwood and Williams, 2018] weakly disagrees with the informativeness metrics. It is possible that this is because of the different regressors (sklearn. enwemble.eeeee..eemmmm....e. e wrr.diinntoosttinnnllsssttssttte ..inear__aaa model.
sklearn.1inn linear_men model. Lit..iiiieeeeee LinearRegression, and sklearn [Pedregosa al., 2011)) used in predicting factors from codes, showing that random seeds and hyperparameters of the metrics may matter more than the models when additional predictors need be trained evaluate the learning methods. This result indicates the advantage of Oinjective over Gremactablee
However, it is important to note the limitations of these experimental results. Since the representations were learned from data and not fully controlled, it is possible that such results are due to the choices of datasets, learning algorithms, hyperparameters, and optimization errors. A high rank correlation coefficient between two metrics in this specific setting cannot guarantee that these metrics always measure the same property, that they rank imperfect representations similarly in other settings. To gain a deeper understanding of these metrics, it is preferable to analyze their minimizers theoretically (Theorem 1) or test them in fully controlled environment (Section 5).
E.5 Computation time of metrics
Finally, we report the computation time of the considered metrics in Table 7 to support our claim that the proposed metrics are much faster than those that require training additional predictors and hyperparameter tuning. We can see that, in an extreme case, the calculation of the DCI metrics [Eastwood and Williams, 2018] using GradientBoostingglassiiee [Pedregosa around 15 minutes, while other metrics can be calculated within seconds. This computation time may be acceptable if the metrics are only used in the evaluation phase, but it is not feasible use them as learning objectives even in derivative-free optimization.
al., 2011 1] takes
52

---

Table 8: Factor-wise modularity metrics on 3D Cars [Reed al., 2015]
Product approx.
Constancy
Rad.
MAD
Var.
Diam.
MPD
Elevation (4)
VAE
0.49
0.87
0.97
0.24
0.90
3-VAE
0.52
0.86
0.97
0.27
0.90
Factor
0.50
0.86
0.97
0.25
0.90
3-TCVAE
0.50
0.87
0.97
0.25
0.90
Azimuth (24)
VAE
0.62
0.91
0.99
0.39
0.94
3-VAE
0.60
0.91
0.98
0.37
0.93
Factor VAE
0.57
0.90
0.98
0.32
0.93
3-TCVAE
0.65
0.91
0.99
0.42
0.94
Object (183)
VAE
0.87
0.97
1.00
0.75
0.98
3-VAE
0.84
0.97
1.00
0.71
0.98
Factor VAE
0.85
0.97
1.00
0.72
0.98
3-TCVAE
0.86
0.97
1.00
0.75
0.98
E.6 wistor-wise modularity metrics
An advantage of the proposed modularity metrics is that which is impossible for those metrics that entangle modularity and informativeness. The results were reported in Tables but underperform on others, and different modularity metrics may rank learning methods differently (see also Appendix D.5).
evaluate each factor separately,
11. We found that some learning methods may outperform others one factor
For example, on MPI3D [Gondal et al., 2019] (Table 11), B- B-V [Higgins et al., 2017] has the highest scores (radius, MAD, variance, diameter, and MPD) on the horizontal and vertical axis factors, but the lowest scores (radius and diameter) on the object size, camera height, and background color factors. However, measured by the MAD and MPD, it still has the highest scores these factors. This means that VAE may generally encode the object size, camera height, and background color well compared to other considered methods, but has small number of outliers. We believe that such fine-grained evaluation can guide the design of learning objectives, data collection, and further refinement of trained representation learning models.
53

---

Table 9: Factor-wise modularity metries on dSprites [Matthey al., 2017]
Product approx.
Constancy
Rad.
MAD
Var.
Diam.
MPD
Shape (3)
VAE
0.74
0.89
0.98
0.55
0.92
3-VAE
0.70
0.88
0.98
0.50
0.92
Factor VAE
0.73
0.89
0.98
0.53
0.92
3-TCVAE
0.72
0.88
0.98
0.53
0.92
Scale (6)
VAE
0.71
0.90
0.98
0.51
0.93
3-VAE
0.55
0.88
0.98
0.30
0.92
Factor VAE
0.67
0.90
0.98
0.45
0.93
3-TCVAE
0.77
0.90
0.98
0.59
0.93
Orientation (40)
VAE
0.92
0.97
1.00
0.84
0.98
3-VAE
0.86
0.98
1.00
0.74
0.98
Factor VAE
0.95
0.98
1.00
0.90
0.99
3-TCVAE
0.88
0.97
1.00
0.77
0.98
Position X (32)
VAE
0.71
0.90
0.98
0.51
0.93
3-VAE
0.66
0.92
0.99
0.43
0.95
Factor VAE
0.62
0.89
0.98
0.39
0.92
3-TCVAE
0.67
0.91
0.99
0.44
0.94
Position Y (32)
VAE
0.68
0.91
0.99
0.47
0.94
3-VAE
0.66
0.92
0.99
0.44
0.94
Factor VAE
0.64
0.90
0.98
0.41
0.93
3-TCVAE
0.67
0.90
0.98
0.45
0.93
54

---

Table 10: Factor-wist modularity metrics on 3D Shapes [Burgess and Kim, 2018]
Product approx.
Constancy
Rad.
MAD
Var.
Diam.
MPD
Floor hue (10)
VAE
0.85
0.97
1.00
0.72
0.98
3-VAE
0.87
0.97
1.00
0.75
0.98
FactorVAE
0.75
0.93
0.99
0.57
0.95
3-TCVAR
0.99
1.00
1.00
0.97
1.00
Wall hue (10)
VAE
0.85
0.97
1.00
0.73
0.98
3- VAE
0.94
0.99
1.00
0.87
0.99
FactorVAB
0.78
0.94
0.99
0.60
0.96
3-TCVAE
0.82
0.94
0.99
0.68
0.96
Object hue (10)
VAE
0.77
0.95
0.99
0.59
0.96
3-VAE
0.75
0.92
0.99
0.56
0.95
FactorVAE
0.75
0.93
0.99
0.56
0.95
3-TCVAE
0.77
0.92
0.99
0.59
0.95
Scale (8)
VAE
0.80
0.95
0.99
0.65
0.96
3- VAE
0.56
0.91
0.98
0.32
0.93
FactorVAE
0.79
0.93
0.99
0.63
0.95
3-TCVAE
0.80
0.95
1.00
0.63
0.97
Shape (4)
VAE
0.59
0.91
0.98
0.35
0.93
3-VAE
0.62
0.91
0.98
0.38
0.93
FactorVAE
0.77
0.93
0.99
0.60
0.95
3-TCVAE
0.69
0.92
0.98
0.47
0.94
Orientation (15)
VAE
0.95
0.99
1.00
0.91
0.99
3- VAE
0.94
0.99
1.00
0.89
0.99
FactorVAE
0.89
0.98
1.00
0.80
0.99
3-TCVAE
0.72
0.91
0.98
0.52
0.94
55

---

Table 11: Factor-wist modularity metries on MPI3D [Gondal et al., 2019]
Product approx.
Constancy
Rad.
MAD
Var.
Diam.
MPD
Object color (6)
VAE
0.52
0.92
0.99
0.27
0.94
3-VAE
0.51
0.97
0.99
0.26
0.97
Factor VAE
0.66
0.94
0.99
0.43
0.96
3-TCVAE
0.57
0.92
0.99
0.33
0.94
Object shape (6)
VAE
0.88
0.97
1.00
0.77
0.98
3-VAE
0.94
1.00
1.00
0.89
1.00
Factor VAE
0.93
0.99
1.00
0.87
0.99
3-TCVAE
0.95
1.00
1.00
0.91
1.00
Object size (2)
VAE
0.70
0.93
0.99
0.49
0.95
3-VAE
0.63
0.98
1.00
0.40
0.98
Factor VAE
0.68
0.94
0.99
0.46
0.95
3-TCVAE
0.75
0.96
1.00
0.56
0.97
Camera height (3)
VAE
0.48
0.87
0.97
0.23
0.90
3-VAE
0.34
0.95
0.99
0.11
0.96
Factor VAE
0.58
0.85
0.96
0.34
0.90
3-TCVAE
0.69
0.91
0.99
0.47
0.94
Background color (3)
VAE
0.60
0.92
0.99
0.36
0.94
3-VAE
0.34
0.96
0.99
0.11
0.97
Factor VAE
0.76
0.94
0.99
0.58
0.96
3-TCVAE
0.59
0.93
0.99
0.34
0.95
Horizontal axis (40)
VAE
0.72
0.93
0.99
0.52
0.95
3-VAE
0.74
0.99
1.00
0.55
0.99
Factor VAE
0.69
0.92
0.99
0.47
0.94
3-TCVAE
0.73
0.94
0.99
0.54
0.96
Vertical axis (40)
VAE
0.67
0.91
0.99
0.45
0.94
3-VAE
0.75
0.99
1.00
0.56
0.99
Factor VAE
0.73
0.93
0.99
0.53
0.95
3-TCVAE
0.60
0.93
0.99
0.36
0.95
56

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: "A theoretical connection between logical definitions of disentanglement and quantitative metrics" was detailed in Appendices A and B. "A systematic approach for converting a first-order predicate into a real-valued quantit" was introduced in Section 3. "The metrics induced by logical definitions," " which were introduced in Section 4. "have strong theoretical guarantees. which was supported by Theorem 1. "The effectiveness of the proposed metrics" was empirically validated in Section 5 "by isolating different aspects of disentangled representations" and further investigated in Appendix E. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We discussed the limitations in Appendix D.7. Another limitation is that we only studied the properties of the minimizers of the metrics (Theorem 1). Research the behavior of these metries on imperfect representations is limited (Appendix D.5). We claimed that some proposed metrics can serve as differentiable learning objectives, but empirical evaluation is out of the scope of this work. Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., the approach was only tested few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
57

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification; All definitions of the proposed terms, theoretical results, and detailed proofs were provided in Appendices A to C. The main theoretical result was stated in Theorem and instantiated in Section 4.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The contribution is primarily the new theoretical framework. For experiments, we provided code snippets in Appendix D.6. We provided the detailed data configuration used in Section in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce
the model (c.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access the model is limited in some way (e.g., ., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results. 5. Open access to data and code
58

---

Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provided the code to reproduce the modularity metrics in Table 2. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/pu
blic/ htc//uides//// Sguidee/Coeeuumin htt///ooeSubiiiiseee wwsssssiooloooooooooooooo PapPolllyyy....... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines for more
(https: http:/ipsscccpuupp details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results of the proposed metrics reported in Table 2 are deterministic and do not contain randomness. We reported the mean values of 10 random trials in Table 4, and the variances are negligible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
59

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should careful not to show tables or figures symmetric enror bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See Appendix E. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpps:/neeutt httpubicc/thiiccsiiiiiiiiiciiaii Answer: [Yes]
Justification: We reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix D.8.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
60

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Justification: This paper mainly focused on the theory of disentangled representation learning.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with
necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com/ da tasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
61

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
62


Reviews:
[{"review_id": "FKqfFcuExP", "reviewer": "Reviewer_GqiT", "summary": "This study introduces a systematic approach to quantify properties of representation learning models. By translating logical definitions into quantitative metrics, the paper evaluates two key properties: modularity and informativeness. Two sets of metrics are derived for each property, one based on approximation and the other on distance computation and aggregation. Theoretical analysis of these metrics is conducted, including examination of their minimizers. \n\n(Full disclosure: I have reviewed this paper before)", "questions": "1. Heyting algebra is mentioned a couple of times in the paper without a proper definition, which would be good for general ML audience.", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. I think this paper is interesting and innovative and mathematically sound.\n2. The paper is well-written. The math objects are introduced with good intuitions.", "weaknesses": "1. It might be difficult for people with less prior knowledges to read. \n2. It’s unclear if the metrics is practical be to a loss to optimize in practice because the experiments do not train with proposed metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks again for your kind evaluation of our work!\nRegarding your concerns, our answers are as follows.\n\n---\n    It might be difficult for people with less prior knowledges to read. \n---\n\nThank you for pointing this out.\nIn the main body of this paper, we tried our best to avoid using abstract categorical concepts, which were necessary for developing the theory. The proposed technique was explained as simple rules such as \"*replacing the conjunction (logical AND) with the addition*\". We believe that the reader does not need any category theory background to understand the proposed metrics.\n\nTo make the theory more accessible, we added a **non-categorical restatement** of the proposed theory in the revised version, which is less general but easier to follow. However, we have to introduce some necessary algebraic concepts such as homomorphism. We hope this part can help the reader understand not only the proposed technique but also the theory behind it.\n\n---\n    It’s unclear if the metrics is practical be to a loss to optimize in practice because the experiments do not train with proposed metrics.\n---\n\nLearning disentangled representations with the proposed metrics is indeed the immediate step after this work. We would like to point out that some existing metrics are much less practical as learning objectives for two reasons:\n\n1. Some metrics such as DCI `[Eastwood and Williams, 2018]` need to train an additional predictor, which requires hyperparameter tuning and more computation. Evaluation using these metrics is time-consuming. In some cases, the calculation of the DCI metrics using `GradientBoostingClassifier` takes around 15 minutes (See **Appendix F.5**). It is not feasible to use them as learning objectives, even in derivative-free optimization.\n2. For the same reason, many existing metrics are not differentiable, so it is impossible to use them in gradient-based optimization.\n\nTo use the proposed metrics to learn representations, we need to consider what supervision can be used, which is a different topic and beyond the scope of this work.\n\n\n---\n    Heyting algebra is mentioned a couple of times in the paper without a proper definition, which would be good for general ML audience.\n---\n\nWe mentioned Heyting algebra and quantale so that a reader with an algebra background can immediately understand our main idea. It is like saying *group* instead of *a set equipped with an associative, unital, and invertible binary operation*. However, we believe that a machine learning researcher who is not familiar with algebra can still understand the proposed technique because we spelled out all the components of a Heyting algebra (true, false, conjunction, implication, etc.). Roughly speaking, a Heyting algebra may be considered as a **Boolean algebra** without excluded middle. For interested readers, the formal definition of Heyting algebra is given in **Appendix B**.\n\n---\n\nWe hope we addressed your concerns.\n\nPlease let us know if you have any other questions.\nThank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gzIIT45Lq0", "reviewer": "Reviewer_4yVc", "summary": "The paper proposed to establish a connection between logical definitions of disentanglement and quantitative metrics from the perspective of typos theory and category theory. It then propose a metrics for disentanglement with stronger theoretical guarantees and compared it with some state of the art metrics.", "questions": "1, why were the specific modularity metrics used for evaluation? How do they contribute to assessing disentanglement in representation learning?\n2, Line 308 mentioned that the results are transformed isomorphically using $e^{-x}$. What is the purpose of this procedure?", "limitations": "The authors adequately addressed the limitations", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper provided theoretical justification for establishing metrics from typos theory and category theory in disentanglement of representation learning. It then proposed to convert first-order predicate into real-value quantity, which is innovative. The proposed metrics also have practical implications as the author mentioned differentiability and its effectiveness through some experimental results.", "weaknesses": "1, Despite that the authors included as much background information in the appendix, the advanced mathematical concepts makes the paper very hard to follow and limit its accessibility. Moreover, the theoretical connections may not be readily feasible to practical and intuitive interpretations. Hence its use scenarios should be stated. \n2, Even though the paper provides empirical results, its scope and effectiveness may not be fully demonstrated through the limited range of scenarios considered. Therefore its generalizability it not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the innovation of our work!\nWe will answer your questions as follows.\n\n---\n    ... the advanced mathematical concepts makes the paper very hard to follow and limit its accessibility.\n---\n\nThank you for pointing this out.\nWe developed the theory with the help of these mathematical concepts, and we are aware that they are not as well known as other tools such as linear algebra and statistics for machine learning researchers. Therefore, in the main body of this paper, we tried to present the proposed technique (**Table 1**) and theorem (**Theorem 1**) without these terms. We simplified the complex structures into easy-to-follow rules, such as \"*replacing the conjunction (logical AND) with the addition*\", and used many examples to demonstrate the application of this technique. We believe that the reader does not need any category theory background to understand the proposed metrics.\n\nTo make the theory more accessible, we added a **non-categorical restatement** of the proposed theory in the revised version, which is less general but easier to follow. However, we have to introduce some necessary algebraic concepts such as *homomorphism*. We hope this part can help the reader understand not only the proposed technique but also the theory behind it.\n\n---\n    1, why were the specific modularity metrics used for evaluation? How do they contribute to assessing disentanglement in representation learning?\n---\n\nWe evaluated not only modularity but also informativeness (Section 3.3, the middle 5 columns in Table 2). Modularity is arguably the essential property of disentanglement (it is simply called *disentanglement* in `Eastwood & Williams [2018]`), informativeness is a basic property of representation learning, and they are considered to be more important than other properties such as completeness/compactness in practice `[Ridgeway and Mozer 2018, Duan et al. 2020, and Carbonneau et al. 2022]`.\n\n---\n    2, Line 308 mentioned that the results are transformed isomorphically using e^{-x}. What is the purpose of this procedure?\n---\n\nOriginally, we used ($[0, \\infty]$-valued) strict premetrics as the quantitative versions of equality. In this case, $0$ means a property holds, and non-zero numbers mean that a property does not hold (*the lower the better*). However, many existing metrics used $[0, 1]$-valued metrics, where $1$ means a property holds perfectly (*the higher the better*). A reader may feel a subtle cognitive dissonance if we compare them directly. Therefore, we transformed $[0, \\infty]$-valued metrics into a $[0, 1]$-valued metrics using $e^{-x}: [0, \\infty] \\to [0, 1]$.\nWe say it's an isomorphism in the sense that it preserves the order ($a \\leq b$ implies $e^{-a} \\geq e^{-b}$), quantitative operations such as the addition ($e^{-(a + b)} = e^{-a} \\cdot e^{-b}$), and so on.\nIf we don't need to compare the results with existing $[0, 1]$-valued metrics, we can use the original values (e.g., for optimization).\n\n---\n\nWe hope these answers address your concerns.\n\nPlease let us know if you have any other questions.\nThank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FFH6JMVn8V", "reviewer": "Reviewer_GEk9", "summary": "The paper consider the connection between logical definitions and quantitative metrics, proposing a systematic approach to design metrics from logical definitions. Particularly, the paper is focused on the measure of disentanglement. The paper theoretically justifies the correspondence between logical definitions of disentanglement and quantitative metrics via topos theory and enriched category theory and empirically demonstrates the effectiveness in isolating various aspects of disentangled representations compared to existing metrics.", "questions": "Could you provide more evidence, either empirical or theoretical, to show the proposed metrics dominate the existing ones universally? Or if the proposed metrics do not outperform the existing ones universally, when are they better?", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea presented in the paper, especially the systematic method of converting logical predicates into quantitative metrics, is novel and very interesting. The authors also theoretically support this idea with topos theory and enriched category theory, which are sophisticated and advanced mathematical frameworks not commonly employed in machine learning literature.", "weaknesses": "The superiority of the proposed metrics over the the existing ones does not seem fully validated, either empircally or theoretically. The evidence provided does not show the proposed metrics dominate the existing ones universally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of the novelty and soundness of this work!\nWe'd like to address your concerns as follows.\n\n---\n    The superiority of the proposed metrics over the the existing ones does not seem fully validated, either empircally or theoretically. The evidence provided does not show the proposed metrics dominate the existing ones universally.\n---\n\nPlease let us summarize the benefits of the derived metrics over the existing ones:\n\n### **No failure modes**\n\nFirst, the biggest advantage is that the derived metrics are rooted in the logical definitions of the property we want to quantify and are governed by **Theorem 1**. Therefore, we know that minimizing these metrics won't lead to wrong answers. In contrast, several existing metrics have \"failure modes\", which means that these metrics can wrongly score representations (see `Carbonneau et al., [2022]`, `Mahon et al., [2023]`). This was theoretically supported by **Theorem 1** and empirically validated in **Table 2**.\n\n### **Differentiability**\n\nIn the literature, a new evaluation metric is often introduced along with a new representation learning method `[Carbonneau et al., 2022]`, but it is usually unproven that the method can optimize the new metric. One reason is that many existing metrics are not differentiable (because, e.g., they need to train a predictor `[Higgins et al., 2017, Kim and Mnih, 2018, Eastwood and Williams, 2018]`) so we cannot directly optimize them. By choosing specific aggregators, we can obtain differentiable metrics for a property, which allows us to directly optimize the property using gradient-based optimization. This was supported by the existence of differentiable metrics in **Eqs. (17), (25), and (27)**.\n\n### **Weakly supervised evaluation**\n\nWe revealed that it is possible to evaluate some modularity metrics using only similarity information because the modularity is invariant to bijections. This is impossible for some existing metrics. We demonstrated weakly supervised modularity metrics in **Appendix F.2**.\n\n### **Efficiency**\n\nSome existing metrics such as DCI need to train an additional predictor, which requires hyperparameter tuning and is usually computationally expensive. High computation cost may be acceptable if the metrics are only used in the evaluation phase, but it is not feasible to use them as learning objectives even in derivative-free optimization. In contrast, the derived metrics are more efficient. This was empirically supported by **Table 7** in **Appendix F.5**.\n\n### **Fine-grained evaluation**\n\nWe proposed to evaluate modularity and informativeness separately. **Table 2** shows that existing single-score metrics cannot distinguish modularity and informativeness, which provides less information about the representations. We further demonstrated how to use the proposed metrics to diagnose issues of representation learning methods in a more fine-grained manner in **Appendix F.6**.\n\n---\n\nWe will further clarify the benefits of the proposed metrics in the revised version.\n\nPlease let us know if you have any other questions or suggestions.\nThank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H7LAliXOkH", "reviewer": "Reviewer_sy6j", "summary": "This paper investigates relating logical definitions of disentanglement and existing quantitative metrics via formal derivations of novel metrics.", "questions": "see above", "limitations": "see above", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The topic is very interesting and it seems the authors were very rigorous in their investigations especially concerning the amount of all of the background material and derivations.", "weaknesses": "However, the paper is very poorly motivated, structured and written. It is very difficult to follow the authors along what they are trying to achieve (the motivation), what they are doing (their ideas) and the details on how this is connected with other research (context via related work) and how this should be used for future ML research/ what the significance is of their work for the future. E.g. the Introduction dives right into formal background notations without any motivation/overview of what the authors wish to achieve/investigate in the work. \n\nMaybe I have misunderstood some things, but what exactly are the results of the section 3? I.e which of the many metrics are the relevant ones and what do they mean intuitively? The authors provide many  examples of specific “implementations” (for lack of a better word here) and then conclude by statements such as “Upon analyzing the metrics above, it becomes evident that what we need is not the best approximation  itself (e.g., the mean) but rather the approximation error “ on page 7. So it seems the previous derivations are irrelevant for the overall goal (what this is is further unclear). So I wonder whether we really require this material then and can remove it, e.g., for the sake of more details on experiments (see below). \n\nOverall, I noticed the supplementary materials has much more relevant information that I believe needs to be in the main text. E.g. related works as these are necessary to understand the context of the work. \n\nFurther, it was also very difficult for me to understand the experiments of the main paper. The experimental setup is not described in the main paper (there is not even a caption to Table 2). This needs to be greatly improved. There seem to be a lot of information in the supplementary concerning experiments and I would suggest moving this into the main paper. Overall, it would help me if the authors could summarize again what they were investigating with these experiments.\n\nThe claims made in the experiments of the main paper are not backed up by the experiment shown in Table 2, e.g. \"The metrics derived from equivalent definitions may differ in terms of computation cost and differentiability.\". Where are the results for this? If these can only be found in the supplementary currently, I think they should also be moved to the main text.\n\nI am also puzzled by all of the experiments in the supplementary material (on interesting datasets), but that are not mentioned in the main text (unless I have missed this). Was there a reason for this? \n\nLastly, I am missing a thorough discussion of the overall findings and particularly potential limitations of the findings/proposed metrics.\n\nOverall, it is too difficult to assess the quality of the work and proposed ideas with the current structure. I would recommend putting some time into rewriting and restructuring to make this more clear to the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed review of our work!\nWe'd like to address your concerns as follows.\n\n---\n    However, the paper is very poorly motivated, structured and written. It is very difficult to follow the authors along what they are trying to achieve (the motivation), what they are doing (their ideas) and the details on how this is connected with other research (context via related work) and how this should be used for future ML research/ what the significance is of their work for the future. E.g. the Introduction dives right into formal background notations without any motivation/overview of what the authors wish to achieve/investigate in the work.\n---\n\nPlease allow us to clarify the motivation and why we chose the current presentation.\n\nOne of the major problems in the current research of disentangled representation learning is that we lack a clear, logical definition of properties such as modularity and informativeness, and **we don't know if a metric truly quantifies a property or not** (abstract `l.4`).\nThis is the research problem we want to address in this paper.\n\nWe started the introduction section with a discussion on the parallel between loss function and function equality, which all machine learning researchers are familiar with, to show our goal: \"**measuring and optimizing other properties like this**\" (`l.29`) *by connecting logical definitions and quantitative metrics and extending this parallel to other properties*.\nThis is the main idea of this paper.\n\nWe used informativeness as an example in Section 1.1, which is an important property in (disentangled) representation learning.\nWe used this example to show that there could be multiple ways (e.g., injectivity and left-invertibility) to formally define a property (e.g., informativeness), and we need to deal with **logical operations** (e.g., implication), **quantifiers** (e.g., universal quantifier and existential quantifier), composition, and identity, which are the **basic building blocks** that will be used in the following sections.\n\nWe then contextualized our work in Section 1.2 (and further explained related work in detail in Appendix D), gave a logical definition of an essential property of disentanglement (modularity), provided a concrete example to help the reader understand this concept, and introduced the research questions we want to answer: **how to find a (preferably differentiable) metric for a logically defined property**.\nDue to the page limit, we had to move the detailed discussions on related work to Appendix D.\nIf we have a chance, we will move some of them back to the introduction section.\n\nIn Section 2, we immediately answered the questions we asked, and we summarized the proposed technique in **Table 1**.\nThe \"meta-theorem\" was stated in **Theorem 1**.\nThe rest of this paper is just explanations and concrete instantiations of this technique.\nWe tried our best to present this work in a way that a machine learning researcher may get the gist of this work by reading the first three pages, and a practitioner can implement the derived metrics using the technique summarized in Table 1.\n\nTherefore, the introduction section is not just the notation; We chose the examples very carefully to show the **goal, motivation, basic concepts, research questions, and our main idea**.\nWe are aware that this may not be a conventional way to present a work in the machine learning community, but we believe that the current \"*let the math speak for itself*\" style is more suitable for this work and straight to the point.\nWe are glad that `Reviewer GqiT` stated that `The paper is well-written. The math objects are introduced with good intuitions`, but we also know that this writing style is not for every reader.\nWe will try our best to further improve the readability of the introduction section for the machine learning audience.\n\n---\n    Maybe I have misunderstood some things, but what exactly are the results of the section 3?\n---\n\nSection 3 demonstrated the use of the technique proposed in Section 2 and presented the modularity metrics (**Propositions 2 and 3**), their concrete instantiations (**Eqs. (17) and (25)**), informative metrics (**Definitions 8 and 9**), and an example of Theorem 1 (**Eq. (28)**).\n\n\"We need the approximation error but not the approximation itself\" can be interpreted as \"we can calculate the variance without calculating the mean\".\n**Metrics derived from logically equivalent definitions and using different aggregators may have different computational costs, sensitivity to outliers, and learning dynamics.**\nThey are all relevant and useful in different scenarios (e.g., robustness evaluation, gradient-based optimization), and further research is needed to understand their characteristics.\nPlease see Appendix E.3 for further explanation.\n\n---\n    Further, it was also very difficult for me to understand the experiments of the main paper.\n---\n\nThe experiments in the main text were meant to confirm Theorem 1: **minimizers of the derived metrics must satisfy the properties the metrics quantify**.\nWe focused on the theoretical exposition of our work, and the experiments were indeed supplementary to demonstrate the benefits of the derived metrics.\nWe believe that the proposed logic-metric theory is our main contribution, which was supported by the formal proofs.\nNevertheless, we will follow your suggestions and clarify the meaning of experiments by moving some materials from the supplementary material to the main text in the revised version.\nThank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper is a borderline case.

The positive end is that the authors are taking a very different approach to studying the notion of disentanglement. In general, I think (and the reviewers agreed) that such creative, "large step" research is valuable. 

A particular aspect of such research is that one paper often cannot fully explore the direction. That means that, as some reviewers noted, the empirical significance is questionable, and indeed it's somewhat unclear what the methodological implications may be. Generally then, the paper relies on its mathematical development for a priori motivation of the approach (rather than testing predictions as a posteriori motivation). I think this is fine for a first paper on a novel direction.

The trouble is that no reviewer feels qualified to assess the mathematical content. Indeed, I also do not feel qualified. 

I am recommending acceptance on the basis that novel and ambitious approaches to hard problems should be encouraged. However, I emphasize to the authors that for this line of work to have impact, significant follow up work will be required to "bring it down from the mountain" and convincingly demonstrate its practical importance.
Decision: Accept (poster)