Title: Enriching Disentanglement: From Logical Definitions to Quantitative Metrics
Venue: NeurIPS 2024
Accepted: yes
Keywords: Disentanglement

Representation Learning

Logic

Metric

Algebra

Category Theory

Topos Theory
Abstract: Disentangling the explanatory factors in complex data is a promising approach for generalizable and data-efficient representation learning. While a variety of quantitative metrics for learning and evaluating disentangled representations have been proposed, it remains unclear what properties these metrics truly quantify. In this work, we establish algebraic relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metrics. Concretely, we introduce a compositional approach for converting a higher-order predicate into a real-valued quantity by replacing (i) equality with a strict premetric, (ii) the Heyting algebra of binary truth values with a quantale of continuous values, and (iii) quantifiers with aggregators. The metrics induced by logical definitions have strong theoretical guarantees, and some of them are easily differentiable and can be used as learning objectives directly. Finally, we empirically demonstrate the effectiveness of the proposed metrics by isolating different aspects of disentangled representations.

Enriching Disentanglement: From Logical Definitions to Quantitative Metrics

Abstract

Disentangling the explanatory factors in complex data is a promising approach for generalizable and data-efficient representation learning. While a variety of quantitative metrics for learning and evaluating disentangled representations have been proposed, it remains unclear what properties these metrics truly quantify. In this work, we establish algebraic relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metrics. Concretely, we introduce a compositional approach for converting a higher-order predicate into a real-valued quantity by replacing (i) equality with a strict premetric, (ii) the Heyting algebra of binary truth values with a quantale of continuous values, and (iii) quantifiers with aggregators. The metrics induced by logical definitions have strong theoretical guarantees, and some of them are easily differentiable and can be used as learning objectives directly. Finally, we empirically demonstrate the effectiveness of the proposed metrics by isolating different aspects of disentangled representations.

1 Introduction

In supervised learning , we usually use a real-valued cost function ℓ : Y × Y → R ≥ 0 to measure how close an output f ( x ) of a function f : X → Y is to a target label y , i.e., ℓ ( f ( x ) , y ) , to quantify the cost of inaccurate prediction. Then, we can use the total cost over a collection of input-output pairs to measure the performance of this function. From a functional perspective, this construction induces a quantitative metric L : [ X,Y ] × [ X,Y ] → R ≥ 0 between functions: 1

where g : X → Y is a ' ground-truth function ' that maps each input x to its target label y . This metric can be used as both learning objective and evaluation metric for the learning model f : X → Y . What does L ( f, g ) quantify? It quantifies the extent to which two functions f and g are equal :

Considering the equality as a predicate, we can observe a parallel between

(binary-valued) equality = Y : Y × Y →{⊤ , ⊥} and (real-valued) cost ℓ : Y × Y → R ≥ 0 , 3

function equality = [ X,Y ] : [ X,Y ] × [ X,Y ] →{⊤ , ⊥} and total cost L : [ X,Y ] × [ X,Y ] → R ≥ 0 .

universal quantifier (' for all ') ∀ x ∈ X and summation ∑ x ∈ X , and

We would like to ask: Is it possible to measure and optimize other properties in the same way?

Masashi Sugiyama

In representation learning [Bengio et al., 2013], measuring and optimizing the performance of a learning model becomes a non-trivial task. The quality of a model cannot always be measured by how close it is to a fixed ground truth. Instead, we often need to consider the properties of the model architecture or learned representation itself, such as convexity [Amos et al., 2017], uniformity [Wang and Isola, 2020], invariance [Kvinge et al., 2022], and equivariance [Lee et al., 2019, Brehmer et al., 2023]. A proper comprehension of what constitutes good representations and how to assess their quality is important for designing suitable models, learning objectives, and evaluation metrics.

Disentangled representation learning: definitions, metrics, and methods Disentanglement is an important property in representation learning, which intuitively means that different explanatory factors in data should be encoded separately [Bengio et al., 2013]. However, disentanglement has no universally agreed-upon formal definition [Higgins et al., 2018, Suter et al., 2019, Shu et al., 2020, Fumero et al., 2021], and it is typically viewed not as a single property but rather as a combination of several requirements [Ridgeway and Mozer, 2018, Eastwood and Williams, 2018, Do and Tran, 2020, Tokui and Sato, 2022]. While many metrics for measuring disentanglement have been proposed [Carbonneau et al., 2022], it remains unclear what properties these metrics truly quantify and how they can be optimized directly. Often, a new evaluation metric is introduced along with a new learning method, but it is usually unproven that the method can optimize the new metric [Higgins et al., 2017, Kim and Mnih, 2018, Chen et al., 2018, Li et al., 2020]. This lack of theoretical understanding makes it difficult to design learning models that can effectively learn disentangled representations.

A logical and algebraic approach to defining and measuring disentangled representations Recently, Zhang and Sugiyama [2023] proposed a general and abstract definition of disentanglement, shedding light on the common structures underlying the algebraic, statistical, and topological definitions of disentanglement. It was shown that the abstract concept of product [Mac Lane, 1978] underlies an essential property of disentanglement called modularity [Ridgeway and Mozer, 2018], and other properties of learning models, such as informativeness [Eastwood and Williams, 2018], can also be defined abstractly using only the composition and identity of morphisms. Following this algebraic approach, we aim to derive theoretically grounded quantitative metrics of disentanglement from the logical definitions of the desired properties, extending the parallel between Eqs. (1) and (2).

Contributions In this paper, we focus on logically defined properties of disentangled representation learning, such as modularity and informativeness (Section 2). We introduce a compositional approach to converting a higher-order equational predicate into a real-valued quantity (Table 1), which serves as a quantitative metric of the extent to which a function satisfies the predicate (Section 3). Our analysis on the relationship between the logical definitions and the induced quantitative metrics provides theoretical guarantee on the properties of the optimal functions (Theorem 1). Then, we demonstrate the usefulness of this conversion method by deriving quantitative metrics for measuring properties of disentangled representations, and we analyze these metrics in terms of computation, optimization, and differentiability (Section 4). Lastly, we compare the derived metrics with several existing ones in a fully controlled experiment and demonstrate that the proposed metrics are able to isolate different aspects of disentangled representations (Section 5).

2 Logical definitions of disentangled representations

In this section, let us first take a closer look at the logical definitions of two properties of disentangled representation learning - informativeness [Eastwood and Williams, 2018] and modularity [Ridgeway and Mozer, 2018], which are arguably more important than other properties [Carbonneau et al., 2022]. We limit our discussion to sets and functions, but the generalization to other morphisms, such as equivariant, stochastic, or continuous functions, is straightforward.

2.1 Informativeness: injectivity or retractability of a learning model

Being informative, expressive, faithful, or useful is a basic requirement for learned representations [Bengio et al., 2013]. We want a representation learning model to preserve explanatory factors in data that are informative to the downstream tasks. For functions, this criterion could be formulated as follows: If two factors y and y ′ are different, then their representations m ( y ) and m ( y ′ ) extracted by a function m : Y → Z should be different too. This means that the function m should be injective :

Definition 1 (Injective function) . A function m : Y → Z is injective if

Alternatively, because injective functions are precisely functions with retractions (left inverses) [Lawvere and Rosebrugh, 2003, Chapter 2], we can measure the retractability instead:

Definition 2 (Retractable function) . A function m : Y → Z has a retraction h : Z → Y if

Note that these properties are predicates p injective , p retractable : [ Y, Z ] →{⊤ , ⊥} on the set [ Y, Z ] of all functions from Y to Z . Analogous to using the total cost in Eq. (1) to measure function equality in Eq. (2), if we want to measure the injectivity in Eq. (3) or retractability in Eq. (4), we need to find quantitative counterparts of the implication → , universal quantifier ∀ , and existential quantifier ∃ used in their logical definitions. Generally, it is desirable to extend the parallel between Eqs. (1) and (2) to other predicates by finding quantitative operations corresponding to logical connectives and quantifiers. This correspondence allows us to construct and analyze quantitative metrics for machine learning models in a compositional manner [Boole, 1854].

2.2 Modularity: product structure preserved by a learning model

Modularity [Ridgeway and Mozer, 2018] is an essential property of disentangled representation learning, which means that the explanatory factors in data, such as the color and shape of an object, are separated into independent components in the learned representation [Bengio et al., 2013].

As shown in Fig. 1, modularity can be defined as follows. We assume that data with multiple explanatory factors (e.g., color and shape) is generated via a function g : Y → X from a product Y := Y 1 × Y 2 of factors . An encoder f : X → Z is a function to a product Z := Z 1 × Z 2 of codes . Then, an encoder is said to be modular if it can reconstruct the product structure , such that the composition m := f ◦ g : Y → Z of the generator g and the encoder f is a product function. 4

red

circle

blue

circle

red

triangle

blue

triangle

red

blue

circle

triangle

(1

,

0)

(0

,

0)

(1

,

1)

(0

,

1)

0

1

0

1

factors

observations

codes

Y

color

×

Y

shape

X

Z

color

×

Z

shape

g

m

:=

f

◦

g

=

m

color

×

m

shape

f

Formally, being a product function is also a property that can be represented as a predicate: Definition 3 (Product function) . Let Y := Y 1 × Y 2 and Z := Z 1 × Z 2 be products of sets. A function m : Y → Z is a product function if

Example 1. Let us compare the following two functions from Y := { 0 , 1 } 2 to Z := R 2 :

where a , b , c , and d are arbitrary real numbers. According to Definition 3, only m ′ = m 1 , 1 × m 2 , 2 is a product function, whose first/second output depends only on the first/second input.

In Example 1, although m is not a product function, we want to address the following questions:

(Metric) Can we quantify the extent to which it resembles a product function?

(Approximation) Can we find a product function that is closest to it?

(Differentiability) Can we make it slightly closer to a product function?

Answers to these questions will be given in the following sections.

3 Enrichment: from logic to metric

In Appendices A and B, we describe in detail the theory of converting a higher-order predicate into a real-valued quantity. In this section, we only introduce the conversion procedure using concrete examples and present the theoretical results. A summary of the conversion is given in Table 1.

First of all, let us clarify the terms predicate and quantity. In the realm of classical logic, a predicate p : A → {⊤ , ⊥} on a set A is a function from the set A to the set {⊤ , ⊥} of binary truth values. For example, the predicates p injective , p retractable , and p product in Definitions 1 to 3 are functions from the set [ Y, Z ] of functions to the set {⊤ , ⊥} . They are logical definitions of some properties of functions. On the other hand, in this work, a quantity q : A → [0 , ∞ ] on a set A is defined as a function to the set [0 , ∞ ] of extended non-negative real numbers. The quantities associated with a predicate will serve as quantitative metrics for the property defined by the predicate.

3.1 From equality predicate to strict premetric

In this work, a predicate of central importance is the equality predicate = A : A × A → {⊤ , ⊥} [Mazur, 2008]. A quantity associated with the equality predicate should be a strict premetric:

Definition 4 (Strict premetric) . A strict premetric on a set A is a function d A : A × A → [0 , ∞ ] that

3.2 From logical operation to quantitative operation

Next, let us have a look at the logical connectives and quantifiers used in the definitions of properties. The product of sets and functions plays a significant role in this work. Two functions f, g : C → A × B to a product are equal if and only if all their component functions are equal:

Note that the conjunction ∧ : {⊤ , ⊥} × {⊤ , ⊥} → {⊤ , ⊥} , a logical connective, is used in Eq. (9). To obtain a corresponding quantity, we replace it with the addition + : [0 , ∞ ] × [0 , ∞ ] → [0 , ∞ ] :

The universal quantifier on a set A is a specific (second-order) predicate ∀ A : {⊤ , ⊥} A →{⊤ , ⊥} on the set {⊤ , ⊥} A of predicates. We can replace it with the supremum sup : [0 , ∞ ] A → [0 , ∞ ] . We can also choose a function from the (i) maximum , (ii) sum , (iii) mean , and (iv) mean square when the set A is finite. More generally, we can replace it with a quantity ▽ A : [0 , ∞ ] A → [0 , ∞ ] on the set [0 , ∞ ] A of quantities that satisfies some conditions, which we refer to as a (universal) aggregator . Intuitively, a universal aggregator should output 0 if and only if all inputs are 0 . Therefore, the median, mode, and range are non-examples. Different choices of aggregators yield metrics with different characteristics in computation and optimization. For example, the function equality predicate

converts to a quantity whose aggregator ▽ is not limited to the sum (cf. Eqs. (1) and (2)):

Dually, we also need to consider the disjunction ∨ and the existential quantifier ∃ . We replace them with the minimum and the infimum , respectively. Lastly, we replace the implication a → b with the (truncated) subtraction b ´ a := max { b -a, 0 } . These operations are illustrated in Fig. 2.

(a) predicate and quantity

(b) conjunction

(c) disjunction

(d) implication

3.3 From compound predicate to compound quantity

Following Table 1, we can convert any compound predicate defined using equational predicates and logical operations into a corresponding compound quantity defined using strict premetrics and quantitative operations. Our main result on their relationship is as follows:

Theorem 1. Let p : A →{⊤ , ⊥} be a predicate on a set A , and let q : A → [0 , ∞ ] be a quantity converted from p according to Table 1. Then, for any a ∈ A , q ( a ) = 0 implies p ( a ) = ⊤ . Conversely, for any a ∈ A , p ( a ) = ⊤ implies q ( a ) = 0 if and only if p does not contain the implication.

The implication is special because we must sacrifice logical equivalence for the sake of continuity, which is necessary for gradient-based optimization. We will explore this through a concrete example regarding injectivity in Section 4.3 and discuss it in detail in Appendices B and D.

3.4 (Sub)homomorphism from metric to logic

Finally, for readers interested in the theoretical background, we briefly introduce the following algebraic concepts and a proof sketch underlying Table 1 and Theorem 1.

Definition 5 (Zero predicate) . The zero predicate ζ : [0 , ∞ ] → {⊤ , ⊥} := x ↦→ ( x = 0) is a function that maps 0 to true ⊤ and any positive value to false ⊥ .

Definition 6 ((Sub)homomorphism from a quantity to a predicate) . Let A be a set. A quantity q : A → [0 , ∞ ] on the set A is homomorphic to a predicate p : A →{⊤ , ⊥} via the zero predicate ζ : [0 , ∞ ] →{⊤ , ⊥} if ζ ◦ q = p , and is subhomomorphic to p if ζ ◦ q → p . 6

Definition 7 ((Sub)homomorphism from a quantitative operation to a logical operation) . Let n ∈ N be a natural number. An n -ary quantitative operation α : [0 , ∞ ] n → [0 , ∞ ] is homomorphic to a logical operation β : {⊤ , ⊥} n →{⊤ , ⊥} via the zero predicate ζ : [0 , ∞ ] →{⊤ , ⊥} if ζ ◦ α = β ◦ ζ n , and is subhomomorphic to β if ζ ◦ α → β ◦ ζ n . 7

Definition 8 ((Sub)homomorphism from an aggregator to a quantifier) . Let A be a set. An aggregator α A : [0 , ∞ ] A → [0 , ∞ ] on the set A is homomorphic to a quantifier β A : {⊤ , ⊥} A →{⊤ , ⊥} via the zero predicate ζ : [0 , ∞ ] → {⊤ , ⊥} if ζ ◦ α A = β A ◦ ζ A , and is subhomomorphic to β A if ζ ◦ α A → β A ◦ ζ A . 8

Homomorphic quantities, quantitative operations, and aggregators can be illustrated as follows:

Based on these algebraic concepts, we can say that strict premetrics are homomorphic to equality predicates, addition is homomorphic to conjunction (since the sum is zero if and only if both addends are zero), minimum is homomorphic to disjunction, truncated subtraction is subhomomorphic to implication, and universal aggregators are homomorphic to the universal quantifier.

Theorem 1 means that any compound quantity is (sub)homomorphic to the corresponding compound predicate if each component (quantities, quantitative operations, aggregators) is (sub)homomorphic to the corresponding component (predicates, logical operations, quantifiers). For implication, we use the truncated subtraction, which is only subhomomorphic, since there is no continuous operation that is homomorphic to implication (see also Appendix D).

More abstractly and concisely, we can say that we replace the Heyting algebra of truth values {⊤ , ⊥} with a quantale of extended non-negative real numbers [0 , ∞ ] , and we replace the quantifiers ∀ and ∃ with aggregators ▽ and inf (see also Appendix B). In this way, we can derive quantitative metrics for any logically defined properties of learning models compositionally .

4 Quantitative metrics of disentangled representations

In this section, we demonstrate how to apply the conversion method introduced above to derive quantitative metrics for measuring the modularity (Definition 3) and informativeness (Definitions 1 and 2) of disentangled representations.

In Sections 4.1 and 4.2, we introduce modularity metrics based on two approaches and discuss their differences in terms of computation and optimization. We point out that the main obstacle lies in the optimization step, resulting from the existential quantifiers in the definition. Then, we show that we can derive easily computable and differentiable metrics from a logically equivalent definition. In Section 4.3, we introduce informativeness metrics and present a result of Theorem 1.

4.1 Modularity metrics via product approximation

We begin with modularity , which is an essential property of disentangled representation learning. Recall that modularity can be defined using the product function (Definition 3). For easier reference, we provide the following diagram, which shows the domains and codomains of the functions involved in the upcoming discussion:

From Definition 3, we can derive the following metric:

Definition 9 (Product approximation) . Let m : Y → Z be a function from a product Y := Y 1 × Y 2 of sets to another product Z := Z 1 × Z 2 of sets. The extent to which m resembles a product function can be measured by a distance between m and its best product function approximation:

The derivation of q product from p product follows the conversion described in Table 1: replacing the equality = [ Y,Z ] : [ Y, Z ] × [ Y, Z ] →{⊤ , ⊥} with a strict premetric d [ Y,Z ] : [ Y, Z ] × [ Y, Z ] → [0 , ∞ ] and the existential quantifiers ∃ with the infimum operators inf .

This modularity metric can be interpreted as a distance from a point m ∈ [ Y, Z ] to a subset { m 1 , 1 × m 2 , 2 | m 1 , 1 ∈ [ Y 1 , Z 1 ] , m 2 , 2 ∈ [ Y 2 , Z 2 ] } ⊂ [ Y, Z ] of product functions (cf. the Hausdorff distance [Lawvere, 1986, Tuzhilin, 2016]). Following from Theorem 1, q product ( m ) = 0 if and only if p product ( m ) = ⊤ . This means that the minimizers of this metric are precisely product functions.

However, we still face two obstacles: the product operation and the minimization problem. For the product operation, we can employ Eqs. (10) and (12) to rewrite q product into a more computable form:

Proposition 2. The quantity q product ( m : Y → Z ) equals

where the functions m ∗ 1 , 1 : Y 1 → Z 1 and m ∗ 2 , 2 : Y 2 → Z 2 are given by

A detailed derivation can be found in Appendix C. Note that we can obtain the optimal product function approximation m ∗ 1 , 1 × m ∗ 2 , 2 explicitly via Eqs. (17) and (18). Intuitively, we need to find an approximation of a (multi)set of codes with one factor fixed and other factors varying, and then we use the aggregation of all the approximation errors as a modularity metric.

The second obstacle - the minimization problem - still needs to be addressed. Since the code spaces Z 1 and Z 2 can be infinite sets, the minimization problem may not have a closed-form minimizer or even an exact solver. Even if an exact solver exists, the solution may not be differentiable with respect to the inputs. Let us examine some concrete examples of q product by choosing different aggregators ▽ in Eqs. (17) and (18). In the following three examples, we assume that the code spaces Z 1 and Z 2 are Euclidean spaces equipped with the usual Euclidean distances.

Example 2. If the aggregator ▽ is the supremum , the best approximation is the center of the smallest bounding sphere [Megiddo, 1983], and the approximation error is the radius .

This metric has the advantage of being definable even when the factor spaces Y 1 and Y 2 are infinite sets, and it can be computed using either randomized [Welzl, 1991] or exact [Fischer et al., 2003] algorithms. However, it is not easy to calculate its gradient. Thus, we cannot use it as a learning objective and directly optimize it using gradient-based optimization.

Example 3. If the aggregator ▽ is the mean , the best approximation is the (geometric) median [Weiszfeld, 1937], and the approximation error is the mean absolute deviation around the median .

It is known that there is no exact algorithm for obtaining the geometric median [Cockayne and Melzak, 1969], but it can be effectively approximated using convex optimization [Cohen et al., 2016]. The geometric median has found applications in robust estimation in the fields of statistics and machine learning [Meer et al., 1991, Minsker, 2015, Pillutla et al., 2022, Guerraoui et al., 2023].

Example 4. If the aggregator ▽ is the mean square , the best approximation is the mean , and the approximation error is the variance . In this case, q product ( m ) can be simplified to

The variance is easier to compute and differentiate than the radius of the smallest bounding sphere and the mean absolute deviation around the median, but it is also more susceptible to outliers and noise. Further work could explore the theoretical implications of these metrics, especially in cases where only partial combinations of factors or noisy annotations are available.

Then, let us revisit our motivating example in Example 1:

Example 5. Let us consider the function m : { 0 , 1 } 2 → R 2 in Eq. (6). Its best product function approximation is

because m ∗ 1 , 1 (0) = 2 is the center/median/mean of the set { m 1 (0 , 0) = 1 , m 1 (0 , 1) = 3 } , and so on. The modularity metric is a distance between m and m ∗ .

4.2 Modularity metrics via constancy

Upon analyzing the metrics above, it becomes evident that what we need is not the best approximation itself (e.g., the mean) but rather the approximation error (e.g., the variance) - a measure of the constancy of a set of codes. Following this insight, our next objective is to formulate a modularity metric that eliminates the need for an optimization step. Zhang and Sugiyama [2023] have proved that a function is a product function if and only if the curried functions 9 of its component functions are constant, as shown in the following example:

Example 6. Consider the functions m,m ′ : { 0 , 1 } 2 → R 2 in Eqs. (6) and (7) and the curried functions of their second component functions m 2 , m ′ 2 : { 0 , 1 } 2 → R :

The curried function ̂ m 2 : { 0 , 1 } → [ { 0 , 1 } , R ] is not constant, while ̂ m ′ 2 is constant with value { 0 ↦→ c, 1 ↦→ d } ∈ [ { 0 , 1 } , R ] (and so is ̂ m ′ 1 ), indicating that m ′ is a product function.

Based on this fact, we propose an alternative approach for measuring modularity:

Definition 10 (Constancy of curried function) . Let m 1 and m 2 be the component functions of a function m : Y → Z from a product Y := Y 1 × Y 2 of sets to another product Z := Z 1 × Z 2 of sets. The extent to which m resembles a product function can be measured by the constancy of the curried functions of m 1 and m 2 :

where q const is a quantity for constant functions.

To complete this construction, we adopt the following definition and metric of the constant function:

Definition 11 (Constant function) . A function f : A → B is constant if

which can be measured by

This constancy metric q const only needs to compute pairwise distances between the outputs, requiring | A | 2 times distance computation but no optimization. Incorporating q const into q const-curry , we can get the following metric:

Proposition 3. The quantity q const-curry ( m : Y → Z ) equals

Here are two examples of q const and q const-curry using different aggregators ▽ in Eqs. (25) and (26).

Example 7. If the aggregator ▽ is the maximum , q const is the diameter (the maximum pairwise distance) of the outputs. In this case, q const-curry ( m ) can be simplified to

Example 8. If the aggregator ▽ is the mean square , q const is the mean pairwise squared distance, which equals the variance . In this case, q const-curry ( m ) coincides with Eq. (19).

In summary, Eqs. (19) and (27) are easily computable and differentiable metrics, and their minimizers are precisely product functions. They do not contain any hyperparameters or stochastic components and thus can serve as both learning objectives and evaluation metrics.

4.3 Informativeness metrics

If an encoder f : X → Z is constant, mapping everything to the same value, according to Definition 3, it is perfectly modular. However, a constant encoder is also completely useless. In this subsection, we shift our focus to the property of informativeness -ameasurement of usefulness.

Informativeness is not a unique requirement for disentangled representations. Other representation learning paradigms, such as contrastive learning [Jaiswal et al., 2020, Wang and Isola, 2020] and metric learning [Musgrave et al., 2020], also emphasize the importance of mapping dissimilar data to far-apart locations in the representation space. While one could integrate this requirement into a single disentanglement score (e.g., [Higgins et al., 2017, Kim and Mnih, 2018]), we argue that it is better to evaluate the usefulness of representations separately for a more fine-grained assessment [Carbonneau et al., 2022].

One straightforward way to measure informativeness is to measure how much we can invert the encoding process:

Definition 12 (Retraction approximation) . Let m : Y → Z be a function. The extent to which m is retractable can be measured by a distance between the composition of m and its best retraction approximation and the identity function:

This metric q retractable is derived from Definition 2 following the conversion procedure in Table 1. This informativeness metric also involves an optimization step similar to the modularity metric q product , potentially introducing randomness or higher computation costs. Note that we may use a parameterized subset of the set [ Z, Y ] of all functions from codes Z to factors Y , such as the set of linear functions. Then, the problem becomes a regression/classification problem, and the metric is the performance of the predictor. A number of existing works adopted this approach and used the accuracy, the area under the ROC curve (AUC-ROC), or the mean squared error (MSE) to measure the informativeness [Ridgeway and Mozer, 2018, Eastwood and Williams, 2018, Eastwood et al., 2023]. However, such metrics necessitate additional hyperparameter tuning and are more likely to exhibit varying behavior across different implementations [Carbonneau et al., 2022].

It raises the question of whether we can measure the informativeness of an encoder without approximating its retraction. We propose to measure informativeness by directly measuring the injectivity of the encoding process:

Definition 13 (Contraction) . Let m : Y → Z be a function. The extent to which m is injective can be measured by how much m contracts pairs of inputs:

This metric q injective is derived from Definition 1 following the conversion procedure in Table 1. According to Theorem 1, we know that q retractable ( m ) = 0 if and only if m is retractable. However, q injective ( m ) = 0 implies the injectivity of m but not the other way around:

In other words, a minimizer of q injective is required to be non-contractive , which is a stronger condition than being injective. For example, let us consider the function m : [0 , 1] → R := y ↦→ 0 . 01 × y . Although it is injective, its outputs are less distinguishable from each other in terms of the Euclidean distance. Therefore, q injective still assign a non-zero value to this function.

Although not all injective functions necessarily minimize q injective , according to Theorem 1, we can guarantee that minimizing q injective will not lead to non-injective functions. Moreover, q injective does not require training regressors or classifiers to approximate the retraction. Consequently, it does not need any time-consuming hyperparameter tuning or cross-validation like existing informativeness metrics [Eastwood and Williams, 2018, Ridgeway and Mozer, 2018].

a [Higgins et al., 2017] b [Kim and Mnih, 2018] c [Chen et al., 2018] d [Eastwood and Williams, 2018]

5 Experiments

In this section, we empirically demonstrate the effectiveness of the proposed metrics. Following Carbonneau et al. [2022], we did not learn representations on datasets but directly defined functions m : Y → Z from factors to codes, which allows us to capture typical failure patterns.

We evaluated modularity metrics based on (i) the radius of the smallest bounding sphere, (ii) mean absolute deviation (MAD) around the median, (iii) variance, (iv) diameter, and (v) mean pairwise distance (MPD) introduced in Sections 4.1 and 4.2. We evaluated informativeness metrics based on retraction approximation using the maximum error (ME), mean absolute error (MAE), and mean squared error (MSE), and we calculated the contraction discussed in Section 4.3. To compare with existing metrics [Higgins et al., 2017, Kim and Mnih, 2018, Chen et al., 2018, Eastwood and Williams, 2018], we transformed the results isomorphically using e -x : [0 , ∞ ] → [0 , 1] , meaning that 1 is the perfect score. The results are shown in Table 2, and our observations are as follows.

If a representation is given a perfect score by a proposed metric, it must satisfy the property that the metric quantifies , which confirms our theoretical result. In Table 2, the light cells show that the proposed metrics can assign a perfect score when the function truly satisfies the properties, which is indicated by ✓ and ✗ . The dark cells are supposed to be perfect scores, but they fall short due to the limited expressiveness of the linear models used for the approximation. Meanwhile, some existing metrics that only provide a single score may entangle modularity and informativeness.

The metrics derived from equivalent definitions may differ in terms of computation cost and differentiability. Concretely, the radius, MAD, ME, MAE, and MSE are not differentiable due to the inner optimization problem, while the variance, diameter, MPD, and max/mean contraction are differentiable. In terms of computation, they are much faster than metrics requiring hyperparameter tuning, such as DCI [Eastwood and Williams, 2018]. Further comparisons are provided in Appendix E.

Different metrics may rank imperfect representations differently, even though they have exactly the same minimizers. This difference can lead to differences in risk preferences, sensitivity to outliers, and learning dynamics when these metrics are used as learning objectives. Illustrations and further discussion can be found in Appendix D.

Further, the proposed metrics can be used in weakly supervised or fine-grained evaluation. See Appendix E for detailed data configuration and further experimental results.

6 Conclusion

In this work, we developed a systematic and rigorous method for converting logical definitions of properties of representation learning models into quantitative metrics (Table 1). We applied this method to assess two important and distinct properties of disentangled representations: modularity and informativeness. We derived two families of metrics for each property based on their logically equivalent definitions. We theoretically analyzed the minimizers of these metrics (Theorem 1) and compared their differences in terms of computation cost and differentiability. Future research could compare metrics derived from different aggregators and design appropriate models to optimize these metrics with minimal supervision.

Acknowledgments

We thank Paolo Perrone for generously sharing insights on Markov categories enriched in divergence spaces. We thank Ken Sakayori for reviewing an earlier version of Appendices A and B and providing constructive suggestions. We thank Zhiyuan Zhan for insightful discussions on metrics, topology, measures, and optimization of function properties, and for reviewing and proofreading some parts of Appendix D. We thank Masahiro Negishi for valuable discussions on disentanglement metrics and weakly supervised disentanglement. We thank Jingwen Fu for checking the algebraic concepts in Section 3. We also thank Johannes Ackermann, Xin-Qiang Cai, and Tongtong Fang for their valuable feedback on the manuscript.

YZ was supported by JSPS KAKENHI Grant Number 22KJ0880. MS was supported by JST CREST Grant Number JPMJCR18A2 and a grant from Apple, Inc. Any views, opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and should not be interpreted as reflecting the views, policies or position, either expressed or implied, of Apple Inc.

References

Jiˇ rí Adámek, Horst Herrlich, and George Strecker. Abstract and Concrete Categories: The Joy of Cats . John Wiley and Sons, 1990. URL http://www.tac.mta.ca/tac/reprints/ar ticles/17/tr17abs.html . A.1

K Amer. Equationally complete classes of commutative monoids with monus. Algebra Universalis , 18:129-131, 1984. URL https://doi.org/10.1007/BF01182254 . 18

Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference on Machine Learning , 2017. URL http://proceedings.mlr.press/v70/amos17b. html . 1, D.1

Steve Awodey. Category Theory . Oxford University Press, 2010. URL https://doi.org/10 .1093/acprof:oso/9780198568612.001.0001 . A.1

Giorgio Bacci, Radu Mardare, Prakash Panangaden, and Gordon Plotkin. Propositional logics for the Lawvere quantale. Electronic Notes in Theoretical Informatics and Computer Science , 3, 2023. URL https://doi.org/10.46298/entics.12292 . https://arxiv.org/abs/ 2302.01224 . B.9, D.2

Giorgio Bacci, Radu Mardare, Prakash Panangaden, and Gordon Plotkin. Polynomial Lawvere logic. arXiv preprint , 2024. URL https://arxiv.org/abs/2402.03543 . D.2

Samy Badreddine, Artur d'Avila Garcez, Luciano Serafini, and Michael Spranger. Logic tensor networks. Artificial Intelligence , 303:103649, 2022. URL https://doi.org/10.1016/j. artint.2021.103649 . https://arxiv.org/abs/2012.13635 . D.2

Nikita Balabin, Daria Voronkova, Ilya Trofimov, Evgeny Burnaev, and Serguei Barannikov. Disentanglement learning via topology. In International Conference on Machine Learning , 2024. D.1

Han Bao and Masashi Sugiyama. Calibrated surrogate maximization of linear-fractional utility in binary classification. In International Conference on Artificial Intelligence and Statistics , pages 2337-2347, 2020. URL http://proceedings.mlr.press/v108/bao20a.html . D.2

Han Bao, Clay Scott, and Masashi Sugiyama. Calibrated surrogate losses for adversarially robust classification. In Conference on Learning Theory , pages 408-451, 2020. URL http: //proceedings.mlr.press/v125/bao20a.html . D.2

Michael Barr and Charles Wells. Category Theory for Computing Science , volume 1. Prentice Hall New York, 1990. URL https://www.math.mcgill.ca/triples/Barr-Wells-ctc s.pdf . A.1

Sugato Basu, Arindam Banerjee, and Raymond J Mooney. Semi-supervised clustering by seeding. In International Conference on Machine Learning , 2002. URL https://dl.acm.org/doi/1 0.5555/645531.656012 . E.2

Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference on Machine Learning , 2019. URL https://proceedings.mlr.press/v97/behrmann19a.html . D.1

Itaï Ben Yaacov. On the expressive power of quantifiers in continuous logic. arXiv preprint , 2022. URL https://arxiv.org/abs/2207.01863 . D.2

Itaï Ben Yaacov and Alexander Usvyatsov. Continuous first order logic and local stability. Transactions of the American Mathematical Society , 362(10):5213-5259, 2010. URL https: //doi.org/10.1090/S0002-9947-10-04837-3 . https://arxiv.org/abs/08 01.4303 . D.2

Itaï Ben Yaacov, Alexander Berenstein, C. Ward Henson, and Alexander Usvyatsov. Model Theory for Metric Structures , page 315-427. London Mathematical Society Lecture Note Series. Cambridge University Press, 2008. URL https://doi.org/10.1017/CBO9780511735219.011 . D.2

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(8): 1798-1828, 2013. URL https://doi.org/10.1109/TPAMI.2013.50 . https: //arxiv.org/abs/1206.5538 . 1, 1, 2.1, 2.2, D.1

Merrie Bergmann. An Introduction to Many-Valued and Fuzzy Logic: Semantics, Algebras, and Derivation Systems . Cambridge University Press, 2008. URL https://doi.org/10.1017/ CBO9780511801129 . B.1, D.2

Mikhail Bilenko, Sugato Basu, and Raymond J Mooney. Integrating constraints and metric learning in semi-supervised clustering. In International Conference on Machine Learning , 2004. URL https://dl.acm.org/doi/10.1145/1015330.1015360 . E.2

George Boole. An Investigation of the Laws of Thought: On Which Are Founded the Mathematical Theories of Logic and Probabilities . Cambridge University Press, 1854. URL https: //doi.org/10.1017/CBO9780511693090 . 2.1, D.2

Tai-Danae Bradley, John Terilla, and Yiannis Vlassopoulos. An enriched category theory of language: From syntax to semantics. La Matematica , 1(2):551-580, 2022. URL https://doi.org/10 .1007/s44007-022-00021-2 . https://arxiv.org/abs/2106.07890 . A.3

Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal representation learning. In Neural Information Processing Systems , 2022. URL https: //openreview.net/forum?id=dz79MhQXWvg . D.2

Johann Brehmer, Pim De Haan, Sönke Behrends, and Taco Cohen. Geometric algebra transformer. In Neural Information Processing Systems , 2023. URL https://openreview.net/forum ?id=M7r2CO4tJC . 1, D.1

Leo Breiman, Jerome Friedman, Richard A. Olshen, and Charles J. Stone. Classification and Regression Trees . Routledge, 1984. URL https://doi.org/10.1201/9781315139470 . D.1

Chris Burgess and Hyunjik Kim. 3D shapes dataset, 2018. https://github.com/deepmin d/3d-shapes (Apache License 2.0). 6, 5, E.3, 7, 10

Matteo Capucci. On quantifiers for quantitative reasoning. arXiv preprint , 2024. URL https: //arxiv.org/abs/2406.04936 . D.2

Marc-André Carbonneau, Julian Zaidi, Jonathan Boilard, and Ghyslain Gagnon. Measuring disentanglement: A review of metrics. IEEE Transactions on Neural Networks and Learning Systems , 2022. URL https://doi.org/10.1109/TNNLS.2022.3218982 . https://arxiv.org/abs/2012.09276 . 1, 2, 4.3, 4.3, 5, D.1

Hugo Caselles-Dupré, Michael Garcia Ortiz, and David Filliat. Symmetry-based disentangled representation learning requires interaction with environments. In Neural Information Processing Systems , 2019. URL https://proceedings.neurips.cc/paper/2019/hash/36e 729ec173b94133d8fa552e4029f8b-Abstract.html . D.1

Chen Chung Chang. Algebraic analysis of many valued logics. Transactions of the American Mathematical society , 88(2):467-490, 1958. URL https://doi.org/10.2307/1993227 . D.2

Chen Chung Chang and H Jerome Keisler. Continuous Model Theory , volume 58 of Annals of Mathematics Studies . Princeton University Press, 1966. URL https://doi.org/10.1515/ 9781400882052 . D.2

Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Neural Information Processing Systems , 2018. URL https://proceedings.neurips.cc/paper/2018/hash/1ee3dfcd8a064 5a25a35977997223d22-Abstract.html . 1, 2, 5, D.1, 5, E.3, 6, 7

Shuxiao Chen, Edgar Dobriban, and Jane H Lee. A group-theoretic framework for data augmentation. The Journal of Machine Learning Research , 21(245):1-71, 2020. URL http://jmlr.org/p apers/v21/20-163.html . D.1

Yiting Chen, Zhanpeng Zhou, and Junchi Yan. Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. In International Conference on Learning Representations , 2024. URL https://openreview.net/forum ?id=4bSQ3lsfEV . A.1

Kenta Cho and Bart Jacobs. Disintegration and Bayesian inversion via string diagrams. Mathematical Structures in Computer Science , 29(7):938-971, 2019. URL https://doi.org/10.1017/ S0960129518000488 . https://arxiv.org/abs/1709.00322 . A.1

Simon Cho. Categorical semantics of metric spaces and continuous logic. The Journal of Symbolic Logic , 85(3):1044-1078, 2020. URL https://doi.org/10.1017/jsl.2020.44 . D.2

Ernest J Cockayne and Zdzislaw A Melzak. Euclidean constructibility in graph-minimization problems. Mathematics Magazine , 42(4):206-208, 1969. URL https://doi.org/10.108 0/0025570X.1969.11975961 . 4.1

Michael B Cohen, Yin Tat Lee, Gary Miller, Jakub Pachocki, and Aaron Sidford. Geometric median in nearly linear time. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 9-21, 2016. URL https://doi.org/10.1145/2897518.2897647 . 4.1

Taco Cohen. Equivariant convolutional networks . PhD thesis, University of Amsterdam, 2021. URL https://hdl.handle.net/11245.1/0f7014ae-ee94-430e-a5d8-37d03d8d1 0e6 . D.1

Taco Cohen and Max Welling. Learning the irreducible representations of commutative Lie groups. In International Conference on Machine Learning , 2014. URL https://proceedings.ml r.press/v32/cohen14.html . D.1

Taco Cohen and Max Welling. Transformation properties of learned visual representations. In International Conference on Learning Representations , 2015. URL http://arxiv.org/ab s/1412.7659 . D.1

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International Conference on Machine Learning , 2016. URL http://proceedings.mlr.press/v48/cohenc16 .html . D.1

Geoffrey SH Cruttwell, Bruno Gavranovi´ c, Neil Ghani, Paul Wilson, and Fabio Zanasi. Categorical foundations of gradient-based learning. In Programming Languages and Systems , pages 1-28. Springer International Publishing, 2022. URL https://doi.org/10.1007/978-3-030 -99336-8_1 . https://arxiv.org/abs/2103.01931 . A.1

Haskell B. Curry. Some philosophical aspects of combinatory logic. In Studies in Logic and the Foundations of Mathematics , volume 101, pages 85-101. Elsevier, 1980. URL https: //doi.org/10.1016/S0049-237X(08)71254-0 . 9

Francesco Dagnino and Fabio Pasquali. Logical foundations of quantitative equality. In Logic in Computer Science , 2022. URL https://doi.org/10.1145/3531130.3533337 . D.2

Hennie Daniels and Marina Velikova. Monotone and partially monotone neural networks. IEEE Transactions on Neural Networks , 21(6):906-917, 2010. URL https://doi.org/10.110 9/TNN.2010.2044803 . D.1

Artur S. d'Avila Garcez, Krysia B. Broda, and Dov M. Gabbay. Neural-Symbolic Learning Systems . Springer, 2002. URL https://doi.org/10.1007/978-1-4471-0211-3 . D.2

Pim de Haan, Taco Cohen, and Max Welling. Natural graph networks. In Neural Information Processing Systems , 2020. URL https://proceedings.neurips.cc/paper/2020/ hash/2517756c5a9be6ac007fe9bb7fb92611-Abstract.html . A.1, D.1

Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wuthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=8VXvj1QNRl1 . D.1

Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representations. In International Conference on Learning Representations , 2020. URL https://openreview .net/forum?id=HJgK0h4Ywr . 1, D.1

Andrew Dudzik. Quantales and Hyperstructures: Monads, Mo'Problems . PhD thesis, University of California, Berkeley, 2017. URL https://arxiv.org/abs/1707.09227 . B.9

Andrew Joseph Dudzik and Petar Veliˇ ckovi´ c. Graph neural networks are dynamic programmers. In Neural Information Processing Systems , 2022. URL https://openreview.net/forum?i d=wu1Za9dY1GY . A.1

Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=By-7dz-AZ . 1, 1, 2, 4.3, 4.3, 2, 5, D.1, 5, 6, 7, E.4, E.5

Cian Eastwood, Andrei Liviu Nicolicioiu, Julius Von Kügelgen, Armin Keki´ c, Frederik Träuble, Andrea Dittadi, and Bernhard Schölkopf. DCI-ES: An extended disentanglement framework with connections to identifiability. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=462z-gLgSht . 4.3, D.1

Ronald Fagin, Ryan Riegel, and Alexander Gray. Foundations of reasoning with uncertainty via real-valued logics. Proceedings of the National Academy of Sciences , 121(21), 2024. URL https://doi.org/10.1073/pnas.2309905121 . D.2

Daniel Figueroa and Benno van den Berg. A topos for continuous logic. Theory and Applications of Categories , 38(28), 2022. URL http://www.tac.mta.ca/tac/volumes/38/28/3 8-28abs.html . D.2

Kaspar Fischer, Bernd Gärtner, and Martin Kutz. Fast smallest-enclosing-ball computation in high dimensions. In European Symposium on Algorithms , pages 630-641. Springer, 2003. URL https://doi.org/10.1007/978-3-540-39658-1_57 . 4.1

Brendan Fong and David I Spivak. An Invitation to Applied Category Theory: Seven Sketches in Compositionality . Cambridge University Press, 2019. URL https://doi.org/10.1017/ 9781108668804 . https://arxiv.org/abs/1803.05316 . A.1, B.1, B.6, B.8

Tobias Fritz. A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statistics. Advances in Mathematics , 370:107239, 2020. URL https://doi.org/ 10.1016/j.aim.2020.107239 . https://arxiv.org/abs/1908.07021 . A.1

Soichiro Fujii. Ordered semirings and subadditive morphisms. arXiv preprint , 2023. URL https://arxiv.org/abs/2311.03862 . B.9

Marco Fumero, Luca Cosmo, Simone Melzi, and Emanuele Rodolà. Learning disentangled representations via product manifold projection. In International Conference on Machine Learning , 2021. URL http://proceedings.mlr.press/v139/fumero21a.html . 1, D.1

Bruno Gavranovi´ c, Paul Lessard, Andrew Joseph Dudzik, Tamara von Glehn, João Guilherme Madeira Araújo, and Petar Veliˇ ckovi´ c. Position: Categorical deep learning is an algebraic theory of all architectures. In International Conference on Machine Learning , 2024. URL https://openreview.net/forum?id=EIcxV7T0Sy . A.1

Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. In Neural Information Processing Systems , 2019. URL https://proceedings.neurips.cc /paper/2019/hash/d97d404b6119214e4a7018391195240a-Abstract.html . https://github.com/rr-learning/disentanglement_dataset (Creative Commons Attribution 4.0 International License). 6, 5, E.3, 7, E.6, 11

Ian Goodfellow, Honglak Lee, Quoc Le, Andrew Saxe, and Andrew Ng. Measuring invariances in deep networks. In Neural Information Processing Systems , 2009. URL https://procee dings.neurips.cc/paper/2009/hash/428fca9bc1921c25c5121f9da7815cd e-Abstract.html . D.1

Rachid Guerraoui, Nirupam Gupta, and Rafael Pinot. Byzantine machine learning: A primer. ACM Computing Surveys , 2023. URL https://doi.org/10.1145/3616537 . 4.1

Petr Hájek. Metamathematics of Fuzzy Logic , volume 4 of Trends in Logic . Springer, 1998. URL https://doi.org/10.1007/978-94-011-5300-3 . D.2

Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature , 585(7825):357-362, 2020. URL https://doi.org/10.1038/s415 86-020-2649-2 . https://numpy.org . D.6

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Sy2fzU9gl . 1, 4.3, 2, 5, D.1, D.2, 5, E.3, E.4, 6, 7, E.6

Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint , 2018. URL https://arxiv.org/abs/1812.02230 . 1, D.1

Aapo Hyvärinen and Erkki Oja. Independent component analysis: Algorithms and applications. Neural networks , 13(4):411-430, 2000. URL https://doi.org/10.1016/S0893-608 0(00)00026-5 . D.1

Isao Ishikawa, Takeshi Teshima, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Universal approximation property of invertible neural networks. Journal of Machine Learning Research , 24(287):1-68, 2023. URL https://www.jmlr.org/papers/v24/22-0384. html . D.1

Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies , 9(1):2, 2020. URL https://doi.org/10.3390/technologies9010002 . https://arxiv.org/ab s/2011.00362 . 4.3

Peter Johnstone. Sketches of an Elephant: A Topos Theory Compendium . Oxford University Press, 2002. A.2

Max Kelly. Basic concepts of enriched category theory. In London Mathematical Society Lecture Note Series , volume 64. Cambridge University Press, 1982. URL http://www.tac.mta.ca /tac/reprints/articles/10/tr10.html . A.3

Maurice G. Kendall. A new measure of rank correlation. Biometrika , 30(1/2):81-93, 1938. URL https://doi.org/10.2307/2332226 . E.4

Hamza Keurti, Hsiao-Ru Pan, Michel Besserve, Benjamin F Grewe, and Bernhard Schölkopf. Homomorphism AutoEncoder -learning group structured representations from observed transitions. In International Conference on Machine Learning , 2023. URL https://pr oceedings.mlr.press/v202/keurti23a.html . D.1

Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning , 2018. URL http://proceedings.mlr.press/v80/kim18b.html . 1, 4.3, 2, 5, D.1, D.2, 5, E.3, E.4, 6, 7

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations , 2014. URL https://openreview.net/forum?id=33X9 fd2-9FyZd . http://arxiv.org/abs/1312.6114 . E.3

Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In International Conference on Machine Learning , 2020. URL https://proceedings.mlr.press/v119/kohler20a.html . D.1

Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques . MIT press, 2009. URL https://mitpress.mit.edu/9780262013192/ . D.1

Solomon Kullback and Richard A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics , 22(1):79-86, 1951. URL https://doi.org/10.1214/aoms /1177729694 . https://www.jstor.org/stable/2236703 . D.2

Henry Kvinge, Tegan Emerson, Grayson Jorgenson, Scott Vasquez, Timothy Doster, and Jesse Lew. In what ways are deep neural networks invariant and how should we measure this? In Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=SC D0hn3kMHw . 1, D.1, D.2

William Lawvere. Adjointness in foundations. Dialectica , pages 281-296, 1969. URL https://doi.org/10.1111/j.1746-8361.1969.tb01194.x . http: //www.tac.mta.ca/tac/reprints/articles/16/tr16abs.html . B.10

William Lawvere. Metric spaces, generalized logic, and closed categories. Rendiconti del seminario matématico e fisico di Milano , 43:135-166, 1973. URL https://doi.org/10.1007/BF02 924844 . http://www.tac.mta.ca/tac/reprints/articles/1/tr1abs.html . A.3, A.3, B.1, B.9, D.2

William Lawvere. Taking categories seriously. Revista colombiana de matematicas , 20(3-4):147178, 1986. http://www.tac.mta.ca/tac/reprints/articles/8/tr8abs.html . 4.1

F William Lawvere and Robert Rosebrugh. Sets for Mathematics . Cambridge University Press, 2003. URL https://doi.org/10.1017/CBO9780511755460 . 2.1, A.2

Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In International Conference on Machine Learning , 2019. URL http://proceedings.mlr.pr ess/v97/lee19d . 1, D.1

Tom Leinster. An informal introduction to topos theory. arXiv preprint , 2010. URL https: //arxiv.org/abs/1012.5647 . A.2

Tom Leinster. Basic Category Theory . Cambridge University Press, 2014. URL https://doi. org/10.1017/CBO9781107360068 . https://arxiv.org/abs/1612.09375 . A.1

Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progressive learning and disentanglement of hierarchical representations. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=SJxpsx rYPS . 1, D.1

Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. In Neural Information Processing Systems , 2019a. URL https://proceedings.neurips.cc/paper/2019/hash/1b4 86d7a5189ebe8d8c46afc64b0d1b4-Abstract.html . D.1

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning , 2019b. URL https://proceedings.mlr.press/v97/locatello19a.html . D.1, 14, E.3

Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning , 2020. URL http://proceedings.mlr.press/v119/locatel lo20a.html . D.2

Jan Łukasiewicz. O logice trójwarto´ sciowej (On three-valued logic). In Selected Works , volume 11 of Studies in Logic , pages 87-88. North-Holland Publishing Company, 1920. D.2

Jan Łukasiewicz and Alfred Tarski. Untersuchungen über den Aussagenkalkül (Investigations on the propositional calculus). Comptes Rendus des Séances de la Société des Sciences et des Lettres de Varsovie, Class III , 23, 1930. D.2

Saunders Mac Lane. Categories for the Working Mathematician . Springer, 1978. URL https: //doi.org/10.1007/978-1-4757-4721-8 . 1, A.1, A.1

Saunders Mac Lane and Ieke Moerdijk. Sheaves in Geometry and Logic: A First Introduction to Topos Theory . Springer, 1994. URL https://doi.org/10.1007/978-1-4612-0927-0 . A.2

Louis Mahon, Lei Shah, and Thomas Lukasiewicz. Correcting flaws in common disentanglement metrics. arXiv preprint , 2023. URL https://arxiv.org/abs/2304.02335 . D.7

Gregorz Malinowski. Many-valued logic and its philosophy. In Handbook of the History of Logic , volume 8, pages 13-94. Elsevier, 2007. URL https://doi.org/10.1016/S1874-585 7(07)80004-5 . D.2

Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. DeepProbLog: Neural probabilistic logic programming. In Neural Information Processing Systems , 2018. URL https://proceedings.neurips.cc/paper/2018/hash/dc5d637ed 5e62c36ecb73b654b05ba2a-Abstract.html . D.2

Radu Mardare, Prakash Panangaden, and Gordon Plotkin. Quantitative algebraic reasoning. In Logic in Computer Science , 2016. URL https://doi.org/10.1145/2933575.2934518 . D.2

Radu Mardare, Prakash Panangaden, and Gordon Plotkin. Fixed-points for quantitative equational logics. In Logic in Computer Science , 2021. URL https://doi.org/10.1109/LICS52 264.2021.9470662 . D.2

Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=Syx72jC9tm . D.1

Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dSprites: Disentanglement testing sprites dataset, 2017. https://github.com/deepmind/dsprites-dataset (Apache License 2.0). 6, 5, E.3, 7, 9

Barry Mazur. When is one thing equal to some other thing? In Proof and Other Dilemmas: Mathematics and Philosophy , page 221-242. Mathematical Association of America, 2008. URL https://doi.org/10.5948/UPO9781614445050.015 . https://people.math. harvard.edu/~mazur/preprints/when_is_one.pdf . 3.1

Peter Meer, Doron Mintz, Azriel Rosenfeld, and Dong Yoon Kim. Robust regression methods for computer vision: A review. International journal of computer vision , 6:59-70, 1991. URL https://doi.org/10.1007/BF00127126 . 4.1

Nimrod Megiddo. The weighted Euclidean 1-center problem. Mathematics of Operations Research , 8(4):498-504, 1983. URL https://doi.org/10.1287/moor.8.4.498 . 2

Stanislav Minsker. Geometric median and robust estimation in Banach spaces. Bernoulli , 21(4): 2308-2335, 2015. URL https://doi.org/10.3150/14-BEJ645 . 4.1

Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. In Neural Information Processing Systems , 2022. URL https: //openreview.net/forum?id=7b7iGkuVqlZ . D.1

Milton L. Montero, Jeffrey Bowers, Rui Ponte Costa, Casimir JH Ludwig, and Gaurav Malhotra. Lost in latent space: Examining failures of disentangled models at combinatorial generalisation. In Neural Information Processing Systems , 2022. URL https://openreview.net/forum?i d=7yUxTNWyQGf . D.7

Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=qbH974jKUVy . D.1, D.7

Christopher J. Mulvey. &. Supplemento ai Rendiconti del Circolo Matemàtico di Palermo. Serie II , 12:99-104, 1986. URL https://zbmath.org/0633.46065 . B.9

Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In European Conference on Computer Vision , pages 681-699, 2020. URL https://doi.org/10.1007/ 978-3-030-58595-2_41 . 4.3

Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning , 2023. URL https://proceedings.mlr.press/v202/navon23a. html . D.1

Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On the calibration of multiclass classification with rejection. In Neural Information Processing Systems , 2019. URL https://proceedings.neurips.cc/paper/2019/hash/571d3a9420bfd9219 f65b643d0003bf4-Abstract.html . D.2

Matthew Painter, Adam Prugel-Bennett, and Jonathon Hare. Linear disentangled representations and unsupervised action estimation. In Neural Information Processing Systems , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/9a02387b02ce7de2d ac4b925892f68fb-Abstract.html . D.1

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, highperformance deep learning library. In Neural Information Processing Systems , 2019. URL https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2 bfa9f7012727740-Abstract.html . https://pytorch.org . D.6, E.3

Edward Pearce-Crump. Categorification of group equivariant neural networks. arXiv preprint , 2023. URL https://arxiv.org/abs/2304.14144 . A.1

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011. URL https://jmlr.org/papers/v12/pedr egosa11a.html . https://scikit-learn.org . E.4, E.5

Paolo Perrone. Markov categories and entropy. IEEE Transactions on Information Theory , 2023. URL https://doi.org/10.1109/TIT.2023.3328825 . https://arxiv.org/ abs/2212.11719 . A.1, D.2

David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière. Disentangling by subspace diffusion. In Neural Information Processing Systems , 2020. URL https://proceedings.neurips. cc/paper/2020/hash/c9f029a6a1b20a8408f372351b321dd8-Abstract.ht ml . D.1

Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing , 70:1142-1154, 2022. URL https: //doi.org/10.1109/TSP.2022.3153135 . 4.1, 13

Jean-Eric Pin. Tropical semirings. In Idempotency , pages 50-69. Cambridge University Press, 1998. URL https://doi.org/10.1017/CBO9780511662508.004 . B.7

Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. In Neural Information Processing Systems , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/e449b9317dad9 20c0dd5ad0a2a2d5e49-Abstract.html . D.1

Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. In Neural Information Processing Systems , 2015. URL https://proceedings.neurips.cc/pap er/2015/hash/e07413354875be01a996dc560274708e-Abstract.html . 6, 5, E.3, 7, 8

Mark D. Reid and Robert C. Williamson. Composite binary losses. Journal of Machine Learning Research , 11(83):2387-2422, 2010. URL http://jmlr.org/papers/v11/reid10a.ht ml . D.2

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning , 2015. URL https://proceedings.mlr.press/v37/ rezende15.html . D.1

Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. In Neural Information Processing Systems , 2018. URL https://proceedings.neur ips.cc/paper/2018/hash/2b24d495052a8ce66358eb576b8912c8-Abstract. html . 1, 1, 2, 2.2, 4.3, 4.3, D.1, D.2, E.2

Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=OKcJ hpQiGiX . D.7

Bernhard Schölkopf and Julius von Kügelgen. From statistical to causal learning. In International Congress of Mathematicians . EMS Press, 2022. URL https://doi.org/10.4171/icm2 022/173 . https://arxiv.org/abs/2204.00607 . D.1

Prithviraj Sen, Breno WSR de Carvalho, Ryan Riegel, and Alexander Gray. Neuro-symbolic inductive logic programming with logical neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , 2022. URL https://doi.org/10.1609/aaai.v36i8.20795 . D.2

Dan Shiebler, Bruno Gavranovi´ c, and Paul Wilson. Category theory in machine learning. arXiv preprint , 2021. URL https://arxiv.org/abs/2106.07032 . A.1

Daniel Shiebler. Compositionality and Functorial Invariants in Machine Learning . PhD thesis, University of Oxford, 2023. URL http://doi.org/10.5287/bodleian:DE1aDx4Zw . A.1

Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=HJgSwyBKvr . 1, D.2, E.2

Joseph Sill. Monotonic networks. In Neural Information Processing Systems , 1997. URL https://proceedings.neurips.cc/paper/1997/hash/83adc9225e4de b67d7ce42d58fe5157c-Abstract.html . D.1

Ingo Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26:225-287, 2007. URL https://doi.org/10.1007/s00365-006-0662-3 . D.2

Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning , 2019. URL http://proceedings.mlr.press/v97/ suter19a.html . 1

Seiya Tokui and Issei Sato. Disentanglement analysis with partial information decomposition. In International Conference on Learning Representations , 2022. URL https://openreview.n et/forum?id=pETy-HVvGtt . 1, D.1

Loek Tonnaer, Luis A Pérez Rey, Vlado Menkovski, Mike Holenderski, and Jacobus W Portegies. Quantifying and learning linear symmetry-based disentanglement. In International Conference on Machine Learning , 2022. URL https://proceedings.mlr.press/v162/tonna er22a.html . D.1

Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International Conference on Machine Learning , 2021. URL https: //proceedings.mlr.press/v139/trauble21a.html . D.7, D.7

Todd Trimble. An elementary approach to elementary topos theory, 2019. URL https: //ncatlab.org/toddtrimble/published/An+elementary+approach+to+ elementary+topos+theory . A.2

Alexey A. Tuzhilin. Who invented the Gromov-Hausdorff distance? arXiv preprint , 2016. URL https://arxiv.org/abs/1612.00728 . 4.1

Elise van der Pol, Herke van Hoof, Frans A Oliehoek, and Max Welling. Multi-agent MDP homomorphic networks. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=H7HDG--DJF0 . D.1

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature methods , 17(3):261-272, 2020. URL https://doi.org/10.1038/s41592-019-068 6-2 . https://scipy.org . E.4

Kiri Wagstaff, Claire Cardie, Seth Rogers, and Stefan Schrödl. Constrained k-means clustering with background knowledge. In International Conference on Machine Learning , 2001. URL https://dl.acm.org/doi/10.5555/645530.655669 . E.2

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning , 2020. URL https://proceedings.mlr.press/v119/wang20k.html . 1, 4.3

Endre Weiszfeld. Sur le point pour lequel la somme des distances de n points donnés est minimum (on the point for which the sum of the distances to n given points is minimum). Tohoku Mathematical Journal, First Series , 43:355-386, 1937. URL https://doi.org/10.1007/s10479-0 08-0352-z . 3

Emo Welzl. Smallest enclosing disks (balls and ellipsoids). In New Results and New Trends in Computer Science , pages 359-370. Springer, 1991. URL https://doi.org/10.1007/BF b0038202 . 4.1, 12

Zhenlin Xu, Marc Niethammer, and Colin A Raffel. Compositional generalization in unsupervised compositional representation learning: A study on disentanglement and emergent language. In Neural Information Processing Systems , 2022. URL https://openreview.net/forum?i d=ZEQ5Gf8DiD . D.1

Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Nanning Zheng. Towards building a group-based unsupervised representation disentanglement framework. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=YgPq Nctmyd . D.1

Yang Yuan. On the power of foundation models. In International Conference on Machine Learning , 2023. URL https://proceedings.mlr.press/v202/yuan23b.html . A.1

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Neural Information Processing Systems , 2017. URL https://proceedings.neurips.cc/paper/2017/hash/f22e4747da1aa27e3 63d86d40ff442fe-Abstract.html . D.1

Sharon Zhang, Amit Moscovich, and Amit Singer. Product manifold learning. In International Conference on Artificial Intelligence and Statistics , 2021. URL http://proceedings.mlr. press/v130/zhang21j.html . D.1

Yivan Zhang and Masashi Sugiyama. A category-theoretical meta-analysis of definitions of disentanglement. In International Conference on Machine Learning , 2023. URL https: //proceedings.mlr.press/v202/zhang23ak.html . 1, 4.2, A.1, D.1

Sharon Zhou, Eric Zelikman, Fred Lu, Andrew Y Ng, Gunnar Carlsson, and Stefano Ermon. Evaluating the disentanglement of deep generative models through manifold topology. In International Conference on Learning Representations , 2020. URL https://openrevi ew.net/forum?id=djwS0m4Ft_A . D.1

Contents

List of Figures

A Preliminaries

In this paper, we used abstract mathematical tools such as category theory and topos theory to develop a theory of the relationship between logical definitions and quantitative metrics. However, this level of abstraction may be unfamiliar or even intimidating to some readers, and sometimes unnecessary for machine learning practitioners. Therefore, we have used only the most basic algebraic concepts, such as homomorphism, in the main text. For readers interested in the mathematical background, we provide a brief introduction to the basic categorical concepts in this section.

A.1 Basic category theory

Category theory is a branch of mathematics that studies mathematical structures in an abstract way, which is suitable for identifying and organizing common patterns across various fields of mathematics [Mac Lane, 1978, Adámek et al., 1990, Awodey, 2010]. It has found applications in many fields, including computer science [Barr and Wells, 1990], probability theory [Cho and Jacobs, 2019, Fritz, 2020, Perrone, 2023], and machine learning [de Haan et al., 2020, Shiebler et al., 2021, Cruttwell et al., 2022, Dudzik and Veliˇ ckovi´ c, 2022, Shiebler, 2023, Yuan, 2023, Pearce-Crump, 2023, Chen et al., 2024, Gavranovi´ c et al., 2024].

The most fundamental concept is that of a category :

Definition 14. A category C = (Obj , Hom , ◦ , id) consists of

a collection Obj of objects,

a set Hom( A,B ) of morphisms between objects, 10

a composition function ◦ : Hom( B,C ) × Hom( A,B ) → Hom( A,C ) for each triple of objects, and

an identity morphism id A ∈ Hom( A,A ) for each object,

subject to

associativity: ( h ◦ g ) ◦ f = h ◦ ( g ◦ f ) and

identity: id B ◦ f = f = f ◦ id A .

A crucial example is the category Set of sets and functions. However, what is of particular interest is not the category itself but its relationships with other categories. Building on the concepts of the functor and natural transformation , whose definitions are omitted here, we can develop tools to better understand the properties of a category.

Moreover, we can define objects in terms of their relations with other objects, employing what is known as universal construction . For example, a terminal object 1 in a category is an object such that for any object A , there exists a unique morphism e A : A → 1 to it, which we call a terminal morphism . In Set , any set {∗} with only one element is a terminal object. Based on the concept of the terminal object, a global element of an object B is defined to be a morphism b : 1 → B from a terminal object.

We write b A : A → B as an abbreviation for the constant morphism b ◦ e A : A e A - - → 1 b - → B with value b : 1 → B . These concepts will be used to develop our theory in Appendix B. Other important universal constructions include the product , pullback , and exponential . The concept of the product is of great importance in disentangled representation learning [Zhang and Sugiyama, 2023].

Regarding the pullback, we need to mention the following useful lemma:

Lemma 4 (Pullback lemma) . Suppose that in the following commutative diagram, the right square is a pullback.

Then, the left square is a pullback if and only if the outer rectangle is a pullback.

This lemma is usually left as an exercise in textbooks [e.g., Mac Lane, 1978, p. 72, Exercise 8, Leinster, 2014, Exercise 5.1.35]. A proof can be found in Fong and Spivak [2019, Proposition 7.3]. We need to use this lemma to (de)compose pullbacks. As a side note, we use the asterisk f ∗ g to denote the pullback of g along f .

A.2 Elementary topos theory

Topos theory studies categories that, in some sense, exhibit behavior akin to the category of sets and functions [Lawvere and Rosebrugh, 2003]. Topos theory has found applications in geometry, topology, and logic [Mac Lane and Moerdijk, 1994, Johnstone, 2002, Leinster, 2010, Trimble, 2019]. In this work, we only explore its relation to logic.

To formally define a topos , two essential concepts are those of the subobject and subobject classifier. A subobject of an object C is simply a monomorphism b : B ↣ C to the object C . The subobject classifier is defined as follows:

Definition 15. In a finitely complete category, a subobject classifier is a universal subobject ⊤ : 1 ↣ Ω such that for every subobject b : B ↣ C , there exists a unique morphism χ b : C → Ω such that b is a pullback of ⊤ along χ b . The morphism χ b is called the classifying morphism of b .

Alternatively, we can state that

Proposition 5. A subobject classifier is precisely a terminal object in the category of monomorphisms and pullbacks.

Then, we can study the morphisms to the object Ω :

Definition 16. In a category with a subobject classifier ⊤ : 1 ↣ Ω , a predicate on an object C is a morphism p : C → Ω .

Based on Definition 15, we can state that subobjects of an object C are classified by predicates on C .

For example, in Set , subobjects are subsets, a function from a singleton {∗} to a two-element set is a subobject classifier, which is usually denoted by ⊤ : {∗} → {⊤ , ⊥} , a predicate on a set C is a function p : C →{⊤ , ⊥} , and a subset precisely corresponds to its indicator function.

Among various equivalent definitions of a topos, a concise one is as follows:

Definition 17. An elementary topos is a finitely complete and cartesian closed category with a subobject classifier.

Despite its concise definition, a great number of logical structures can be derived from it, which will be explored in Appendix B.

A.3 Enriched category theory

Enriched category theory generalizes the concept of the category by replacing the sets of morphisms with objects in a suitable category [Kelly, 1982]. It has been used to better understand a wide range of domains, from metric spaces [Lawvere, 1973] to language [Bradley et al., 2022].

Let us dive into the definition of an enriched category:

Definition 18. Acategory C = (Obj , Hom , ◦ , id) enriched in a monoidal category ( V , ⊗ , I ) consists of

a collection Obj of objects,

a hom-object Hom( A,B ) ∈ Obj V between objects,

a composition morphism ◦ : Hom( B,C ) ⊗ Hom( A,B ) → Hom( A,C ) for each triple of objects, and

an identity element id A : I → Hom( A,A ) for each object,

subject to associativity and identity.

Comparing Definitions 14 and 18, we can say that a (locally small) category is a category enriched in the category Set of sets and functions. Enrichment is a way to describe the additional structures of morphisms and the properties that need to be respected by composition.

An example is a preorder, which can be seen as a category enriched in the category of boolean values. Another example is a Lawvere metric space [Lawvere, 1973], which is a set with a premetric that satisfies the triangle inequality. Note that the transitivity of a preorder and the triangle inequality of a Lawvere metric are described by their composition morphisms, respectively.

In this work, we use enrichment to describe the association of a set of morphisms with additional operations, such as a strict premetric and aggregators.

B Theory

In this section, we detail the theory of converting logical definitions into their corresponding quantitative metrics based on elementary topos theory and enriched category theory.

B.1 Subobject quantifier and quantizer

Since our main goal is to develop a multi-valued (possibly continuous and differentiable) quantification of properties defined by a certain type of logic, we begin with a category E that has sufficient structures to allow the desired logical operations and build the quantification upon these structures.

Firstly, recall that a subobject classifier Ω , if exists, is the representing object of the subobject functor Sub E , such that a subobject b : B ↣ C corresponds to a unique classifying morphism χ b : C → Ω , and an external operation on the set Sub E ( C ) of subobjects that is natural in the object C corresponds to an internal operation .

For example, the intersection

corresponds a natural transformation

which, because the hom-functor Hom E preserves limits, is isomorphic to a natural transformation between hom-functors

which, by the Yoneda lemma, is isomorphic to an internal operation on the subobject classifier Ω :

Note that the subobject classifier, the classifying morphisms, and those internal operations are determined uniquely up to isomorphism. However, in order to obtain a multi-valued quantification, the requirement for uniqueness might be too restrictive. Thus, we propose to study a weakened concept instead:

Definition 19. In a finitely complete category, a subobject quantifier is a subobject o : 1 ↣ Ψ such that for every subobject b : B ↣ C , there exists at least one morphism ϕ b : C → Ψ such that b is a pullback of o along ϕ b . The morphism ϕ b is called a quantifying morphism of b . If the category has a subobject classifier ⊤ : 1 ↣ Ω , the quantizer κ : Ψ → Ω of the subobject quantifier o is the classifying morphism of o .

More succinctly, we can state that (cf. Proposition 5)

Proposition 6. A subobject quantifier is a weakly terminal object in the category of monomorphisms and pullbacks.

Thus, there is a unique morphism χ b : C → Ω classifying a subobject b : B ↣ C of an object C , but there could be multiple morphisms ϕ b : C → Ψ quantifying this subobject.

It is provable that the domain of a terminal object ⊤ in the category of monomorphisms and pullbacks (Proposition 5) must be a terminal object 1 in the category E , but not all weakly terminal objects in the category of monomorphisms and pullbacks (Proposition 6) are monomorphisms out of a terminal object. We choose Definition 19 because we want only one global element o : 1 ↣ Ψ to be designated to the truth value ⊤ : 1 ↣ Ω .

Here, we give two examples to motivate this definition of subobject quantifier. One is related to three-valued logic [Bergmann, 2008, Fong and Spivak, 2019, Exercise 2.34]:

Example 9. In Set , the function

which maps the element ∗ in a singleton set {∗} (a terminal object in Set ) to an element yes in a three-element set { no , maybe , yes } is a subobject quantifier. For any subset B of a set C , a quantifying morphism ϕ b : C → Ψ is a function that maps all elements in the subset B to yes and all other elements to either maybe or no .

The other is related to metric spaces [Lawvere, 1973] and will be our running example in the following subsections:

Example 10. In Set , the function 0 : {∗} → [0 , ∞ ] selecting the number 0 out of the set [0 , ∞ ] of extended non-negative real numbers is a subobject quantifier. The quantizer is a function

which maps 0 to ⊤ and any non-zero number to ⊥ .

Intuitively, with a subobject quantifier, there is only one way to be true, but there may be many ways to be false. In Set , a quantizer κ maps multiple ' degrees of truth ' from a potentially large, even infinite set Ψ to a smaller set Ω of truth values, hence the name.

Next, we define the counterpart of the concept of predicate (Definition 16):

Definition 20. In a category with a subobject quantifier o : 1 ↣ Ψ , a quantity on an object C is a morphism q : C → Ψ .

Since we weakened the requirement for uniqueness, there is no one-to-one correspondence between subobjects and quantities. However, they are still related as follows:

Lemma 7. In a category with a subobject classifier ⊤ : 1 ↣ Ω , a subobject quantifier o : 1 ↣ Ψ , and a quantizer κ : Ψ → Ω , a quantity q : C → Ψ on an object C is a quantifying morphism of a subobject q ∗ o of the object C , which is isomorphic to a subobject ( κ ◦ q ) ∗ ⊤ .

Proof. q ∗ o and ( κ ◦ q ) ∗ ⊤ are both subobjects of C because pullbacks preserve subobjects. Their isomorphism follows from the pullback lemma.

Lemma 8. In a category with a subobject classifier ⊤ : 1 ↣ Ω , a subobject quantifier o : 1 ↣ Ψ , and a quantizer κ : Ψ → Ω , a quantity q : C → Ψ on an object C is a quantifying morphism of a subobject b : B ↣ C if and only if κ ◦ q = χ b , where χ b is the classifying morphism of the subobject b .

Proof. Necessity follows from the pullback lemma and the uniqueness of the classifying morphism; sufficiency follows from Lemma 7.

The following relationship between a quantity and the quantizer is also useful:

Lemma 9. In a category with a subobject classifier ⊤ : 1 ↣ Ω , a subobject quantifier o : 1 ↣ Ψ , and a quantizer κ : Ψ → Ω , for any quantity q : C → Ψ on an object C , q = o C if and only if κ ◦ q = κ ◦ o C = ⊤ C , where o C is the constant morphism o ◦ e C : C e C - - → 1 o - → Ψ with value o : 1 → Ψ .

Proof. This is due to the universal property of pullback o of ⊤ along κ .

We can see that the hom-functor on the quantizer Hom E ( -, κ ) : Hom E ( -, Ψ) ⇒ Hom E ( -, Ω) is a natural transformation which maps the quantities Hom E ( C, Ψ) on an object C to the predicates Hom E ( C, Ω) on C , which are precisely subobjects of C .

B.2 Equality and premetric

Next, we take a closer look at a concrete and important predicate - equality - and its corresponding quantities.

Definition 21. In a category with a subobject classifier ⊤ : 1 → Ω , the equality predicate = C : C × C → Ω on an object C is the classifying morphism of the diagonal morphism ∆ C : C ↣ C × C := ⟨ id C , id C ⟩ .

By Lemma 8, a quantity d C : C × C → Ψ is a quantifying morphism of ∆ C if and only if κ ◦ d C = = C , depicted in the following diagram:

In Set , we have the following definitions:

Definition 22. A premetric on a set C is a binary function d C : C × C → [0 , ∞ ] such that

Or equivalently, d C ◦ ∆ C = 0 C , depicted in the following diagram:

Definition 23. A strict premetric on a set C is a premetric d C : C × C → [0 , ∞ ] such that

Or equivalently, ∆ C is a pullback of 0 along d C :

In other words, strict premetrics are precisely quantifying morphisms of the diagonal morphism ∆ C in the category Set with 0 : {∗} → [0 , ∞ ] as a subobject quantifier.

Note that the symmetry and the triangle inequality

which make d C a metric , are not required. The addition + and the order ≥ on the set [0 , ∞ ] are not needed to define a strict premetric. However, they are necessary for defining other operations and properties, which will be discussed in the next subsection.

B.3 Preorder

There is a preorder ⊆ C of inclusion defined on the set Sub E ( C ) of subobjects of an object C : for two subobjects a : A ↣ C and b : B ↣ C , a ⊆ C b if and only if there exists a morphism f : A → B such that a = b ◦ f :

This preorder on the subobjects Sub E ( C ) induces a preorder on the predicates Hom E ( C, Ω) via the isomorphism. We can generalize this construction and define a preorder on other hom-sets:

Definition 24. In a category E with pullbacks, the inclusion preorder ⊆ C on the set Sub E ( C ) of subobjects of an object C and a subobject m : S ↣ T induce a preorder ⪯ m C on the hom-set Hom E ( C, T ) via pullback of m : for any two morphisms f 1 , f 2 : C → T , f 1 ⪯ m C f 2 if and only if f ∗ 1 m ⊆ C f ∗ 2 m .

Based on this definition, we can explore the preorders on any hom-sets. From now, we assume that E is a category with necessary structures that we need.

Then, the preorder of predicates is ⪯ ⊤ C on Hom E ( C, Ω) , and the preorder of quantities is ⪯ o C on Hom E ( C, Ψ) . By Lemma 7, we know that for two quantities q 1 , q 2 : C → Ψ , q 1 ⪯ o C q 2 if and only if ( κ ◦ q 1 ) ⪯ ⊤ C ( κ ◦ q 2 ) , which means that Hom E ( C, κ ) is an order-preserving function from (Hom E ( C, Ψ) , ⪯ o C ) to (Hom E ( C, Ω) , ⪯ ⊤ C ) .

Next, we will explore the structures of Hom E ( C, Ω) and Hom E ( C, Ψ) . To begin with, ⊤ C is a top in Hom E ( C, Ω) , and o C is a top in Hom E ( C, Ψ) , because id C is a top in Sub E ( C ) .

The inclusion preorder on Sub E ( C ) also has a bottom - the initial morphism i C : 0 ↣ C from an initial object 0 in the category E to the object C . Then, its classifying morphism ⊥ C : C → Ω is a bottom in Hom E ( C, Ω) . It can be proven that ⊥ C is a constant morphism with value ⊥ : 1 → Ω , which is the classifying morphism of the initial/terminal morphism 0 ↣ 1 . Any quantifying morphism ψ C : C → Ψ of i C is a bottom in Hom E ( C, Ψ) , but it is not necessarily a constant morphism.

0

1

1

C

Ψ

Ω

i

C

⌟

o

⌟

⊤

ψ

C

⊥

C

κ

(50)

The preorder on the global elements plays a special role:

Example 11. In Set , ⪯ ⊤ 1 , also denoted by ⊢ , is a preorder on the set {⊤ , ⊥} with only one nonidentity relation ⊥⊢⊤ .

Example 12. In Set , ⪯ 0 1 is a preorder on the set [0 , ∞ ] where n ⪯ 0 1 0 for any number n , and m ⪯ 0 1 n and n ⪯ 0 1 m for any positive numbers m and n .

By definition, ( {⊤ , ⊥} , ⊢ ) and ([0 , ∞ ] , ⪯ 0 1 ) are equivalent. However, we can consider a suborder of ([0 , ∞ ] , ⪯ 0 1 ) , e.g., the usual ' greater than or equal to ' ≥ total order, to further differentiate positive numbers. Note that 0 remains the top in this suborder ([0 , ∞ ] , ≥ ) .

B.4 Operation: algebra over the product endofunctor

In an elementary topos E , the subobjects Sub E ( C ) of an object C not only forms a preorder by inclusion but also are equipped with certain set operations (e.g., intersection and disjoint union), which are defined in terms of the universal properties of their corresponding order operations (e.g., meet and join). Further, these operations are reflected in the structures of the subobject classifier (e.g., conjunction and disjunction).

Here, we establish a link between the structures of the subobject classifier and those of a subobject quantifier. Our primary result is as follows:

Theorem 10. Consider a category with a subobject classifier ⊤ : 1 ↣ Ω , a subobject quantifier o : 1 ↣ Ψ , and a quantizer κ : Ψ → Ω .

Let n be a natural number. Let β : Ω n → Ω be an n -ary logical operation on Ω , and let α : Ψ n → Ψ be an n -ary quantitative operation on Ψ .

For i ∈ { 1 , . . . , n } , let p i : C → Ω be a predicate on an object C , and let q i : C → Ψ be a quantity on the object C such that p i = κ ◦ q i . Let p = ⟨ p 1 , . . . , p n ⟩ be the tupling of the predicates, and let q = ⟨ q 1 , . . . , q n ⟩ be the tupling of the quantities. Let b : B ↣ C := ( β ◦ p ) ∗ ⊤ be the subobject classified by β ◦ p , and let a : A ↣ C := ( α ◦ q ) ∗ o be the subobject quantified by α ◦ q .

∗

A

α

1

1

B

β

∗

1

1

C

Ψ

n

Ψ

C

Ω

n

Ω

a

⌟

α

∗

o

⌟

o

b

⌟

⌟

q

id

C

α

κ

n

κ

p

β

β

∗

⊤

⊤

(51)

Then, we have

For convenience, we call an n -ary operation α : Ψ n → Ψ (as an algebra over the product endofunctor ( -) n ) homomorphic to β : Ω n → Ω via a morphism κ : Ψ → Ω if

and subhomomorphic to β : Ω n → Ω if it satisfies the condition

Proof. (i) follows from the property of tupling and product: κ n ◦ q = ⟨ κ ◦ q 1 , . . . , κ ◦ q n ⟩ = ⟨ p 1 , . . . , p n ⟩ = p .

(ii) states that if α is subhomomorphic to β , then β ◦ p is the classifying morphism of the subobject quantified by α ◦ q .

(iii) means that α being homomorphic to β is a stronger condition than being merely subhomomorphic to β .

=

κ

◦

o

◦

o

∗

α

(pullback) (61)

(iv) shows the relationship between the predicate β ◦ p and the quantity α ◦ q when α is homomorphic to β .

κ ◦ α ◦ q (63) = β ◦ κ n ◦ q (homomorphism) (64) = β ◦ p (i) (65)

0

1

2

3

4

5

0

1

2

3

4

5

'p

n

q

r

n

'

0

s

1

´

n

1

n

e

´

n

n

(v) means that if α is homomorphic to β , then α ◦ q is a quantifying morphism of b .

In summary, if α is subhomomorphic to β , then a is included in b ((ii)); if α is homomorphic to β , then a and b are isomorphic ((iii) and (v)). We consider this weaker condition because subhomomorphic but non-homomorphic operations may exhibit favorable properties in other aspects, such as continuity. We will discuss several concrete examples in the following subsections.

B.5 Negation

First, let us take a closer look at a unary logical operation negation ¬ : Ω → Ω , which is defined as the classifying morphism of ⊥ : 1 → Ω . Recall that ⊥ is the classifying morphism of 0 ↣ 1 .

Let us consider a unary quantitative operation ∼ : Ψ → Ψ . If ∼ is homomorphic to ¬ via the quantizer κ , it means that ¬ ◦ κ = κ ◦ ∼ , or the following diagram commutes:

Let us consider the set [0 , ∞ ] in Set . A quantitative operation ∼ : [0 , ∞ ] → [0 , ∞ ] homomorphic to the negation ¬ : {⊤ , ⊥} → {⊤ , ⊥} is a function

which maps 0 to a non-zero number n 0 and any non-zero number to 0 . 11 However, this function is discontinuous at 0 . On the other hand, if we consider a quantitative operation ∼ subhomomorphic to the negation ¬ , then the only requirement is that for all n , ∼ ( n ) = 0 implies n > 0 , or, by contraposition, ∼ (0) > 0 . Continuous choices include the hinge function n ↦→ 1 ´ n = max { 1 -n, 0 } , reciprocal function n ↦→ 1 n , and exponential decay function n ↦→ e -n (see Fig. 3). Note that the latter

two are actually homomorphic to the constant false ⊥ because their outputs are always non-zero. The hinge function 1 ´ q , as discussed later, can be seen as derived from the implication p →⊥ .

In this way, if we have a quantity q for a predicate p , we can obtain a quantity ∼ q for the negation ¬ p of the predicate as well. If the quantitative operation ∼ is subhomomorphic but not homomorphic to the logical operation ¬ , we can guarantee that for any x , ∼ q ( x ) = 0 implies ¬ p ( x ) = ⊤ , but not vice versa.

B.6 Conjunction

Next, let us move on to an important binary logical operation conjunction ∧ : Ω × Ω → Ω , which is defined as the classifying morphism of ⟨⊤ , ⊤⟩ : 1 ↣ Ω × Ω . Similarly, we consider a binary quantitative operation ⊗ : Ψ × Ψ → Ψ homomorphic to the conjunction ∧ via the quantizer κ :

By abuse of notation, the conjunction ∧ also denotes a binary operation on the set Hom E ( C, Ω) of predicates, such that for any two predicates p 1 , p 2 ∈ Hom E ( C, Ω) ,

The same goes for the quantitative operation ⊗ .

For the conjunction, we can prove a stronger result:

Theorem 11. Consider a category with a subobject classifier ⊤ : 1 ↣ Ω , a subobject quantifier o : 1 ↣ Ψ , and a quantizer κ : Ψ → Ω .

Let the conjunction ∧ : Ω × Ω → Ω be the classifying morphism of ⟨⊤ , ⊤⟩ : 1 ↣ Ω × Ω , and let ⊗ : Ψ × Ψ → Ψ be a binary operation on Ψ homomorphic to the conjunction ∧ via the quantizer κ .

Let p 1 , p 2 : C → Ω be two predicates on an object C , and let q 1 , q 2 : C → Ψ be two quantities on the object C such that p 1 = κ ◦ q 1 and p 2 = κ ◦ q 2 . Let p = ⟨ p 1 , p 2 ⟩ be the pairing of the predicates, and let q = ⟨ q 1 , q 2 ⟩ be the pairing of the quantities. Let b : B ↣ C := ( p 1 ∧ p 2 ) ∗ ⊤ be the subobject classified by p 1 ∧ p 2 .

Then, ⊗ is a quantifying morphism of ⟨ o, o ⟩ , and b is a pullback of ⟨ o, o ⟩ along ⟨ q 1 , q 2 ⟩ .

Proof. If q 1 ⊗ q 2 = o C , then ( κ ◦ q 1 ) ∧ ( κ ◦ q 2 ) = ⊤ C , which leads to ⟨ κ ◦ q 1 , κ ◦ q 2 ⟩ = ⟨⊤ , ⊤⟩◦ e C = ⟨⊤ C , ⊤ C ⟩ due to the universal property of pullback. Then, we have κ ◦ q 1 = κ ◦ q 2 = ⊤ C due to the universal property of pairing, and consequently q 1 = q 2 = o C according to Lemma 9, i.e., ⟨ q 1 , q 2 ⟩ = ⟨ o, o ⟩ ◦ e C . Therefore, ⟨ o, o ⟩ is a pullback of o along ⊗ . According to Theorem 10, b is a pullback of o along q 1 ⊗ q 2 . Then, following the pullback lemma, b is a pullback of ⟨ o, o ⟩ along ⟨ q 1 , q 2 ⟩ .

In other words, the pullback square symbols in Eq. (73) are unambiguous - the top row, the left column, the right column, the top-left square, and the top-right square are all pullbacks.

Note that for any quantities a, b, c : C → Ψ , we have

due to the associativity, commutativity, and unitality of the conjunction, but the quantitative operation ⊗ is not required to satisfy these properties, i.e., it is possible that

̸

̸

̸

However, in the following examples, we mainly consider quantitative operations ⊗ such that these properties are satisfied. In such cases, (Hom E ( C, Ψ) , ⊗ , o C ) forms a commutative monoid, and consequently Hom E ( C, κ ) is a monoid homomorphism from it to (Hom E ( C, Ω) , ∧ , ⊤ C ) .

In Appendix B.3, we introduced preorder structures on the predicates Hom E ( C, Ω) and the quantities Hom E ( C, Ψ) . The conjunction ∧ is a commutative monoidal structure compatible with the preorder (Hom E ( C, Ω) , ⪯ ⊤ C ) because it is the meet operation. For the set Hom E ( C, Ψ) of quantities, we can choose the quantitative operation ⊗ to be the meet operation as well. Alternatively, we can only require it to be compatible with the preorder, in the sense that the monoid product is order-preserving: for any quantities q 1 , q ′ 1 , q 2 , q ′ 2 ∈ Hom E ( C, Ψ) , if q 1 ⪯ o C q ′ 1 and q 2 ⪯ o C q ′ 2 , then q 1 ⊗ q 2 ⪯ o C q ′ 1 ⊗ q ′ 2 . In other words, we require (Hom E ( C, Ψ) , ⪯ o C , ⊗ , o C ) to be a symmetric monoidal preorder [Fong and Spivak, 2019, Definition 2.2].

Example 13. Let us consider two commutative monoidal structures on the preorder ([0 , ∞ ] , ≥ ) . The max operation max : [0 , ∞ ] × [0 , ∞ ] → [0 , ∞ ] is the meet, i.e., cartesian product, while the addition + : [0 , ∞ ] × [0 , ∞ ] → [0 , ∞ ] is a monoidal product. They are both semicartesian because the top 0 is the unit.

Note that ([0 , ∞ ] , + , 0) , ([0 , 1] , × , 1) , and ([1 , ∞ ] , × , 1) are isomorphic to each other with the following isomorphisms:

We can also induce a monoidal structure on a set if it is isomorphic to a monoid:

Lemma 12. Let f : A ⇄ B : g be a pair of bijections between sets A and B . If ( B, ⊗ B , I B ) is a monoid, then ( A, ⊗ A := g ◦ ⊗ B ◦ ( f × f ) , I A := g ◦ I B ) is also a monoid.

Proof. Associativity:

Left unitality:

In this way, we can obtain a richer choice of monoidal structures on the set [0 , ∞ ] beyond the addition (see Fig. 4):

Example 14 (Semicartesian monoidal product) .

In summary, the max operation on [0 , ∞ ] can be regarded as a continuous conjunction , whereas a semicartesian monoidal product, such as the addition, can be viewed as a soft max .

B.7 Disjunction

Dually, the disjunction ∨ : Ω × Ω → Ω reflects the join of the inclusion preorder of subobjects. Similarly, we want to find a quantitative operation ⊕ : Ψ × Ψ → Ψ homomorphic to the disjunction ∨ via the quantizer κ :

Using the same technique as in Lemma 12, we can obtain several monoidal structures on the set [0 , ∞ ] homomorphic to the disjunction (see Fig. 5):

Example 15 (Semicocartesian monoidal product) .

However, we usually choose the quantitative operation ⊕ to be the join operation min of the preorder ([0 , ∞ ] , ≥ ) . In this way, ⊗ distributes over ⊕ , i.e., for any quantities a, b, c : C → Ψ , we have

A typical example is the min-plus semiring ([0 , ∞ ] , min , +) [Pin, 1998].

B.8 Implication

Finally, let us construct a quantitative counterpart of the implication → : Ω × Ω → Ω , which is right adjoint to the conjunction ∧ . For global elements a, b, c : 1 → Ω , this means that

There are two ways to construct a quantitative operation ⊸ : Ψ × Ψ → Ψ corresponding to the implication → . One way is to find a quantitative operation ⊸ homomorphic to the implication → via the quantizer κ :

Example 16. For the set [0 , ∞ ] , a quantitative operation ⊸ : [0 , ∞ ] × [0 , ∞ ] → [0 , ∞ ] homomorphic to the implication → : {⊤ , ⊥} × {⊤ , ⊥} → {⊤ , ⊥} is a function

where f : (0 , ∞ ] → (0 , ∞ ] is an arbitrary function to non-zero numbers. Note that this function is discontinuous at the line a = 0 and b > 0 (cf. Appendix B.5).

The other way is to find a quantitative operation ⊸ right adjoint to an operation ⊗ homomorphic to the conjunction ∧ , i.e., the internal hom of the monoidal closed preorder [Fong and Spivak, 2019, Definition 2.79].

Example 17. For the meet-semilattice ([0 , ∞ ] , ≥ , max , 0) , the function

is right adjoint to the max because

While this function is subhomomorphic to the implication, it is still discontinuous at the line a = b . Example 18. For the monoidal preorder ([0 , ∞ ] , ≥ , + , 0) , the truncated subtraction ´ (a.k.a. monus [Amer, 1984])

is right adjoint to the addition because

The truncated subtraction is continuous and subhomomorphic to the implication. Note that the hinge function n ↦→ max { 1 -n, 0 } = 1 ´ n for the negation can be interpreted as the quantitative operation derived from ¬ n = n →⊥ , where 1 is homomorphic to the constant false ⊥ .

Similarly to Lemma 12, if two symmetric monoidal preorders are isomorphic and one of them is closed, we can induce that the other is also closed:

Lemma 13. Let f : A ⇄ B : g be a pair of isomorphisms between symmetric monoidal preorders ( A, ⪯ A , ⊗ A , I A ) and ( B, ⪯ B , ⊗ B , I B ) . If ( B, ⪯ B , ⊗ B , I B , ⊸ B ) is closed, then ( A, ⪯ A , ⊗ A , I A , ⊸ A := g ◦ ⊸ B ◦ ( f × f )) is also closed.

Proof. For all a, b, c ∈ A , we have

This means that ⊸ A is right adjoint to ⊗ A .

In this way, we can find the internal homs corresponding to the monoidal products discussed in Appendix B.6 (see Fig. 6).

As a side note, we can use the quantitative operations discussed above to define quantitative operations for other logical connectives. For example, the logical equivalence a ↔ b can be represented as ( a → b ) ∧ ( b → a ) (bi-implication), ( ¬ a ∨ b ) ∧ ( ¬ b ∨ a ) (conjunctive normal form (CNF)), or ( a ∧ b ) ∨ ( ¬ a ∧ ¬ b ) (disjunctive normal form (DNF)), and its quantitative operations can be defined accordingly. Some examples are shown in Fig. 7.

B.9 Heyting algebra, quantale, and ordered semiring

Now, having constructed the logical operations and their corresponding quantitative operations, we can compare the structures of the subobject classifier with those of a subobject quantifier.

It is known that the global elements of the subobject classifier with the logical operations form a Heyting algebra (Ω , ⊢ , ⊤ , ⊥ , ∧ , ∨ , → ) :

Definition 25. A Heyting algebra is a cartesian closed bounded lattice.

In categorical terms, ⊤ is the terminal object, ⊥ is the initial object, ∧ is the product, ∨ is the coproduct, and → is the exponential in the preorder (Ω , ⊢ ) (as a thin category).

We weakened the requirements to construct the algebraic structures of the subobject quantifier, which usually forms what is called a quantale (Ψ , ⪯ , 1 , 0 , ⊗ , ⊕ , ⊸ ) [Mulvey, 1986, Dudzik, 2017]:

Definition 26. A (unital) quantale is a monoidal closed suplattice.

This means that we can consider a preorder (Ψ , ⪯ ) on the subobject quantifier as a thin category, where 1 is the terminal object, 0 is the initial object, ⊗ is a monoidal product and not necessarily the product, ⊕ is still the coproduct, and ⊸ is the internal hom right adjoint to the monoidal product ⊗ . An example is the Lawvere quantale ([0 , ∞ ] , ≥ , 0 , ∞ , + , min , ´ ) [Lawvere, 1973, Bacci et al., 2023]. Then, the quantizer κ : Ψ → Ω is a homomorphism preserving some or all the structures.

Note that due to the adjoint functor theorem and the fact that left adjoints preserve colimits, the product distributes over the coproduct in a Heyting algebra, and the monoidal product distributes over the coproduct in a quantale. We can further relax the requirement for ⊕ to be the coproduct and instead consider a monoidal product (Appendix B.7). If we still require the distributivity for the two monoidal structures ⊗ and ⊕ , the algebraic structure is an ordered semiring [Fujii, 2023]. Further investigation is left for future work.

B.10 Quantifier: algebra over the exponentiation endofunctor

Up to this point, our focus has been on n -ary operations in propositional logic (Appendix B.4). Next, we introduce the universal quantification ∀ and existential quantification ∃ used in predicate logic and their quantitative counterparts.

Externally, the universal quantification and the existential quantification are right and left adjoint to the pullback of projection, respectively [Lawvere, 1969]. Internally, the universal quantifier ∀ D : Ω D → Ω and existential quantifier ∃ D : Ω D → Ω are given by morphisms from the power object Ω D of an object D to the subobject classifier Ω .

Recall that the power object Ω D is also an exponential object into the subobject classifier Ω , so the universal quantifier ∀ D and existential quantifier ∃ D can also be viewed as algebras over the exponentiation endofunctor ( -) D of exponentiation on the subobject classifier Ω . Then, we can consider algebras on the subobject quantifier Ψ homomorphic to them, which serve as quantitative counterparts of these quantifiers.

Our main result is as follows (cf. Theorem 10):

Theorem 14. Consider an elementary topos with a subobject classifier ⊤ : 1 ↣ Ω , a subobject quantifier o : 1 ↣ Ψ , and a quantizer κ : Ψ → Ω .

Let p : C × D → Ω be a predicate on a product, and let q : C × D → Ψ be a quantity such that p = κ ◦ q . Let ̂ p : C → Ω D and ̂ q : C → Ψ D be their exponential transposes.

Let β D : Ω D → Ω be a predicate, and let α D : Ψ D → Ψ be a quantity homomorphic to β D via the quantizer κ , i.e., β D ◦ κ D = κ ◦ α D .

Let b : B ↣ C := ( β D ◦ ̂ p ) ∗ ⊤ be the subobject classified by β D ◦ ̂ p , and let a : A ↣ C := ( α D ◦ ̂ q ) ∗ o be the subobject quantified by α D ◦ ̂ q .

A

1

B

1

C

Ψ

D

Ψ

C

Ω

D

Ω

a

⌟

o

b

⌟

̂

q

id

C

α

D

κ

D

κ

p

β

D

⊤

(113)

Then, we have

Proof. (i) follows from the property of exponential.

(ii) shows the relationship between the predicate β D ◦ ̂ p and the quantity α D ◦ ̂ q when α D is homomorphic to β D .

(iii) means that β D ◦ ̂ p is a classifying morphism of a .

̂

(117)

(ii) (118)

(pullback) (119)

(composition) (120)

(iv) means that α D ◦ ̂ q is a quantifying morphism of b .

Definition 27 (Universal aggregator) . A universal aggregator ▽ D : Ψ D → Ψ is a quantity that is homomorphic to the universal quantifier ∀ D : Ω D → Ω :

Definition 28 (Existential aggregator) . An existential aggregator △ D : Ψ D → Ψ is a quantity that is homomorphic to the existential quantifier ∃ D : Ω D → Ω :

Example 19. For the set [0 , ∞ ] , the canonical choices of universal aggregator ▽ D and existential aggregator △ D are sup and inf . If the set D is finite, we can also use sum and mean as the universal aggregator. Non-examples of universal aggregator include median and mode , which are not homomorphic to the universal quantifier.

Note that the universal quantifier and existential quantifier are commutative up to isomorphism:

However, we are free to choose different aggregators for different objects that are not necessarily commutative. For example, the sum of max is usually not equal to the max of sum.

B.11 Enrichment

Lastly, we describe the conversion based on enrichment.

First, let us define the enriching category:

Definition 29. Let (Ψ , ⪯ , ⊗ , ⊕ , ⊸ ) be an internal quantale object in Set . We define ΨSet to be a category whose objects are tuples consisting of a set C , a Ψ -valued strict premetric d C on C , and universal and existential aggregators on C :

and morphisms from ( A,d A , ▽ A , △ A ) to ( B,d B , ▽ B , △ B ) are functions f : A → B .

Definition 30. A bifunctor ⊠ on ΨSet is given by the Cartesian product of sets and functions, together with the following products of strict premetrics and aggregators:

Proposition 15. The product d A ⊠ d B of strict premetrics given in Definition 30 is again a strict premetric.

Proof. This is a result of Theorem 11. Consider the following diagram:

∆ A × ∆ B is a pullback of o along ⊗◦ ( d A × d B ) because ⟨ o, o ⟩ is a pullback of o along ⊗ according to Theorem 11, ∆ A × ∆ B is a pullback of ⟨ o, o ⟩ along d A × d B , and we can apply the pullback lemma. ∆ A × ∆ B is isomorphic to ∆ A × B , which means that ⊗◦ ( d A × d B ) is a strict premetric.

Based on this definition, we can show that

because × and ⊗ are associative up to isomorphism.

Further, we have a universal aggregator

because composition is associative.

The singleton ( {∗} , o {∗}×{∗} , id {∗} , id {∗} ) is the unit of the bifunctor ⊠ . In this way, (ΨSet , ⊠ , {∗} ) forms a monoidal category.

However, note that the bifunctor ⊠ is not symmetric because ▽ A ⊠▽ B and △ A ⊠△ B are not necessarily symmetric, and d A ⊠ d B ∼ = d B ⊠ d A if and only if the monoidal product ⊗ of the quantale Ψ is commutative.

Since ΨSet is a monoidal category, we can consider a strict monoidal functor F Ψ : Set → ΨSet , which equips products of sets with product strict premetrics and product aggregators:

Such a monoidal functor induces a functor from a ( Set -enriched) category to a ΨSet -enriched category, called the base change of enriching category. This means that we have a systematic way to equip a set [ A,B ] of morphisms with a strict premetric

and an existential aggregator which is compatible with the product.

Then, Theorem 1 is a special case of this enrichment. The relationship between predicates and quantities follows from Theorems 10 and 14.

B.12 Summary

Finally, to accommodate readers without a background in category theory, we present the instantiated definitions and theoretical results free of categorical terminology. The non-categorical proofs are omitted. We will be using the following functions:

the zero predicate ζ : [0 , ∞ ] →{⊤ , ⊥} := x ↦→ ( x = 0) ,

the postcomposition ζ A : [0 , ∞ ] A →{⊤ , ⊥} A := q ↦→ ζ ◦ q of the zero predicate.

the product ζ n : [0 , ∞ ] n →{⊤ , ⊥} n : ( q 1 , . . . , q n ) ↦→ ( q 1 = 0 , . . . , q n = 0) , and

Definition 31 (Quantity) . Aquantity q : A → [0 , ∞ ] is homomorphic to a predicate p : A →{⊤ , ⊥} if p = ζ ◦ q :

A quantity q is subhomomorphic to a predicate p if ζ ◦ q → p

:

Example 20. A strict premetric d A : A × A → [0 , ∞ ] is a quantity on the product set A × A homomorphic to the equality predicate = A : A × A →{⊤ , ⊥} .

Definition 32 (Quantitative operation) . Let n ∈ N be a natural number. A quantitative operation α : [0 , ∞ ] n → [0 , ∞ ] is homomorphic to a logical operation β : {⊤ , ⊥} n →{⊤ , ⊥} via the zero predicate ζ if ζ ◦ α = β ◦ ζ n :

A quantitative operation α is subhomomorphic to a logical operation β via the zero predicate if ζ ◦ α → β ◦ ζ n :

The relationship between quantitative operations and logical operations is as follows (Theorem 10): Proposition 16. Let n ∈ N be a natural number. For i ∈ { 1 , . . . , n } , let p i : A → {⊤ , ⊥} be a predicate, and let q i : A → [0 , ∞ ] be a quantity. Let p : A → {⊤ , ⊥} n := ⟨ p 1 , . . . , p n ⟩ be the tupling of the predicates, and let q : A → [0 , ∞ ] n := ⟨ q 1 , . . . , q n ⟩ be the tupling of the quantities. Let α : [0 , ∞ ] n → [0 , ∞ ] be a quantitative operation, and let β : {⊤ , ⊥} n →{⊤ , ⊥} be a logical operation. Assume that for all i ∈ { 1 , . . . , n } , q i is homomorphic to p i . Then,

if α is homomorphic to β , α ◦ q homomorphic to β ◦ p ; and

if α is subhomomorphic to β , α ◦ q subhomomorphic to β ◦ p .

In fact, we can also show that if for all i ∈ { 1 , . . . , n } , q i is subhomomorphic to p i , and α is subhomomorphic to β , then α ◦ q is subhomomorphic to β ◦ p .

Definition 33 (Aggregator) . An aggregator α A : [0 , ∞ ] A → [0 , ∞ ] is homomorphic to a quantifier β A : {⊤ , ⊥} A →{⊤ , ⊥} via the zero predicate ζ if ζ ◦ α A = β A ◦ ζ A :

Example 21. A universal aggregator ▽ A : [0 , ∞ ] A → [0 , ∞ ] is a function such that

Example 22. An existential aggregator △ A : [0 , ∞ ] A → [0 , ∞ ] is a function such that

The relationship between aggregators and quantifiers is as follows (Theorem 14):

Proposition 17. Let p : A × B → {⊤ , ⊥} be a predicate, and let q : A × B → [0 , ∞ ] be a quantity. Let ̂ p : B →{⊤ , ⊥} A and ̂ q : B → [0 , ∞ ] A be the exponential transposes of p and q . Let α A : [0 , ∞ ] A → [0 , ∞ ] be an aggregator, and let β A : {⊤ , ⊥} A →{⊤ , ⊥} be a quantifier. Then, if q is homomorphic to p , and α is homomorphic to β , then α A ◦ ̂ q is homomorphic to β A ◦ ̂ p .

We have a compositional way to assign a strict premetric and an aggregator to a product of sets: Proposition 18. Let d A : A × A → [0 , ∞ ] and d B : B × B → [0 , ∞ ] be strict premetrics. Then,

Proposition 19. Let ▽ A : [0 , ∞ ] A → [0 , ∞ ] and ▽ B : [0 , ∞ ] B → [0 , ∞ ] be universal aggregators. Then,

is a universal aggregator on the product set A × B .

C Proofs

C.1 Proposition 2

Proof.

where

C.2 Proposition 3

Proof.

=

2

2

y

1

∈

Y

1

y

∈

Y

2

y

′

2

∈

Y

=

2

2

y

1

∈

Y

1

y

∈

Y

2

y

′

2

∈

Y

=

1

2

y

′

2

∈

Y

2

y

∈

Y

1

y

2

∈

Y

d

(

m

(

y

, y

)

, m

(

y

, y

′

2

))

d

(

m

(

y

, y

)

, m

(

y

, y

′

2

))

d

(

m

(

y

)(

y

)

,

m

(

y

′

2

)(

y

))

Z

Z

Z

1

1

1

1

̂

1

1

2

1

1

1

2

2

1

̂

1

1

1

1

1

D Discussions

D.1 Background

Defining and measuring the properties of learning models is a core topic in machine learning, especially representation learning [Bengio et al., 2013]. A proper comprehension of what constitutes good representations and how to assess their quality is important for developing suitable learning objectives and evaluation metrics. To define these properties, many important concepts are given by equational predicates , such as independence of random variables, extensively used in statistical learning and causal learning [Hyvärinen and Oja, 2000, Koller and Friedman, 2009, Schölkopf and von Kügelgen, 2022], and equivariance of learning models, reflecting the symmetries and structures of the data [Cohen and Welling, 2016, Zaheer et al., 2017, Higgins et al., 2018, Maron et al., 2019, de Haan et al., 2020, Cohen, 2021, van der Pol et al., 2022, Navon et al., 2023].

Considerable efforts have been put into designing model architectures that perfectly satisfy specific properties, such as monotonicity [Sill, 1997, Daniels and Velikova, 2010], invertibility [Rezende and Mohamed, 2015, Behrmann et al., 2019, Ishikawa et al., 2023], convexity [Amos et al., 2017], and equivariance [Lee et al., 2019, Brehmer et al., 2023]. However, hard-coding multiple properties into a model by design could be challenging [Köhler et al., 2020]. Hence, it is desirable to devise quantitative metrics to directly measure these properties, even if the models do not have the properties built-in [Goodfellow et al., 2009, Chen et al., 2020, Kvinge et al., 2022]. Ideally, these metrics should be easily computable or even differentiable, allowing us to directly optimize the properties.

Disentangled representation learning [Bengio et al., 2013], our main focus of this paper, is such a field where defining and measuring the desired properties are not straightforward tasks [Carbonneau et al., 2022, Zhang and Sugiyama, 2023]. It has been suggested that disentangling the underlying explanatory factors in complex data is a promising approach for reliable, interpretable, generalizable, and data-efficient representation learning [Locatello et al., 2019a,b, Montero et al., 2021, Dittadi et al., 2021, Xu et al., 2022]. However, in contrast to the wealth of results regarding invariant and equivariant layers, the exploration of designing a ' disentangled layer ' has been relatively limited. One reason is that disentanglement was not considered a singular property but rather a combination of several requirements. The absence of a clear definition and appropriate metrics for disentanglement has created a gap between the learning objectives and evaluation metrics. A new evaluation metric is often introduced along with a new representation learning method [Carbonneau et al., 2022], but it is usually unproven that the method can optimize the new metric, and the metric truly quantifies the alleged property [Higgins et al., 2017, Kim and Mnih, 2018, Chen et al., 2018, Li et al., 2020].

To formally define disentanglement, a line of research utilized group theory and representation theory [Cohen and Welling, 2014, 2015, Higgins et al., 2018], with a focus on the direct product of groups . Thanks to the rich algebraic structure, it becomes possible to derive various model architectures and learning objectives from the equational requirements of the product and equivariance [Caselles-Dupré et al., 2019, Pfau et al., 2020, Quessard et al., 2020, Painter et al., 2020, Miyato et al., 2022, Yang et al., 2022, Tonnaer et al., 2022, Keurti et al., 2023]. Another approach adopted a topological perspective, using concepts such as the product manifold to define disentanglement [Zhou et al., 2020, Fumero et al., 2021, Zhang et al., 2021, Balabin et al., 2024]. However, theoretically comparing different approaches has been a challenging task.

To quantitatively measure disentanglement, Ridgeway and Mozer [2018] proposed three concepts called modularity, compactness, and explicitness , which were defined verbally but not mathematically. Eastwood and Williams [2018] proposed similar three criteria called disentanglement, completeness, and informativeness and corresponding evaluation metrics. However, it was unclear what properties these metrics truly quantify. Additionally, due to the necessity for additional training of classifiers along with hyperparameter tuning and the involvement of non-differentiable regressors such as the random forest [Breiman et al., 1984], it is impossible to directly optimize these metrics using gradient-based optimization. Recently, Eastwood et al. [2023] extended this framework with two new metrics called explicitness/ease-of-use and size based on the functional capacity. Do and Tran [2020] introduced metrics for informativeness, separability, independence, and interpretability from an information-theoretic perspective, while Tokui and Sato [2022] introduced a new metric in terms of uniqueness, redundancy, and synergy based on partial information decomposition. These metrics have been mainly used during the evaluation stage, after a model is trained with other learning objectives.

D.2 Related work

Equivariance The work by Kvinge et al. [2022] might be the closest to our approach in spirit. They directly converted equivariance , an equational predicate, to a quantitative metric and analyzed their relationship (Proposition 3.2). In contrast, based on our proposed conversion method, we can use the following definition and metric:

Definition 34 (Equivariant function) . Let A , B , and C be sets. A function f : A → B is equivariant to actions (any binary functions) · A : C × A → A and · B : C × B → B if

which can be measured by

Calibration More broadly, the study of the relationship between different metrics in statistical learning is called calibration analysis [Steinwart, 2007, Reid and Williamson, 2010, Ni et al., 2019, Bao and Sugiyama, 2020, Bao et al., 2020]. Our work can be seen as an extension of the concept of the calibration to a wider range of properties defined by equational predicates.

Disentanglement metric In disentangled representation learning, metrics similar to Eq. (26) have been proposed by Higgins et al. [2017], Kim and Mnih [2018]. Their metrics also fix one factor and vary all others and calculate some constancy metrics (the mean pairwise distance in Higgins et al. [2017] and the variance in Kim and Mnih [2018]). However, both studies took an indirect approach, involving the training of a classifier to predict the fixed factor. Consequently, the resulting metrics are not differentiable anymore and entangle modularity and informativeness. In this work, we argue that it is better to measure these two properties separately.

Weakly supervised disentanglement Ridgeway and Mozer [2018] proposed and investigated the similarity supervision and argued that such supervision is easy to obtain via crowdsourcing. Shu et al. [2020] further studied this type of supervision based on distribution matching and referred it as match pairing. Other weaker forms of supervision were also investigated, such as the number of changed factors [Locatello et al., 2020] or paired data with unknown intervention [Brehmer et al., 2022]. Given that our theory can establish connections between logical definitions and quantitative metrics, it holds promise for deriving disentanglement metrics for various types of weak supervision based on logical inference.

Multi-valued logic Aristotelian logic assumes that every proposition is either true or false, adhering to the principle of bivalence . The law of Aristotelian logic can be algebraically represented on the set { 0 , 1 } of binary truth values [Boole, 1854], known as the two-element Boolean algebra . The exploration of non-Aristotelian logic involves investigating logical systems that relax or modify this strict binary valuation, allowing for a broader range of truth values and accommodating various forms of uncertainty, vagueness, or context-dependence in reasoning [Hájek, 1998, Malinowski, 2007, Bergmann, 2008].

The mathematical study of multi-valued logic can date back to the seminal work by Łukasiewicz in 1920, who introduced a third truth value interpreted as ' possibility ' and symbolized by 1 2 . Łukasiewicz [1920] examined several principles in this three-valued logic such as the principles of identity, implication, syllogism, and contradiction, and discussed its theoretical and practical importance in indeterministic philosophy and deductive sciences. Later, Łukasiewicz and Tarski [1930] proposed propositional calculus , a theory of propositions with values from the real interval [0 , 1] , which is now also commonly known as Łukasiewicz logic . Łukasiewicz logic involves new continuous logical connectives such as strong/weak conjunction and disjunction.

Furthermore, Chang [1958] studied the algebraic systems for many-valued logic , called MV-algebras . Chang and Keisler [1966] then proposed continuous model theory , also referred to as compact-valued logic [Ben Yaacov, 2022], where the truth values can be in arbitrary compact Hausdorff spaces and a wide variety of quantifiers was studied. Later, Ben Yaacov et al. [2008] studied model theory for metric structures and proposed (real-valued) continuous first-order logic [Ben Yaacov and Usvyatsov,

2010], where the space of truth values is a closed, bounded interval of real numbers with the order topology (e.g., [0 , 1] ), and suggested that we only need two canonical quantifiers sup and inf . From a categorical perspective, Cho [2020] developed categorical semantics of metric spaces and continuous logic by introducing the notion of continuous subobject classifier , and Figueroa and van den Berg [2022] studied a topos of continuous logic using the notion of hyperdoctrine .

On the other hand, from a categorical perspective, Lawvere [1973] showed that a generalized metric space, also known as a Lawvere metric space , is a category enriched over what is now commonly called the Lawvere quantale ([0 , ∞ ] , ≥ , + , 0) , i.e., the set [0 , ∞ ] of extended non-negative real numbers equipped with addition + as a (semicartesian) monoidal product and truncated subtraction ´ as the internal hom. In other words, a Lawvere metric space is a set A equipped with a function d : A × A → [0 , ∞ ] such that for all a ∈ A , we have 0 ≥ d ( a, a ) or d ( a, a ) = 0 (identity), which makes d a premetric , and for all a, b, c ∈ A , we have d ( b, c ) + d ( a, b ) ≥ d ( a, c ) (composition), which means that d satisfies the triangle inequality .

Recently, Mardare et al. [2016] took an equational approach to quantitative algebraic reasoning , which was later also referred to as quantitative equational logic [Mardare et al., 2021], by introducing approximate equality predicates = ε indexed by rational numbers ε (i.e., a = ε b if a and b are at most ε apart), and suggested that this approach essentially involves working with enriched Lawvere theory . Dagnino and Pasquali [2022] provided a logical ground to quantitative reasoning in the categorical language of Lawvere's doctrines by viewing distances as equality predicates in linear logic . Bacci et al. [2023] further studied the natural deduction systems of propositional logics for the Lawvere quantale and introduced what was later called affine Lawvere logic , including Łukasiewicz logic and Ben Yaacov's continuous propositional logic. Bacci et al. [2024] extended affine Lawvere logic to polynomial Lawvere logic by allowing multiplication as an extra logical connective. These studies are on propositional logic and do not involve predicates and quantifiers. Recently, Capucci [2024] studied a spectrum of quantifiers in [0 , ∞ ] -valued quantitative predicate logic.

In the context of machine learning, these relatively recently developed logics have yet to prove their practical importance. While these innovative approaches often hold theoretical promise, they need to demonstrate tangible benefits in real-world applications. Key areas where these new logics might eventually make an impact include neuro-symbolic reasoning and logic/probabilistic programming [d'Avila Garcez et al., 2002, Manhaeve et al., 2018, Sen et al., 2022, Badreddine et al., 2022, Fagin et al., 2024], which hold promise for integrating low-level perception with high-level reasoning, improving model interpretability, enhancing training efficiency, and enabling more robust decisionmaking processes. However, widespread adoption and validation through practical use cases are necessary to establish their true value and effectiveness in the machine learning landscape.

Under this background, let us contextualize our proposed methodology for deriving [0 , ∞ ] -valued quantitative metrics from logical definitions. We highlight three characteristics of our framework:

We allowed not only metrics, but strict premetrics , such as the relative entropy (Kullback-Leibler divergence) [Kullback and Leibler, 1951, Perrone, 2023] widely used in machine learning, as the real-valued counterparts for the equality predicates;

We focused on whether the metrics are zero or not and the (differentiable) optimization of the derived metrics, because our main goal is to guarantee that the minimizers of the derived metrics satisfy the predicate;

We included a wide range of real-valued quantifiers (or aggregators in our terms) beyond sup and inf , such as mean and mean square, as long as they are homomorphic to the two-valued quantifiers, because the derived metrics may have nicer properties or even analytical solutions.

(a) homomorphic

(b) adjoint to max

(c) less coverage

(d) more coverage

(e) negation, disjunction

(f) min, hinge

(g) min, reciprocal

(h) fraction

(a) bi-implication

(b) CNF

(c) DNF

(d) fraction

D.3 Implication and equivalence

In Section 3, we only used the truncated subtraction ´ as a quantitative operation for the implication → (Table 1). In Theorem 1, we noted that the implication is special because if a predicate involves the implication, then not all elements satisfying the predicate minimize the corresponding quantities (Fig. 2). In Appendix B.8, we discussed other possible quantitative operations ⊸ corresponding to the implication → .

In Fig. 8, we showed eight alternative quantitative operations for the implication. In the first row, the first one is homomorphic to the implication, which means that its minimizers are exactly those that satisfy the predicate (Example 16); the second one is right adjoint to the max (Example 17); and the other two are variants of the truncated subtraction (Example 18), which have more or less coverage. In the second row, we used the logical equivalence between a → b and ¬ a ∨ b to define quantitative operations for the implication using quantitative operations for the negation and disjunction. For example, we can use the hinge function 1 ´ n and min , which lead to min { 1 ´ a, b } , or the reciprocal function 1 and ab , which lead to 1 a b 1 = b .

Similarly, we can use logically equivalent expressions of the logical equivalence a ↔ b , such as ( a → b ) ∧ ( b → a ) (bi-implication), ( ¬ a ∨ b ) ∧ ( ¬ b ∨ a ) (conjunctive normal form (CNF)), and ( a ∧ b ) ∨ ( ¬ a ∧ ¬ b ) (disjunctive normal form (DNF)), to derive quantitative operations for the equivalence, shown in Fig. 9.

center

median

mean

(a) central point

diameter

(b) pairwise distance

Note that a quantitative operation homomorphic to the implication cannot be continuous everywhere, which is undesirable for gradient-based optimization. For example, the following quantity also measures the injectivity of a function m : Y → Z :

̸

This quantity aggregates distances between pairs of different inputs mapped to the same outputs. However, unlike q injective introduced in Section 4.3, it is not differentiable with respect to the function m : Y → Z . Thus, we cannot use it to improve the injectivity of a function by gradient descent.

D.4 Constant function

Note that there are two logically equivalent definitions of a constant function (to a non-empty set). One is based on the equality between all pairs, as in Definition 11. The other is based on the constant output value (see Fig. 10):

Definition 35 (Constant function with value) . A function f : A → B is a constant function with value b ∈ B if

which can be measured by

This quantity q const-v finds a central point in the codomain that best approximates all the outputs of a function, which is similar to the approach we discussed in Section 4.1. In fact, we can prove that if we use q const-v in q const-curry , we will end up with the same quantity q product .

(a) small radius, large variance

(b) large radius, small variance

7

.

62

MAD

3

.

97

2

.

82

Var.

1

.

93

2

.

78

5

.

56

3

.

63

MPD

2

.

69

MAD

7

.

12

Var.

49

.

02

MAD

0

.

73

6

.

55

MAD

33

.

54

MPD

49

.

81

Var.

0

.

51

6

.

89

Var.

15

.

82

MPD

6

.

12

5

.

50

MPD

2

.

44

9

.

66

MAD

1

.

81

4

.

68

Var.

0

.

85

2

.

28

4

.

56

7

.

75

MPD

1

.

42

MAD

8

.

51

Var.

46

.

15

MAD

1

.

73

4

.

86

MAD

39

.

66

MPD

47

.

86

Var.

1

.

86

5

.

63

Var.

16

.

83

MPD

5

.

11

7

.

19

MPD

0

.

94

D.5 Rank of imperfect representations

It is worth noting that Theorem 1 only guarantees that the minimizers of different quantitative metrics derived from the same logical definition are the same, but imperfect representations, whose evaluation results are non-zero, may be ranked differently by different metrics.

For example, Fig. 11 illustrates two constancy metrics, the radius of the smallest bounding sphere and the variance, on two sets of points in R 2 , where one set has a small radius but a large variance, while the other has a large radius but a small variance. More examples are presented in Fig. 12, and such results can also be observed in Table 2. This difference can lead to differences in risk preferences, sensitivity to outliers, and learning dynamics when these metrics are used as learning objectives. Further investigation of the characteristics of these metrics for imperfect representations is left for future work.

D.6 Implementation

Thanks to advanced indexing (e.g., NumPy [Harris et al., 2020] and PyTorch [Paszke et al., 2019]) and analytical solutions to some optimization problems (e.g., Eq. (19)), some of the proposed metrics can be easily implemented, even as Python one-liners.

For example, the following function implements a family of modularity metrics:

Here, y and z are NumPy arrays of shape (factor, index) ; aggregate can be max , mean , or sum ; deviation can be a function calculating the radius of the smallest bounding sphere, 12 mean absolute deviation around the geometric median, 13 variance, diameter, or mean pairwise distance. Please note, however, that the deviation function can be computationally expensive, depending on the dimension of the codes.

D.7 Limitations

Lastly, we discuss several aspects that are not covered in this work and potential directions for future research.

Function equality A collection of input-out pairs { ( x i , y i ) } n i =1 ∈ ( X × Y ) n may not define a function g : X → Y for two reasons: First, the set of all inputs X 0 := { x i } n i =0 is unlikely to enumerate all possible inputs (i.e., X 0 ⊊ X ), especially when the cardinality of the domain X is infinite (e.g., R ), so the data may only define a partial function g : X ⇀ Y or a function from a smaller domain g 0 : X 0 → Y . Second, the inputs may not be distinct, e.g., when an input is given multiple labels by different annotators, so the data may define a multi-valued function . The extension from functions to relations or stochastic maps is an important future direction of our work.

Partial combinations A more general issue is learning and evaluating disentangled representations given only a subset of all combinations of factors, which is common when dealing with a large number of factors [Träuble et al., 2021, Montero et al., 2021, 2022, Roth et al., 2023]. It is crucial to evaluate and justify whether a metric computed on partial combinations of factors is a reliable proxy for the performance of the model on unseen combinations.

Unknown projections Another common scenario is when the extracted representation is not properly aligned with the underlying factors. For example, a model may extract a three-dimensional representation z ∈ R 3 for two factors y ∈ [0 , 1] 2 , and it can project to (( z 1 , z 2 ) , z 3 ) or ( z 1 , ( z 2 , z 3 )) . How can we determine which is better, without enumerating all possible projections? Finding the optimal assignment [Mahon et al., 2023] and correcting a pre-trained model post hoc [Träuble et al., 2021] based on the proposed metrics are interesting future directions.

D.8 Broader impact

This paper focuses on the theoretical aspects of disentangled representation learning, and we do not foresee any immediate negative societal consequences. However, we would acknowledge that disentanglement is closely related to data-efficiency and fairness, potentially sparking discussions on ethical considerations. Besides, the application of category theory may facilitate the transfer and integration of knowledge across disciplines, fostering closer connections between various fields of study, even beyond the machine learning community.

(a) true factors Y

deviation

approximation

contraction

(b) entangled codes Z

(c) product approximation of an encoder m : Y → Z (d) linear approximation of its retraction h : Z → Y

6

4

2

0

2

4

6

(a) original distribution

0

2

4

6

0.0

0.2

0.4

0.6

0.8

1.0

(b) integral transform

uniform distribution

0.0

0.2

0.4

0.6

0.8

1.0

3

2

1

0

1

2

3

(c) inverse transform

standard normal distribution

3

2

1

0

1

2

3

3

2

1

0

1

2

3

(d) orthogonal transformation

standard normal distribution

3

2

1

0

1

2

3

0.0

0.2

0.4

0.6

0.8

1.0

(e) integral transform

uniform distribution

0.0

0.2

0.4

0.6

0.8

1.0

6

4

2

0

2

4

6

(f) inverse transform

transformed distribution

0

2

4

6

E Experiments

In this section, we provide the detailed data configuration used in Section 5 and further experimental results.

E.1 Synthetic data

We used a simple synthetic setup to simulate entanglement of factors and common failures patterns. Concretely, we used a Cartesian product Y := { 0 , 0 . 1 , . . . , 1 } 3 of three sets as the underlying factors (Fig. 13a). We used a random rotation matrix R to entangle factors and componentwise exponential as a non-linear transformation. We composited this procedure twice and used an affine transformation to normalize the outputs (Fig. 13b). That is, we used the following function as the data generating process:

Note that this function is injective but not a product or linear.

We used the following functions as the function m : Y → Z :

The rotation operation entangles factors but can be (linearly) inverted. The duplicate encoder has a modular decoder (projections), but itself is not modular. The redundancy encoder is both modular and informative, but not all codes can be decoded. The constant encoder is perfectly modular but not informative.

E.2 Weakly supervised modularity metrics

We briefly comment on the possibility of employing weak supervision for measuring disentangled representations.

For supervised disentanglement metrics we discussed in Section 4, the necessary data consists of observation-factor pairs ( x, y ) , representing a generator g : Y → X . In order to evaluate an encoder f : X → Z , we compose it with a generator and study the properties of the composition m : Y → Z := f ◦ g .

It is worth noting that p product and p injective are equational predicates, which means that they are invariant to bijections . Similarly, q product and q injective are invariant to isometries . This implies that the exact values of the factors are not important; we only need to know if two factors are equal or not. Hence, we only need weak supervision of the form ( x, x ′ , y i = Y i y ′ i ) so that we can construct some equivalence classes of factors, and we can still calculate or approximate Eqs. (19) and (27). Note that this type of supervision has been partially investigated by Ridgeway and Mozer [2018], Shu et al. [2020].

For example, suppose we have an object A (e.g., red circle) and an object B (e.g., red triangle), and all we know is that objects A and B have the same color. Based on such weak information, we can still construct an equivalence class containing objects with the same color as object A. Then, we can regularize an encoder f : X → Z by minimizing the variance of the color representations over this equivalence class (Eq. (19)). In this way, the modularity of the encoder can be improved. The challenge arises when there is noise or only partial combinations, which is an interesting future work direction. In such cases, we may need to use semi-supervised clustering to group the data [Wagstaff et al., 2001, Basu et al., 2002, Bilenko et al., 2004].

To validate this idea, we conducted experiments where we only used a random sample of pairs and their similarities. Table 3 is an excerpt of Table 2, showing only the proposed modularity metrics, and Table 4 shows these metrics calculated using only similarity supervision. We reported the mean values of 10 random samples of pairs, and the variances are negligible. Comparing Tables 3 and 4, we can observe that weakly supervised metrics may overestimate imperfect representations, but they can still maintain the ranks. This observation suggests the potential utility of employing weak supervision for both learning and evaluating disentangled representations using the proposed metrics.

a [Higgins et al., 2017] b [Kim and Mnih, 2018] c [Chen et al., 2018] d [Eastwood and Williams, 2018]

E.3 Evaluation of existing models on image datasets

We also report the results of several widely used unsupervised disentangled representation learning methods (VAE [Kingma and Welling, 2014], β -VAE [Higgins et al., 2017], FactorVAE [Kim and Mnih, 2018], and β -TCVAE [Chen et al., 2018]) evaluated on four image datasets ( 3D Cars [Reed et al., 2015], dSprites [Matthey et al., 2017], 3D Shapes [Burgess and Kim, 2018], and MPI3D [Gondal et al., 2019]) in Table 5.

We used a public PyTorch implementation [Paszke et al., 2019] of these methods and used the same encoder/decoder architecture with the default hyperparameters described in Locatello et al. [2019b] for all methods for a fair comparison. We used linear projection to find the most informative representations for each factor. The experiments were conducted on a NVIDIA Tesla V100 GPU.

Before analyzing these results, it is important to note that the evaluation of these learning models is not meant to be a proof of the correctness of the proposed metrics, since we cannot tell whether a bad result is due to the insufficiency of a learning method, to the quality of the datasets, or to the problem of the evaluation, if we have no theoretical guarantee for the metrics. We can trust the results of the proposed metrics because the properties of their minimizers are guaranteed by Theorem 1.

From Table 5 we can observe that the considered learning methods do not exhibit significant difference in terms of modularity and informativeness. This result supports the theoretical finding of Locatello et al. [2019b] that unsupervised learning of disentangled representations by matching the distributions of observations is fundamentally impossible (see also Fig. 14) as well as their empirical finding that there is no evidence that learning disentangled representations in an unsupervised manner is reliable.

E.4 Kendall tau distance between metrics

To analyze the relationship between these metrics, we report the Kendall tau distance [Kendall, 1938, Virtanen et al., 2020] averaged over experimental settings in Table 6. The Kendall tau distance is a correlation measure for ordinal data valued in [ -1 , 1] which counts the number of pairwise disagreements between two ranking lists. Values close to 1 indicate strong agreement, and values close to -1 indicate strong disagreement.

From Table 6 we can observe that even though different metrics derived from the same logical definition may rank imperfect representations differently (see also Fig. 12), they still have positive correlations with each other, indicating that they measure the same property. The metrics proposed by Higgins et al. [2017] and Kim and Mnih [2018] have the highest correlations with each other (except for themselves), and we hypothesize that this is because they are both based on the pairwise distance

a [Higgins et al., 2017] b [Kim and Mnih, 2018] c [Chen et al., 2018] d [Eastwood and Williams, 2018]

a [Higgins et al., 2017] b [Kim and Mnih, 2018] c [Chen et al., 2018] d [Eastwood and Williams, 2018]

approach. The DCI disentanglement metric [Eastwood and Williams, 2018] weakly agrees with the modularity metrics. However, the DCI informativeness metric [Eastwood and Williams, 2018] weakly disagrees with the informativeness metrics. It is possible that this is because of the different regressors ( sklearn.ensemble.GradientBoostingClassifier , sklearn.linear_model.LinearRegression , and sklearn .linear_model.QuantileRegressor [Pedregosa et al., 2011]) used in predicting factors from codes, showing that random seeds and hyperparameters of the metrics may matter more than the models when additional predictors need to be trained to evaluate the learning methods. This result indicates the advantage of q injective over q retractable .

However, it is important to note the limitations of these experimental results. Since the representations were learned from data and not fully controlled, it is possible that such results are due to the choices of datasets, learning algorithms, hyperparameters, and optimization errors. A high rank correlation coefficient between two metrics in this specific setting cannot guarantee that these metrics always measure the same property, or that they rank imperfect representations similarly in other settings. To gain a deeper understanding of these metrics, it is preferable to analyze their minimizers theoretically (Theorem 1) or test them in a fully controlled environment (Section 5).

E.5 Computation time of metrics

Finally, we report the computation time of the considered metrics in Table 7 to support our claim that the proposed metrics are much faster than those that require training additional predictors and hyperparameter tuning. We can see that, in an extreme case, the calculation of the DCI metrics [Eastwood and Williams, 2018] using GradientBoostingClassifier [Pedregosa et al., 2011] takes around 15 minutes, while other metrics can be calculated within seconds. This computation time may be acceptable if the metrics are only used in the evaluation phase, but it is not feasible to use them as learning objectives even in derivative-free optimization.

E.6 Factor-wise modularity metrics

An advantage of the proposed modularity metrics is that we can even evaluate each factor separately, which is impossible for those metrics that entangle modularity and informativeness. The results were reported in Tables 8 to 11. We found that some learning methods may outperform others on one factor but underperform on others, and different modularity metrics may rank learning methods differently (see also Appendix D.5).

For example, on MPI3D [Gondal et al., 2019] (Table 11), β -VAE [Higgins et al., 2017] has the highest scores (radius, MAD, variance, diameter, and MPD) on the horizontal and vertical axis factors, but the lowest scores (radius and diameter) on the object size, camera height, and background color factors. However, as measured by the MAD and MPD, it still has the highest scores on these factors. This means that β -VAE may generally encode the object size, camera height, and background color well compared to other considered methods, but has a small number of outliers. We believe that such fine-grained evaluation can guide the design of learning objectives, data collection, and further refinement of trained representation learning models.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: ' A theoretical connection between logical definitions of disentanglement and quantitative metrics ' was detailed in Appendices A and B. ' A systematic approach for converting a first-order predicate into a real-valued quantity ' was introduced in Section 3. ' The metrics induced by logical definitions, ' which were introduced in Section 4, ' have strong theoretical guarantees, ' which was supported by Theorem 1. ' The effectiveness of the proposed metrics ' was empirically validated in Section 5 ' by isolating different aspects of disentangled representations ' and further investigated in Appendix E.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We discussed the limitations in Appendix D.7. Another limitation is that we only studied the properties of the minimizers of the metrics (Theorem 1). Research on the behavior of these metrics on imperfect representations is limited (Appendix D.5). We claimed that some proposed metrics can serve as differentiable learning objectives, but empirical evaluation is out of the scope of this work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All definitions of the proposed terms, theoretical results, and detailed proofs were provided in Appendices A to C. The main theoretical result was stated in Theorem 1 and instantiated in Section 4.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The contribution is primarily the new theoretical framework. For experiments, we provided code snippets in Appendix D.6. We provided the detailed data configuration used in Section 5 in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provided the code to reproduce the modularity metrics in Table 2. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results of the proposed metrics reported in Table 2 are deterministic and do not contain randomness. We reported the mean values of 10 random trials in Table 4, and the variances are negligible.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ? Answer: [Yes]

Justification: We reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper mainly focused on the theory of disentangled representation learning.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates relating logical definitions of disentanglement and existing quantitative metrics via formal derivations of novel metrics.\n\nSTRENGTHS:\nThe topic is very interesting and it seems the authors were very rigorous in their investigations especially concerning the amount of all of the background material and derivations.\n\nWEAKNESSES:\nHowever, the paper is very poorly motivated, structured and written. It is very difficult to follow the authors along what they are trying to achieve (the motivation), what they are doing (their ideas) and the details on how this is connected with other research (context via related work) and how this should be used for future ML research/ what the significance is of their work for the future. E.g. the Introduction dives right into formal background notations without any motivation/overview of what the authors wish to achieve/investigate in the work. \n\nMaybe I have misunderstood some things, but what exactly are the results of the section 3? I.e which of the many metrics are the relevant ones and what do they mean intuitively? The authors provide many  examples of specific “implementations” (for lack of a better word here) and then conclude by statements such as “Upon analyzing the metrics above, it becomes evident that what we need is not the best approximation  itself (e.g., the mean) but rather the approximation error “ on page 7. So it seems the previous derivations are irrelevant for the overall goal (what this is is further unclear). So I wonder whether we really require this material then and can remove it, e.g., for the sake of more details on experiments (see below). \n\nOverall, I noticed the supplementary materials has much more relevant information that I believe needs to be in the main text. E.g. related works as these are necessary to understand the context of the work. \n\nFurther, it was also very difficult for me to understand the experiments of the main paper. The experimental setup is not described in the main paper (there is not even a caption to Table 2). This needs to be greatly improved. There seem to be a lot of information in the supplementary concerning experiments and I would suggest moving this into the main paper. Overall, it would help me if the authors could summarize again what they were investigating with these experiments.\n\nThe claims made in the experiments of the main paper are not backed up by the experiment shown in Table 2, e.g. \"The metrics derived from equivalent definitions may differ in terms of computation cost and differentiability.\". Where are the results for this? If these can only be found in the supplementary currently, I think they should also be moved to the main text.\n\nI am also puzzled by all of the experiments in the supplementary material (on interesting datasets), but that are not mentioned in the main text (unless I have missed this). Was there a reason for this? \n\nLastly, I am missing a thorough discussion of the overall findings and particularly potential limitations of the findings/proposed metrics.\n\nOverall, it is too difficult to assess the quality of the work and proposed ideas with the current structure. I would recommend putting some time into rewriting and restructuring to make this more clear to the reader.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-30"}

comments: SUMMARY:
The paper consider the connection between logical definitions and quantitative metrics, proposing a systematic approach to design metrics from logical definitions. Particularly, the paper is focused on the measure of disentanglement. The paper theoretically justifies the correspondence between logical definitions of disentanglement and quantitative metrics via topos theory and enriched category theory and empirically demonstrates the effectiveness in isolating various aspects of disentangled representations compared to existing metrics.

STRENGTHS:
The idea presented in the paper, especially the systematic method of converting logical predicates into quantitative metrics, is novel and very interesting. The authors also theoretically support this idea with topos theory and enriched category theory, which are sophisticated and advanced mathematical frameworks not commonly employed in machine learning literature.

WEAKNESSES:
The superiority of the proposed metrics over the the existing ones does not seem fully validated, either empircally or theoretically. The evidence provided does not show the proposed metrics dominate the existing ones universally.

QUESTIONS:
Could you provide more evidence, either empirical or theoretical, to show the proposed metrics dominate the existing ones universally? Or if the proposed metrics do not outperform the existing ones universally, when are they better?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

comments: SUMMARY:
The paper proposed to establish a connection between logical definitions of disentanglement and quantitative metrics from the perspective of typos theory and category theory. It then propose a metrics for disentanglement with stronger theoretical guarantees and compared it with some state of the art metrics.

STRENGTHS:
The paper provided theoretical justification for establishing metrics from typos theory and category theory in disentanglement of representation learning. It then proposed to convert first-order predicate into real-value quantity, which is innovative. The proposed metrics also have practical implications as the author mentioned differentiability and its effectiveness through some experimental results.

WEAKNESSES:
1, Despite that the authors included as much background information in the appendix, the advanced mathematical concepts makes the paper very hard to follow and limit its accessibility. Moreover, the theoretical connections may not be readily feasible to practical and intuitive interpretations. Hence its use scenarios should be stated. 
2, Even though the paper provides empirical results, its scope and effectiveness may not be fully demonstrated through the limited range of scenarios considered. Therefore its generalizability it not clear.

QUESTIONS:
1, why were the specific modularity metrics used for evaluation? How do they contribute to assessing disentanglement in representation learning?
2, Line 308 mentioned that the results are transformed isomorphically using $e^{-x}$. What is the purpose of this procedure?

LIMITATIONS:
The authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

comments: SUMMARY:
This study introduces a systematic approach to quantify properties of representation learning models. By translating logical definitions into quantitative metrics, the paper evaluates two key properties: modularity and informativeness. Two sets of metrics are derived for each property, one based on approximation and the other on distance computation and aggregation. Theoretical analysis of these metrics is conducted, including examination of their minimizers. 

(Full disclosure: I have reviewed this paper before)

STRENGTHS:
1. I think this paper is interesting and innovative and mathematically sound.
2. The paper is well-written. The math objects are introduced with good intuitions.

WEAKNESSES:
1. It might be difficult for people with less prior knowledges to read. 
2. It’s unclear if the metrics is practical be to a loss to optimize in practice because the experiments do not train with proposed metrics.

QUESTIONS:
1. Heyting algebra is mentioned a couple of times in the paper without a proper definition, which would be good for general ML audience.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12