A Pattern Language for Machine Learning Tasks
Authors: Benjamin Rodatz
Ian Fan
Tuomas Laakkonen
Neil John Ortega
Thomas Hoffmann
Vincent Wang
Venue: Accepted by TMLR ()

Abstract: We formalise the essential data of objective functions as equality constraints on composites of learners. We call these constraints ``tasks'', and we investigate the idealised view that such tasks determine model behaviours. We develop a flowchart-like graphical mathematics for tasks that allows us to; offer a unified perspective of approaches in machine learning across domains; design and optimise desired behaviours model-agnostically; and import insights from theoretical computer science into practical machine learning.
As preliminary experimental validation of our theoretical framework, we exhibit and implement a novel ``manipulation'' task that minimally edits input data to have a desired attribute. Our model-agnostic approach achieves this end-to-end, and without the need for custom architectures, adversarial training, random sampling, or interventions on the data, hence enabling capable, small-scale, and training-stable models.

Published in Transactions on Machine Learning Research (12/2025)
A Pattern Language for Machine Learning Tasks
Benjamin Rodatz
Compositional Intelligence, Quantinuum, University of Oxford
Ian Fan
Jump Trading, London
Tuomas Laakkonen
MIT
Neil John Ortega Compositional Intelligence, Quantinuum
Thomas Hoffmann
Artidis AG
Vincent Maag-Maacianica HAlLab, Philosophy, Orford
Reviewed on OpenReview: https: // openreview. net/ forum? id= 10s POUHC
Abstract
We formalise the essential data objective functions as equality constraints on composites of learners. We call these constraints "tasks", and we investigate the idealised view that such tasks determine model behaviours. We develop a flowchart-like graphical mathematics for tasks that allows to; offer a unified perspective of approaches in machine learning across domains; design and optimise desired behaviours model-agnostticllly and import insights from theoretical computer science into practical machine learning. As preliminary experimental validation of our theoretical framework, we exhibit and implement a novel "manipulation" lask that minimally edits input data to have a desired attribute. Our model- agnostic approach achieves this end-to-end and without the need for custom architectures, adversarial training, random sampling, or interventions on the data, hence enabling capable, small-scale, and training-stable models.
1 Introduction
The primary instrument for shaping the behaviour machine learning (MT.) systems is the objective function. We model objectives through a pattern language which the essential data of a loss is re-expressed as an equational constraint between typed learners. Let O., Oa be the parameter spaces of model enc and dec. The reconstruction loss for an autoencoder can then be written (for 0g E Oc. 0s Qa) as
arg min (Ea-D [D(ecco, (enco, (z)), 0.,0g
where we take the expected value over data distribution D on feature space X' of statistical divergence D (such cross-entropy log- likelihood) between two expressions that, under ideal conditions, should be equal: the decoding of the encoding of some data and the original æ. In our language, we focus on the two expressions that want to be equal, and we call this equational constraint a task.
This work was done while all authors worked Quantinuum LTD.

---

Published in Transactions on Machine Learning Research (12/2025)
In practice, designing good objective function incorporates many technical choices, such as the architecture, the statistical divergence, and the choice of training data (Ciampiconi et al. 2023 Richardson 2022 Terven
let al. 2023). However, these choices are often made by heuristies, or rationalised lised post hoc. While such
choices are sometimes required to make training tractable, they are not always relevant to understanding the final behaviour of the trained model. Instead, we propose and investigate the idealised view that tasks determine model behaviour.
To explore the expressive power of tasks, abstract away implementation details such architecture and training by idealising models to be universal function approxinators that can, in principle, perfectly optimise objective functions: in other words, we consider learners to be typed function-variables. Thus each task can be viewed purely as an equational constraint, on the behaviour of the learners, exactly analogous to equational constraints on the possible values of variables in algebra (Definition 2.6
Unpacking this bird's eye perspective of deep learming for a mathematical audience, the view we propose is that deep learning nonlinear representation theory: instead of linear maps representing groups, smooth maps represent higher algebraic structuros such as coloured PROPs (Yau 2008) for typed function composition, and the role of gradient descent is to convert data and compute into instances of these functorial representations. Unpacking the same perspective as a value proposition for practitioners, we suggest that deep learning solutions may be constructed declaratively, by writing down task equations that capture what we want them to do in terms we understand, letting neural networks and gradient descent figure out how to satisfy those equations by themselves.
We represent these task equations using string diagrams, which are a natural and recurring syntactic choice in mathematical domains concerned with formal reasoning about the behaviour of processes composed sequentially and in parallel (Selinger) 2011}; for us, these diagrams expose how multiple objectives interact across composed learners, which is precisely the regime in which informal engineering choices become brittle. We develop a pollern language with these diagrams, grouping logether objectives behaviourally, and visually simplifying formal reasoning with their tigher-algebrer compositional structure as topological manipulation of string diagrams.
Our overarching aim is to provide theoretical foundations and an expressive language that bridges existing methods and enables rigorous reasoning, supported by practical demonstrations designed to illustrate rather than benchmark empirically. The theory we present is deliberately broad and architecturo-ggnostic, and the examples we provide are designed to provoke and inspire rather than exhaustively demonstrate. Large-scale empirical validations or competitive benchmarking against established architectures, though important for broader practical adoption, fall outside the immediate scope of this paper. For small-scale demonstrations, we have developed an online tool in which equations can be specified as string diagrams, and which can generate the equivalent code implementatior
1.1 Contributions
We make three primary contributions this work. Our first, theoretical contribution the formalisation of common but informal standard procedure in deep learning, which may be summarised as the following recipe: Characterise desired behavior via equational constraints (tasks) between learners Implement tasks by treating neural networks as universal approximators - Convert equations to loss function by a choice of hyperparameters, namely weighted sum of constituent losses and choice of divergences.
Recipe: from specification to objective
Specify desired behaviours diagrammatically before committing to architectures.
Compile tasks to objectives via weighted divergences on the codomains involved.
Refactor competing designs algebrsically (using /rriiattttio//eiieeeeett compare guarantees.
Train and sssess using the same task equations, yielding contract-alignee tests by construction.
Thiis is available at https://patlsng.wereee wwwcel. appl and the source code he found at
patlang-ede editor
2

---

Published in Transactions on Machine Learning Research (12/2025)
Our second, theoretical contribution is demonstration that within our framework, we may analyse, predict, and design behaviours of models. By diagrammatic algebraic reasoning, we can analysc the behaviour of complex models via synthetic relationships bet ween tasks we call specialisation (Definition 2.8) and refinement (Definition (.14], grounded by well-understood tasks we call patterns (Section 2.4]. A Altogether, we may use these techniques to understand and compare the behaviour of models before committing to potentially costly training. Example 2.12 and Propositions 2.17 3.6 and 3.8 illustrate the kinds of formal reasoning our language enables.
Our third, practical contribution and proof-of-concept is the implementation of novel task we call manipulation [Section 3), which formalises (Bancilhon & Spyratos 1981) the problem of viewing and editing a targeted attribute of data while "leaving other aspects the same". As examples in image domains, we change only the colour of a shape (Figure 3 change the value of a handwritten digit without affecting other stylistic properties (Figure 40; and change only whether a person smiling in an image (Figure Sp. Even in these toy domains, we observe range of benefits we expect scale: by following our recipe we obtain architecture-agnosiic (Table style-transference models without the need for randomness, adversarial training, modality- and architeecurr--seeciic interventions, with good interpolation properties (Figure
An applied example of specification to objective
1. Start with natural-language requirement, e.g., "recolour a shape without altering anything else". Identify the observable attributes (colour) and the parts preserve (all other image properties).
2. Introduce learners that witness the behaviour: a reader get for the attribute, a writer put that edits, plus auxiliory encoder-
s/ /ecoodes when couvenient.
3. Write the desired behaviour equations (CLASSIFY, PUTGET, GETPUT, UNDOABILITY; Task 3.1) so that each equality captures a promise we want the system to satisfy.
4. Compile each equation into a loss term by choosing divergences and weights, producing an implementable optimisation objoctive.
Train learners with standard optimisers and re-use the same equations as diagnosties: measure attribute flip sccuracy with CLASSIPY, and check non-target preservation with GETPUT/UNDOABIIS
1.2 Positioning within the literature
Decision-thecocticc ML. Classical risi-minimisation views such ERM and MLE treat objectives scalar summaries of performance (Vapnik) 1998), while oplimisation-centric Bayesian perspectives (Knoblauch et al.
2022 analyse posterior design; marketplace-style multi-model frameworks such as Machine Learning Markets (Storkey et al., 2011) provide yet another lens how interacting objectives equilibrate. Our task language complements these approaches by reasoning about how multiple such objectives compose and constrain one another across interacting learners.
Graphical models and probabilistic programming. Plate diagrams and graphical models capture conditional independences and factorisations (Koller & Friedman 2009). By contrast, our wires track typed computational flows with learnable morphismss making them better suited to multi-loss design questions rather than inference over fixed structures; we point to probabilistic programming when stochastic semantics are the focus.
Neurosymbolic and compositional modules. Neurosymbolic systems ( Avila Garcez & Lamb 2020) often pair neural back-ends with symbolic engines or constraints. Our calculus operates one abstraction layer above, equipping designers with equational templates for objectives that can then be instantiated with neural modules, rather than hard-coding symbolic reasoning.
Bidirectional transformations and lenses. programming and vicw-update rescarch (Abou-Salch et al. 2018 Nakano 2021 The diagrammatic set sotting lets treat these laws as ready-made task equations, preserving their provenance while compiling them directly into MI. optimisation objectives.
The manipulation case study imports laws from bidirectional
3

---

Published in Transactions on Machine Learning Research (12/2025)
Adjacent style-transfer and disentanglement methods. The manipulation case study is closely related to style-transfer pipelines such as CycleGAN (Zhu et al., 2017) and disentanglement frameworks like 3-VAE (Higgins et al, 2017), which pursue related goals but with different guarantees: the former emphasise distributional alignment, the latter global latent factorisation. Our manipulation task targets local, per-instance altribule control with explicit behavioural contracis; Section 3 clarifies these distinctions formally.
2
Tasks and patterns
2.1
Notation
String diagrams provide typed, topological syntax for wiring learners; sequential compasition corresponds to placing boxes left-to-right, and parallel composition stacks wires vertically, We use the generators in Figure throughout the paper.
RM -7- N
RM (+)
20 M
7
Function
Singleton space
Delete
Data from
Sequential composition
F: RMM N/
2 (4)
R (6)
X: (x) - aM
(a f): 24 4020
N
MM
MM
M 28
(x)
+
NM
RM
RM
aN
MM
-0
N
R2
Vector
Swap
Copy
Independent X,
Parallel composition
C...V 0: (M) NN
A:
MM)
(X xx: (+) ((M4N) ( w): ()::):((4)) g(4+0) ((
Figure 1: Wires carry typed data, black boxes are learners, white boxes are fixed functions. Vectors are depicted as triangular nodes with only output wires, reminiscent of bra-ket notation; (co)associative (co)monoids such as copy /oey/eelete and add/zero appear circular nodes. Encoders and decoders may be drawn trapezia, and distributions thick bars. This grammar sufficient for reading every diagrammatic derivation in the sequel.
In terms of symbolic notation, we use X, y for feature spaces and domains that serve as carriers for inputs and outputs of learners; X,Y Y for random variables on feature spaces used in expectations and probabilistic reasoning; for data distributions on X' that supply samples for training and evaluation; @ for parameter spaces that specify learnable families (e-g., pecoders/decodeee equations into optimisation objectives; and or, 8 for hyperparametric constants used in weighted sums of objectives. We use monospace to refer to patterns and their constituent learners, and SMALL CAPS to refer to the constituent task equations patterns for ease of reference when reasoning. Additionally, we use colours (like this or this) to distinguish different types of data or components in diagrams.
D for divergences losses that compile task
2.2 An introduction via worked examples
Rather than starting with abstract definitions, we will begin gently with series of worked examples to introduce the core ideas of our framework in action, and to make the reader comfortable with string- diagrammatic representation and reasoning. While the formal details (Appendix A) involve category thcory, the power of string diagrams lies in their intuitive visual nature: by reading the diagrams as flowcharts from left to right, practitioners can leverage these diagrams to reason about ML. tasks without needing to fully grasp the underlying mathematical formalism.
As running example, we present the reconstruction loss of autoencoders once again.
Example 2.1. For a hyperparametric choice divergence D, where X is an input datatype, and LAT the datatype of the latent space, the two components of the empirical risk minimisation of the autoencoder task consist of (1) applying the encoder enc (typed X + LAT) followed by the decoder dec (typed LAT + X) to inputs drawn from source Dx of data over the datatype X, which should be equal to (2) the original x..
4

---

Published in Transactions on Machine Learning Research (12/2025)
Recall that by the Universal Approximation Theorem, seek to treat learners finetion-variablee So abstract away the parameter spaces 0, and simply depict the learners as black-boxes.
X
LAT
enc,dec
X
Ex [D (dec (encpe (a))
X
enc
dec
x
(2)
(1)
dec (enc (x))
The diagrammatic notation is formally equivalent to the traditional symbolic notation with the addition of type-information about inputs and outputs. In summary, nodes depicted as various shapes are functions, and wires are datatypes which can be understood as carrying information.
We observe that the function-variable perspoctive immediately suggests two ways of specialising such tasks viewed equational objectives: by imposing additional lasks, or by specifying architectural choices.
Example 2.2 (Perceptual losses as additional objective). Adding perceptual loss L X R220 to autocncoder is common example of task specialisation by additional objectives. Adding additional objectives means that the learners should opLimise the original lasks as well as the new ones. In practice, multiple tasks may combined into a single objective function by means of weighted summation with positive hyperparametric coefficients 3, but we abstract away such choices in our visual presentation leave only the equational data. Depicting processes with no learnable parameters white boxes, we have:
LAT
X
enc
dec
X
LAT
ezc, dec dec
cnc
X
my
anc,dac
onc
dec
argming 9 (aEgza (D(dec (ancge (x))) a)]
argmingg. ,d (E (Deaog (ance, (a)), a)))
m
+ (D(L(docn, (oeep (x))) 0)])
Example 2.3 (Residuation as an architectural choice). A common example an architect specialisation is adding a residual learner, where learner N of type X X is transformed into Nres == (2))
depicted as:
Nres
X
In an exact analogy with symbolic variable substitution, specialisation in this manner diagrammatically rewrites a function-variable in place as another diagram with matching input-output lype constraints (which The full formal semantics for such substitutions are provided Appendix A.2 Intuitively, since a universal function approximator can be any function, it can be a particular function (or subset of functions) of the same input-outpus type by rewrite. Since such architectural choices generally correspond to imposing inductive biases, these rewrites usually one-way.
may itself contain a cunction-variabll
Although our framework is too abstract in its current formulation to remark directly properties during training, we may still reason about the relative expressivity of learners by specialisation alone, which we later expand to compare different tasks (Definition 2.14
Example 2.4 (Residuations do alter expressivity). In the presence of negation, a nonresiduated learner A can learn anything its residuated version can, and vice versa, the following cycle of specialisations
5

---

Published in Transactions on Machine Learning Research (12/2025)
demonstrates:
x
my
my
d A
A(x) A(x)
A(x) A(z) -
So, specialisations allow us to analyse and compare the behaviour of models in terms of their constituent tasks. More productively, specialisations also give foothold for explaining and constructing model variants. As demonstrative example, we may obtain variational autoencoder (VAE) (Kingma & Welling 2022) from a regular autocncoder by the two kinds of specialisation we have described.
Example 2.5 (VAE). Suppose we wish to construct a probabilistic autocncoder. As first step, we may achieve this by specialising the output type of the encoder to be a space (mean, variance) of parameters for Gaussians, and specialising the decoder to be internally structured as the sequential composite of sampling from a Gaussian of the input (mean, variance) followed by learner.
(Ja, ar)
LAT
dec
X
GDC
samp.
dec
X
By inspection, there are two salient cases which models trained according to the above task may degenerate into deterministic behaviour. The first is when the encoder exclusively chooses very small variance encodings, and the second is when the encoder chooses means y that are very far apart: both cases, the decoder receives point-like encodings.
Both cases are addressed by specialisation with additional normalisation task which requires the outputs of the encoder to be close to the parameters of the unit Gaussian (0, 1).
(g,,
enc
|
enC
(0.1)
(NORMALISE)
Surprisingly, this behaviourally declarative presentation VAEs is equivalent to the traditional probabilistic form when the statistical divergence is KT. RRocca, 2021
Altogether, these examples illustrate how tasks give a foothold for inspecting and designing desired behaviours, and how specialisations may be used to pro-empt unwanted behaviours. We further claborate on the use of this form of reasoning for informing task design in Appendix B in the context of oblaining guarantees for the manipulation task.
2.3 Tasks, formally
We assume the following contextual data, omitted there no confusion. Let X, denote datatypes; Σ a set of processos f, each of which has (possibly empty) learnable parameter datatypes py. An atomic task is process-theorelic equational constraint on learners specilying that. f should behave like g all inputs. The objective function of an atomic task & of type X + equipped with distribution Dx corresponds to a for some choice of statistical divergence D.
map P + I that sends π + Ee-Dxx (D (D spec);; A learner can optimise multiple atomic tasks simultaneonsly by optimising a combination a the atomic objective functions (commonly obtained by taking a weighted sum).
Definition 2.6 (Tasks). An atomic task is a tuple (f. Dx p), where f.g: g: X are composite processes of Σ, Dx distribution over X, and p S Py ① P, is a space of trainable parameters. We indicate the system f and specification as sys, and spec,, and similarly the domain X and codomain Y as dom(y)) and cod(p). A compound task t (or just task) is non-empty set of tasks. As have seen in previous examples, we

---

Published in Transactions on Machine Learning Research (12/2025)
notate task @ a collection of atomic tasks sys, spec,; where superscripts on the harpoons indicate which learnable parameters are governed by each atomic task.
Tasks become concrete objective functions by hyperparametric choice of divergences for atomic tasks, followed by a combination via a weighted sum with hyperparameter coefficients, or more generally compound function. As such, & particular objective function that instantiales a task is one where the choices for the measure of statistical divergence and compound function have been made, Beyond these hyperparameters, tasks and objective functions may be viewed as informationally oquivalent.
Definition 2.7 (Objective function). Let o = {(fi. 9s. Dz. Di. be a compound task with N atomic lasks. Let. E Σ be a learner of Φ An objective function for is a tuple (Φ;, D, where 4 = {(ft. P.. p.) para(() S p&} is the set of all tasks on which is optimised, D is set of statistical divergences D D(peer): cod((p) x cod((p) R20, and the compound function is function (IR (I220) - R20 that is differentiable, and lypically non-decreasing in each argument.
Definition 2.8 (Specialisation). A specialisalion of lask + is task obtained by imposing additional constraints on 4 We identify two primary forms, both maintaining strengthening original behavioural guarantecs, with architectural specialisation additionally enabling explicit engincering of inductive biases into models.
Objective Specialisation adding further tasks, represented formally as extending the task set: Sseec = 4 U {Ki)er, where each dbs is an additional atomic or compound task imposing further behavioural constraints on learners.
Architectural Specialisation: Let & contain an atomic task involving a learner f E with type X + Y. lask 4' is an architectural specialisation of if f is diagrammatically substituted with a composite process C[f]= : X ++ Y such that C[f] E E- and dom(C(f)) X, cod(CTff) = Y. Here, C(J) is a well-typed composite in the free symmetric monoidal category over E, constructed by replacing f with a structured process diagram using additional (possibly parameterised) components, while preserving type compatibility. The substitution is interpreted as a one way rewrite rule on diagrams: f ~+ C[f], valid in the universal approximator regime (Appendix A.2
Throughout the following sections, we present selection of tasks and relate them to their respective objective functions, organised by learning paradigm. The following examples are not exhaustive, but serve to cast. familiar ML regimes in diagrammatic form, helping readers build intuitions about the notation. We illustrate classification and regression Pattern 2.9 for supervised learning; autoencoding [Ptttern 2.10 and energy minimisation [Pattern 2.13 for unsupervised learning; GAN Pattern 2.11 and VAE Example 2.5 for generative modelling; and CycleGAN Example 2.12 for transfer learning.
2.4 Patterns are "nice" tasks
On the view that behaviour can be captured equationally, it would be desirable to have a bank of correspondences between tasks and their behaviours. We do not expect there to be general and systematic method to translate between natural-language behavioural specifications and tasks; if such a method existed, then all of deep learning would be reduced to hyperparameter search. There are often many different ways to approach problem ML, much like there is no single correct way to write software, or design a building. This suggests to us the view of pallerns: some lasks are well understood, usable modularly and in many contexts, and easily modifiable, and such tasks can be viewed design patterns borrowing term from software engineering (Beck & Cunningham, 1987 In this section, we suggest some examples of patterns that correspond to well-studied methods and paradigms in ML, and we show how to use patterns as an accessible basis to analyse models.
Pattern 2.9 (classification). Given a data-label pair (d, 1) drawn from Dx with labels I E L,, a classifier cls : D + is a function that solves the classification task, in which it seeks to reconstruct the label
2 And before that, architecture and urban design Alexander al. 1977

---

Published in Transactions on Machine Learning Research (12/2025)
from the data:
D
cla
x
This can be done by minimising the corresponding objective function E(L.)=Dx (ccls(d)) [)] for some measure of statistical divergence D on the label space, which may be continuous to encompass regression.
we have seen, given a data distribution Dx over X and some latent space LAT,, an autoencoder consists of an encoder enc X LAT and decoder dec : LAT + X which cooperate to reconstruct the identity the observed distribution:
Pattern 2.10 (autoencoding).
LAT
dec
X
enc
dec
X
This corresponds to minimising Ex (D(de((nn(())).))
Pattern 2.11 (GAN). Given a data distribution Dx over X and noise distribution W over N, a generative adversarial network (GAN) consists of a generalor gen N + X and a discriminalor dsc X - [0, 1]. The prosaic explanation that "the discriminator seeks to distinguish real data from fake data while the generator aims to fool the discriminator" translates dircctly into a. task description: where indicates "real" and indicates "fake":
dac
|.
dec
gen
dac
N
N
dsc
In other words, the discriminator dsc seeks to minimise some positive combination of the teriss Ex [Ddsc((r), 1)] and Ez-Dx (ooDDxDDdssee [D(dsc(gen(t)))O] while the generator gen seeks to minimise E Dx ((dsc(gen(()))11)
For the sake of compactness, on the right hand side, we compressed the two adversarial tasks into one figure where the discriminator and generator pull into opposite directions.
2.5 Analysing complex tasks
We can analyse the intended functions of models by viewing them as composites of simple patterns. We have already seen in Example 2.5 how a VAE is analysable by inspection as a specialised regular autoencoder. As a second example, a CycleGAN two GANs on different distributions, whose generators are mutually autocncoders by a cycle consistency loss (Zhu al. (017). This suggests that the generators encode the distributions into each other reversible manner, and indeed this kind of style transfer between distributions is what CycleGAN does in practice.
Example 2.12 (CycleGAN). Below, i,j are nonequal indices taking values in {0,1}, where Do and Dj are different distributions on the same space X: typically these are two classes of images.
dsc;
X,
X
dsc:
GAX conditions
gen
dac;
X
X
gen;
dac;
Xj
|
4
gen
gen
X
gen.
gen
X
Cyole consistency lass
We can also upgrade informal intuitions into formal derivations. For example, on the account of Ranzato et al. 2007 broad class of unsupervised learning techniques including PCA and k-means
8

---

Published in Transactions on Machine Learning Research (12/2025)
specialisations of the energy minimisation task, which may be considered autoencoding task from latent. "code" space subject to the representations minimising measure of "energy". We can in fact formalise the relationship between these forms of unsupervised learning and autoencoding, showing that under mild assumptions, they are the same the computational limit.
Pattern 2.13 (energy minimisation). energy minimisation consists of; three types of systems: D(ata), C(ode), and R two learnable processes: an encoder enc D - C' and a decoder dec C + D; two user-supplied energy functions: Eenc C R220 and Edsc D D + R and a user-supplied constant E I220. Provided a distribution of imputs y D, and a distribution on C the system seeks to minimise Eyy~y,zzz (7 (e 2) + Edsc (y,, d(=)))- Such a lask is called code-ertracling when 2 enc()), in which case, diagrammatically, this corresponds to the following task:
D
Edec
X TY
enc,dec
dec
y
Z
Eand
To formally relate code-extracting energy minimisation and autoencoding, we introduce a relationship between tasks called refinement, which states that perfectly solving & allows one to construct perfect solutions for V. When & and refine each other, the tasks are "the same the computational limit"; a perfect autoencoder is perfect code-extracting energy minimiser, and vice versa. These relationships for the behaviour and relative power of concrete implementations of tasks.
proxy
Definition 2.14 (Refinement and equivalence of tasks). Task & refines task V if, by treating the atomic lasks as equations, the processes of & may be composed to satisly the equations of V. & and V are equivalent if they refine one another, which we denote + = !
Now we aim to prove that autoencoders and energy-minimisers are identical in the computational limit: that a perfect deterministic autoencoder yields a perfect energy-minimiser and vice versa.
Lemma 2.15. For all well-typed f, 9, and for any positive linear combination a: I220 x R20 + R20;
x
a
X
X
9
Proof. For the forward refinement, is positive lincar combination, have for all E Dx that (((()) = (fx) + a2 g(x) = 0. If ʃ and y are constant-function 0, are done. Other- wise, positivity implies that which is the desired task. For the backwards refinement, if f(x) = = g(x) for all E Dx then a(f(z).g(z)) = a1 f(x) + a2 g(z)= 0.
f(x) = g(x) = O, and since a1 a2 E R20, then f(r) = 0 = g(x),
Lemma 2.16. For a real-valued pairwise mcasure D D(Y) x D(Y) + R20 on the space of distributions
over Y, the positivity axiom 32) = > Vi = }2 implies, for (almos p all f.g: X + Y and Dx:
x
H
D
= x H
x H =
X
Proof. For the forward refinement, we assume that (Lf((x).g(Dx)) 0. By positivity of D, f(Dx) g(Dx), which is the right hand task. For the backward refinement, if f(Dx) = (Dx), then, by positivity, D(f(Dx). =
"When the function space containing and g large, the edge case where f and g are nonconstant and is mcasure- theoretically negligible. Since we concerned with behaviour the computational limit, this an acceptable assumption for our purposes.
9

---

Published in Transactions on Machine Learning Research (12/2025)
Proposition 2.17. If enc and dec deterministic and Eeac and Edec positive (e.g. metrics statistical divergences), then energy minimisation autoencoding.
Proof. As and dec are functions, may copy them through their outputs to re-express energy minimisation as:
dec
X-Y
enc, dec
e110
dec
V
EDC
dec
CBC
Eaa
enC
By Lemma 2.15 this is equivalent to:
enc,dec
enc, dec
y doc
dec
Eene
Recall that Eane and Edac are positive by definition, hence minimisations as:
Lemma 2.16 allows us to express the two
dec,dec
enc, enc,dec
The left task is autoencoding, so we have that energy minimisation refines autoencoding. For the other direction, we observe that autoencoding refines the right task by postcomposing both sides with enc, and as the left and right tasks together are equivalent to energy minimisation, we have the claim.
2.6 Proof of concept: Tasks from specifications the stack
In this section and the following [Section 3/, we will provide two examples of utilising the proposed perspective to design new tasks. First, we will propose and experimentally verify a set of tasks that is restrain two interacting learners to perform the push and pop operations of a stack. While by itself the stack is not particularly interesting, this section is more used as first introduction working with tasks to create desired behaviour. In Section 3 we will introduce more involved tasks for manipulating attributes in data.
The well-known data structure stack consists of two interacting operations: push and pop.
Task 2.18 (stack). Given data distribution Dx over X and pretrained autoencoder (enc: X - LAT, dec: LAT + X), stack consists of emply slack 1: x + S, push operation psh S x X S and a pop operation pop: S + S x X. We recursively define the distribution:
D = 1 enc(Dx)
psh(d e D, Dx) enc(x)
meaning D is stack of arbitrary size paired with encoded element from Dx.
Then psh and pop have to obey the following rules:
pop
psh, pap
E
pop
(EMPty)
psh
pop
(PsuPOP)
Dj (pop (+x); : x 0))
(s,x)>~ [D2 (popg (pued (e, æ)), (e, x))]
For completeness, and illustrate the ergonomic necessity of our diagrammatic notation, we display the formulaically obtained hyperparameterised objective function of stack in standard notation below. The
10

---

Published in Transactions on Machine Learning Research (12/2025)
appeal of the diagrammatic notation will become even more obvious in Section 3
(Ux)) + (a (2(po (push (
argmin a.v.x
EMPPY
PsiPor
Where 6, vh, X are the parameters of push, pop and to be learnt; a, 8 are hyperparametric positive weighting coefficients for each task; D1, D2 hyperparametrie statistical divergences for X.
Given the purely demonstrational role of this experiment, will do proof-of-concepl dataset Fiiuure 3) inspired by Spriteworld (Watters et al. 2019) consisting of images depicts a single shape with varying propertics. We observe that the stack works exactly as expected (sce Figure 2 For additional implementational details, see Section C.1
simple analytic
Original
images
Push
Push
Push
Push
Pop
Pop
Pop
Pop
Output Images
In this example, we train stack (alongside autoencoder) to store the latent vectors of Spriteworld shapes. With an image latent size 16 and stack vector size 64, it is able to retain information to faithfully restore up to 4 shapes.
Figure 2:
While the basic stack task not particularly interesting, it does provide a fully differentiable data structure which may be useful in composite with other tasks.
3 The manipulation task
In this section, design and theoretically analyse manipulation, novel task which aims to view and edit property of data without explicit guidance. In [Section 4 will experimentally validate the task. There are many models that behave like mani manipulation, e.g. unsupervised sentiment translation [Li et al. 2018 Sudhakar et al. 2019) in the text domain and prompt-based photo editing (Hertz et al.) 2022 Kawar et al. 2023) in the image domain. While most of these approach this question on the architectural level to impose the desired behaviour, we will instead approach this problem via the objective function constraining the learning process. To constrain the behaviour of manipulation, we demonstrate another ability of our framework: importing insights from computer science more broadly via the process-theone manipulation, we reference the field of Bidirectional Transformations, which studies consistency between different overlapping representations of data (Abou-Salch et al. 2018). A special case is the view-update problem (Bancilhon & Spyratos) 1981) originally proposed for databases: how do we algebraically characterise reading-out and updating an attribute E A from some data d E D? There are a family of solutions called lenses which are parameterised by algebraic laws of varying strength (Nakano 2021), which we take inspiration from below:
perspective. For
11

---

Published in Transactions on Machine Learning Research (12/2025)
Task 3.1 (manipulation). Let D be distribution pairs (z, with x X and attribute labels A, and let Dx denote its marginal on X. A modeller also specifies larget distribution A over desired attributes. A manipulation consists of a pair of operations (get X + A, put X x A + X) that can be understood as reading and writing respectively. The two operations must satisfy the following tasks with respect to D and the modeller's choice of A:
get
X
put,get
A
X
X
(CLASSIFY)
put get
(PUtCET)
(d,a) (D1 (get (d), a)]
(d,a)) d A 2D2 (get (put (d, a²)), a!))
get.
put,
X
X
put
(GRTPUT)
(UNDOARILPTT)
get
put
E E X1 d [DP (put,, (d, get (d)), d)]
(ddaa)~x1p X ADD (put.gs (put (d, a'), get (d)), d)]
To provide an intuition for each of the tasks, assume that the data consists of images each containing single shape labelled with the colour of the shape. CLASSIFY allows us to use get read out the colour of a shape. PUTGET says that first editing the colour of shape (say, from red to blue) and then immediately reading out that colour returns the edited colour (blue). GETPUT says that reading out the colour of a shape (say, red) followed by editing the shape to have the same colour (i.e., an edit that leaves red unchanged) is the same as doing nothing, UNDOABILITY says that edits can be undone; using the first put to change the colour of a shape (say from red to blue), and then editing again with a second put to restore the original, read-out. colour of the shape (red) must restore the original image.
Once again, for completeness, we display the hyperparameterised objective function of manipulation in standard notation below:
CLASSRY
PurGer
aE a (D/ (get,(x)). + (x. XA (D (get beet,pput (put, (2), + DE (DF (put (a, get (x)),, " XA (DA (put (put (put, (a, a'), get, (x))))
argming.s
GerPer
UNDOASILITY
Where & parameterises get and v/ parameterises put; a, 1, 7,7,0 are positive weights for each task; D D are divergences over attributes; D D on the data space X; Dy the marginal of over data; and is chosen so that the desired attribute values lie in supp(AA). Section evaluates these contracts via. attributo-flip accuracy and non-target preservation checks.
3.1 Theoretical analysis
Beyond providing an intuitive language for objective functions, the proposed diagrammatic language can also be used to prove properties about particular choice of objective function. In this section, we will prove two facts about the manipulation task that let us make predictions about its most likely behaviour after training.
First, we argue that an idealised model satisfying the manipulation task which attribute information is fully disentangled from the rest of the data is inherently challenging rcalise in practico. This perspective differs from global disentanglement frameworks such as 3-VAE (Higgins et al., (017), which seek latent factorisations up to dataset-wide symmetries, whereas contracts only demand local, per-instance
12

---

Published in Transactions on Machine Learning Research (12/2025)
controllability. This is because, in complex data domains, isolating and inverting attribute-specific structure often entails recovering detailed, latent. factors of variation. Such a disentangled inversion effectively requires the system to act as a conditional generator that precisely modifies only the attribute of interest while leaving other features unchanged.
Despite this, we still expect the manipulator to perform well in realistic settings. Specifically, we show that the manipulation objective refines the structure of CycleGAN, meaning that under ideal training it should match or exceed the behavioural guarantees of standard CycleGAN system. Whether such performance is actually achieved in practice depends on empirical training success, which investigate in Section 4
3.1.1 Proof Bayesian inversion
In the ideal case, solve manipulation, model would completely separate the information about the attribute it is manipulating from all other information. If it managed to do this, it could manipulate exactly the attribute while leaving everything else is. For this to be possible, there must exist complement datatype C' such that our original data distribution D can be scparated into two independent distributions A and C. Assuming that this is possible, we will show that (a) being able to do such a separation would freely give a manipulator and (b) that this manipulator would refine Bayesian inversion, task that is known be difficult for complex data domains.
First, define:
Definition 3.2 (Balanced entropy of an attribute). Given a distribution of data D on space D, we say that distribution A over space A represents an entropy-balanced. attribute of D if there exists complement type C with distribution C such that have the equality in distributions D = C x A up an isomorphism of the underlying spaces D ~ (C' A).
If there exists such an isomorphism, then we know that there must exist two functions that observe the isomorphism. We have:
Definition 3.3 (Latent split autoencoder). Let the attribute A induce balanced attribute distribution D on data D, and cls D + A be perfect classifier for the attribute A. Then the functions enc : D - C x A and dec D + C x A that observe this isomorphism between the two underlying spaces satisfy the following three conditions:
enc,dec
D
60C
dec
D
(1)
C
enc,dec
C
dec
6DC
(2)
A
A
enc
D
(3)
A
The two functions are suggestively named ene and dec as the first condition indeed specifies that they have act as an autoencoder on D. We will show that given such an idealised autoencoder, we can freely, ie. without any additional training, satisfy manipulation:
Proposition 3.4 (Manipulation from latent split autoencoder). Let be distribution with balancod altribute A. Then latent split autoencoder refines manipulation.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Proof. We can define:
D
D
put
dec
get
BIC
A
We now have to show that this indeed satisfies manipulation.
For this, we show that the put and get observe the restrictions of manipulation: PUTGET:
AIG
Equation
(Delete naturality)
GETPUT:
dec
(copy naturality)
(copy-delete counitality)
(autoencoder)
UNDOABILITY:
deo
WYC
ABC
SBC
Equation
(GEtPUT)
Thus, solving the information separation problem would also give a solution to manipulation.
We will now show that this is difficult by proving that this manipulation freely provides a Bayesian inversion to classifier from to A.
We define:
Definition 3.5 (Bayesian Inversion in Markov Categories). The Bayesian inversion CCoo & Jacobs 2019
nLab authors 2024a of a stochastic map f : X + Y with respect to a distribution P on is a stochastic map f : X such that, in distribution:
X
fi
X
X
f
P
P
f
(in dist.)
Y
We show:
Proposition 3.6 (Manipulation as Bayesian inversion). If a deterministic discriminative classifier cls D + A induces a balanced attribute cls(D) over A with respect D, then the manipulator specified above refines Bayesian inversion clst A + D.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Proof. We show that the put composed with an independent copy of the data source D is the Bayesian inversion of cls.
diss.)
(By def.)
(By Definition 3.2
(Gn dist.)
By Definition 3.2
(Counit)
(Copy)
(autoencoder, by def.)
As such, we have shown that the idealised solution to the manipulation problem would also give a solution to Bayesian inversion, which is generally considered a challenging problem machine learning. Given this inherent difficulty, one might reasonably question why we expect this setup to succeed. Before providing empirical evidence, we will first develop theoretical intuition by relating manipulation to the established lask of CycleGAN.
3.1.2 Proof CycleGAN
CycleGANs solve similar task as manipulation, translating between two distributions. In fact, with the ionilional regularisation terms, strong manipulation a refinement of CycleGAN, giving us more guarantees by avoiding certain failure cases.
To prove this relationship formally, we have add additional rogularisation terms to the manipulation task. While these theoretically justified, in practice they less train stable. Therefore, in our experiments, will use the vanilla manipulation
starting point.
Task 3.7 (strong manipulation add-ons). The strong manipulation task consists of all the tasks con- tained in manipulation, well as:
put,get A
X
put
(FAKR)
A
A
put
(PurPur)
dac
put
X
A
put
dsc
4
A
X
dac
put
(Foot)
X
dsc
(TRUE)
A
The PUTPUT task (which, paired with PUTGET and GETPUT strictly stronger than UNDOABILITY in that it is algebraically implied) says that the effect of putting twice is the same discarding the effect of the first edit and only keeping the last edit. In conjunction with PUTGET and GETPUT, this creates what is known in the literature as a very well-behaved lens. The TRUE, FAKE, and FOOL. tasks introduce a discriminator component dsc, which forms a GAN pattern with respect to put as the generator. When well-trained, this forces the outputs of put to lie in-distribution. As in general there are no algebraic or equational laws that. characterise arbitrary distributions of data, using GANs in this way is a generic recipe for shaping outputs of generators to behave well in-distribution.
To show that strong manipulation refines CycleGAN, we require one additional assumption: generator in senerative-adverearri setting is optimal if and only if its output distribution is equal to the original
15

---

Published in Transactions on Machine Learning Research (12/2025)
distribution. This is natural assumption for such settings, and has been proven formally [Goodfellow et al, 2020, Theorem 1) by Goodfellow et al for an appropriate choice of measure of statistical divergence. We may now prove a weaker variant of refinement, relative to optimal (rather than perfect) tasks.
Proposition 3.8 (Strong Manipulation and CycleGAN). Given appropriate choices of statistical divergence measures, optimal solutions to the strong manipulation task can be used to construct optimal solutions to the CycleGAN task.
Proof. Let. Do, Dj be two distributions over the same type. We create a combined space D = {(x, i) x D E {0,1) where the label indicates which distribution the data point came from.
Assume have functions (get, put, dsc) that optimally satisfy the strong manipulation task for this new dataset. Then show that
construct optimal CycleGAN from these functions.
For E (0, j 1 i, can construct generators Go and G1 as follows:
G
put
First, we show that the cycle-consistency requirements of CycleGAN are satisfied:
X
G
jj
put
put
put
(Def)
(PutPut)
put
X
get
(Claseify)
(GetPut)
Next, we show that these generators are optimal the GAN sense. As outlined above, we say a generator is optimal and only if its output distribution matches the target distribution. Since put optimally trained with the discriminator dsc, (., i) produces outputs indistinguishable from D,.
By the PUTGET property:
get
put
get
X
(Def)
(PutGet)
Combined with the CLASSIFY property, this ensures that G; produces outputs that belong to the correct. distribution D,
Thus, the generators Go and G1 derived from strong manipulation optimal solutions to the CycleGAN task.
However, the converse does not hold: CycleGAN is not a refinement. of strong manipulation. For example, a CycleGAN could learn generators that apply a consistent reversible transformation (like horizontal flipping) while prescrving cycle-consistency, but such a solution would violate the PUtPuT property of strong manipulation. This demonstrates that strong manipulation provides stronger guarantees about the behavior of its learned functions than CycleGAN does.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Despite the additional guarantees, strong manipulation does not guarantee that it indeed only changes little as necessary to go from one distribution to the other. To showcase this, we can consider a degenerate example where no unique solution to manipulation can exist: imagine data with four objects circle, square, red and green. There is no unique mapping between shapes and colors. Without further information, it is therefore impossible to say what would mean to change as little as possible to go from shapes to colors. Yet, despite these theoretical concerns, in practice, manipulation often converges to the desired behavior, similar to CycleGAN, which has even fewer guarantees.
As comparison to a broader vein in the literature than CycleGAN, we consider Disentangled Rep- resentation Learning (DRL). Following Wang et al. (2024), two common formulations of DRL are via Bengio et al. (2013) (the "intuitive" definition), and Higgins et al. 2018 (the group-theoretic definition). The latent-split autoencoder construction we have presented is a flat, vector-wise DRL formalisation of the first definition, discntangling a single attribute. Proposition 3.4 constructs a put as m(x, = y semigroup-action on the disentangled attribute space, which conforms with Higgins et al's group-action definition as conservative generalisation. A converse refinement. (and hence, task equivalence) holds if are granted style- projeeproeettin map D Dj~, where D/- ~ is the input space D quotiented by the equivalence relation d ~ d' A Vaaa put((d,a) a) = put(d', a). As a sketch, this obtains D = D/~ XA by the correspondences enc(d) + (r(d).ge(dd)) and dec(s, a) + put(s.a).
While a comprehensive comparative study is beyond the scope of this work, we remark in passing that Manipulation appears to be different in aims from DRL, and this difference in intent manifests downstream as differences in use. In essence, DRL seeks global coordinate charts of data with low interdependence and invariances, whereas Manipulation aims merely at local, per-instance on-manifold control. Intuitively, the former is asking for more than the latter, since it is possible to navigate terrain locally without global map, and that is what Proposition 3.4 demonstrates formally. In practice, the unsupervised factor discovery of DRL aims to obtain block diogo-liagyyllly separate latents identifiable up to gauge choice of rotations, whereas Manipulation anchors the attribute coordinate and allows for detection and enforcement of cross-factor causal influence by edit-commutation, reminiscent of Scholkopf et al. (2021)'s 's causal DRL.
4 Experimental validation of manipulation
In this section, we provide simple but illustrative demonstrations of the manipulation task, designed primarily to validate thcoretical prodictions about task-driven behaviour controlled settings. While these examples do not represent state-of-the-art performance, they clearly illustrate how our framework systematically predicts model behaviour from task structure alone.
We will first provide some toy examples of the vanilla man manipulation task, setting in which we can also adapt to distributions without balanced entropy. Then we will showease the manipulation task with more realistic data, manipulating attributes of faces. For technical implementation details, see Appendix C
4.1 Experimental Results I: Simple attributes of synthetic and real-world data
We first demonstrate initial proofs-of-concept of the manipulation task on a simple analytic dataset F Figure 3 inspired by Spriteworld (Watters et al., 2219), and on MNIST (Fiuuee 4). In the former, each image depicts a single shape with varying properties, and labelled by two attributes: shape - circle, square or triangle and colour red, green or blue. For each attribute, we train a get/put pair according to the mamipulation task specification.
These initial demonstrations synthetic data illustrate the potential generalisation properties theoretically predicted by our formal task analysis, particularly the ability of models to interpolate smoothly between attribute states. We emphasise that our goal here is not performance optimisation, but rather to transparently illustrate theoretical predictions in a comprehensible setting.
We thank anonymous reviewer for bringing this field our attention for a constructive comparison.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Tringgee
Rectangle
Orginal
Red
Green
Blue
Figure 3: An imput Spriteworld image alongside a spectrum outputs exhibiting the ability of the put to manipulate a single attribute of the input while preserving its other properties. Additionally, the model able to generalise by interpolating to attribute values unseen during training, in this case producing orange and cyan shapes, whereas during training, it only sees red, green or blue shapes. (further details in Section C.2
Original
1
2
3
4
5
6
7
8
9
D
\
3
4
5
6
7
8
9
6
0
2
3
4
5
6
7
8
9
7
0
1
2
3
4
5
6
7
8
9
S
0
/
2
3
4
5
6
7
8
9
4
O
3
3
4 5 6
7
9
Figure Outputs of a put trained against an MNIST classifier. The put preserves several graphological aspects, such as stroke weight, slant, and angularity. This represents qualitative evidence support our prediction that put a class-conditioned generative model behaves as style-preserving edit.
18

---

Published in Transactions on Machine Learning Research (12/2025)
4.2 Experimental Results II: Derived attributes of synthetic data
The following demonstrations explore theoretically anticipated challenges in manipulating complex, derived attributes attributes-parriiulll parttuularrt entropy imbalances known to cause issues in more practical, large-scale contexts. We intentionally simplify and explicitly control our synthetic scenario (Spriteworld) to provide clear, instructive validation of our theoretical insights into the interplay between task constraints and model behaviours.
Often in practice we are interested in complex, non-explicit attributes that are derived from those labelled in the data: for example, "eligibility for a loan" may be derived from other explicit attributes of people in database by operationally opaque classifier, with unknown range, distribution, and dependencies other attributes. A known challenge in manipulating derived attributes is unequal entropy in attribute classes (Chu et al. 2017), which may cause models such as CycleGANs to hide data imperceptibly, making them particularly vulnerable to adversarial attacks. Various solutions have been proposed, including masks Wu et al., 2024), blurring (Fu et al.) 2019) and compression (Dziugaite et al., 2016). We demonstrate via a modification of manipulation (Task 3.1 Figure 6) that our framework permits the design and implementation of end-to-end approaches to editing complex attributes without interventions on the data.
min(1, + 0.6)
if shope circle
bc
m0.( max (0.2, cs))
shape square
max(0, cs 0.6)
if shape triangle
Figure 5: To illustrate the concepts of derived attributes and unequal entropy, consider an attribute on the Spriteworld data called blue-circleness, which broadly measures how similar a shape is to a blue circle. We define blue-circleness (bc) function of explicit attributes shape and colour; assign continuous colour score cs E 00, 1] based on the hue, where red = 0 and blue = 1. To illustrate unequal entropy in this example, the class 0 has higher entropy than 0.4 because there are more shapes that have be-value 0. So manipulating a shape with bo-value 0 to 0.4 must lose information.
Task 4.1 (complement manipulation). Inspired by the complement of symmetric lenses (Hofmann et al., 2011), we introduce a complement to put, changing its type S + S x Let (d, a) X be distribution over some data d D, each labelled with an attribute E A. complement manipulation C) fulfilling the following rules:
consists of pair (get: : D + A, put D AAC
X
get
get
get
put
put,get
X
x
C
H
get
X
X
X
X
get
get
put.get
put
A
put
put
A
|
C |
C
The idea of the complement is that it provides the manipulation with a scratchpad C to keep track of additional data. As none of the tasks check the output of the complement, the put and get can use freely to store relevant data.
While the complement manipulator successfully demonstrates our theoretical predictions small scales, we stress that this controlled scenario is illustrative of underlying theoretical mechanisms rather than directly
19

---

Published in Transactions on Machine Learning Research (12/2025)
Origial
can manipulate derived attributes such as blue-ciroleness, by using the complement as scratchpad to record a correspondence between data points (further details in Section C.2 while preserving attributes such as position and size.
Figure 6: A complement manipulator Task 4.1
indicative performance at scale. The value here lies primarily in transparently validating predictions about how task structure can address known issues (e.g., entropy imbalance) without requiring explicit architectural interventions.
4.3 Experimental results III: Interpretability applications on real-world data
In this section, further illustrate theoretical predictions of our task framework by applying manipulation lasks to somewhat more complex, yet still controlled, real-world dataset (CelebFaces Altributes). These experiments demonstrate how specific theoretical insights, such as the introduction of linearity constraints, can systematically yield interpretable and theoretically anticipated behaviours in practice, albeit at exploratory scales.
In the same way that we would expect latent space "filtered through" the probabilistic structure of Gaussians yield good sampling properties [Example 2.5). we would expect that enforcing linear structure on the latent space would yield linear properties. So specialised the put to be simple vector addition in the latent space of an autoencoder FFigure 7), for the relatively complex Targe-scale CelehPaces Alfributes dataset. We found that restricting put to be linear in this way not only increased training stability, but indeed exhibiled contimuous interpolation in generated outputs between normally discrete class labels (Figure 8), and class-sensitive separation of latent space embeddings in the autoencoder. We consider this to be compelling evidence that since our framework is agnostic, implementation details may be enginecred to obtain additional desirable properties without compromising behavioural specifications.
dec
put
gut
Figure 7: Recalling that architectural choices are a form of specialisation by diagrammatic substitution, the linear put is a specialisation of a generic put as an autoencoder lask-bound pair learners enc and dec, along with a put' that computes single shift vector be added into the latent space, depending only on the label value. enc, dec, and put' are trained simultancously along with the manipulation tasks, and intuitively this pressures the autoencoder pair to adapt their latent representations to fit. the needs of the broader manipulation task.
These illustrative results support our theoretical claim that structured tasks systematically induce predictable model behaviours: in this case, linearity enabling continuous latent interpolations. Although the complexity of the chosen dataset moves beyond simple synthetic data, we explicitly note these demonstrations remain exploratory validations intended primarily as proofi-ef-principle for theoretical predictions rather than definitive large-scale benchmarks.
Taken together, the illustrative examples presented across synthetic and small-scale real-world datasets serve primarily as concrete demonstrations of our framework's theoretical predictive power, rather than comprehensive empirical evaluations of practical performance scalability. Such demonstrations confirm the
20

---

Published in Transactions on Machine Learning Research (12/2025)
Original
Not emiling
Smiling
Autoencoder Only
Smillng Not smilling
Linear Manipulator
(a)
(b)
Figure 8: Left: Outputs of a linear manipulator trained face image data paired with a binary omile/no-smine label. The remarkable aspect of this experiment is that the original data only carried binary smile/no-smile no-smile labels, and that the linear structure in the specialisation of the put admits continuons interpolation. Right: A comparison of the spread of the latent embeddings of images from the validation set when pre-training an autoencoder and then training a (linear) classifier on the latent space (top), vs. when trained with a linear manipulator (bottom). We find that linear put automatically separates latent space embeddings of classes: the graphs depict the relative density of embeddings along the direction of the classifiers' weight vectors, normalised that each combined data spread is centred and has unit variance (details in Section C.3).
clarity and utility of our theoretical framework, and explicitly highlight opportunities for future exploration at larger scales and across broader practical domains.
5 Concluding discussion
5.1
Summary
We introduced diagrammatic language for representing and reasoning about the behaviour of machine learning models in terms of tasks, viewed as the essential data of objective functions. By leveraging category theory and string diagrams, our work establishes a cross-disciplinary formal bridge between theoretical computer science and practical machine learning, providing new conceptual tools for analyzing ML systems and permitting the transfer of insights between traditionally separate fields.
The proposed framework allows capturing existing tasks in machine learning, providing intuitive insights rooted in mathenatical rigour. We identify set of widespread and well-understood tasks, which we call patterns. We can analyse some tasks as composites of patterns (Example 2.12) while other tasks can be understood as specialisations of patterns [Example 2.5). The rewrite system inherent to string diagrams, allows us to identify relationships between different tasks and formalise intuitions Prropsiiion 2.17 Proposition 3.8).
Beyond theoretical insights, the proposed language also allows the creation of new training paradigms. As preliminary empirical validation of our theory's utility and potential, we introduced a novel task type called manipulation that produces class-conditioned and style-preserving generative model counterpart for given classifier. In the image domain, were able to verify predicted behaviours (Section 4.1), and we demonstrated the ability to design novel end-to-end capabilities, such as end-to-end editing of complex attributes (Section 4.2 and the imposition of linear structure on latent space representations (Section 4.3), which allowed continuous interpolation between discrete class labels on real data, and separated latent space embeddings of different classes. Notably, this was achieved without adversarial training conditions, random sampling, preprocessing of data, or hardcoded interventions the architectures, i.c.: our framework enables capable, small-scale, and braining-stable models.
21

---

Published in Transactions on Machine Learning Research (12/2025)
5.2 Relation to extant work
Regarding our nascent theoretical framework as a whole, the style of engineering beginning from tasks is already common practice in many fields of ML, and we sought here to place these practices on a more rigorous footing, and to probe their strengths and limitations. Our mathematical approach draws broadly from the field of Applied Category Theory (Fong & Spivak] 2019a), particularly the use of string diagrams for the higher-algeeraic data of concurrently and serially composed functions, which enables compact representation and reasoning with otherwise cumbersome symbolic equivalents when dealing with multiple learners in tandem. To our knowledge, our concern with the composition of tasks among many learners distinguishes our aims and formal choices from approaches that employ similar mathematical formulations, both within the category-theoretic literature (cf. Gavranović et al. 2004)), ), and without (ef. the variational generalisation of Bayesian inference presented Knoblauch et al. (2022)
While our approach is essentially neurosymbolic in spirit, it does not fit neatly into the mainstream triad of neurosymbolic approaches (d'Avila Garcez & Lambl 2020); do not encode symbolic data for neural operations, nor do we interface neural approaches with symbolic engines, nor are we hardcoding expert knowledge representations. Moreover, our aims differ: while neurosymbolic approaches often seek to manipulate symbolic data systematically by neural mcans, our framework operates at a higher level of abstraction, seeking to use the systematicity of higher-algebraic equational characterisations as a means to shape the neural ends. Hence our perspective may complement existing approaches structure in machine learning.
Regarding manipulation in particular, this was to our knowledge the first practically demonstrated synthesis of insights from Bidirectional Transformations as a subfield of database theory (Abou-Saleh et al. 2018) with ML. While explicitly neurosymbolic approaches have been tried for similar editing tasks before (sce e.g. Smet
let al. (2023)), ) owing to the influence of database theory in our approach, to our knowledge our statement. and execution of this task enjoys the maximal permissible generality and implementation agnosticism among similar attributo-cditing tasks, without sacrificing rigour and systematicity.
5.3
Limitations and Prospects
5.3.1 Scaling and Empirical Validation
Our experimental results are necessarily preliminary and limited scale, intended primarily as sanity checks and first demonstrations rather than definitive benchmarks. Recognising this, we note explicitly that further empirical exploration, while beyond our current scope, would be valuable and necessary to fully assess practical scalability and broader utility. Future work should rigorously test our theoretical predictions at larger scales and in comparison with established state-of-the-art frameworks.
Concerning manipulation in particular, an immediately evident limitation of this practical demonstration is a lack of exploration of how the difficulty of training such ensembles of learners behaves at scale, with respect to more complex and multimodal datasots, and with wider range of architectures. While none of the products of our experimentation are stale-of-the-arl with respect to specific applications, we believe the variety and promise of these results serve compelling validation of our theoretical framework's utility and potential.
Concerning applications of manipulation beyond the image domain, we report on some sketch experiments in sentiment- maniiulatinniiiiiiiinnnnnn on text in [ppenddx D where we also comment on the nature technical difficulties to be overcome the application of manipulation to complex domain data, and offer an explanation for mode collapses observed during training by empirically relating manipulation to other generative classification approaches.
22

---

Published in Transactions on Machine Learning Research (12/2025)
5.3.2 Complexity and Theoretical Formalism
We acknowledge that our reliance on categorical formalisms and abstract mathematical structures, while foundationally valuable, may impose an initial barrier to practitioners less familiar with these tools. However, we believe that such rigour is precisely what enables clear theoretical insights and structured reasoning about model behaviour. To aid accessibility, we have provided intuitive explanations accompanying formal constructs and a friendly diagrammalic synlax that can be used productively without detailed understanding of the formal semantics. We anticipate practitioners can benefit from engaging with these theoretical underpinnings incrementally.
Addressing the theoretical framework of lasks more broadly, our reliance on equational characterisa is double-edged. On one hand, is uncommon to find such characterisations of mathematical systems of interest. as they are usually defined by more direct means, and this presents theoretical limitation. On the other hand, it appears that the strength of equational characterisations when applied to ML lies in imposing structure "the way to learn to solve a problem" rather than on the solutions or problems themselves (Sutton 2019) This suggests promising future possibilities of our mathematical framework in bridging structural-symboot approaches from computer science more broadly with methods that can effectively leverage computation.
5.3.3 Theoretical Gaps
While we have demonstrated that, in certain cases, tasks can determine behaviours, there is thcoretical gap in the converse analysis of behaviours of trained models in terms of their basic tasks.
Our theoretical framework currently best handles static, equilibrium conditions of oasi-optimistiin, providing robust insights into structural behaviour under idealised limits. Dynamic adversarial interactions, such as those arising in GAN-based architectures, introduce complexity that our present theoretical treatment explicitly acknowledges but does not fully resolve. This represents a fruitful avenue for future theoretical expansion, underscoring the potential for this framework to evolve and integrate more sophisticated dynamical systems analysis.
Future theoretical developments will also seek to incorporate other aspects of ML. for example, relating to work that focuses on the choice of model architecture (Khatri et al. 2024 and interactions with the underlying data distribution (Bronstein el al. 2021). While our current experiments focus on demonstrating our framework's validity, future practical developments will explore applications to more complex, real-world ML challenges, where we envision our approach informing arcas such AutoML, interpretable AI, and formal verification of ML systems: the compositional nature of our task-based framework naturally aligns with neural architecture search by potentially informing principled search strategies for optimal model architectures, and the explicit representation of model behaviours as equational constraints could enhance interpretability and facilitate formal verification.
Acknowledgements
TL is supported by the US Department of Energy under Award No. DE-SC0020264. VW-M supported by the Cosmos Institute, under the Cosmos Senior Research Fellowship.
References
Faris Abbou-Salhh, James Cheney, Jeremy Gibbons, James McKinna, and Perdita Stevens. Introduction to Bidirectional Transformations, pp. 1 28. Springer International Publishing, Cham, 2018 ISBN 978-3-319- 79108-1. doi:
1. URL https:///oi. 100./91077777-1111 3- 319--19--9999 1119-79188-11
Christopher Alexandor, Sara Ishikawa, and Murray Silverstein. A Pattern Language: Towns, buildings, construction, volume 2 of Center for Enwironmental Structure Series. Oxford University Press, New York, 1977.
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic Variational Video Prediction, March 2018. URL https://doi. 48550/arx100 1710. 11252
23

---

Published in Transactions on Machine Learning Research (12/2025)
John C. Bacz and Jade Master. Open Petri Nets. Mathematical Structures in Computer Science, 30 (3):314-341) March 2020. ISSN 0960-1295, 1469-8072. doi: 10.1017/50000112552000000 URL http:
wwxzv.orgg/bb//800... 05415
Francois Bancilhon and Nicolas Spyratos. Update semantics of relational views. ACM Trans. Database Syst., 6(4):557 575, dec 1981. ISSN 0362-5915. doi: 10.1145/1190253116641 URL https://6oi.org//10 319628 319634
Kent Beck and Ward Cunningham. Using Pattern Languages for Object-Oriented Programs. Technical Report CR-87-43, September 1987.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnelt (eds.), Advances in Newrul Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL pttp:////..ooceengg....eo pdf
cc/paper files/ 2alles/paerr/215//11ee
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation Learning: A Review and New Perspectives. JEEE Transactions on Pallern Analysis and Machine Intelligence, 35(8):1798 1828, August. 2013. ISSN 3839. doi: 1A.1099TTAAM2221...
Guillaume Boisseau and Pawel Sobociński. String Diagrammatic Electrical Circuit Theory. Electronic Proceedings in Theoretical Computer Science, 372:178-191 191, November 2022. ISSN 2075-2180. doi: 10.4204/ EPPTCS337.113 URL http:://arxivorrgeeeee http:/bb//11000 07763
Filippo Bonchi, Pawel Sobociriski, and Fabio Zanasi. A Categorical Semantics of Signal Flow Graphs. In CONCUR 2014 Concurrency Theory 25th International Conference, Rome, Italy, September 2014. URL
https://hal science/hal-02114418
Filippo Bonchi, Pawel Sobocinski, and Fabio Zanasi. Interacting Hopf Algebras. Journal of Pure and Applied Algebra, 221 Jamuary 2017. ISSN 00224049. doi:
200..06.00. URL
Filippo Bonchi, Robin Piedeleu, Pawel Sobociíski, and Fabio Zanasi. Graphical Affine Algebra. In 2019 34th Annual ACM/IEEF Symposium on Logic in Computer Science (LICS), pp. 1 1-12, June 2019. doi:
URL https://dol.orr/1u 1109/LICS
Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G. Willcocks. Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Basee and Autoregressive Models. IEEE Transactions on Pallern Analysis and Machine Intelligence, 44(11) 3:7377 November 2022. ISSN 0162-8828, 2160-9292, 1939-3539. doi: 20000/TTAMML2221.1111111. URL https:///ott 10. 48550/
arXiv. 04922
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozcfowicz, and Samy Bengio. Generating Sentences from Continuous Space. In Proceedings of The 20th SIGNLL Conference Compulational Natural Language Learning, pp. 10 21, Berlin, Germany, 2016. Association for Computational Linguistics. URL https:://do..
doi: 10.
Michael M. Bronstcin, Joan Bruna, Taco Cohen, and Petar Veličković. Goometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges, May 2021. URL httpps///doi http::///ooooo/1.../1 wtt.:o....//..///............//. (88550/arivv255 2104 13478
Kenta. Cho and Bart Jacobs. Disintegration and Bayesian Inversion via. String Diagrams. Mathematical Structures in Computer Science, (7):9938 971, August 2019. ISSN 0960-1295, 1469-8072. doi: 10.1017/ 0U960122518000488 URL https:///oii
Casey Chu, Andrey Zhmoginov, and Mark Sandler. Cyclegan, a master of steganography. arXiv preprint 1712 02950, 2017. URL htttp:://bbi.oor//104885.85 wwt.:xiww.......0001
24

---

larxiv..org/abs//155... Published in Transactions on Machine Learning Research (12/2025)
Lorenzo Ciampiconi, Adam Elwood, Marco Leonardi, Ashraf Mohamed, and Alessandro Rozza. A survey and taxonomy of loss functions in machine learning, January 2023. http::////a.....////04400/000//1
2301.05579
Bob Coecke and Ross Duncan. Interacting quantum observables: categorical algebra and diagrammatics.
New Journal of Physics, 1344)00 April 2011. doi: 00////666-263/////4//
URL https :
didx.doi.orr wwwooo/////// 1088/ 1 2630/13/4/043016
Bob Coecke and Aleks Kissinger. Picturing Quantum Processes: First Course in Quantum Theory and
Diagrammatic Reasoning. Cambridge University Press, Cambridge, 2017. ISBN 978-1-107- doi: 10.
1017/97813162199117 URL https://www cambridge
Bob Coccke, Mchrnoosh Sadrzadch, and Stephen Clark. Mathematical Foundations for Compositional 4394
Distributional Model of Meaning, March 2010. CRL
Geoffrey SH Cruttwell, Bruno Gavranović, Neil Ghani, Paul Wilson, and Fabio Zanasi. Categorical loundations of gradient-basee based learning. In Programmingg Languages and Systems: 31st European Symposium on Programming, ESOP 2022, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2022, Munich, Germany, April 2-7, 2022, Proceedings, pp. 1-28. Springer International
ooc.lii.orro/ooo Publishing Cham, 2022. URL httph:tt /tdps:://do...o. oro//rg////0 10. 48550/arX11 210..
Artur d'Avila Garcez and Luis C. Lamb. hupss//arxvv.org/ abb.a/b2...077662222 December 2020.
Neurosymbolic
AT: The 3rd
Wave.
Gintare Karolina Dxiugaite, Zoubin Ghahramani, and Daniel M. Roy. A study of the effect of JPG compression adversarial images, August 2016. URL htttpp::/ddi.org/// http:///....005//...........
Brendan Fong and David I. Spivak. An Invitation to Applied Category Theory: Seven Sketches in Composi- lionalily. Cambridge University Press, edition, July 2019a. ISBN 978-1-108-66888 11-0--660---99----1----------0 978-1-108-4822955 doi: 10.1017/9781108666
978-1-108-7118--1
Brendan Fong and David I. Spivak. Supplying bells and whistles in symmetric monoidal categories., August 2019b. URL https://doi..rgg htt.:/i.or//...../////iaiiii. 1908. 02633
Thomas Fox. Coalgebras and cartesian categories. Communications in Algebra, 4(7):665 667, January 1976a.
ISSN 0892-7872. doi: 10.1080/00992777760022222
Thomas Fox. Coalgebras and cartesian categories. Communications in Algebra 4(7):665 667, January 1976b. ISSN 0092-7872, 1532-4125. 009278776088221277
doi: 10. 1011089/0092787768822222 URL URLhtttss: /ttp://:////1..011110000000
Tobias Fritz. A synthetic approach Markov kernels, conditional independence and theorems on sufficient statistics. Advances in Mothematics, 370:107239, August 2020. ISSN 00018708. doi: 10.1016/j..im..100 2020.
107239. URL http::://boo.orr///0000//////// aim.. w.m..0.0..m..m....m....m........ 107239
Tobias Fritz, Tomáš Gonda, and Paolo Perrone. De Finetti's Theorem in Categorical Probability. Journal of Stochastic Analysis 244,, November 2021. ISSN 2689-6931. doi: 2.31339/josaa.24.066 URL http
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. Geometry- consistent generative adversarial networks for one-sided unsupervised domain mapping. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Bruno Gavranović, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João G. M. Araújo, and Petar Veličković. Categorical deep learning: An algebraic theory of architoctures, February 2024. URL https 7/doi 1awrg///...55//aaaae 24332.13332
25

---

Published in Transactions on Machine Learning Research (12/2025)
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commanications of the A ACM, 63(11): 139 144, 2020. URL
Nathan Haydon and Pawel Sobociúski. Compositional Diagrammatic First-Order Logic. In Veikko Pietarinen; Peter Chapman, Leonie Bosveld-de Smet, Valeria Giardino, James Corter, and Sven Linker (eds.), Diagrammatic Representation and Inference, Lecture Notes in Computer Science, pp. 402 402-418, Cham, 2020. Springer International Publishing. ISBN 978-33-00--54249-8. doi: 10.1007/978-33000--5522 URL http://doorr///01000//////222360
32.
Jules Hedges. String diagrams for game theory, March 2015. URL http:///arxxv..og//aa//15533
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to- prompt image editing with cross-attention control. In The Fleventh International Conference on Learning Representations, 2022. URL https://doi.orr//00 48550 405.000055000//.....0 22t......2...01600
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 8-vae: Learning basic visual concepts with a constrained variational framework. In Proceedings of the 5th International Conference on Learning Representations, 2017.
Trina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a Definition of Disentangled Representations, December 2018.
Martin Hofmann, Benjamin Pierce, and Daniel Wagner. Symmetric lenses. ACM SIGPLAN Notices, 46(1): 371-384, 2011.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Newral networks, 2(5):359 366, 1989.
Bart Jacobs, Aleks Kissinger, and Fabio Zanasi. Causal inference by string diagram surgery. In Foundations of Software Science and Computation Structures: 22nd International Conference, FOSSACS 2019, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2019, Progue, Czech Republic, April 6- 11, 2019, Proceedings 22, pp. 313 329. Springer, 2019. URL https://aoi.org 10. 1007/978--3-000-1722---
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosscri, and Michal Trani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CV Conference on Computer Vision and Pottern Recognition (CVPR), pp. 6007-6011 June 2023.
Nikhil Khatri, Tuomas Laakkonen, Jonathon Liu, and Vincent Wang-Mascianicn On the anatomy of attention, 2024.
Diederik P. Kingmna and Max Welling, Auto-Encoding Variational Bayes, December 2022. URL https:
//doi //doi.org//i 10. 48550/8555350 awrvvv..vvv. 1312.6114
Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An Optimization-centricc View on Bayes' Rule: Reviewing and Generalizing Variational Inference. Journal of Machine Learning Research, 23:1-109, 2022.
Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. MIT Press, Cambridge, MA, 2009.
Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1865-1874, 2018. URL httpp:///ol org/10. 1.0.118633/1/111-1116
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
26

---

Published in Transactions on Machine Learning Research (12/2025)
Robin Lorenz and Sean Tull. Causal models in string diagrams, April 2023. URL https://do1.org//1 48550/arXiv 2304.07638
Martin Markl, Steve Shnider, and Jim Stasheff. Operads in Algebra, Topology and Physics. American Mathematical Society, Providence, RI, July 2007. ISBN 978-0-8218-4362-8.
Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation, 2021.
Keisuke Nakano. A Tangled Web of 12 Lens Laws. In Reversible Compulation: 19th International Conference, RC 2021, Virtual Beent, July 7 8, 2021, Proceediigss, pp. 185 203, Berlin, Heidelberg, July 2021. Springer- Verlag. ISBN 978-3-030-79836-9, doi: 10.1007/978 1111007978---000 3-000 79837-6
Andrew Ng and Michael Jordan. On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes. In Advances in Neurol Information Processing Systems, volume 14. MIT Press, 2001.
nLab authors. Bayesian inversion. Revision 9
http:://aaaaaboro//////////////t June 2024a.
nLab authors. Markov category. httpp:///eeaaaaa...gggaaaa////ar July 2024b. Revision
56
Prakash Panangaden.
The Category of Markov Kernels. Electr. Notes Theor. Comput. Sci., 22:171-187.
December 1999. doi: 10.1016/51171-0061111)/ (055)000044 URI http:////.a..aoo00000000//////// 80602-4
Dusko Pavlovic. Monoidal computer i: Basic computability by string diagrams. Information and Computation, 226:94-118 116, 2013. ISSN 0890-5401. doi:
1tt.://////..111......000000 sciencedirect. aataacttieett.ooo/cccccaaaaaaaaa htt.://we.................eooooa Special Issue: Information Security Resource.
URL bttts://wwx
Dusko Pavlovic. Programs Diagrams: From Categorical Computability to Computable Categories. Springer, 1st ed. 2023 edition edition, September 2023. ISBN 978-3-031-3482666. URL https://doi. orrelirrg//110 wt..g//1..885000 arXiv 2208 03817
Boldizsár Poór, Quanlong Wang, Razin A. Shaikh, Lia Ych, Richic Yeung. and Bob Coecke. Completeness for arbitrary finite dimensions of ZX W-calculus, a unifying caleulus, April 2023. URL http://arxiv.orrg
0/5/2332..2221
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training, 2018. URL https://api. semanticscoolar. wwwpaaatticccooaa.ooooo//rrppppe 493132455
Aare'oo Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. unified energy-based framework for unsupervised learning. In Marina Meila and Xiaotong Shen (eds.), Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, volume 2 of Proceedings of Machine Learning Research, pp. 371 379, San Juan, Puerto Rico, 21 24 Mar 2007. PMLR. URL https:://proceedings mlr. v2/ranzato07a.hta html
mlr.press/ press/
Oliver E. Richardson. Loss the inconsistency of a probabilistic dependency graph: Choose your model, not your loss function. In Gustau Camps-Valll Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 2706-2735 2735. PMIR, 28 30 Mar 2022. URL httpps://proceeding.. .mlr
prees/v1B//rrccarrrso2....1
Joseph Rocca. Understanding Variational Autoencoders (VAEs). VttEa,,,ttss///oaaadddaaaaaaaa http:///oontdddaaccccccoooooootd aaniattonnlanttoooodeerssse////o
March 2021.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Bernhard Scholkopf, Francesco Locatello, Stefan Baner, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the TEEE, 109(5), 2021. ISSN 0018-9219.
Peter Selinger. A Survey of GraphicalLangungeee for Physics, pp. 289-355. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011. ISBN 978-3-642- 642-12821-9, doi: 0011077/978-3--12-22822
Languages for Monoidal Categories. Tn Bob Coecke (ed.), New Structures
4. URL https:///oii 3-0
Lennert De Smet, Pedro Zuidberg Dos Martires, Robin Manhaeve, Giuseppe Marra, Angelika Kimmig, and Lue De Readt. Neural probabilistic logic programming in discrele-continuous domains. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, pp. 529-538. PMLR, July 2023.
Pawel Sobociíski. Graphical Linear Algebra, June 2015. URL hha.lattps//ggepptaaaalilarrrrrr
Amas Storkey, Won-Min Seong, and Christopher K. I. Williams. Machine learning markets. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pp. 716-722 2011.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. "Transforming" delete, retrieve, generate approach for controlled text style transfer. In Proccedings of the 2019 Conference Empirical Methods in Natural Language Prucessing and the 9th International Joinl Conference on Natural Language Processing (EMNLP-IJCNE) pp. 3269 3279, 2019. URL https://doi. org/10. 48511.....0//1111111 48550//aKiv....../
Richard Sutton. The Bitter Lesson, 2019.
Jua Juan Terven, Diana M. Cordova-Esparza, Alfonso Ramirez-Pedrawa, and Edgar A. Chavez-Urbiola. Loss Functions and Metrics in Deep Learning, September 2023. URL https://doii /ttp::////oo..////////// www..o.....././////.// 48550/ arr.v.rriii... 2307 0269
Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, New York, 1st edition edition, September 1998. ISBN 978-0-471-03003-4.
Xin Wang, Hong Chen, Si'ao Tang, Zihao Wu, and Wenwu Zhu. Disentangled Representation Learning, June 2024. Comment: Accepted by IEEE Transactions Pattern Analysis and Machine Intelligence.
Wang- Masng-Maciianca. String Diagrams for Teat. 2023.
University of Oxford,
Vincent Wang- Jonathon Liu, and Bob Coecke. Distilling Text into Circuits, January 2023. URL
http://arxiv. org/ hwwpg//bb//31..... 10595
Nicholas Watters, Loic Matthey, Sebastian Borgeaud, Rishabh Kabra, and Alexander Lerchner. Spriteworld: A flexible, configurable reinforcement learning environment, 2019. URL httts:://tttt// com domm/eeemind////// spriteworld/
Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270-20) 280, 1989. doi: 10.1162 htwee////oo1/// http///cc..199.....3 1.8......77. URL https://dot
org/10 10. 1162/neco..o.. 1989. 1.
Sidi Wu, Yizi Chen, Samuel Mermet, Lorenz Hurni, Konrad Schindler, Nicolas Gonthier, and Loie Landrieu. Stegogan: Leveraging steganography for non- nijebijective mmage-to-image translation, March 2024. URL
https://doi.org:
http://000..0r/11.4455555555 48550/ arxiv. 2403.20142
Donald Yau. Higher dimensional algebras via colored PROPs, September 2008. URL https://doi.org:
48550/arXiv.08m 2161
Jun-Yan Zhu, Tacsung Park, Phillip Isola, and Alexci A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, 2017.
28

---

Published in Transactions on Machine Learning Research (12/2025)
A String diagrams for tasks
Section 1 in the main text alrcady introduces the working grammar of our diagrams. Here we collect the broader categorical background and semantics. String diagrams are a formal diagrammatic synlax that lake semantics in symmetric monoidal categories, and they find usage in a broad variety of fields the generators from Figure for convenience:
We reiterate
AM4 -
gAr
カーーーー g
Function
Singleton space
Delete
Data from X'
Sequential composition
F: T.
N
= (+)
e: +
{+)
X: (+) - RM
(g f) ad - RC
-. RM
(6)
RM
aM
gM
RN
RM
RC
Vector
Swap
Copy
Independent N,Y (x x >): (x) x(M44))
Parallel composition
() (M+NN)
RM MMM+
a): k(440) 2(++D)
For conventional reasons that were not by our choice, vectors are depicted as triangular nodes with only output wires, reminiscent of bra-ket notation. (Co)associative (co)monoids, such as copy-delete and add-zero, are specially depicted as circular nodes as is common in applied category theory. In this work, encoders and decoders are sometimes depicted as "bottlenecking trapezia", as is common in ML, and distributional states are given their own notation as thick bars
An attractive characteristic of string diagrams is that visually intuitive equivalences between information flows are guaranteed to correspond to symbolic derivations of behavioural equivalence: tedious algebraic proofs of equality between sequentially- and compoit- processes are suppressed and absorbed by (processive) isotopies of diagrams. In the diagrammatic syntax it is conventional to notate such isomorphisms as plain equalities. Interested readers are referred to (Selinger 2011 for the relevant mathematical foundations.
(1 ())
~ (1 e 0) (1 01 o (f
[dentity, interchange]
(1 1 1) (f (fee)
[Brald naturality]
= (1 6 (1 of e /
(1
[Copy naturality]
9
8 1) ftlffeeeee fl (1e)))
[Braid naturality]
A.1 Categorical semantics of task diagrams
The functional effect of the construction below is to extend the category of continuous maps between Euclidean spaces with global elements that behave as probability distributions instead of points. We presume familiarity with symmetric monoidal categories and their graphical calculi (Selinger 2011).
Let CartSSp denote the coloured PROP Yau 2008) of continuous maps between Euclidean spaces, where the tensor product is the cartesian product i.c. CartSp is cartesian monoidal.
Let BorclStoch denote the Markov category (Cho & Jacobs 2019
Fritz 2020 of stochastic kernels |Panaggaden 1999 between Borel-measurable spaces. Stochastic kernels in particular subsume the continuous maps between Euclidean spaces.
As a Markov category, in the terminology of Fong & Spivak 2019b) BorelStoch supplics cocommutative comonoids. By Fox's theorem (Fox) 1976b) cartesian monoidal calegories are precisely those isomorphic their own categories of cocommutative comonoids. Hence there (semicartesian) functorial embedding of CartSp into BorelStoch sending V to N (equipped with the usual Borel measure), and continuous maps
$Including linear and affine algebra
[Sobocifskii 2015
Bonchi et al. 2017 2019) first order logic (Hyyon & Sobociñski
2020). caussal models [Lorewx & Tull
2023
Jacobs al.
2019). signal
flo graphs (Bonchi al. 2014), electrical circuits
Boisscau & Sobociñiski] 2022), garue theory
(odges 2015
petri nets BBaz & Master 2020), probability theory (Ditz all
2021 formal linguistics (Coccke et al. 2010
Wang-Masciante Mascianica al. 2023
Wang-Mascianle 2023 quantum thcory (oocoke &
Duncan 2011
Coecke Kissinger
Poor al. 2023 and aspects machine learning such backpropagation Cruttwell
29

---

Published in Transactions on Machine Learning Research (12/2025)
to deterministic continuous maps. We declare semantics to be taken in the category to be generated
by the image of this embedding along with the probability distributions D {*) - R.V. where (*) is the singleton monoidal unit of BorelStoch.
A.2 Categorical semantics for idealised universal approximators
As we are concerned with bchaviour, not implementation details, we idcalise all neural networks as perfect universal approxinators, which we may formulate diannggdiigaammticallllyy in monoidal closed category, borrowing calluator-notatan from (Pavlovic, 2013 2023). In essence, we are assuming that architectures are sufficiently expressive to optimise whatever tasks we give them; in practice, the conditions under which architectures become universal approximators can be mild (Hornik et al.) 1989), and the idealisation is increasingly true-in-practice in the contemporary context of increasing data and compute.
Definition A.1 (Learner). Let X,Y denote imput and output types. A process Ω X Y with parameters in para( R (for sufficiently large n) is universal approrimator or learner wher 0
X
Y
Y
Vfx=rafep
f
X
Ω
The parameter space could represent e.g. the phase space of weights and biases of neural network.
Example A.2. By visual convention, we use colours to indicate diffferent data types of wires. We depict processes with no free parameters as white boxes, and learners as black-boxes with variable labels indicate distinct shared parameters. The following composite process has function f with learnable parameters, and three neural nets: the two labelled share parameter in the space and the one labelled 3 takes a parameter in Q. In this paper, we favour the shorthand on the right.
p
Ω
Ω
Ω
Q
f
f
The universal approximation theorem, suitably idealised, manifests as the capacity for a black-box learner to diagrammatically substituted for any other composite diagram with equal input and output, including those composites that contain other learners. For example, recall the linear put below, which may be viewed as substituting particular composite of put', enc, dec, and addition in place of put:
GBC
dec
put
This ability is referred to in this work intermittently specialisation, and expressive reduction in (Khatri et al. 2024) where the concept. first appeared. For the sake of completeness, reproduce the relevant. construction that gives category-theoreti semantics to universal approximators and specialisation below, along with standard definitions, with the authors' permission.
Definition A.3 (PROP). PROP is a strict symmetric monoidal category generated by a single object r: every object is of the form
8 x
PROPs may be generated by, and presented as signatures (E, E) consisting generating morphisms with arity and coarity in N, and equations relating symmetric monoidal composites of generators.
Ww adapt the shape of the universal approximators clearly indicate the parameters.
30

---

Published in Transactions on Machine Learning Research (12/2025)
Definition A.4 (Coloured PROP). A multi-sorted or coloured PROP with set of colours C has monoid of objects generated by E.
Definition A.5 (Cartesian PROP). By Fox's theorem Fox 1976a), a cartesian PROP is one in which every object (wire) is equipped with a cocommutative comonoid (copy) with counit (delete) such that all morphisms the category are comonoid cohomomorphisms.
Definition A.6 (symmmetric, unital) coloured operad). Where (V, J) is symmetric monoidal category and C denotes set of colours C; coloured operad O consists of:
For each n E N and each (n + 1)-tuple , an object O(ci, ((c1s"";,,iin) e V
For each e E C, morphism I. :e:. + O c) called the identily of e
For each (n + t)-tuple (c1....nic) and n other tuples (d} ....))))) (df ...d.,))) composition morphism
O ;n: A (d) dz ) (d ... ... O(} di. ...d -.
for all n N, all tuples of colours, and each permutation a S, the symmmetric group on n, a morphism:
-..ei Gnic)
The o² must represent Sn, and composition must satisfy associativity and unitality in S,,invariant manner. Construction A.7 (Hom-Operad of coloured PROP). Where (P,0, , I) is a coloured PROP with colours Ep. we construct Op, the hom-operod of P. We do so in two stages, by first defining an ambient operad, and then restricting to the operad obtained by collection of generators. Let the ambient symmetric monoidal category be (Set, x, {+)). Let the colours to be the set of all tuples (A,B), each denoting pair of tuples (A1 An, Bi @ B₂) of Ai, Bi Cp.
The tuple ((A), BB).......
.......(4 (A", (A",B" "", (A, is assigned the set IP(A',B) x... ... X "(A,,"" + P(A,B) E Set; the set of all generated functions from the product of homsets P(A', B') to the homset P(A,B).
(A.B) (+ (P(A,B) + P)] is the identity functional that maps f:A - B in P(A, B) to itself.
The composition operations correspond to function composition in Sct, where [X - Y -| [X Z] sends (J.x-r, - (g f))x- appropriately generalised to the multi-argumenl case. The permutations are similarly defined, inheriting their coherence conditions from the commutativity isomorphisms of the categorical product x.
The generators are:
For every f P(A,B) that is a generator of P, define corresponding generator of type {*}
[P(I.I) + P(A, B)], which is the functional (- + (f ()) that sends endomorphisms of the monoidal unit of P to their tensor with f, viewed as an element of the set [P(I.T) - P(A,
For every pair of tuples ((X², Y!)... (X Y")); (A, B)) and ((J²,K)) K²) ... (J" K"); (B, C)) in Co. corresponding sequential composition operation of type:
P Y P(A.B)) II P(J) K') P(B.C) jEn
mL
+ I( II P(X Y") x II P(J) K)) P(A,
nn
Which maps pairs of functionals (F. P(X: -P
G. II P(J (.)-K.))P(BB.)) to the functional
which sends p¹ X - Y and q¹ x YJ to G(p ..mmm o F qa). qn)
31

---

Published in Transactions on Machine Learning Research (12/2025)
An analogous parallel composition for every pair of tuples, which sends pairs of functionals (F.G) to G(pr." "Pm)) 00 F(g .a))).
Remark A.8. For technical reasons involving scalars (the endomorphisms of the monoidal unit), this construction only works in semicartesian settings, i.e. where the monoidal unit is also terminal, but that is sufficiently general to admit our use cases, which are primarily in cartesian monoidal settings (Fox 1976a and semicartesian Markov categories for probabilistic settings (nLab authors 2024b).
Example A.9. Construction A.7 can be thought of as bridging diagrams with their specific algebraic descriptions using just the basic constructors o, @; the hom-operad (when notated suggestively in the usual tree-notation, found e.g. in Markl et al. (2007) plays the role of the syntactic tree of o, & operators. For instance, given the composite morphism operad-state in Op is:
(0 f) in PROP P, the corresponding diagram and E) (1A 10
[P( P(A AM
FF P(A
[P(1, PL (0)]
[P(7, PA
(P(),)) PLA P1
(F - PLA 8]]
11
[P(1. 73 - P(B,
Since the PROPs CartSp and its free tensoring cartesian, III,I) is singleton containing only the identity of the monoidal unit, so in the settings we are concerned with, may simplify colours of the form [)(,,)) + P(A, B)] to just P(A, B), and operad-states {+} + P(A, B) are in bijective correpondence with morphisms f: + B of P; the fact that all f A - B are representable as operad states follows by construction, since any ʃ in P by definition expressible in terms of the generators of P, and sequential and parallel composition o, @. As we assume homsets are alrcady quotiented by the equational theory of P and the symmetrie monoidal coherences, our operadic representations inherit them: for example, we obtain interchange equalities such as the one below for free:
P(A.
PLA,
P(A 8,
P(A, R))
P(R,
P(A
P(C,
PP
PCC, B)
P(B, D)
P[B,
P(D,
P(D, P)
Definition A.10 (Universal approximators and specialisation). A morphism of coloured PROP P of type (A, B) containing universal approximators as black-boxes of types At + Bis is morphism ((A1, B²) ".)...(AA)) B"); (A, B)) of Op, and by construction, vice versa. Specialisation corresponds to precom- position in Op.
Example A.1I. The inputs of open morphisms Op correspond to "typod holes", and operadic precomposi- tion corresponds to "filling holes", with contents that may themselves also contain typed holes. This precisely formalises the intuition that expressive reductions correspond to the ability of universal approximator to
32

---

Published in Transactions on Machine Learning Research (12/2025)
simulate anything, including composites containing other universal approximators.
F(A, A)
P(A B C
B+
F(A P,
P(A B C.RRRCC
P(A 9
P(G. G)
PLR
P(B
PLC, G)
Remark A.12. The extension of the current. theory to accommodate parameter sharing between universal approximators is conceptually straightforward but technically involved. Parameter sharing corresponds to the ability to reuse i.e. copy data between open wires in the operad Op, which amounts to having a cartesian operad.
B Strong manipulation
The basic manipulation admits pathological counterexamples, which we may block by imposing additional tasks as regularisation terms. This section further illustrates a form of legalistic thinking using tasks: by thinking of ways that "noncooperative" the behaviour that the modeller desires. By identifying these counterexamples and constructing additional tasks that block them, the modeller may iteratively improve the behaviour of the model. For illustration, consider the following examples of pathological behaviour that satisfy basic manipulation, again in the setting of editing the colour of shape.
"naughty" learners might seek to satisfy tasks without cxhibiting
Example B.1 (Flipping). Consider put that changes the colour of shape as desired, but then vertically flips the shape, If the classifier get is insensitive to the position and orientation of the shape, then CLASSIFY, PUTGET, and GETPUT are satisfied. Morcover, since a vertical flip its own inverse, composing two puts as in the UNDOABILTTY lask will not detect this aberration. Speaking in more general terms, if there are properties that get insensitive to, there must be additional guardrails to ensure that put preserves these other properties as the identity, rather than one of potentially very many self-inverse symmetries.
Example B.2 (Adversarial decorations). While the classifier get may be perfect in-distribution, there are no guarantees about its behaviour out of distribution, for example, when given images with multiple shapes, where it might only classify the leftmost shape. So, it is possible that put learns to make edits that take the resulting image out-of-distribution for example, by adding red circle next to a blue square to fool the classifier into outputting "red". This would satisfy CLASSIFY, PUTGET, and GETPUT. If the put recognisc and undo its own decorations, then UNDOABILITY will also be satisfied. Speaking more generally, we require additional guardrails to ensure that put returns something in-distribution.
The strong manipulation adds additional tasks to manipulation. A strong manipulator has to satisfy the original four tasks plus the following four:
33

---

Published in Transactions on Machine Learning Research (12/2025)
Task B.3 (strong manipulation add-ons). The strong manipulation task consists of all the tasks
tained in manipulation, we well as:
H
get,
X
put
(FAKR)
A
A
put
(PotPut)
dac
put
X
A
put
dac
A
A
X
dac
put
(Foot)
X
dsc
(TRUE)
A
The PUTPUT task (which strictly stronger than UNDOABILITY that it algebraically implied) says that the effect of putling twice is the as discarding the effect of the first edit and only keeping the last edit. In conjunction with PUTGET and GETPUT, this creates what is known in the literature as very well-behaved lens, which blocks Example B.1 and similar modifications of the data get is insensitive to.
The TRUE, FAKE, and FoOL tasks introduce a discriminator component dsc, which forms a GAN pattern with respect to put as the generator. When well-trained, this forces the outputs of put to lie in-distribution. As in general there are no algebraic or equational laws that characterise arbitrary distributions of data, using GANs this way is a generic recipe for shaping outputs of generators behave well in in-disriibution.
Remark B.4 (Why basic manipulation is preferable in practice). We have observed informally that conditions such as those in basic manipulation where learners are cooperative and there are only learners on the LHS appear to be more stable during training. We suggest a sketch reason why: in the tasks of strong manipulation, PUTPUT has put occur on both the LHS and RHS, which establishes a nontrivial dependence the current position parameter-space of put in the process of finding a solution. Similarly, the GAN rules of strong manipulation establishes adversarial mutual dependencies in the parameters of dsc and put. Conceptually, these dependencies create dynamical systems on the paths that the learners take over the course of training in parameler-space, which may for instance include stable orbits and chaotic behaviour, and may highly sensitive to initial conditions. A further elaboration of "static" versus "dynamic" tasks in tandem with the ability express equivalent tasks is potentially useful for creating train-stable models with equivalent behaviour, but this is beyond the scope of this paper, and left for future work.
C Experiment Details
C.1 Stack
This experiment uses an autoencoder architecture for processing 32 x RGB images, with a latent space size of 16 dimensions. The encoder consists of four convolutional layers, each with 64 channels and channel multiplier of 1. Additionally, stack of 64 latent. features is processed through MLP with 256 hidden units. Training is performed on GPU, with batch size of 64, learning rate of 10-4, weight decay of x 10 10-2, and gradient clipping at 1. The model trains for 100,000 steps, logging every 10,000 steps. Input images are converted to Lensors and scaled to floating-poit precision, and a fixed random seed of 0 ensures reproducibility. The stack space S has to be n times larger than the latent space Lat for the stack to be able to hold up to n items.
C.2 Spriteworld
For the Spriteworld experiment, we procedurally generate 32x32 images containing a single shape with the following attributes:
34

---

Published in Transactions on Machine Learning Research (12/2025)
Attributes
Possible Values
Shape
{ Ellipse, Rectangle, Triangle
Hue
{ Red 018, Green: 8518, Blue 17018 }
Saturation
64-255
Value
64-255
Background Color
Black
Width & Height
5-27
X & Y position 5-27
Only the first two attributes, shape and hue, are changed in the manipulation task, but all unchanged propertics are intended to be preserved by the transformation. We use an autoencoder with a CNN/DCNN architecture to embed each image into latent space:
Parumeter
Value
Latent Size
32
Layers
Hidden Channels
64
Kernel Size
5x5
Stride
Activation Function
LeakyReLLUO01) followed by BatchNorm
We train separate get/put models for each of the three concepts: shape, colour, blue-circleness. Each model uses the encoder of the autoencoder to embed input images into latent space, and only sees the labels for the particular attribute it manipulating.
For shape and colour, the get model uses a linear classifier from the latent space (of size 32) to output logit values, one for each possible value. The put model maps one-hot vector of the input value vector in latent space that is added to the embedding. This new embedding is then decoded by the autoencoder.
For blue-circlenes, the get model uses a linear classifier from the latent space to a single output value from zero to one (we do not use a sigmoid output layer to restrict the output). The put model uses a complement of size 8. It concatenates the onc-hot value vector with the image embedding and the complement vector (using a default trainable complement vector one not provided) and passes that through a linear layer Lo get a new embedding vector (which is then decoded) and complement vector.
In total, these modcls contain 644,130 parameters. All models trained simultancously according to the autoencoding and manipulation rules, along with PUTPur. At each step, batch of images is generated, along with four batches of random values, containing random labels for the shape and colour, and random real numbers for the blue-cireloness, uniformly sampled from [-0.1, 1.1] and then clamped to [o, 1]. The loss function is a weighted sum the losses from each atomic task in order to balance the signal from the image loss with the signal from the classifier loss:
Hyper-parameter
Value
Task
Weight
Steps
100,000
AUTOENCODING
100
Batch Size
512
GETPUT
1
Optimiser
AdamW
PUTPUT
1
Learning Rate
10-3
UNDO
10
Weight Decay
10-2
PUTGET
Gradient Clipping
(clement-wise)
(bluc-circleness)
10
Image Loss
La + 0.25 Lı
(shape and colour)
1
Discrete Value Loss
Binary cross-entropy
CLASSIFICATION
Continuous Value Loss
Mean squared error
(blue-circleness)
10
Seed
(shape and colour)
1
35

---

Published in Transactions on Machine Learning Research (12/2025)
C.3 Faces
For the faces experiment, we use the CelebFaces Albribules datasel, [Liu et al. 2015), with off-the-shelf
data augmentation method called TrinenlAugment (Müller & Hutter 2021 Again, we use an autoencoder
with a CNN/DCNN architecture to embed cach image:
Parameter
Value
Latent Size
128
Layers
Hidden Channels
8, 16, 32, 64, 128
Kernel Size
Stride
Activation Function Leaky followed by BatchNorm
We train linear get/ put models for the binary concept of "Smiling", resulting in total of 1,071,749 parameters. The loss function is weighted sum of the losses from each atomic task:
Hyper-perammter
Volue
Task
Weight
Steps
100,000
AUTOENCODING
10
Batch Size
GETPUT
Optimiser
AdamW
PUTPUT
Learning Rate
10-3
UNDO
1
Weight Decay
10-2
PUTGET
1
Gradient Clipping
1 (element-wise)
CLASSIFICATION
1
Image Loss
L2 + 0.2 L1 + SSTM
Value Loss
Binary cross-entropy
Seed
C.4 MNIST
We trained the manipulation task on the MNIST dataset, using the digit label as the property. The get operated directly on images, while put was trained to act on the latent space of an autoencoder, as in option (1) of Section 4.3 The images input as 28 28 matrices, flattened to 784-dimensional vectors, and the labels are provided 10-dimensional vectors with onc-hot encoding. All of the components structured multilayer perceptrons. The hyperparaneteers are given below:
enc
put
get
Input Dimension
784 = 28 x 28
32
42 = 32 + 10
784 = 28 x 28
Output Dimension
32
784 = 28 x 28
32
10
Hidden Dimensions
(128, 128, 64)
{64, 128, 128]
{128, 128, 128]
{64, 64) ReLU
Hidden Activations
ReL.U
ReL.U Sigmoid
ReLU
Final Activation
Sigmoid
Sigmoid
Softmax
With these architectural components, we trained four tasks: (a) training get supervised, (b) training get given pre-trained put, enc, and dec, (c) training put given pre-trained get', enc, and dec, and (d) training get match previous get'. These were trained using the manipulation rules, well PurPur, and additional regularization term we denote ENTROPY. The loss function of ENTROPY is given by
L CENTROY [[[[[get(enc(())] (([gge((nn((z))
where x is a batch of input images, H(+) is the entropy of a categorical distribution, and the expectation is approximated by the mean each batch. The idea behind ENTROPY is to encourage the output of get
36

---

Published in Transactions on Machine Learning Research (12/2025)
to be well-distributed across labels (by maximizing the entropy of the mean distribution) but to be sure of each label (by minimizing the entropy for each specific input). For task (d), labels were generated for the CLASSIFY rule using get'. Each task is trained by minimizing a weighted linear combination of the rules. We give the hyperparameters, rule weights, and loss functions for cach these below.
(a)
(b)
(c)
(d)
Optimizer
Adam
Adam
Adain
Adam
Learning Rate
0.001
0.001
0.0001
0.001
Epochs
20
20
20
20
Weight
Loss
Weight
Loss
Weight
Loss
Weight
Loss
CLASSIFY PUTGET
CF
CE
10
CE
10
CE
GETPUT
10
L2
10
L2
PUTPUT
10
L2
10
L2
UNDOABILITY
10
L2
10
L,2
ENTROPY
LENTROPY
We also have an additional task (e) of training enc and dec unsupervised. This was done using the Adam optimizer, with a learning rate of 0.001 for 80 epochs. The reconstruction loss was given by
C(r, L2(x, f) + SSIM(2,, #))
where SSIM is the structure similarity image metric.
To produce Figure an enc, dec, put, and get were trained using (a) - (e) + (c), followed by training (c) for an additional 40 epochs. A slightly larger (but still MLP-based) model, where both put and get act on the latent space of the autoencoder, was used to achieve better visual quality. The hyperparameters are detailed below:
enc
dec
put
get
Input Dimension
784
28 28
32
42
32
10
32
Output Dimension
32
784 = 28 28
32
10
Hidden Dimensions
{128, 128, 128,
{32, 32, 128,
{256, 256}
{256}
32, 32}
128, 128)
Hidden Activations
ReLU
ReLU
ReLU Sigmoid
ReLU
Final Activation
Sigmoid
Sigmoid
Softmax
enc, dec, and put were used to manipulate six examples picked from the dataset, putting each of the ten classes onto each example. The examples were cherrypicked to provide maximum stylistic contrast across the sample but were not selected for maximum style transfer accuracy similar level was observed across the entire dataset. Code for all of these models, as well as the training schedules of tasks (a)-(e), are provided in the supplementary matcrial.
D Examining Manipulation in complex domains
D.1 Manipulation for text sentiment
We atlempled to fine-tune an exlant strong solution for text-sentimenl modification by additionally imposing the constraints manipulation on top of the original objective function. Our findings suggest that additionally imposing the constraints of manipulation on architectures that are already performant docs not make an appreciable difference Tabll 1). We pretrained the Blind Generative Style Transformer (B-GST) model of (Sudhakar al. 2019) which takes in the non-stylistic components of sentence and the target.
37

---

Published in Transactions on Machine Learning Research (12/2025)
sentiment, and outputs the sentence generated in the target style. This was done until we achieved baselines higher than the original ones reported by the authors of the model. Afterwards, we continued training under three different conditions: (1) resuming training with only the original objective, (2) only using objective functions from manipulation, and (3) using both. For (1) and (3), we reused the same objective in the original paper. In all experiments, we used the YELP dataset used by (Li et al. 2018), reusing the same train-dev-test split they used. It consists of 270K positive and 180K negative sentences for the training set, 2000 sentences each for the dev set, and 500 sentences each for the test set. Furthermore, we used the human gold standard references they provided for their test set. B-GST uses sequence length of 512, 12 attention blocks each with 12 attention heads. We used 768-dimensional internal states (keys, queries, values, word embeddings, positional embeddings). We tokenized the input text using Byte-Pair Encoding (BPE).
We used the same input autoencoding and output decoding used in (Sudhakar et al. 2019) across all experiments. For the get of the manipulation task, we used the PyTorch version of the pretrained Transformer by HuggingFace, which uses the OpenAT GPT model pretrained by (Radford & Narasimhan 2018) on the BookCorpus dataset which contains over 7000 books with approximately 800M words. We trained it on a sentiment classification task using the YELP dataset reaching 98% accuracy on the test set. The get was fixed for the entire duration of training conditions (2) and (3) above. For the put of manipulation, we used the B-GST model to generate text with a specified sentiment. This was a computational bottleneck for training conditions (2) and (3) as autoregressive decoding is required to generate model inputs for PUTGET and UNDOABILITY in manipulation. We used 'teacher forcing' or 'guided approach' (Bengio et al. 2015 Williams & Zipser 1989) whenever we computed the reconstruction loss of put. Additionally, for training conditions (2) and (3), we only used PUTGET, GETPUT, and UNDOABILITY from manipulation. We used weighted sum of the losses computed for each of these and the original reconstruction loss if present the weights can be considered as training hyperparameters. For (2), we used (PUTGET=5, GETPUT=20, UNDOABILITY=200), while for (3), we used (PUTGET=5, GETPUT=10, UNDOABILITY=25, B-GST=30). Code for all of the models, training schedules, and hyperparameter values for training conditions (1)-(3) are also provided in the supplementary material.
MODEL
GLEU BLEUSRC
BLEUREF
ACC (FASTTEXT)
E-GST-pretrainee
11.869
74.563
52.770
84.6
B-GST-only
11.426
74.876
52.549
85.7
manipulation-only
11.712
74.428
52.646
84.1
B-GSTvnanipulation
11.338
74.608
52.836
85.1
Human Reference 100.00
58.158
100.00
67.6
Table 1:
We pretrained the Blind Generative Style Transformer (B-GST) model (Sudhakar et al. 2019 based on the Delete-Retrieve -eneeerrte (Li et al.) 2018) framework for sentiment modification until we recovered higher baselines than reported by the authors of the model (GLEU=11.6, BLEUsnc=71.0) and we continued training in three different conditions: (1) keeping the original objective, (2) only using objective functions obtained from manipulation, and (3) using both. We report no statistically significant differences in scores, even under continued training. The table reports the results of training conditions (1) and (2) for additional epoch, and (3) for two epochs.
38

---

Published in Transactions on Machine Learning Research (12/2025)
D.2 Characterising Manipulation as generative classification
Training manipulation autoregressively (e.g. when instantiating the learners as transformers for sequential data) is slow due to autoregressing twice for PurPuT and Undoability. Moreover, our attempts to autoregressively manipulate the sentiment of LMDB reviews often resulted in form of posterior collapse where put ignored the attribute and behaved as the identity function on text. Conceptually, this is because the identity function satisfies GETPUT, PUTPuT and UNDOABILITY, and while the identity fails PUTGET, failing on one component of the combined loss function does not provide a strong enough incentive to move away from the identity in parameter-space.
Notably, these shortcomings mirror that of VAEs, which also suffer from posterior collapse [Bond-Taylor et al. 2022) in highly structured domains such as video (Babaeizadeh et. al., 2018) and text (Bowman el al., 2016). This suggested to us that puts may be generative classifiers, which could potentially explain why mode collapse was occurring in complex domains. Borrowing terminology from (Ng & Jordan 2001), classifiers are discrimánative if they seek to learn the conditional distribution p(ald) of attributes given data (as in the classification pattern), and otherwise they are generalive if they seek to learn the joint distribution p(d, a) (as, for example, a VAE). The tradeoffs between the two types are well studied, e.g. performance-wise, generative models may converge faster with limited data, but discriminative models often achieve lower asymptotic error, and is well known that learning generative models is harder (Vapnik) 1998).
Manipulation cannot be viewed directly as generative classifier, as the put admits extraneous conditionali- sations on a reference and a target attribute. So we resorted to an empirical comparison of manipulation and VA as a known generative classifier specifically, we compared their ability to approximate the Bayesian inverse, and we found their performance comparable. To demonstrate this, we tried three ways to train an informationally identical cls' given an initial cls, provided access to unlabelled data to obtain a distribution of pairs (d, cIs(d)) by: (a) directly training cls by classification, (b) training a generative classifier, in our case VAE, and (c) training manipulation around cls-as-get to obtain put, and then train cls' to satisly the tasks of manipulation except for CLASSIFY. Repealing this process several limes, would expect to see some loss of accuracy due to imperfect Bayesian inversion. Indeed, we see in Figure 9 that (b) and (c) have similar decays in accuracy, indicating that manipulation and VAEs have similar performance in this case. This experiment was performed using the trained components obtained from the MNIST experiment (Section C.4), and in addition to the tasks (a-e) we (f) trained a VAF, to learn the joint distribution of images and labels produced by get', and we (g) trained a get supervised using labels and images generated from the VAE. The VAE encoder and decoder MLP trunk and two linear heads for generating the means and log- variances of the latent space, or the image and labels, respectively. The latent space is comprised of independent normally distributed variables as in (Kingna & Welling 2022), and is sampled using the standard reparameterization trick. The hyperparameters of the architecture are given below:
also based on multilayer perceptrons cach is comprised of an
VAE Encoder
VAE Decoder
Input Dimension
794 28 x 28 10
32
Hidden Dimensions
{128, 128, 64]
{64, 128, 128]
Hidden Activations
ReLU
ReLU
Head 1 Dimension
32
784 = 28 x 28
Head 1 Activation
Sigmoid
Head 2 Dimension
32
10
Head 2 Activation
Softmax
Three loss functions were used for tasks (f) and (g) the reconstruction loss of the autoencoder, which can be separated into label loss and an image loss, the K-L divergence regularization term Cx of the VAE (Kingma& & Welling 2022.. ; and the CLASSIFY loss of get. The training hyperparameters are given as follows:
39

---

Published in Transactions on Machine Learning Research (12/2025)
Optimizer
Adam 0.001
Adam
Learning Rate
0.001
Epochs
40
20
Weight
Loss
Weight
Loss
CLASSIFY
CE
Image Reconstruction
100
L2
Label Reconstruction
CE
K-L Divergence
0.5
LKL
order to evaluate the three methods, we trained the tasks in the following order. At each step, the component being trained (e.g. get, put, etc) was initialized randomly (the previous weights were discarded). Measurements of the test accuracy made after each (a), (b), (d), (g) training run, and used to produce
F Figure 9
get 2
Bee
(a) (d)
1 (8)
- (d)
225 188 and +- ger
(a)
(e)
1 (6) 1 (e)
+ (c)
(b) + (b)
. 3VV // K 128
(a) (1) 7 36
1
1
(g)
40

---

Published in Transactions on Machine Learning Research (12/2025)
100%
80%
Accuracy
60%
(a) get + get'
40%
(b) get + VAE get'
(c) get - put + get'
0
1
2
3
4
5
Steps
Figure 9: We tried three ways to distil an informationally identical cls' given an initial cls depicted are the results of training successive MNIST classifiers using methods (a), (b) and (c) given above. 'steps' refers to the number times this process was repeated. We observe that the degradation of accuracy is approximately the same for both (b) and (c), which we consider evidence that manipulators and VAEs have similar performance characteristics. Both models had roughly the same number of parameters (300K) and were based on the same MLP architecture, We ran 20 repetitions of each method, the shaded regions represent one standard deviation (method (a) had standard deviation of less than 1%).
41


Reviews:
[{"review_id": "xxCv1O6FBA", "reviewer": "Reviewer_R3pr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To me, the main strengths of the paper are: \n\n1) The way the manuscript is written is actually good and very decent, so one can actually understand what the authors are trying to communicate, how this new diagrammatic language works, and what the main purpose of the proposed approach is. The work has probably been revised quite a lot, and I feel it could be even longer with certain quality materials transported to the Appendix. The visual explanations and examples are well-chosen, as well as the general structure of the paper, with the put-push actions, the way the loss is \"translated\", leaving the critical manipulation task for the end of the methodology section. \n\n2) Empirical results and experiments in general match very well with the examples and the cases considered for their analysis. Perhaps some extra results could've been included, but given the theoretical/review spirit of the paper, maybe is fine like this. Additionally, I also appreciated the fact that not only are very simplistic vanilla models considered, but a minimum degree of complexity is added to the examples and ML methods (i.e., cycleGAN).\n\n3) The conclusion/discussion and the limitation section are a good point of the paper, helping to close down the communication story rather than just summarising contributions.\n\nHowever, I unfortunately see critical flaws in the paper in three main directions that are difficult to ignore for any experienced reader in certain ML/AI models:\n\n1) **Cross-disciplinary contribution and the spirit of \"translating\" the syntax of certain models into the new diagrammatic reasoning language:** One of my feelings (or self-asked questions) while reviewing from the first moment is that the work was somehow introducing an existing formulation from other research disciplines in ML/AI (i.e., Mathematical Structures in CS, Concurrency Theory or Quantum Theory and Diagrammatic Reasoning). This is of course, fine, and promoting cross-disciplinary works and building bridges between key disciplines is critical in these days. However, I think that the way it is done is not entirely correct, or at least \"too cold\" for me, without actually explaining to the reader where all of these new ideas come from, what the problems and issues that motivated their development are, and why there are advantages when considering them for ML/AI models. Many of these questions are unanswered, and I could just speculate what their responses are.\n\n2) **Related work and lack of ties with existing relevant literature.** The introduction is just too short and lacks a lot of citing work, including references to SOTA methods and some long-considered ideas in the ML/AI community. The effort in the last section 5 is OK, when the approach is framed between neurosymbolic methods and compositional function models, but no comparison, mention, or deeper comments are considered all along the manuscript, which is actually a critical flaw of the paper. In this way, I ask myself a lot if including all of this would make the paper too long for TMLR, and it could be better to target venues or journals on the other side (experienced ones on diagrammatic reasoning rather than pure ML/AI). But this last point is just a not-well-informed opinion of a ML-reseacher reviewer. \n\n3) **The main goal of manipulating tasks on attributes and already existing ideas.** It is clear from my perspective that the main goal (as said more or less by authors) is to be able to capture/isolate via these manipulation tasks the function operations applied to certain attributes of the data while modelling or training representation learning models with some type of losses. As I said previously, these sorts of ideas are almost as old as the representation learning topic itself, and in the last decades have been particularly treated in the context of disentanglement. The fact that this is not considered (I think just lightly mentioned once in some paragraph), makes the paper very weak to be accepted in the current form, as it confuses the reader and does not align with well-known literature. Additionally, the advantages/weaknesses of the proposed approach should be compared in the empirical results with such a family of SOTA methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new *diagrammatic language* for representing specific tasks in certain types of machine learning models (i.e., autoencoders, generative models, etc). Despite the fact that specific (string) diagrammatic algebraic reasoning has been derived in previous literature, linked to applied category theory out of the usual ML/AI conference/journal circuit, we can see the work as an attempt to *translate* certain types of models to be expressed in this particular string-based language. As authors acknowledge in Section 5, the approach is *neurosymbolic* in spirit and matches the idea of task-composition while modeling ML/AI problems, which is something that has been explored before in the literature. Empirical results are partially synthetic and partially applied to small-size datasets for the considered examples.\n\nThe key idea for introducing such diagrammatic language in the context of ML models is (from this reader's point of view) to isolate certain modelling tasks on certain data attributes induced by the model itself or the chosen loss. In this way and using what is named in the diagrammatic language as \"manipulation\", the approach should be to constraint on \"tasks\" over \"attributes\", such one can induce certain behaviors in the performance of the model itself. Two examples of this sort of manipulation are shown in Figures 3 and 7, where \"graphicological\" properties on MNIST digits can be perceived, and on smiling/not-similing attributes of certain images of people. From the perspective of a ML/AI practitioner or researcher and in a broad sense, this sort of reasoning or manipulations are equivalent to the idea of \"disentanglement\" in representation learning. Which has been around for decades already. Disentanglement is just the modelling of the independent factors of variation in the data, so once modelled or captured, one can condition on them to obtain the outputs or predictions with the desired properties.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In my opinion, the manuscript is currently closer to a reject-score than one towards acceptance. To lean my decision on the positive side, I would like to see major changes on the three points included as \"weaknesses\", where I stated clearly what are the missing elements or the critical flaws that I am concerned about.\n\n### Minor issues\n1. Title in sub-section 5.2 -> \"extant\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6413ZI011x", "reviewer": "Reviewer_6dng", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* The writing is very clear, and much of the organisation of the paper is well thought out. Aside from one specific problem case described below, I did not have a problem understanding the points the authors were trying to get across. There are plenty of examples and thorough explanations of the core ideas.\n* The proposed framework represents a novel way of thinking the structure of machine learning models, tasks, and the associated objective functions. It is very common to represent modern machine learning architectures and pipelines in a diagrammatic form, but often these diagrams are developed ad hoc and lack a formal syntax and semantics, therefore resulting in a loss of information or ambiguity about how the diagram should be interpreted. The proposed framework overcomes this issue and it is demonstrated one can formally reason about architectures in this diagrammatic form. I can see this being quite an effective tool, similar to how , e.g., plate diagrams and UML have been indispensible in graphical models and software engineering research (and practice), respectively.\n* The technical aspects of the paper seem to be correct. I am not so accustomied to the diagrammatic style of the proofs (which of course cannot be avoided in this case), but I am reasonably confident all the results are correct. The various simple propositions and examples also serve as good pedagogical material for people wanting to build off this work.\n\n## Weaknesses\n* There are many positive aspects to the presentation, but I think introducing the diagrammatic frameworks only via examples is counter-productive. I found it quite hard to follow what the diagrams actually meant until I happened on the figure at the start of Appendix A. I would strongly recommend this is moved into this main paper.\n* The positioning of the framework with respect to previous work could be improved. The stated aim of the proposed framework is to put typical ML engineering practices on a more rigorous footing. Although it is not common to discuss it in application oriented deep learning papers, there are well-established frameworks for motivating the selection of various loss functions and macro-structures for learning algorithms. In particular, the statistical decision-theoretic foundations of machine learning are quite well-developed, providing a strong motivation to use methods based on, e.g., maximum likelihood estimation, Bayesian inference, empirical risk minimisation, and some adversarial learning settings. Various extensions of these ideas exist when specifically considering systems of multiple models; see, e.g., the work on Machine Learning Markets [A]. A discussion on the expressive power of the proposed framework compared to existing graphical model approaches would also be useful.\n* The main claims of the paper seem to centre around developing a framework that better enables descriptive and prescriptive analysis of machine learning algorithms, but I am not convinced this has been accomplished. It is stated that the proposed framework can be used to \"understand and compare the behaviour of models before committing to potentially costly training.\" However, there are not really any methodical ways in which the framework can be applied; I had been expecting that properties of the diagrams would correspond to properties one usually expects of machine learning methods (e.g., consistency, existence of a solution, optimality, stability, etc). Instead, the examples of propositions used to justify this claim mainly focus on formalising existing learning algorithms in the proposed framework.\n\n[A] Storkey, Amos. Machine Learning Markets. In AISTATS, 2011.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission claims to have three main contributions:\n* Formalisation of a \"common but informal standard procedure\" concerned with characterising the desired behaviour of different learned components of a system and using this to determine what objective functions should be used to train such components.\n* Within this formalisation, it is demonstrated how the diagrammatic algebraic reasoning of the proposed framework can be used to analyse, predict, and design behaviours of models.\n* As a case study, the `manipulation` task is considered. This requires a model to modify an attribute of a data point while leaving all other attributes unchanged. Several examples of this task are explored: coloured shapes, stylised MNIST digits, and smiling faces.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The change that I would most like to see is an expanded related work section and some revision of discussion w.r.t. prior work at the other places in the paper. This would address the second weakness given above.\n* As mentioned above, I think jumping straight into examples of the proposd diagrammatic framework does not work well; clarity would be improved by introducing the basic building blocks currently listed in Appendix A.\n* Rewording the claims to better reflect the capabilities of the proposed framework, or some additional results demonstrating methodical ways for how typical properties of machine learning algorithms can be extracted from the diagrammatic representations.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cTajsh528c", "reviewer": "Reviewer_E5wW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. The paper is generally well-written. The introduction starts from a very basic concept, then the main idea of this paper is presented. \n\n2. The proposed language is indeed universal for well-known machine learning tasks. \n\n3. Some analysis is also provided.\n\nCons:\n\n1. In the introduction, it might be better to emphasize the 'pattern language', otherwise, readers might be confusing what the main aim of this paper should be.\n\n2. The considered cases can be better categorized. For example, self-supervised learning, supervised learning, augmentation, etc.\n\n3. Can the authors show how to use plain language to teach people to use this pattern (obtaining a new machine learning task)?\n\n4. Figures should have captions.\n\n5. It might be better to follow some common notations used in the machine learning community. For example, \\mathcal{D} will be used to represent distribution, \\mathcal{X} is used to represent the feature space, and X is used to represent a random variable. Your notation system is no problem, but consistency between your naming system and machine learning community will make this paper have more potential readers.\n\n6. Can we hire some people to justify and evaluate the new language proposed in this paper?\n\n7. It might be better to show the function of each smallest component in the figures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "It is interesting to see an alternative to demonstrate machine learning tasks. This paper provides one unified way to express machine learning tasks within a new pattern language. Perhaps, one benefit would be that this kind of language can be easier to accept by people outside of statistics? I would like to support this paper simply due to its novelty, although it is still unclear what the potential impact can be.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In the introduction, it might be better to emphasize the 'pattern language', otherwise, readers might be confusing what the main aim of this paper should be.\n\n2. The considered cases can be better categorized. For example, self-supervised learning, supervised learning, augmentation, etc.\n\n3. Can the authors show how to use plain language to teach people to use this pattern (obtaining a new machine learning task)?\n\n4. Figures should have captions.\n\n5. It might be better to follow some common notations used in the machine learning community. For example, \\mathcal{D} will be used to represent distribution, \\mathcal{X} is used to represent the feature space, and X is used to represent a random variable. Your notation system is no problem, but consistency between your naming system and machine learning community will make this paper have more potential readers.\n\n6. Can we hire some people to justify and evaluate the new language proposed in this paper?\n\n7. It might be better to show the function of each smallest component in the figures.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)