Sparling: End-to-End Spatial Concept Learning via Extremely Sparse Activations
Authors: Kavi Gupta
Osbert Bastani
Armando Solar-Lezama
Venue: ICLR 2026 Poster (2026)
Keywords: machine learning
sparsity
interpretability
optimization
identifiability
Primary area: interpretability and explainable AI
TL;DR: We prove it is possible to identify an extremely sparse intermediate latent variable with only end-to-end supervision, and introduce Sparling, an extreme activation sparsity layer and optimization algorithm that can learn such a latent variable

Abstract: Real-world processes often contain intermediate state that can be modeled as an extremely sparse activation tensor. In this work, we analyze the identifiability of such sparse and local latent intermediate variables, which we call motifs. We prove our Motif Identifiability Theorem, stating that under certain assumptions it is possible to precisely identify these motifs exclusively by reducing end-to-end error. Notably, we do not assume identifiability of parameters, but rather of a latent intermediate representation output by a local model, thus allowing these representations to be arbitrarily complex functions of the input. Additionally, we provide the Sparling algorithm, which uses a new kind of informational bottleneck that enforces levels of activation sparsity unachievable using other techniques. We confirm empirically that extreme sparsity is necessary to achieve good intermediate state modeling. On synthetic domains, we are able to precisely localize the intermediate states up to feature permutation with $>90\%$ accuracy, even though we only train end-to-end.

SPARLING: END-TO-END SPATIAL CONCEPT LEARN- ING VIA EXTREMELY SPARSE ACTIVATIONS
Kavi Gupta
Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology Cambridge, MA 02139, USA kavigemit edu
Osbert Bastani
Department of Computer and Information Science University of Pennsylvania Philadclphia, PA 19104, USA
obastani&sedd upenn. edu
Armando Solar-Lezama.
Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology
Cambridge, MA 02139, USA asolar@csail eeu
ABSTRACT
Real-world processes often contain intermediate state that can be modeled an extremely sparse activation tensor. In this work, we analyze the identifiability of such sparse and local latent intermediate variables, which we call motifs. We prove our Motif Identifiability Theorem, stating that under certain assumptions it is possible to precisely identify these motifs exclusively by reducing end-to-end error. Notably, we do not assume identifiability of parameters, but rather of latent intermediate representation output by local model, thus allowing these represen- tations to be arbitrarily complex functions of the input. Additionally, we provide the SPARLING algorithm, which uses a new kind of informational bottleneck that enforces levels of activation sparsity unachievable using other techniques. We confirm empirically that extreme sparsity is necessary to achieve good interme- diate state modeling. On synthetic domains, are able precisely localize the intermediate states up to feature permutation with > 90% accuracy, even though we only train end-to-end.
INTRODUCTION
A hallmark of deep learning is its ability to learn useful intermediate representations of data from end-to-end supervision backpropagation. However, these representations are often opaque- values in the intermediate vectors generally do not map to meaningful concepts. As consequence, there has been great deal of recent interest in concept bottleneck models (Koh al., 2020), which guide models towards meaningful concepts at intermediate layers. Training these models either relies on supervision of the intermediate concepts or on designing algorithms capable learning these from end-to-end supervision. The latter is desirable since supervision of concepts is only possible in domains where they are known a priori, yet a key advantage of deep learning is the capability of learning representations beyond handcrafted knowledge. However, learning concepts end-to-end daunting task there huge space of possible concepts that could produce the same labeled input/output mapping.

---

X
M
Y
g
h
a)
072634
h*
b)
(RNA sequence)
g
(protein binding sites)
(splice sites)
Included training
included training
Included training
Figure 1: (a) Example of the DIGITCIRCLE domain, alongside (b) cartoon of the splicing problem. The input is mapped by the ground truth g" function to the motif map m of the positions of every digit/protein binding sites, which itself mapped by the ground truth h* function the output y". the sequence 072 63 634/plice sites. Only and y² are available during training; the goal is to reconstruct 9" and h* Note that splicing, unlike DIGITCIRCLE, the motifs can overlap. The M dots indicate the representation as described in Section 2 which is one-hot encoding at each location.
recent work in genomics (Gupta et al., 2024) has demonstrated a capability similar to the one de- scribed above: it learns "motifs" (intermediate spatial concepts) from end-to-end training signal. In that work, the motifs represented locations in RNA where proteins bind, which can be assumed to be sparse, and for which there are some approximate models. The paper did extensive validation to demonstrate that end-to-end training led to more accurate motif models. This result was surprising because even though additional direct information about the motif's was provided, the prediction of these motifs became more aligned with independent experimental measurements. Inspired by this result, we analyze some theoretical questions raised by this approach: mainly under what conditions is it possible faithfully recover an intermediate signal. We find that we can dispense with the approximate starting motif models altogether by making stronger assumptions about the domain.
In this work we advance theorem demonstrating that, assuming that true process contains inter- mediate concepts which are sparse, local, and necessary and sufficient to produce the output, it is possible to learn them end-to-end (up to simple transformations) without intermediate supervision. Notably, apart from the model before the sparse intermediate layer being local, we do not impose any constraints model architecture
To learn the "correct" spatial concepts, or motifs, we need to impose significant structure on the intermediate representation. As example, consider the DIGITCIRCIF task shown in the top of Figure 1 In this task, the input is a noisy image of a circle of digits, and the label is a list of the digits read counterclockwise starting from the smallest. Then, motif is digit in the input image x, and the intermediate representation (which we call the motif space) consists of binary indicator j 1} for each position (i,j) and digit e indicating whether occurs at (i,j) in x. These kinds of problems also arise in other domains for another example (illustrated at the bottom of Figure D predicting where RNA is spliced is a key problem genomics, and splice sites can be predicted based on which proteins bind to the RNA sequence; binding sites can be determined based on local RNA sequence motifs (Gupta et al., 2024). In the context these examples, we wish characterize set of assumptions under which it is statistically possible to learn motifs from end-to-end supervision alone. To demonstrate that these assumptions are satisfiable, we advance an algorithm that is capable of identifying motif's from end-to-end supervision.
As such, do not claim identifiahility of parameters, but rather of the function, as described by in- put/output behavior.
2

---

Our key insight is that spatial concepts typically have two key properties: (i) locality- ie.., motif m[i. c| only depends on the imago in a window around the corresponding spatial position ælt. j], and (ii) pparsity-ie... since there are far fewer spatial concepts than pixels, only tiny fraction of motif activations are nonzero. We prove a theoretical result that locality and sparsity, together with several reasonable assumptions about the distribution of training examples, suffices to recover the motif space from a statistical standpoint. This result requires that we can find the true optimizer of the loss; thus, we additionally provide an algorithm for optimizing end-to-end error in sparse model, and demonstrate its efficacy on three datasets.
Contributions. We present three main contributions in this paper. First, we provide a proof of our Motif Identifiability Theorem: that sparse local latent variables are identifiable. We attempt to make as few assumptions as possible about the structure of the relationships between the inputs, motifs, and output, assuming only that the motif patches are separated and are both necessary and sufficient to computing the output. We do not make any further assumptions regarding the structure of the functions relating the motif's and the output (e.g., limited number of layers). Second, we describe the SPARLING algorithm, which allows for training models with an extreme sparsity constraint. We accomplish this via layer that sets activations below some threshold equal to zero; this threshold is iteratively updated to achieve a target sparsity level (e.g., 99%). In order to address the unstable optimization landscape this produces high sparsity values, our optimization algorithm anneals the target sparsity over time. Finally, we demonstrate several domains in which SPARLING can correctly identify the intermediate latent variable. These domains, while synthetic. demonstrate that the identifiability guarantee we proved is achievable in practice.
Related work. We summarize the related work here, and provide more detailed discussion in Appendix[A] A Most existing work on learning interpretable latent representations assume some prior knowledge about the representations, including both concept bottleneck models and the Genomies work mentioned above. The recently proposed "Language in a Bottle" technique (Yang et al.) 2023) proposes to address this problem by using large language models (L.LMs) to identify intermediate concepts; however, this is only applicable to certain domains. Our theoretical work is connected to the statistical literature on identifiability, which asks whether the "true" parameters of a model can be recovered from data. Indeed, prior work has proposed algorithms that guarantee identification of latent variable models such as Hidden Markov Models (HMMs) (Yoon) 2009) and Probabilistic Context-Free Grammars (PCFGs) (Hsu et al. 2012). While the problem is similar, are interested in the deep learning setting where the latent concepts form the intermediate layer between two arbitrary models (presumably neural networks). Then, our theoretical results establish assumptions on the models and data distributions under which we can guarantee recovery of the "true function". This problem is similar to that of nonlinear Independent Component Analysis (ICA) (Hyvärinen) et al. 2023 Khemakhem et al. 2020), where the goal is identifying independent components mixed by some nonlinear function. However, we attempt to make much more limited assumptions of the "mixing function" and show that small end-to-end error is sufficient to imply recovery of the latent concepts. While our algorithm is not guaranteed to achieve small end-to-end error, this is a useful theorem as verifying low end-to-end error is trivial given a test set. Additionally, we find that in our experiments we do achieve low end-to-end error.
2 PRELIMINARIES
We are interested in settings where intermediate activations represent latent variables corresponding to semantically meaningful concepts in the problem. To this end, consider the case where the ground truth is represented as a function f X - Y composed f = h²" o g" of two functions g" X + M and h" M + Y. We call the latent space M the motif space.
We consider the task of training and h to accurately model 9" and h using only end-to-end data D = [ax, (()x)= : x ~ Dx} (i.e., enforcing only that their composition f h Å accurately models fo Importantly, we assume no access to data on M (in particular, which components of
We the term "image" and assume 2 dimensions in running examples, but our results apply any tensor input, which could be ID sequence, 3D, etc.
'W do not consider noise for the purposes of this paper. The result could be modified to handle IID Bernouli noise in the emror function by replacing the end-to-end emror with end-to-end error minus irreducible error in the theorem statement.
3

---

F( c ( a (eggNNeetlloVvviiiiiivirngi (ylvight)(igh)) (1-
470841
22780
c cu U aI U () + 30 a 6 U. 7
27946
(5)5 (2)+++0 +2++ (7)a6~
Figure 2: Two examples of inputs (images), outputs (sequences in titles), and our 9 predictions for seed=1 (colored dots) for DIGITCIRCLE, LATEX-OCR, and AUDIOMIISTSSQQUNNCC For LATEX-OCR we provide the output twice, first as the sequence of commands generated by the network and second as the translation of those commands into LaTeX. We place dot for every maximal motif, colored/labeled by the channel that appears in (c.g., the Oth channel A or #00, ist is B or #01, etc.). Stars indicate sites where non-maximal motifs present well.
M are active for any particular input). Our goal is establish the conditions under which this task is possible and to present algorithm to derive and h. Specifically, focus the case where g" and ğ exhibit the properties of locality and sparsity as described below.
We assume that elements of X C Rd and M S (0, 1)dx(x) are tensors, where [d] = (1,... d). and where / = [Di] x [i] is set of spatial indices, is the number of input channels, and is the number of kinds of motif. addition, Y is a discrete label space. M acts as one-hot encoding for each motif, depicted in Figure with the last channel corresponding In the formulation for our theorem, M is assumed to be binary, but of course during training. M can be treated as a real-valued tensor; additional step should be added to discretize it to binary for the purposes of applying the theorem.
which motif is which.
As running example, consider the DIGITCIRCLE task in Figure 1 where the input monochrome image and the label Y is a sequence of digits in that image from left to right. In this example, Dj D2 the height and width of the image, and d = 1 for the monochrome channel. The motif embedding spgtts--peiffiallyy. m c] encodes whether digit e occurs position (i. Then, g predicts whether digit occurs at each position in the input image, and h" extracts the digit sequence by reading off values of counterclockwise starting from the smallest digit).
X is
E M has n = 10 motif kinds, and encodes the occurrences of
In our example, it is easy to see that g" satisfies the key properties of locality (i.e., its prediction m[i, j. c) depends on the input image x window around (i, and sparsity (i.e., most of the motif values are m[i.j.c) 0). We describe these two properties in more detail below.
Locality We define the set & "local" motif models to capture convolutional models. SPAR- LING relies on locality to treat different parts of the input as independent, alongside the MOTIF- SUFFICIENCY assumption we introduce later.
Formally, define the set G relative to some convolutional radius N. We then say g G cor- + {0, 1)"h such that g(x) c| (z) where p(i) = {i r... + r) X ... x (i r .... + r) is the local of indices. We also de- fine the "motif cell" (i) being twice as wide {i 2r... 2r x (i 2r i i 2r}: corresponding to all locations whose motif footprints overlap the motif at Ipal (p)(1) is
responds to a local version gi E R (22++1)x..xx22-1111"))))).
'Note: while we define this concept relative to convolution, it can he generalized to graph convolutions other local operations, just with different definitions of p and pa
4

---

constant in i and appears in error bound. In the of DIGITCIRCLE, we have that 8 so p(i) = {i 8... + 8)2 and p2(i) = {i 16... 16}2.
Sparsity Let the number of motifs for channel in given activation pattern
g be
#m(m) = Eier [(m[i,c) cj * 0). We can then define the mean value of this over the dataset for a given motif function as # = E ( Let ((t) - #o(t) for both elements M and G. Let the density of a model be ö(g) = # (g) [l x [n]] and 0" 8(g"). We refer to 1 - 0(g) as the sparsity of g. In general, in our experiments 6" tends to be extremely low: for example, since we an average of 4.5 digits for the DIGITCIRCLE domain, and the images are 100x 100, and there are 10 kinds of motif, so 8 4.5 x 10-5 fraction of motifs are nonzero.
3 MOTIF IDENTIFIABILITY THEOREM
The claim that one can reconstruct an intermediate variable from end-to-end data sounds impossible, there several ways that such intermediate variable be non-unique. In this section, establish a small set of conditions under which such an intermediate variable might be non-unique and demonstrate that if none of these conditions apply, motifs can be identified.
3.1 THEOREM STATEMENT
We define Motif Identifiability as property of data distribution Dx and mechanism g" hh, In- tuitively, it says that for any estimate f h of f*, if ff has low end-to-end enror, then must have low motif error (i.e., 9 is a good estimate of g²). In other words, if we are able to learn a model conclude that we have correctly
on (I, y") data that achieves good end-to-end error, then estimated mn even we do not have any data on m". Formally, in Section 3.3 we define three MOTIE-SUFFICIENCY and a-MOTIF- MOOTIF-NECCSSSTYY such that if #max Ip2ln
properties:
all these properties hold, then for some k = 0
we have
#-
Vg g, h ..))())) = 8" + (ve> (ve>0,E(h 0, g) <E = (g) < ke
where E is end-to-end error and Em is motif error, defined Section 3.2 Note that 8(ğ) 8* enforces extreme sparsity, as 8 is must be extremely small due to OVERLAPPING and thus corresponds to an extreme sparsity setting; practice 8* is found via the adaptive sparsity algorithm, as described in Section
For simplicity, we describe our error metrics and assumptions if 1, that is, there is only one kind of motif. We provide multi-kind versions of these formally in Appendix C
3.2
ERROR METRICS
We define error metrics for both end-to-end error and motif error in two ways: a mathematically simple definition for our proof, and more intuitive definition for our empirical findings (see Sec- tion 5.1). We demonstrate that these are equivalent modulo a constant factor in Appendix For our proofs, we define and-to-end етrоr as exact match: E(f) =
+ y*).
Defining the motif eтror metric, Em. is more complex. In particular, the definition of equivalence needs to account for 9 placing the motifs at slightly different locations, or permuting the motif channels. Thus, we only check that the predicted point be within the motif cell of given true motif. In this section, we assume there is only one channel, so there is no channel permutation problem, but Appendix [ we handle channel permutations by taking minimum over all possibilities.
For our proofs, we define motif enror using intersection-over-uu prerrettionovve-uuiior-nnneeeeee metric. For the "intersection" this metric we use the number of true motif cells g (x) covered by a unique motif in f(x). To define this, we first define the function v (i) to be the number of motifs in the motif cell surrounding in in = ((x):
(i)
Σ ('P2(()
1(M # 0))
5

---

We then define u("(=), (x)) be the number of motif cells in the true motif pattern g (*)) that are covered by exactly one motif in the predicted motif pattern ((z).
m m3)
- (mm # OA u. (1) = =1
RE!
In Figure 2 we represent these with circles (stars correspond to ones with more than one match).
We then take the expectation of u over the dataset to get our "intersection" value. For our "union" value, we take the maximum of the expected number of motifs produced by g and g: max(#(0) (g), #(g)). The result is our metric
mm (g) ENND EV (((()).
max m ((),
This metric is directionally correct under all circumstances, rewarding 9 that produce motifs that overlap cells of g" with a lower erro
3.3 ASSUMPTIONS
We have three main assumptions, under which our theorem applies. Formal versions of these as- sumptions can be found in Appendix
TON-OVERLAPPING This assumption states that motifs cannot appear too near each other; specifically that any motifs' p2( cannot overlap. This assumption, written, technically excludes some of our domains hecause of how large the distance needs to be between two motifs be met. This assumption is kept strong for simplicity, and can probably be weakened in future work if an additional assumption stating that motifs' pattern must use portions of the entire p(i) cell is added.
for this condition
MOTIF- SUFFICIEECY We wish to ensure that the motifs are sufficient to predict the output, that is, there is information necessary predict the output not captured by the locations of the motifs. To accomplish this, we assert that the specific pixels representing motifs must be independent of Pm.(m, the overall structure of the input (i.e., the spatial positioning of each motif relative to the overall image and other motifs). In the context of DIGITCIRCLE, this assumption corresponds to the fact that the patterns for each digit generated independently of the procedure which decides which digit goes where. There is an additional assumption that the background (the part of the image that does not correspond motif's) must be translation invariant, property that is satisfied by independent-and and-ieetiiai-i--iiiilalaal distributed noise, but also by selections of clips from a larger object.
This is our main assumption, which pairs with sparsity and locality to mean that motifs are indepen- dent entities rather than simply correlated subfeatures of some larger picture.
This assumption states that kind of motif is entirely ignored (or treated as interchangeable with another kind) by the true h² model. The assumption is designed to be very weak, allowing for variety of cases in which individual motifs being perturbed does not alter the output; but does require that in cases whose probabilities sum to 2 a, motif pattern my must be perturbable by deleting altering single motif into plausible pattern my such that these have different outputs h" (m1) * h² (m2).
--MOTIF-NECESSHE
This assumption is generally easy to satisfy with > 0.1 in domains like DIGITCIRCLE AU- DIOMNIST SQQUENCE where perturbations that alter the output are common and objects with more degrees of freedom (e.g., more digits) have correspondingly lower probability. See Appendix B.3 for practical example of how to bound a.
'If' we assume KON-OVERLAPPIN we also have E(g") =0
6 At present, we assume no overlap at all between (2(1) cells because we must allow the possibility that a motif is perfectly predictable from, e.g., one pixel in the top left corner of p(i). which is. in nearly every realistic case, impossible. We leave formalizing more complex version of the assumption that excludes this possibility while allowing overlapping cells to future work.

---

3.4 PROOF SKETCH
We give a proof of this theorem in Appendix E In short, we proceed by contrapositive, assuming high motif error. We then establish via counting argument that since 8(g) d", any motif error must either be due to false negatives or confusion (a channel of 9 being used for two different motifs). In both cases, we then establish that this error must apply to some fraction of all motif sites (via MOTIF-S SUFHIIENCP))))) then establish that this should lead to perturbation described in O-MOTIF-NECESSITT with some proportional probability, and thus to end-to-end error.
METHODS
SPARLING trains models with Spatial Sparsity Layers using the Adaptive Sparsity Algorithm. A python implementation of SPARLING available in the esparling pypi package.
Spatial Sparsity Layer This layer is the last step in the computation of and enforces its sparsity. We define a spatial sparsity layer to layer with parameter whose forward pass is computed
Sparse, (z) = ReLU(2 -)
Importantly, is treated as a constant in backpropagation and thus not updated by gradient descent. Instead, we update t using exponential moving average of the quantiles of batches
ln pln-1 + (1 p))(zn: 5),
where in is the value of L on the nith iteration, En is the nth batch of inputs to this layer, / is the momentum (we use y 0.9), the target density, and q Bxd, x...xdxxx xn R -> Th
is the standard torch. quantile function. q is applied across all dimensions except the last: produces a value for each channel that represents the threshold u for which the proportion of elements above in the tensor at that channel is 8. We describe alternative in Appendix K.3 Since fn is fit to the data distribution, we can treat this as a layer that enforces that Ò has sparsity of 1-6. 5. Finally, always include affine batch normalization before this layer to increase training stability. We provide an analysis on the necessity of this addition in Appendix K.2
Algorithm 1 Train Loop (f, D. M, B, dr, Supdate)
To - for t = 10... do TRAINSTEP(F, D.ffDDDD DBBB((+)) Ti Ti-1 Bdr if Bt mod M = 0 then
A, + VAL.DATE(f}
if A, > T₁ then
(f. t (f.5 x Supdate; Ac)
Adaptive Sparsity Algorithm We found that applying an extreme sparsity requirement (very low 6) upon initial training of the network leads to the network getting stuck in local minimum due to a lack of learning signal. To resolve this, we use a technique inspired by simulated annealing and reduce slowly over time. Annealing hyperparameters is known technique (Sønderby et al., (016)), but we tie this annealing to end-to-end validation accuracy in order to be flexible to training schedule. As shown in Algorithm[1 [ we add step to our training loop that checks validation accuracy A, and reduces the density whenever it exceeds a target Te, reducing T3 over time. Our experiments use evaluation frequency M 2 109, batch size B = 10, dy 10-7, and δupdate 0.75.
5
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
We describe our three new domains below. See Fig examples of each domain.
For numerical stability, we accumulate hatches such that |z, 18 2 10C before running this update

---

DIGITCIRCLE domain. The input is 100 x 100 monochrome image with 3-6 unique digits placed in a rough circular pattern, with some noise being applied to the image both before and after the numbers are placed. The output y is the sequence of digits in counterclockwise order, starting with the smallest number. The latent motifs layer m is the position of each digit: which can be represented 100 x 100 x 10 tensor with 3-6 nonzero entries. Note that we have no access during training and validation to the concept of a digit as an image, nor to the concept of digit's position.
LATEX-OCR domain. As a more realistic test, we take inspiration from Deng et al. (2016) and present the task of synthesizing LATEX code from images. This task is an OCR task like DIGIT- CIRCLE, but with variation in digit rendering (size, aliasing) and a more complex h"
AUDIOMNIIST domain. In this domain, we synthesize short clips of audio representing sequences of 5-10 digits over a bed of noise. The task is predict the sequence of characters spoken. Here, we test if motif models can generalize: we train and validate with AUDIOMNIST (Becker al. 2018)) samples from Speakers 1-51 and test with samples from Speakers 52-60.
Splicing domain. We also considered the splicing domain discussed in Gupta et al. 2024). Since does not satisfy our assumptions from Section 3.3 SPARLING is not able to precisely identify the motifs, but does perform substantially better than random chance. See Appendix M for our results.
Architecture and training. Our neural architecture is adapted from that of Deng et al. (2066). For DIGITCIRCLE, we make 9 have a 17 x 17 overall window, by layering four residual units (He et al. 2016), each containing two 3 x 3 convolutional layers. We then map to a 10-channel bottleneck where our Spatial Sparsity layer is placed. Our architecture is a max pooling, followed by similar architecture to Deng al. (2016). We keep the LSTM row-encoder, but replace the attention decoder with a column-based positional encoding followed by Transformer (Vaswani al., 2017) whose encoder and decoder have 8 heads and 6 layers. Throughout, except in the bottleneck layer, we use width of 512 for all units. For LATEX-OCR we use the same architecture but with 32 motif's (to account for the additional characters) and a 65 65 overall window (to account for the larger characters, though we find 33 x 33 does not change the results substantially). For AUDIOMNISTSEQQENCE we process the audio via a spectrogram with a sample rate of 8000 and 64 channels, use a 33-wide ID resnet stack for 9 and a transformer for h. We generate training, validation, and test sets randomly. For efficiency, LATEX-OCR is looped on 10 training samples, the rest are infinite. We use a batch size of 10 and learning rate of 10-5, Our validation and test sets both contain 10s examples. Details on computational usage in Appendix N.2
Error Metrics For our empirical analysis, we use more granular error metrics. First, we define end-to-end error normalized edit distance:
EDITDISTANCE(Y".FO max((1"),1ffxx))
E2EE(f) = =ZD
Next, we disaggregate motif error's false positives, false negatives, and mis-identified motifs into three separate metrics: False Positive (FPE), False Negative (FNE), and Confusion Error (CE) (con- fusion eтror occurs when multiple motif channels are confused, this is always zero if n = 1). Ap- pendix contaans formal definitions of these metries and Appendix [ contaiins proof that these metrics are bounded within a constant multiplicative factor of Em
5.2 RESULTS
Motif error. We show our three metrics of motif error Figure for each of our models on each do- main. Motif errors for our model average below 10% for all our domains, except in the case of FNE on LATEX-OCR. The generally low motif errors, despite only training and validating end-to-end, demonstrate that our algorithm achieves Motif Identifiability on all three domains. This property even holds when generalizing to unseen samples in the AUDIOMNISTSEQUENC viding evidence that SPARLING is genuinely learning the motif features rather than memorizing. The one case where our model has high error, FNE LATEX-OCR, demonstrates the importance of the G-MOTTI- require identification of fraction bars or all of () +. For more details, see Figure and Appendix HH Interestingly, this only affects the unimportant digits; this is because our proof is still mostly valid if some motifs are never used: they can simply be treated part of the background instead.
experiment, pro-
assumption: recognizing LATEX text in the space we generated does not
8

---

DigitCircle
AudioM VISTSequence
IPE
IPC
Menil cinen (5]
HPC
Mooil Caner
7.5
5.0
0.0
Figure 3: Motif Error, across three different metrics. Bar height depicts the mean across seeds, in- dividual dots represent seed, the emoo har represents 95% bootstrap CI. AUDIOMNISSSSQUUNN has an FPE of exactly O. High FNE on LATEX-OCH is due to fraction bars, parentheses, and plus signs not being recognized in all cases since it is possible infer the output without access to these. For comparison of our technique to less-sparse models, see Figure
DigitCircle
AudioMNiSTSequeenc
FPE
FPE
8C-
FPE
60
40
E2EE
Emist
20
E26E
End
E288
0.09
0.05
0.03
0.02
0.009
0.005
0.02
0.009
0.005
0.003
0.002
0,0009
2
0.7
0.4
0.2
0.1
618)]
6[8]
[%)
Figure 4: Motif and end-to-end error metrics versus 5. Note that the x axis is a reversed log-scale, since the adaptive sparsity algorithm starts with high density and narrows exponentially.
Examples. Figure Shows shows few examples for one of our models' intermediate layers. As can be seen, all digits are appropriately identified by our intermediate layer, with very few dots (in these examples, none) falling away from digit. Note that the activations are consistent from sample to sample-for example, in DIGITCIRCLE, motif C is used for digit 6 in both images.
Necessity Extreme Sparsity Figure shows our error metrics plotted against the sparsity, with the zzaxis reversed to show progression in training time we anneal 5. As expected, as 5 decreases, FPE decreases and FNE increases. More interestingly, we note a trade-off between E2EE and CE: as 5 decreases, E2EE increases and CE decreases substantially. This demonstrates trade-off between a more accurate overall model, which benefits from greater information present and a more accurate motif model, which benefits from tighter entropy bound. Furthermore, CE is often substantially higher for even a 2-3x increase in 8, demonstrating the need for extreme sparsity. This validates the Motif Identification Theorem, which relies on d(ğ) 8" make its guarantees.
End-to-Bal End error As seen in Figure 5 SPARLING tends to produce higher end-to-end errors than baseline Non-Sparse model. We hypothesize that this is because our constraint on the information flow requires the model to "commit" to choice on whether or not given site is true motif. To
DigitCirele
La OCR
AudioM\ISTSequence
Crcer (50)
0.5
Sparking
Nonrsparse
Retrained
Sporling
Non-Sperzee
Retrained
Sparling
Non-Sparse
Retreined
Figure 5: Retrained tends perform as well as slightly worse than Non Non-Sparse, making up most of the gap from SPARLING. The apparent improvement from Non-Sparse to Retrained should not be interpreted as real, the numerical difference is tiny and the sample accuracies overlap.

---

test our hypothesis, we investigate the Retrained setting, in which remove the bottleneck, freeze the motif model 9. and finetune à on the training set until convergence. The Retrained setting tends to perform similarly to the Non-Sparse setting, demonstrating that the motif model itself provides signal sufficient to learn the end-to-end function, and validating our hypothesis.
Original
Perturbed
O
g
h Marae""0) (2)) ((Niggi))))
h (racc("(0))eeerr(( htrp://weeeetttt.oooooaetatretee (e)tiitt
Figure 6: Illustration of h model via perturbations of the motif layer. Left column the original, unmodified flow, with the 3 motifs annotated. On the right, we have the flow when perturbing these motifs; altering each site to 2, a, and respectively. We then run on the altered motif layer, and note that each the corresponding positions in the output has been modified.
Sensitivity of h model to perturbations To demonstrate that the model is behaving as expected, we look at the motifs layer of an example from LATEX-OCR and manipulate so that certain motifs are changed to others. We then run and ensure that the output is altered as expected. Figure depicts a version of this experiment. As more quantified notion of how "accurate" is, we perform the following experiment: take 1000 dataset examples where f correctly predicts the output (this done for simplicity so perturbations can be considered with respect to the true output), then take all motifs of given type (e.g., 3 in the example in Figure 6 and perturb them to some other value 8 Then we check to see if the output changed as expected, using exact match accuracy I on AUDIOMMISSTSRQURNNC, demonstrating that is mostly behaving expected
We observe accuracies of 99.3% on DIGITCIRCLE, 86.1 % on LATEX-OCR and 93.4%
CONCLUSION
We prove that Motif Identification is solvable under certain assumptions. Additionally, we demon- strate SPARLING, a practical algorithm to learn end-to-end models that have sparse intermediate layer. Finally, we demonstrate that Motif Identifiability is not solely theoretical: SPARLING achieves interpretable and accurate motifs with zero direct supervision on the motifs across three domains.
"We take change a digit that already exists (since all training examples exclusively contain unique digits), in LATEX- OCR we always start and end with digit or letter.
the resulting motif pattern remains in-distribution: in DIGITCIRCLE
Foor LATEX- OCRX and AUDIOMNISTSEQQUENC this is find and replace; for DIGITCIRCLE may also need reorder the output canonicalize it.
LLATEX- -OCR probably has higher failure rates because the "wrong subchannel" is being selected, since in the LaTeX domain allow multiple channels map each motif. this means for some characters the small and large forms heing identified distinctly
10

---

ACKNOWLEDGMENTS
We thank Christopher B. Burge, Phillip A. Sharp, Chenxi Yang, and Kayla McCue for helpful discussions. This work was supported by NSF grant numbers CCF-1918839 (to A. S-L.) and CCF- 1917852 (to O.B.).
REFERENCES
Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 15:15516-,5528, 2022.
Yusuf Aytar, Lluis Castrejon, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Cross-modal scene networks. IEEE transactions on pattern analysis and machine intelligence, 40(10))2303- 2314, 2017.
Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Interpreting and explaining deep neural networks for classification of audio signals. CoRR, abs/1807.03411 2018.
Paschalis Bizopoulos and Dimitrios Koutsouris. Sparsely activated networks. IEEE Transactions on Newral Networks and Learning Systems, 32(3): 1304-1330,,3,,, 2020.
Joachim Bona-Pellissiee François Bachoc, and François Malgouyres. Parameter identifiability of deep feedforward relu neural network. Machine Learning, 1
2023.
Hervé Bourlard and Yves Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59(4):291-294, 1988.
Jack Brady, Roland S Zimmermann, Yash Sharma, Berhaard Schölkopf, Julius Von Kügelgen, and Wicland Brendel. Provably learning object-centric representations. International Conference on Machine Learning, pp. 5038-3062 PMLR, 2023.
Yuntian Deng, Anssi Kanervisto, and Alexander Rush. What you get what you see: visual markup decompiler. arXiv preprint 04X1:18609.04938, 10:32-37, 2016.
Guillaume Desjardins, Aaron Courville, and Yoshua Bengio. Disentangling factors of variation via generative entangling. arXiv preprint 121.55444 2012.
James Enouen and Yan Liu. Sparse interaction additive networks via feature interaction detection and sparse selection. Advances in Neural Information Processing Systems, 35:13 35:13908-1392) 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: /803.03635, 2018
Kavi Gupta, Chenxi Yang, Kayla McCue, Osbert Bastani, Phillip A Sharp, Christopher B Burge, and Armando Solar-Lezama. Improved modeling of maa-iinding protein motifs in an interpretable neural model of ma splicing. Genome Biology, 25(1):23, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer. 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2016.
D Hsu, Sham M Kakade, and Percy Liang. Identifiability and unmixing of latent parse trees.
Advances in neural information processing systems, 25, 2012.
Aapo Hyvärinen, Ilyes Khemakhem, and Hiroshi Morioka. Nonlinear independent component anal- ysis for principled disentanglement in unsupervised deep learning. Patterns, 4(10), 2023.
11

---

Kishore Jaganathan, Sofia Kyriazopoulou Panagiotopoulou, Jeremy F McRae, Siavash Fazel Dar- bandi, David Knowles, Yang I Li, Jack A Kosmicki, Juan Arbelaez, Wenwu Cui, Grace B Schwartz, et al. Predicting splicing from primary sequence with deep learning. Cell, 176(3): 535-548, 2019.
Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs, and Rajesh Ranganath. Have we learned to explain?: How interpretability methods can learn to encode predictions their inter- pretations. In International Conference on Artificial Intelligence and Statistics, pp. 1459-1467.
PMIRR 2021.
Nan Jiang, Wenge Rong, Baolin Peng, Yifan Nie, and Zhang Xiong. An empirical analysis of different sparse penalties for autoencoder in unsupervised feature learning. In 2015 international joint conference on neural networks (IJCNN) pp. 1-8. IEEE, 2015.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International conference on artificial intelli- gence and statistics, pp. 2207-2217. PMLR, 2020.
Diederik Kingma and Max Welling. Stochastic gradient vb and the variational auto-encoder. In Second International Conference on Learning Representations, ICLR, volume 19, pp. 121, 2014.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. International Conference on Machine Learning, pp. 5338-5348. PMLR, 2020.
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. Conference on Causal Learning and Reasoning, pp. 428-484. PMLR, 2022.
Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julies and Quentin Bertrand. Synergies between disentanglement and sparsity: Gen- eralization and identifiability in multi-task learning. In International Conference Machine Learning. pp. 18171-18206 PMLR, 2023.
Ismael Lemhadri, Feng Ruan, Louis Abraham, and Robert Tibshirani. Lassonet: A neural network with feature sparsity. The Journal of Machine Learning Research, 22(():5633-566 5661, 2021.
Susan E Liao, Mukund Sudarshan, and Oded Regev. Machine learning for discovery: deciphering splicing logic. bioRxiv, pp. 2022- 10, 2022.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pp. 4114 PMLR, 2019.
Rongrong Ma, Jianyu Miao, Lingfeng Niu, and Peng Zhang. Transformed 11 regularization for learning sparse deep neural networks. Neural Networks, 119:286-298, 2019. ISSN 0893-6080. doi: http://ccooooeeeoeeeeeee////333. com/ http://ceeccccraccceeee/eeee
000.0...00.............. URL https::/www.scienneeee
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. Advances in neural information processing systems, 27, 2014.
Gemma E Moran, Dhanya Sridhar, Yixin Wang, and David M Blei. Identifiable deep generative models via sparse decoding. arXiv preprint arXiv:2 /0. 10804 2021.
Andrew Ng. Cs294a lecture notes: Sparse autoencoder, Winter 2011. URL https::/veel Stanfford. edu/ cctpc/clssssssss//////// aaakaa/spaseaaae wpw.sseuutooncooee.... pd
Qihan Ren, Jiayang Gao, Wen Shen, and Quanshi Zhang. Where we have anrived in proving the emergence of sparse interaction primitives in dnns. In The Twelfth International Conference on Learning Representations, 2024.
12

---

Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse reg- ularization for deep neural networks. Neurocomputiigg. 241:81-89 89, 2017. ISSN 0925-2312. doi: http:://ch.oo/////.ooo//ooomooot
URL ORLLhttee sciencedirectt. https://ww..ciencccrreet.
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint arX1v:1611.07450, 2016.
Sofia Serrano and Noah Smith. Is attention interpretable? arXiv preprint arXiv: 2019.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034. 2013.
Casper Kaae Sønderby, Tapani Raiko, Lars Maalee, Søren Kaac Sønderby. and Ole Winther. Ladder variational autoencoders. Advances neural information processing systems, 29, 2016.
Yiyang Sun, Zhi Chen, Vittorio Orlandi, Tong Wang, and Cynthia Rudin. Sparse and faithful expla- nations without sparse models. arXiv preprint arXiv:2402.09702 2024.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Sociery: Series B (Methodological), 58(1):267-225) 288, 1996.
Alex M Tseng, Gökcen Eraslan, Nathaniel Lee Diamant, Tommaso Biancalani, and Gabriele Scalia. A mechanistically interpretable neural-network architecture for discovery of regulatory genomics. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL htttps::/prooeeediing htwp://eeeedinn....eieieeeeeeeee www.ps....r..........
pdf
Danru Xu, Dingling Yao, Sébastien Lachapelle, Perouz Taslakian, Julius von Küigelgen, Francesco Locatello, and Sara Magliacane. sparsity principle for partially observable causal representation learning, arXiv preprint arXiv:2403.08335, 2024.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable im- age classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19187-19197, 2023.
Byung-Jun Yoon. Hidden markov models and their applications in biological sequence analysis. Current genomics, 402-)14 2009.
Mateo Espinosa Zarlenga, Pictro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Gian- nini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. arXiv preprint arXiv:2209 09056, 2022.
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. Advances in neural information processing systems, 35:16411-164221 2022.
Kai Zhong, Zhao Song, Prateek Jain, Peter L Bartlett, and Inderjit Dhillon. Recovery guaran- tees for one-hidden-layer neural networks. In International conference on machine learning, pp- 4140-4149. PMLR, 2017.
13

---

A ADDITIONAL RELATED WORK
Learning RNA/DNA motifs In Gupta et al. (024) the authors introduce the concept of Sparse Adjusted Motifs. Specifically, they model the problem of splieing as a two stage process, in which first proteins bind the RNA sequence, and then cause the sequence to be spliced at certain points. Using end-to-end data of a sequence annotated with splicepoints, as well as baseline models of protein binding patterns in RNA, they are able to improve these models of protein binding. To accomplish this they use the baseline model to predict protein binding affinity, then apply SPARLING, a sparse layer, with sparsity of 26. They then modify this with neural network trained residually, allowing it to only influence nonzero sites, then apply another SPARLING layer with sparsity 5. this work; we eschew the complexity off the Adjusted Motif model and instead consider the sparse layer by itself. In Tseng et al. (2024) and Liao et al. (2022), the authors learn motifs without intermediate supervision, but in these cases they heavily restrict the model class of the motif models, requiring them to be 1-layer convolutions.
Concept bottleneck models. Previous work also learns models with intermediate features that correspond to known variables. Some techniques, such Concept Bottleneck Models (Koh et al. 2020) and Concept Embedding Models (Zarlenga et al. (022), involve additional supervision with existing feature labels. Other techniques, such as Cross-Model Scene Networks (Aytar al.] 2017), use multiple datasets with the same intermediate representation. The Language in a Bottle technique (Yang et al., )023) uses LLMs to identify intermediate concepts; however this is only applicable to certain domains (c.g., asking an LLM to produce the protein binding motifs in an RNA sequence will result in it providing a list of motif finding tools, not motifs). In this work, we do not require the presence of additional datasets or annotations.
Identifiability The problem of identifiability. in which the behavior of some component of function is inferred via the behavior of the overall function, under some assumptions, is typically set up as an attempt to infer the values of specific parameters up to some isomorphism. In |Hsu et al. (2012), the parameters are those of PCFG expressing a distribution over sequences and the behavior of the function is the computation of a moment of this distribution (with infinite data). In Bona- Pellissier al. (2033). the parameters are those of multi layer ReLU network, identifiability is established with infinite data under several assumptions relating to the network as a piecewise linear the
function. Other work such as Zhong al. (2017) focuses on strong convexity guarantees neighborhood of the true parameters, which is a far stronger claim as it leads to plausible inference algorithms; though the model class is restricted to 1 layer neural networks. In AAhjaa et al. (2022), the result of sparse perturbations (perturbations of only some variables) to the latent variables is given, which enables identifiability; this differs from our D-MOTTF-NECESSSE assume the existence of perturbations that affect the observable output, rather than requiring access to these modified outputs as part of the dataset. In our case, we are attempting to infer the motif function I rather than any particular parameter, also up to isomorphism. As a result, we make weaker architectural assumptions about and h. However, the property we attempt to establish is stronger than identifiability with infinite data: we wish to show that the error in identification of is bounded by a multiple of the end-to-end error. While this does not immediately lead to an inference algorithm, it implies that any inference algorithm that preserves our sparsity constraint while achieving low error will be valid algorithm for identifying the true g".
in that we only
Neural Input Attribution. SPARLING is useful for identifying the relevant parts of an input. One existing technique that accomplishes this goal is saliency mapping (Simonyan et al. 2013 Selvaraju et al. 2016), which uses gradient techniques to find which parts of the input affect the output most. Another technique, analyzing the attention weights of an attention layer (Mnih et al. 2014). only works with single layer of attention and does not necessarily produce valid or complete explana- tions (Serrano& & Smith 2019). Additionally, Amortized Explanation Techniques produce a subset of features that form a "local explanation," i.e., features sufficient to produce prediction (Jethani cet al., (022). The main benefit a sparse annotation provides over these techniques is unconditional independence: when using sparsity, you have the ability to make the claim "region x/r] of the input is not relevant to the output prediction, regardless of the rest of the input x[F]". This is a direct attention techniques which
result of sparsity and locality and is unavailable when using saliency inherently condition on the values you provide for x (F]. Techniques such as Sparse Explanation Values (Sun et al. 2024) do not have this guarantee and so while they apply to a wider variety of
14

---

model structures, they can thus only reason about local perturbations, providing local explanations of changes in behavior.
Disentangled representations, Disentangled representations ones where different components of the representation encode independent attributes of the underlying data (Desjardins et al., 2012 Higgins al. 2016). Locatello et al. (2119) suggests there universal solutions to this problem, and all attempts require some prior about the kinds of representations being disentangled. We focus here on prior regarding sparsity and locality.
Informational bottleneck. Other work also constrains the information content of the interme- diate representation in neural network. Strategies include constraining the dimension of the representation-ex. e.g. PCA and autoencoders with low-dimensional representations (Bourlard & Kamp 1988), or adding noise-e.g., variational autoencoders (Kingma & Welling/ 2014). How- ever, these approaches often encourage entangling features to communicate them through smaller number of channels, and as such do not always learn interpretable representations of an intermediate state.
Sparse activations. Note that this notion of sparsity differs from sparse parameters (Tibshirani 1996 [Scardapane et al., 2017, Frankle & Carbin, 2018, Ma et al.
2019 Lemhadri et al., 2021
Lachapelle et al. 2223)) sparse causal graphs (Moran et al. 2021 |
Lachapelle et al. 2022 Enouen
& Liu 2022 Ren et al. 2024), and sparse jacobians (Zheng et al. 2022 Brady al. 2023): instead
this line of work attempts to constrain the information content of an intermediate representation by encouraging sparse activations- -.e.., each component of the representation is zero for most inputs. Sparse parameters serve different objectives and require different strategies to be used effectively. As sparse parameters only provide interpretability for single two-layer models, they are generally used for efficiency rather than interpretability in larger models. In terms of imposing sparsity, differ- ent techniques must again be used as sparse activation patterns depend on the input, so occasional pruning e.,. Frankle & Carbin 2018 is insufficient. Strategies for achieving sparse activations include imposing an L1 penalty on the representation a penalty on the KL divergence between the representation 's distribution and low-pro now-probability Bernoulli distribution (Jiang et al. 2015). How- ever, these techniques typically only achieve 50%-90% sparsity, whereas SPARLING can achieve 99.9%. We directly compare with these in Appendix K. Additionally, (Bizopoulos & Koutsouris 2020) uses a quantile-based activation limit equivalent to both our ablations (see Appendix K.2) combined, but in the simpler context of linear and 9 models. Similarly, Xu et al. (2024) provides an identifiability result given sparse activations, but in the context of affine model, whereas allow arbitrary nonlinearity.
B FORMAL ASSUMPTIONS
We assume our data generation process is represented by graphical model x m² y": intu- itively, the motifs m are sampled first, and then x and y" are sampled conditioned on m². This allows us describe our assumptions as constraints on P(x|mm)) and (m").
There are several ways for this graphical model to be non-unique given the joint distribution p(x", , y²); we identify set of assumptions that excludes any possible such non-uniqueness. To ensure the pixels (or audio samples, nodes, etc.) corresponding to each motif are not just part of some larger image, we assume motifss cannot appear near each other OOO--OVELAAPPINC) and P(x|m²) must be easily decomposed into factors in order to constrain the relationship between and m ensuring that z a product of distributions describing the footprints of motifs (MOTIF- SUFFICIENCY). This is our main assumption, analogous to Markovian assumption in a Hidden Markov Model. Next, we exclude the possibility that some motifs are always redundant: a-MOTIF- NECESSITY describes the relationship between m² and y", asserting that all motifs are important in some cases; in other words, h² cannot systematically ignore any motif or treat any two motifs as interchangeable. This assumption ensures the definition of "motif" is restricted to concepts that are possible to learn from end-to-end data, analogous to a full-rank covariance assumption in Linear Regression.
While these constraints may appear strict, they fit problems where g" identifies small local patterns in the i.g. motifs such as the individual digits in DIGITCIRCLE that are all used least
15

---

sometimes by h². However, they do not fit the splicing domain (primarily NON-OVERLAPPING and MOTII=-UFFICIENCYY, necessitating the additional data used by Gupta et al. (2024).
B.I NON-OVERLAAPING
We assume that motif cells cannot overlap in samples drawn from Dx VI X, (x) > 0 Vi, g ( 00 where i E gf (g #0.
B.2 MOTIF-SUFIIIENCY
We assert that probability (x(x) decomposes to independent distributions for each patch p(i) for which g [x) * 0 and background probability covering all non-patch inputs. Formally, define the probability of x given that it produces the motif pattern m as
P|alg" (x) = mm] = II P((z((())) Po (x[r(m))) J.Em
where Pj is distribution over "foreground" parts of the input (those containing motifs) and Po(a:) is distribution over "background" x, and we are taking a marginal x[[(m]], where r(m) = I x [d] \ USem p(i) the set of all indices not in any motif footprint. We then represent P[æ] EmeM P(zlg"(x) (x) mmp where Pm.. (m) is our distribution over m.
We also require that P. be translationally invariant. Specifically, for all sets L C I and all offsets o Z' such that {i + o: L.) C 7, have [z) Pa(a((i + o joint distribution should be the same at each location regardless of translation. This property holds for all datasets created by clipping random components of larger datasets, e.g., clipping sequences of RNA from the genome or snippets of text from a book. See Appendix G. for a motivating counterexample.
T.)))- That is, the
B.3 Q-MOTIT-NECESSIH
We assert that the motifs are important, that is, all of them are "used" to compute the output. Being "used" implies two properties. First, we assert that perturbations to motifs must cause h" to produce different outputs. However, this is insufficient, because it is possible for a perturbation to cause h" to produce different output, but for this perturbation to be "correctible" by some alternate h. To exclude this possibility, we add our second property, that our distribution is structured in such manner that all perturbations result in motif pattern that remains in-distribution. These two conditions are sufficient, but they are too strong to apply to most problems of interest, so we relax them by requiring that they co-occur in only some fraction of motif patterns. Our assumption is parameterized by this o motif importance with a higher a implies that motif errors will lead to end-to-end errors on higher fraction of the input.
We begin by defining a perturbation function R(m1) that relates a motif m. to set m2 E R(m1) which corresponds to mi with a motif deleted. We then define (((m)) {m2 R(mi) h" (mı) * h" (m²)); that is the subset of A(m,) such that the output is changed. The idea here that the model not accurately reporting given motif means that it cannot distinguish the condi- tions that lead to mi and m2 from each other, but these have different outputs, so it must be incorrect at least one these cases.
Ideally, we might want R' to map every mi to some unique m2 with identical probability. If this is the case, we can guarantee that the model is incorrect either on mi or m2 and therefore is incorrect in 50% of cases. However, in practice, this is far too strong assumption. A given motif pattern might have no motifs to delete, or multiple possible delctions, and additionally, many m, might lead to the same m2 post perturbation.
As such, we define lem: let nodes ( and (Bm hmEM, where Am has input flow P. (m) and Bn, has output flow P.,.mm (m). Add unconstrained connections between Am, and Bm, if and only if m2 E R' (m1)) Our assumption is that the max flow this graph is least
NECESSIIT as an assumption over the following bipartite flow prob-
16

---

As intuition, the flow this graph represents how much weight put each perturbation, allowing us to ensure that we always start and end at a high-probability section of the distribution, with the output constraints preventing from "overusing" any particular m2.
In practice, for both DIGITCIRCLE and AUDIOMINISTSEQUEENCE, we can prove a-MOTIF- NECESSITY for high (above 0.25). This works because while several possible deletions dif- ferent m, can lead to the same M2, the my each have more degrees of freedom and thus lower probability.
Consider simplified version of DIGITCIRCLE where there are 3-6 identical digits randomly placed in line rather than circle, with 10 possible positions but two digits can be directly adjacent. We have for an with 5 digits
P(m.) 1
[P digits)]
4
1
(counted via enumeration]
21
so P(m1) 84
whereas for m2 with digits
P(m))
[P(4 digits)]
4
[counted via enumeration
70
so Pmm1) 280 and thus P(m1) = P(mz). We end up being unable to send the full flow, with each my corresponding between 1 to my elements, and in fact are only able send 0.196 units of flow, out of the available 0.25. Repeating this for m, with 4 and 6 digits (we cannot delete digits when starting with 3), we end up with an overall a 0.465.
C MULTIPLE MOTIF KINDS
We have to modify several definitions to handle the case of multiple channels. However, none of these changes modify the fundamental character of the theorem. Our provided proof (Appendix E for the more general case.
One useful definition is that of the set of true motifs: define w. (m) to be set of indices corresponding to motif of channel e: we(m) = {i E Ee, m(i,c] * 0}, we have that i m = 3e, (m))
C.I
MOTIF ERROR
Since there multiple channels, and there is no way for know priori what the appropriate assignment of motif to channel is, the predicted motifs models should be deemed equivalent to the ground truth model which is known when test-if there exists channel assignment for which they are equivalent.
We follow a similar metric the one described in Section 3.2 except channel-specifi and then minimized over all assignments 7 [r]] + [n]] of channels of to channels in g
Our definition of modified by identifying channel
Vaa (T,)))
Σ (( ('EP(())
We do not require T to be a permutation, as in practice we might want to allow extra channels in case we do not know the exact number. In this case, the metric will only provide good score if real g motif is split up among channels of but not if single channel e' of Ò corresponds to multiple channels of g which would he loss of information
17

---

We then modify to be the number of motif cells of channel in the true motif pattern ga(x) that are uniquely covered by a motif channel c'.
uim m c.
Σ 1m (. - cml,,i
iE!
We then take the sum of this over all channels c' of 9 and corresponding channels T(c') of g², and then take an expectation over the dataset to get our "intersection" value, the expected number of true motif cells covered by unique predicted motif of the corresponding channel.
EgwD Σ E (x),
Our union value is unchanged, leading to the metric:
min (1 EEND EEd u(g(x)) (0(z),g0) g (x),d, ((((()) max # (g), #*)
Em (g)
C.2 MOTIE-SUFFICIENC ASSUMPTION
We slightly modify our assumption to
(x) m]
II Pc(a(o(()) Po (=(r(m)))
cE(m).
which is identical except that Py is replaced by Pa. which is distinct for each channel.
C.3 C-MMOTI-NECESSITY ASSUMPTION
Adding more channels means we need to consider multiple perturbation functions. Specifically, consider two classes of perturbation functions R(mi) that relate pairs of motif patterns, those where E R(m1) correspond to mi with motif of a particular channel deleted, and those where my R(m1) corresponds to mi with particular motif of channel e₁ mutated into a motif of channel
One additional subtlety that we also require flexibility shifts in the perturbed motif's position, ensuring that the precise positions of motifs are not determined by the rest of the motifs, precluding situation where, e-g., motifs are aligned to grid, and the learned ğ "sneaks through" information about a motif's s channel via fff- positioning. Note that this means a-MOTIF-NN MOTIF-NECCESSIT is tied to locality and in particular only makes sense when g² is local within p2 as defined in Section
Let A 2 2r} x { 1-2r...2r
2r) be the constant set such that (2(1) i + A.
Our formal definition then is over R. set of perturbation relations defined as follows:
For all d1 E A and let there be some R E R such that m2 E R(m1) if and only if there exists some E such that my and m2 agree everywhere except that m, + di][ei] A and m2(p2
0
For all d1, d2 e A and e1 * C2 let there be some R e R such that m2 E R(m1) and only if there exists some i E such that my and m2 agree everywhere except that mi(i + d1 dr][ct] + 0 and m² + d2 [[2]] + O
Then for each R R assert the existence of some maximum flow of on the corresponding R'.
For the proofs, we conceptualize this flow as a probability distribution (mmg) mi) which corresponds to the flow from m m2 as fraction of P(m;), with (mLmmm) taking the remainder of the flow. Since it corresponds to this flow, it must satisfy the properties
(mm S Pm (m22) MR
and
Σ
(mm2/mm) Pm (m)
MM
18

---

D EVALUATION METRIC DETAILS
D.1 PRELIMINARIES
We now define FPM and MM motif sets, along with the C function.
Predicted motifs. For a given predicted motif tensor m, we define P(m) = {(t, mfi, c| > 0}
be the set of motifs predicted in ri, where i I E [n]. Typically, motifs (()()) for our estimated motif model ÿ.
interested the set of
Footprint identification. Let C 1 M + TU (1) be function that identifies the motif cell that a given index is within, or 1 otherwise:
C(i'mm") = + i' E p2(1)
By NON-OVERLAPPING this is always unique, but we can extend the definition to be coherent otherwise by giving flexibility to choose an arbitrary such i:
(C(i', m²) =i = 1' P2(1) (C(i', m²) = = Vi, @ P2())
False Positive Motifs. We now have the ability define our first class of motifs: false positive motifs. These are predicted motifs that do not correspond to any real motifs:
FPM(m, m²) ((('(c)) P(m): C(²,m") = 1}.
We denote the remaining motifs by
Pi (m, P(m) \ (PM(m,mm)
Maximal Motifs First, we need to define the set of all predicted motifs that cover the same footprint as a given predicted motif. We do so the Am,m function, which takes a given predicted motif (assumed to overlap some footprint) and returns all others covering the same footprint:
A (i'c) = ((i",))
.)) E P(m): C(i", m²) C(i', m²)}
Now we can define maximal motifs are predicted motifs that are maximal in the footprint they cover:
MM(m,, m²) {t Pi,m, m²) ml[t] L'EA,..m
m
We can also define non-maximal motif's are predicted motifs that are non-maximal in the footprint they cover:
NMM(n, m²) {t Pi m²):
m +
max m[tT}
e'Em,.m" (6)
However, ignore non-maximal motifs entirely for the purposes of analysis, under the reason- ing that these are trivially removable in practice.
D.2 MOTIF ERROR METRIC
We then define three motif error metrics that we use empirically in evaluating our learned ğ models.
First, the false positive error (FPE) is the percentage of motifs that are false positive motifs.
En-D [IFM (x))
FP)
E (()()))]
Second, the false negative error (FNE) is the percentage of true sites that are not covered by any motif.
FNEp(O) = Ex-D[H((, E ((g(())) F(1',) E P(g(x)) / p2(1))]
Finally, the confusion error (CE) is defined as follows: (i) rearrange ğ's channels to best align them with g², (ii) compute the percentage of maximal motifs in footprint of true motif that do not correspond to the true motif's channel:
min [ (x))))
CEp(g)
T:je [m]] (c)
19

---

(m,, m²) represents the motifs that do not match ground truth under rearrangement
co (m, m²) = {t E MM(n,, m²) -mat, (t, Ctt, m m)))
and mat,((, 6*) is a function that checks whether the two motif index tuples match under channel rearrangement T.
low FPE/ENE implies that the model is identifying relevant portions of the input, while a low CE implies that the model classifies these components motifs correctly.
E PROOF OF MOTIF IDENTIFIABILITY THEOREM
The following is formal proof of the Motif Identifiability Theorem. The term #max is used in this proof to denote max. #c
E.1 PROOF SKETCH
We proceed by contrapositive, starting with the assumption that (g) ke and then proving that {hoog 2 e. We first demonstrate (Lemma E.2.1 that high motif error implies either a high number of false negatives for some channel c (true motif cells that have coverage by () or simultaneously a low number of false positives and some channels C1, 02 such that there is some high number cells of both that covered by the same channel c' of g. This theorem is proven by a simple counting argument, relying only on the fact that (g)) = 8*. We then prove in each of the two resulting cases that the property holds for some fraction of motif cells in general, using VON-OVERIAN APPRIAPPING. We then apply a- MOTIF-NECESSITY to each case, demonstrating that 9 does distinguish different inputs that must lead different values of y h' (g"())) Since 9 cannot distinguish these inputs, neither can h o 9. and thus in one of the two cases error must arise. Thus, in both cases, we conclude that E(h
and
h g) 2 E.
E.2 LEMMAS
E.2.1
SOURCES OF MOTIF ERROR DICHOTOMY
First, we define a few quantities, representing the number of times a true cell is covered negatively, covered once, covered multiple times.
FN (c) = Ex,m"
Σ (1)g(o)(1) = 0)
(Ewe(m")
COy(c, d' =Ez,m-
Σ
1 (1)(a)))) = 1 (t,))
(Ewe(m²)
(M2 Ez,m"
Σ iEwe(m*)
1 (1g(a)) >
We also define the quantity
FP, = Er,m= [x)
where
F
Σ
1(0(=);,,d]=1 1)
The claim we wish to establish is
vh E M + Y, E G, 5()) = 8" A (m) 2 ke
= (c, 31)
V (Ee, C2 min((C
A))(FP, 213)
20

---

For
#*kr kc
32 2n(n 1)
and
0/32 31 4# A
contrapositive, assuming that (Ve, (N,()) (Vez, C2 c', min(CO, d)), proof relies on none of our assumptions and is just about counting the outputs of g.
We proceed by
B1) and Note that this c c') < P2) V (FP, n311) both hold.
Bounding CM and FP
First, we bound CM and FP. Specifically, we establish that
Eco((ed) = Σ Ex m
Σ Σ (1pc)))) 1A (c, =
(EWe(m²)
Σ Ex
(10)())()=
(EWe(I)))
Σ
co, +2 Σ CM₂(c) Σ Ex,,m m
+2
Σ
((Vg(o) = + 2 (1) (vo(e) (1) > >1)
(Ew)(m²)
E m
Σ (i(c)(()
Σ E m
((x)(E,c)
FP,
# FPg
FPg + Σ COg(ccc + Σ CM,(c) #*
Σ (C) + Σ Co,((0)) + Σ CM,(c) Σ Ex, m Σ
cCe
(Ewe(I))))
=#"
FNN((C) + C0,((c,d + Σ CM,(c) FPg + Σ Cog((cd) + (M,(c)
2CMM(())
FN)((0) 2 FP Σ (M2(c)
And thus we have a bound on CM and FP in terms of FN. Note that this means we can eliminate the FP, n,3 disjunction from our premises as we now know that FP, S E. (c) n81.
Low FN implies high CO
From above we have
FNNg(C) + Eco,(c,d) + Σ CM,(c)= #
0.0"
From this and the previous result it is clear that
Σ FNy(c) + Eco,(od)2 #*
We then can state
Σ C0yle.d) 2 #' -2 Σ FNg(c)
21

---

High CO implies low &m We now define the following function [n]] - [n]] assigning the
"proper channel" of given channel of 9 as
x(d)) = arg max COg(c,e')
Assume that Ven, c2, d, min (0, c)) - 32. We then have that
Ve # cOgle,d) min(C0,(c.)),
s B2
Finally, we have that
Σ C0,(c,d) n(n 1)32 + Σ (Og()(
6g
We then express
c0,(((())))
Σ Σ C0g(c,c)
c'
Σ
Ex,m
1 (1)(a)(1) = 1 1 (1,))
c c'(=(c)==c
(m
S #
# Em (g)
where the last step is viable as max #" # # as 8(g) 8* We thus have that
[c),(a S n(n 1)32 #" # E..
and therefore
E (g) S 1) + #
(cy)
c
Final proof We then add the assumption Vc, S 33. This means that
20 #* -
Putting this together with the above, we have
*mEm (0) S nnn - 1)33 2n31 S n(n - 1)82++ = (n( S 2m(n - 11332 = #*ke "e
Thus ending our proof
E.2.2 COROLLARY: MOTIF ERROR ALL POSITIONS
We define the extended footprint cell as function Φ 1 1 x| mapping a location to the set of locations in the input whose output is in (1)
(11) (1" 3i' E (2()), 1" p(())
Now, we establish that motif error in some percentage of positions implies a consistent probability of motif error every time the motif shows up, regardless of skeleton. First, define Po. to be distribution over regions of size Φ(i) defined as
Po.(n) = P[=[((1) = nlg" (x) cl # 0]
We can use MOTIF-SUFFICIENC to break this down as (letting o be the relative position of i within
7
Po.(n) = P[x[o(1)] = nlg" (x) [i, c 0
Pa(nlpptt) ol]P%(=(((()) \ ((() n((0(1) \ \p()) o]]
22

---

We implicitly use VON-OVERLAPPING when assume that ((1) p(i) is entirely over the back- ground. The specific property here is that o(1) n p(i') = for all i, 7 E m 1' + i. This follows from NON-OVERLAPPING have that
)(()) # = 3i", 1" E np('))(t)
3i","" E v(i) ni" E p(i')
3," E p2(i), Ep(i)) Ai" E
Zj. E P2(1) A AI", E
Zj.j E P2(1) A (p()) # 0
3j.j E p2(1) AJ A
pa(1) (22()') #0
Which is the exact condition given in NON-OVERLAPPING.
Note that this is no longer in any way dependent on due to the translational invariance of Po- Therefore, we have that (n.)(n) = Pe()). and this consistent at all locations that c appears, regardless of the skeleton.
We also define q 2 x A + 2Ax(n] [n] be be a function that takes a vector u and offset d and returns the map q(u, d) such that q(u, [d] A Vi' + d, q(u d) = Let Q(u) = (a(u, d) : d E A}
Claim The claim we wish to establish is
vh E M + Q, G,6(0) 8 A (g) 2 ke
Ee, P (((1) 0/1 p.. B
#max
V
301, c', min P (p(n) Q(ea)) Pol
B2 # max
cf(en)))
(FP, 2
False negative case We now start with the assumption that FN 2 B. We have that
(C = Eue,
Σ
(1)g()) = 0)
NEW, (m))
Σ Bur.
Σ
1 (()g(w)(0 0)lg"(>)) = m (m)
imw)(m²)
E Bx,m-
(1) 0)lg m] Pm
iem
P [1((ga))) (i) =]]g" (x) m Pm (m)
TEm
Σ TEm
P (((n) 0/n ~ Po (m)
(0()) = 0/1 ~ Pc] Σ Σ
Pn (m)
m
smm
= P (0(n) = 0/1
P. E
LiE
P (m(n) 0/1 P. #c
23

---

and thus we can conclude that P (((n) = 0/77 ~ Po
#max
Confusion Case
In this case, we have two properties, first that we have some e₁ and 0o such that
C0g(e,,) 2 B2 A (02 d') 2 32
and the second that
FP, S 81
First, we use similar argument to the previous case to establish that
COy(c,c)) = Ex,,-
Σ
(10((s((1) = (1))
(mw))
Eg.= Σ
1(Ug(=)(1) = 1 ( 11g"(())))))))) = m (m
iEwe(m²)
By.m+ (1) (0) 11)"(x)) m] Pm (m)
iem
P [I(r)()))) (1) us(o) 1)1g"(()) (x:) ml P... (m)
imm
((n) Q(e ~ P. (m)
iEm
= p(n) E Q((c)) ~ P. ΣΣ Pma (m)
m iEm
P (b()) e Q(ex)) Pa E
liem
= + Q(cx))n P.] Ho
and thus we can conclude that P (00)) Q(ex))l ~ Po
8
82_ f fci, C2). #mase
E.2.3 LEMMA: INDISTINGUISHABLE LOCAL-TO-GLOBAL
Statement: given pairing scheme vh, a predicate Ç : TR [x|]| TB some i. 0, and that for all (mmg//m)) > that are an i-OFT-BY-ONE PAIR and for all IR x can assume
P(((x)(m), + P(c(x))m2, xn] 2
we can prove that
1
P(((x)) 2 ax
We begin by multiplying by P[æn|m]] P[æn[m2] (these equal because my and m2 agree outside of (i))
([)m)) ((((x))ma, xnl P( 2 k P[
([(()) + P(((x), mn|ma] 2 x P(an|m)]
and integrating
/ Plc(#), xn/milda + / PI((s), 2 K / Plænlmildeg P(((x))\ml) + PI((x))ma) 2
24

---

We then multiply both sides by (m and sum:
Σ
{mmmm))Pa(())(((()))))))) P((())))) P((())(m22)
>
(mm2/114 (m1))
M
We have that
LHS = Σ
( ( (((()))ma)))
mI-m2EM
Σ (m)
Σ
MIMMEMM
MiMMEMM
Σ ((mma)m))P(((s)))
Σ {(mm))
M2EM
M2EM
2 P(((=))+
Σ P
M
- P[((x)] + P((())
P(((x)) 2 LHS
RHS
Σ
(mm2 (m)
MM
Σ {m2))
M2EM
2 aK
and therefore, we have that
P[((x)] 2
2 ka
which completes our proof.
E.2.4 LEMMA: FALSE NEGATIVES
Given some ÿ. K < 2
1 such that
P(ğ(m) 0/71
P. 2
we have that for all h.
E(h o g) 2
2 an
We now proceed with our proof. Let v be the pairing scheme corresponding to = e, and 822 = 0, and dj d2 O. Fix any mi, m2, such that (mmmg mm) being an COFF-BY-ONT PAIR and
ER R (00) We can now see that
P(f(x) + (m1m,, 2 Pif(x) v' 0/m1 øR.
-[f(()) = O. m. T (()( = 0/m ØR
PIf(x) vvl((((1)) 0. mm 2n (() 0/7 ~ P.]
2 P(f(x) # v'(((((i) = 0, m1.2R|K
Once we know FR and that (()(1)) = 0, we know that f(x) entirely dependent on øæ and not on e(p())) because we have access via g()(i)) to all values of g(x) that influenced by zfp(i)). As such, we can replace fxx) with A(xR). Thus, have
Pff(x) # y" 3m Z P[((zn) # y"
By the translational property of P, we know that P(g(m) = 0/n ~ Pa. is a definable, fixed, quantity, and applies any random variable n = =[((i))- We thus have that P(ğ(m) 0|m
25

---

Po 2-1 nó as otherwise 9 would not be capable of having density of ó on the whole image. We can safely assume nó < 2 since otherwise the NON-OVERLAPPIN would be violated, since the minimum size of motif cell is 3 (I-dimensional, radius 1). Thus we can assume that
P(ğ(m) 0/m Po)
2 2
and thus have the same property
P(f(x) + y [R] 2 P[((RR) + y 122
We have that he' (m1) + ht (m2). We can then proceed
Plf(x) # y" [R] - Plf(x) # m2,, xRl
2 <(P()(RR) * [ yy
2 x(((A(ERR) * h" (m1))
P\M(FR) + h² (m2 xR)
+ <(1(A(ER) * +
h" (m2))
2 x(I()(nn) # h² (m1) V ((xx) # h (m2))
=K
As such, we can now apply Lemma E.2.3 to get the statement
P(í(a)+v'1 2 ka
which completes our proof
E.2.5 LEMMA: CONFUSION
Given cJ, C22 and d', k. and some such that
P(ğ(n) E Qm Pa]2 P(o(12) E Q(ee))122 Pogl 2
have that for all h
[[f(x) Fッ" FV" 0j 2 aK
|
We now proceed with our proof.
We proceed in the proof of Lemma E.2.4 with few variations. Consider the pairing scheme
s corresponding to 11 Cc, and 822 Cogy and d1 arg maxa P(ğ(n) q(u, d) mh P.,]
and d2 arg max P()(12) q(u, d)l12 Poz]. We have that Plo(m) q(u, -d mm P(g(1)) - 1m Pozl x/ A]. Let x/|
Let (m1, m2) be an i-OFF-RY-ONE PAIR such that (m))mm) > O. Fix FR (x( Consider
Pif(x) + 5" V (x) > 0R] Plf(x) A y² (x) 0/m2, We have that h² (m1) + h (m2). We also know that
Pif(2) + n² v FP, (x)
2 Plf(x) + h (m,1 (F) > 0/m, ænl
We can then analyze
Plf(z) # hm(mm) V FP, (x) > Q|11Iæ] 2 Plf(x) # h2 (m1) VF, (z) > 0,9(zl0(i d:))) = ((u.0)mm1,.)
P(f(>) + h² (mi) V FP, (z) 0/m,, - di)) q(u, P)] di)
2 P(f(2) + h'' (m1) V FP g (x) > 0/m, =n. ((ii - dz))) = ()k'
where the last step comes from the fact that my has its motif at + d,, and therefore, و should activate at i. Finally, if we define A(I to be the value h(m) takes when mli'| ((x[d(i')] for all a motif cell of m² other than i and otherwise, we have that
f(x) + MxRR) = FP, 00
because if is equal to any other value, that indicates that ğ sending some values through non- motif cell channels. We thus have that
f(x) # h2 (m1) (x) 0 Mxx) + h (m11 FP) > 0
26

---

Thus, have that P[f(z) + hi (m1)) V (x) 0/11, TR] 2 K P(M(R)) + h (m1) V ( 0/m1, RRl
and by an identical argument
Pif(x) + h" (m2) (x) 0/12, 2 n (()(())) + h" (m22) V ( m2 FRl
We now proceed by cases. Either ((rn) + h (m1); which case
P[{(RR) + (m1) V (x)( > 2. =1 or fxx) = (ma) and thus A(xR) * ht" (m2) and thus
P[((ER) # h (m²) V FP FP,(x) > 0/ RRl =1
In either case, we have
P(f(x) # y FP, ( 0/m1 znl + P(f(z) # y" V FPg(2)) > 0|m2, k2
Applying Lemma E.2.3 to this statement, we get that we have
P(f(x) + FP ( 0) 2 k'a
completing our proof
E.3
MAIN PROOF
The statement is reproduced below:
9, h. 8(g) = 8" (ve 00 <6 = Em( < ke)
Vg
Let
k: 16# mww |
#*
and then fix 9 such that 8(g) = 8 and > 0. Assume towards contradiction that the statement
E(h o 9) < = Em (g) < kc is false. We then have E(h g) < and (m)g) 2 kc. Using
Corrolary E.2.2 we have two cases.
E.3.1 FALSE NEGATIVE CASE
We have that there is some for which
P P(ō(n) 0/n Po 2
31
# max
Applying Lemma E.2.4 we have that
o(h 2 a B1
a 32
a # Ke
2 #max 8:42
16 max
In(n -
max
which is a contradiction with (h o 9) <e
E.3.2 CONFUSION CASE
We have that there exist some Cı, C22 d' such that
min cE(er.c2)
P ((()) E Q(c2)))) Pol
# max
and also,
FP, S n1
Applying Lemma E.2.5 we have that
1
32 # max
Elf(x)
v VP FP)(2)) > 2
|
We also know that
E|FP, [x]] < 31
and therefore
3
82
1
#*ke ke
24 max
E(f)
31
A #
A #mx
8
n(n D)} #max
a
which is contradiction with E(h 0) e, thus completing proof.
27

---

F MOTIF ERROR EQUIVALENCE
In this section, prove that our proof's enror metric is only ever off by constant factor from empirical error metries.
F.1 FORMAL STATEMENT
For all G such that 6(ğ) = 0
mm (0) S = (N & EA ()E S c A CB( S 2c mm (g) SE (NE()) A CB()) S 2
F.2
CORRESPONDENCE WITH QUANTITIES FROM LEMMA B.2.
First, note that
#m = -mnn #
Σ Σ C0g(c,)
c((osr(e))
Then, note that
FNE(O) Σ. FNg(C)
#
FP
FPE
#(0)
by inspection. The case of CE more complicated, due to the presence of MM. Inspecting the denominator, we have
MM m") + NMM m²)l P()) [F, m")
and therefore
9" ((MM((())) (x))] (1FF(()))(())
Additionally, note that if assume there ties in the max computation (or alterna- tively, they are broken in some systematic way rather than leading to duplicates), we know that
MM m (((i,e) P(g 3(''c') E P()(z)) : i' p2(1)
and thus
([MM(((=)). #1 FNE(g))
Letting Qm, m²) ((i,c) ((g" :3),)) 3(,,) E P()(x)) /' E p2(1)) we can break this down into a dichotomy
Q(ū,m²) Q2 L (2(
where
Qi( = ((i,c) E (z)) "1('' E P((())): E
(²(m,m²) = ((i,c) E (x) (1 # (22 E P(ā(x)) i pz))))
We have that
Bl|Q2(0(z), (z))1 (x))))
Σ CM,(c)
28

---

Finally, we can see that
|conf~ m²)l -
E K(i',c)) (mn, : E p2(1)}]
(1,0)EQ
Ar m Q2(
m +
Σ
H('' E conf, p2()))
(20))
A m (222(1)22222222022 m +
Σ
1(3' T(d)) -cA
# 1)
(Me))QQ
Ay (m) m3 Q2 m²)l +
Σ
Σ
(()m(0) = (1 11 =1
(Me)) d'fice)=
(0]c (x))1] , Σ (c2(c)+) Σ E
C0glac))
d(r(ce)=
X Σ CM₂(C)+ + Ecog(e,))
- E cf(((c)me
Σ C0glc,c))
A Σ (c2(c) + #* Σ FNg(c) CM)
Σ Σ C0g(c,d
d((r(c))=e
where Ay (m, m²) and X are some constants in [0, 1]. Since (min, A(x)) (min, B(x)) S min, (A(x) B(x)) (min, A(x)) + (max, B(x)),
have that
min (((])ccoo((((()).))))))) (0(*), (x)) # (m + Σ (c,(c) - FNN((C) - (M,
for some A E [0, 1] and thus
min Ellconf, (o(x), ("(c)) (x))1 = # (m) - A2 Σ (c,(c) - -FFN(())
(N,(C)
= = A Σ FN (C) ΣFN,(O)
= - (1+ + (1+ FN)(())
E. CM,(c) E,CM,(O) S ^2, we have As E 10, 1]. We thus have that for some A2 E [0, 1J. and since λa 12 PNole)
CE(g) = #* Em)
(1 + FN)
#1(1 FNE(O))
Em( - (1 + FFBB()
- FNB(O)
F.3 MAIN PROOF
Forward direction Assume (m) S 6. We have that
We proceed by using the quantities from above, bounding FNE,
FN
FNE(g)
#*
# Em (g) d2 E. CM2(c) - min, El (((=). (x))
#*
< Eg)
<6
29

---

m 1
m 2
A
99.99%
00.01%
y = " "A"
y = ""
Figure Counterexample motivating the g(m22) S Pmm(m)) (m2) requirement
Since we know from Section E.2.1 that FP, E. FNy(c) we have
FP
FPE(O)
#
< >e (C)
#
S
If then clearly CE(g) < 2c. If c < 2 we have that FNE(O)
1 and thus
2
CE(g) Em (0) (1+ + FN)(()(()
- -NNE(O)
2 (0)
(1 (x)
< 28m (ō)
$ 2e
desired
Backward Direction Assume that CE(g) < jec and FNE(9) S f.c. We then have that Em CE(g) FNE(g)) + (1 FNE(()
CE(g) + 2F)
G COUNTEREXAMPLES FOR LESS INTUITIVE ASSUMPTIONS
G.1 TRANSLATIONAL INVARIANCE OF BACKGROUND DISTRIBUTION
We assume that P. is translationally invariant. For an example of what happens if this assumption broken, consider a version of DIGITCIRCLE where one digit always appears the left side of the image, and is not read as part of the output y". While one might think this would lead to the possibility of high motif error without high end-to-end error, the locality of 9 ensures that the motif is predicted correctly despite not being used, as ý does not "know" that the motif will not be useful. However, PA were not translationally invariant, it would be possible for e-g., the background to be systematically darker on the left side of the image, with the motif prediction being slightly off center to take this into account and not report the motif it is in the darker region. This would not affect end-to-end error but would affect motif error.
G.2 No INCREASE IN PROBABILITY MASS FOR PERTURBATIONS
To demonstrate the necessity of the ((m2) & Pm (m)) requirement, consider the domain shown in Figure 7 which has exactly M (mi,m²) with my having probability (depicted is
30

---

014
5000 3013
21 85%)
13
10%)
119
32T4
4.1%
E176
11000250
6.04
119.0 11m1
1100
LPA
5.1%
1204 (3.00 (3.0%)
ATC 1P%
TA
130.09%
1.5%
rant
F9
2 70%)
8716
175
11.4%
DACS
3 17%)
1.59
1.8%
111
T0N
1.9%
100e (170%)
M2501
#3817
Figure 8: Confusion Matrix of 10k unseen samples computed for seed=1 across all domains. Each row represents a true motif being recognized and column represents a channel in the model's s motif output. False positive and false negative motifs are placed into the none rows and columns, respec- tively. Each row labeled by the percentage of motifs falling into the row, and each row's cells are then normalized to add to 1. We then permute to align along the diagonal. For LATEX-OCH we use more channels than there are symbol types so we merge channels together for display and analysis.
= 0.01%). Clearly, this domain trivially satisfies NON-OVERLAPPING and HOTIF-SUFFICIER
as there is only one motif. Additionally, if we let (mm2)mm) = and s ((-Im) _mmm) = 1, we have that ri clearly satisfies the support requirement and since ((m2) = 1 - 1, we have that Σ g(m) = - L so this domain satisfies the MMTTIF-NECESSTI
assumption with a 1 However, we have that we can set ((x) 0 and h(x)= "A" ", giving E(h o g) 6. and (g) 1. This clearly breaks our proof since we can make/ arbitrarily small while not changing a much as it converges to 1.
H CONFUSION
Figure X depicts appropriately permuted confusion matrices for each domain. Our model generally assigns each true motif to channel or set of channels in the sparse layer. The main exception is that in LATEX-OCR, the fraction bar is never recognized, and () are only sometimes recognized. In other seeds, exhibits similar behavior to ()
I SPARSITY AS AN INFORMATION BOUND
1.1 CONNECTION TO INFORMATION BOUND
Sparsity induces an information bound by limiting the amount of information in the interme- diate representation. Specifically, if we let X be random variable for the input, and M g" (x) be the motif layer, we have that we can bound the mutual information between inputs and motifs as M) < H(M), where H(-) is entropy. Thus, to bound mutual infor- mation, it is sufficient to bound HMM). We first can break into per-channel components: H(M) < E,, (M]]i
let &,. denote the density of channel c at position i, and n 2 H(Mii, M[.. # 0) be a bound on the amount of entropy in each nonzero activation (see Appendix 1.2). Then we apply the chain rule to get H(M[i,)) c)) < H(B(5,,.) + n7,,, where B--) is the Bernoulli distribution. Thus, H(M) < Ei. (B((,.)) + SSmm,,weere S is the size of
the image in pixels and is the number of channels, and is defined as in section 2 Finally, using
31

---

MT
ST
2.0
2
Binning error delta [%)
1.5
4
8
1.0
16
0.5
0.0
-0.5
10
6 10
10
6 x 10-5
Density [%]
Density [%]
Figure 9: Increase in error when binning. Each series represents different bin count, annotated in the legend. Density log-scaled and reversed to indicate training progress. MT is the model tracked in the rest of the paper, ST is the model defined in Appendix
K.3 K.3
Jensen's inequality (as (B(t)) is concave):
M) & H(M) S (B(H) + 16).
Since the computed bound monotonic function in where as - 0, the bound approaches O. we can see that sparsity bound can be used as an informational bottleneck for any information bound of user's choosing.
1.2 ENTROPY UPPER BOUND
To compute our entropy upper bound, we must first compute 7), as defined in Section [.1 this, bin the nonzero activations into 2 bins by quantile. We 17 to be the smallest value of k that does not substantially affect the accuracy of the model (we consider 0.1% to be a reasonable threshold for this purpose). Figure shows the result of this experiment, averaged across 9 seeds. The general downward trend in error caused by binning as density decreases demonstrates that reducing the number of motifs reduces the importance of the precise magnitudes. For the purposes of entropy bounding, we can use 17 = log(16) = 4b.
To compute
J PREDICTING MOTIF ERROR.
Figure shows the relationship between the motif errors and the overall end-to-end error for DIG- ITCIRCL... There is no relationship for FPE, but there is a positive relationship for CE, implying that strategy where one trains several models and then chooses the one with the best validation error a good way to reduce CE and thereby improve motif quality. This provides further evidence for Motif Identifiability (though the primary evidence for this remains that this model is able to achieve low FPE and CE in general, as training itself focuses on reducing end-to-end error via the loss func- tion). While this may seem contradict the result in Section 5.2 it in fact does not. Within single model. tightening the density has inverse effects on end-to-end error and CE, but separately, some models in general more less accurate.
32

---

20
End-to-End Error [%)
10
2
10
100
101
100
10-1
10a
FPE [%]
FNE [%]
CE [%]
Figure 10: Model error versus FPE and CE, at 1.1x the minimum sparsity. All are log-scaled to highlight the low-error region. Fach dot represents single model training seed.
K COMPARISONS BETWEEN SPARLING AND OTHER TECHNIQUES FOR SPARSITY
In this section we compare to alternatives of the SPARLING model. For all comparisons, we keep the model architecture fixed and only modify the Sparse layer.
K.1 BASELINES
We consider two other approaches to ensuring the creation of sparse motifs, both taking the form of auxiliary regularization losses. both cases, we vary loss weight to analyze how that affects error and sparsity. First, consider L1 loss. In our implementation, layer followed by ReLU, The output of the ReLU then used in an auxiliary L1 los This approach is discussed in Jiang et al. (2015). We also consider using K L-divergence loss as Jiang jtt al. (2013). The approach to apply a sigmoid, then compute a KL-divergence between the Bernoulli implied by the mean activation of the sigmoid and target sparsity value (we use 99.995% to perform a direct comparison). While this usually is done across the training data (Ng. 2011), we instead enforce the loss across all positions and channels, but per-batch (the mean sparsity should be similar in each batch). Our other modification, in order to induce true sparsity, is to, after the sigmoid layer (where the loss is computed), subtract 0.5 and apply ReLU layer.
affine batch normalization
Table shows the results of using L1 as method for encouraging sparsity. There are two weight regimes, where when À < 1, we end up with high density (relative the theoretical minimum) low error, and when À 2 2, we end up with high-error model. Even in the latter case, the L, loss does not consistently push density down to the level of SPARLING, suggesting it might be insufficiently strong learning signal. In our experiments, the K L-divergence was unable to achieve density below 0.1%, even when we used a loss weight as high as À 10y and x 10 steps (much more than necessary for convergence of the L1 model). Thus, conclude that it is unsuitable for encouraging the kind of sparsity we are interested in.
K.2 ABLATIONS
We consider two ablations: First, is the batch normalization place before sparse layer essary? Second, is the adaptive sparsity algorithm we use necessary? These ablations are only is the domain where simpler techniques would work best.
evaluated on DIGITCIRCLE
We find that including batch normalization before the sparsity layer is crucial. Without batch normalization layer, over 9 runs, the best model gets an E2BE of 71% (the best ST model gets an
This approach parameterizes the same model class SPARLING: both ReL.U U in forward pass
33

---

L1
SPARLING
0.1
==1
À
==5
10
MT
FPB [%]
99.99
99.90
91.25
95.99
97.63
1.48 [0.07-4.23]
FNE [%]]
0.00
0.00
58.09
73.12
84.51
0.42 [0.25-0.67]
CE [6]
50.34
47.84
45.65
50.85
33.82
1.16 [0.03-3.39]
E2EB [%]
0.68
2.85
70.31
75.00
73.20
0.74 [0.47-1.15]
Density [%]]
37
4.6
0.021
0.031
0.028
0.005
Table 1: Results of L, experiment on DIGITCIRCLE. As L1 increases, the density decreases, but end-to-end error becomes >> 50%, and CR/FPE never improve the level of SPARLING. SPARLING is able to keep error low while achieving lower density than L1 with any À value we tried.
10%
1%
MT Error
0.1%
0.01%
FPE
FNE
CE
E2E
0.01%
0.1%
1%
10%
ST Error
Figure 11: SPARLING using MT (as in the main figures) ST
error of 52.48%); in essence, it is not able to learn the task at all. Additionally, annealing (Algo- rithm I is clearly necessary: when started with the annealing algorithm's final and penultimate values, the model converged to E2EE values of 68% and 71 % respectively.
K.3 SINGLE THRESHOLD
In this section, consider variation to the quantile function. We call this the single threshold (ST) sparsity approach, as opposed to the multiple thresholds (MT) technique described in Section 4 where take the quantile across the entire input (batch axis, dimensional axes, channel axis). In this case, the channels can have differing resulting densities that average together to the target 6. More precisely, use the quantile function GsT that
IR Bxd, X.. xn R. - E, implemented such
Σ 1(1bbbii, qsr(2,p)).
BSC
As seen Figure II ST performs substantially worse in terms of CE and E2EE, while performing better with respect FPE. Without the constraint that the motifs have equivalent density across
34

---

each channel, some motifs are being used to represent multiple digits, which substantially increases confusion error, but also reduces false positives. In general, the MT model is superior as it has reasonable FPE and substantially lower CE/E2EE.
L COMPARISON TO DIRECTLY LEARNING THE MOTIFS
SPARLING [mean]
DIRECT [mean]
Ratio [of means]
DigitCirele LaTeX-OCR
1.24
0.01
0.01
6.55
0.12
0.02
LaTeX-OCR [without +O]
2.96
0.10
0.03
AudioMNISTSSequoncctriin
5.41
0.61
0.11
AudioMNNISSSeqqenncttess
8.01
4.28
0.53
Table 2: Error [%] and ratios between errors. All are computed as a mean across seeds
The purpose of SPARLING is to be able to learn intermediate state without having to have access to any training data on the intermediate state. this section, we analyze how well it does this goal, by comparing it to DIRECT, a setting where we train and evaluate on the intermediate state directly. Specifically, we construct datasets for each task single motifs and train and test models on these datasets, then also test SPARLING these datasets.
In the case of DIGITCIRCLE and LATEX-OCR, DIRECT a trivial task as there no distributional shift in the motif samples used to train and evaluate the model effectively, DIRECT is tested on the training set. Thus, DIRECT gets ~00 error.
However, the AUDIOMNISTTEQQEER eπror on the training sample distribution but a much higher 4.28% error on the testing sample distri- bution. Meanwhile, SPARLING increases substantially less, from 5.41% to 8.01 %. This is because the error in SPARLING comes from two sources: the underlying uncertainty in prediction shares with the DIRECT technique, and epistemic uncertainty related to the problem of identifying motif's from end-to-end data. This latter error evidently does not scale linearly with the difficulty of the underlying task.
task, the DIRECT has non-negligible error, with 0.61%
M SPLICING DOMAIN
We also consider the original splicing domain, hypothesizing that on a domain that does not satisfy assumptions in Section 3.3 SPARLING will not perform well but perform better than chance. To keep things simple, we use the Jaganathan et (2019) architecture the model and simple convolutional stack identical to the adjustment model from Gupta et al.) (2024) as the Ò model. To ensure our experiment is picking up on real signal, will exclude the local splice site motifs (LSSI sites) from the set of true motifs for the purposes of analysis, as these sites can be found trivially from the end-to-end data, instead, we only evaluate on the other protein binding sites.
100
Sparling
T
Random Baseline
80
60
40
20
FNE
E2EE
Figure 12: Results on the splicing domain. Results are presented per error metric for both 4 runs of SPARLING and 95% CI of boostrap mean of 10 runs of a matched randomized baseline.
35

---

SPARLING achieves reasonable end-to-end performance, but does not perform well the other three domains on motif prediction (see Figure 12) However, we find that it consistently outperforms random chance baseline in the most important error metric, CE CE-indicating that it is correctly classifying motifs. The other error metrics are more mixed, while outperforms the baseline in FPE, it underperforms it in FNE suggesting that the model is producing duplicate activations, which leads to insufficient coverage of the motifs. Overall, this is consistent with our hypothesis that while Motif Identifiability is only possible given certain assumptions, SPARLING capable of picking up some signal even when these assumptions are not met.
N REPRODUCIBILITY
N.1 REPRODUCIBILITY REPOSITORY
repository that will allow you to reproduce the results in this paper can be found https:
77
N.2 COMPUTE USAGE
All our experiments were performed on NVIDIA GeForce GTX 1080 Ti (12GiB VRAM) or Quadro RTX 5000 (16GiB VRAM) GPUs. On average, DIGITCIRCLE experiments took 4 hours each to train, LATEX-OCR experiments took 14 days each to train, and AUDIOMNISTSEQUENCC experi- ments took days train. In total, we used about 350 GPU days of compute for the experiments reported in the body of the paper, 250 GPU days for the experiments referenced in footnotes/the appendix, and 200 GPU days of compute for exploratory experiments that were not referenced.
36


Reviews:
[{"review_id": "sZ5Spg6ET0", "reviewer": "Reviewer_oPKW", "summary": "This paper studies when and how semantically meaningful spatial concepts (called motifs) can be recovered purely from end-to-end supervision.\n\nIt introduces a Motif Identifiability Theorem, showing that under assumptions of locality, sparsity, non-overlap, patch independence, and α-motif-importance, one can identify the true latent motif map from low end-to-end error.\n\nBuilding on this, the authors propose SPARLING, an algorithm that enforces extreme activation sparsity through an adaptive thresholding layer and annealed sparsity schedule.\n\nEmpirically, SPARLING is evaluated on synthetic and semi-realistic datasets—DIGITCIRCLE, LATEX-OCR, and AUDIOMNISTSEQUENCE—achieving accurate motif localization (> 90%) without direct supervision.", "questions": "N/A", "limitations": "", "rating": null, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "The paper provides a new theoretical formulation of identifiability for sparse local latent variables—a valuable bridge between statistical identifiability theory and deep concept-bottleneck learning. The emphasis on end-to-end identifiability (without explicit supervision) is conceptually novel.", "weaknesses": "1. The paper is not well-structured and well-written. The mathematical formalism is often dense and difficult to parse. Many places are hard to follow.\n\n2. The theorem statements are long and self-referential but lack rigorous statement.\n\n3. Figures are under-explained; axes and variables are often unlabeled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R9OjonHfux", "reviewer": "Reviewer_YS9h", "summary": "The paper proves that motif identification is solvable under three main assumptions. it motivates that understanding motifs can help improvements of deep learning models in many real-world tasks such as RNA modeling and Latex OCR. The theoretical proofs seems sound and experiments are supporting the main general proofs, and the assumptions of the paper seems realistic for the real-world tasks.", "questions": "My questions are provided in weakness section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper has several strengths:\n\n* **Addressing an interesting problem:** Although I am not deeply familiar with this line of research, the problem of recognizing and understanding motifs appears to be a very interesting and meaningful task. The paper effectively motivates this problem through real-world applications such as RNA modeling, which are indeed highly relevant today.\n\n* **Sound assumptions:** The assumptions made in the paper, aside from the non-overlapping constraint, seem reasonable. The idea that most motifs are important is both intuitive and realistic for the majority of relevant tasks.\n\n* **Theoretical proofs:** While I am not fully familiar with the related literature, the theoretical proofs appear to be well supported, with clear reasoning and detailed supplementary arguments.\n\n* **Experimental setup:** The experiments are well designed and align closely with the paper’s mot", "weaknesses": "I have only some minor weaknesses on presentation and details of the model:\n* **1)** In real-world tasks, it is unclear how the model distinguishes between motifs and noise. As suggested by the paper, the mechanism by which the model identifies this difference is not explicitly explained.\n\n* **2)** The description of the neural architecture is largely missing. Apart from a single equation related to sparsity in the Methods section, there is no clear definition of the model’s architecture or how it is implemented for learning.\n\n* **3)** Lack of visualizations: Figures 1 and 2 provide clear and informative illustrations of motifs, but it would be helpful to include visualizations showing whether the model actually “attends to” or decodes these motifs. Beyond reporting accuracies, some interpretable visualization or analysis could demonstrate that the model is indeed identifying motifs.\n\n* **4)** The title suggests that sparse activations play a central role, yet the connection between sparsity, motif learning, and the main theorem is not clearly articulated. This link should be made more explicit in the presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kfJ2gZB133", "reviewer": "Reviewer_uhfz", "summary": "This paper theoretically proves the motifs (or latent variables) of a task can be accurately identified by a neural network that is trained end-to-end. Specifically, the paper proves an upper bound for the motif error when the end-to-end error is low under certain assumptions. The paper shows that this kind of motif identifiability can be achieved by enforcing extremely sparse activations in the training process and empirically validates the algorithm on several synthetic tasks including digit circle recognition, latex-ocr, and audio sequence identification.", "questions": "1.\tIs there a relation between the identification of sparse spatial motifs and the superposition hypothesis[cite1] in mechanistic interpretability? I would appreciate a discussion on this comparison.\n\n2.\tWhat concrete modifications to the assumptions or the proof would allow overlapping motifs (common in detection with occlusion) or spatially correlated backgrounds? Are there any partial thoughts or empirical evidence with controlled overlap?\n\n3.\tHow sensitive is SPARLING to $\\delta$ misspecification (as stated in Weaknesses 1)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe scope of the paper is clear: it focuses on problems in which latent variables are sparse non-overlapping spatial positions. Based on this, the paper addresses an interesting problem: can a model automatically identify these latent variables in its hidden representation when trained end-to-end?\n\n2.\tThe paper has good theoretical contribution: an identifiability result for local, extremely sparse spatial concepts learned end-to-end, under explicit assumptions. The bound that relates end-to-end error to motif error is valuable.\n\n3.\tNovel algorithm design: Besides the theoretical results, the paper proposes a practical algorithm to boost the sparsity of intermediate representations via per-channel quantile thresholds and annealing. This sparsity ratio is often unattainable by standard regularizers.", "weaknesses": "1.\tThe theorem requires $\\delta(\\hat{g})=\\delta^\\*$. However, in practice, $\\delta^*$ might be unknown. There is no robustness analysis for misspecified $\\delta$.\n\n2.\tExperiments remain synthetic (including LATEX-OCR). Real-world validations (e.g., natural scene text OCR, object detection) are absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MagW9IY8Ia", "reviewer": "Reviewer_g8Sy", "summary": "The paper introduces the Motif Identifiability Theorem, which proves that under conditions of locality, extreme sparsity, and certain distributional assumptions, a model with low end-to-end error must also accurately recover the true motifs if its bottleneck matches the real motif density. Building on this theory, the authors propose SPARLING, a training method that enforces highly sparse intermediate activations to learn spatial motifs directly from data. Using a quantile-based spatial sparsity layer with annealed target density and per-channel thresholds, SPARLING achieves over 90 percent correct motif localization on three semi-synthetic datasets: DIGITCIRCLE, LATEX-OCR, and AUDIOMNISTSEQUENCE.", "questions": "- Many scenes have structured, non-stationary backgrounds. Could data augmentation (random crops/phase shifts) approximate patch-independence enough to recover motifs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Without direct supervision of intermediate motifs, the proposed method achieves interpretable and accurate motifs while maintaining competitive end-to-end performance across three evaluated tasks.\n- The work shows interesting trade-offs with their empirical results. As shown in figure 4, increasing sparsity reduces the confusion error (CE) and motif channels become better separated. However, it slightly increases end-to-end task error (E2EE), given that model's information bottleneck is tighter.\n- As shown in appendix K.1, the work compares Sparling with L1 regularization as a baseline to encourage sparsity. Although by using a higher $\\lambda$ one can decrease the density using L1 regularization, the E2EE significantly increases. Using Sparling however, the work shows that it can achieve density lower than $\\lambda =10$ of L1 regularization but with E2EE comparable with $\\lambda=0.1$.\n- The work tests the proposed method on domains that violates the assumptions. As shown in appendix M, in splicing domain that doesn't met the assumptions, although CE is high, it shows that it outperforms random chance baseline, highlighting that the method is able to capture meaningful signals even when its assumptions are not satisfied.", "weaknesses": "-  All the core results are evaluated on synthetic or synthetic-ized tasks (digits in circles, LaTeX OCR with controlled rendering, spoken digits). The work would be much stronger with a real-world dataset. Tasks like identifying small-objects such as ships on open water from satellite images. It would be interesting to see end-to-end training from scene labels such as vessels being present could discover localized motifs of where they are without providing bounding boxes.\n - As stated in Appendix N, approximately 350 GPU days were required for the main experiments. This is a substantial computational cost considering the relatively modest problem sizes, which limits the method’s practicality for larger-scale applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L8bMVRYnuG", "reviewer": "Reviewer_sThB", "summary": "The authors propose an end-to-end spatial concept learning framework that identifies interpretable intermediate representations (called motifs) through extremely sparse activations. The paper introduces a Motif Identifiability Theorem, proving that under certain assumptions (locality, sparsity, and independence), true latent motifs can be recovered solely from end-to-end supervision. They further present the adaptive sparsity algorithm, which gradually enforces ultra-sparse activations during training to achieve interpretability without supervision. Experiments on synthetic datasets  show that SPARLING can localize intermediate states with over 90% accuracy, bridging theoretical guarantees and practical learning.", "questions": "How sensitive is SPARLING’s performance to the chosen sparsity target (e.g., 99% vs 99.9%)—does extreme sparsity always help motif identifiability?\n\nCan the authors clarify whether the Motif Identifiability Theorem can be extended to overlapping or correlated motifs, and if so, what modifications would be needed to the current assumptions?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper establishes a clear theoretical foundation with the Motif Identifiability Theorem (Section 3), offering formal conditions where sparse, local latent variables are identifiable. This bridges a long-standing gap between interpretability and end-to-end deep learning by providing provable guarantees rather than heuristic explanations.\n\nFrom an engineering perspective, the Spatial Sparsity Layer and Adaptive Sparsity Algorithm (Section 4) are elegant and practical. They show how to gradually enforce 99%+ sparsity while keeping models trainable, making the framework useful beyond the synthetic domains tested.", "weaknesses": "The experiments mainly rely on highly synthetic datasets—DIGITCIRCLE, LATEX-OCR, and AUDIOMNISTSEQUENCE (Section 5.1). This leaves uncertainty about whether SPARLING’s assumptions (e.g., NON-OVERLAPPING or PATCH-INDEPENDENCE) hold in real tasks such as natural images or genomics data.\n\nThe theorem depends on strong constraints (Section 3.3 – lines 269–324) like NON-OVERLAPPING motifs and PATCH-INDEPENDENCE, which may not be satisfied in practical domains. For example, line 270 explicitly admits that “this assumption… excludes some of our domains because of how large the distance needs to be.”\n\nFigure 5 and the discussion in lines 432–480 show SPARLING’s end-to-end error is higher than non-sparse baselines, but the explanation remains speculative (“we theorize that this is because our constraint… requires the model to ‘commit’”). There is also no direct quantitative comparison with modern interpretability methods such as saliency or concept bottleneck models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)