Tversky Neural Networks: Psychologically Plausible Deep Learning with Differentiable Tversky Similarity
Authors: Moussa Koulako Bala Doumbouya
Dan Jurafsky
Christopher D Manning
Venue: ICLR 2026 Poster (2026)
Keywords: machine learning
psychology
neural networks
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A differentiable parameterization of Tversky (1977)'s theory of psychological similarity, and derived neural network building blocks

Abstract: Work in psychology has highlighted that the geometric model of similarity standard in deep learning is not psychologically plausible because its metric properties such as symmetry do not align with human perception of similarity. In contrast, Tversky (1977) proposed an axiomatic theory of similarity with psychological plausibility based on a representation of objects as sets of features, and their similarity as a function of their common and distinctive features. This model of similarity has not been incorporated as a general-purpose building block in deep learning, in part because of the challenge of incorporating discrete set operations.     In this paper, we develop a differentiable parameterization of Tversky's similarity that is learnable through gradient descent, and derive basic neural network building blocks such as the Tversky projection layer, which unlike the linear projection layer can model non-linear functions such as XOR.     Through experiments with image recognition and language modeling neural networks, we show that the Tversky projection layer is a beneficial replacement for the linear projection layer.     For instance, on the NABirds image classification task, a ResNet-50 with a Tversky projection layer trained from scratch achieves a 2.36 percentage point accuracy improvement over the linear layer baseline.     With Tversky projection layers, GPT-2's perplexity on PTB decreases by 7.8%, and its parameter count by 34.8%.     Finally, we propose a unified interpretation of both projection layer types as computing similarities of inputs to learned prototypes, along with a novel visualization technique. Crucially, Tversky's set-based representation enables the algebraic specification of semantic fields, which we illustrate with lexical and visual stimuli. Our work offers a new paradigm for neural networks that are not only more accurate and efficient, but also interpretable under an established theory of psychological similarity.

Published as a conference paper at ICLR 2026
TVERSKY NEURAL NETWORKS: PSYCHOLOGICALLY PLAUSIBLE DEEP LEARNING WITH DIFFERENTIABLE TVERSKY SIMILARITY
Moussa Koulako Bala Doumbouya
Dan Jurafsky
Christopher D. Manning
Department of Computer Science, 353 Jane Stanford Way; Stanford, CA 94305 (moussa, jurafsky, manning @stanford.eee edu
1.4
1.59
1.78
2.27
2,30
45.85
45,89
59,47
61.26
02.88
0
0
0
0
0
V
O
O
O
0
O
/
7
)
/
7
1
1
1
1
4
1
a
4.
2
2
7
2
2
5
3
5
3
3
3
3
3
3
7
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
S
5
5
S
5
5
S
5
S
5
S
5
6
6
V
6
6
6
6
6
7
7
R
7
2
7
7
7
7
7
7
7
2
6
R
8
8
4
8
8
8
8
8
8
8
8
9
9
7
9
2
9
9
9
9
9
9
Figure 1: NABirds and MNIST stimuli sorted from low to high salience (Equation 2) in TverskyResNet-50. More salient stimuli exhibit more object features and greater informational content, familiartty and goodness of form, in alignment with human judgment (Tversky, 1977).
ABSTRACT
Work in psychology has highlighted that the geometric model of similarity standard in deep learning is not psychologically plausible because its metric properties such as symmetry do not align with human perception of similarity. In contrast, Tversky (1977) proposed an axiomatic theory of similarity with psychological plausibility based on representation of objects as of features, and their similarity as a function of their common and distinctive features. This model of similarity has not been incorporated as general- purppos building block in deep learning, part because of the challenge of incorporating discrete set operations. In this paper, we develop a differentiable parameterization Tversky's similarity that is learnable through gradient descent, and derive basic neural network building blocks such the Tversky projection layer, which unlike the linear projection layer can model non-linear functions such XOR. Through experiments with image recognition and language modeling neural networks, we show that the Tversky projection layer is beneficial replacement for the linear projection layer. For instance, the NABirds image classification task, a ResNet-50 with a Tversky projection layer trained from scratch achieves 2.36 percentage point accuracy improvement over the linear layer baseline. With Tversky projection layers, GPT-2'ss perplexity on PTB decreases by 7.8%, and its parameter count by 34.8%. Finally, we propose a unified interpretation of both projection layer types computing similarities of inputs to learned prototypes, along with a novel visualization technique. Crucially, Tversky's 's set- based representation enables the algebraic specification of semantic fields, which we illustrate with lexical and visual stimuli. Our work offers a new paradigm for neural networks that not only more accurate and efficient, but also interpretable under an established theory psychological similarity.'
https: / / thub. www/noounoouuuaaaoooooooooooooaa wwtp://teeeeeeeeeeeeeeeee
r2026.6

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
The geometric model of similarity is ubiquitously used in modern neural networks. Their architectures include operations that assess the similarity between objects using metric similarity functions such as the vector dot product or cosine similarity. For instance, linear projection layers, also known as dense, or fully connected, output the dot- pot-product similarity of the input vector with the columns of its weight matrix, which are concept prototypes. Classification modules typically normalize these similarities with SOFTMAX to form valid probability distributions (Bridle, 1990; LeCun et al., 1998). Convolution layers of image recognition neural networks compute the dot product of image regions and convolution kernels, which represent prototypical patterns. LSTM gates compute dot product similarities of combinations of inputs, hidden states, and cell states with prototypical input, forget. and output gating patterns (Hochreiter and Schmidhuber, 1997; Graves et al., 2005; Graves, 2013). Semantic word embedding models such as word2vee (Mikolov et al., 2013a) and Glo (Pennington al., 2014) compute the dot-product and cosine similarity of word representations with prototypical embeddings to predict words. Neural language models apply the same classification mechanism to output discrete words or subwords at each time step (Bengio et al., 2003; Sutskever al., 2014; Devlin et al., 2019; Radford et al., 2018; OpenAl et al., 2024). Attention weights (Bahdanau et al., 2015; Vaswani et al., 2017) are the normalized dot-product similarity of queries and keys, which are themselves obtained using linear projection layers. Geometric similarity representations have also been widely used in machine learning objective function design for over 200 years, from least squares methods (Legendre, 1806) to modern approaches such as 1.2 loss (Hadsell et al., 2006), Large Margin Cosine Loss (Wang et al., 2018), and Noise Contrastive Estimation (van den Oord et al., 2018; Chen et al., 2020), spanning both supervised and self-supervised learning paradigms.
However, Tversky (1977) famously challenged this geometric representation of similarity, showing its psychological implausibility due to its fundamental inability to model phenomena such as humans' asymmetric judgment of similarity. For instance, "We say that 'the son resembles the father' rather than 'the father resembles the son '" He addressed this problem with his axiomatic theory based feature matching process in which objects are represented as sets of features and their similarity is measured as a linear combination of measures of their common and distinctive features. Tversky's similarity has been shown to model human judgment more accurately than metric similarity measures in various empirical studies (Gati and Tversky, 1984; Ritov et al., 1990; Siegel al., 1982; Rorissa, 2007). Moreover, over the past decade, several works have highlighted the limitations of the geometric model of similarity and related parallelogram model of semantic relations (Garg et al., 2015; Chen et al., 2017: Peterson et al., 2020; Zhou et al., 2022), and advocated for more psychologically principled similarity measures such as Tversky's model, which has strong empirical support.
Nevertheless, a differentiable expression of Tversky's ' similarity suitable as general-purpose purpose neural network building block has not been proposed to date, preventing its ubiquitous integration in gradient- based machine learning (Rumelhart et al., 1986; LeCun et al., 1998; Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017: He et al., 2016; Radford et al., 2018; OpenAl et al., 2024). The formulation of differentiable Tversky similarity function is non toivtrivial because it employs measures of set intersections and differences, which are not diffcrentiable with respect to object features comprising those sets. To address this gap, we propose novel representation of features as vectors of the same dimensionality as object vectors, and a dual representation of objects both as vectors and as sets, such that an object is the set of features with which it has a positive dot product. This representation of objects and features enables us to define differentiable measures of set intersections and differences, which we use to construct differentiable Tversky similarity function suitable for deep learning. We also propose the Tversky Projection neural layer, analogous to the linear projection layer but computing Tversky rather than dot-product similarities of their input to prototypes.
We introduce our proposed differentiable similarity function, and basic neural network modules, along with a novel method for interpreting and visualizing deep projection layers in Section 3. Section demonstrates the effectiveness of Tversky Projection layers in state-of-the-ar architectures, including ResNet-50 and GPT-2, on image recognition and language modeling tasks. Our experiments show that the use of Tversky projection layers can lead to significant improvements in both accuracy and parameter efficiency. In particular, we show that the use of a Tversky Projection layer instead of the final linear projection layer in ResNet-50 can result in up to 2.36 percentage points accuracy improvement in image recognition tasks. We also show that replacing linear projection layers in the language modeling head, and attention blocks in GPT-22 can result in 7.8% reduction in perplexity
2

---

Published as a conference paper at ICLR 2026
and 34.8% reduction in parameter count. Section 4.4 provides qualitative analyses of Tversky projection layers, including results highlighting their principled explainability under Tversky's 's theory of similarity, which enables the algebraic specification of human-interpreetbbl semantic fields. Overall the main contribution of this work is to provide a foundation for efficient and interpretable neural networks based established theory of psychological similarity.
2 RELATED WORK
Here briefly discuss prior work related to Tversky neural networks.
Tversky Loss for Semantic Segmentation: Salchi et al. (2017) propose a Tversky-based loss for 3D image segmentation, treating voxel predictions and ground truths as features. Unlike their approach, our definition of features is not restricted to pixels or voxels, and is applicable in any data modality and any neural network layer.
Semantic Similarity Networks: Rahnama and Hüllermeier (2020) learn a semantic similarity network via two-level architecture of binary feature extractors followed by Tversky similarity layer, but their approach is limited to similarity learning and requires stimulus pairs annotated with similarity scores and fine-grand tnee-gaaied feature annotations. Our method uses learnable feature bank to measure feature presence and salience, requires no such annotations, and serves as general-purpose building block usable in any neural network.
Mixttree-o-EExeer Networks: Tversky layer indicator functions act as hard routers, making Tversky layers sparsely activated modules analogous Mixture-of-ENye Ex-eertstt (MoE) gating networks (Jacobs et al.. 1991; Shazeer et al., 2017; Fedus et al., 2022; Hwang et al., 2026). We discuss the related "dead feature" issue in Section
Prototype Visualization:
ProtoPNet (Chen et al., 2019) and ProtoTree (Nauta et al., 2021) learn latent patch representations compared against convolutional feature maps to infer classification evidence, but restricted to convolutional patches and cannot visualize fully connected layer parameters. Our method visualizes the parameters of any projection layer, fully connected or Tversky, directly in input space at any network depth.
Algebraic Semantic Fields: Vector arithmetic over word representations can probe semantic concepts such analogies (Mikolov et al., 2013b; Molino et al., 2019), but has limited expressivity, notably failing to characterize antonyms (Ali et al., 2024; Faruqui et al., 2015; Mrkšić et al., 2016; Ono et al., 2015). In contrast, our set-based representations enable the specification of arbitrary semantic fields using set-algebraic expressions in any domain and modality (Section 4.4).
METHODS
3.1 TVERSKY'S SIMILARITY
Tversky's model of similarity (Tversky, 1977) has emerged as influential theoretical representation of human similarity judgment, supported by extensive empirical evidence from cognitive psychology (Tversky and Gati, 1982; Gati and Tversky, 1984; Goldstone, 1994; Medin al., 1993; Rorissa, 2007). His work challenged the geometric model of similarity by demonstrating that humans systematically violate metric axioms such as minimality, symmetry, and triangle inequality when assessing similarity, and proposed theoretical framework in which objects are represented as sets of features and their similarity assessment as feature matching process. Formally, Tversky's 's asymmetric model of similarity of object to object b defined feature sets A and B is function F of their common and distinctive features. Tversky showed in his axiomatic framework that F is a linear combination of measures of its set parameters, namely: the common features of and 6, the distinctive features of a, and the distinctive features of b (Equation 1). In Section 3.2, we introduce a differentiable parameterization of this function, making suitable for gradient-based machine learning.
b(a, = F(An B, B, B A) (f(An - B) - A)
(1)
3

---

Published as a conference paper at ICLR 2026
3.2 DIFFERENTIABLE TVERSKY SIMILARITY
This section presents our proposed differentiable parameterization of Tversky's similarity function, which is constructed with representation of features vectors of the same dimensionality as objects, and a dual representation of objects as vectors and as sets.
Dual Representation of Objects as Vectors and as Sets: Given the learnable finite universe Z of feature vectors f& E gd and an object represented as the vector E R2 we propose I fu to be the scalar measure of feature f& in æ, and second representation of x as the set X {fi f2 fE >> 0}
of features with which z has positive dot product.
Salience: Tversky hypothesized that the relative salience of stimuli, or prominence of their features, determines the direction of asymmetry in human's judgment of similarity. The less salient stimulus (e.g. the son) assessed versa. Following Tversky's 's theory and our proposed representation, the salience of features in an object A, which is the sum of the measures of all features present in the object f(A) (Equation 2).
more similar the more salient stimulus (e.g. the father) than vice
f(A)=
> k=1 fx L(a fx 0]
[0]
(2)
Feature Set Intersections: To measure the common features of objects A and B, f(A n B) (Equation 3), we propose function to aggregate measures of the features present in both and and experiment with values min, max, product, mean, gmean and softmin. This function corresponds to the intersection reduction hyperparameter of Tversky neural modules.
12] k= (a fu. b- fx) [aa fo > 0Ab fo ~
f(AnB)=
Feature Set Difference: f(A B) is a measure of features present in A but not present in B (Equation 4). We propose an alternate form of this measure that accounts for features that are present in both A and B, but in greater amount in A (Fquation 5). These two measures of set difference respectively correspond to the values ignorematch and substractmatch of the difference reduction hyperparameter of Tversky neural modules.
P'(A ((A-B) Em
IN (a-
fit) [a fo > 0Λb. S S
S'AA B) ('A B)
Σ
[0] (a fa - b. Sa) 1 12>00a b.
(5)
k=1
3.3 TVERSKY NEURAL NETWORK MODULES
We propose two basic building blocks for Tversky neural networks, the Tversky Similarity Layer, which is analogous to metric similarity functions such as dot product or cosine similarity, and the Tversky Projection Layer, analogous the contemporary fully connected layer.
This layer, formalized in Equation 6, calculates the similarity of object a E Rdd to object b E Rd. Its learnable parameters are a feature bank Ω and the a, B and 0 scalar parameters of Tverky's contrast model of similarity (Equation 1).
Tversky Similarity Layer:
Rd x Rd
R
f(AnB) ʃ(A - B)
(a,
(6)
(a, b)
-
Lf(B A)
Tversky Projection Layer: This non-linear projection (Equation 7), calculates the similarity of its
input IRd each prototype the ordered set of p prototypes II, E Rdd, yielding vector in R
Rd
RP
50.0,3.0 (a, Ho)
p.a........))
S0.0.300 (a. II)
(7)
SP.a.3 (a, 1p
4

---

Published as a conference paper at ICLR 2026
3.4 INTERPRETATION OF PROJECTION LAYERS
Both the linear and Tversky projection layers output vectors in which each dimension is the similarity of the imput to prototype; however, they differ in the employed similarity function. Linear projection layers compute dot- fot-product similarities. This is an application of the geometric model of similarity, in which the features of objects and prototypes are scalar values arranged in a Cartesian coordinate space and similarity is measured as the oriented length of the object vector's projection onto the prototype vector. Dot product similarity satisfies metric axioms such as symmetry and triangle inequality that have been empirically shown to not align with human perception, and is provably incapable of modeling asymmetric relations. Tversky projection layers compute Tversky similarity with learnable parameters prototype vectors II, feature vectors 22 and Tversky's 's contrast model's weights a, 3. and 8. The number of features [S2] can be varied without affecting the output dimensionality p= [II]
3.5 TVERSKY FEATURE SHARING
Tversky feature banks and prototype banks can be shared across various layers in a neural network in semantically justifiable ways. For instance, in Tversky GPT-2, the output projection layers in attention blocks, and the final language modeling head can all share the same feature bank Ω as this parameter is semantically compatible across all those layers: it represents token features. Tversky projection layers that are language modeling heads can also use token embeddings as token prototypes. This parameter sharing strategy is similar weight rying widely used in language models employing a linear projection layer to classify output tokens (Inan et al., 2017). Multiple Tversky similarity or projection layers can share their feature or prototype bank if semantically compatible. They can also share the same feature bank while maintaining separate prototypes, or keep all their parameters separate. Our results Section 4.2 show that Tversky feature sharing can result a dramatic reduction of neural network parameter count while improving their performance.
3.6 DATA-DOMAIN VISUALIZATION PROJECTION LAYERS
We introduce a novel method of visualizing projection layer parameters such as the weights of fully connected layers, and the prototypes and features of Tversky projection layers to enable their interpretation in the same domain as data stimuli. Our method is based on the hypothesis that those prototypes and features are concepts that should be recognizable in the data domain, and is applicable regardless of the depth at which those projection layers employed in deep neural networks. Our proposed visualization method consists of specifying the projection parameters as tensors of the same shape as the input data. These input-domain-seee -onai--ppeccffeee parameters are forwarded through the neural network just like data stimuli up to the layer prior to the layer in which they are used. Their obtained vector representation is subsequently used perform the projection. This approach significantly differs from prior approaches of interpreting neural network parameters and decisions based on the visualization of activations, optimization of input stimuli, or construction of adversarial examples (Zeiler and Fergus, 2014; Mordvintsev et al., 2015; Zhang and Zhu, 2018; Selvaraju et al., 2020; Zhang and Liang. 2020; Fan et al., 2021). While our proposed method offers the clear advantage of visualizing projection parameters, it comes with the limitation that parameters specified in data-space are typically larger in size than their original counterparts, which increases the effective number of trainable parameters. These parameters also need to be forwarded along with every data batch, which induces additional training-tine time computation cost (See Figure 5 in Appendix A). We use this technique to qualitatively compare the parameters of linear projection layer and a Tversky projection layer of neural networks trained to recognize handwritten digits in our qualitative analysis experiment presented in Section 4.4, which revealed that Tversky Projection layers' parameters far more interpretable than the ones of the contemporary fully connected layer (see Figure 3).
4 EXPERIMENTS
Here we demonstrate the utility of our proposed differentiable Tversky similarity function in machine learning. First, we show by construction that a single Tversky projection layer can model the XOR function and report empirical results on the same learning task (Section 4.1). Then, in Sections 4.2 and 4.3, we conduct empirical experiments with of neural networks for language modeling and image recognition in which we compare baseline neural networks with their counterparts
5

---

Published as a conference paper at ICLR 2026
employing Tversky projection layers. Finally, in Section 4.4, conduct qualitative analyses demonstrating the interpretability of Tversky neural networks.
4.1 MODELING XOR WITH A SINGLE TVERSKY PROJECTION LAYER
As first experiment, we construct single Tversky projection layer that computes the XOR func- tion, which is not computable by a single linear projection due the required non-linear decision boundary. Figure 2 shows the constructed projection, and its data and parameter vectors along with their sel-centric interpretation. to learn XOR with gradient descent under various hyperparameter conditions, we find that: Some initializations prototypes and features lead to convergence failure; Initializing those parameters from uniform distribution leads to higher convergence probability compared to normal and orthog- onal initialization; Normalizing prototype and object vectors deteriorates convergence probability; product and substracimatch work best values of the infersection reduction and difference reduction hyperparameters: Convergence probability doesn't increase monotonically with the feature bank size; and finally, Tversky projection can model XOR with little one feature. See Appendix C.
empirical experiments in which we train Tversky projection
x
p
2x3
x²
0
x
pb
5/ = [s(x D.,,.p'),oo s(x' p\)]
Figure 2: (left) A Tversky projection layer with 11 learnable parameters that computes the XOR function. The input is one the four 2-digit binary numbers encoded as x IR. The output vector ÿ' 106 Øi] contains the Tversky similarity of x' to prototypes pd and p' computed using features f and fl, and coefficients 8, a and B, such that øi > 30 < zor(x") = 1. (middle) Vector representation of data x¹ (with x not marked in the figure but at the origin), prototypes p', and features f'. Notice positive dot-proddcts: x² c..fl x²2, 222 f0,p= pf and p².f2. (right) Set representation and Tversky feature matching. Objects are represented as the subset features with which they have (. (f" J. x and 3 are more similar to pd than to pl, which has two distinctive features penalizing similarity; thus zor([0, 0)) and zor[[11 1]) = 0. xx' and x² are more similar to p', with which they each share one feature; thus zor((0, 1]) and zor([1,0)
a positive dot product: x²" = 0. x { x² {f)). x3 =
4.2 LANGUAGE MODELING WITH TVERSKYGPT-2
Table 1: Perplexity (PPL) and parameter counts (Params) of GPT2 and TverskyGPT2 trained for the language modeling task on the PTB dataset. Perplexities are reported on the held out test split. Features
Model Init Tied Proto
Params
PPL
A Params
APPL
baseline tversky-head
finetuned
033 184
30.52
finetuned
16384
175 620 099
28.33
+7.7%
-7.2%
haseline
finetuned
Y
124-439 808
18.31
tversky-head
finetuned
32 768
149 605 635
18.62
+20.2%
+1.7%
bascline
scratch
N
163 037 184
111.79
worsky-alllllayer
scratch
4
591 655
98.22
-28.5
-12.1
bascline
scratch
Y
124 439 808
112.81
Sorsky-alllllayee
scratch
192
81 140
103.99
-34,8%
-7.8

---

Published as a conference paper at ICLR 2026
4.2.1 TASK AND DATA
We compare the baseline GPT-2 small (Radford et al., 2019) model with its Tversky variants on the Penn Treebank (PTB) language modeling benchmark (Marcus et al., 1993).
4.2.2 METHOD
The following TverskyGPT-2 variants were considered: Iversky-head, which replaces GPT-2's language modeling head's stack of linear layers with Tversky projection layer, and Iversky-all, which also replaces GPT-2's intermediate attention blocks' output projections with Tversky projections. Tversky feature sharing is employed across all Tversky projection layers. For all models, we experiment with prototype tying (tied not tied) and initialization (random vs OpenAl's is released weights) while training for 50 epochs on PTB's s training set, and validating on PTB validation set. The baseline models, and best Tversky models are evaluated PTB's held-out set.
4.2.3 RESULTS
Tversky language models matched or surpassed baseline perplexity in all settings except the one in which pre-trained weights and prototype tying are employed (Table 1). In that setting, Tversky prototypes and features are still randomly initialized, while the baseline fully connected layers are initialized from the pretrained token embedding matrix. The perplexity gap between the best Tversky neural network and baseline was higher the tversky-al all configuration trained from scratch, with 7.8% reduction in perplexity when prototypes are tied to input embeddings. and 12.1% reduction when they are not. In the tied prototype setting, the best Tversky model also has 34.8% fewer parameters. See Appendix B for validation results.
4.3 IMAGE RECOGNITION WITH TVERSKY RESNET-50
4.3.1 TASK AND DATA
We experiment with replacing the final linear projection layer in ResNet-50 (He et al., 2016) with a Tversky projection and report the accuracy of the baseline and Tversky variants on the MNIST (LeCun al., 1998) handwritten digit classification task and the NABirds (Van Horn et al., 2015) bird species classification task.
4.3.2 METHOD
We compare the baseline ResNet-50 to TverskyResNet-500 which replaces the fully connected layer with Tversky projection layer. Both models are evaluated under different training conditions. the Pretrained Backbone setting, convolutional weights are initialized from ImageNet: otherwise they randomly initialized. In the Frozen Backbone setting, backbone parameters are fixed and only the final projection module is optimized. Tversky prototype and feature banks are randomly initialized in all cases, and multiple feature bank sizes are explored. Because MNIST and Nirdrss do not provide official three-way way splits, randomly sample 10% of the training data to construct validation set and reserve the official test splits for final evaluation. Each configuration is trained with multiple random seeds, affecting parameter initialization and train/validation partitioning. Model selection performed using mean validation accuracy across seeds, and the selected configuration is evaluated on the held-out standard test set. Experimental details, including hyperparameter grids, optimization settings, data preprocessing, augmentation, model selection protocol, and final evaluation results are provided in Appendix Sections E and
4.3.3 RESULTS
Classification accuracies reported in Table show that Tversky vision models can match or surpass baseline accuracy under the specified experimental conditions. The exception is the frozen backbone setting on NABirds, where the baseline marginally outperforms TeerskyResNet-550. In particular, when trained from scratch, RverskyResNet-50 improves test accuracy by 2.36 percentage points.
In this experiment, TeerskyResNet-500 has more parameters than the baseline because, in addition to the prototype bank (analogous the fully connected layer weights), it also includes feature
7

---

Published as a conference paper at ICLR 2026
Table 2: Accuracy of ResNet-50 (Baseline) and TverskyResNet-500 (Tversky) on the tasks of MNIST handwritten digit classification and NABirds bird species classification. Pretrained (True when weights are initialized from ResNeet50/mmaeNet, False when they are randomly initialized). Frozen (True when only the final projection layer is finetuned, False when the entire model is finetuned)
MNIST
NABirds
Pretrained
Frozen
Tversky
Baseline
Tversky
Baseline
False
False
99.56 + 0.06
99.54 + 0.04
65.20 = 0.26
62.84 + 0.45
True
False
99.60 0.05
99.56 + 0.04
82.96 = 0.07
82.37 + 0.25
True
True
86.66 + 0.17
86.64 + 0.14
38.73 + 0.20
40.25 + 0.28
bank. While the increased parameter count of Teersky could contribute to the observed accuracy improvement, our results in Section 4.2 show that the use of T T'errky neural networks can simultaneously result decreased parameter count and increased performance compared to baseline.
4.4 QUALITATIVE ANALYSIS OF TVERSKY NEURAL NETWORKS
Here we present results showing the interpretability Tversky neural networks. We discuss the explainability of learned prototypes, the quantification of salience, and the algebraic specification of semantic fields enabled by Tversky neural networks.
4.4.1 METHOD
Salience Measure: To assess whether salience, as computed using Equation 2, is intelligible to humans, we rank MNIST and NABirds examples by their salience score.
To visually compare the prototypes learned by Tversky Neural Networks and classical neural networks, we employ the prototype visualization method described in Section 3.6 and illustrated in Appendix A. This method is used to display the handwritten digit prototypes learned by VisualMNISTNet and VisualTversky MNISTNet, two neural networks inspired by LeNet-5 (LeCun et al., 1998). Further details are provided in Appendix D)
Prototype Visualization:
Semantic Fields: Tversky Neural Networks' representation of objects as sets of features allows the algebraic specification of interpretable semantic fields. A semantic field is subset of features resulting from set expressions that are interpretable by humans. For instance, An C describes the semantic field capturing the common features of A and B that absent from C. To visualize semantic field
F, we retrieve examples 2; with the highest scores in that semantic field: s = Σ1eeF fi I,.
4.4.2 RESULTS
Salience: Figure shows MNIST and NABirds examples ranked from low to high salience according to Equation 2. Higher-ranked stimuli exhibit more object features and greater informational content, familiarity, and goodness of form, consistent with Tversky (1977)'s theory.
Figure 3 shows that Tversky prototypes exhibit stroke patterns similar to human handwriting, such as lines and curves, more clearly than those learned by linear projection layers, which exhibit texture patterns that are difficult to interpret. This lack of interpretability, even in this simple domain without background textures. represents a significant limitation of prior approaches. Tversky prototypes are also more salient than individual instances: they exhibit more features and appear to combine the features of all possible instances of their class. For example, the prototype for the handwritten "7" has both an upper-left serif and a middle horizontal stroke. The prototypes for the digit "1" and "9" include vertical lines slanted to the right and to the left. The prototype for class "9" combines both open and closed top loops. Figure 16 shows the evolution of learned prototypes over training epochs for both VisualMNISTNG and Tversky VisualMN We observed a > B in several, but not all, trained Tversky neural networks (see Appendix F.4). indicating that they weigh the distinctive features of individual instances more heavily than those of prototypes when assessing the similarity of instances prototypes, which is in line with Tversky's theory.
Prototypes:
8

---

Published as a conference paper at ICLR 2026
Visualization of Projection Parameters in Data-Domain
2
3
4
6
6
7
8
Examples Instances
2
4
5
6
1
8
9
|
2
3
4
6
7
8
5
Baseline FC3 Prototypes
Tversky Prototypes
Figure 3: Visualization of prototypes using our input domain projection parameter specification method. Top: 3 examples for each class. Bottom: 10 columns of the final linear projection layer of VisualMNISTTNrt, and 10 prototype vectors of the Tversky Projection Layer of TNvesSy Both models achieve 99% accuracy on the validation set. Handwritten digit features are more perceptible to humans in Tversky prototypes compared the linear projection prototypes.
Semantic Fields:
Tversky Neural Networks' set-based representation enables the algebraic specifi- cation of semantic fields, which we illustrate in Figure 4 across three modalities. Lexical stimuli (A:love, B:like): A n B captures shared evaluative sentiment; A B captures romance and un- conditional love; and B - A captures similarity and comparison. Handwritten digits: Common features of eights retrieve other eights; distinctive features capture six-like lower stroke and an upper arch prominent in twos, respectively. Bird species: Common features of Least and Semipalmated Sandpiper retrieve both species; distinctive features retrieve species sharing relevant visual traits (yellow vs. black legs, drooped vs. straight bill). Table 3 and Appendix B.4, B.5 show that semantic fields spanning adjective degree (positive, comparative, superlative), word senses (plant, sanction), morphology (past tense, plural, progressive), metonymy (White House, Pentagon), analogy (king man:: : ?wwoman). sentiment, polarity and antonymy are expressible as concise. human-readable algebraic set expressions over TverskyGPT-2 token representations.
8
8
6
8 8 8 8
2
2
2
8
8
2
2
Avarity
6
6
8
8
8
2
3
Figure 4: Semantic fields defined as common (A n B) and distinctive B, A) features of lexical and visual stimuli. left: TverskyGPT-2 tokens love and like. Distinctively, love associated with romance and unconditional love, while like is associated with similarity and comparison. Their common features capture shared evaluative sentiment. middle: Two handwritten eights (A) and (B) from MNIST. Their common features retrieve other eights similar to both. A's distinctive features capture six-like lower stroke, while B's capture an upper right arch, also prominent in twos. right: A Least Sandpiper (A) and a Semipalmated Sandpiper (B) from NABirds. Their common features retrieve other sandpipers. A's distinctive features capture its yellow legs and fine drooped bill, while B's capture its black legs and blunt straight black bill.
5 DISCUSSION
Hyperparameter Tuning: Our experiments suggest the existence of an optimal feature bank size for Tversky projection layers for given learning task, though determining this value a priori

---

Published as a conference paper at ICLR 2026
Table 3: Semantic concepts, their algebraic specification using set operations TverskyGPT-2 tokens, and top tokens ranked by semantic score in the resulting semantic field. Tversky neural networks enable the algebraic specification interpretable semantic concepts. (Also see Tables 8-12)
Concept
Semantic Expression
Top-scoring tokens
Adjective
bad n good worse better worst best
lousy, evil, crappy, poor, shitty, nice, terrible, horrible, decent, mediocre, valid, great, excellent
Comparative
worse n better bad good worst - best
sharper, nicer, clearer, smarter, wiser, smoother, safer, hotter, happier, righter, louder, preferable
Superlative
worst n best worse happiest, safest, quickest, deadliest, finest, hottest, busiest. toughest, darkest, farkest.fastest, brightest, holiest
better bad good
Industrial Plant
plant aceae vegetation vegetable planting flower herb vine crop tree mushroom plantation - leaves
facility, stall, brate, factory, Laboratories, Shed, implant, Berm, Sew, Manufacturing, reactor, Diesel, refinery, strain, distribut, planting, Industrial, microbial, actory, reactors, Unit, Install, depot, spill, Cutter, Indust, Bot. Nuclear, ineries
remains an open question. We encourage practitioners to apply standard hyperparameter tuning to further improve performance, especially since Tversky networks are novel architecture, whereas the baselines we compare against have benefited from years of community-drinn refinement.
Convergence: Under particular conditions, some features or prototypes may remain permanently inactive and receive zero gradient throughout training. This phenomenon is analogous to the "dead expert" issue in MoEs (Dai et al., 2024). We illustrated this issue in the XOR domain, which contains only four data points (see Figure 13, bottom row). However, we did not observe this issue in our language modeling experiments, or our image recognition models which showed consistent convergence over several random seeds (Appendix E and F).
Our experiments with GPT-2 on PTB showed that weight tying can be beneficial in language modeling heads employing both Tversky and linear projections. Our interpretation of projection layers as measuring similarity between stimuli and prototypes explains the adequacy of weight tying in language models: token embeddings are prototypes, and the language modeling head predicts output tokens by measuring the similarity of output representations to these prototypes.
Interpretation of Weight Tying Language Models:
CONCLUSION
This work introduces a differentiable similarity function based Tversky's theory of psychological similarity, and derives basic neural network building blocks, specifically the Tversky similarity and projection layers, offering an alternative to the geometric similarity functions that underlie most modern neural architectures. Our experiments show that Tversky projection layers can improve over linear projection layers performance and parameter efficiency in image recognition and language modeling tasks. Tversky's set- bet-based representation also enables the algebraic specification of semantic fields, which are uuman-interpretable feature subsets that support fine-grained analysis of learned representations in any modality. For instance, querying the distinctive features of Least Sandpiper and a Semipalmated Sandpiper retrieves birds with yellow legs and drooped bill, and birds with black legs and straight black bill, respectively. We also propose a novel data-domain visualization method enabling interpretation of learned prototypes and features directly in input space at any network depth. Taken together, these contributions suggest shift in the similarity paradigm underlying deep learning: from geometric comparison toward explicit feature- matching grounded in an established theory of psychological similarity.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Amos Tversky. Features of Similarity. Psychological Review, 84(4):327 -52 1977. ISSN 0033- 295X, 1939-1471. doi: 10.103770033-2255X..A 84.437777 URL httpt::tp////m http::///ddo..o....////////// 0033 295X. 84 34.4 4 327.
John Bridle. Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition. In Françoise Fogelman Soulié and Jeanny Hérault, editors, Neurocomputing: Algorithms, A Architectures and Applications, pages 227-236. Springer, Berlin, Heidelberg, 1990. ISBN 97--8774-44----15555533 doi: 10.11077/978-3-64--7113333 https: /ttp///...///1111////9////////// 3-0
URL
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- Learning Applied to Document Recognition. Proceedings of the IEEE, 86(1 1):2278-23224 1998. ISSN 0018- 9219,1538-2256. doi: 10. 109/5.777776. URL https: /oii wtlddoioorooo wwgg/0..g.0.. 010//..1100...1.......... 726779
Sepp Hochreiter and Jürgen Schmidhuber. Long Thort-Term Memory. Neural Computation, (8):1735-1780, 1997. ISSN 0899-7667, 1530-888X. doi: 10.1162/neco. https:/ /ttps:://do.... ww..orggggg. wtt.g//....16//eeeeeeeee 1997. 9. 1735.
URL
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition. In Proceedings of the 1 15th International Conference on Artificial Neural Networks: Formal Models and Their Applications Volume Part II, ICANN' 05, pages 799-709-999 Berlin, Heidelberg, 2005. Springer-Verie Verlag. ISBN 3540287558. doi: 10.5555/198609..190 1986220.. URL https: /oii. org/10. 555..55555/55 198 6079 9 1986220.
Alex Graves. Generating Sequences With Recurrent Neural Networks, 2013. URL https:// wwt.:oorggbbbss/s.ooeeo
0850.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representa- tions in Vector Space. In Proceedings of Workshop at ICLR, 2013a. doi: 10.48550/arXiv. 1301.3781. URL https::////.. /arxiviiiiv. org/ abs/ 110111111100 3781.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global Vectors for Word Representation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1533-1543, Doha, Qatar, October 2014. Association for Computational Linguisties. doi: 10.3115/ v1/D14-1164 URL http://aaaaaattttaeaeac otgg///-/////11122222010
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A Neural Probabilistic Language Model. Journal of Machine Learning Research, ((febb::137-111: 2003. ISSN 1533-7928. URL hwtph:tt/////..... /twp///www../.mmm/ jmlr .rg/
html.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to Sequence Learning with Neu- ral Networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Wein- berger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL http4//R/tttss///////eeeeonooeot neurips wwwpeeurrssssppppeeeeeeeeeeeeeee cc/paper files/paper/ 220.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19 1423. URL https: http://aaaanntooo.... org/ Arg/N19- M 1423/.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Lan- guage Understanding by Generative Pre-Training. Technical report, OpenAI, 2018. htL //n. www.oo.en.....oooo wttpl//cw///eeeeeeeeeee wwtcchhooerr/eeeee banggaggeuudderrttadinnnng pdf. GPT-1.
11

---

Published as a conference paper at ICLR 2026
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro. Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung. Dave Cummings. Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Auy Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontio-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu. Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, ukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, ukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger. Vishal Kuo, Michael Lampe, Ikai Lan. Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko. Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalie Staudacher, Felipe Petroski Such, Natalic Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak. Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian. Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 Technical Report, 2024. URL https: attp///axxvvv///////////ee//aaav org/ 2200. 207..08774......
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations, 2015. URL https: /ttps///axxvvvvo org/ abs/ /etss/109..000 0473.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: /ttp///eeccccecceeeeeeepppc neurips. ce/ cc/paper. soroerrfflee__ee pappr/2017/ file/ 3.le/ff5eee 3fffee2334444 44ee2dee5eede 91 1 84 Papp .pdf.
12

---

Published as a conference paper at ICLR 2026
Adrien Marie Legendre. Nouvelles Méthodes pour la Détermination des Orbites des Comètes: Avec un supplément contenant divers perfectionnemens de ces méthodes et leur application aux deux cométes de 1805. Courcier, Paris, 1806. URL https: hawp::/a/aallaaaaaaaaa .nff fafffff/ase fr/ark: /121 1.1 8/ /tpttppp///. 6k 5209372.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality Reduction by Learning Invariant Mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (06), volume 2. pages 1735-1742, USA, 2006. IEEE Computer Society. ISBN 0-7695-2597- 100.
0. doi: 10.1 N0.1009/CVR.2206... 100. URL https: : /tiiiiiiooo/// org /10 1109/ CWw..
Hao Wang, Yitong Wang. Zheng Zhou, Xing Ji, Dihong Gong. Jingchao Zhou, Zhifeng Li, and Wei Liu. CosFace: Large Margin Cosine Loss for Deep Face Recognition. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5265-5274, 2018. doi: 10. 1109/CVPR..018.000552 URL www.h:tppps..mmm 2018/htang http:///eeooonnoooaeeeeaaaaaaaaa
httpttpp::///pppacceeeeees thecvf. com/ commmnt_entr_. html.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, 2018. URL
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Hal Daumé and Aarti Singh, edi- tors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020. URL https: https://proceedingg www.mmr.reesseeee v11 chen20j..o .hmml.
Itamar Gati and Amos Tversky. Weighting Common and Distinctive Features Perceptual and Conceptual Judgments. Cognitive Psychology, 16(3):341-370, 1984. ISSN 0010-0285, doi: 10100160011--03544490000-------1 URL https: /twii//oiio/// /ttp://.......111111000001000000 (84) 90C13-6.
Ilana Ritov, Itamar Gati, and Amos Tversky. Differential Weighting of Common and Distinctive Components. Journal of Experimental P Psychology: General, 119(1):30-44 41, 1990. ISSN 0096- 3445, 2922. doi: 10. (0.1007/0096-3444 119.1...... URL https: http://oo.ooo....000// 01166-3445119.....
Paul S. Siegel, David M. McCord, and Alice Reagan Crawford. An Experimental Note Tversky's "Features of Similarity". Bulletin of the Psychonomic Society, 19(3):141-142, 1982. ISSN 0090-5054. doi: 10.3758/BFO333022. URL sattps://Got wtt.:oii....oooooooooooooooooooo wttp:///....////03333333333
Abebe Rorissa. Relationships Between Perceived Features and Similarity Images: A Test of Tversky's Contrast model. Journal of the American Society for Information Science and Technology,
58(10):1401-1448, August 2007. ISSN 1532-2882. doi: 10. 10.1002/asi.20606 URL https:
Ankita Garg, Catherine G. Enright, and Michael G. Madden. On Asymmetric Similarity Search. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pages 649-654, IEEE, 2015, doi: 10.1 00.109//CMLA22015..222 URL https: /ttp:///oo....//111///1019/// ICMLA. 2015. 128.
Dawn Chen, Joshua C. Peterson, and Thomas Griffiths. Evaluating Vector-Spare Models of Analogy. In Proceedings of the 39th Anmual Conference of the Cognitive Science Society, pages 1746-1751. Cognitive Science Society, 2017. URL htwps:ttp///eettm 04416.
Joshua C. Peterson, Dawn Chen, and Thomas L. Griffiths. Parallelograms Revisited: Exploring the Limitations of Vector Space Models for Simple Analogies. Cogmition, 205, December 2020. ISSN 0010-0277. doi: 10. co0.//..ogniiion.2000 wwt.2022..14400000 URL https: //oii. org 10160/0..... cognition. 2020. 104440.
Kaitlyn Zhou, Kawin Ethayarajh. Dallas Card, and Dan Jurafsky. Problems with Cosine a Measure of Embedding Similarity for High Frequency Words. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association
13

---

Published as a conference paper at ICLR 2026
for Computational Linguistics (Volume 2: Short Papers), pages 401 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 111666//112222222llssstrr URL https: httpp:s//alaantolooyy wwt.oogg//// www.......soo........e shorr.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning Representations by Pack-Proppggting Errors. Nature, 323(6088):533-555) 1986. ISSN 0028-0836, 1476-4687. doi:
10.1038/323533a0. URL. https::////. / /doiod.../// wwo...og//11. C 0233/33333330.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learn- ing for Image Recognition. In 2016 IEFE Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 770-778. IEEE, 2016. doi: 10.1109/CVPR.2016.2000 URL BtLLtttss//oppaaceeee tttpppeaccccsss.eec......eeeeeee thecvf. com/ Desscuual_eerrnnng_cceee paper. html.
Seyed Sadegh Mohseni Salchi, Deniz Erdogmus, and Ali Gholipour. Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks. In Qian Wang, Yinghuan Shi, Hcung-ll Suk, and Kenji Suzuki, editors, Machine Learning in Medical Imaging, pages 379-387, Cham, 2017. Springer International Publishing. ISBN 978 3-319---3338388 9 doi: 10. 978-3-319-67389- 44. URL https:: htpps://do.. wttp://.......1///////////////// 0-1---11007777----1---7-33--3-3- 44.
Javad Rahnama and Eyke Hüllermeier. Learning Tversky Similarity. In Marie-Jeanne Lesot, Susana Vieira, Marek Z. Reformat, João Paulo Carvalho, Anna Wilbik, Bernadette Bouchon- Meunier, and Ronald R. Yager, editors, Information Processing and Management of Uncertainty in Knowledge-Baie Based Systems: 18th International Conference, IPMU 2020, Lisbon, Portugal, June 1519, 2020, Proceedings, Part II, volume 1238 of Communications in Computer and Information Science, pages 269-280. Springer Nature Switzerland AG, 2020. ISBN 978-3- 030-50143-3. doi: 10.1007/978 3-030-5010----33 21. URL htttps://dot http://dooo.rg11.100000000 978-3-030-50143
10.1077998-3-000-5014433
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive Mixtures of Local Experts. Neural Computation, 3(1):79-87, 1991. ISSN 0899-7667, 1530-888X. doi: 10.1 10.1162//eco.19911
URL https::::: /doi. wot.doooooooooooooooooooooooeeee wog/ 11t.111122mccoooo 1991. 3
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer In International Conference on Learning Representations, 2017. URL /tttps://peerrviieee net/
ckMDqlg.
William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 23(120): 1-39, 2022. ISSN 1532-4435. doi: 10.5555/3586589338877777 URL https: / jmmrr...... org/ papers/v23/21-000 / 0992221--999001 html.
Sukjun Hwang, Brandon Wang, and Albert Gu. Dynamic Chunking for End-to-End Hierarchical Sequence Modeling. In The Fourteenth International Conference on Learning Representations, 2026. URL https: : http/pppnreeeeeeeee net. forum? 1.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This Looks Like That: Deep Learning for Interpretable Image Recognition. In H. Wal- Beygelzimer, F. d'Alché-Bue,
lach, H. Larochelle, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., /
Fox, and R. Garnett, editors,
2019. URL https: htttps://pzeet httt:///ptttpttccppttct/t/c/sest neurips. ce/ Pwwpee5555pppppppppp pdff
Meike Nauta, Ron van Bree, and Christin Seifert. Neural Prototype Trees for Interpretable Fine-grained Image Recognition. In 2021 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14933-14943, Los Alamitos, CA, USA, June 2021. JEEE. doi: 10.1 N0.11099CVPPR4443.222...000 WweRo02l/stmm//aaaraaeeeeee Neural wttprraallpttttppeeeeeeee Trees Trees_for____e for for_Interpreetblleee paper. html. html.
URL https: httpss:/oppecccesssss theevf com/ content/
Fine- Gra i ned Tmage @ org.reeeeee....i.......aoooooooo
14

---

Published as a conference paper at ICLR 2026
Tomas Mikolov, Wen-tau Yih, and Geoftrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff, editors, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 746-751, Atlanta, Georgia, June 2013b. Asso- ciation for Computational Linguistics. URL htwph:ttp//.......... a anthology. org/ N1 3-10099--
Piero Molino, Yang Wang, and Jiawei Zhang, Parallax: Visualizing and Understanding the Semanties of Embedding Spaces via Algebraic Formulae. Marta R. Costa-jussà and Enrique Alfonseca, edi- tors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Sys- tem Demonstrations, pages 165-180, Florence, Italy, July 2019. Association for Computational Lin- URL ht https: / ac aalaannthhogyy. org/P1 900199-320000 3028/.
guisties. doi:
Muhammad Ali, Yan Hu, Jianbin Qin, and Di Wang. Antonym vs Synonym Distinction using InterlaCed Encoder NETworks (ICE-NET). In Yvette Graham and Matthew Purver, editors, Findings of the Association for Computational Linguisties: EACL 2024, pages 1462-1473, St. Juliam's s. Malta, March 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/2024 findings-eact.99 URL https: /tt///clannttt http://aaaaaatttoooooooo
org 2024 Findings-eacl 99/.
Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. Retrofitting Word Vectors to Semantic Lexicons. In Rada Mihalcea, Joyce Chai, and Anoop Sarkar, editors, Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1606-1614 Denver, Colorado, MayJune 2015, Association for Computational Linguistics. doi: 10.3115/1/N15- 11844 URL https: / aacl anthologoy www.ggg....oggooooooooo - 1184/.
Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašié, Lina M. Rojas-Barahone Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. Counter-fitting Word Vectors to Linguistic Constraints. Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 142-148, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16-100 aclanthology. org/N16 6 1018/.
URL https: //
Masataka Ono, Makoto Miwa, and Yutaka Sasaki. Word Embedding- based Antonym Detection using Thesauri and Distributional Information. In Rada Mihalcea, Joyce Chai, and Anoop Sarkar, editors, Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 984-989 Denver, Colorado, URL
Maylune 2015. Association for Computational Linguistics. doi: 10.3115/v1/N1s https: / http/aclaaaa/ aalaaaaannnoooyyyyooo
Amos Tversky and Itamar Gati. Similarity, Separability, and the Triangle Inequality. Psychological Review, 13-2)12 1982 ISSN 0033-295X, 1939-1471, doi: 10. 10.103770033-2958.89222 URL http:: /tt::///ooi../ http:/oi......g///// 1037 2tt.....1110000003553133 89 123..
Robert L Goldstone. The Role of Similarity in Categorization: Providing Groundwork. Cognition,
1252)11 1994. ISSN 0010-0277, 1873-7838. doi: 10. 10110660000--22779990006----5-5 URL
hittps://dof 0ttp:///..../1..0000000000000000 90006555-
(94) 90065-5.
Douglas L Medin, Robert L Goldstone, and Dedre Gentner. Respects for Similarity, Psychological Review, 100(2):254-778. 1993. ISSN 0033-295X, 1939-1471. doi: 10. 00.10770003--955X.0022.22 URL
/100g//1..00 1037 00077/0033-2555 2. 254.
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying Word Vectors and Word Classifiers:
Loss Framework for Language Modeling. International Conference on Learning Representa- tions, 2017. URL https:://// http///ppeeeeeieeeeeeeeeee .nt/ wor/moormmmmmm
Matthew D Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. In European Conference Computer Vision (ECCV), pages 818 1-38833.000 Springer, 2014. doi: 10.1007/ 978-3-319- 10590-1 10590-1_53. URL https: doi w0t. 3-1---1059----1-------1-11-11111 10590-1 53.
15

---

Published as a conference paper at ICLR 2026
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going Deeper into Neural Networks. Google Research Blog, 2013. URL httpa:////reeeeeee oot Accessed: 2026-03-01,
Quan-Shi Zhang and Song-Chun Zhu. Visual Interpretability for Deep Learning: A Survey, Frontiers of Information Technology & Electronic Engineering, 19(1):27-39, 2018. ISSN 2095-9184, 2095-9230. doi: 10.1631/FITEE.1 1700808. URL / 1700808.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks Gradient-Based Local- ization. Internatiooaal Journal of Computer Vision, 128(2):336 3590866--55,99 2020. ISSN 1573-1405, doi: 10. (007/s|
URL https:th/.... /twi/iiio///o otg.:/oo.ooooooooooooooooooo /10 1007 wwt.://///111111110
01228-7.
Qiyang Zhang and Dong Liang. Visualization of Fully Connected Layer Weights in Deep Learning CT Reconstruction, 2020. URL https:t/arxii...tt
Feng-Lei Fan, Jinjun Xiong, Mengzhou Li, and Ge Wang. On Interpretability of Artificial Neural Networks: A Survey. JEEE Transactions on Radiation and Plasma Medical Sciences, 5(6): 741-760, 2021. doi: 10. 10.1109/TRPMS 2021.35622223336442 URL httph: http:s:://oo.oooooe http:////oooooo/..1111111 TRPMS 2021. 3066428.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. Technical report, OpenAl, 2019. URL https://cddi openai. com wow/beetee-laaeeeeeeeeeeeeeeeeea languuueeeeeeeeeeeeeeeeeeeeeeeee saaeeeemolllssaaasne mode 1s
pdf.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313 1993. ISSN 0891-2017. URL httph::////// /atsss//alllttee /aclanthoooooyyoooooooooooooolaa .rrg J93 2000/-004..
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessic Barry, Panos Ipeirotis, Pictro Perona, and Serge Belongie. Building Bird Recognition App and Large Scale Dataset With Citizen Scientists: The Fine Print in Fine-Grained Dataset Collection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 595-604, IEEE, 2015. doi: 10.11099CVPR.2055.29966
URL https::/opeeacceeee cvpr_2015/htmll /aorsemmmooooBBBur http::aaa.....nan............... html.
wtw.s/ccccsss....te.eee tww.cfeeee..cco wwt.c/ooccoooooooo content
Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang. Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Expeee Language Models. In Lei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280-1297, Bangkok, Thailand, August 2024. Association for Computational URL // ho ogy org//024.
Linguisties. doi: 10. 10.18655//1/2224.alll acl-long. 70/.
16

---

Published as a conference paper at ICLR 2026
A
ILLUSTRATION OF DATA-DOMAIN VISUALIZATION OF PROJECTION LAYERS
Projection Parameters Specified in input Domain
Input Data
pd
Layer
g(.
Layer n-1
P: -(p!
Projection Parameters
Computed Projection Parameters
Output Projection
Figure 5: Illustration of proposed data-domain visualization of projection parameters. Left: Classical deep neural network with a deep projection layer (Layer n) and its projection parameters illustrated as blue vectors. Right: Our proposed method. Projection parameters are specified as tensors of the same shape as the input data (p,"), and forwarded through the neural network up to layer n-1 (the g(.) sub-network). The obtained vectors = g(p?) are used as projection parameters in Layer n. Using this method, the effective neural network parameter count is larger because pf is typically larger than p.. However, this method enables visualizing the projection parameters in the input domain.
17

---

Published as a conference paper at ICLR 2026
B EXPERIMENT 006: TTVERSK/JPPT LANGUAGE MODEL ON PTB
B.1 INITIALIZATION: OPENAI RELEASED WEIGHTS
Validation Perplesity
diflaenne
igamcmmtch
sulstractunnch
intereaciion
min
puodact
min
pusluc
Inin
modkl
lic-peto
PP1.
PP1.
P95.
101.
PPL
PPI.
P91.
PP1.
F91.
PPL
frntuned
haseline
169097184
$4.85
14.85
$4.85
34,85
$4.85
34,85
34.85
34,85
34,85
34,85
Irstuned
bonnky-hend
1034
165033610
5244
2011
$1.30)
57.54
76.73
61.59
51.90
58.15
55.37
120122
linduned
bonkyyheud
2045
164610051
20,92
54.44
41.23
46.79
53.
48.71
41.28
47.91
66.05
86.26
lisduned
N
4055
166182915
34.47
38.52
34.44
40.32
40.32
42.82
34.54
41.39
64.33
$7.10
fintuned
henky-hend
8192
169326643
32.56
32.80
32.53
18
54.
38.89
33.49
39.09
61
48.73
fineuned
tomky-hadd
12388
17242477
$2.84
12.92
12.82
40.52
52,
41.15
92.81
41.39
00.34
$2.26
fintuned
bonkyyhend
16784
175620990
33.62
36.33
33.62
99.75
-32.1
40.71
35.63
40.67
77.18
$2.52
linduned
bonky-heud
32768
188283611
38.35
40.34
38.98
49.35
33 94
49.45
38.79
47.17
109. 100.10
62.57
lindunned
hennkyull-laya
1004
11-232399
345.50
308.61
142.82
154 144.14
219.63
172.13
144.59
155.65
638
297.61
linetuned
benky-al-laap
2045
115015791
318.23
172.70
115.15
130.06
166.49
121.00
114.88
109.46
257 50
253
linetnned
benky all Hayer N
4055
116591655
95.05
143.67
95.51
121.15
124.50
122.75
95.79
121.51
157.91
203.5 54
linetned
weaky all Haaver N
8192
119737363
91.40
94.86
91.44
93.09
99.75
93.14
91.45
93.55
149.20
142.79
lnnetieed
weelyy all Hayer N
12288
122683111
90.39
94.49
90.79
92.54
94,00
92.95
90.74
92.40
147
133.41
fnnetned
iemily all laaye N
16384
126026839 59
90.51
96.41
91.22
95.39
91. 08
95.64
91.25
95.06
156.30
finetined
ivenyy Deniy-all-layeet layer N
32768
13861175
95.00
101
96.44
100.88
92.63
Table Validation perplexity comparison with init=finetuned and tie-proto=N.
Validation Perplesity
differznce
Igtorematch
substractmatch
intersaction
m133
muann
min
product
gresen
mass
min
product
model
tie-peeto
frcturos
params
PV1.
F11.
PH1.
F91.
fnntuned
haseline
398439008
119.99
'19.99
'19.99
119.99
119.99
'19.99
'19.99
$19.99
'19.99
Roenned
Ivenly head
Y
1024
125226243
90.22
39.25
56.30
61.32
09.65
61.29
37.66
61.08
$1.07
94.75
Roeeneed
ieesly head
Y
2008
126012675
59.95
25.69
56.18
67.96
43.19
71.83
57.38
68.19
46.20
55,05
Rneeneed
merlly head
Y
4095
127585539
23.26
27.15
25.01
24.54
23,91
23.37
24.08
23:22
30.54
Aneeneed
memlly hezd
Y
8192
130151267
21.17
22.31
21.25
21.44
22.37
21.43
21.25
21.30
22.03
23.24
Anneuned
menky--hed
12388
133829895
22.64
21.09
21.33
20.90
21.32
08
21:35
21
21.81
Anneneed
memky-hrad
16184
192022733
21.86
20.85
31
20/77
21.14
20.85
21:12
22.98
fintuned
nomky-herd
149505635
fintuned
beniyellllllgg layer
741.72
finsunal
benskyall-lapp Hayer
2048
finstnnd
besky-lll110
76.37
linetned
wesky-ul--
8192
69.75
117.50
lineuned
weskky-ll
75.85
lineuneed
ivesky Hayer
78.85
100.12
ivesly Hayer
100014375
Table 5: Validation perplexity comparison with init=finetuned and tie-proto=Y.
18

---

Published as a conference paper at ICLR 2026
B.2 INITIALIZATION: RANDOM WEIGHTS
Validation Perplexity
ditference
Igtoremanch
suboraztmacch
Insersection
greea
IDAX
steat
min
product
gieca
mIX
min
product
Lait
model
Uè groto
Teatures
PPL
P9L
PPL
PPL
PPL
PPL
PEL
PPL
PPL
PPL
scrnchh
baseline
163057184
134.06
134.06
134,06
134.06
134 06
134,06
134.05
134,06
134,16
134,06
srnnch
bamky-hhrd
1034
1638236 9
146.
218.46
146,
157.30
184. 36
155.25
145.65
158.31
495
168. 5D
scnach
bunky-head
2348
164610051
147.91
192.12
148.02
145.72
171 54
155.64
147.64
145.66
$20.49
140.48
scnnch
4396
166182915
168.12
161 04
167.65
176.67
162.97
157.44
167.05
155.93
147.83
133.47
heod
8192
189328643
210.0
204.28
217.75
182.50
175.63
182
215.21
182.47
205.55
154.21
scanch
temik--hood
12288
173474571
227.80
178.52
241.49
212.82
186 96
212.26
236 236.99
212.04
203.83
srnnch
bamky-hrou
16384
175620069
220
202.26
250.62
228.47
213.85
231 28
248
231 A9
228, 31
200.96
scnach
benkydhcuu
52768
183201011
182.45
162.13
190.04
190.92
252.01
198. 72
187.12
192.09
223.08
218.1
114232559
138.92
127.19
347.79
nenig--al--Ilg
2016
115018791
121.05
135.35
120029
125.96
130.72
126
120.58
128,17
163.99
24%
scnnch
remkg-all-llpe
4086
116591655
128.89
127.62
121.11
125.97
117.59
134,16
121
133.86
136 27
225
sennch
remky-lll-layrr
8192
11975753
128. 21
134.91
129.87
176
118.05
136. 73
179 459445
136.31
155
140.68
sencch
renky-all-layer
12288
122883111
130.54
131.29
152.46
127.68
125.05
122 80
132.42
122.72
159.
157.27
mersly
16.884
1250288299
128.37
131.9
128.01
163.26
scrnnhh
weniky-all- Ilyerr
32768
1336 751
1
132.35
130.26
129.51
136.46
Table 6: Validation perplexity comparison with init=seratch and tie-proto=N.
Validation Perplesity
difference
igtorematch
substroomacch
interuation -+
prrezen
mizn
min
prsduct
gracen
max
mrn
min
prodher
Init
model
bi- Re-prrte
Featuree
parmms
P91.
191.
1V1.
195.
PPT.
191.
PVL.
PPL.
191.
P9L.
scranch
baseline
1244 99906
136.04
136.04
136.04
136
135.04
136.04
136.04
136.04
136.04
136.04
Reenky lead
1024
12515:33
402.21
213.07
118.51
155.58
192.72
135.15
149.01
135.25
3011255
271.20
scrach
mvesky head
2048
125012615
147.91
179.89
148.12
145.80
109.92
155.
147.83
145.83
175.95
154.26
scnach
bunkky-hual
4096
127585539
167.44
168.53
168.55
150.11
161.85
151.59
168.00
140.89
152.11
145.52
senach
bunky leaa
8192
130731267
197.06
207
39
204.79
181.11
168. 20
179.76
201.95
190.99
183.96
156.57
senach
benky-
12258
133876995
212.50
211.17
223.91
202.53
179.87
204.57
221.02
202.63
214.04
163.64
scnach
Werky head
16384
137022723
210.90
152.18
227.04
217.16
204.55
214.09
226.02
216.75
238.05
scrach
werky head
32768
149605635
162.49
151.25
17L.88
176.24
216.54
scrach
wesky all Hayer
1024
75634983
142.00
176.76
136.77
152.44
182.22
154.00
138.65
152.51
250.44
389 389.16
smach
benkydll-laps
2048
76421415
152.41
141.77
130.55
136.58
146.27
144.
130.90
137.51
180.40
352.45
knach
renky -laser
4096
77994279
129.4
157.46
129.64
131.57
128.54
130.26
129.66
130.6]
146-93
175.75
snach
nenky all Haaver
5192
#1142007
156.73
145.4
139.16
125536
125,05
125.95
139.13
125.96
151.85
151.07
snacc
mesky all Hayer
12268
#4285725
140.86
156.1
144.50
129.38
134.95
127.85
144.51
125.45
156.44
140.26
stath
mesky all Haver
16384
87431400 403
143.59
160.99
148.72
130.69
140.35
129
148.77
130.50
153.27
staach
meekly all Hayer
32768
(000 100014335
154.05
140.75
150.33
Table 7: Validation perplexity comparison with inif=scratch and tie-proto=Y.
19

---

Published as a conference paper at ICLR 2026
B.3
COMPUTATION TIMES AND RESOURCES
(Pracess) v. tversky. lwwheadmeaaaaieeeeeeeeea featuree_oonnt
50300
Rola Time (Pocee
10300
20000
10200
tvensky festure
2
4,00
6,800
8,800
10,00
12,00
14,80
18,000
Figure 6: Sample of wall-clock times in seconds for TverskyGPT2 experiments PTB function
of the feature bank size. 2 GPUs were used for each run. GPU models varied. They included NVIDIA 1100-XXM4400GBB NVIDIA HI00 80GB HBM3, and NVIDIA RTX 6000 Ada Generation.
20

---

Published as a conference paper at ICLR 2026
B.4 VISUALIZATION OF SEMANTIC FIELDS IN TVERSKYGPT2'S SET-CENTRIC TOKEN REPRESENTATION
Figures 7.8 ,9 .10 and 11 show examples of visualizations of semantic fields formed using set expressions on TverskyGPT2 tokens represented as sets. For each token, the top 1000 features (sorted by dot-product with the token) are considered, and represented as the smallest colored circles. Tokens are represented as yellow circles, and connected to the features they comprise with grey lines. All distinct intersections of features are colored the same color, and connected to a circle of the same color representing distinctive semantic field. Semantic fields of interest are annotated with a text box listing the top 50 tokens by semantic score (See Section 4.4) in that semantic field, permitting its visualization. A force-directed graph layout is employed to position the circles corresponding to tokens, features, and semantic fields. Note that the character (U+0120) represents space in GPT2 tokenization, distinguishing tokens that appear at the beginning of words from other tokens.
TverskyGPT2 Tokens: Orthographic Markings, Homophony, Polysemy
Bound Morpheme (e.g. baking,
Royalty, Upper Case
baked, bakers, baker, bakes)
c. 2077
GEmperor Strptoes OPriios sines Couitan doals 0reas Guare, Grule GEmpras, Cas.
ken, KER, ching, kis idns, ides, pirig, iy nets, lorra, soen, treading, chus, geres doms, King, liet, cating, sared, dhing, rith, ides, frides, laky, time, ling. hing pres.
fires
dduca-1 Ouenees Galcsendes donngdon celsings dhtankey, Mastic Gerven can.
ÖKing
deright detigeion dolans, Orteccules
king
Royalty, Lower Case
Religious Royalty
C.n. 5322
Oemsena, soutt, trame dpriess
Kings Kavi Quees draNd, Okin, Kneht Saudh kobesit Maker Rsterend, Jastic Lants
recrs
Osees anse Otyres Gumpdoera
Davies,
Dcimppar, States oigatzun, Roderts Grings Ditagdaans Land Bore, York ditrr in, lngend, Revillation,
King
Usis, drswee, sing. Olaen, dachat diees, Goyeant peror des, Ginegan, achat uttans
Gking
Figure 7: Visualization of the semantic fields formed by the distinctive features related tokens. Notice that Tve2 tokens capture semantics related to orthographie markings, and morpho- logical function. The king token (as in baking) is related to other suffixes such as ked, ker and kes (as in baked, baker and bakes)
21

---

Published as a conference paper at ICLR 2026
TverskyGPT2 Tokens: Polysemy
Gegre Galke,, Guniiko, Gauspic, ccording, Gnodd, Gconvol, Gmathemat Giiked, Gchalleng, such. Chorizont, ometimes, Oshenan, Omoseu, Gproble, Gincent, 014, Gdestro, Gtraged, esque, Uniiee ttes, Gtyr, Gsuch, Gdescrib, Gdefic, 8x-,, Gvuiner, Gresembling, Gsacrilic, Ganaloazuus, Opromot, Gdisadvant, thumbnails, Gdistingu, Gjuven
Glike
Bloves, dloving, GLoving, Gloved, Coffection, Opassion, Giover, Gdespise, Gromance, Ghato, Glovers, Ghates, GRomantic,
diove, BLOVE,
Greverence, Ghatred, Gcherish, Glisses, Greaouuy,, Gdevation, Ghating, sual Gdialia, Gai, Griate Gerave, Giescination, Cadultary, Okissing, Opassions, Ganvy, Clov, dlonging, Gadiiiation, BLover, Caiss, Clust, Orriondshhpp &Desire, marriago, Ogratitudo, Opatriotsm, Gtrave
Glove
Figure 8: Visualization of the semantic fields formed by the distinctive features of tokens like (like - love) and love (love like). like, distinctively from love, which also captures the sense of sentiment, captures the sense of similarity, with tokens such akin, alike, unlike and reminiscent.
TverskyGPT2 Tokens: Verb Forms
Past Participle
Infinitive
Ogann
aee diuener tin salts On
Ggo
Ond Cdurrs, Seaw Chrant, Duins Oreee dene chain
rakitan Osena, thiius Ind siiht ene Saree devernen
Ggone
dreart gre Goome, dity dctiderr Ovees, Center Soers, Chaees Gursernt, Shepp Gary Bodate Dave repariy Chappin
ADRA Octiters Unetcatee Dlays
Öwent
Ggoing
Past Tense
Present Participle
Giscina d8 Oacing Ohion Coverato,
Opinng Ogenns going douing Da Creassmme Sentual Camene, Coatherrs
devied apan, parsund
Gaareed Coneress Grawiled Gertdree danx
Grtspping sataiig diaiing tee Buarred dis-ein Scaskic Greeng Sooing Giihhg, Sint bryes.
Butaran fggrting Muonder dlouding
Goonve
Garm verra Gits, inoeet
Drown, Derr
Figure 9: Visualization of the semantic fields formed by the distinctive features of tokens go (go gone went going), gone (gone - go went going), went (went - gone - go - going), and going (going- gone went go). respectively specifying the concept of verb forms infinitive, past participle, past tense and present participle.
22

---

Published as a conference paper at ICLR 2026
TverskyGPT2 Tokens: Degrees of Comparison of Adjectives
Adjective
Superlative
Coec: Cgecs, bed good, GOcDo, doed, Goced.
Gnoes, Otherss, Goanden, Pec; dafitty celarss
droppiee doeet, dwaas
foce lert, denceus Oivil agn Biseghty
drrest, oar, Rec, Rasociet
Gaurkos Olaito (handest Oprastet
dicces, Duttes Ordundder stren, deseloorn
Gbad
Oynen, Gecolan, Gmaptic sU
Gooclen Oatrngest Goweset Cheanies
Gbest
Guidert Cortatent, snees Gohcrnet Gralest,
Ggood
Grars cans, Gitnpaas ats drodlct,
Gworst
Comparative
Gworse
Conepee Criwe Sceare Camatie: Dutees
Souicies Grigher Castoggrr Spoore, Grudee
Gbetter
decraaned ansr Gualee, doigges Gparkrabls, draster drines oother deeathiee Gttickes d6611 dessie, diaares bock andic Giacker
Figure 10: Visualization of the semantic fields formed by the distinctive features of the common features of tokens bad and good (bad good - worse better worst - best), worse and better (worse n better Weettrr worse - better). These semantic fields algebraically specify the concepts of adjective, comparative, and superlative.
bad good worst best), and worst and best (worst n Nbest bad good
TverskyGPT2 Tokens: Distinctive Features of Synonyms
6.0,4101
Glescher Gteschers, CTeacters, Gdassmate, student Student, Gclesenomms Gfaacting, Studeres, &cisomen, Geretructons, deccher 6ana: Ginstruttt 6dtudents, intendent, dprofessors, Oclassmatts, 6student, Parents, Gtzetbooks. dstuders, dkindengatton, Opratessor, Beurticuunn sheel, Cleir CFucuity, Coupuiittandert, dSchcols. Charenta Casperineddrt,, cdue. Clegiaator.
Öteacher
Gcoach
5985
Gznech, Coach, dCarch, Ganaches, Gooeching, Gcoaches, dtranee Gonammate Football Otcammates, ESMN kihick, Sport, Train OHtar bough,
Girere: Gtramers, Players, Gluilaeds Gamhet,
Gatuders,, Gtheraper
dcoptais, juries diinem Bosordinator, OTeams, Garummer, Gacorebcerd, Gaalccecck GMourinh, Greferen, Sports, lain Ogoalls, Ogoaknepp: dscorer. Oberth, looes, GFtayott, dJancio, lopp, dSPORTS, CToura, cheat, Gdefemaeman
Gpidde, Gguides, guice, GGuide CGuides, Gquiding, &Comearion. Oguidance, dcomesss, Obookier Ocheddii Groadmap. Obroch, Cguid, diMaraal Giatunal, boud, Gaccomp, acsontzen, DGu delines DrsgonMingarmm Gpamprit, GFAG, insetie, nuls Gprmee, Ósafepuant Opridefres, denligtt, OMarifest, Gtrave, finder. Gencychpedia. Descort, Omeenee, Gaccompany, Gratructions, ddirections, escription. Spoiler Goompanian
SJ9,5008
Gmentor, Omentors, Ctutor, Gment, Goalaabartoo,, Gdiscple, Geiesamata, Gteammate, conservancy, ahine, Gmuse, Gasprentioo, Gadvisor, avorite OMent, DragonMagaaine, Gadvas, GLeian, Ginem, SuysblanstorenndOiii urus, Ogunu. Groommate soDeiveerDDat, rentices, Greet, 45x. clus, hetti, Ginds, HHUD, doanlid, Gmoses, tackle, umbledore, Gadvisors, &colleague, nen, Gunemplay, Geonrada, ModLoeder, Ohikk
22_822377
Gmentor
Gguide
peer,
Figure 11: Visualization the semantic fields capturing the distinctive senses of close synomyms couch, leacher, mentor, and guide. Notice that the distinctive features of coach (coach- Leacher- mentor guide) shows terms related to sport such as trainer, teammate, Football and ESPN whereas the distinetive features of teacher (teacher coach mentor guide) show terms related to formal education such as educator, classroom, instructor, lecturer and professor.
23

---

Published as a conference paper at ICLR 2026
Sayy Ouha
fhe crree Free cooid Shi Sanccss
poas 2ins the Esanci de
ns duabiiy
sutAA raann Wing stubi Rlwnn
thg
Gman
Coircs.
paris R00W, sn' deio dan
Daee
piyvl Overs park Qunn port
Dan, lend Theiu Feen (V
Cffuuee Oniete
Gking
denoea, dbartst dedtris Shati hale
ory Beng See Dalree
sulyy deyel Stagles,
Gqueen
Gwoman
One eeatt in Caltoe timer Gerts
Saenk ouaens Quen 20 wee Carvers
Ohe Fyeend Bedkas Dvar
Soreern wooe sale Gantromess Charoen
Canterss
soreel Coloeen
Calees Cameens Clomm eno
ling can 230ps Mtryet
Visualization of the semantic fields capturing the distinctive feature of tokens man, woman, king and queen, and the distinctive semantics of the common features of man and king. man and woman, woman and gueen, queen and king, and man and woman.
Figure 12:
24

---

Published as a conference paper at ICLR 2026
B.5 ALGEBRAIC LEXICAL SEMANTIC EXPRESSIONS OVER TVERSKYGPT2'S TOKENS
Tables 8-12 present semantic fields spanning a broad taxonomy of concept types, algebraically specified as set expressions over TverskyGPT-2 token representations.
Table 8: Examples of semantic fields TverskyGPT-2. Part 1/5: Morphology and Lexical sense
Category
Concept & Expression
Top-scoring tokens
Past tense
walked, ran, wandered, crawled, Became, smelled, stumbled, persisted, danced, prevailed, sprang, resided, risked, exited, sailed, overcame, froze, toured, apologized, leapt
ran n walked - run walk running walking
Morphology
Progressive
walking, running, running, Running, Running, wandering, walking, crawling, Walking, marching, barking, rocking, whining, campaigning, kicking. stumbling, driving, hiking, touring, hopping
-ran rran-nakedddddded walked-
run walk
Plural noun
cats, dogs, puppies, kittens. Dogs, dogs, pets, tigers, pige, Cats, chimpanzees, lions, goats, cats, Animals, Horses, reptiles, wolves, dolphins, wolves
dogs O cats dog cal
Negation prefix
improbable, misunder, unfair, unlikely, suspic, unethical, unusual, unsus, Seym, ModLoader, unexpected, uthor, unforeseen, theless, aution, unorthodox, untled, icult, effff, unnatural
(umlikely likely) n (unfair fair) (unusual usual)
Like (comparison) unlike n like dislike
unlike, like, LIKE, Like, like, Like, Unlike, Unlike, ccording, akin, reminiscent, agre, incent, whereas, eatures, ecause, Seym, nodd, sacrific, tery
Lexical sense
Like (Evaluation) like disikee unlike
dislike, LIKE, disliked, liking, like, like, reluct, mathemat, Like, Like, despise, disdain, suspic, blat, agre, comr, hating, disapprove, disadvant, ometimes
Sanction (permit) sanction punistuments
sanction, sanctioned, sanctions, enance, sponsorship. sponsoring, spons, ction, displayTest Text, legitim, č, sponsor, BCC, authorizing, decree, patronage, awarding
Grant,
Prosecut disapproval
shaming disband
sanction, sanctioned, sanctions, sanction approve - permit allow authorize
Sanction (punish)
punishments, displayText,, Prosecut, discipl, disadvant, disapproval, sovere, č condem, shaming, reperc, tsky, disband, ENDED, enance, retribution
25

---

Published as a conference paper at ICLR 2026
Table 9: Examples of semantic fields TverskyGPT-2. Part 2/5: Metonymy and Analogy
Category
Concept Expression
Top-scoring tokens
White House (government)
House, White, White, House, WHITE, Senate, Houses, Senate, HOUSE, Oval, Congressional, Speaker, artisan, Democrats, Capitol, Commons, congressional, Republican, Democratic, Congress
Metonymy
WhitehHouse building home
residence archilecture
Hollywood (film industry) Hollywood Cali fornia city neighborhood - subarb
Hollywood, ollywood, Film, Spielberg, Actor, Movie, movie, Movies, Movie, Disney, Celeb, MGM, filmmaking. cinematic, TMZ, Kubrick, Cannes, filmmakers, Oscars, actresses
Pentagon (military) Pentagon - shape geometry building architecture
Pentagon, DOD, Military, Mattis, USAF, Lockheed, Kremlin, Guant, neocons, Defense, CIA, psey, CLASSIFIED, anamo, Petraeus, FBI, Defense, Clapper, Army, Kissinger
king:man ?:woman (king man) n woman
king. Woman, queen, Queen, Duchess, princess, woman, Woman, Queen, Lady, ournal, woman, queens, women, goddess, female, Women, foundland, kings, heroine
Analogy
?:China (Cairo Egypt) China
Cairo:Egypt:: ::
Cairo, Beijing, China, Shanghai, China, Chinese, ijing. Chinatown, Manila, stanbul, Sina, Riyadh, Beirut, Xin, ongyang, Havana, Guang, Islamabad, Moscow, atown
Cairo:Egypt ::
?:Brazil (Cairo Egup() Brazil
Brazil, Cairo, Brazil, Janeiro, Rio, Brazilian, Sao, querque, Spain, Lisbon, Uruguay, Buenos, Paulo, iago, razil, Paris, Mexico, Bras, Urug, Manila
?:woman (actor man) O women
actor:man ::
actress, actresses, Actress, actor, Actor, Woman, Actor, woman, heroine, prostitute, vati, comedian, dancer, filmmaker, actors, emale, woman, contrace, xual, girlfriend
doctor:hospital :: ?:school
doctor, doctor, school, school, School, teacher, psychiatrist, dentist, physician, veterinarian, educator, mathemat, SCHOOL, Doctors, soDeliveryDate chool, Teacher, tutor, Doctors, teachers
(doctor hospital) O school
26

---

Published as a conference paper at ICLR 2026
Table 10: Examples of semantic fields TverskyGPT-2. Part 3/5: Domain, Relation and Sentiment
Category
Concept & Expression
Top-scoring tokens
Academic title
professor, doctor, doctor, psychiatrist, Professor, lecturer, Professor, physicist, Doctor, scientist, iologist, professors, physician, veterinarian, rabbi, mathematician, dentist, psychologist, Doctor, biologist
Domain
professor n doctor student
hospital medicine
Legal language courthlaw- -wwrrrhwwwwwddesppppooee sports game- play field
court, law, law, Court, Court, Law, Law, COURT, courts, court, LAW, laws, Courts, statute, Lawyers, Justice, courthouse, Attorney, Legal, lawy
Programming
Python, Java, Python, Java, Django, python, ython, Clojure, python, Scala, РHP, Lua, Ådi Drupal, Lua, java, Jakarta, Unix, Lisp, Haskell
- snake - coffee- animal drink
Python
Sibling brothernsister- father- son daughter family
sister, brother, sisters, sibling, brother, Brother, Sister, brothers, Brother, cousin, siblings, cousins, Sisters, comrade, niece, brethren, Brothers, nephew, neighbour, friend
Relation
Spouse husband wife partner - family - child
husband, wife, Wife, husband, husbands, wife, wives, spouse, fian, spouses, fiance, bride, Mrs, widow, wives, marital, Bride, granddaughter, niece, boyfriend
Negative emotion anger n sadness happy joy fear anviety
sadness, anger, DoyablelnstoreAAdOnliin Anger, terness, fury, bitterness, sorrow, indignation, oneliness, resentment, cynicism, mourning, disgust, loneliness, engeance, remorse, grief, saddened, anguish
Sentiment
Mild positive
nice, good, nice, good, Good, Nice, Good, Nice, nicer, GOOD, bad, nifty, romeda, sane, pleasant, decent, Friendly, osuke, positives, shitty
good O nice great excellent per feet best wonder
Surprise
shock, surprise, shock, shocks, surprises, Surprise, Shock, Shock, aston, surpr, shocked, surprised, surprising. shocking, bombshell, aback, miracle, stunned, dazz, startled
surprise O shock fear anger sadness disgust
27

---

Published as a conference paper at ICLR 2026
Table 11: Examples of semantic fields TverskyGPT-2. Part 4/5: Polarity
Category
Concept & Expression
Top-scoring tokens
Bravery (positive)
brave, courageous, isSpecialOrderable, mathemat, trave, advoc, misunder, suspic, streng, challeng, comr, toget, enthusi, bravery, psychiat, valiant, dilig, successfully, Honest, redes
(brove U afraid) n (good - bad U positive negative)
Polarity
Bravery (negative)
afraid, destro, ModLoader, ashamed, fearful, disadvant, defic, misunder, BAD, reluct, isSpecial scared, psychiat, suspic, Fear, terrified, inventoryQuantiy,, trave, frightened, bad
(brove U afradd) n (bod good negative - positive)
Usefulness (positive)
useful, good, good, Useful, GOOD, worthwhile, helpful, Good, usefulness, cellent, mathemat, fruitful, valuable. challeng, indispensable, invaluable, ngth, Excellent, useless, usable
(useful U useless) n (good bad)
Usefulness (negative) (useful U Usscles)) n (bad - good)
useless, bad, BAD, disadvant, worthless, bad, ineffective, unus, unemploy, defic, Bad, pointless, lousy, Worse, faulty, superflu, undesirable, counterproductive, destro, Evil
Temperature (cold) (cold @ hot) n (small big U low high)
cold, cold, Cold, Low, chilly, LOW, colder, low, Cold, misunder, Small, nodd, small, Low, small, shallow, Small, defic, horizont, low
Temperature (hot) (cold + hot) n (big small U high low)
hot, mathemat, hotter, Hot, big, HOT, Hot, cryst, hot, HUGE, hots, Big, dirty, contrace, cold, misunder, hots, hottest, confir, BIG
Wealth rich t poor
poor, rich, poor, Poor, Poor, richer, poorer, Rich, defic, rich, wealthier, impoverished, richest, wealthy, Rich, poorest, needy, affluent, mathemat, wealthiest
Antiquity ancient @ modern
modern, Modern, Ancient, cients, ancient, antiquity, modern, Ancient, prehistoric, medieval, Antiqu, ieval, archaic, Modern, historic, Medieval, mathemat, contemporary, indo, millennia
28

---

Published as a conference paper at ICLR 2026
Table 12: Examples of semantic fields GPP22 Part 5/5: Antonymy. While unhappy appears the semantic field of happy (highlighted in red), set difference isolates the distinct senses of cach pole: happy unhappy and unhappy happy.
Category
Concept & Expression
Top-scoring tokens
(semantic field of happy)
happy, happy, Happy, happiest, happier, Happy, pleased, unhappy, ecstatic, thrilled, satisfied, joyful, delighted, glad, happiness, grateful, thankful, Happiness, happily, cheerful, amused, joy, proud, merry. jealous, Honest, reluct, rejoice, honoured, triumphant
happy
Antonymy
(semantic field of unhappy) unhoppy
unhappy, dissatisfied, discont, untled, dissatisf, disple, dissatisfaction, happy, happiest, disgruntled, unsatisf, reluct, happier, discontent, displeasure, miserable, unlucky, Seym, disillusion, uneasy, appiness, distraught, undesirable, Rebell, contrace, unemploy, disadvant, suspic, annoyed, irritated
(happy contrast to unhappy) happy unhappy
happy, happy, Happy, Happy, happiest, happier, pleased, thrilled, ecstatic, glad, delighted, satisfied, joyful, grateful, thankful, proud, happily, happiness, amused, cheerful, Happiness, joy, Merry, honoured, smiles, Honest, stoked, pleasant, rejoice, merry
(unhappy in contrast to happy)
unhappy, dissatisfied, untled, dissatisf, discont, disple, dissatisfaction, unsatisf, disgruntled, displeasure, discontent, reluct, unlucky, disillusion, undesirable, unemploy, disadvant, miserable, contrace, unwelcome, Seym, intolerable, unworthy, defic, distraught, demoral, irritated, Rebell, uneasy, estranged
undappy happy
29

---

Published as a conference paper at ICLR 2026
C EXPERIMENT 007: TVERSKYXORNET ON XOR
C.I TASK AND DATA
In this experiment, we train single Tversky projection layer to learn the XOR function via gradient descent under various hyper-parameter conditions. Figure 13 shows that some initializations of Tver- sky projection layers may lead gradient descent optimization failure. This experiment empirically analyses its sensitivity to hyperparameters when trained to model the XOR function.
C.2 METHOD
To empirically estimate the sensitivity of Tversky projection layer's convergence to its hyperparame- ters, train 12 960 models consisting of the following combinations of hyperparameters:
6 intersection reduction methods (min, max, product, mean, gmean, softmin);
2 difference reduction methods (ignorematch, substractmatch);
2 normalization modes: [false, true);
6 feature counts: (1, 4, 8, 16, 32);
3 prototype initialization distributions (uniform, normal, orthogonal);
3 feature bank initialization distributions /uniform, normal, and orthogonal };
9 random seeds.
Each model is trained for 1000 epochs. Tables 13, 14, 15, and 16 reports the average and standard error of the convergence indicator (whether accuracy is 100%) of the trained models marginalized by various combinations of hyperparameters. We refer to this variable as convergence probability p(conw) in our results.
C.3 RESULTS
Initialization of prototypes and features Initializing both features and prototypes by sampling from the uniform distribution resulted in the highest convergence probability (Table 14).
Reduction of measures of intersections and differences resulted in the highest convergence probability. (Table 13)
Using product and substractmatch
Normalization In this experiment, normalizing prototype and object vectors prior to calculating Tversky similarity decreased the convergence probability. (Table 15)
Feature count Tversky projection layer successfully modeled zor with as little as I feature. p(conw) maximal with 16 features, but not did monotonously increase with feature count.
While linear projection layers cannot learn non-linear decision boundaries without composition with non-linear activation functions, Figure 14 shows that single Tversky projection layers learn complex non-linear decision boundaries even in low-dimensional vector space.
30

---

Published as a conference paper at ICLR 2026
TrenkpptiPPee nuceonatal poop) Ialee (bettom) cases
Figure 13: Example of learned ProjstyProjection layers optimized to model the xor function. While multiple solutions are possible (top row), some initializations do not lead convergence (bottom row). The decision boundaries were drawn as follows. Random input vectors [x.y] are uniformly sampled in the range [-2,-2] to [2,2] the tip each vector is ploted representing the trained Tversky Projection layer's output. Tversky projection layers modeling the vor function, which has boolean domain and range extend zor to the real vector space. Successful models should show data points [0.0] and [1,1] I] in purple (class 0), and [0,1] and [1,0] in yellow (class 1)
colored dot, with the color
Sucossstul TwwnkyyOOPNet Decaion Dounderie
Sestume
Reauuees
16 Photunns
32 Sealuneo
64 Featunoo
128 Saulanoo
258 faalares
Figure 14: Example of successful Tversky decision boundaries modeling the XOR function with 4, 8. 16, 32, 64, 128 and 256 features. The presence of clusters of features are frequent in overparameterized models.
31

---

Published as a conference paper at ICLR 2026
Table 13: Mean + std error of loss and accuracy for TverskyXORNet with intersection reduction methods (An B) and difference reduction methods (A B). Each corresponds the aggregate of 3x3x2x6x9 = 972 training sessions with: fbank and prototype initialization methods (rthoonl. normal, uniform), 2 normalization modes (normalized, and not normalized), 6 feature bank sizes (1, 2, 4, 8, 16, 32), and 9 random seeds. In each case, models were trained for 1000 epochs
AnB A
loss
acc
best acc
p(conv)
product mean max
substractmatch
972
0.27 + 0.01
0.83 + 0.01
1.0
0.53
0.02
substractmatch
972
0.28
0.01
0.001
1.0
0.51
0.02
ignorematch
972
0.35 0.01
0.78
1.0
0.47 + 0.02
max
substractmatch
972
0.01
0.80
1.0
0.41 0.02
softmin
substractmatch
972
0.35
0.00
1.0
0.42 0.02
min
ignorematch
972
0.31
0.78
1.0
0.42 0.02
softmin
ignorematch
972
0.34
0.78
1.0
0.38 0.02
mean
ignorematch
972
0.00
0.73
1.0
0.26
product
ignorematch
972
0.01
0.75
1.0
0.23
min
substractmatch
972
0.000
0.000
1.0
0.000
gmean
ignorematch
972
nan = nan
0.00
0.5
0.00
gmean
substractmatch
972
nan = nan
0.00
0.5
0.00 0.00
Table 14: Mean + std error of loss and accuracy for TverskyXORNe with 3 fbank and prototype initialization methods (fbank imit, and proto init). Each row corresponds to the aggreate of 6x2x2x6x9 1,296 training sessions with: 6 intersection reduction methods (min, max, product, mean, gmean, softmin) and 2 difference reduction methods (ignorematch, substractmatch), 2 normalization modes (normalized, and normalized), feature bank sizes (1, 2, 8. 16, 32), and 9 random seeds. In each case, models were trained for 1000 epochs.
fbank init
proto init
n
loss
acc
best ace
p(conv)
uniform uniform normal
uniform
1296
0.27 + 0.01
0.78 + 0.01
1.0
0.41 + 0.01
normal
1296
0.36 - 0.01
0.72
0.01
1.0
0.34 0.01
uniform
1296
0.34 0.01
0.74
1.0
0.32 + 0.01
normal normal
1296
0.35+00001
0.74
1.0
0.31 0.01
uniform
orthogonal
1296
0.39
0.70
1.0
0.30
orthogonal
uniform
1296
0.01
0.72
1.0
0.00
normal
orthogonal
1296
0.01
0.73
1.0
0.28
orthogonal
normal
1296
0.37
0.73
1.0
0.26
orthogonal
orthogonal
1296
0.39 = 0.01
0.71
1.0
Table 15: Mean + std eиror of loss and accuracy for TverskyXORNet with normalization modes (normalized, and not normalized). Each row corresponds the aggreate of 3x3x6x2x6x9 5832 training sessions with: 3 fbank and prototype initialization methods (rthoognal,, normal, uniform). intersection reduction methods (min, max, product, mean, gmean, softmin) and difference reduction methods (ignorematch, substractmatch), 6 feature bank sizes (1, 2, 4, 8, 16, 32), and 9 random seeds. each case, models were trained for 1000 epochs.
normalize
loss
best
p(conv)
False
5832
0.33
0.00
0.74 0.00
1.0
0.34 - 0.01
True
5832
0.38 = 0.00
0.72 + 0.00
1.0
0.27 + 0.01
32

---

Published as a conference paper at ICLR 2026
Table 16: Mean + std error loss and accuracy for TverskyXORNet with 6 feature bank sizes (1, 2, 4, 8, 16, 32). Each row corresponds to the aggreate of 3x3x6x2x2x9 1944 training sessions with: 3 fbank and prototype initialization methoossorthhooonal, normal, uniform). 6 intersection reduction methods (min, max, product, mean, gmean, softmin) and 2 difference reduction methods (ignorematch, substractmatch), 2 normalization modes (normalized, and not normalized), and random seeds. In each case, models were trained for 1000 epochs.
fbank size
loss
acc
best acc
p(conv)
16.0
1944
0.25 + 0.01
0.79
0.00
1,0
0.42 + 0.01
8.0
1944
0.28 + 0.01
0.78 + 0.00
1.0
0.39 + 0.01
4.0
1944
0.30
0.01
0.76
1.0
0.38 + 0.01
32.0
1944
0.31 0.01
0.00
1.0
0.33 0.01
2.0
1944
0.00
0.67 1.77
1.0
0.20
1.0
1944
0.54
0.61 + 0.00
1.0
0.12
33

---

Published as a conference paper at ICLR 2026
D EXPERIMENT 009: VISUULL][VVESKY JMNIISSTT
JMNISTNET
Two neural networks, VIsTTT,, and Visua//VerskyMNISTTI (Figure 15) are trained to perform MNIST handwritten digit classification. The neural network architectures share the same convolutional feature extraction stack yielding a 36-dimensional vector representation of the input image. They differ in how those vectors are projected onto the 10-dimensional output vectors corresponding to 10 digit classes. VisualMNISTNe employs stack of 120, 84, and 10 unit linear projection layers following LeCun al. (1998). However, the ReLU activation function used instead of the logistic sigmoid. VisualT vesualt MerIST employs a single Tversky Projection layer with prototypes and features. Both neural networks employ the method described in Section 3.6, facillitating qualitative analysis of the learned prototypes and features.
Two additional neural networks, MNISTNet and Visual counterparts, but do not employ our parameter visualization method are also trained to serve as baseline for accuracy. These neural networks have fewer parameters as the projection parameters specified in their vector form, which is more compact.
which are identical to their
MNISTNet is three times smaller than LeNet-5 due to the design of its convolution stack, which outputs 36-dimensional vector which the concatenation of three 12-dimensional vectors obtained by averaging 3 convolutional feature maps across their spatial dimensions.
TverskyMNISTNNe, with only 7K parameters is also 3 times smaller than MNISTNet because it employs a single Tversky Projection layer instead of a stack linear projection layers and non-linear activation functions. All 4 neural networks trained for 1000 epochs with batch size of 500. Dropout is applied to the output the convolution stack at the rate of 0.05. Adam optimizer is used with learning rate 0.002, and weight decay rate 0.00001.
Projection Params
28x28x1
Specified input Domain
cony 3x3, 12
conv 3x3, 12
12
10x28x28
spatial mcan
Visual FC3 Prototypes
pool 2x2
cony 3x3, 12
conv 3x3, 12
10x28x28 Visual Tversky Prototypes
12
spatial mcan
pool 2x2
conv 3x3, 12
conv 2x2, 12
20x28x28 Visual Tversky Features
12
spatial mean
36
fe1, 120
Tversky Proj.
fc2, 84
fc3, 10
10
10
Figure 15: The neural network architecture used in qualitative analysis experiments.
34

---

Published as a conference paper at ICLR 2026
Table 17: Parameter count and accuracy of baseline and tversky convolutional neural networks trained
for MNIST handwritten digit classification. The "visual" variants specify tversky prototypes and features, and the third fully connected layer's parameters the input space as 28x28 matrices. All models were trained for 1000 epochs.
Model
Params
Valid ACC
MNISTNet
21 394
99.1%
VisualMNISTNet
28 394
99.1%
TverskyMNISTNet
7023
98.7%
VisuaIT
29 463
98.7%
FC3 Projection
Tversky Projection
epoch
epoch 200
epoch 400
epoch 600
Figure 16: Learned visual prototypes in the baseline and tversky variants of the neural network
described in Figure 15 after epochs 0, 200, 400 and 600.
35

---

Published as a conference paper at ICLR 2026
E EXPERIMENT 101: JRVERSKYJRESNETTS ON MNIST
Models cross a variety of hyperparameters (batch size, grad norm clipping threshold) and model initialization (random/image
weights) and param update (update/freeze back one) strategies were trained with 5 random seeds. Each seed corresponds to a different model initial initialization, and different train/val splits of the training sets (See Section E.1) All models were trained for 200 epochs. The SGD optimizer was applied with learning rate 0.03, momentum 0.9, weight decay 0.0001. MNIST images were also up sampled to x 56 input size.
The mean and standard deviation of model accuracies in the train and validation portions are used for model selection. (See Section E.2)
The best Tversky and baseline models in each model initial and parameter update strategy were selected and evaluated on the test set to produce the final mean and standard deviation of model accuracies on the test set. (See Section E.3)
E.I HYPERPARAMETER SEARCII
Results are organized by batch size and gradient clipping combinations. Bold entries indicate the best performing model within each (Pretrained, Freeze) condition. Accuracies are shown as mean +: std.
Table 18: Results for Batch Size=32, Gradient Clipping=1
Pretrained
Freeze
Model
Seeds
Train Acc (%)
Valid Acc (%)
True
False
baseline
5
100.00 = 0.00
99.50 = 0.08
True
False
tversky (fbank=2)
64.59 + 17.82
63.70 + 17.64
True
False
tversky (fbank=3)
76.65 + 13.52
76.00 + 13.28
True
False
tversky (fbank=5)
5
96.20 5.21
95.80 + 5.07
True
False
tversky (fbank=8)
98.06 4.32
97.38 4.70
True
False
tversky (fbank=13)
5
100.00 + 0.00
99.52
True
False
tversky (fbank=21)
100.00 0.00
99.48 0.06
True
False
tversky (fbank=34)
5
100.00 + 0.00
99.47
True
False
tversky (fbank=55)
100.00 0.00
99.51 + 0.08
True
True
baseline
84.87 0.13
84.42 + 0.30
True
True
tversky (fbank=2)
20.37 7.09
19.78 + 6.78
True
True
tversky (fbank=3)
33.37 8.99
33.08 + 8.97
True
True
tversky (fbank=5)
5
51.57
51.86 + 12.02
True
True
tversky (fbank=8)
5
73.40 6.94
73.14
True
True
tversky (fbank=13)
72.51 19.63
71.65 + 19.82
True
True
tversky (fbank=21)
5
81.44 1444-7 7.65
80.46 7.62
True
True
tversky (fbank=34)
86.89 1.48
84.73
True
True
tversky (fbank=55)
5
88.12 + 2.47
85.58 + 1.91
False
False
baseline
100.00 + 0.00
99.52 + 0.07
False
False
tversky (fbank=2)
13.95 + 5.60
1 5.63
False
False
tversky (fbank=3)
9.89 0.04
9.68 + 0.32
False
False
tversky (fbank=5)
5
20.45
19.9477.45
False
False
tversky (fbank=8)
33..4
4
False
False
tversky (fbank=13)
5
60.64
00.23 25.37
False
False
tversky (fbank=21)
100.00
0.00
9..47
False
False
tversky (fbank=34)
5
100.00
0.00
99.48
False
False
tversky (fbank=55)
100.00
0.00
9..04
36

---

Published as a conference paper at ICLR 2026
Table 19; Results for Batch Size=32, Gradient Clipping=10
Pretrained
Freeze
Model
Seeds
Train Acc (%)
Valid Ace (%)
True
False
baseline
100.00 0.00
99.51 + 0.07
True
False
tversky (fbank=2)
5
45.18 = 14.84
44.28 + 14.45
True
False
tversky (fbank=3)
60.70 = 12.54
60.06 + 13.00
True
False
tversky (fbank=5)
5
86.50 5.37
86.12
True
False
tversky (fbank=8)
97.96 4.56
97.44 4.61
True
False
Iversky (fbank=13)
100.00 0.00
99.48 0.05
True
False
tversky (fbank=21)
100.00 0.00
99.51 + 0.08
True
False
tversky (fbank=34)
1.000
99.49
True
False
tversky (fbank=55)
5
100.00 0.00
99.50 0.05
True
True
baseline
5
84.87 0.14
84.31 = 0.37 0.37
True
True
tversky (fbank=2)
1
17.66 8.29
True
True
tversky (fbank=3)
5
19.86 +
19.54 11.60
True
True
tversky (fbank=5)
18.53 4.88
+
True
True
tversky (fbank=8)
31.47 29.29
4
True
True
tvensky (fbank=13)
5
39.00
58.56 34.78
True
True
tversky (fbank=21)
39.44 40.59
True
True
tversky (fbank=34)
5
+3977
41.78 39.19
True
True
tversky (fbank=55)
41.11 1 42.76
39.92 + 41.30
False
False
baseline
100.00 + 0.00
99.47 + 0.01
False
False
tversky (fbank=2)
9.89 + 0.04
9.68 + 0.32
False
False
tversky (fbank=3)
9.68 + 0.32
False
False
tversky (lbank=5)
9.89
9.68
False
False
tversky (fbank=8)
5
+
29.59 30.68
False
False
tversky (fbank=13)
45.66 40.90
40.12
False
False
tversky (fbank=21)
5
49.18 49.55 - 46.21
False
False
tversky (fbank=34)
66.46 46.06
66.23
False
False
tversky (fbank=55)
5
64.07 + 49.20
63.66 + 48.97
37

---

Published as a conference paper at ICLR 2026
Table 20; Results for Batch Size=256, Gradient Clipping=1
Pretrained
Freeze
Model
Seeds
Train Acc (%)
Valid Ace (%)
True
False
baseline
100.00 0.00
99.61 + 0.06
True
False
tversky (fbank=2)
5
41.14 17.28
40.66 + 17.05
True
False
tversky (fbank=3)
48.55 + 12.96
48.07 = 12.93
True
False
tversky (fbank=5)
5
88.23 + 10.96
87.96 + 10.89
True
False
tversky (fbank=8)
94.18 8.69
93.55 8.79
True
False
tversky (fbank=13)
100.00 0.00
99.58 + 0.11
True
False
tversky (fbank=21)
5
100.00 0.00
99.59 + 0.09
True
False
tversky (fbank=34)
1.000
99.62 + 0.06
True
False
tversky (fbank=55)
5
100.00 0.00
99.61 = 0.06
True
True
baseline
5
87.35 = 0.15
86.02 = 0.56
True
True
tversky (fbank=2)
37.59 1.61
37.46 1.42
True
True
tversky (fbank=3)
50.81 4.25
50.79 4.15
True
True
tversky (fbank=5)
70.79 + 3.38
70.75 3.77
True
True
tversky (fbank=8)
83.31 = 2.68
82.32 2.57
True
True
tvensky (fbank=13)
5
87.81 + 0.25
86.01 0.42
True
True
tversky (fbank=21)
89.47 - 0.43
85.99 0.65
True
True
tversky (fbank=34)
5
92.36 + 0.13
True
True
tversky (fbank=55)
94.67 = 0.17
86.85 = 0.20
False
False
baseline
100.00 = 0.00
99.34 0.07
False
False
tversky (fbank=2)
24.69 11.52
24.19 + 11.06
False
False
tversky (fbank=3)
2 12.33
20.13 12.13
False
False
tversky (Ibank=5)
26.43 19.91
26.31 + 19.87
False
False
tversky (fbank=8)
5
4 47.77
False
False
tversky (fbank=13)
88.32
8
False
False
tversky (fbank=21)
5
94.26 8.47
93.69
False
False
tversky (fbank=34)
100.00 0.00
99.33
False
False
tversky (fbank=55)
5
100.00 + 0.00
99.34
38

---

Published as a conference paper at ICLR 2026
Table 21: Results for Batch Size=256, Gradient Clipping=10
Pretrained
Freeze
Model
Seeds
Train Acc (%)
Valid Ace (%)
True
False
baseline
100.00 0.00
99.61 + 0.05
True
False
tversky (fbank=2)
5
36.98 5.49
36.54 5.68
True
False
tversky (fbank=3)
50.81 + 11.99
49.92 + 12.54
True
False
tversky (fbank=5)
5
94.09 5.41
93.91 + 5.22
True
False
tversky (fbank=8)
100.00 0.00
99.55 + 0.06
True
False
Iversky (fbank=13)
100.00 0.00
99.59 + 0.11
True
False
tvensky (fbank=21)
5
100.00 0.00
99.58 + 0.08
True
False
tversky (fbank=34)
1.000
99.58 + 0.07
True
False
tversky (fbank=55)
5
100.00 0.00
99.57 0.04
True
True
baseline
5
87.35 0.11
86.05 0.54
True
True
tversky (fbank=2)
36.14 1.69
36.09 - 2.19
True
True
tversky (fbank=3)
51.82 6.07
51.92 6.48
True
True
tversky (fbank=5)
69.91 4.43
69.68 + 4.72
True
True
tversky (fbank=8)
82.00 + 2.80
81.04
True
True
tvensky (fbank=13)
5
0.17 87.86 - 0.17
86.17
True
True
tversky (fbank=21)
89.42 0.07
85.96 0.53
True
True
tversky (fbank=34)
5
92.31 = 0.17
86.80 0.37
True
True
tversky (fbank=55)
94.66 0.07
86.81 = 0.41
False
False
baseline
100.00 = 0.00
99.28 + 0.09
False
False
tversky (fbank=2)
10.41 0.71
10.44 + 1.13
False
False
tversky (fbank=3)
10.15 0.57
10.10 1.00
False
False
tversky (Ibank=5)
12.69 + 4.81
12.47
False
False
tversky (fbank=8)
5
30.80 + 9.63
30.43 9.64
False
False
tversky (fbank=13)
34.91 + 18.04
False
False
tversky (fbank=21)
5
52.81 16.38
52.36 16.47
False
False
tversky (fbank=34)
72.70
False
False
tversky (fbank=55)
5
96.24 + 8.41
95.34 8.70
39

---

Published as a conference paper at ICLR 2026
E.2 MODEL SELECTION
Table 22: Summary: Best Baseline and Tversky Models for Each Configuration
Best Baseline
Best Tversky Batch
Pretrained
Freeze
Valid Acc (%)
Batch
Clip
Valid Ace (%)
Clip
FBank
True
False
99.61 + 0.05
256
10
99.62 + 0.06
256
34
True
True
86.05 = 0.54
256
10
86.85 = 0.20
256
False
False
99.52 = 0.07
32
99.54 + 0.09
32
55
E.3 EVALUATION OF BEST MODELS
Table 23: Summary: Best Baseline and Tversky Models for Each Configuration
Best Daseline
Best Tversyy Test Acc (%)
Pretrained
Freeze
Thin Ace (8)
Valid Ace (16
Text Acc (6)
Batch
Clip
Train Acc (%)
Valid Acc (96
Borch
Clip
Filank
Tue
Filse
100.00 + 0.00
9961 + 0.05
92.06 0.04
256
10
100.00 0.00
50.02 = 0.06
50.00 0.06
256
34
Tuee
True
00.24 0.27
85.8 + 0.80
85.64 0.14
256
10
98.18 0.36
56, 80 = 0:23
86.06 0.37
256
55
False
Filse
100.00 =: 0.00
99.52 =: 0.07
92.54 0.04
32
100,00 0.00
30.54 = 0.00
09.56 = 11.0
32
55
40

---

Published as a conference paper at ICLR 2026
F EXPERIMENT 103: [TVERSKY]RESNET-S ON NABIRDS
ResNet-50 and Rverssy were trained for 600 epochs with SGD (learning rate 0.03, mo- mentum 0.9, weight decay 10 10-4, gradient clipping threshold 10) and a cosine annealing learning rate schedule. We used a batch size of 256 and dropout of 0.3. Input images were resized to 600 x 600 pixels and randomly cropped to 448 448 during training; at test time, center crop of 448 448 was used. Training images were augmented with random horizontal flips, and all images were normalized using ImageNet mean and standard deviation. The Tversky projection layer used product and agnorematch as the intersection and difference reduction hyperparameters respectively, with feature bank sizes drawn from the Fibonacei sequence: 610, 987, 1597, 2584, 4181, 6765, and 10946. Each configuration parameters, shuffling of training data, and partitioning of train/validation data. 10% of the training split was used validation set. Test results reported the held-out standard NABirds test set. All accuracies are reported as mean + std.
trained with multiple seeds, corresponding to different initializations of model
F.1 PAPER-PARRMETER SEARCII
Table 24: Results for Batch Size=256, Gradient Clipping=10
Pretrained
Freeze
Model
Seeds
Train Acc (%)
Valid Acc (%)
True
False
baseline
99.97 = 0.01
86.71 + 1.13
True
False
tversky (fbank=610)
99.96 = 0.01
85.83 + 1.19
True
False
tversky (fbank=987)
99.97 + 0.01
86.71 0.49
True
False
tversky (fbank=1597)
99.97 + 0.02
86.18 + 0.72
True
False
tversky (fbank=2584)
99.98 0.01
86.12 = 0.51
True
False
tversky (Ibank=4181)
99.98 = 0.01
86.50 + 0.43
True
False
tversky (lbank=6765)
99.97 = 0.02
87.03 1.35
True
False
tversky (fbank=10946)
99.96 = 0.02
86.22 + 0.64
True
True
baseline
79.33 = 0.25
51.88 + 0.44
True
True
tversky (fbank=610)
96.40 = 0.42
48.21 1.27
True
True
tversky (fbank=987)
98.67 + 0.34
48.23 0.50
True
True
tversky (fbank=1597)
99.20 + 0.29
50.24 0.99
True
True
tversky (fbank=2584)
99.30 0.06
51.52 2 0.02
True
True
tversky (fbank=4181)
99.49 = 0.08
50.93 + 2.09
True
True
tversky (lbank=6765)
99.56 = 0.08
51.37 1.33
True
True
tversky (fbank=10946)
99.49 + 0.10
50.74 + 0.77
False
False
baseline
99.99 + 0.01
68.67 + 1.19
False
False
tversky (fbank=610)
99.89 = 0.05
70.41 + 0.73
False
False
tversky (fbank=987)
99.92 + 0.05
70.55 0.74
False
False
tversky (fbank=1597)
99.93 + 0.04
71.14 0.86
False
False
tversky (fbank=2584)
99.93 0.00
71.12 0.79
False
False
tversky (fbank=4181)
99.89 - 0.02
70.53 + 1.20
False
False
tversky (fbank=6765)
99.91 = 0.02
71.09 + 1.57
False
False
tversky (fbank=10946)
99.91 + 0.06
69.07 1.12
41

---

Published as a conference paper at ICLR 2026
F.2
MODEL SELECTION
Table 25: Summary: Best Baseline and Tversky Models for Each Configuration
Best Baseline
Best Tversky Batch
Pretrained
Freeze
Valid Acc (%)
Batch
Clip
Valid Ace (%)
Clip
FBank
True
False
86.71 + 1.13
256
10
87.03 + 1.35
256
10
6765
True
True
51.88 = 0.44
256
51.52 + 0.02
256
10
2584
False
False
68.67 = 1.19
256
10
71.14 + 0.86
256
10
1597
F.3 EVALUATION OF BEST MODELS
Table 26: Summary: Best Baseline and Tversky Models for Each Configuration
Best Daseline
Best Tversky Test Acc (9)
Pretrained
Freeze
Train Acc (%)
Valid Acc (9)
Test Acc (6)
Batch
Clip
Train Acc (8)
Valid Acc (9)
Borch
Clip
Filank
Tue
Filse
20.90 0.01
86.11 + 1.13
62.37 0.25
256
10
100.00 0.0
67.00 = 1.35
62.06 0.07
256
10
6765
Tuee
Thue
96.02 0.34
5188 0.44
10.25 0.28
256
10
99.54 0002
51.47 = 0.30
38.71 0.20
256
10
2584
False
Filse
20.99 = 0.00
68.07 = 1.19
02.84 = 0.45
256
10
99.97 000
71.14 0.50
66.20 0.26
256
10
1597
42

---

Published as a conference paper at ICLR 2026
F.4
EXAMPLE OF LEARNED a, B AND B
dlagnostics/alpha valid/epoch_ v. diagnostics/beta
acc.
0,702
0.704
0.706
0.708
0,710
0.712
0.714
0,716
0.718
1.7
1.66
1.62
1.6
1.58
1.54
1.5
1.46
0
B
a
<
1.42
dia di beta
1.42
1.48
1.46
1.48
1.50
1552
1.54
1.50
1.50
1.00
Figure 17: Top: Example learned a, 3 and values over training iterations a TverskyResNet-500 on NABirds trained from scratch. These parameters are learned via gradient descent alongside all other model parameters. Bottom: Comparison of final learned a (X-axis) and (y-axis) across 3 runs of the same configuration with different seeds; point color represents validation accuracy. In all three models > 8, consistent with Tversky's 's theory that the distinctive features of instances are weighted more heavily than those of prototypes. While we observed a >> 3 in several successful models, all accurate models had this property.
43

---

Published as a conference paper at ICLR 2026
F.5 COMPUTATION RESOURCES
Table 27: Duration of model training sessions in wall-clock time (h:mm:ss), grouped by model type,
GPU type, GPU count, and feature bank size (Tversky only). Note that the measured durations depend on various factors external to the architecture and hyperparameters of models, including I/O
access time network drives, which fluctuates with cluster usage. Std is reported when N 1.
Model
GPU
#GPU
[bank N
Mean
Sid
Min
Max
Baseline
NVIDIA SXM4-40GB
28:32:18
15:55:15
10:09:22
37:56:42
Baseline
NVIDIA A100-SXM4-800
4
10:00:14
0:29:02
9:15:07
10:28:44
Baseline
NVIDIA RTX A6000
10:45:45
10:45:45
10:45:45
Tversky
NVIDIA 1100-XX44400GB
3
4181
9:14:14
9:14:14
9:14:14
Tversky
NVIDIA 4100-SXM4-40GB
6765
9:24:51
9:24:51
9:24:51
Tversky
NVIDIA -100-5XM4440GB
2584
9:57:01
9:57:01
9:57:01
Tversky
NVIDIA 1100-SXM4400GB
4
4181
10:00:42
10:00:42
10:00:42
Tversky
NVIDIA A100-SXM4440GB
6765
10:11:54
10:11:54
10:11:54
Tversky
NVIDIA A100-SXM4440GB
10946
10:01:52
10:01:52
10:01:52
Tversky
NVIDIA -100-SXM4-80GB
610
9:12:21
0:27:41
8:52:46
9:31:56
Tversky
NVIDIA 1100-SXM4-800G
2
987
9:46:35
9:46:35
9:46:35
Tversky
NVIDIA A1000SXX44800GH
2
1597
8:34:17
8:34:17
8:34:17
Tversky
A100-SXM4-800T
2584
9:31:55
9:31:55
9:31:55
Tversky
NVIDIA A1000-XXM4-800-
2
10946
8:24:21
8:24:21
8:24:21
Tversky
NVIDIA A100-SX.44-8000
4
610
9:34:20
0:24:56
9:12:26
10:08:32
Tversky
NVIDIA 1100-SXM4-00GB
987
9:41:57
0:35:41
9:12:16
10:25:39
Tversky
NVIDIA A100-SXM4-800EB
1597
9:21:46
0:28:12
8:40:10
9:52:03
Tversky
NVIDIA A100-SXM4-8OGB
2584
9:46:25
0:29:38
9:11:34
10:20:07
Tversky
NVIDIA 300-SXM4-S0GBB
4
4181
9:27:43
0:27:41
8:45:03
10:09:11
Tversky
NVIDIA A100-SXM4-80GE
4
6765
9:28:17
0:33:22
8:56:45
10:08:13
Tversky
A100-SXM4-S00
9:26:21
0:43:17
8:38:53
10:28:57
Tversky
NVIDIA H200
2
987
6:25:57
6:25:57
6:25:57
44


Reviews:
[{"review_id": "go4rDK62nh", "reviewer": "Reviewer_wB3R", "summary": "This paper challenges the geometric model of similarity (dot-product/ cosine similarity) largely used in deep learning, proposing instead Tversky's psychologically-grounded similarity measure that permits asymmetry and other violations of metric axioms observed in human similarity judgment. The key contribution is a differentiable version of Tversky's similarity and corresponding neural layers that can replace dense layers in standard architectures. The approach is based on the introduced dual representation of an object: objects are represented both as vectors and as sets of features from a learnable finite universe, where an object's set comprises those feature vectors with which it has positive dot product. This set-theoretic representation enables measuring similarity through feature set intersections and differences, forming the basis for Tversky Similarity and Projection layers. The authors demonstrate that: (1) a single Tversky layer can model XOR, impossible for linear layers; (2) Tversky layers can improve performance in GPT-2 language modeling (PTB dataset) and ResNet-50 image classification (MNIST, NABirds). Qualitative analyses reveal interpretable properties: salience scores (sum of dot-products of an object's vector with all features in its set) align with \"goodness of form,\" learned (input-level) prototypes are human-interpretable, and semantic fields (e.g., adjectives, verb forms) emerge from set algebra. Additionally, they introduce a data-domain visualization method for projection parameters, though this approach scales poorly. \n\nThe work is conceptually novel, well-written, and really promising but could benefit from modern architecture evaluation, clearer differentiability explanation, and stronger scalability analysis.", "questions": "- Please see questions already stated in weaknesses :)\n- Methods:\n    - As the author’s mention in the discussion, the size of  $\\Omega$  is hyperparameter and needs to be tuned as everything else. However, do the authors have an intuition if there could be a relationship between the number of prototypes and the universe size?\n    - Question about the set representations: A feature $f_k$ of the learnable finite universe is in the set of object $x$ if their dot-product is larger than zero. What does it then mean to include into this set representation a feature $f_j$  with $x\\cdot f_j = 1e^-10$ and a feature $f_q$ with $x\\cdot f_q = 1e^-10$?\n- Experiments\n    - Tversky layer can model XOR: In Tables 4-7 we can an observe rather small convergence ratios, could the authors comment on that?\n    - Language-modeling: In Table 1, one can observe that only replacing the prediction head with a tversky head results in more features and parameters compared to replacing all projection layers with Tversky projections. This makes sense as the projection head itself must learn to represent all the semantic structure needed for predicting the next token. When all projection layers are replaced, the encoder and attention blocks co-adapt with the new similarity function, so the features can specialize efficiently, and fewer prototypes are sufficient. However, could you elaborate on the feature number and parameter number if we compare tversky-all with and without prototyp tying? Why do we observe such a difference?\n    - Image classification:\n        - Would it be possible to show some image classification results with some a pretrained Vision Transformer. E.g., a vision-encoder of a CLIP model, or Dinov3 model (available on huggingface)?\n        - Why are the pretrained and frozen setting so bad in performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Original and well-motivated idea connecting human similarity judgments to machine-learned representations. Clear novelty of First differentiable implementation of Tversky similarity for neural networks.\n- The paper is well-written, clearly structured, and engaging to read. I learned a lot. \n- The XOR demonstration (Section 3.1, Figure 1) with its thorough validation convincingly shows that a single Tversky projection can model non-linear decision boundaries without composition with activation functions. This is a genuine advantage over linear layers.\n- Demonstrates potential in both vision and language modeling contexts.\n- Qualitative analysis really interesting.\n- Acknowledges computational cost of data-domain visualization (Figure 5) and it is honest about the limitations.", "weaknesses": "- Gradient flow through indicator functions: Equations 2-5 rely on indicator functions [a·fₖ > 0] which have zero derivative almost everywhere. It's unclear how gradients propagate during training—are you using straight-through estimators, sigmoid approximations, or another approach? Providing implementation details or code would help to understand you work.\n- Convergence of training: Tables 4-7 show convergence failure rates of 47-77% across many hyperparameter settings, with some producing NaN results. Understanding what causes these failures would strengthen the work. How did convergence look for the other main experiments?\n- Evaluation on larger, more recent benchmarks would strengthen claims. Current experiments use MNIST (toy dataset), PTB (1M tokens from 1993), and NABirds (48K images with domain overlap with ImageNet birds), which are ideal for a proof of concept. However, testing on standard modern benchmarks like ImageNet, CIFAR-100, or WikiText-103 would better demonstrate competitiveness against well-tuned baselines.\n- Similarly, validation on current architectures would show broader applicability. While ResNet-50 and GPT-2 small are solid choices for controlled experiments, evaluating on Vision Transformers (ViT, CLIP, DINOv2) or larger language models would support the \"state-of-the-art\" framing (L82).\n- Multiple seeds are reported for XOR experiments, but it's unclear if main experiments used multiple runs. Would it be possible to comment on this issue or if available report the means and standard deviations?\n- Tied vs untied feature banks remain unexplored. While prototype tying is ablated in Table 1, feature bank sharing is always enabled. Testing both configurations would be informative.\n- Data-domain visualization might be impractical at scale. For NABirds with 555 classes, input-space parameterization would require 83.5M parameters (555×224×224×3) compared to 1.1M for standard approaches—a 75× increase. Could the authors comment on the scalability constraints and potential overfitting risks?\n- Reporting accuracies: Using relative improvements (e.g., 24.7%) rather than absolute accuracy points can sometimes overemphasize small differences. Both would be informative.\n- Methods like ProtoPNet and ProtoTree (prototype learning domain) seem related to the data-domain visualization contribution but aren't discussed in the introduction or chapter 2.5. Could the authors comment on that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FS4fFaIpBN", "reviewer": "Reviewer_XcN8", "summary": "The paper introduces a new perspective on classification via prototype learning, where classifier decisions are made by taking the argmax over similarities to class prototypes.  Similarity is defined as an asymmetric match function between the object and a prototype over a learned feature bank whose dimensionality can be set.  While asymmetric similarity has been used in some prior applications (e.g., nearest neighbor classification based on engineered features), to my knowledge it has not been used in end-2-end CNN training, and the application is novel.", "questions": "It would be good to know if the Tversky head is effective as a classifier when attached to a backbone not trained for classification (e.g., a self-supervised backbone) and if in that case it would also outperform a baseline linear readout.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Strengths:\n* Strong originality of entire prototype learning framework based on learned asymmetric comparison, which can be integrated in end-to-end training of various deep architectures. \n* The asymmetry weights, prototype dimensionality, and the feature bank are treated as learnable (free) parameters.\n* Demonstrates improved performance on several benchmarks, including NABirds, indicating applicability to problems with hundreds of classes. While there is no demonstration of extension to thousands of classes, in this case I do not see as a limitation, because one of the central future applications of this training approach may be application to classification in relatively small domains with highly similar categories.\n* A strength of the approach, whichcould be better discussed, is the potential to constrain the directionality of the asymmetry weights (e.g., setting β>α when omissions matter more, α>β when extras hurt more) depending on domain. For domains with low intra-class variance where 'core' features count (quasi necessary), penalizing omissions  of object vs. prototype might be more sensible; the converse for broader, more fuzzy categories.", "weaknesses": "(Minor) Weaknesses:\n\n* It would be nice to have a small ablation experiment isolating the specific contribution of asymmetry by setting the asymmetry weights to zero and learning only the overlap (intersection) term in Equation 1.\n* Can be extended to allow learning only the asymmetry terms with the overlap term to zero. Taken together we can understand if performance gains rely mainly on similarities or differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tFfkSNs44a", "reviewer": "Reviewer_ymVH", "summary": "The paper proposes an alternative to the commonly geometric formulation of similarity through using Tversky similarity. The authors propose a differentiable formulation of representations for this purpose, along with an implementation thereof (Tversky Projection neural layer). In addition, an approach for interpreting Tversky networks through visualization of projection layers in the data domain are proposed. Experiments on the XOR, Penn Treebank text data, and MNIST are performed to demonstrate the method.", "questions": "- Assuming contextualized models, would they be able to learn Tversky-type similarity?\n- The notion of feature bank Ω could be communicated more clearly in Section 2.3. From Section 2.2 these are feature vectors $f_k$ and Ω parametrizes the function $f$, so $f$ essentially an embedding layer (as also mentioned more clearly later in Sec. 2.4.1).\n- “For our vision experiments, no tuning was performed” (l. 459): No tuning *of hyperparameters* if I read Section 3.3 correctly, is that correct?\n\n**Editing*\n- “$α, β, andθ$” (l. 181)\n- “fullly” (l. 217)\n- “weights are initialized from ImageNet” (l. 327) - ImageNet -> ResNet?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Clear and well-founded scientific motivation for the proposed novel methods. The idea of using Tversky similarity for deep learning is to my knowledge a novel contribution.\n- Empirical confirmation of the proposed neuralized Tversky approach, in both synthetic experiments and large-scale studies\n- The paper includes an honest and balanced discussion section and transparently raises limitations throughout the paper.\n- The experiments appear reproducible and are for the most part clearly described (see below).", "weaknesses": "- While I like the motivation of Section 3.1 (XOR), the section and associated Figure 1 read rushed and can be confusing in its current state. I think the main take-home is that the Tversky projection layer can in principle learn the XOR function. I would suggest focusing on introducing the problem clearly (which is currently done in the caption), and then discuss some of the limitations encountered during optimization, i.e., convergence and hyperparameter selection.\n- While I think the experiments are overall good, they remained superficial at times. Some quite interesting points were raised in the discussion, e.g. the idea of editing prototypes for increased robustness, which would have been great to demonstrate also empirically. \n- Some claims regarding interpretability are not clearly supported by evidence, e.g. “handwriting, such as lines and curves, more clearly than those learned by linear projection layers” (l. 413) and “Tversky Projection layers’s parameters are far more interpretable than the ones of the contemporary fully connected layer“ (l. 217). A more fair comparison could be to compare attributions in input space, e.g. [Mon18] as the Tversky Projection layers explicitly tie the projection to being decodable from the input. \n- The description of the proposed visualization technique in Section 2.5 should be more clear, e.g. through the use of formulas or a conceptual visualization, e.g. moving Fig. 5 to the main alongside a more clear description of how the method works. For example, the sentence “limitation that parameters specified in data-space are typically larger in size than their original counterparts, which increases the effective number of trainable parameters.“ (l. 211-213) is difficult to parse, i.e. how are the projection parameters specified, is the visualization technique trained or is this a post-hoc interpretability approach?\n\n\n**Comment**\n- I believe the proposed framework could be quite useful in the context of textual similarity, where it was shown that standard encoders and cosine similarity do not match human similarity judgments very well [Rei19]. Follow-up work on interpreting these dot-product embeddings could reveal quite simplistic feature matching strategies, even after aligning predictions to human similarity judgements [Vas24]. Here a Tversky layers may provide a more effective similarity readout.\n\n\n**References**\n- [Rei19] Reimers, N., & Gurevych, I. (2019, November). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3982-3992).\n- [Vas24] Vasileiou, A., & Eberle, O. (2024, June). Explaining Text Similarity in Transformer Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 7852-7866).\n- [Mon18] Montavon, G., Samek, W., & Müller, K. R. (2018). Methods for interpreting and understanding deep neural networks. Digital signal processing, 73, 1-15.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JxnhlU7G6Z", "reviewer": "Reviewer_c2vD", "summary": "The authors propose how to use Tversky similarity (from cognitive psychology) instead of the geometric model of similarity, which is usually used in deep learning. They perform experiments on text and image domains.", "questions": "* “Some initializations of prototypes and features lead to convergence failure” - are normal and orthogonal initialization the ones that the authors refer to?\n* Why didn’t the authors present the example graphical results for NABirds (like the ones in Fig. 2 for MNIST)?\n* Also, see the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* I really like the idea of using Tversky similarity to create layers for neural networks. Tversky’s theory is very logical and simple and I think that it has a huge potential for the neural network domain. \n* The paper is well-written and easy to follow. It is beneficial that the authors presented a small XOR example at the beginning. \n* The salience experiment was interesting.", "weaknesses": "* The paper lacks a clear related work section. While some works are described in the introduction, it would be good to include a dedicated section. Also, there are works that use Tversky’s similarity model or methods inspired by it for different purposes in the deep learning domain, and it would be beneficial to mention these works and acknowledge that using Tversky’s model of similarity is not completely new in deep learning - even if for different purposes.\n* “This lack of interpretability, even in this simple domain without background textures, represents a significant limitation of prior approaches.” - the authors say that their approach improves interpretability. While it is visible that for MNIST (simple objects, with large differences) the prototypes look reasonable, I am not so sure that they would be that pronounced for the second dataset used in the study - regarding different breeds of birds. \n* For the vision experiments, it would be nice to include the results for a ViT as well - to represent two leading architecture families in vision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)