Tversky Neural Networks: Psychologically Plausible Deep Learning with Differentiable Tversky Similarity

## Abstract

Abstract Work in psychology has highlighted that the geometric model of similarity standard in deep learning is not psychologically plausible because its metric properties such as symmetry do not align with human perception of similarity.
In contrast, Tversky ( 1977 ) proposed an axiomatic theory of similarity with psychological plausibility based on a representation of objects as sets of features, and their similarity as a function of their common and distinctive features.
This model of similarity has not been used in deep learning before, in part because of the challenge of incorporating discrete set operations.
In this paper, we develop a differentiable parameterization of Tversky’s similarity that is learnable through gradient descent, and derive basic neural network building blocks such as the Tversky projection layer , which unlike the linear projection layer can model non-linear functions such as xor .
Through experiments with image recognition and language modeling neural networks, we show that the Tversky projection layer is a beneficial replacement for the linear projection layer.
For instance, on the NABirds image classification task, a frozen ResNet-50 adapted with a Tversky projection layer achieves a 24.7% relative accuracy improvement over the linear layer adapter baseline.
With Tversky projection layers, GPT-2’s perplexity on PTB decreases by 7.8%, and its parameter count by 34.8%.
Finally, we propose a unified interpretation of both types of projection layers as computing similarities of input stimuli to learned prototypes for which we also propose a novel visualization technique highlighting the interpretability of Tversky projection layers.
Our work offers a new paradigm for thinking about the similarity model implicit in modern deep learning, and designing neural networks that are interpretable under an established theory of psychological similarity.

## 1 Introduction

The geometric model of similarity is ubiquitously used in modern neural networks. Their architectures include operations that assess the similarity between objects using metric
similarity functions such as the vector dot product or cosine similarity.
For instance, linear projection layers, also known as *dense*, *feed forward*, or *fully connected*, output the dot-product similarity of the input vector with the columns of its weight matrix, which are concept prototypes.
Classification modules typically normalize these similarities with softmax to form valid probability distributions (Bridle, 1990; LeCun et al., 1998).
Convolution layers of image recognition neural networks compute the dot product of image regions and convolution kernels, which represent prototypical patterns.
LSTM gates compute dot product similarities of combinations of inputs, hidden states, and cell states with prototypical *input*, *forget*, and *output* gating patterns (Hochreiter and Schmidhuber, 1997; Graves et al., 2005; Graves, 2013).
Semantic word embedding models such as word2vec (Mikolov, 2013) and GloVe (Pennington et al., 2014) compute the dot-product and cosine similarity of word representations with prototypical embeddings to predict words.
Neural language models apply the same classification mechanism to output discrete words or subwords at each time step (Bengio et al., 2000; Sutskever, 2014; Devlin, 2018; Radford et al., 2018; Achiam et al., 2023).
Attention weights (Bahdanau et al., 2014; Vaswani, 2017) are the normalized dot-product similarity of *queries* and *keys*, which are themselves obtained using linear projection layers.
Geometric similarity representations have also been widely used in machine learning objective function design for over 200 years, from least squares methods (Legendre, 1806) to modern approaches including L2 loss (Hadsell et al., 2006),
Large Margin Cosine Loss (Wang et al., 2018),
and
Noise Contrastive Estimation objective (van den Oord et al., 2018; Chen et al., 2020). These objectives span both supervised and self-supervised learning paradigms.

However, Tversky (1977) famously challenged this geometric representation of similarity, showing its psychological implausibility due to its fundamental inability to model phenomena such as humans’ asymmetric judgment of similarity.
For instance, “We say that *‘the son resembles the father’* rather than *‘the father resembles the son’*.”
He addressed this problem with his axiomatic theory based on a feature matching process in which objects are represented as sets of features and their similarity is measured as a linear combination of measures of their common and distinctive features.
Tversky’s similarity has been shown to model human judgment more accurately than metric similarity measures in various empirical studies (Gati and Tversky, 1984; Ritov et al., 1990; Siegel et al., 1982; Rorissa, 2007).
Moreover, over the past decade, several works have highlighted the limitations of the geometric model of similarity and related parallelogram model of semantic relations (Garg et al., 2015; Chen et al., 2017; Peterson et al., 2020; Zhou et al., 2022), and advocated for more psychologically principled similarity measures such as Tversky’s model, which has strong empirical support.

Nevertheless, a differentiable expression of Tversky’s similarity function has not been proposed to date, which prevents its incorporation into neural networks trained using gradient-based methods
(Rumelhart et al., 1986; LeCun et al., 1998).
The formulation of a differentiable Tversky similarity function is non-trivial because it employs measures of set intersections and differences, which are not differentiable with respect to object features comprising those sets.
To address this gap, we propose a novel representation of features as vectors of the same dimensionality as object vectors, and a dual representation of objects both as vectors and as sets, such that an object is the set of features with which it has a positive dot product. This representation of objects and features enables us to define differentiable measures of set intersections and differences, which we use to construct a differentiable Tversky similarity function suitable for deep learning. We also propose the Tversky Projection neural layer, which is analogous to the linear projection layer but is based on Tversky’s model of similarity.

We introduce our proposed differentiable similarity function, and basic neural network modules, along with a novel method for interpreting and visualizing deep projection layers in Section [2](https://arxiv.org/html/2506.11035v2#S2).
Section [3](https://arxiv.org/html/2506.11035v2#S3) demonstrates the effectiveness of Tversky Projection layers in state-of-the-art architectures, including ResNet-50 and GPT-2, on image recognition and language modeling tasks. Our experiments show that the use of Tversky projection layers can lead to significant improvements in both accuracy and parameter efficiency.
In particular, we show that the use of a Tversky Projection layer instead of the final linear projection layer in ResNet-50 can result in up to 24% accuracy improvement in image recognition tasks.
We also show that replacing linear projection layers in the language modeling head, and attention blocks in GPT-2 can result in a 7.5% reduction in perplexity and 34.8% reduction in parameter count.
Section [3.4](https://arxiv.org/html/2506.11035v2#S3.SS4) provides various qualitative analyses of Tversky projection layers, including results highlighting their principled explanability under Tversky’s theory of similarity.
Overall the main contribution of this work is to provide a foundation for
efficient and interpretable neural networks based on an established theory of psychological similarity.

## 2 Methods

### 2.1 Tversky’s Similarity

Tversky’s model of similarity (Tversky, 1977) has emerged as an influential theoretical representation of human similarity judgment, supported by extensive empirical evidence from cognitive psychology (Tversky and Gati, 1982; Gati and Tversky, 1984; Goldstone, 1994; Medin et al., 1993; Rorissa, 2007). His work challenged the geometric model of similarity by demonstrating that humans systematically violate metric axioms such as minimality, symmetry, and triangle inequality when assessing similarity, and proposed a theoretical framework in which objects are represented as sets of features and their similarity assessment as a feature matching process.
Formally, Tversky’s asymmetric model of similarity of object $a$ to object $b$ defined as feature sets $A$ and $B$ is a function $F$ of their common and distinctive features. Tversky showed in his axiomatic framework that $F$ is a linear combination of measures of its set parameters, namely: the common features of $a$ and $b$, the distinctive features of $a$, and the distinctive features of $b$ (Equation [1](https://arxiv.org/html/2506.11035v2#S2.E1)).
In Section [2.2](https://arxiv.org/html/2506.11035v2#S2.SS2), we introduce a differentiable parameterization of this function, making it suitable for gradient-based machine learning.

$$ $\displaystyle S(a,b)=F(A\cap B,A-B,B-A)=\theta f(A\cap B)-\alpha f(A-B)-\beta f(B-A)$ (1) $$

### 2.2 Differentiable Tversky Similarity

This section presents our proposed differentiable parameterization of Tversky’s similarity function, which is constructed with
a representation of features as vectors of the same dimensionality as objects, and
a dual representation of objects as vectors and as sets.

Dual Representation of Objects as Vectors and as Sets:
Given the learnable finite universe $\Omega$ of features vectors $f_{k}\in\mathbb{R}^{d}$, and an object represented as the vector $x\in\mathbb{R}^{d}$,
we propose $x\cdot f_{k}$ to be the scalar measure of feature $f_{k}$ in $x$, and a second representation of $x$ as the set
$X=\{f_{k}\in\Omega|x\cdot f_{k}>0\}$ of features with which $x$ has a positive dot product.

Salience:
Tversky hypothesized that the relative salience of stimuli, or prominence of their features, determines the direction of asymmetry in human’s judgment of similarity. The less salient stimulus (e.g. the son) is assessed to be more similar to the more salient stimulus (e.g. the father) than vice versa.
Following Tversky’s theory and our proposed representation, the salience of features in an object $A$, which is the sum of the measures of all features present in the object is $f(A)$ (Equation [2](https://arxiv.org/html/2506.11035v2#S2.E2)).

$$ $f(A)=\sum\nolimits_{k=1}^{|\Omega|}a\cdot f_{k}\cdot\mathbbm{1}[a\cdot f_{k}>0]$ (2) $$

Feature Set Intersections:
To measure the common features of objects $A$ and $B$, $f(A\cap B)$ (Equation [3](https://arxiv.org/html/2506.11035v2#S2.E3)), we propose a function $\Psi$ to aggregate measures of the features present in both $a$ and $b$, and experiment with values *min*, *max*, *product*, *mean*, *gmean* and *softmin*.
This function corresponds to the
*intersection reduction* hyperparameter of Tversky neural modules.

$$ $\displaystyle f(A\cap B)$ $\displaystyle=\sum\nolimits_{k=1}^{|\Omega|}\Psi\bigl(a\cdot f_{k},b\cdot f_{k}\bigr)\times\mathbbm{1}\bigl[a\cdot f_{k}>0\land b\cdot f_{k}>0\bigr]$ (3) $$

Feature Set Difference:
$f(A-B)$ is a measure of features present in $A$ but not present in $B$ (Equation [4](https://arxiv.org/html/2506.11035v2#S2.E4)). We propose an alternate form of this measure that accounts for features that are present in both $A$ and $B$, but in greater amount in $A$ (Equation [5](https://arxiv.org/html/2506.11035v2#S2.E5)). These two measures of set difference respectively correspond to the values *ignorematch* and *substractmatch* of the *difference reduction* hyperparameter of Tversky neural modules.

$$ $\displaystyle f^{i}(A-B)$ $\displaystyle=\sum\nolimits_{k=1}^{|\Omega|}\bigl(a\cdot f_{k}\bigr)\times\mathbbm{1}\bigl[a\cdot f_{k}>0\land b\cdot f_{k}\leq 0\bigr]$ (4) $\displaystyle f^{s}(A-B)$ $\displaystyle=f^{i}(A-B)+\sum\nolimits_{k=1}^{|\Omega|}\bigl(a\cdot f_{k}-b\cdot f_{k}\bigr)\times\mathbbm{1}\bigl[b\cdot f_{k}>0\land a\cdot f_{k}>b\cdot f_{k}\bigr]$ (5) $$

### 2.3 Tversky Neural Network Modules

We propose two basic building blocks for Tversky neural networks, the *Tversky Similarity Layer*, which is analogous to metric similarity functions such as dot product or cosine similarity, and the *Tversky Projection Layer*, analogous to the contemporary *fully connected layer*.

##### Tversky Similarity Layer:

This layer, formalized in Equation [6](https://arxiv.org/html/2506.11035v2#S2.E6), calculates the similarity of object $a\in\mathbb{R}^{d}$ to object $b\in\mathbb{R}^{d}$. Its learnable parameters are a feature bank $\Omega$, and the $\alpha$, $\beta$ and $\theta$ scalar parameters of Tverky’s contrast model of similarity (Equation [1](https://arxiv.org/html/2506.11035v2#S2.E1)).

$$ $\displaystyle\mathcal{S}^{\Omega,\alpha,\beta,\theta}(a,b)\colon\begin{cases}\mathbb{R}^{d}\times\mathbb{R}^{d}&\longrightarrow\,\,\,\mathbb{R}\\ (a,b)&\longmapsto\,\,\,\begin{bmatrix}\theta\\ -\alpha\\ -\beta\\ \end{bmatrix}\cdot\begin{bmatrix}f(A\cap B)\\ f(A-B)\\ f(B-A)\\ \end{bmatrix}\end{cases}$ (6) $$

##### Tversky Projection Layer:

This non-linear projection (Equation [7](https://arxiv.org/html/2506.11035v2#S2.E7)), calculates the similarity of its input $a\in\mathbb{R}^{d}$ to each prototype in the ordered set of $p$ prototypes $\Pi_{i}\in\mathbb{R}^{d}$, yielding a vector in $\mathbb{R}^{p}$.

$$ $\displaystyle\mathcal{P}^{\Omega,\alpha,\beta,\theta,\Pi}(a)\colon\begin{cases}\mathbb{R}^{d}&\longrightarrow\,\,\,\mathbb{R}^{p}\\ a&\longmapsto\,\,\,\begin{bmatrix}\mathcal{S}^{\Omega,\alpha,\beta,\theta}(a,\Pi_{0})\\ \mathcal{S}^{\Omega,\alpha,\beta,\theta}(a,\Pi_{1})\\ \vdots\\ \mathcal{S}^{\Omega,\alpha,\beta,\theta}(a,\Pi_{p-1})\\ \end{bmatrix}\end{cases}$ (7) $$

### 2.4 Interpretation of Projection Layers:

Both the linear and Tversky projection layers output vectors in which each dimension is the similarity of the input to a prototype; however, they differ in the employed similarity function.
Linear projection layers
compute dot-product similarities. This is an application of the geometric model of simlarity, in which the features of objects and prototypes are scalar values arranged in a cartesian coordinate space and similarity is measured as the oriented length of the object vector’s projection onto the prototype vector. Dot product similarities have metric properties such as asymmetry and triangle inequality that have been empirically shown to not align with human perception, and proven to be incapable of modeling asymmetric relations.
Tversky projection layers
compute Tversky similarity with learnable parameters prototype vectors $\Pi$, feature vectors $\Omega$, and Tversky’s contrast model’s weights $\alpha,\beta,and\theta$. The number of features $|\Omega|$ can be varied without affecting the output dimensionality $p=|\Pi|$.

#### 2.4.1 Tversky Feature Sharing

Tversky feature banks and prototype banks can be shared across various layers in a neural network in semantically justifiable ways.
For instance, in a Tversky GPT-2,
the output projection layers in attention blocks, and the final language modeling head can all share the same feature bank $\Omega$ as this parameter is semantically compatible across all those layers: it represents token features.
Tversky projection layers that are language modeling heads can also use token embeddings as token prototypes. This parameter sharing strategy is similar to *weight tying* widely used in language models employing a linear projection layer to classify output tokens (Inan et al., 2016).
Multiple Tversky similarity or projection layers can share their feature or prototype bank if semantically compatible. They can also share the same feature bank while maintaining separate prototypes, or keep all their parameters separate.
Our results in Section [3.2](https://arxiv.org/html/2506.11035v2#S3.SS2) show that Tversky feature sharing can result in a dramatic reduction of neural network parameter count while improving their performance.

### 2.5 Data-Domain Visualization of Projection Layers

We introduce a novel method of visualizing projection layer parameters such as the weights of fully connected layers, and the prototypes and features of Tversky projection layers to enable their interpretation in the same domain as data stimuli.
Our method is based on the hypothesis that those prototypes and features are concepts that should be recognizable in the data domain, and is applicable regardless of the depth at which those projection layers are employed in deep neural networks.
Our proposed visualization method consists of specifying the projection parameters as tensors of the same shape as the input data. These input-domain-specified prototypes and features are forwarded through the neural network just like data stimuli up to the layer prior to the layer in which they are used. Their obtained vector representation is subsequently used to perform the projection.
This approach significantly differs from prior approaches of interpreting neural network parameters and decisions based on
the visualization of activations,
optimization of input stimuli,
or construction of adversarial examples (Zeiler and Fergus, 2014; Mordvintsev et al., 2015; Selvaraju et al., 2016; Zhang and Zhu, 2018; Zhang and Liang, 2020; Fan et al., 2021).
While our proposed method offers the clear advantage of visualizing projection parameters, it comes with the limitation that parameters specified in data-space are typically larger in size than their original counterparts, which increases the effective number of trainable parameters. These parameters also need to be forwarded along with every data batch, which induces additional training-time computation cost (See
Figure [5](https://arxiv.org/html/2506.11035v2#A2.F5) in Appendix
[B](https://arxiv.org/html/2506.11035v2#A2)).
We use this technique to qualitatively compare the parameters of a linear projection layer and a Tversky projection layer of neural networks trained to recognize handwritten digits in our qualitative analysis experiment presented in Section [3.4](https://arxiv.org/html/2506.11035v2#S3.SS4), which revealed that Tversky Projection layers’s parameters are far more interpretable than the ones of the contemporary fullly connected layer (see Figure [2](https://arxiv.org/html/2506.11035v2#S3.F2)).

## 3 Experiments

Here we demonstrate the utility of our proposed differentiable Tversky similarity function in machine learning.
First, we show by construction that a single Tversky projection layer can model the xor function and report empirical results on the same learning task (Section [3.1](https://arxiv.org/html/2506.11035v2#S3.SS1)).
Then, in Sections [3.2](https://arxiv.org/html/2506.11035v2#S3.SS2) and [3.3](https://arxiv.org/html/2506.11035v2#S3.SS3), we conduct empirical experiments with state-of-the-art neural networks for language modeling and image recognition in which we compare baseline neural networks with their counterparts employing Tversky projection layers.
Finally, in Section [3.4](https://arxiv.org/html/2506.11035v2#S3.SS4), we conduct qualitative analyses demonstrating the interpretability of Tversky neural networks.

### 3.1 Modeling XOR with a Single Tversky Projection Layer

As a first experiment, we construct a single Tversky projection layer that computes the xor function, which is not computable by a single linear projection due to the required non-linear decision boundary.
Figure [1](https://arxiv.org/html/2506.11035v2#S3.F1) shows the constructed projection, and its data and parameter vectors along with their set-centric interpretation.
In empirical experiments in which we train a Tversky projection to learn xor with gradient descent under various hyperparameter conditions, we find that:
Some initializations of prototypes and features lead to convergence failure;
Initializing those parameters from a uniform distribution leads to higher convergence probability compared to normal and orthogonal initialization;
Normalizing prototype and object vectors deteriorates convergence probability;
*product* and *substractmatch* work best as values of the *intersection reduction* and *difference reduction* hyperparameters;
Convergence probability doesn’t increase monotonically with the feature bank size;
and finally, Tversky projection can model xor with as little as one feature. See Appendix [C](https://arxiv.org/html/2506.11035v2#A3).

Figure: Figure 1: (left) A single tversky projection layer with 11 learnable parameters that computes the xor function. The input is a 2 digit binary number encoded as $x^{i}\in\mathcal{R}^{2}$. The output $\hat{y}^{i}$ is such that $txor(x^{i})\iff s(x^{i},p^{1})>s(x^{i},p^{0})$, where $s$ is the proposed similarity function. (middle) Input vectors $x^{i}$ along with features $f^{i}$, and prototypes $p^{i}$ modeling the $xor$ function. The dot products $x^{1}\cdot f^{1}$, $x^{2}\cdot f^{0}$, $p^{1}\cdot f^{0}$ and $p^{1}\cdot f^{1}$ are positive. (right) Feature vectors define a universe in which input vectors $x^{0}=\{\}$, $x^{1}=\{f^{1}\}$, $x^{2}=\{f^{0}\}$, $x^{3}=\{\}$, and learned prototype vectors $p^{0}=\{\}$, $p^{1}=\{f^{0},f^{1}\}$, are represented as set of features. By convention, an object (input or prototype) contains a feature if its dot product with the feature vector is positive. By Tversky’s contrast model of similarity, $s(x^{3},p^{0})>s(x^{3},p^{1})$, and $s(x^{2},p^{1})>s(x^{2},p^{0})$ therefore $txor([1,1])=0$ and $txor([1,0])=1$. Similarly for other inputs, $txor([0,0])=0$ and $txor([0,1])=1$.
Refer to caption: x1.png

### 3.2 Language Modeling with TverskyGPT-2

Table: Table 1: Perplexity (PPL) and parameter counts (Params) of GPT2 and TverskyGPT2 trained for the language modeling task on the PTB dataset. Perplexities are reported on the held out test split.

#### 3.2.1 Task and Data

We compare the baseline GPT-2 small (Radford et al., 2019) model with its Tversky variants on the Penn Treebank (PTB) language modeling benchmark (Marcus et al., 1993).

#### 3.2.2 Method

The following TverskyGPT-2 variants were considered:
*tversky-head*, which replaces GPT-2’s language modeling head’s stack of linear layers with a Tversky projection layer, and
*tversky-all*, which also replaces GPT-2’s intermediate attention blocks’ output projections with Tversky projections.
Tversky feature sharing is employed across all Tversky projection layers.
For all models, we experiment with prototype tying (tied vs not tied) and initialization (random vs OpenAI’s released weights) while training for 50 epochs on PTB’s training set, and validating on PTB validation set.
The baseline models, and best Tversky models are evaluated on PTB’s held-out test set.

#### 3.2.3 Results

Tversky language models matched or surpassed baseline perplexity in all settings except the one in which
pre-trained weights and prototype tying are employed
(Table [1](https://arxiv.org/html/2506.11035v2#S3.T1)). In that setting, Tversky prototypes and features are still randomly initialized, while the baseline fully connected layers are initialized from the pretrained token embedding matrix.
The perplexity gap between the best Tversky neural network and baseline was higher when training from scratch, with a 7.8% reduction in perplexity when prototypes are tied to input embeddings, and a 12.1% reduction when they are not. In the tied prototype setting, the best Tversky model also has 34.8% fewer parameters.
See
Appendix [D](https://arxiv.org/html/2506.11035v2#A4) for validation results, and Table [1](https://arxiv.org/html/2506.11035v2#S3.T1) for test results.

### 3.3 Image Recognition with TverskyResNet50

#### 3.3.1 Task and Data

We experiment with replacing the final linear projection layer in ResNet50 (He et al., 2016) with a Tversky projection and report the accuracy of the baseline and Tversky variants on
MNIST (LeCun et al., 1998) handwritten digits
and NABirds (Van Horn et al., 2015) bird species classification tasks.

#### 3.3.2 Method

ResNet-50 and TverskyResNet-50, which uses a Tversky Projection instead of the final fully connected layer, are trained with the same hyperparameters (200 epochs; batch sizes: (NABirds, 256), (MNIST, 1024); lr: 0.03; momentum 0.9; weight decay $10^{-8}$; dropout: 0.1; and gradient clipping threshold $10$). In the *Pretrained backbone* condition, ResNet50 is initialized with pretrained ImageNet weights. In the *Freeze Backbone* condition, the convolution layer weights are not updated during training. Tversky prototypes and features are randomly initialized in all settings.
Tversky feature intersection and difference reduction hyperparameters (*product* and *ignorematch*) were informed by our results in Section [3.2](https://arxiv.org/html/2506.11035v2#S3.SS2). We chose Tversky feature bank sizes 20 and 224 for MNIST and NABirds respectively.
Unlike in experiment [3.2](https://arxiv.org/html/2506.11035v2#S3.SS2), which employs a strict train/validation/test protocol, no extensive hyperparameter tuning, and validation was employed in this experiment because the employed datasets, despite having sizeable test sets, do not have official distinct validation sets.
Nonetheless the following results demonstrate the existence of settings under which Tversky vision models match or surpass baseline accuracy on official train/test splits.

#### 3.3.3 Results

Classification accuracies reported in table [2](https://arxiv.org/html/2506.11035v2#S3.T2) show that Tversky vision models can match or surpass baseline accuracy under the specified experimental conditions. In particular, when Tversky projection was used as a domain adapter with a frozen backbone, $8.5\%$ and $24.7\%$ improvement over baseline was observed.
In this experiment, the Tversky neural networks have more parameters than the baseline, because their parameters also include a feature bank in addition to the prototype bank while the baseline only contains a prototype bank, corresponding to the weights of the fully connected layer.
While the increased parameter count of TverskyResNet-50 could contribute to the observed accuracy improvement, our results in Section [3.2](https://arxiv.org/html/2506.11035v2#S3.SS2) show that the use of Tversky neural networks can simultaneously result in decreased parameter count and increased performance compared to baseline.

**Table 2: Accuracy of Resnet-50 (Baseline) and Tversky-Resnet-50 (Tversky) on the tasks of MNIST handwritten digit classification and NABirds bird species classification. *Pretrained* (True when weights are initialized from ImageNet, False when they are randomly initialized). *Frozen* (True when only the final projection layer is finetuned, False when the entire model is finetuned)**
|  |  | MNIST | NABirds |  |  |
| --- | --- | --- | --- | --- | --- |
| Pretrained | Frozen | Tversky | Baseline | Tversky | Baseline |
| True | True | 62.3 | 57.4 | 44.9 | 36.0 |
| True | False | 99.3 | 99.2 | 83.7 | 83.6 |
| False | False | 98.6 | 98.0 | 65.7 | 61.3 |

### 3.4 Qualitative Analysis of Tversky Neural Networks

Figure: Figure 2: Visualization of prototypes using our input domain projection parameter specification method. Top: 3 examples for each class. Bottom: 10 columns of the final linear projection layer of VisualMNISTNet, and 10 prototype vectors of the Tversky Projection Layer of TverskyMNISTNet. Both models achieve 99% accuracy on the validation set. Handwritten digit features are more perceptible to humans in Tversky prototypes and features compared to the linear projection prototypes.
Refer to caption: x3.png

Here we present results showing the interpretability of Tversky neural networks.
We discuss the explainability of learned prototypes, the quantification of salience, and the algebraic specification of semantic fields enabled by Tversky neural networks.

#### 3.4.1 Method

Prototype Visualization:
To visually compare the prototypes learned by Tversky Neural Networks and classical neural networks,
we employ the prototype visualization method described in Section [2.5](https://arxiv.org/html/2506.11035v2#S2.SS5) and illustrated in Appendix [B](https://arxiv.org/html/2506.11035v2#A2).
This method is used to display the handwritten digit prototypes learned by VisualMNISTNet and VisualTverskyMNISTNet,
two neural networks inspired by LeNet-5 (LeCun et al., 1998).
Further details are provided in Appendix [F](https://arxiv.org/html/2506.11035v2#A6).

Salience:
To assess whether salience, as computed using Equation [2](https://arxiv.org/html/2506.11035v2#S2.E2), is intelligible to humans,
we rank MNIST examples by their salience score.

Semantic Fields:
Tversky Neural Networks’ repesentation of objects as sets of features allows the algebraic specification of *semantic fields*.
A semantic field is a subset of features resulting from set expressions that are interpretable by humans.
For instance, $A\cap B-C$ describes the semantic field capturing the common features of $A$ and $B$ that are distinct from $C$.
To visualize a semantic field $F$, we rank examples $x_{i}$ by their score in that field:
$s=\sum_{f_{k}\in F}{f_{k}\cdot x_{i}}$.
This enables the visualization of semantics captured by TverskyMNISTNet’s representation of images and TverskyGPT2 representation of tokens.

#### 3.4.2 Results

Prototypes:
Figure [2](https://arxiv.org/html/2506.11035v2#S3.F2) shows that Tversky prototypes exhibit stroke patterns similar to human handwriting,
such as lines and curves, more clearly than those learned by linear projection layers, which exhibit texture patterns that are difficult to interpret.
This lack of interpretability, even in this simple domain without background textures, represents a significant limitation of prior approaches.
Tversky prototypes are also more *salient* than individual instances:
they exhibit more features and appear to combine the features of all possible instances of their class.
For example, the prototype for the handwritten “7” has both an upper-left serif and a middle horizontal stroke.
The prototypes for the digit “1” and “9” include vertical lines slanted to the right and to the left.
The prototype for class “9” combines both open and closed top loops.
Figure [15](https://arxiv.org/html/2506.11035v2#A6.F15) shows the evolution of learned prototypes over training epochs for both VisualMNISTNet and TverskyVisualMNISTNet.
We observed that $\alpha>\beta$ in trained Tversky neural networks (see Appendix [G](https://arxiv.org/html/2506.11035v2#A7)),
indicating that they weigh the distinctive features of individual instances more heavily than those of prototypes, which is in line with Tversky’s theory.

Salience:
Figure [3](https://arxiv.org/html/2506.11035v2#S3.F3) shows MNIST examples ranked from low to high salience according to Equation [2](https://arxiv.org/html/2506.11035v2#S2.E2).
This ranking aligns with Tversky’s empirical psychological observations that humans perceive stimuli exhibiting more *goodness of form* as more salient.

Semantic Fields
Table [3](https://arxiv.org/html/2506.11035v2#S3.T3) shows examples of semantic fields defined with TverskyGPT2 tokens, and top scoring tokens in those semantic fields. These results and additional results in Appendix [E](https://arxiv.org/html/2506.11035v2#A5) show that interpretable semantic fields such as those capturing the concepts of *adjectives*, *comparatives*, *superlatives*, *verb forms*, or specific word *senses* can be specified algebraically with Tversky Language models.
Similar results can be achieved with Tversky vision models. Figure [16](https://arxiv.org/html/2506.11035v2#A6.F16) shows two MNIST digits, and examples that illustrate their common and distinctive features.

**Table 3: Semantic concepts, their algebraic specification using set operations on TverskyGPT2 tokens represented as sets, and top tokens ranked by semantic score in the resulting semantic field. Tversky neural networks enable the algebraic specification of interpretable semantic concepts.**
| Concept | Semantic Expression | Top-scoring tokens |
| --- | --- | --- |
| Adjective | $bad\cap good-worse-better-worst-best$ | *lousy*, *evil*, *crappy*, *poor*, *shitty*, *nice*, *terrible*, *horrible*, *decent*, *mediocre*, *valid*, *great*, *excellent* |
| Comparative | $worse\cap better-bad-good-worst-best$ | *sharper*, *nicer*, *clearer*, *smarter*, *wiser*, *smoother*, *safer*, *hotter*, *happier*, *tighter*, *louder*, *preferable* |
| Superlative | $worst\cap best-worse-better-bad-good$ | *happiest*, *safest*, *quickest*, *deadliest*, *finest*, *hottest*, *busiest*, *toughest*, *darkest*, *fastest*, *brightest*, *holiest* |
| Industrial Plant: | $plant-aceae-vegetation-vegetable-planting-flower-herb-vine-crop-tree-mushroom-plantation-leaves$ | *facility*, *stall*, *brate*, *factory*, *Laboratories*, *Shed*, *implant*, *Berm*, *Sew*, *Manufacturing*, *reactor*, *Diesel*, *refinery*, *strain*, *distribut*, *planting*, *Industrial*, *microbial*, *actory*, *reactors*, *Unit*, *Install*, *depot*, *spill*, *Cutter*, *Indust*, *Bot*, *Nuclear*, *ineries* |

Figure: Figure 3: MNIST digits sorted from low to high salience (Equation [2](https://arxiv.org/html/2506.11035v2#S2.E2)). Like humans, the underlying Tversky Neural Network perceives stimuli exhibiting more *goodness of form* as more salient.
Refer to caption: exp_009/mnist-salience.png

## 4 Discussion

##### Hyperparameter Tuning:

Our language modeling experiments suggest the existence of an optimal feature bank size, though determining this value *a priori* remains an open question.
In the XOR setting, we observed that Tversky networks can be sensitive to hyperparameters, underscoring the importance of standard tuning practices.
For our vision experiments, no tuning was performed; our goal was simply to demonstrate the existence of Tversky models that can match or surpass baseline accuracy.
We encourage practitioners to apply standard hyperparameter tuning to further improve performance, especially since Tversky networks are a novel architecture, whereas the baselines we compare against have benefited from years of community-driven refinement.

Weight tying in language models is adequate.
Our experiments with GPT-2 on PTB showed that weight tying can be beneficial in both language modeling heads employing Tversky and linear projections.
Our interpretation of projection layers as measurement of similarity of stimuli to prototypes explains the adequacy of weight tying in language models: Token embeddings are prototypes, and the language modeling head classifies output tokens by measuring their similarity to those prototypes. This practice with Tversky projection layers is further principled given the underlying theory of similarity.

Data-domain projection parameter specification could enable interpretable editing.
For instance, excluding handwritten sevens with the mid horizontal stroke from VisualTverskyMNISTNet’s training set would result in a prototype without that stroke, and high error rates on handwritten sevens with that stroke. The resulting model could be edited by adding a feature specifying the stroke, and editing the prototype to include the stroke. This neural network editing approach could enable the understanding and eradication of dangerous model biases.

## Conclusion

This work introduces a differentiable similarity function based on a psychologically plausible theory of similarity.
The Tversky similarity layer and Tversky projection layer serve as basic neural network building blocks implementing this theory.
We illustrated the Tversky projection layer in simplified settings that permit qualitative analysis, as well as in larger-scale image recognition and language modeling tasks.
Our experiments provide compelling evidence that our introduced neural modules are suitable for psychologically plausible deep learning while offering principled explainability, performance gains and parameter efficiency.

## References

- Tversky [1977]
Amos Tversky.
Features of similarity.
*Psychological review*, 84(4):327, 1977.
- Bridle [1990]
John S Bridle.
Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition.
In *Neurocomputing: Algorithms, architectures and applications*, pages 227–236. Springer, 1990.
- LeCun et al. [1998]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.
Gradient-based learning applied to document recognition.
*Proceedings of the IEEE*, 86(11):2278–2324, 1998.
- Hochreiter and Schmidhuber [1997]
Sepp Hochreiter and Jürgen Schmidhuber.
Long short-term memory.
*Neural Computation*, 9:1735–1780, 1997.
- Graves et al. [2005]
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber.
Bidirectional LSTM networks for improved phoneme classification and recognition.
In *International conference on artificial neural networks*, pages 799–804. Springer, 2005.
- Graves [2013]
Alex Graves.
Generating sequences with recurrent neural networks.
*arXiv preprint arXiv:1308.0850*, 2013.
- Mikolov [2013]
Tomas Mikolov.
Efficient estimation of word representations in vector space.
*arXiv preprint arXiv:1301.3781*, 3781, 2013.
- Pennington et al. [2014]
Jeffrey Pennington, Richard Socher, and Christopher D Manning.
Glove: Global vectors for word representation.
In *Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)*, pages 1532–1543, 2014.
- Bengio et al. [2000]
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent.
A neural probabilistic language model.
*Advances in neural information processing systems*, 13, 2000.
- Sutskever [2014]
I Sutskever.
Sequence to sequence learning with neural networks.
*arXiv preprint arXiv:1409.3215*, 2014.
- Devlin [2018]
Jacob Devlin.
Bert: Pre-training of deep bidirectional transformers for language understanding.
*arXiv preprint arXiv:1810.04805*, 2018.
- Radford et al. [2018]
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.
Improving language understanding by generative pre-training.
Technical report, OpenAI, 2018.
URL [https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf).
OpenAI Technical Report.
- Achiam et al. [2023]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Bahdanau et al. [2014]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.
Neural machine translation by jointly learning to align and translate.
*arXiv preprint arXiv:1409.0473*, 2014.
- Vaswani [2017]
A Vaswani.
Attention is all you need.
*Advances in Neural Information Processing Systems*, 2017.
- Legendre [1806]
Adrien Marie Legendre.
*Nouvelles méthodes pour la détermination des orbites des comètes: avec un supplément contenant divers perfectionnemens de ces méthodes et leur application aux deux comètes de 1805*.
Courcier, 1806.
- Hadsell et al. [2006]
Raia Hadsell, Sumit Chopra, and Yann LeCun.
Dimensionality reduction by learning an invariant mapping.
In *2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06)*, volume 2, pages 1735–1742. IEEE, 2006.
- Wang et al. [2018]
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.
Cosface: Large margin cosine loss for deep face recognition.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 5265–5274, 2018.
- van den Oord et al. [2018]
Aaron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding.
*arXiv preprint arXiv:1807.03748*, 2018.
- Chen et al. [2020]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.
A simple framework for contrastive learning of visual representations.
In *International conference on machine learning*, pages 1597–1607. PMLR, 2020.
- Gati and Tversky [1984]
Itamar Gati and Amos Tversky.
Weighting common and distinctive features in perceptual and conceptual judgments.
*Cognitive Psychology*, 16(3):341–370, 1984.
- Ritov et al. [1990]
Ilana Ritov, Itamar Gati, and Amos Tversky.
Differential weighting of common and distinctive components.
*Journal of Experimental Psychology: General*, 119(1):30, 1990.
- Siegel et al. [1982]
Paul S Siegel, David M McCord, and Alice Reagan Crawford.
An experimental note on tversky’s “features of similarity”.
*Bulletin of the Psychonomic Society*, 19:141–142, 1982.
- Rorissa [2007]
Abebe Rorissa.
Relationships between perceived features and similarity of images: A test of Tversky’s contrast model.
*Journal of the American Society for Information Science and Technology*, 58(10):1401–1418, 2007.
- Garg et al. [2015]
Ankita Garg, Catherine G. Enright, and Michael G. Madden.
On asymmetric similarity search.
In *2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA)*, page 649–654, Miami, FL, USA, December 2015. IEEE.
ISBN 978-1-5090-0287-0.
doi: 10.1109/ICMLA.2015.128.
URL [http://ieeexplore.ieee.org/document/7424392/](http://ieeexplore.ieee.org/document/7424392/).
- Chen et al. [2017]
Dawn Chen, Joshua C. Peterson, and Thomas L. Griffiths.
Evaluating vector-space models of analogy.
*arXiv preprint arXiv:1705.04416*, June 2017.
URL [http://arxiv.org/abs/1705.04416](http://arxiv.org/abs/1705.04416).
- Peterson et al. [2020]
Joshua C Peterson, Dawn Chen, and Thomas L Griffiths.
Parallelograms revisited: Exploring the limitations of vector space models for simple analogies.
*Cognition*, 205:104440, 2020.
- Zhou et al. [2022]
Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card, and Dan Jurafsky.
Problems with cosine as a measure of embedding similarity for high frequency words.
*arXiv preprint arXiv:2205.05092*, 2022.
- Rumelhart et al. [1986]
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.
Learning representations by back-propagating errors.
*Nature*, 323(6088):533–536, 1986.
- Tversky and Gati [1982]
Amos Tversky and Itamar Gati.
Similarity, separability, and the triangle inequality.
*Psychological review*, 89(2):123, 1982.
- Goldstone [1994]
Robert L Goldstone.
The role of similarity in categorization: Providing a groundwork.
*Cognition*, 52(2):125–157, 1994.
- Medin et al. [1993]
Douglas L Medin, Robert L Goldstone, and Dedre Gentner.
Respects for similarity.
*Psychological review*, 100(2):254, 1993.
- Inan et al. [2016]
Hakan Inan, Khashayar Khosravi, and Richard Socher.
Tying word vectors and word classifiers: A loss framework for language modeling.
*arXiv preprint arXiv:1611.01462*, 2016.
- Zeiler and Fergus [2014]
Matthew D Zeiler and Rob Fergus.
Visualizing and understanding convolutional networks.
In *Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13*, pages 818–833. Springer, 2014.
- Mordvintsev et al. [2015]
Alexander Mordvintsev, Christopher Olah, and Mike Tyka.
Inceptionism: Going deeper into neural networks.
*Google research blog*, 20(14):5, 2015.
- Selvaraju et al. [2016]
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra.
Grad-cam: Why did you say that?
*arXiv preprint arXiv:1611.07450*, 2016.
- Zhang and Zhu [2018]
Quan-Shi Zhang and Song-Chun Zhu.
Visual interpretability for deep learning: a survey.
*Frontiers of Information Technology & Electronic Engineering*, 19(1):27–39, 2018.
- Zhang and Liang [2020]
Qiyang Zhang and Dong Liang.
Visualization of fully connected layer weights in deep learning ct reconstruction.
*arXiv preprint arXiv:2002.06788*, 2020.
- Fan et al. [2021]
Feng-Lei Fan, Jinjun Xiong, Mengzhou Li, and Ge Wang.
On interpretability of artificial neural networks: A survey.
*IEEE Transactions on Radiation and Plasma Medical Sciences*, 5(6):741–760, 2021.
- Radford et al. [2019]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.
Language models are unsupervised multitask learners.
*OpenAI blog*, 1(8):9, 2019.
- Marcus et al. [1993]
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz.
Building a large annotated corpus of English: The Penn Treebank.
*Computational Linguistics*, 19(2):313–330, 1993.
URL [https://aclanthology.org/J93-2004/](https://aclanthology.org/J93-2004/).
- He et al. [2016]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 770–778, 2016.
- Van Horn et al. [2015]
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie.
Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 595–604, 2015.

## Appendix A Modeling 2 Digits Binary Addition with a Tversky Projection Layer

Figure: Figure 4: A tversky layer that adds 2 binary digits. Inputs are encoded in $\mathcal{R}^{3}$ as $x^{0}=[0,0,0]$, $x^{1}=[0,1,0]$, $x^{2}=[1,0,0]$ and $x^{3}=[1,1,1]$. The result of the addition corresponds to the prototype to which the input is most similar. Compared to the txor model, this model employs one additional dimention to enable the introduction of a feature $f^{2}=[0,0,1]$ only shared by $x^{3}$ and $p^{2}=[1,1,0.5]$. In this configuration, we have $tadd([0,0])=0$, $tadd([0,1])=tadd([1,0])=1$ and $tadd([1,1])=2$
Refer to caption: x4.png

## Appendix B Illustration of Data-Domain Visualization of Projection Layers

Figure: Figure 5: Illustration of our proposed data-domain visualization of projection parameters. Left: Classical deep neural network with a deep projection layer (Layer n) and its projection parameters illustrated as blue vectors. Right: Our proposed method. Projection parameters are specified as tensors of the same shape as the input data, and forwarded through the neural network up to layer n-1. The obtained vectors are used as projection parameters in Layer n. Using this method, the effective neural network parameter count is larger. However, this method enables visualizing the projection parameters in the input domain.
Refer to caption: x6.png

## Appendix C Training a Tversky Projection to Learn XOR with Gradient Descent

### C.1 Task and Data

In this experiment, we train a single Tversky projection layer to learn the xor function via gradient descent under various hyper-parameter conditions.
Figure [6](https://arxiv.org/html/2506.11035v2#A3.F6)
shows that some initializations of Tversky projection layers may lead to gradient descent optimization failure.
This experiment empirically analyses its sensitivity to hyperparameters when trained to model the xor function.

### C.2 Method

To empirically estimate the sensitivity of Tversky projection layer’s convergence to its hyperparameters, we train $12\,960$ xor models consisting of the following combinations of hyperparameters:

- •
6 intersection reduction methods {*min*, *max*, *product*, *mean*, *gmean*, *softmin*};
- •
2 difference reduction methods {*ignorematch*, *substractmatch*};
- •
2 normalization modes: {false, true};
- •
6 feature counts: {1, 2, 4, 8, 16, 32};
- •
3 prototype initialization distributions {uniform, normal, orthogonal};
- •
3 feature bank initialization distributions {uniform, normal, and orthogonal };
- •
9 random seeds.

Each model is trained for 1000 epochs. Tables [4](https://arxiv.org/html/2506.11035v2#A3.T4), [5](https://arxiv.org/html/2506.11035v2#A3.T5), [6](https://arxiv.org/html/2506.11035v2#A3.T6), and [7](https://arxiv.org/html/2506.11035v2#A3.T7) reports the average and standard error of the convergence indicator (whether accuracy is 100%) of the trained models marginalized by various combinations of hyperparameters. We refer to this variable as convergence probability $p(conv)$ in our results.

### C.3 Results

##### Initialization of prototypes and features

Initializing both features and prototypes by sampling from the uniform distribution resulted in the highest convergence probability (Table [5](https://arxiv.org/html/2506.11035v2#A3.T5)).

##### Reduction of measures of intersections and differences

Using *product* and *substractmatch* resulted in the highest convergence probability. (Table [4](https://arxiv.org/html/2506.11035v2#A3.T4))

##### Normalization

In this experiment, normalizing prototype and object vectors prior to calculating Tversky similarity decreased the convergence probability. (Table [6](https://arxiv.org/html/2506.11035v2#A3.T6))

##### Feature count

Tversky projection layer successfully modeled $xor$ with as little as 1 feature. $p(conv)$ was maximal with 16 features, but not monotonously increase with feature count.

While linear projection layers cannot learn non-linear decision boundaries without composition with non-linear *activation* functions, Figure [7](https://arxiv.org/html/2506.11035v2#A3.F7) shows that a single Tversky projection layers can learn complex non-linear decision boundaries even in low-dimensional vector space.

Figure: Figure 6: Example of learned TverskyProjection layers optimized to model the xor function. While multiple solutions are possible (top row), some initializations do not lead to convergence (botton row). The decision boundaries were drawn as follows. Random input vectors [x,y] are uniformly sampled in the range [-2,-2] to [2,2] the tip of each vector is ploted as a colored dot, with the color representing the trained Tversky Projection layer’s output. Tversky projection layers modeling the $xor$ function, which has a boolean domain and range extend $xor$ to the real vector space. Successful models should show data points [0,0] and [1,1] in purple (class 0), and [0,1] and [1,0] in yellow (class 1)
Refer to caption: x7.png

Figure: Figure 7: Example of successful Tversky decision boundaries modeling the XOR function with 2, 4, 8, 16, 32, 64, 128 and 256 features. The presence of clusters of features are frequent in overparameterized models.
Refer to caption: x8.png

**Table 4: Mean $\pm$ std error of loss and accuracy for TverskyXORNet with 6 intersection reduction methods ($A\cap B$) and 2 difference reduction methods ($A-B$). Each row corresponds to the aggregate of 3x3x2x6x9 = 972 training sessions with: 3 fbank and prototype initialization methods(orthogonal, normal, uniform), 2 normalization modes (normalized, and not normalized), 6 feature bank sizes (1, 2, 4, 8, 16, 32), and 9 random seeds. In each case, models were trained for 1000 epochs**
| $A\cap B$ | $A-B$ | n | loss | acc | best acc | p(conv) |
| --- | --- | --- | --- | --- | --- | --- |
| product | substractmatch | 972 | $0.27\pm 0.01$ | $0.83\pm 0.01$ | 1.0 | $0.53\pm 0.02$ |
| mean | substractmatch | 972 | $0.28\pm 0.01$ | $0.82\pm 0.01$ | 1.0 | $0.51\pm 0.02$ |
| max | ignorematch | 972 | $0.35\pm 0.01$ | $0.78\pm 0.01$ | 1.0 | $0.47\pm 0.02$ |
| max | substractmatch | 972 | $0.32\pm 0.01$ | $0.80\pm 0.01$ | 1.0 | $0.44\pm 0.02$ |
| softmin | substractmatch | 972 | $0.35\pm 0.01$ | $0.80\pm 0.01$ | 1.0 | $0.42\pm 0.02$ |
| min | ignorematch | 972 | $0.31\pm 0.01$ | $0.78\pm 0.01$ | 1.0 | $0.42\pm 0.02$ |
| softmin | ignorematch | 972 | $0.34\pm 0.01$ | $0.78\pm 0.01$ | 1.0 | $0.38\pm 0.02$ |
| mean | ignorematch | 972 | $0.42\pm 0.01$ | $0.73\pm 0.01$ | 1.0 | $0.26\pm 0.01$ |
| product | ignorematch | 972 | $0.41\pm 0.01$ | $0.75\pm 0.01$ | 1.0 | $0.23\pm 0.01$ |
| min | substractmatch | 972 | $0.50\pm 0.00$ | $0.68\pm 0.00$ | 1.0 | $0.02\pm 0.00$ |
| gmean | ignorematch | 972 | $nan\pm nan$ | $0.50\pm 0.00$ | 0.5 | $0.00\pm 0.00$ |
| gmean | substractmatch | 972 | $nan\pm nan$ | $0.50\pm 0.00$ | 0.5 | $0.00\pm 0.00$ |

**Table 5: Mean $\pm$ std error of loss and accuracy for TverskyXORNet with 3 fbank and prototype initialization methods (*fbank init*, and *proto init*). Each row corresponds to the aggreate of 6x2x2x6x9 = 1,296 training sessions with: 6 intersection reduction methods (min, max, product, mean, gmean, softmin) and 2 difference reduction methods (ignorematch, substractmatch), 2 normalization modes (normalized, and not normalized), 6 feature bank sizes (1, 2, 4, 8, 16, 32), and 9 random seeds. In each case, models were trained for 1000 epochs.**
| fbank init | proto init | n | loss | acc | best acc | p(conv) |
| --- | --- | --- | --- | --- | --- | --- |
| uniform | uniform | 1296 | $0.27\pm 0.01$ | $0.78\pm 0.01$ | 1.0 | $0.41\pm 0.01$ |
| uniform | normal | 1296 | $0.36\pm 0.01$ | $0.72\pm 0.01$ | 1.0 | $0.34\pm 0.01$ |
| normal | uniform | 1296 | $0.34\pm 0.01$ | $0.74\pm 0.01$ | 1.0 | $0.32\pm 0.01$ |
| normal | normal | 1296 | $0.35\pm 0.01$ | $0.74\pm 0.01$ | 1.0 | $0.31\pm 0.01$ |
| uniform | orthogonal | 1296 | $0.39\pm 0.01$ | $0.70\pm 0.01$ | 1.0 | $0.30\pm 0.01$ |
| orthogonal | uniform | 1296 | $0.37\pm 0.01$ | $0.72\pm 0.01$ | 1.0 | $0.29\pm 0.01$ |
| normal | orthogonal | 1296 | $0.37\pm 0.01$ | $0.73\pm 0.01$ | 1.0 | $0.28\pm 0.01$ |
| orthogonal | normal | 1296 | $0.37\pm 0.01$ | $0.73\pm 0.01$ | 1.0 | $0.26\pm 0.01$ |
| orthogonal | orthogonal | 1296 | $0.39\pm 0.01$ | $0.71\pm 0.01$ | 1.0 | $0.24\pm 0.01$ |

**Table 6: Mean $\pm$ std error of loss and accuracy for TverskyXORNet with 2 normalization modes (normalized, and not normalized). Each row corresponds to the aggreate of 3x3x6x2x6x9 = 5832 training sessions with: 3 fbank and prototype initialization methods(orthogonal, normal, uniform). 6 intersection reduction methods (min, max, product, mean, gmean, softmin) and 2 difference reduction methods (ignorematch, substractmatch), 6 feature bank sizes (1, 2, 4, 8, 16, 32), and 9 random seeds. In each case, models were trained for 1000 epochs.**
| normalize | n | loss | acc | best acc | p(conv) |
| --- | --- | --- | --- | --- | --- |
| False | 5832 | $0.33\pm 0.00$ | $0.74\pm 0.00$ | 1.0 | $0.34\pm 0.01$ |
| True | 5832 | $0.38\pm 0.00$ | $0.72\pm 0.00$ | 1.0 | $0.27\pm 0.01$ |

**Table 7: Mean $\pm$ std error of loss and accuracy for TverskyXORNet with 6 feature bank sizes (1, 2, 4, 8, 16, 32). Each row corresponds to the aggreate of 3x3x6x2x2x9 = 1944 training sessions with: 3 fbank and prototype initialization methods(orthogonal, normal, uniform). 6 intersection reduction methods (min, max, product, mean, gmean, softmin) and 2 difference reduction methods (ignorematch, substractmatch), 2 normalization modes (normalized, and not normalized), and 9 random seeds. In each case, models were trained for 1000 epochs.**
| fbank size | n | loss | acc | best acc | p(conv) |
| --- | --- | --- | --- | --- | --- |
| 16.0 | 1944 | $0.25\pm 0.01$ | $0.79\pm 0.00$ | 1.0 | $0.42\pm 0.01$ |
| 8.0 | 1944 | $0.28\pm 0.01$ | $0.78\pm 0.00$ | 1.0 | $0.39\pm 0.01$ |
| 4.0 | 1944 | $0.30\pm 0.01$ | $0.76\pm 0.00$ | 1.0 | $0.38\pm 0.01$ |
| 32.0 | 1944 | $0.31\pm 0.01$ | $0.76\pm 0.00$ | 1.0 | $0.33\pm 0.01$ |
| 2.0 | 1944 | $0.45\pm 0.01$ | $0.67\pm 0.00$ | 1.0 | $0.20\pm 0.01$ |
| 1.0 | 1944 | $0.54\pm 0.01$ | $0.61\pm 0.00$ | 1.0 | $0.12\pm 0.01$ |

## Appendix D Language Modeling with GPT-2 and Tversky-GPT-2 on PTB

### D.1 Initialization: OpenAI Released Weights

**Table 8: Validation perplexity comparison with init=finetuned and tie-proto=N.**
|  | Validation Perplexity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | difference $\rightarrow$ | ignorematch | substractmatch |  |  |  |  |  |  |  |  |  |  |  |  |
|  | intersection $\rightarrow$ | gmean | max | mean | min | product | gmean | max | mean | min | product |  |  |  |  |
| Init | model | tie-proto | tie-fbank | features | params | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL |
| finetuned | baseline | N |  |  | 163037184 | 34.85 | 34.85 | 34.85 | 34.85 | 34.85 | 34.85 | 34.85 | 34.85 | 34.85 | 34.85 |
| finetuned | tversky-head | N | N | 1024 | 163823619 | 52.42 | 70.11 | 51.80 | 57.66 | 76.73 | 61.59 | 51.90 | 58.15 | 85.37 | 120.22 |
| finetuned | tversky-head | N | N | 2048 | 164610051 | 40.92 | 54.44 | 41.23 | 46.79 | 53.00 | 48.71 | 41.28 | 47.91 | 66.95 | 86.26 |
| finetuned | tversky-head | N | N | 4096 | 166182915 | 34.47 | 38.52 | 34.44 | 40.32 | 40.32 | 42.82 | 34.54 | 41.89 | 64.80 | 57.10 |
| finetuned | tversky-head | N | N | 8192 | 169328643 | 32.56 | 32.80 | 32.53 | 38.80 | 34.50 | 38.89 | 32.49 | 39.09 | 61.66 | 48.72 |
| finetuned | tversky-head | N | N | 12288 | 172474371 | 32.84 | 32.92 | 32.82 | 40.82 | 32.32 | 41.18 | 32.81 | 41.29 | 70.24 | 52.26 |
| finetuned | tversky-head | N | N | 16384 | 175620099 | 33.62 | 36.33 | 33.62 | 39.75 | ^∗32.31 | 40.71 | 33.63 | 40.67 | 77.18 | 52.32 |
| finetuned | tversky-head | N | N | 32768 | 188203011 | 38.36 | 40.34 | 38.98 | 49.35 | 33.94 | 49.48 | 38.79 | 47.17 | 109.10 | 62.37 |
| finetuned | tversky-all-1layer | N | N | 1024 | 114232359 | 145.50 | 208.61 | 142.82 | 154.14 | 219.63 | 172.13 | 144.59 | 155.65 | 638.28 | 297.61 |
| finetuned | tversky-all-1layer | N | N | 2048 | 115018791 | 118.23 | 172.70 | 115.15 | 120.06 | 166.49 | 121.00 | 114.88 | 119.46 | 257.80 | 253.91 |
| finetuned | tversky-all-1layer | N | N | 4096 | 116591655 | 95.65 | 143.67 | 95.51 | 121.13 | 124.50 | 122.75 | 95.79 | 121.51 | 157.91 | 203.54 |
| finetuned | tversky-all-1layer | N | N | 8192 | 119737383 | 91.40 | 94.86 | 91.44 | 93.09 | 99.75 | 93.14 | 91.45 | 93.53 | 149.20 | 142.79 |
| finetuned | tversky-all-1layer | N | N | 12288 | 122883111 | 90.39 | 94.49 | 90.79 | 92.54 | 94.00 | 92.98 | 90.74 | 92.40 | 147.16 | 133.41 |
| finetuned | tversky-all-1layer | N | N | 16384 | 126028839 | 90.51 | 96.41 | 91.22 | 95.39 | 91.08 | 95.64 | 91.23 | 95.06 | 156.30 | - |
| finetuned | tversky-all-1layer | N | N | 32768 | 138611751 | 96.00 | 101.08 | 96.44 | 100.78 | 92.68 | - | - | - | - | - |

**Table 9: Validation perplexity comparison with init=finetuned and tie-proto=Y.**
|  | Validation Perplexity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | difference $\rightarrow$ | ignorematch | substractmatch |  |  |  |  |  |  |  |  |  |  |  |  |
|  | intersection $\rightarrow$ | gmean | max | mean | min | product | gmean | max | mean | min | product |  |  |  |  |
| Init | model | tie-proto | tie-fbank | features | params | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL |
| finetuned | baseline | Y |  |  | 124439808 | ^∗19.99 | ^∗19.99 | ^∗19.99 | ^∗19.99 | ^∗19.99 | ^∗19.99 | ^∗19.99 | ^∗19.99 | ^∗19.99 | ^∗19.99 |
| finetuned | tversky-head | Y | N | 1024 | 125226243 | 90.22 | 39.28 | 86.30 | 61.22 | 69.68 | 61.29 | 87.64 | 61.08 | 81.07 | 94.78 |
| finetuned | tversky-head | Y | N | 2048 | 126012675 | 59.95 | 28.69 | 56.18 | 67.96 | 43.19 | 71.83 | 57.38 | 68.19 | 46.20 | 58.08 |
| finetuned | tversky-head | Y | N | 4096 | 127585539 | 23.26 | 27.15 | 24.01 | 24.54 | 28.91 | 23.37 | 24.08 | 23.22 | 30.84 | 35.15 |
| finetuned | tversky-head | Y | N | 8192 | 130731267 | 21.17 | 22.31 | 21.25 | 21.44 | 22.37 | 21.43 | 21.23 | 21.50 | 22.03 | 23.24 |
| finetuned | tversky-head | Y | N | 12288 | 133876995 | 21.07 | 22.64 | 21.09 | 21.33 | 20.99 | 21.32 | 21.08 | 21.35 | 21.98 | 21.81 |
| finetuned | tversky-head | Y | N | 16384 | 137022723 | 20.86 | 21.86 | 20.85 | 21.12 | 20.77 | 21.14 | 20.85 | 21.12 | 22.88 | - |
| finetuned | tversky-head | Y | N | 32768 | 149605635 | 20.70 | 21.33 | 20.68 | 20.82 | 20.46 | - | - | - | - | - |
| finetuned | tversky-all-1layer | Y | N | 1024 | 75634983 | 134.68 | 188.07 | 132.79 | 174.71 | 212.09 | 166.07 | 132.21 | 174.75 | 741.72 | 321.28 |
| finetuned | tversky-all-1layer | Y | N | 2048 | 76421415 | 101.62 | 182.26 | 102.64 | 131.28 | 164.13 | 134.14 | 103.46 | 130.69 | 235.55 | 228.58 |
| finetuned | tversky-all-1layer | Y | N | 4096 | 77994279 | 77.13 | 87.87 | 76.87 | 96.56 | 109.65 | 97.91 | 76.94 | 100.72 | 198.39 | 190.20 |
| finetuned | tversky-all-1layer | Y | N | 8192 | 81140007 | 70.07 | 88.16 | 69.75 | 80.32 | 83.06 | 79.63 | 69.71 | 79.65 | 97.48 | 117.50 |
| finetuned | tversky-all-1layer | Y | N | 12288 | 84285735 | 67.61 | 79.07 | 66.50 | 76.51 | 73.95 | 76.66 | 66.51 | 75.85 | 101.56 | 97.24 |
| finetuned | tversky-all-1layer | Y | N | 16384 | 87431463 | 69.54 | 79.76 | 69.22 | 78.32 | 73.03 | 79.33 | 69.22 | 78.85 | 100.12 | - |
| finetuned | tversky-all-1layer | Y | N | 32768 | 100014375 | 73.33 | 99.82 | 73.12 | 84.67 | 67.19 | - | - | - | - | - |

### D.2 Initialization: Random Weights

**Table 10: Validation perplexity comparison with init=scratch and tie-proto=N.**
|  | Validation Perplexity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | difference $\rightarrow$ | ignorematch | substractmatch |  |  |  |  |  |  |  |  |  |  |  |  |
|  | intersection $\rightarrow$ | gmean | max | mean | min | product | gmean | max | mean | min | product |  |  |  |  |
| Init | model | tie-proto | tie-fbank | features | params | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL |
| scratch | baseline | N |  |  | 163037184 | 134.06 | 134.06 | 134.06 | 134.06 | 134.06 | 134.06 | 134.06 | 134.06 | 134.06 | 134.06 |
| scratch | tversky-head | N | N | 1024 | 163823619 | 146.74 | 218.66 | 146.10 | 157.30 | 184.36 | 155.74 | 145.65 | 158.21 | 496.50 | 168.50 |
| scratch | tversky-head | N | N | 2048 | 164610051 | 147.92 | 192.12 | 148.02 | 145.72 | 171.54 | 155.64 | 147.64 | 145.66 | 320.49 | 140.48 |
| scratch | tversky-head | N | N | 4096 | 166182915 | 168.12 | 161.04 | 167.65 | 176.67 | 162.97 | 157.44 | 167.05 | 155.93 | 147.83 | 133.47 |
| scratch | tversky-head | N | N | 8192 | 169328643 | 210.03 | 204.28 | 217.75 | 182.50 | 173.63 | 182.08 | 215.71 | 182.47 | 205.87 | 154.21 |
| scratch | tversky-head | N | N | 12288 | 172474371 | 227.69 | 178.52 | 241.49 | 212.82 | 186.98 | 212.26 | 236.39 | 212.04 | 203.83 | 163.90 |
| scratch | tversky-head | N | N | 16384 | 175620099 | 229.84 | 202.26 | 250.62 | 228.47 | 213.86 | 231.28 | 248.62 | 231.49 | 228.31 | 200.96 |
| scratch | tversky-head | N | N | 32768 | 188203011 | 182.45 | 162.13 | 190.04 | 190.92 | 252.01 | 198.72 | 187.12 | 197.99 | 223.08 | 218.13 |
| scratch | tversky-all-1layer | N | N | 1024 | 114232359 | nan | 173.25 | 126.67 | 138.92 | 169.06 | nan | 127.19 | 139.09 | 271.09 | 347.79 |
| scratch | tversky-all-1layer | N | N | 2048 | 115018791 | 121.06 | 135.85 | 120.29 | 123.94 | 130.72 | 126.68 | 120.58 | 124.17 | 163.99 | 249.04 |
| scratch | tversky-all-1layer | N | N | 4096 | 116591655 | 120.89 | 127.62 | 121.11 | 123.97 | ^∗117.59 | 124.16 | 121.26 | 123.86 | 136.27 | 225.91 |
| scratch | tversky-all-1layer | N | N | 8192 | 119737383 | 128.32 | 134.91 | 129.87 | 126.38 | 118.06 | 126.72 | 129.45 | 126.31 | 155.86 | 140.68 |
| scratch | tversky-all-1layer | N | N | 12288 | 122883111 | 130.54 | 131.29 | 132.46 | 127.68 | 125.03 | 127.69 | 132.42 | 127.72 | 159.30 | 137.27 |
| scratch | tversky-all-1layer | N | N | 16384 | 126028839 | 129.24 | 131.03 | 131.82 | 128.37 | 132.29 | 128.33 | 131.94 | 128.01 | 163.26 | - |
| scratch | tversky-all-1layer | N | N | 32768 | 138611751 | 129.11 | 132.33 | 130.26 | 129.51 | 136.46 | - | - | - | - | - |

**Table 11: Validation perplexity comparison with init=scratch and tie-proto=Y.**
|  | Validation Perplexity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | difference $\rightarrow$ | ignorematch | substractmatch |  |  |  |  |  |  |  |  |  |  |  |  |
|  | intersection $\rightarrow$ | gmean | max | mean | min | product | gmean | max | mean | min | product |  |  |  |  |
| Init | model | tie-proto | tie-fbank | features | params | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL | PPL |
| scratch | baseline | Y |  |  | 124439808 | 136.04 | 136.04 | 136.04 | 136.04 | 136.04 | 136.04 | 136.04 | 136.04 | 136.04 | 136.04 |
| scratch | tversky-head | Y | N | 1024 | 125226243 | 149.22 | 213.67 | 148.81 | 155.98 | 193.72 | 153.13 | 149.07 | 156.29 | 360.26 | 271.20 |
| scratch | tversky-head | Y | N | 2048 | 126012675 | 147.91 | 179.89 | 148.12 | 145.80 | 169.92 | 155.16 | 147.83 | 145.83 | 175.95 | 154.26 |
| scratch | tversky-head | Y | N | 4096 | 127585539 | 167.44 | 168.53 | 168.55 | 150.11 | 161.85 | 151.59 | 168.02 | 149.89 | 152.11 | 145.52 |
| scratch | tversky-head | Y | N | 8192 | 130731267 | 197.06 | 207.89 | 204.79 | 181.21 | 168.29 | 179.76 | 201.95 | 180.99 | 183.96 | 156.37 |
| scratch | tversky-head | Y | N | 12288 | 133876995 | 212.50 | 211.17 | 223.91 | 202.83 | 179.87 | 204.57 | 221.02 | 202.63 | 214.04 | 165.64 |
| scratch | tversky-head | Y | N | 16384 | 137022723 | 210.90 | 152.18 | 227.04 | 217.16 | 204.58 | 214.09 | 226.02 | 216.78 | 238.05 | - |
| scratch | tversky-head | Y | N | 32768 | 149605635 | 162.49 | 151.23 | 171.88 | 176.74 | 216.54 | - | - | - | - | - |
| scratch | tversky-all-1layer | Y | N | 1024 | 75634983 | 142.00 | 174.76 | 138.77 | 152.44 | 182.22 | 154.00 | 138.65 | 152.31 | 250.44 | 389.16 |
| scratch | tversky-all-1layer | Y | N | 2048 | 76421415 | 132.41 | 141.77 | 130.55 | 136.88 | 146.27 | 144.29 | 130.69 | 137.51 | 180.40 | 332.45 |
| scratch | tversky-all-1layer | Y | N | 4096 | 77994279 | 129.46 | 137.46 | 129.64 | 130.57 | 128.84 | 130.26 | 129.66 | 130.61 | 146.93 | 175.75 |
| scratch | tversky-all-1layer | Y | N | 8192 | 81140007 | 136.73 | 145.34 | 139.16 | ^∗125.86 | 128.05 | 125.98 | 139.13 | 125.96 | 151.85 | 151.07 |
| scratch | tversky-all-1layer | Y | N | 12288 | 84285735 | 140.86 | 156.21 | 144.50 | 128.38 | 134.98 | 127.85 | 144.61 | 128.45 | 156.44 | 140.26 |
| scratch | tversky-all-1layer | Y | N | 16384 | 87431463 | 143.59 | 160.09 | 148.72 | 130.69 | 140.38 | 129.88 | 148.77 | 130.50 | 153.77 | - |
| scratch | tversky-all-1layer | Y | N | 32768 | 100014375 | 154.05 | - | - | 140.73 | 150.33 | - | - | - | - | - |

## Appendix E Examples of Semantic Fields in TverskyGPT2’s Set-Centric Token Representation

Figures
[8](https://arxiv.org/html/2506.11035v2#A5.F8)
,[9](https://arxiv.org/html/2506.11035v2#A5.F9)
,[10](https://arxiv.org/html/2506.11035v2#A5.F10)
,[11](https://arxiv.org/html/2506.11035v2#A5.F11)
and [12](https://arxiv.org/html/2506.11035v2#A5.F12) show examples of visualizations of semantic fields formed using set expressions on TverskyGPT2 tokens represented as sets.
For each token, the top 1000 features (sorted by dot-product with the token) are considered, and represented as the smallest colored circles.
Tokens are represented as yellow circles, and connected to the features they comprise with grey lines.
All distinct intersections of features are colored in the same color, and connected to a circle of the same color representing a distinctive semantic field. Semantic fields of interest are annotated with a text box listing the top 50 tokens by semantic score (See Section [3.4](https://arxiv.org/html/2506.11035v2#S3.SS4)) in that semantic field, permitting its visualization.
A force-directed graph layout is employed to position the circles corresponding to tokens, features, and semantic fields.
Note that the Ġ character (U+0120) represents space in GPT2 tokenization, distinguishing tokens that appear at the beginning of words from other tokens.

Figure: Figure 8: Visualization of the semantic fields formed by the distinctive features of related tokens. Notice that TverskyGPT2 tokens capture semantics related to orthographic markings, and morphological function. The $king$ token (as in baking) is related to other suffixes such as $ked$, $ker$ and $kes$ (as in baked, baker and bakes)
Refer to caption: x9.png

Figure: Figure 9: Visualization of the semantic fields formed by the distinctive features of tokens $like$ ($like-love$) and $love$ ($love-like$). $like$, distinctively from $love$, which also captures the sense of sentiment, captures the sense of similarity, with tokens such as *akin*, *alike*, *unlike* and *reminiscent*.
Refer to caption: x10.png

Figure: Figure 10: Visualization of the semantic fields formed by the distinctive features of tokens $go$ ($go-gone-went-going$), $gone$ ($gone-go-went-going$), $went$ ($went-gone-go-going$), and $going$ ($going-gone-went-go$), respectively specifying the concept of verb forms *infinitive*, *past participle*, *past tense* and *present participle*.
Refer to caption: x11.png

Figure: Figure 11: Visualization of the semantic fields formed by the distinctive features of the common features of tokens $bad$ and $good$ ($bad\cap good-worse-better-worst-best$), $worse$ and $better$ ($worse\cap better-bad-good-worst-best$), and $worst$ and $best$ ($worst\cap best-bad-good-worse-better$). These semantic fields algebraically specify the concepts of *adjective*, *comparative*, and *superlative*.
Refer to caption: x12.png

Figure: Figure 12: Visualization of the semantic fields capturing the distinctive senses of close synomyms $coach$, $teacher$, $mentor$, and $guide$. Notice that the distinctive features of $coach$ ($coach-teacher-mentor-guide$) shows terms related to sport such as *trainer*, *teammate*, *Football* and *ESPN* whereas the distinctive features of $teacher$ ($teacher-coach-mentor-guide$) show terms related to formal education such as *educator*, *classroom*, *instructor*, *lecturer* and *professor*.
Refer to caption: x13.png

Figure: Figure 13: Visualization of the semantic fields capturing the distinctive feature of tokens $man$, $woman$, $king$ and $queen$, and the distinctive semantics of the common features of $man$ and $king$, $man$ and $woman$, $woman$ and $queen$, $queen$ and $king$, and $man$ and $woman$
Refer to caption: x14.png

## Appendix F [Visual]MNISTNet, and [Visual]TverskyMNISTNet,

Two neural networks, VisualMNISTNet, and VisualTverskyMNISTNet (Figure [14](https://arxiv.org/html/2506.11035v2#A6.F14)) are trained to perform MNIST handwritten digit classification. The two neural network architectures share the same convolutional feature extraction stack yielding a 36-dimensional vector representation of the input image. They differ in how those vectors are projected onto the 10-dimensional output vectors corresponding to 10 digit classes. VisualMNISTNet employs a stack of 120, 84, and 10 unit linear projection layers following LeCun et al. [1998]. However, the ReLU activation function is used instead of the logistic sigmoid. VisualTverskyMNISTNet employs a single Tversky Projection layer with 10 prototypes and 20 features. Both neural networks employ the method described in Section [2.5](https://arxiv.org/html/2506.11035v2#S2.SS5), facillitating our qualitative analysis of the learned prototypes and features.

Two additional neural networks, MNISTNet and TverskyMNISTNet, which are identical to their *Visual* counterparts, but do not employ our parameter visualization method are also trained to serve as baseline for accuracy. These neural networks have fewer parameters as the projection parameters are specified in their vector form, which is more compact.

MNISTNet 3 times smaller than LeNet-5 due to the design of its convolution stack, which outputs a 36-dimensional vector which are the concatenation of three 12-dimensional vectors obtained by averaging 3 convolutional feature maps across their spatial dimensions.

TverskyMNISTNet, with only 7K parameters is also 3 times smaller than MNISTNet because it employs a single Tversky Projection layer instead of a stack linear projection layers and non-linear activation functions.
All 4 neural networks are trained for 1000 epochs with a batch size of 500. Dropout is applied to the output of the convolution stack at the rate of 0.05. Adam optimizer is used with learning rate 0.002, and weight decay rate 0.00001.

Figure: Figure 14: The neural network architecture used in our qualitative analysis experiments.
Refer to caption: x15.png

**Table 12: Parameter count and accuracy of baseline and tversky convolutional neural networks trained for MNIST handwritten digit classification. The "visual" variants specify tversky prototypes and features, and the third fully connected layer’s parameters in the input space as 28x28 matrices. All models were trained for 1000 epochs.**
| Model | Params | Valid ACC |
| --- | --- | --- |
| MNISTNet | 21 394 | 99.1% |
| VisualMNISTNet | 28 394 | 99.1% |
| TverskyMNISTNet | 7 023 | 98.7% |
| VisualTverskyMNISTNet | 29 463 | 98.7% |

Figure: Figure 15: Learned visual prototypes in the baseline and tversky variants of the neural network described in Figure [14](https://arxiv.org/html/2506.11035v2#A6.F14) after epochs 0, 200, 400 and 600.
Refer to caption: x16.png

Figure: Figure 16: Two MNIST digits: a two (A), and a one (B). A grid of top-9 instances ranked by semantic score in the $A-B$, $A\cap B$ and $B-A$ semantic fields are shown (see Section [3.4](https://arxiv.org/html/2506.11035v2#S3.SS4)). The distinctive features of A exhibit curviness. The distinctive features of B exhibit a vertical stroke slanted to the right.
Refer to caption: x17.png

## Appendix G Example of learned α \alpha , β \beta and θ \theta

Figure: Figure 17: Example of learned $\alpha$, $\beta$ and $\theta$ values over the training iterations of a Tversky-Resnet-50 on the NABirds dataset. As per Tversky’s hypothesis of prototypicality, $\alpha$ > $\beta$ in the trained model. The distinctive features of instances are weighted more than the distinctive features of prototypes.
Refer to caption: exp_001.02/learned-theta-alpha-beta.jpg

## Appendix H Computation Times and Resources

### H.1 Resnet-50 / NABirds Experiments

**Table 13: Device use and wall-clock times for the NABirds experiments**
| Model | Time | Devices |
| --- | --- | --- |
| Resnet-50 | 2h 43m 4s | 4 x NVIDIA RTX A6000 |
| Resnet-50 | 3h 34m 37s | 4 x NVIDIA RTX A6000 |
| Resnet-50 | 2h 45m 47s | 4 x NVIDIA RTX A6000 |
| Resnet-50 | 3h 25m 7s | 4 x NVIDIA RTX A6000 |
| Tversky-Resnet-50 | 2h 43m 17s | 4 x NVIDIA RTX 6000 Ada Generation |
| Tversky-Resnet-50 | 2h 58m 29s | 4 x NVIDIA RTX 6000 Ada Generation |
| Tversky-Resnet-50 | 2h 41m 46s | 4 x NVIDIA RTX 6000 Ada Generation |
| Tversky-Resnet-50 | 2h 57m 21s | 4 x NVIDIA RTX 6000 Ada Generation |

### H.2 GPT2 Experiments

Figure: Figure 18: Sample of wall-clock times in seconds for TverskyGPT2 experiments on PTB as a function of the feature bank size. 2 GPUs were used for each run. GPU models varied. They included NVIDIA A100-SXM4-40GB, NVIDIA H100 80GB HBM3, and NVIDIA RTX 6000 Ada Generation.
Refer to caption: exp_006/gpt2_experiment_wallclock_times.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Please see questions already stated in weaknesses :)
- Methods:
    - As the author’s mention in the discussion, the size of  $\Omega$  is hyperparameter and needs to be tuned as everything else. However, do the authors have an intuition if there could be a relationship between the number of prototypes and the universe size?
    - Question about the set representations: A feature $f_k$ of the learnable finite universe is in the set of object $x$ if their dot-product is larger than zero. What does it then mean to include into this set representation a feature $f_j$  with $x\cdot f_j = 1e^-10$ and a feature $f_q$ with $x\cdot f_q = 1e^-10$?
- Experiments
    - Tversky layer can model XOR: In Tables 4-7 we can an observe rather small convergence ratios, could the authors comment on that?
    - Language-modeling: In Table 1, one can observe that only replacing the prediction head with a tversky head results in more features and parameters compared to replacing all projection layers with Tversky projections. This makes sense as the projection head itself must learn to represent all the semantic structure needed for predicting the next token. When all projection layers are replaced, the encoder and attention blocks co-adapt with the new similarity function, so the features can specialize efficiently, and fewer prototypes are sufficient. However, could you elaborate on the feature number and parameter number if we compare tversky-all with and without prototyp tying? Why do we observe such a difference?
    - Image classification:
        - Would it be possible to show some image classification results with some a pretrained Vision Transformer. E.g., a vision-encoder of a CLIP model, or Dinov3 model (available on huggingface)?
        - Why are the pretrained and frozen setting so bad in performance?
soundness: 3
strengths: - Original and well-motivated idea connecting human similarity judgments to machine-learned representations. Clear novelty of First differentiable implementation of Tversky similarity for neural networks.
- The paper is well-written, clearly structured, and engaging to read. I learned a lot. 
- The XOR demonstration (Section 3.1, Figure 1) with its thorough validation convincingly shows that a single Tversky projection can model non-linear decision boundaries without composition with activation functions. This is a genuine advantage over linear layers.
- Demonstrates potential in both vision and language modeling contexts.
- Qualitative analysis really interesting.
- Acknowledges computational cost of data-domain visualization (Figure 5) and it is honest about the limitations.
summary: This paper challenges the geometric model of similarity (dot-product/ cosine similarity) largely used in deep learning, proposing instead Tversky's psychologically-grounded similarity measure that permits asymmetry and other violations of metric axioms observed in human similarity judgment. The key contribution is a differentiable version of Tversky's similarity and corresponding neural layers that can replace dense layers in standard architectures. The approach is based on the introduced dual representation of an object: objects are represented both as vectors and as sets of features from a learnable finite universe, where an object's set comprises those feature vectors with which it has positive dot product. This set-theoretic representation enables measuring similarity through feature set intersections and differences, forming the basis for Tversky Similarity and Projection layers. The authors demonstrate that: (1) a single Tversky layer can model XOR, impossible for linear layers; (2) Tversky layers can improve performance in GPT-2 language modeling (PTB dataset) and ResNet-50 image classification (MNIST, NABirds). Qualitative analyses reveal interpretable properties: salience scores (sum of dot-products of an object's vector with all features in its set) align with "goodness of form," learned (input-level) prototypes are human-interpretable, and semantic fields (e.g., adjectives, verb forms) emerge from set algebra. Additionally, they introduce a data-domain visualization method for projection parameters, though this approach scales poorly. 

The work is conceptually novel, well-written, and really promising but could benefit from modern architecture evaluation, clearer differentiability explanation, and stronger scalability analysis.
weaknesses: - Gradient flow through indicator functions: Equations 2-5 rely on indicator functions [a·fₖ > 0] which have zero derivative almost everywhere. It's unclear how gradients propagate during training—are you using straight-through estimators, sigmoid approximations, or another approach? Providing implementation details or code would help to understand you work.
- Convergence of training: Tables 4-7 show convergence failure rates of 47-77% across many hyperparameter settings, with some producing NaN results. Understanding what causes these failures would strengthen the work. How did convergence look for the other main experiments?
- Evaluation on larger, more recent benchmarks would strengthen claims. Current experiments use MNIST (toy dataset), PTB (1M tokens from 1993), and NABirds (48K images with domain overlap with ImageNet birds), which are ideal for a proof of concept. However, testing on standard modern benchmarks like ImageNet, CIFAR-100, or WikiText-103 would better demonstrate competitiveness against well-tuned baselines.
- Similarly, validation on current architectures would show broader applicability. While ResNet-50 and GPT-2 small are solid choices for controlled experiments, evaluating on Vision Transformers (ViT, CLIP, DINOv2) or larger language models would support the "state-of-the-art" framing (L82).
- Multiple seeds are reported for XOR experiments, but it's unclear if main experiments used multiple runs. Would it be possible to comment on this issue or if available report the means and standard deviations?
- Tied vs untied feature banks remain unexplored. While prototype tying is ablated in Table 1, feature bank sharing is always enabled. Testing both configurations would be informative.
- Data-domain visualization might be impractical at scale. For NABirds with 555 classes, input-space parameterization would require 83.5M parameters (555×224×224×3) compared to 1.1M for standard approaches—a 75× increase. Could the authors comment on the scalability constraints and potential overfitting risks?
- Reporting accuracies: Using relative improvements (e.g., 24.7%) rather than absolute accuracy points can sometimes overemphasize small differences. Both would be informative.
- Methods like ProtoPNet and ProtoTree (prototype learning domain) seem related to the data-domain visualization contribution but aren't discussed in the introduction or chapter 2.5. Could the authors comment on that?

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: It would be good to know if the Tversky head is effective as a classifier when attached to a backbone not trained for classification (e.g., a self-supervised backbone) and if in that case it would also outperform a baseline linear readout.
soundness: 4
strengths: Strengths:
* Strong originality of entire prototype learning framework based on learned asymmetric comparison, which can be integrated in end-to-end training of various deep architectures. 
* The asymmetry weights, prototype dimensionality, and the feature bank are treated as learnable (free) parameters.
* Demonstrates improved performance on several benchmarks, including NABirds, indicating applicability to problems with hundreds of classes. While there is no demonstration of extension to thousands of classes, in this case I do not see as a limitation, because one of the central future applications of this training approach may be application to classification in relatively small domains with highly similar categories.
* A strength of the approach, whichcould be better discussed, is the potential to constrain the directionality of the asymmetry weights (e.g., setting β>α when omissions matter more, α>β when extras hurt more) depending on domain. For domains with low intra-class variance where 'core' features count (quasi necessary), penalizing omissions  of object vs. prototype might be more sensible; the converse for broader, more fuzzy categories.
summary: The paper introduces a new perspective on classification via prototype learning, where classifier decisions are made by taking the argmax over similarities to class prototypes.  Similarity is defined as an asymmetric match function between the object and a prototype over a learned feature bank whose dimensionality can be set.  While asymmetric similarity has been used in some prior applications (e.g., nearest neighbor classification based on engineered features), to my knowledge it has not been used in end-2-end CNN training, and the application is novel.
weaknesses: (Minor) Weaknesses:

* It would be nice to have a small ablation experiment isolating the specific contribution of asymmetry by setting the asymmetry weights to zero and learning only the overlap (intersection) term in Equation 1.
* Can be extended to allow learning only the asymmetry terms with the overlap term to zero. Taken together we can understand if performance gains rely mainly on similarities or differences.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - Assuming contextualized models, would they be able to learn Tversky-type similarity?
- The notion of feature bank Ω could be communicated more clearly in Section 2.3. From Section 2.2 these are feature vectors $f_k$ and Ω parametrizes the function $f$, so $f$ essentially an embedding layer (as also mentioned more clearly later in Sec. 2.4.1).
- “For our vision experiments, no tuning was performed” (l. 459): No tuning *of hyperparameters* if I read Section 3.3 correctly, is that correct?

**Editing*
- “$α, β, andθ$” (l. 181)
- “fullly” (l. 217)
- “weights are initialized from ImageNet” (l. 327) - ImageNet -> ResNet?
soundness: 3
strengths: - Clear and well-founded scientific motivation for the proposed novel methods. The idea of using Tversky similarity for deep learning is to my knowledge a novel contribution.
- Empirical confirmation of the proposed neuralized Tversky approach, in both synthetic experiments and large-scale studies
- The paper includes an honest and balanced discussion section and transparently raises limitations throughout the paper.
- The experiments appear reproducible and are for the most part clearly described (see below).
summary: The paper proposes an alternative to the commonly geometric formulation of similarity through using Tversky similarity. The authors propose a differentiable formulation of representations for this purpose, along with an implementation thereof (Tversky Projection neural layer). In addition, an approach for interpreting Tversky networks through visualization of projection layers in the data domain are proposed. Experiments on the XOR, Penn Treebank text data, and MNIST are performed to demonstrate the method.
weaknesses: - While I like the motivation of Section 3.1 (XOR), the section and associated Figure 1 read rushed and can be confusing in its current state. I think the main take-home is that the Tversky projection layer can in principle learn the XOR function. I would suggest focusing on introducing the problem clearly (which is currently done in the caption), and then discuss some of the limitations encountered during optimization, i.e., convergence and hyperparameter selection.
- While I think the experiments are overall good, they remained superficial at times. Some quite interesting points were raised in the discussion, e.g. the idea of editing prototypes for increased robustness, which would have been great to demonstrate also empirically. 
- Some claims regarding interpretability are not clearly supported by evidence, e.g. “handwriting, such as lines and curves, more clearly than those learned by linear projection layers” (l. 413) and “Tversky Projection layers’s parameters are far more interpretable than the ones of the contemporary fully connected layer“ (l. 217). A more fair comparison could be to compare attributions in input space, e.g. [Mon18] as the Tversky Projection layers explicitly tie the projection to being decodable from the input. 
- The description of the proposed visualization technique in Section 2.5 should be more clear, e.g. through the use of formulas or a conceptual visualization, e.g. moving Fig. 5 to the main alongside a more clear description of how the method works. For example, the sentence “limitation that parameters specified in data-space are typically larger in size than their original counterparts, which increases the effective number of trainable parameters.“ (l. 211-213) is difficult to parse, i.e. how are the projection parameters specified, is the visualization technique trained or is this a post-hoc interpretability approach?


**Comment**
- I believe the proposed framework could be quite useful in the context of textual similarity, where it was shown that standard encoders and cosine similarity do not match human similarity judgments very well [Rei19]. Follow-up work on interpreting these dot-product embeddings could reveal quite simplistic feature matching strategies, even after aligning predictions to human similarity judgements [Vas24]. Here a Tversky layers may provide a more effective similarity readout.


**References**
- [Rei19] Reimers, N., & Gurevych, I. (2019, November). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3982-3992).
- [Vas24] Vasileiou, A., & Eberle, O. (2024, June). Explaining Text Similarity in Transformer Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 7852-7866).
- [Mon18] Montavon, G., Samek, W., & Müller, K. R. (2018). Methods for interpreting and understanding deep neural networks. Digital signal processing, 73, 1-15.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: * “Some initializations of prototypes and features lead to convergence failure” - are normal and orthogonal initialization the ones that the authors refer to?
* Why didn’t the authors present the example graphical results for NABirds (like the ones in Fig. 2 for MNIST)?
* Also, see the weaknesses.
soundness: 3
strengths: * I really like the idea of using Tversky similarity to create layers for neural networks. Tversky’s theory is very logical and simple and I think that it has a huge potential for the neural network domain. 
* The paper is well-written and easy to follow. It is beneficial that the authors presented a small XOR example at the beginning. 
* The salience experiment was interesting.
summary: The authors propose how to use Tversky similarity (from cognitive psychology) instead of the geometric model of similarity, which is usually used in deep learning. They perform experiments on text and image domains.
weaknesses: * The paper lacks a clear related work section. While some works are described in the introduction, it would be good to include a dedicated section. Also, there are works that use Tversky’s similarity model or methods inspired by it for different purposes in the deep learning domain, and it would be beneficial to mention these works and acknowledge that using Tversky’s model of similarity is not completely new in deep learning - even if for different purposes.
* “This lack of interpretability, even in this simple domain without background textures, represents a significant limitation of prior approaches.” - the authors say that their approach improves interpretability. While it is visible that for MNIST (simple objects, with large differences) the prototypes look reasonable, I am not so sure that they would be that pronounced for the second dataset used in the study - regarding different breeds of birds. 
* For the vision experiments, it would be nice to include the results for a ViT as well - to represent two leading architecture families in vision.


Decision: Accept (Poster)