ProtoPairNet: Interpretable Regression through Prototypical Pair Reasoning
Authors: Rose Gurung
Ronilo Ragodos
Chiyu Ma
Tong Wang
Chaofan Chen
Venue: NeurIPS 2025 poster ()
Keywords: deep learning
interpretability
prototype-based neural network
case-based reasoning
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: We present Prototypical Pair Network (ProtoPairNet), a novel interpretable architecture that combines deep learning with case-based reasoning to predict continuous targets. While prototype-based models have primarily addressed image classification with discrete outputs, extending these methods to continuous targets, such as regression, poses significant challenges. Existing architectures which rely heavily on one-to-one comparison with prototypes lack the directional information necessary for continuous predictions. Our method redefines the role of prototypes in such tasks by incorporating prototypical pairs into the reasoning process. Predictions are derived based on the input's relative dissimilarities to these pairs, leveraging an intuitive geometric interpretation. Our method further reduces the complexity of the reasoning process by relying on the single most relevant pair of prototypes, rather than all prototypes in the model as was done in prior works. Our model is versatile enough to be used in both vision-based regression and continuous control in reinforcement learning. Our experiments demonstrate that ProtoPairNet achieves performance on par with its black-box counterparts across these tasks. Comprehensive analyses confirm the meaningfulness of prototypical pairs and the faithfulness of our model’s interpretations, and extensive user studies highlight our model's improved interpretability over existing methods.

ProtoPairNet: Interpretable Regression through
Prototypical Pair Reasoning
Rose Gurung University of Maine rose.gunme rose.gurung maine.edu
Ronilo Ragodos
Chiyu Ma Dartmouth College
University of New Hampshire Konilo.Ragodoo unh.edu
chiyu.ma. @artmouutheduu
Tong Wang
Chaofan Chen
Yale University
University of Maine chaofam maine.edu
tong. wwng..ang.tw687000 yale.ed
Abstract
We present Prototypical Pair Network (ProtoPairNet), novel interpretable architec- ture that combines deep learning with uuse-based reasoning to predict continuous targets. While prototype-bet based models have primarily addressed image classifica- tion with discrete outputs, extending these methods to continuous targets, such as regression, poses significant challenges. Existing architectures which rely heavily one-to-ons comparison with prototypes lack the directional information neces- sary for continuous predictions. Our method redefines the role of prototypes in such tasks by incorporating prototypical pairs into the reasoning process. Predictions are derived based on the input's relative dissimilarities to these pairs, leveraging intuitive geometric interpretation. Our method further reduces the complexity the reasoning process by relying on the single most relevant pair of prototypes, rather than all prototypes in the model is versatile enough to be used in both vision-based regression and continuous control in reinforcement learning. Our experiments demonstrate that ProtoPairNet achieves performance on par with its black-box counterparts across these tasks. Comprehensive analyses confirm the meaningfulness of prototypical pairs and the faithfulness of our model's l interpretations, and extensive user studies highlight our model's improved interpretability over existing methods.
done in prior works. Our model
Introduction
Deep neural networks have achieved remarkable success across various domains, from advancing
autonomous vehicles 31 to supporting critical decisions in healthcare 5 46 39 and criminal justice [6]. As these applications expand into increasingly sensitive and high-stakes areas, the demand for models with transparent and interpretable decision-make makiing processes has become crucial. Prototype-
28 23), which inherently interpretable by design, address this challenge through case-based reasoning [1 a natural form of human reasoning. These models generally perform one-to-one comparisons between input image and prototypical cases learned during training. They then aggregate evidence of similarity to these prototypes, using a linear layer to arrive at the final classification decision.
based models [1] 44
33
26
12
43
Despite recent advances in prototype-based models for image classifications with discrete tar- gets, extending these models to tasks with continuous targets, such as regression 224 16 21] face two primary issues: First, one-to-one comparisons between an input image and individ- ual prototypes only tell that the input image is similar to some prototypes and thus should have label similar to the labels of those prototypes. This is insufficient for regression, where
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

predictions in continuous space would require additional context, such as whether the out- put should be greater or less than each prototype's s label. This shown the top of Fig- ure D where the input image has comparable similarities to both prototypes, making it un- clear whether its label should be greater or smaller than those of the individual prototypes. Second, using linear layer at the end to com- bine prototype labels via a similarity- average can cause unwarranted deviations. For example, even when an input matches a proto- type exactly, the predicted value may deviate from the prototype's s label due to interference from irrelevant prototypes, even if their contri- butions to the output is small, An illustration for the second issue be found in Appendix B
Reasoning with one to one comparton:
Clecbrriyy
palitys bdits
Sayed large
Reasoning Peointy pical pair:
Inspired by these challenges, we propose the Prototypical Pair Network (ProtoPairNet), novel architecture that redefines the role of pro- totypes in regression tasks through case-based reasoning. Unlike traditional one-to-one image- to-prototype comparisons, ProtoPairNet intro- duces prototypical pairs- pairs of learned pro- totypes that jointly govern the direction and mag- nitude of the predicted label. Each prototypical pair defines an axis along which the regression output varies linearly. By comparing an input image with the most relevant prototype pairs identified by the model, ProtoPairNet captures both proximity
liscarty
slong thi
Prodedd SUM
ispul isnnge sppcers obder thes
Figure 1: Reasoning process of how prototype- based models predict the age of an input image: traditional one-to-one comparison (top) vs. pro- totypical pairs reasoning (bottom). One-to-one comparisons lack directional cues whether the predicted label should exceed or fall below the prototype's 's label, leading to ambiguity in explana- tions.
the axis defined by the pro- totype pair and relative positioning within the range defined by the pair. As illustrated in the bottom of Figure [ for age prediction, the input image is projected onto line defined by a pair of prototypes in the latent space. Our model is designed to ensure that age varies linearly along this line and therefore it can use the input's relative position between the prototypes to determine the predicted age. This intuitive geometric interpretation of prototypes further eliminates the need for separate linear layer, allowing predictions to rely solely on relevant prototypical pairs. This refined representation enables more insightful and effective reasoning process in continuous spaces, making ProtoPairNet particularly suited for regression tasks.
To demonstrate the versatility of ProtoPairNet. we evaluate it on two distinct domains: age prediction supervised learning task) and car racing (a behavioral cloning task with reinforcement learning expert). Experimental results show that ProtoPairNet achieves performance competitive with black- box baselines in both settings. Meanwhile, multiple user studies further highlight that ProtoPairNet improves interpretability over conventional prototype-basee comparisons. Additionally, the illustrations of our reasoning processes and global analyses empirically demonstrate the consistency and faithfulness of the prototype representations.
models relying on one-to-one prototype
Related Work
Posthoc explanation methods for computer vision, such as activation maximization 13 27 image
perturbation 19 29), attention heatmaps 42 (3), and saliency visualizations [38 4 47 40 41).
often fail to faithfully explain the reasoning processes of deep neural networks, as their explanations may not accurately reflect the model's 's decision-making
[2].
Prototype-basee approaches offer a transparent prediction process through case-based reasoning. These models compare small set of learned latent feature representations called prototypes (the "cases") with the latent representations of test image to perform classification. For example, the Prototypical Part Network (ProtoPNet) [II employs one-to-one comparisons with class-specific prototypes using L2 distance. Each prototype is trained to closely resemble feature patches from its own class while being dissimilar to patches from other classes. The Transparent Embedding Space Network (TesNet) [44) modifies the original ProtoPNet by utilizing a cosine similarity metric to

---

compute similarities between image patches and prototypes in a latent space. Deformable ProtoPNet decomposes the prototypes into smaller patches capture pose variations. Other works
12
reduce the reliance on class-specific prototypes, thereby minimizing the number of prototypes required. Recent advancements have focused on improving quality of prototypes 7
33
34
45
42
91. enhancing interpretability through modified architectures [22, learning prototypes
as distributions [35] [0], and extending prototype-based models to vision transformers [23], text
classification 25 [8] and reinforcement learning with discrete action spaces [30].
However, existing work on prototype-base based models has predominantly focused on predicting dis- crete outcomes, and very few address the prediction of continuous outcomes such as regression. models addressing regression tasks, Hyperspherical Prototype Networks 24 sacrifice interpretability by defining prototypes as abstract points disconnected from real data, thereby reducing the transparency of predictions. ExPeRT 17 and INSightR-Net [16] mitigates this by anchoring prototypes to real data and calculating predictions as similarity-weigle weighted averages of all prototype labels, akin to ProtoPNet [1] and its variants. However, this approach in regression tasks remains vulnerable to interference from unrelated prototypes, potentially leading to inaccurate predictions. Soototyee-rapper Networks (PWNet) [2I]. which can used for continuous control in reinforcement learning, face limitations in adaptability and scalability due to their reliance manual prototype selection and fixed layer weights. Furthermore, they inherit the same vulnerability from similarity-we limiiarity-eiightedd averaging, where an input identical to prototype can yield different prediction due to the influence of other prototypes (see Appendix B). These works primarily rely on one-to-one comparisons, which only provides limited information for regression tasks and introduces ambigu- ity into the reasoning process. In contrast, our model makes regression predictions by comparing input images with the most relevant prototypical pairs it has learned. Our ProtoPairNet enhances interpretability by reducing dependence on simillrity-weighhedd averages of all prototype labels.
Among prototype-bae
Method
2.1 Architecture Overview
Praneppial poirs correpadding labek:
Op-Pehh PPa Pu 1o, F. enl (2y). Siy C 2
srigg squatie
Pawuth
Prodktinn
Falr
Prediceed 24/8
Feature Encodur
You
Hother
Paliig
relecaace branch
KIDO RERUIR (D
Figure 2: Model architecture for ProtoPairNet. The pink bubbles show the comparison of each prototypical pair with the input in the latent space. This information is used by both the prediction branch (green) and the pair relevance branch (red) to produce the final prediction.
Figure 2 provides an overview of the ProtoPairNet architecture. Our model consists of fea- ture encoder f that encodes an input image to latent vector branches: prediction branch and a pair relevance branch. Both branches utilize prototypical pairs (P1, P2).... (P24-1, )2*), .... (Pm-1, Pm), where m is the number of prototypes, and is an even integer, resulting in m/2 prototypical pairs. For each = I..., m/2, the k-th prototypical pair (p2*-1, )22) defines axis, which we call the k-th ProtoPair axis, along which the predicted
f(x), followed by two
label increases linearly in the direction of P2* - P2k- 1- Each prototypical pair (P2k-1. P2k) also associated with the prototypical labels (1pzk-1 Upze ) IR? which denote the ground-trute labels of
3

---

the corresponding prototypes. Both the prototypical pairs and their associated labels are learned from the training data. Since the focus of this work is introduce a new paradigm of reasoning using prototypical pairs for continuous target predictions, we have opted to use prototypes that have the same dimensions as the latent vector z, which represents the entire input rather than localized part. The extension to prototypical parts will be investigated in future work.
Given an input image x, our ProtoPairNe compares its latent representation z = f(x) with m/2 prototypical pairs. For each of the k-th prototypical pairs (P2x-1, ( the latent vector is first projected onto the k-thh ProtoPair axis. Let z' (2k- 1,2k) denote the projected point (see Figure ( Since our model is designed to ensure that the target labels vary linearly along the ProtoPair axis, the predicted label of depends on the position of (2k-1,2k) along the k-th ProtoPair axis. We then use the relative distances of z' (2k 11.2k) with P2k-1 and P2k to predict label 9(2k-1,2k)) The prediction branch of our model makes m/2 predictions ((1.2) -... m,)) one for each prototypical pair. Section 2.2 provides more details on the prediction branch.
The pair relevance branch of our model identifies the most relevant prototypical pair for prediction, by first calculating the squared L2 distance between the latent vector z and each k-th ProtoPair axis. This is given by d(2x-1,24)
Iz z' 222221,22) constructed by applying Gumbel-softmax to (1,) ..... -d(m-1.m) to generate the pair relevance model assigns
A one-hot relevance score vector is
score 2(1,2) .... mmm- Since Gumbel-softmax approximates the argmax, relevance of to the prototypical pair that is closest to the latent vector z. The final prediction ÿ corresponds to the output of the prediction branch associated with the closest prototypical pair. Section 2.3 describes the pair relevance branch in more detail.
2.2 Prediction Branch
As shown in Figure 3. we consider pair of prototypes (Pj+ pyy), with j 1.3.... m and j' = + 1. For the k-th prototypical pair, we have j Each prototype Pj is associated with prototypical label Vp; E R. The geometric intuition that, for each latent vector z, and each pair of prototypes (Pj+ Py*) with cor- responding labels (Up,, Up,,) we predict an output ((jd)) by first projecting z onto the k-th ProtoPair axis along the vector py Py. z' (1.9) Mathematically, this projection is given by:
2k
and j'
2k.
with label
with lnbel Aus
D. with lahel yb.
A-th
ProtoPair axis
(0, = Pj + ||z - 11211 cos A
Рj' Pj
(1)
-p
with label BU)
where 9 denotes the angle between the vector z P, and the vector Py Pj-
Figure 3: Geometric interpretation of a prototype pair and latent vector.
Two key assumptions underlie this framework: (1) the labels vary linearly along the k-th ProtoPair axis, and (2) latent representations for the same label lie on a shared hyperplane. As a result, z and z' (.9 will have the same predicted label ((.3)) Using these assumptions and equation ( the prediction U1z.3) is estimated as:
"G.'" = Up, + 1|z -Pill2 cos Upy yp,,
(2)
lp; p
Since we have
(z |12ll || p, [l, Py 112
p,)) (p.- p,) = [(z (B6-pp)" P,)T((;;) - P2)]]
zz - Pill2 cos 0 ||z
(3)
|p; py
we can substitute the right hand side of equation
(3 into the equation
2 for "(d"), obtaining:
[2 p², (py py)] Mr. Dp, Ip, - ly [P, - 1l Sp, + (z - p3)) (py Ps)] (Upy Mpp))
(Ga") = Up,
(py - -g-) ( - -P,)
Using equation 4 the prediction branch takes a latent representation z as input and outputs, for each prototypical pair (P28-11 P2k), a continuous label (,26-

---

This geometric formulation resolves both of the challenges (discussed in Section I of applying existing prototype-basd basedd models to regression. By using equation (4) to compute predictions, our model ensures that the predicted label varies linearly along the ProtoPair axis defined by (P)+ P)r) In particular, equation ( indicates whether the prediction should be greater or less than given prototypical label, based on the input's 's relative position along the axis in the latent space. Additionally, this formulation ensures that when an imput matches a prototype exactly, the prediction will match the prototype's s label mathematically, when we have z Pj "io)" = Vp; or ((j'")) = yp,.. respectively. In the next section, we will discuss how we can scale our model to handle more complex tasks requiring multiple prototypical pairs.
Dj equation ( yields
2.3 Pair Relevance Branch
Since ProtoPairNet can have multiple prototypical pairs, each producing a different prediction label, it is necessary to identify pair that is most suited making the final prediction. This motivates us to define the pair relevance branch, which estimates a relevance score for each prototypical pair. We fur- ther design our model to consider the distance between a latent vector z and each ProtoPair axis, such that the closest ProtoPair axis identifies the most relevant pair for making the final prediction. Mathe- matically, given latent vector z, the pair relevance branch first computes its squared L2 distance to
each k-th ProtoPair axis: d(2k-1,26) z Z (22x -2--1, 1.2k) for all k = I...,, m/2. Then, it computes
a relevance vector r = [F(1.2), F(m- = Gumbel-softmax (1 (-d(1.2);- - -d(m mm))) We use Gumbel-softmal [20] differentiable argmax "approximation" ": has temperature parameter with the property that as T + O, the output of Gumbel-softmax converges to one-hot vectors. Our use of Gumbel-softmax to compute 1(2k-1,2k) for the k-th prototypical pair given by
exp [(dd(xx-112) 1,2k) + 9(24 (12x)=,
T(2k- 1,2k)
(5)
Σ exp (2d + 9(2M-1,,2 )/7]
where (126--1,2X") are i.i.d. random variables sampled from the Gumbel distribution [5 and F is the softmax temperature. The final prediction of ProtoPairNet is given by
m/2
Σ ((28-1,28) 0(22-
The use of Gumbel-softmad means that the final prediction ý is essentially the prediction made by single, most relevant prototype pair. This not only ensures decision sparsity, but also ensures that the final prediction does not suffer from interference of other (irrelevant) prototype pairs.
2.4 Training Algorithm
The training of ProtoPainNet is divided into three stages: (1) optimization via stochastic gradient descent (SGD). (2) projection of prototypical pairs onto training instances, and (3) post-projection optimization. Stage 3 is crucial for ensuring alignment between the projected prototypes and the rest of the network, even slight misalignment in the latent space can lead to significant errors when predicting continuous values. The feature encoder of uuedd in these stages is pretrained on the task-specific dataset beforehand. We use D ((x(") ()) ) to denote a training dataset.
Stage 1. Optimization via stochastic gradient descent (SGD): In the first training stage, we aim to learn meaningful latent space where the latent representations of data points are clustered around semantically similar prototypes and each of them is close to at least one ProtoPair axis. We initialize the prototypical labels using k-means clustering over the labels of the training dataset, to ensure that they represent distinct and well-distributed regions. In this stage, the prototypical labels yp. : ..... 3pm. are fixed, and the parameters the encoder f and the prototypical pairs (P1, P2), ..., (Pm-1, 1, Pm)) are optimized, by minimizing a loss function consisting of a mean absolute error (MAF) or mean square error (MSE), a cluster loss, an axis distance loss, and diversity loss.
We use a cluster loss adapted for regression defined as:
Lost
exp (n yo,) (((0 -P3
5

---

where z(1) = (x) is the latent representation obtained from the encoder f and y( is the label of the input image x! x Also, > 0 is predefined scaling factor, and all other notations are defined in previous sections. The cluster loss encourages the latent representation z of a training image x(+) to be close to prototypes of similar labels, as the squared distance between (9) and each prototype Py is weighted by the closeness of the target label y to the prototypical label l/p,- This weight larger when the target label ااور is closer to the prototypical label V/p,, decaying exponentially as the target label deviates from the prototypical label.
To enforce the geometric structure required for our approach, we use an axis distance loss defined as:
a (b)
(()) 2 (2h 1,20.)
AxisDDist
71
Upa I), and 2 (2h) 1,2kk) being the projection of (6) onto the k*-th ProtoPair axis. The axis distance loss encourages the latent representation (()) to be close to the ProtoPair axis defined by the k*-th prototypical pair. This pair is chosen the one whose associated prototypical label range is the closest to the target label y While the cluster loss ensures that the latent representation of training image stays close to individual prototypes. the axis distance loss ensures that the latent representation of a training image stays close to ProtoPair axis, thereby facilitating the identification of the most relevant pair of prototypes.
with 1- = arg ming min (ly() - Upa--1l ly
Additionally, we use an intra-pair diversity loss that encourages prototypes within each pair to be distinet: LDw
/m/ 1 puell2 Overall, this training stage aims to minimize the total
loss:
Liotal = LMAESSE + A1 + A + A3 CDv-
We use MAE for Age Prediction, MSE for Car Racing, and Å1, A2, A3 are hyperparameters.
Stage 2. Projection of prototypical pairs onto training instances: To visualize the prototypical
pairs {P22-1 ] we need to project them onto pairs of latent representations of training instances. However, simply replacing prototypes with their nearest latent representations as was done in ProtoPNet [ is undesirable, because this could drastically change the direction of the ProtoPair axes, leading to incorrect predictions and performance degradations. To address this issue, we propose a novel prototype projection algorithm, which evaluates whether the latent representations of a pair of training images (x(0), x()) can serve as a good candidate for projecting the k-th prototype pair (P2k-1, P2t) using three metries:
(i) the cosine similarity ag (f,2") between the vectors P2k-1 P2k and (00 z(²':
(i.F)
(Pas 1 Pax) y (269) 201')
P22 - Paxlla (att) "lla where z = f(x(0) and z(t') = f(x(()) denote the latent repre-
sentations of (+) and (²) respectively;
(ii diversity score g(i.") which measures the differences between the latent representations (0) and (f)), as well as the labels y(()) and y)(²) of the training image pair (x(+) x x(()):
3(0,0) 10(0- y (()))12(() -2(()113
(i,x") which measures the average of the squared distances between
(iii) the average axis distance 7/
the k-th ProtoPair axis and the latent representations ((4) and z((); (1,4") (2)
1/2 (l|z(0)
%
(2k 12k-1.20 22) 112 + [|x(() x((')
(2,_1120,)12) 112), where z²(1] ((2- and z ((') 122) are the projections
of z ( and z (8') onto the k-th ProtoPair axis, respectively.
(5.5) measures whether replacing the prototype pairs (P2x-1, Pas) with (z()), (6)) changes the direction of the k-th ProtoPair axis, with higher values indicating less changes in the direction of the ProtoPair axis. The diversity score 3((.") is needed because the prediction becomes numerically unstable if (6) and z(i') are too close to each other, and becomes difficult to
The cosine similarity ok
interpret if ( is too close to y())), The average axis distance 7/ [t.5") is used to find training image pairs that are close to the k-th ProtoPair axis. For each prototype pair (P2k-1 1, P2*), our projection
algorithm selects training image pair (x) x())) that maximizes the cosine similarity aj
(5.2") while

---

Table 1: Age prediction results. Comparision of ProtoPairNet with baseline black-box models (without prototypes) in terms of MAE and R scores across different architectures.
Architecture
RecNet50
NefeeientNet-B40
VGG19
VIT-Small (Patch 16)
MAR
R Score
MAE
R² Score
MAR
R' Score
MAE
R² Score
Baseline
4.71
0.09
0.86
0.004
4.82
0.08
0,86 0.004
4.73 + 0.07
0.86 + 0.008
4.81 + 0.06
0,86 0.004
ProtoPairNett (Ours)
4.59
0.01
0.87
0,001
4.88 0.03
0.86 I 0.0006
4.57 = 0.01
0.87 + 0,001
4,63 0.04
0.87 0.001
ensuring that the diversity score 3(6,5) is sufficiently large and the average axis distance 7 sufficiently small. To reduce the computational complexity, we apply the prototype projection procedure to 50 randomly sampled training image pairs in each training batch.
(2,5")
Once a training image pair (x(1, ) is selected for the prototype pair (P2R-1, P2s), we update the prototype pair to (z(*) 2 and update the corresponding prototypical labels (Vpa-11 Mppc using the ground-truth labels (y((),y())) of the training image pair. We also store the original training images (i) and x(() as (1)(p2)-1) and Vis(pax), where Vis denotes the pixel-space visualization of the corresponding prototype. More details of the projection algorithm, including the pseudo-code, can be found in Appendix G
Stage 3. Post-proe cost-projection optimization: In this stage, we fix the prototype visualizations Vis(p)). Vis(pm) ) and minimize the same objective defined in equation (7) as in stage I, but subject the constraints P, ((Vis((,)) for all j 1, ,,..., This is implemented in code by storing Vis(p,) as parameters in the network with no gradients, and using only (VVis(p)) in the model computations.
This stage is necessary because projecting prototypical pairs onto real data instances may introduce abrupt shifts in their latent space positions. Fine-tuning the model parameters, with the prototypical pairs fixed at the latent representations of their visualizations, ensures that these adjustments integrate smoothly with the overall latent structure and model parameters while preserving the semantic integrity of the projected prototypical pairs.
Experiments
3.1 Case Study 1: Age Prediction
In this case study, we evaluate ProtoPairNet on the UTKKaaee dataset [48] for age prediction, using only the age labels from 23,702 facial images. Ages above 84 are excluded due the limited number of instances in the dataset for these older ages. We apply online augmentations- randoo rotation, flipping, and color jitter-only during training stages and 3; prototype projection (stage 2) is performed on the original (unaugmented) dataset. The training set contains 18,615 images. We use prototype pairs (10 prototypes total) in our main experiments, with ablation studies on loss components and prototype configurations presented in Appendix B
We report the MAEs and R² scores achieved by ProtoPairNet and baseline models (without proto- types) across various architectures, averaged over three runs, in Table Experiments were conducted using widely adopted convolutional neural network (CNN) architectures, including ResNet-50, EfficientNet-B0. and VGG-19, as well as the Vision Transformer (ViT-Small with a patch size of 16). All backbone models were initialized with pre-trained weights from the ImageNet dataset and then finetuned with the UTKFace dataset. Our results, summarized in Table [ demonstrate that ProtoPairNet consistently delivers competitive performance across multiple backbone architectures. For ResNet-50, VGG19, and ViT-Small (Patch 16) backbones, ProtoPairNet achieves lower MAES and higher R² scores, demonstrating superior performance with these architectures while maintaining comparable performance for EfficientNet-BO. Furthermore, ProtoPairNet introduces interpretability across all architectures, highlighting its ability to balance strong predictive performance with enhanced interpretability for regression tasks.
We also compare ProtoPairNet with other prototype-bard based regression models for age prediction. As shown in Table 2 ProtoPairNet outperforms HPN (24), INSightR-Ner [16), and ExPeRT [17] in both MAE and It2 score. HPN was evaluated using its regression variant with two label prototypes, while INSightR-Ner and ExPeRT used 10 prototypes to match our 5 prototype pairs All models use a ResNet-50 backbone and were finetuned using the same protocol as ProtoPairNet. Although HPN performs reasonably (MAE- 5.08), its prototypes not visualizable and thus lack interpretability.
7

---

For INSightR-Net we followed the official setup with backbone pretraining and 1 x patch-based prototypes. To conduct fair comparison with our ProtoPair- Net's non-patch-based design, we also trained an INSightR-NG Net using full latent representations as prototypes. For ExPeRT, we used their original configuration with full-image latent prototypes.
Table 2: Comparison of ProtoPairNet with other prototype-based ResNet50 models for regression using MAE and Rt² scores.
Model
MAE
R score
HPN
5.08 + 0.03
0.84 +00001
INSightR-Ne (1x1)
14.01 +0.01
-0.03 + 0.001
INSighiR-Net (full)
14.81 +0040
-0.08 0.07
ExPeRT
17.36 + 2.79
-0.37 0.41
ProtoPairNet (Ours)
4.59 0.01
0.87 + 0.001
impat
Mist Helevant Prutetyps Pair
pestatope pair
Age
Pockital apc
Tarntyya Age 14
Indtopen Apel
Protic led Las
Age
Prodictd Age
4,05 (uy,,) -11 (151/OL57 2.100
Unxed ELRe
7.35
Figure 4: Reasoning process for age prediction for test image with ground-true truth ages 5
Figure 5: Global analysis for age prediction.
Figure 4 shows how ProtoPairNet predicts the age of a test image, by projecting its latent representa- tion onto the axis defined by the most relevant prototype pair Pı and p2 selected by the pair relevance branch. The prediction branch uses the dissimilarity between the test image and the two prototypes from the most relevant pair to compute the predicted age, in this case, 7.15 for test image with ground truth age 5. Figure presentss a global analysis, displaying three test instances closest to the axis defined by a given prototype pair, arranged by the predicted ages. This analysis demonstrates that the prototype pair effectively captures meaningful variations in the dataset, with the test instances positioned appropriately relative to the prototype pair along the axis. Additional examples of our ProtoPairNet's reasoning process and global analysis can be found in Appendix | and
3.2 Case Study 2: Car Racing
In this case study, we evaluate ProtoPainNet for behavior cloning in the Car Racing task from OpenAl Gym [ using a PPO-trained expert (36). The action space includes three continuous outputs: steering (-1 to 1), acceleration (0 to 1), and braking (0 to 1). The dataset, generated from the expert's black-box policy, contains 36, 107 states (each a stack of four frames), with 25,000 for training and the rest for validation. We evaluate ProtoPairNet as deterministic policy deployed online in the environment. For the reported results, we use two prototype pairs per action dimension-sin pairs in total across steering, acceleration, and braking. We use the same PPO configurations Kenny et al. did in 21 to ensure a fair comparison.
We report the rewards achieved by ProtoPairNet in Table 3 comparing them to those of other regression networks. We compare ProtoPairNet with PWNet (1), which uses hand-picked prototypes, PWNet* 21] which is version of PWNet that uses model-learned prototypes, and k-Means [21) that uses as many clusters as the number of prototypes used in PWNet. In all cases, the rewards were averaged over 5 independent runs, following the same experimental setup described in (1] to ensure a fair comparison. The black-box expert agent achieves an average reward of 221.36, which serves as the reference point for reward comparisons. Notably, ProtoPairNet achieves higher rewards than all other models, including the black-box expert.
Figure 6 illustrates ProtoPairNet's reasoning process for predicting the steering action in a test scenario. The steering predicted by the PPO expert -0.39, while our model predicts -0.34. Despite this slight difference, our model achieves a higher average reward, demonstrating its ability to outperform the expert policy while incorporating interpretability. Additional examples for reasoning process are provided in Appendix K and global analysis examples are in Appendix
8

---

Table 3: Car Racing results. Compari- son of different architectures in terms of rewards.
Prototy pe Pair for steering
Esput image Proficted Steering: -0.44
Pronttppe Storing =0.9
Protutype P. Steentagla, r 0,20
Predicted Steering
matt (4/(4, +4.)
Architecture
Reward
k-Means 21
-2.09 + 0.94
PW-Net* 21
-9.48 +: 2.50
2160))(0.294000
Starr rnn pralicted
PW-Net
220.61 +: 0.70
PPO expert
PrirNea
223,97 + 0.75
00:35
21.65
PPO black box agent 221.36 + 0.96
Figure 6: Reasoning process for Car Racing.
3.3 User Studies
We conducted four IRB-approved user studies with 120 participants (30 for each study) evaluate the interpretability of ProtoPairNet's predictions and reasoning in Age Prediction and Car Racing tasks. In Study 1, participants viewed 10 test face images, each with two reference prototype pairs and their corresponding ages, and selected the pair that they thought better supported their age estimation. Our goal was to assess whether the prototype pairs used by our model aligned with human preferences. A majority of responses (247/300) matched the pairs selected by ProtoPairNet, indicating strong agreement with the model's selection of pairs in pacision-makinn for age prediction. In Study 2, participants were divided into three groups and provided with 10 test images with different levels of information: Group received no references, Group received a single reference face with its age, and Group 3 received a pair of reference faces with their ages. We wanted to see whether providing a pair of references enhances decision-making was significantly lower for Group 3 (4.0) compared to Group 2 (5.96, p < 0.01) and Group 1 (7.85, p < 0.01), indicating that having a pair of references improved decision-making accuracy. For Study 3 and 4, aimed to understand whether users found ProtoPairNet's reasoning process more preferable compared to conventional prototype-base baaseddee-ddddddeeee models' reasoning that uses a single reference example. In Study 3, participants were shown applications of two reasoning processes (one using ProtoPairNet's reasoning that uses a pair of reference faces and the other using a single reference face) to 10 test images. When asked which type of reasoning they preferred, the majority (191 out of 300, < 0.01) favored ProtoPairNet's. Interestingly, participants who preferred a single reference often did so when the reference closely resembled the test image, with one participant noting, "/ think when the faces look so similar to one reference, it is betler to have just one reference than two." In Study 4, participants reviewed 10 car racing scenarios, with a reasoning process that uses pair of reference faces (ProtoPairNet) and another that uses single reference scenario (PWNet (1]). Again, majority (170 out of 300, p < 0.01) preferred ProtoPairNet's reasoning, with pattern similar to Study 3, where those favoring single-reference reasoning did so when the test scenario closely matched the reference. These findings demonstrate that the ProtoPairNet's use of prototype pairs leads to more interpretable explanations and aligns better with human reasoning compared to existing regression models that perform one-to-one comparisons with prototypes (see Appendix A)
The MAE for age estimation
3.4 Occlusion Sensitivity Analysis
While ProtoPairNet's image-level prototypes offer intuitive, high-level interpretability (e.g.. "this face looks more like that of an older person than a younger one"), they do not by themselves reveal which specific regions of an input image are most responsible for the model's prediction. To bridge this gap. we applied occlusion-bas based sensitivity analysis [47] to a ProtoPairNet trained for age prediction, to reveal spatial importance by systematically masking parts of input image and measuring the impact on the model's output.
More specifically, to identify which input region the most relevant prototype pair attends to for a given input image, we sequentially masked input patches and measured the change in its prediction as an importance score. This patch occlusion method, though not built into the core architecture, provides spatial explanations without requiring part-level supervision. We analyzed the top-10 most important patches across 6 randomly chosen test images and found that high-impact regions consistently fall on key facial landmarks such as the eyes, nose, lips, and jawline (see Figure This suggests that our ProtoPairNe consistently focuses semantically meaningful features when making predictions.

---

Table 4: Occlusion sensitivity results showing the average and maximum prediction changes for 6 test images.
Colaionan on
Ferisernge oge
Butuse aniiyyy
Average prediction change
Maximum prediction change
Test image
1
0.773
4.153
fourlwhkgg11 i1
Prodiced It..:
2
0.486
2.904
3 0.376
1.820
4
0.885
3.438
5
0,371
1.497
6
0.172
1.192
Test images (numbered 1-6) are shown on the left, with corresponding heatmaps on the right highlighting facial regions most influential to the model's predictions.
Figure 7:
Quantitatively, as shown in Table for the randomly chosen test images. the average change in prediction across occluded patches ranged from 0.172 to 0.885, whereas the maximum change in prediction resulting from occluding the most important patch ranged from 1.192 to 4.153. The substantial difference between the average and maximum prediction changes resulting from patch occlusions indicates that only a small subset of patches, typically those corresponding to key facial landmarks, has significant impact our ProtoPairNet's output. This further confirms the importance of these regions in our model's decision-making process.
3.5 Ablation Study: Is a Single, Most Relevant Prototype Pair Sufficient for Prediction?
While our ProtoPairNet may contain multiple prototype pairs, the use of Gumbel-softmax in the pair relevance branch means that each prediction of our ProtoPairNet essentially made by single, most relevant prototype pair. To evaluate whether a single pair is sufficient for each prediction, conducted an ablation study replacing
Table 5: Age prediction performance with standard softmax, simple averaging, and Gumbel-softmax pair relevance.
MAE
R² score
Standard softmax
4.49 :: 0.007
0.877 0.0006
Simple average
4.48 : 0.003
0,878 0.0004
Gumbel-softmax (Ours)
4.59 + 0.01
0.87 + 0.001
Gumbel-softmar with (1) standard softmax and (2) simple average of all prototype pairs' predic- tions. We found that using multiple pairs per prediction through standard softmax or simple averaging did not significantly improve the model's performance for age prediction, suggesting that single, most relevant prototype pair identified by Gumbel-softmax is often sufficient for each prediction (see Table 5). More ablation studies found in Appendix B
Limitations, Future Work, and Conclusion
Our approach does not use part- explanations as ProtoPNet [T and its variants do. Leveraging prototypical parts as prototypes could improve interpretability by linking specific regions of the input to meaningful, localized concepts. However, we consider our pair-based formulation a novel contribution on its own. Our method introduces new framework that explains predictions by positioning the latent representation of an input along a meaningful axis defined by a pair of prototypes, capturing smooth transitions and continuous variation across examples for regression tasks. Extending our framework to incorporate part- based reasoning is a promising direction we would like to explore.
To conclude, in this work, we presented the ProtoPairNet, novel model for prototype-based regression that incorporates comparisons with pairs of prototypes. Our code available at phttpss //github. httpa//c.../eeeeee/ooooooaiiiis
10

---

Acknowledgments
We would like to acknowledge funding from the National Science Foundation (NSF) under grants RII Track-2 FEC OIA-2218063 and NSF CAREER IIS- IS--2442399
References
[1] Agnar Aamodt and Enric Plaza. Case-Basee Based Reasoning: Foundational Issues, Methodological Variations, and System Approaches. Al Communications, 7(1):39-59 1994.
[2] Samira Abnar and Willem Zuidema. Quantifying Attention Flow in Transformers. In Pro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1190-4197, 2020.
[3] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity Checks for Saliency Maps. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
[4] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wirs Wise Relevance Propagation. PLOS One, 10(7):e01301 2015.
[5] Alina Jade Barnett, Fides Regina Schwartz, Chaofan Tao, Chaofan Chen, Yinhao Ren, Joseph Lo, and Cynthia Rudin. A case-based interpretable deep learning model for classification of mass lesions digital mammography. Nature Machine Intelligence, 3(12):001-1177, 2021.
[6] Richard Berk. Machine Learning Risk Assessments in Criminal Justice Settings. Springer, 2019.
[7] Andrea Bontempelli, Stefano Teso, Katya Tentori, Fausto Giunchiglia, and Andrea Passerini. Concept-level Debugging of Part-Prototype Networks. The Eleventh International Conference on Learning Representations (ICLR), 2023.
[8] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv:1606.0/ 0ete:16666.154400 5400,, 2016.
[9] Zachariah Carmichael, Suhas Lohit, Anoop Cherian, Michael J Jones, and Walter J Scheirer. Pixel-Grounded Prototypical Part Networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4768 4779, 2024.
[10] Zachariah Carmichael, Timothy Redgrave, Daniel Gonzalez Cedre, and Walter Scheirer. This Probably Looks Exactly Like That: Invertible Prototypical Network. In European Conference Computer Vision (ECCV), pages 221-2400 Springer, 2024.
[11] Chaofan Chen, Oscar Li, Chaofan Tao, Alina Barnett, Jonathan Su, and Cynthia Rudin. This Looks Like That: Deep Learning for Interpretable Image Recognition. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[12] Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10265-10275, 2022.
[13] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing Higher-Layer Features of Deep Network. University of Montreal Technical Report 1341, 2009.
[14] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding Deep Networks via Extremal Perturbations and Smooth Masks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2950-2958, 2019.
[15] Emil Julius Gumbel. Les valeurs extrêmes des distributions statistiques. Annales de I'Institut Henri Poincaré, 555): 1935.

---

[16] Linde S Hesse and Ana IL Namburete. INSightR-Nee Interpretable Neural Network for Regression Using Similarity-Baan Based Comparisons to Prototypical Examples. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 502-511, Springer, 2022.
[17] Linde S Hesse, Nicola K Dinsdale, and Ana II. Namburete. Prototype Learning for Explainable Brain Age Prediction. In Proceedings of the JEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7903-7913, 2024.
[18] Dat Hong. Tong Wang, and Stephen Baek. ProtoryNet- Interpretable Text Classification Via Prototype Trajectories. Journal of Machine Learning Research (JMLR), 24(264):1-39, 2023,
[19] Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols. Perturbation based methods for explaining deep neural networks: A survey. Pattern Recognition Letters, 150:228-2344 2021.
[20] Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparamctrization with Gumbel-Softmax In International Conference on Learning Representations (ICLR), 2017.
[21] Eoin M Kenny, Mycal Tucker, and Julie Shah. Towards Interpretable Deep Reinforcement Learning with FliendyFriendly Prototypes. In The Eleventh International Conference Learning Representations (ICLR), 2023.
[22] Chiyu Ma, Brandon Zhao, Chaofan Chen, and Cynthia Rudin. This Looks Like Those: Illumi- nating Prototypical Concepts Using Multiple Visualizations. Advances in Neural Information Processing Systems (NeurIPS), 36, 2023.
[23] Chiyu Ma, Jon Donnelly, Wenjun Liu, Soroush Vosoughi, Cynthia Rudin, and Chaofan Chen. In- terpretable Image Classification with Adaptive Prototype-based Vision Transformers. Advances in Neural Information Processing Systems (NeurIPS), 37, 2024.
[24] Pascal Mettes, Elise van der Pol, and Cees GM Snoek. Hyperspherical Prototype Networks. Advances in Neural Information Processing Systems (NeurlPS), 32, 2019.
[25] Yao Ming, Panpan Xu, Huamin Qu, and Liu Ren. Interpretable and Steerable Sequence Learning via Prototypes. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 903-913, 2019.
[26] Meike Nauta, Ron van Bree, and Christin Seifert. Neural Prototype Trees for Interpretable Fine-grained Image Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14933-14943, 2021.
[27] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred imputs for neurons in neural networks via deep generator networks. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016
[28] Mateusz Pach, Koryna Lewandowska, Jacek Tabor, Bartosz Zielinski, and Dawid Rymarczyk. LucidPPN: Unambiguous Prototypical Parts Network for User-centric Interpretable Computer Vision. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[29] Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized Input Sampling for Explanation of Black-box Models. In British Machine Vision Conference (BMVC), 2018.
[30] Ronilo Ragodos, Tong Wang. Qihang Lin, and Xun Zhou. ProtoX: Explaining Reinforcement Learning Agent via Prototyping. Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.
[31] Sebastian Ramos, Stefan Gehrig, Peter Pinggera, Uwe Franke, and Carsten Rother. Detecting Unexpected Obstacles for Self-Driving Cars: Fusing Deep Learning and Geometric Modeling. In 2017 IEEE Intelligent Vehicles Symposium (IV), pages 1025-1032. IEEF, 2017.
[32] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1:206-215, 2019.
12

---

[33] Dawid Rymarczyk, Lukasz Struski. Jacek Tabor, and Bartosz Zieliński. ProtoPShare: Pro- totypical Parts Sharing for Similarity Discovery Interpretable Image Classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1420-1430 2021.
[34] Dawid Rymarczyk, Lukasz Struski, Michal Górszczak, Koryna Lewandowska, Jacek Tabor, and Bartosz Zieliński. Interpretable Image Classification with Differentiable Prototypes Assignment. In European Conference on Computer Vision (ECCV), pages 351-368. Springer, 2022.
[35] Sascha Saralajew, Ashish Rana, Thomas Villmann, and Ammar Shaker. A Robust Prototype- Based Network with Interpretable RBF Classifier Foundations. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 20273-20282. 2025.
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint vrXiiv:707..00 7734006477 2017.
[37] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient- based Localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 088866,2222 2017.
[38] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv preprint 2013.
[39] Mohana Singh, BS Vivek, Jayavardhana Gubbi, and Arpan Pal. Prototype-based Interpretable Model for Glaucoma Detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 5056-5065, 2024.
[40]
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- Grad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017.
[41] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 2017.
[42] Andong Tan, Fengtao Zhou, and Hao Chen. Post-hoe Part-prototype Networks. Proceedings of the 41st International Conference on Machine Learning (ICML), pages 47681-47693, 2024.
[43] Chong Wang, Yuyuan Liu, Yuanhong Chen, Fengbei Liu, Yu Tian, Davis McCarthy, Helen Frazer, and Gustavo Carneiro. Learning Support and Trivial Prototypes for Interpretable Image Classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2062 20722, 2023.
[44] liaqi Wang. Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable Image Recognition by Constructing Transparent Embedding Space. In Proceedings of the IEEE/CVE International Conference Computer Vision (ICCV), pages 895-904 2021.
[45] Frank Willard, Luke Moffett, Emmanuel Mokel, Jon Donnelly, Stark Guo, Julia Yang, Giyoung Kim, Alina Jade Barnett, and Cynthia Rudin. This Looks Better than That: Better Interpretable Models with ProtoPNeXt. arXiv preprint 1rXXi:24406114677 2024.
[46] Haoming Yang, Steven Winter, Zhengwu Zhang, and David Dunson. Interpretable Al for relating brain structural and functional connectomes. arXiv preprint arXiv:2210.05672. 2022.
[47] Matthew D Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 1, pages 818-833 Springer, 2014.
[48] Zhifei Zhang. UTKFace Dataset, 2017. [Online]. Available: https: wwwpusaggg.gghuubbb..uu.e io/ UTKFace
Accessed: Feb. 24, 2024.
[49] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning Deep Features for Discriminative Localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2921-2929, 2016.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The thcoretical derivations and experimental results along with the user studies in the appendix supports the claims made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the main paper Section 4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
The theoretical formulation of our approach is discussed in detail in Section 2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification;
The training hyperparameters and the training procedure are described in Appendix Dnepppeccttieee The experimental details are described in Section of the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our code is available htttpp///ggtaaooo//////2///oooo
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.co cc/
Shbllic/guidess/oddeelbmm.ssss wtt.sSubbmiiiiooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ccpppi.ccppulli///grg guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Data splits are discussed in Section 3 of the main paper and Appendix H The hyperparameter settings and ablation studies are listed in Appendix C and Appendix respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All of our results are reported as mean + standard deviation over multiple runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: It is discussed in Appendix M
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted the paper conform, in every aspect, to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts are discussed in Section | of the main paper and in Appendix N
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They are properly cited and credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
The new approach developed in the paper is well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/coodemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The paper includes user studies and example questions are included in Ap- pendix A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The details of the user studies, including the consent process, are described in Appendix A The protocols of the user studies have been approved by our institution' 's IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our method does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

Appendix Table of Contents
Details on User Studies
22
A.I
Study ]
22
A.2
Study 2
22
A.3 Study 3
23
A.4 Study 4
24
B One-to-One Comparisons versus Prototype Pair-based Reasoning
28
C Training Hyperparameters
28
D Training Procedure
29
Ablation Studies
29
Car Racing Model Architecture
30
Projection of Prototypical Pairs
31
H Age Distribution in the UTKFace Dataset
32
More Examples of Reasoning Process for Age Prediction
32
More Examples of Global Analysis for Age Prediction
33
K More Examples of Reasoning Process for Car Racing
34
More Examples of Global Analysis for Car Racing
35
Reproducibility and Empirical Computational Cost
35
Broader Impacts
35
21

---

A Details on User Studies
We conducted four user studies, each involving 30 participants recruited through Prolific. All participants provided informed consent prior to data collection and were fully informed of their rights. The protocols of these user studies have been approved by the IRB. Each study session lasted approximately 10 to 20 minutes, and participants were compensated at rate of $10 per hour.
The goal of the first study with human preferences. The second study aimed to evaluate whether presenting participants with pair of references (i.e., prototypical pair) would help them perform prediction tasks (e.g.. age prediction) more effectively than when given only a single reference (i.c., single prototype). Studies three and four further examined whether our model improved interpretability compared baseline methods that used
assess whether the prototype pairs generated by our model aligned
explanations.
Overall, the results of user studies suggest that providing pairs of explanations (prototypes) not only aligns with human preferences better, but also helps participants perform prediction tasks more effectively and provides clearer explanation of the model's 's reasoning process. The following section provides further details on the examples used and the implementation of each study.
A.1
Study 1
In this study, 30 participants were presented with 10 test photos of human faces, alongside two pairs of reference faces. These reference pairs corresponded to the training examples that our ProtoPairNet'ss learned prototype pairs were projected onto. For each test photo, the reference pair they felt better helped them estimate the age of the person in the photo. Our goal was if the prototype pairs selected by ProtoPairNet for its decision making aligned with those selected by the participants. Some example questions for Study are shown in Figure 12 (a).
asked participants to choose
User Study 1: Pair Selection Aligument Between Users and Our Model for
Coumt
Age Prodiction
250
347
230
150
Hotoodigaas grad
Hanveigrad
Figure 8: Study result.
The result of Study 1 is illustrated in Figure g We observed that the majority of users' reference pair selections were aligned with the reference prototype pairs chosen by ProtoPairNer, with 247 instances (out of 300 responses) of alignment compared to 53 instances (out of 300 responses) of non-alignment. This indicates strong preference for the prototype pairs utilized by the model in its decision-making process for age prediction.
Results:
A.2 Study 2
In this study, we worked with another 30 participants, dividing them into three groups of 10. Each participant was shown 10 test photos of human faces. For each test photo, participants in Group I were not given any reference training examples. Those Group 2 were provided with a single reference training example, the age of the reference, and a dissimilarity score between the test photo and the reference. Participants in Group were given a pair of reference training examples, their ages, and the relative dissimilarity between the test photo and the pair of references. We asked
22

---

participants to estimate the age of each test photo based on the information provided and to rate how useful they found the information making their guesses. This study explored whether providing pair of reference examples enhances decision-making and increases the probability of accurate predictions compared to a single example or none at all. Some example questions for Study
shown in Figure 12 (b) for group 1, Figure 12
(c) for group 2, and Figure T2 (d) for group 3.
(4]
Study MAE Comparison Across Groups
Age
(5)
Study Average Holpfalness Rotings (Droup 1)v5.
Prediction
Reference Images (Group 2)
66:81
0ec2 de:93
Dernode
Ovep
Drap
Figure 9: Study 2 results.
The results of Study are illustrated in Figure 9 As shown in Figure 9 (a), we observed that the mean absolute eтгоr (MAE) when guessing the ages of the test photos was significantly lower for participants in Group 3 (4.0), who were provided with pair of reference examples, compared those in Group 2 (5.96) who were provided with single reference example and Group (7.85) who were provided with no reference example. The participants in Group demonstrated better performance in predicting the age, suggesting that providing a pair of reference examples enhances decision-making accuracy. Additionally, ProtoPairNet outperformed all participant groups, achieving lowest MAE for the test photos. To assess statistical significance, we conducted independent t-tests comparing Group 3 with Group and Group 1. As shown in Table 6 the results indicate statistically significant differences, with Group outperforming Group (t 2.72, = 0.0091) and Group references are statistically significant.
Results:
4.55, 0.0004), confirming that the improvements observed with paired
Furthermore, shown in Figure (b), participants in Group 3 rated the helpfulness of the provided information higher on average (3.7) compared Group (3.14). This indicates that the additional context from two reference examples was perceived more effectivve in aiding participants age predictions, highlighting the importance of multiple references in improving decision- vecision-making..
Table 6: 1-Test results comparing MAEs of Group 3 with Group 2 and Group
Groups
t-Statistic
p- Value (one-tail)
Group 3 Group
-2.72
0.0091
Group 3 Group
-4.55
0.0004
A.3 Study 3
In this study, 30 participants were shown 10 test photos of human faces. For each photo, they were presented with two reasoning processes: one using a pair of reference faces (Justification 1) and the other using single reference face (Justification 2). We asked the participants to choose which reasoning justification they found more effective for predicting the person's 's age. We also added comment box where the participants could briefly state their reasons for their choices. This study aimed to see if human users prefer the reasoning based on two reference examples or a single reference example. An example question for Study 3 is shown in Figure 13 (a).
Results: The result of Study 3 is illustrated Figure 10 The majority of participants (191 out of 300 responses) preferred the reasoning process using pair of reference images (Justification
23

---

User Study 3: Preference for the Ressoning Process using a Pair of Reference Images Reference Image
Coust
258
209
Par weeeecencciinn 3araicaten
qut Bcalinn)
Figure 10: Study 3 results.
1) compared to fewer (109 out of 300 responses) participants who chose the process with a single reference image (Justification 2). Interestingly, based on the comments left by the participants, the cases where participants favored Justification 2 were usually when the reference face in the image closely resembled the test photo. The comment from one such participant was "I think when the faces are closer together on end of the spectrum it is better to have just I reference point.", However, given the inherent uniqueness of human faces, relying on highly specific references is not practical and limits generalization. For the participants who favored Justification 1, they favored it strongly, leaving comments like "Justification I better justifies the prediction because it compares the face with both younger and older faces, suggesting more muanced relationship", ", "Justification I has more data points to validate its decision. Comparing two faces increases confulence compared comparing one face.", etc. This considerable preference underscores the importance of the additional contextual information provided by the pair of references when predicting the ages of human faces.
The result a one-tail (-test on the mean counts of the justifications chosen by the users, dubbed as the "Justification Score," further illustrate that there is a statistically significant preference for the reasoning process of using a pair of references compared to using a single reference as shown in Table [
Table 7: User Study results comparing the Justification Score for using pair of references and the Justification Score for using a single reference.
Justifications
t-Statistic
p- V-Value (one-tail)
Pair of References Justification Score 5.67
2.34 x 10-7
Single Reference Justification Score
A.4 Study 4
In this study, 30 participants were presented with 10 test scenarios from car racing game. For each test scenario, participants were provided with the reasoning process our ProtoPairNet used to predict the level of acceleration (Decision I and Justification 1), and the reasoning process the baseline model PWNet [21] used to predict the level of acceleration (Decision and Justification 2). The "Decision" " here refers to the predicted acceleration and the "Justification" refers to the reasoning process of each model. Participants were asked to select the decision that made more sense to them out of the two and, separately, to choose the reasoning process they preferred. Additionally, we also put comment box after every question for the participants to briefly justify their choices. This study aimed to understand whether users find ProtoPairNet's prediction process more intuitive and preferable compared to the baseline model's approach. An example question for Study 4 is shown in Figure 13 (b).
24

---

User Study & Preference for the Reasoring Process of Our Model vs. PWNet (Decision lustification)
354
200
Destvor (nd) Bration
Figure 11: Study 4 results.
The results of Study 4 are illustrated in Figure 11 We observed that the majority of participants preferred the reasoning process of ProtoPainNet (Decision and Justification 1) over the reasoning process of PWNet (Decision and Justification 2). Specifically, for decisions alone, 165 participants favored ProtoPairNet's decisions compared 135 participants who favored PWNet's decisions. Similarly, for justifications. 170 participants preferred ProtoPairNet's justifications com- pared to 130 participants favoring PWNet's justifications. Based on participant feedback, those who preferred ProtoPairNet's reasoning process highlighted its clarity and the strength of the comparisons drawn, often mentioning that the reasoning process felt more comprehensive and aligned better with their expectations. As in Study 2, the participants who favored Decision and Justification 1 favored it strongly, with comments like "It makes me nervous how Justification and Decision will account for a more curved road predict acceleration. Decision 1 considers both curved and straight road and checks how curved the scenario is before making a decision. The way the first model makes the decision actually good, it checks similar roads and places the sample along the spectrum and matches acceleration. The second model feels like black box, it just says they are slightly similar and spits out a number." " This comment highlights the weakness of PWNet's manually picked prototype for acceleration which is just straight road and is used to predict the acceleration for all scenarios in the environment. On the other hand, the participants who selected PWNet's 's reasoning appreciated its simplicity and as in Study 3, favored it whenever the test scenario looked significantly close the single reference scenario provided. One such participant left the comment, "I prefer having two frameworks of comparison, but Justification 2 having only 2% difference seems more realistic choose". These results emphasize the strength of ProtoPairNet's 's reasoning process in both decision-making and justifications, demonstrating the added value of robust and well-supported reasoning framework.
Results:
The result of a one-tailed (-test on the mean counts of justifications selected by users, or the "Jus- tification Score" demonstrates statistically significant preference for the reasoning process of ProtoPairNet compared to PWNet, as shown in the first row of Table [8 To further analyze participant preferences, we combine decisions and justifications into single metric by multiplying them (i.c., Decision 1 x Justification for ProtoPairNet and Decision x Justification 2 for PWNet), dubbed "Preference Score" the table. This approach highlights cases where participants consistently align with both the Decision and Justification from model, emphasizing the importance of agreement on both aspects. The interaction between Decision and Justification is critical, as alignment with both reflects more robust preference than agreement on either aspect individually. Again, this shows that participants found our ProtoPairNet's reasoning process more intuitive and its decisions better justified, further validating its effectiveness in this study.
25

---

Table t-Test results comparing ProtoPairNet'' scores with PWNet's scores for Justification and Preference.
Models
t-Statistic
p- Value (one-tail)
ProtoPairNet's Justification
Score
2.9
0.0026
PWNet's Justification Score
ProtoPainNet''
Preference
Score
2.6548
0.0051
PWNet's Preference Score
Question
(a)
Question
(c)
Usinn age information two quess penca
image
Below: scores typoaaly range rightr soones indicading
higher dissmiarry.
you quesa the gi of person in abovc picture??
Ape
Age
Dinislaie
Enter
Question
ed. 23
Using age folm alaan of lec gadio the perzar's age.
Adctiionally fily, pieese rate how much you butiave the information about the other fice guees
wealitie usald
you gless the age penson the abovs pktuvet
Question
Fleass quess the age person using the infommaioon image
Déoimele typically foo O with hightr indicating higher dlssm larey.
Question
(b)
Extar for the possens
Age:s
Back
hext
Dioinkurt L12
Dastesbirty 0.11
Enter your guees for the peron's
Question
qq.
Enter your gaesa for the parson's agec
Add
information about
tscrs ped you maie your gunnoc
Figure 12: Example questions for Study (a), and Study 2, Group 1 (b), Group 2 (c), Group (d).
26

---

Question 1
Question
Consiter the ponten right, Plooso solee which tim poatiications bolonn better su supports the mocell prediction for wis personts age, Dr Driiiiarity scoms in the imoges only range from to with highes scons eddeatinng Higher skookniaaiiy
beter
the shee mert
the glae conarie
Doukiion and jest
The model coddes that scceleastonn should empapedd G0R been scnsare adcre between the nowiiig t,to sce varie or whher sccacceen acceerace ergaged en 235 and 77% respect trely, but more uiieer sconario D. Covtrntarrtyy seere range from O uny pe pusliive mann too.. with liihhor sears incioting Higher salanberty.
Justiieution
The model
beceuse netween 53 respectivey
Arnderation
Axumdtart- 2015
dovenlenyy 13.27
juctiication
The rodul vookks thail socolis lus sirealdd Us payed dTis secerael given scmrora docres caneewar cimarr sefeence ecenaro wishh asselecnce ergaged Seniarity soors ragg from Cito tighor stores bnttating
The mode prodicts eis the to be 6130 because the pesson above is somswtet san isterance person whose ago is 2%
Auceltvation
Relvvee
Aocleaajes
Agc T1
saalery
Disioklutt -
Deacent
Which lestio better justifies the prediction made by the machine leamiing
Aust on
Jusl
Just calbn
Pleass brietly justily your choicee:
Pleass bridfy, juitly your cholces:
Backk
Neant
Back
Next
Figure 13: Example questions for Study 3 (a), and Study (b).
27

---

B One-to-One Comparisons versus Prototype Pair-based Reasoning
Reasoning with one-fo-one comparison:
standertey:
Predicted Age
Suudiarity
Ags
9
Teput Beage Oraginal ape
Age 80
Reasoning Protutypical pair:
inleentt legar
Age vartes Wincrrty along this
Prateope Age
Pretatge
Age 40
Focpeted crenbabel
The input image exactly Prutotype predicted age
Predicted
Inpart lnonge Ooiginal age age
One-to-one comparisons prototype pair-based reasoning, when the input image matches prototype exactly.
Figure 14:
Figure demeonsttss why a wieiltrity-weighted average can cause unwarranted deviations, even when input image matches a prototype exactly. As shown at the top of Figure 14 while the input image is the same as the prototype whose label is 9, the 2-prototype model using similarity weighted averaging predicts 9 x 1 + 80 x 0.01 9.8 * 9. In contrast, the prototype pair-based reasoning (used by our ProtoPairNet) addresses this issue effectively. By design, when an input image the same prototype, ProtoPairNet predicts the label of the prototype. This is illustrated in the bottom of Figure
C Training Hyperparameters
Table 9: Hyperparameter settings for ProtoPairNet
Parameter
Weight
MAE
1
Cluster loss Leist
0.008
Axis distance loss CAxisDia
0.008
Diversity loss LDiv
0.001
Gumbel softmax temperature
0.1
This section documents the hyperparameters used to train our ProtoPairNet, as shown in Table The hyperparameters shown in the table are consistent across all the architectures. The hyperparameters chosen using grid search on validation set for both the age prediction task and the racing application.
To evaluate whether the Gumbel-softmax temperature is sufficiently small, we computed the mean J distance between the Gummbel-sfftax outputs and their binarized one-hot versions pnc-hot over the test set of the age prediction task. At the temperature 0.1 used in our experiments, the
28

---

mean distance was 0.02, indicating that the Gumbel softmax outputs are sufficiently close to one-hot encodings. Training Procedure
D
Loaded pretrained backbone
Stage 1: Optimizing via SGD
add-on lapers, l.. prototypical paira pretrained backbors
Objective function:
MAGMSE
, Ciat
2. AvisDist
+2 "
Stage 2:
Prototype Projection
Project prototypical pairs to distinct paira of training instances that are closest the pair axis while maintaining the strongest cosine alignment with the pair and diversity within the pair.
Stage Post-Projection Optimization
Finstune backbons, (, and add-on layens, Fut... with fixed prototypical pairs.
objective
Stago
2, Asiania + 21
MAFAASE
Cld
Trained ProtoPairNet
Figure 15: The procedure for training ProtoPairNet.
The three stages training as discussed in the main paper are shown Figure 15 For all architectures. we use backbone weights pretrained on the corresponding dataset. The encoder f is composed of the backbone fob and add-on layers faid-on- In summary, this defines the basic workflow of our approach: Stage trains the model along with the prototypical pairs, Stage 2 projects them onto selected training instances, and Stage fine-tunes the model with a reduced learning rate while keeping the prototypical pairs fixed at the latent representations of the selected training images.
E
Ablation Studies
The ablation study on loss terms shown in Table 10 reveals that removing the axis distance loss disrupts the projection mechanism, preventing prototypes from aligning with the most cosine-similar pair of training instances. This is evident from the fact that cosine values between prototype pairs before and after projection are no longer close to one. contrast, removing other losses does not affect this alignment, indicating that the axis distance loss plays crucial role in ensuring meaningful prototype placement. While the overall performance impact of removing any single loss term is not significant, these losses collectively help structure the latent space, encouraging the model to learn more useful and interpretable representations.
Table II and Table 12 show that increasing the prototype dimension or the number of prototype pairs does not significantly affect performance. This suggests that these hyperparameters can be adjusted flexibly based on the task without compromising model effectiveness. We find that smaller prototype dimension sufficient for structuring the latent space effectively, while larger dimensions do not provide additional benefits.
29

---

Table 10: Ablation study for age prediction on cluster loss, axis distance loss, and diversity loss.
Cosine alignment
between prototype pairs before and after projection
Removed parameter
MAE
R score
(5 pairs)
None Cluster loss LCIst
4.59 = 0.01
0.87 + 0.001
[0.99, 0.99, 0.99, 0.99, 0.93]
4.78 + 0.15
0.86 + 0.005
[0.99, 0.99, 0.99, 0.96, 0.93] [0.64, 1010, 0.01, 0.73, 0.10]
Axis distance loss LixisDist
4.75 + 0.07
0.86 + 0.001
Diversity loss Liv
4.83 + 0.31
0.86 + 0.010
[0.99, 0.99, 0.96, 0.99, 0.93]
Table 11: Effect of prototype dimension on age prediction performance.
Prototype dimension
MAE
R² score
50 (Used in paper)
4.59 = 0.01
0.87 = 0.001
100
4.71 = 0.13
0.87 = 0.002
200
4.81 = 0.40
0.86 = 0.003
Table 12: Effect of the number of prototype pairs on age prediction performance. Increasing the number of prototype pairs decreases test performance. 5 pairs sufficient and optimal in our case.
Number of prototype pairs
MAE
R score
5 (Used in paper)
4.59 + 0.03
0.87 + 0.001
10
4.85 + 0.18
0.86 + 0.004
20
4.89 = 0.15
0.86 + 0.007
40
5.24 = 0.04
0.84 + 0.001
50
5.58 = 0.50
0.83 0.019
F
Car Racing Model Architecture
Protatypical pairs and corropanding labdl:
PP P. X-.. E Pulo-ol P Pu.. P. and by coo A Dha -Pyl -.. Dia Pr.1
Pakr
Auk as layan accolomatrn
brandi
Frature Rackbane
Foct m
sockrallie 46%
Uving vesabe
Layers
branch
(iagrquaio (5)
Ex Eubiay
Figure 16: ProtoPairNet architecture for Car Racing. There are three separate sets of add-on layers for three output dimensions: steering, acceleration, and braking.
When the output has more than one dimension as in Car Racing (steering, acceleration, and braking), we can use separate set of add-on layers for every dimension in our ProtoPairNet (similar to PWNet
30

---

Algorithm Projection of prototypical pairs
Input: Trained model f. training data D ((x(*) y prototypical pairs ((P2=-1: P24 k=1
)
prototypical labels {(ppe-. Upuz ) and empty prototypical visualizations
((Vis(Pas-1), Vis(p2x))
Output: Updated prototypical pairs ((22- Pus |m labels (1pp.-1 Spm)) 3 m populated
visualizations ((Vis(pz-11, Vis(P2x)) m/2 )
Initialize maximum cosine similarity aj maximum diversity 8 0, minimum distance from pairs 2/ oo for all & = = I.....m.. 2 pairs
2: for each (x ()) E D do
3: Compute latent representations ( s(x(0) and d 1,2k) = -||2 2((6)
(2k 24) 112 for
all k: = 1.-....,mm, m/2 pairs
4: for 1...., m/2 do
for (z()) z ) 1) E Unique pairs do
5:
(t,5") (P 1 Pos) (x -(0))
6:
k
P Pou lla [2
7:
804"
(z))) 2 112
y0) 10(0)
8:
7h
(1,1)
2 (di2s- (22- 1,26)
(2k 1,26)
a 2 ag and g(i,t) > and 7k
(1,5))
(i,t') 7% then
9:
10:
P t 200, P2k + (i)
Up 1 + y(6), y/aa + y()
11:
12:
Vis VPs t- -x((), Vis(p2x) t x()
(i,')
13:
a
+ a
8 gli.5)
15:
% t- davg
16:
endl if
17:
end for
end for
19:
end for
Return {(D2-11, Puk)) m// (1pss-1 3p..) Mm? and {(Vi(pax=-1) Vis(p ))
20:
21 This separates prototypical pairs of different output dimensions different latent spaces. The model architecture in this scenario illustrated Figure 16
G Projection of Prototypical Pairs
The pseudocode for our projection procedure is shown in Algorithm [ with all relevant notations defined therein. We begin by initializing the cosine similarity ag to 0. the maximum diversity 3* to O, and the minimum distance to the ProtoPair axis Tf to infinity. These values are tracked and updated across all batches of the training dataset. For each batch, representations and their distances to all ProtoPair axes. Then, for every k-th prototypical pair, search for candidate pairs of training instances that achieve the highest cosine similarity to the corresponding prototypical pair learned by the model. By doing so, we ensure that the axis formed by these candidate pairs remains well-aligned with the k-th ProtoPair axis. Maintaining high cosine similarity during projection is essential to preserving the orientation of the ProtoPair axis and avoid performance degradation due disruptive changes in direction. Mathematically, the cosine similarity a'
first compute the latent
(t,x") between the k-th ProtoPair axis and the latent direction defined by candidate pair x(() x((
(whose latent representations z(i) 2()), respectively) is given by:
(8,80 (p2k-1 P2a ) (z (())
P2t-1 P2* la (x) ( 1
We compute this for the top random samples within each training batch, restricting the search to individual batches reduce computational cost. Next, for these candidate pairs, compute the
31

---

diversity between the prototypes in these pairs. This is to ensure that the prototypes in the pairs exhibit sufficient stability and avoid cases where the prototypes within a pair may collapse into the same values. Mathematically, the diversity score B(1²") between the latent representations z(4) and (t)), as well as the labels y() and y!t') of a candidate pair x()) and x("), is given by:
glid" (y( - y ( 10 - z 1
(6,2) between the k-th ProtoPair axis and latent 7k
Finally, we compute the average axis distance candidate pair (z(+), 2(²) as:
(()) (6,4') 1 ((12( (2k-1.28) 112 + ||z((') - -z( (t') (2k 1,28) 1223))12)
which measures the proximity between the candidate pair and the corresponding prototypical pair in the latent space. These three metrics are used together to select the pairs that maximize cosine similarity to the corresponding prototypical pair, exhibit sufficient diversity latent and label spaces, and have the smallest average distance to the k-th ProtoPair axis. We then update our model- learned prototypical pairs and their corresponding prototypical labels with these chosen pairs of latent representations and their corresponding labels. We also store the image representations or visualizations of these newly projected prototypical pairs this stage as Vis(P2x-1) and Vis(p where Vis denotes the prototype visualization function. This enables visualizations in the pixel space while also facilitating the next stage of training. Finally, if a candidate pair exceeds the current tracked metrics Og _*,, and ng. we update these values to reflect the optimal candidate, as detailed in Steps 13, 14, and 15 of Algorithm
H Age Distribution in the UTKFace Dataset
Thoh
Validation
Figure 17: Age distribution across the training, validation, and test sets.
Figure 17 shows the distribution of age labels across the training, validation, and test sets. All three sets are skewed toward younger and middle-aged individuals, with relatively fewer examples at older ages. This observation provides context for the interpretability results shown in the figures in Appendix[aand and ! Many instances in the dataset correspond to younger individuals, which naturally align more closely with the lower end of their most relevant prototype pairs. Since our model predicts by projecting an input onto the axis of prototypical pair in the latent space, examples from younger age groups often appear nearer to the left (younger) prototype. This behavior reflects both the data distribution and the model's 's reasoning.
I More Examples of Reasoning Process for Age Prediction
This section provides additional examples illustrating the reasoning process for age prediction. Figure [8 and Figure show the reasoning process of ProtoPairNet for predicting the age of total of 24 random input images. We find that the predicted labels of 77.22 % of test images fall within the label range of their corresponding most relevant prototype pair, while the predicted labels of the remaining 22.78% % lie outside that range. Since our ProtoPairNet makes predictions along the axis defined by prototype pair, test samples may naturally lie either within or outside the label range of the prototype pair; both scenarios are consistent with our formulation.
32

---

Most Aeliaas Pronaepe Fair
Belevand Diansappe
and
Rnbrral Pramlppr
Ape
Protene (ON0EET--SO
Prrk
ligh lingg
Aarty
2
Preteiding 1,,,, puantae--ial -0--2
Mest Relevan Prenidppe
Intal
Prarrpe
2014
4.01,4
2004
5.9999
Suny de Exeecce go 2320
Must Bolncad Prunaype
2/1
Must Bolovse Pootarpe Pair
4,99 1. dssiuss serog "ya 50 2004
Paokeed Age
11.9
200e
Figure 18: Example reasoning processes of how age is predicted by ProtoPairNet for 12 random input images.
J More Examples of Global Analysis for Age Prediction
This section provides additional examples illustrating the global analysis for age prediction. Figure shows the global analysis for 8 pairs of prototypes used in various ProtoPairNet models trained.
33

---

Mext Prstdrpe
Pverree
Prodend jucant ais 2010
belenat Pra
Neit Relense Probitpe
Fadd 2ol
Tredland 30 177-
-0 23 5930
Moe Reterant Pronab
Mar Belenna Franbgr
leg lage
4/M ei dyy
Agr 4y54 lann
How Rekrree Pranobyp
Raiorat Francypu
Aookead may 5,00
223
2647
4-111
20
Nest Kein un Ervecppe Fair
Poonatyae Fai
Prrkrype
Fractpel
Figure 19: Example reasoning processes of how age is predicted by ProtoPairNet for 12 random input images.
K More Examples of Reasoning Process for Car Racing
This section provides additional examples illustrating the reasoning process for car racing. Figure 21 and Figure 22 show the reasoning process of ProtoPairNet for predicting the steering of random input states. Additionally, Figure 23 shows the reasoning process for predicting acceleration (left column) and braking (right column).
34

---

Top inteerr
haaggeha
Instane line
Top Scceee lartanee perponetrpe par
Or Ordeelien
Oriiial ape
Apii
Top cheeri Repa lastanccs thegonetpp
to
haree
City aprcl
oge
Top A ckwer leea Imcanoo
betates to the Bodind
Priters
0080
Figure 20: Global analyses of pairs of prototypes used in ProtoPairNet.
More Examples of Global Analysis for Car Racing
This section provides additional examples illustrating the global analysis for car racing. Figure 24 shows the global analysis for prototype pairs for steering. Additionally, Figure (anreee]]soww the global analysis for pair of acceleration prototypes (left column) and pair of braking prototypes (right column) used by ProtoPairNet.
M Reproducibility and Empirical Computational Cost
We implemented our models using PyTorch and conducted all experiments on high-performance computing cluster using SLURM. Each experiment was run on a single NVIDIA A100 80GB PCIe GPU with CUDA version 12.3, using 2 CPU cores and 64 GB of memory. The full pipeline-includin prototype projection and fine-tuning-tood approximately 6 hours for age prediction (3 runs, batch size 256) and 3 hours for car racing (5 runs, batch size 128). Prototype projection added minimal computational overhead in both tasks. For comparison, HPN (4 took approximately 1.5 days (36 hours) to complete 3 runs of 250 epochs using their default configuration, as reported in their paper. INSightR-Ne Net [16] required approximately hours, while the ExPeRT 17 required just under 2.5 hours to train under similar conditions.
N
Broader Impacts
Model interpretability is essential for the deployment of artificial intelligence (AI) systems in high- stakes environments. This work advances interpretability for tasks requiring continuous target predictions by leveraging prototypical pairs that ground predictions in semantically meaningful examples. By structuring predictions around these visual anchors, our method enables users to understand not only what the model predicts, but also why. Our approach supports applications where trust, transparency, and accountability are crucial, such as medical decision support, autonomous driving, and human-centered AI, and offers path toward more interpretable and human- aliggndd Al systems.
35

---

West
Mast Betroal
Pretated Bante
Relernt Fronetge
Mai Reincal Presstpp
og
Sasligh, LH
Frenoo R.
Bekeral Treeegg BUNTRE
Bakerat Prataagg
MMp.
Myys
Jebouat Presigge
Par storing
Suatfic Goil
Proelond Seerieg HSD
Prodinal
Selecat Trestypa curring
Por sitriigg
Fall
Predane
Bootiidt 20122-111
may- HUM +0,0 muur)
ale
AM6
Nas
Fai stortig
Pretivel Seeriay MJM HJD 1ayi)
Prockos Mariing
Sertiic
Figure 21: Example reasoning processes of how steering is predicted by ProtoPainNet for 12 random input states.
36

---

Pae deniigg
Tanape South des
Sowning by
Pridant
Frentapr Pair weertigg
Fatt for rivtin
Secliig
Sharngg P.
(134-4.)
Xlewiig
Relevnn theieeppe
Meit Beleved Preserpe
Fal muttte
hamee X
Priluted
-. 14 (0y)))
suB
255
Figure 22: Example reasoning processes of how steering is predicted by ProtoPairNet for 6 random input states.
(a)
Mhat Nekeaan
Fax anokeriiee
[b]
Rokeran brakiin
Podoad
Pradind
14,04,44.0
Acaicuries u.t9
Briking 0.11
20.1
Mait Releven Proewtype ferterading
Heteven Fretetgg: Poir cseolratio
Fredited Bradin 16,76,44,
Amsooiinee Froeleed knckeeate
maye (6,0
Deligo 004
1-15
Figure 23: Example reasoning processes of how acceleration is predicted for 2 random input states, left column (a), and how brake is predicted for 2 random input states, right column (b) by ProtoPainNet.
37

---

Top 3 dioed lepie Intanie pritttpe
Top dioek lnlance pretatypepa pax
Uniped wertr
Top 3 deser tapar Seetinn pestaope pair
Figure 24: Global analyses of pairs of prototypes used by ProtoPairNet for steering in car racing.
Top dleeest Lnpur instanes the pastoopeear
Zypaeppn C.
Ooggiaal Prodical
Inai
Figure 25: Global analyses of a pair of prototypes for acceleration, left column (a), and pair for braking used by ProtoPairNet for car racing.
38


Reviews:
[{"review_id": "TRy0kpY9YY", "reviewer": "Reviewer_uvMS", "summary": "This paper introduces ProtoPairNet, a novel interpretable architecture that combines deep learning with case-based reasoning to predict continuous targets, extending prototype-based models from solving classification tasks with discrete outputs to continuous targets such as regression tasks. \nSpecifically, ProtoPairNet leverages prototypical pairs as ProtoPair axis in the latent space: given an input, it makes predictions based on the relative positioning along the ProtoPair axis defined by its most relevant prototype pair. The architecture comprises 1) a prediction branch that uses geometric projection onto axis between prototypical pairs, and 2) a pair relevance branch that selects the most relevant axis for a given sample via Gumbel-softmax. \nProtoPairNet is evaluated on two domains: age prediction (a supervised learning task) and car racing (a behavioral cloning task with a reinforcement learning expert), achieving performance competitive with black-box baselines, while providing transparent, case-based reasoning for predictions.", "questions": "1.\tGeneralization Beyond Vision: Have the authors considered testing ProtoPairNet on non-visual regression tasks (e.g., tabular, timeseries)? If so, what difficulties arise, and how can pair-axes be made interpretable in domains without natural prototypes?\n2.\tOut-of-Distribution Reasoning: How does the model handle inputs whose latent representation falls far outside any prototype pair axis? Is there any mechanism for cases when the prototype pairs are insufficient to cover the data manifold?\n3.\tPotential Failure Modes: This paper would benefit from a more thorough discussion and analysis of potential failure modes.\n4.\tBaseline Clarification: The description of baseline in Table 1 is not sufficiently clear.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Quality\n1. The paper provides very concrete and detailed mathematical derivations as proof, demonstrating strong theoretical support for the proposed method.\n2. The paper provides very concrete and detailed mathematical derivations as proof, demonstrating strong theoretical support for the proposed method.\n3. The approach is rigorous, including a three-stage training process and a non-trivial prototype projection algorithm to ensure semantic and geometrical coherence.\n-Clarity\n1. The paper is well written, with clear and thorough explanations.\n2. All figures and tables contribute positively to readers’ understanding.\n- Significance\n1. The paper tackles an important problem, namely extending prototype-based models from discrete classification to interpretable regression tasks using reasoning-based approaches.\n- Originality\n1. The model architecture and training paradigm can be considered novel.\n\nWeaknesses\n- Quality\n1. The experimental results, while competitive, are only evaluated on a limited set of regression tasks (face age, car racing in paper). Vision is heavily emphasized, no \"classic\" tabular regression or high-dimensional time-series regression tasks are included.\n2. There is a lack of discussion of potential failure modes or cases where one-to-one or traditional regression models might outperform (e.g., highly nonlinear target relationships, out-of-distribution scenarios).\n\n- Clarity\n1. Some descriptions, such as the prototype projection (Section 2.4), are mathematically dense and could be made more accessible, 2. The description of baselines in Table 1 is not sufficiently clear.\n- Significance\n1. While user studies are carefully executed, all evaluation hinges on just two datasets. It is not fully clear how well this method generalizes to hard regression tasks (e.g., non-vision or highly nonlinear problems).\n2. ProtoPairNet’s interpretability hinges on whether its selected prototype pairs are semantically coherent and representative. If the nearest prototype pair for an input does not correspond to a meaningful axis or concept in the data space, the resulting explanation may lack clarity and fail to provide useful human insight.\n- Originality\n1. Underlying notions are well established, though the extension to pairs and projection-based reasoning is new.", "was_revised": true, "final_justification": "Most of my concerns have now been addressed. I believe that incorporating your responses from the rebuttal into the main paper would strengthen the overall quality and clarity of the work. Therefore, I will retain my original score.", "rebuttal": "- Limited benchmarking and generalization beyond vision (Quality weakness 1, Significance weakness 1, Question 1): \n\nThank you for the suggestion! To further evaluate the generalizability of ProtoPairNet, we have expanded our benchmarking to include two additional tasks across diverse domains:\n\nTabular regression: Following your suggestion, we added the California housing prices dataset [1], a classic tabular regression task with continuous outputs and heterogeneous input features. For the black-box baseline, we used a simple fully-connected network with three hidden layers (128, 64, 32 units), each followed by ReLU and 10% dropout. The final layer outputs a single value for predicting house prices.\n\nMedical imaging: We applied ProtoPairNet to the APTOS dataset [2] for predicting the severity of diabetic retinopathy from retinal fundus images. This benchmark presents both high-dimensional input (3×224×224) and clinically relevant labels. For the baseline model we used ResNet50.\n\n| Dataset            | Model               | MAE               | R² score         |\n|--------------------|---------------------|-------------------|------------------|\n| **California housing** | Baseline (No prototypes) | 0.320 ± 0.001      | 0.811 ± 0.004     |\n|                    | HPN                  | 0.275 ± 0.0004     | 0.816 ± 0.001     |\n|                    | ProtoPairNet (ours)        | 0.289 ± 0.003      | 0.796 ± 0.004     |\n| **APTOS**          | Baseline (No prototypes) | 0.301 ± 0.008      | 0.825 ± 0.009     |\n|                    | HPN                  | 0.360 ± 0.007      | 0.728 ± 0.006     |\n|                    | ProtoPairNet (ours)        | 0.278 ± 0.028      | 0.837 ± 0.010     |\n\nIn a tabular domain such as the California housing dataset, we can interpret prototype pairs through their feature values and treat the pair-axes as a semantic contrast between representative conditions (e.g., high vs. low income regions in tabular data). While direct visualization is not possible, these axes can still be interpretable through feature-level analysis or domain-specific context.\n\nWhile ProtoPairNet was primarily designed for vision-based regression tasks, our additional benchmark on the tabular California housing data demonstrates that our ProtoPairNet is more flexible and generalizable than originally intended.\n\nAs shown in the table above, ProtoPairNet performs comparably to baselines in the two additional benchmarks. We hope the new experiments satisfy the requirement of broader evaluation and generalization beyond vision.\n\n- Out-of-distribution reasoning (Question 2):\n\nInputs whose latent representations fall far outside any ProtoPair axis indeed raise important considerations with regard to out-of-distribution (OOD) reasoning, but the challenge of OOD samples is not unique to our model – it is a fundamental limitation faced by virtually all machine learning systems. No model can be expected to reliably reason about regions of the data space that are sparsely populated or unsupported by training examples. In this sense, the concern applies broadly and is not specific to our approach.\n\nHowever, ProtoPairNet is particularly well-positioned to address this issue. Because its predictions are based on distances to prototype pairs, the model provides a direct and interpretable signal for when an input lies far from any axis of comparison. This allows for a natural mechanism for out-of-distribution (OOD) detection: by measuring the distance from a test point to its nearest ProtoPair axis and flagging cases where this distance exceeds a threshold, we can identify inputs that fall outside the region where the model is expected to reason reliably. While we have not yet implemented such an OOD detection module, the structure of ProtoPairNet makes it amenable to this kind of extension, offering a promising path toward increased robustness and interpretability.\n\n- Potential failure modes (Quality weakness 2, Question 3):\n\nThank you for the suggestion. We would include a more thorough discussion of potential failure modes in our final paper. Here are some potential failure modes of our ProtoPairNet model:\n\n1. Out-of-distribution inputs: As previously discussed, if an input's latent representation falls far from all prototype pair axes, the model may produce unreliable outputs. Without explicit mechanisms to detect or adapt to such inputs, the reasoning based on prototype pairs may fail. However, we can use a distance-based threshold to flag such cases and consider this an important direction for future development.\n\n2. Insufficient prototype coverage: If the learned prototype pairs fail to span the full diversity of the data manifold, the model may generalize poorly to underrepresented regions. Learning a sufficient number of diverse prototype pairs should address this issue. In our experiments, we determined the ideal number of prototype pairs for each task using a validation set.  \n\n- Clarity (Clarity weakness 1, Question 4):\n\nWe will describe prototype projection (Section 2.4) and the baseline models in Table 1 more clearly in our final paper.\n\n- Semantic coherence of prototype pairs (Significance weakness 2):\n\nWe agree that the quality of explanations depends on whether the selected prototype pairs form meaningful, interpretable axes. To encourage this, ProtoPairNet is explicitly designed to learn semantically useful latent axes through training, ground each prototype pair in real training examples, and select the most relevant pair at inference time using Gumbel-softmax, which focuses reasoning on a single interpretable dimension. To validate the semantic coherence of prototype pairs, we conducted a user study (Study 1) where participants were asked to choose which of two prototype pairs was more helpful for the age prediction task. The results show that users largely agree with the model’s prototype-pair selection, suggesting that the axes learned by ProtoPairNet often align with human-understandable concepts.\n\n- Originality (Originality weakness 1):\n\nWe agree that ProtoPairNet builds on well-established ideas in prototype-based learning and interpretable modeling. However, most prior work in this space has focused almost exclusively on classification, where similarity to a single prototype can directly inform discrete label decisions. In contrast, regression remains relatively underexplored in prototype-based learning, despite its widespread relevance. Our main contribution is to extend prototype reasoning to regression by introducing prototype pairs and projection-based reasoning. This allows our ProtoPairNet to reason about where an input lies along a semantic axis defined by two prototypes with continuous labels. It addresses key challenges that arise in regression, such as ambiguity in single-prototype similarity and interference from unrelated prototypes, which are not effectively handled by existing prototype-based classification methods.\n\n[1] Pace, R. K., & Barry, R. (1997). Sparse spatial autoregressions. Statistics & Probability Letters, 33(3), 291-297.\n\n[2] Kaggle (2019). APTOS 2019 blindness detection. Kaggle."}, {"review_id": "hVFt1QmD2x", "reviewer": "Reviewer_tFYK", "summary": "This paper introduces ProtoPairNet, a prototype-based framework for interpretable regression. Unlike traditional prototype models that rely on one-to-one comparisons, ProtoPairNet compares inputs to pairs of prototypes, enabling predictions based on the input’s position along the axis between two reference examples. This approach provides directional information needed for accurate continuous predictions. The model also uses a relevance mechanism to select the most appropriate prototype pair for each input, ensuring sparse and interpretable decision-making. Additionally, the authors propose a three-stage training strategy, including a projection of prototypes to real data samples for better visualization. ProtoPairNet is evaluated on age prediction and reinforcement learning-based control tasks, demonstrating competitive performance and improved interpretability over prior methods, as confirmed by multiple user studies.", "questions": "1. Table 2: ProtoPariNet seems to significantly outperform other architectures. Are those SOTA architectures, and were they fully fine-tuned for this task?\n2. Could you propose more complex benchmark for SOTA comparison? Age prediction seems not sufficient to decide the generalizability of this architecture.\n3. MAE and R^2 seem like a limited performance metrics. It would be nice to see a distribution of prediction error, or more exhaustive metrics that describe it (along with comparison to other models).\n4. Would be nice to see some attribution method (like LRP or Integrated Gradients) to justify the explanation reasoning (Is the model really focusing on age, or maybe something else?)", "limitations": "The authors have noted the limitation concerning global prototypes rather than prototype-part based ones (or attribution-based). However, my concern lies with the benchmarking, specifically the datasets and evaluation metrics employed, as well as the generalizability of the approach to more complex scenarios.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Interesting approach for prototypical regression: Prototype-based models have been widely used for classification, but very few extend to regression. ProtoPairNet addresses this by introducing a structured, interpretable framework for continuous prediction tasks. This makes it a novel and valuable contribution to the field of explainable AI.\n- Interpretable Prototypes: Prototypes in ProtoPairNet are projected onto actual training instances, making them easy to visualize and understand. Unlike abstract latent vectors, these prototypes can be shown as real images. This adds transparency and helps users intuitively understand how the model arrived at its prediction.\n- Backbone compatibility: ProtoPairNet is compatible with various standard architectures like ResNet, EfficientNet, and Vision Transformers. This flexibility makes it easy to apply to various domains without redesigning the core model from scratch. This modular design also allows for future extensions.\n- Selection mechanism for explanation clarity: ProtoPairNet uses a selection mechanism to select the most relevant prototype pair for each prediction. This reduces noisy explanations and ensures that the reasoning remains focused and interpretable. \n\nWeaknesses\n- Limited benchmarking: ProtoPairNet is only tested on two tasks: age prediction and a car racing control task. These are relatively constrained problems that may not reflect the challenges of more complex, noisy, or high-dimensional regression domains. Without broader evaluation, the model’s generalizability remains uncertain.\n- Global explanations: Unlike part-prototype models that highlight specific input regions (e.g. facial features), ProtoPairNet uses global prototypes (entire images). This means it cannot indicate which parts of an input were most important for the prediction. As a result, it lacks fine-grained, spatial explanations, which limits its usefulness and interpretability, as we cannot be sure which features are contributing to the prototypical reasoning. It also makes the interpretation subjective to the user, which might be misaligned with how the model really works.\n- Limited use of multiple pairs per prediction: The model predicts using only one prototype pair per input. While this improves interpretability, it may not capture all relevant aspects of the input when multiple semantic factors are relevant. This reasoning could reduce prediction quality in more complex scenarios.", "was_revised": true, "final_justification": "The paper presents an interesting and to some degree interpretable approach to regression using prototype pairs, offering a fresh perspective on explainable AI for continuous prediction tasks. The authors strengthened the submission by additional evaluations. However, limitations remain in the reliance on global prototypes without built-in fine-grained attribution and the mixed performance across domains, justifying a borderline accept.", "rebuttal": "- Limited benchmarking and more complex benchmark (Weakness 1, Question 2, Limitations):\n\nThank you for highlighting this important point. While our initial experiments focused on age prediction and car racing, it is worth noting that both tasks involve high-dimensional visual inputs – 3×128×128 images for age prediction and 4×96×96 stacked frames for CarRacing – making them non-trivial and representative of real-world perceptual challenges. To further evaluate the generalizability of ProtoPairNet, we have expanded our benchmarking to include two additional tasks across diverse domains:\n\nMedical imaging: We applied ProtoPairNet to the APTOS dataset [1] for predicting the severity of diabetic retinopathy from retinal fundus images. This benchmark presents both high-dimensional input (3×224×224) and clinically relevant labels. For the baseline model we used ResNet50.\n\nTabular regression: We applied ProtoPairNet to the California housing dataset [2], a classic tabular regression task with 8-dimensional inputs and continuous outputs. The baseline is a fully connected network with three hidden layers (128, 64, 32 units), ReLU activations, 10% dropout, and a single output neuron for predicting house prices.\n\n\n| Dataset            | Model               | MAE               | R² score         |\n|--------------------|---------------------|-------------------|------------------|\n| **APTOS**          | Baseline (No prototypes) | 0.301 ± 0.008      | 0.825 ± 0.009     |\n|                    | HPN                  | 0.360 ± 0.007      | 0.728 ± 0.006     |\n|                    | ProtoPairNet (ours)        | 0.278 ± 0.028      | 0.837 ± 0.010     |\n| **California housing** | Baseline (No prototypes) | 0.320 ± 0.001      | 0.811 ± 0.004     |\n|                    | HPN                  | 0.275 ± 0.0004     | 0.816 ± 0.001     |\n|                    | ProtoPairNet (ours)        | 0.289 ± 0.003      | 0.796 ± 0.004     |\n\nAs shown above, ProtoPairNet performs comparably to baselines in these new benchmarks. We hope these new results address the request for more benchmarks.\n\n- Global explanations and attribution method (Weakness 2, Question 4):\n\nThank you for the thoughtful comment. Prior work on prototype-based deep neural networks indeed spans two major approaches: using prototypes defined at the global (entire image) level [3, 4] and those defined at the local (patch or part) level [5]. Our choice to adopt global prototypes aligns with the former line of work and is a deliberate design decision, primarily due to the nature of our data and labels. Specifically, for age predictions, learning semantically meaningful part prototypes can be challenging, since a 50-year-old may have facial features that resemble someone younger, making part-based prototypes less reliable.\n\nThat said, we fully agree with the concern about the lack of fine-grained spatial attribution. While ProtoPairNet’s global prototypes naturally provide semantic interpretability at the image level (e.g., \"this face resembles an older person\"), they do not by themselves reveal which parts of the input contributed most to the prediction. Fortunately, this limitation can be addressed via post-hoc attribution methods. For instance, occlusion-based sensitivity analysis [6] or region perturbation methods [7] can help localize which input regions most influence the similarity (or dissimilarity) to a given prototype pair. This approach has precedent: Ragodos et al. [4] also use global prototypes (entire game frames) but apply post-hoc attribution to identify salient regions that drive prototype similarity.\n\nIn our setting, to identify which input region a prototype pair attends to for a given input image, we sequentially masked input patches and measured the change in its prediction as an importance score. This patch occlusion method, though not built into the core architecture, provides spatial explanations without requiring unreliable part-level supervision. We analyzed the top-10 most important patches across 5 randomly chosen test images and found that high-impact regions consistently fall within rows 2–6 and columns 2–6 of the 8×8 patch grid—corresponding to key facial landmarks such as the eyes, nose, lips, and jawline. This suggests the model consistently focuses on semantically meaningful features.\n\nQuantitatively, as shown in the table below, the average change in prediction across occluded patches ranged from 0.38 to 1.07, with maximum changes per image between 1.82 and 4.40, confirming that these regions exert a significantly greater influence on the model’s output. In all cases, these regions corresponded to one of the facial landmarks.\n\n| Test image   | Average prediction change | Max prediction change |\n|---------------|----------------|-------------|\n| Test image 1  | 0.525          | 2.737       |\n| Test image 2  | 0.773          | 4.153       |\n| Test image 3  | 0.378          | 1.820       |\n| Test image 4  | 1.071          | 4.400       |\n| Test image 5  | 0.884          | 3.438       |\n\n- Limited use of multiple pairs per prediction (Weakness 3):\n\nThank you for the insightful comment. We agree that relying on a single prototype pair may limit expressiveness when multiple semantic factors are relevant. However, this design choice enhances interpretability by encouraging reasoning along a single, identifiable axis.\n\nTo evaluate the trade-off, we conducted an ablation study in which we replaced the Gumbel-softmax-based prototype pair relevance branch with (1) a standard softmax and (2) a simple average of all prototype pairs’ predictions. For a given input, the standard softmax weights each prototype pair’s prediction proportional to the negative exponent of the distance between the input’s latent representation to the pair’s axis, while a simple average simply outputs the arithmetic average of all prototype pairs’ predictions. In both cases, the predicted value uses more than one prototype pair. We found that using multiple pairs did not significantly improve performance across our benchmarks, suggesting that a single pair often captures the dominant factor driving the prediction (see the table below).\n\nThis observation is also consistent with findings from prior work [8], which showed that projecting to a single prototype, rather than a weighted combination of multiple prototypes, has minimal impact on performance (see Table 9 of that paper). \n\nHence, we view the use of the most relevant pair not necessarily as a disadvantage, since it maintains predictive performance and at the same time improves interpretability.\n\n| ProtoPairNet Predictions | MAE             | R² score         |\n|--------------------------|----------------|------------------|\n| Standard Softmax         | 4.49 ± 0.007    | 0.877 ± 0.0006   |\n| Simple Average           | 4.48 ± 0.003    | 0.878 ± 0.0004   |\n| Gumbel Softmax (ours)| 4.52 ± 0.010    | 0.876 ± 0.0009   |\n\n- Comparison with other architectures in Table 2 (Question 1):\n\nYes, the baseline architectures we compared against are state-of-the-art visual regression models, and were fully fine-tuned for each regression task using standard training protocols and hyperparameter sweeps to ensure fair comparison.\n\nIt is important to note that, unlike classification, there are relatively few prototype-based architectures specifically designed for visual regression tasks. Our ProtoPairNet is designed specifically for prototype-based regression, and its performance gains reflect the strength of using prototype-pair reasoning in these settings.\n\n- Performance evaluation metrics (Question 3, Limitations):\n\nThank you for the suggestion. For the age prediction task, we did an analysis of the distribution of mean average errors over different age groups using the ResNet50-based ProtoPairNet and the standard (black-box) ResNet50 models. The results are summarized in a table.\n\n| Age group   | ProtoPairNet MAE (ours)   | ResNet50 MAE        | HPN MAE            | Count |\n|-------------|----------------------|----------------------|---------------------|--------|\n| [0, 15)     | 2.06 ± 0.14  | 1.93 ± 0.03  | 2.57 ± 0.44  | 366    |\n| [15, 30)    | 3.52 ± 0.17  | 3.48 ± 0.16  | 4.36 ± 0.37   | 830    |\n| [30, 45)    | 4.80 ± 0.15  | 5.46 ± 0.05 | 5.67 ± 0.54   | 570    |\n| [45, 60)    | 7.14 ± 0.33  | 7.60 ± 0.04 | 7.43 ± 0.38  | 335    |\n| [60, 75)    | 7.37 ± 0.77 | 6.83 ± 0.18  | 7.52 ± 0.06   | 169    |\n| [75, 90)    | 10.26 ± 1.32  | 10.18 ± 0.43  | 10.80 ± 1.10 | 57     |\n\nAs shown in the above table, the ResNet50-based ProtoPairNet performs better than the black-box ResNet50 for two age groups [30, 45) and [45, 60), and performs similarly for the other age groups.\n\n[1] Kaggle (2019). APTOS 2019 blindness detection. Kaggle.\n\n[2] Pace, R. K., & Barry, R. (1997). Sparse spatial autoregressions. Statistics & Probability Letters, 33(3), 291-297.\n\n[3] Kenny, E. M., Tucker, M., & Shah, J. (2023). Towards interpretable deep reinforcement learning with human-friendly prototypes. International Conference on Learning Representations.\n\n[4] Ragodos, R., Wang, T., Lin, Q., & Zhou, X. (2022). ProtoX: Explaining a reinforcement learning agent via prototyping. Advances in Neural Information Processing Systems, 35, 27239-27252.\n\n[5] Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., & Su, J. K. (2019). This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32.\n\n[6] Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. European Conference on Computer Vision.\n\n[7] Petsiuk, V., Das, A., & Saenko, K. (2018) RISE: Randomized Input Sampling for Explanation of Black-box Models. British Machine Vision Conference.\n\n[8] Hong, D., Wang, T., & Baek, S. (2023). Protorynet-interpretable text classification via prototype trajectories. Journal of Machine Learning Research, 24(264), 1-39."}, {"review_id": "4UBKC1GVS1", "reviewer": "Reviewer_gJBu", "summary": "This paper proposes **Prototypical Pair Network (ProtoPairNet)**, a novel model belonging to the class of prototype-based models, which offers the promise of interpretability versus standard black-box models.\n\nProtoPairNet tackles the regression problem instead of the classification problem, which has been the dominant focus of the community. While some prior works have explored regression with prototype-based models, this paper argues that those approaches suffer from one of the following limitations: (1) they rely on similarity-weighted averaging, which introduces interference from unrelated prototypes, or (2) they sacrifice interpretability.\n\nTo address this, the paper introduces two key innovations:\n\n(1) Instead of using single instances as prototypes, the model uses **pairs** of instances, such that the regression output varies linearly along the line connecting the two instances.\n\n(2) Given a test input, the model computes relevance scores with respect to a library of prototype pairs, selects one using a soft mechanism (Gumbel-Softmax), and applies a linear interpolation formula to compute the output from the labels of the selected prototype pair.\n\nIn experiments, the method performs competitively with black-box models and outperforms other previous prototype-based models. User studies also suggest that the model’s reasoning process aligns with that of human participants.", "questions": "## Questions\n\nQ. Why is now a timely moment for this paper? All the ingredients motivating this work seem to have existed already a few years ago.\n\nQ. How are prototype pairs decided? How is the pairing done?\n\nQ. I tend to believe that interpretability is usually gained at the expense of flexibility, so one would expect prototype models (which are interpretable) to perform worse than the black-box models -- which isn't a bad thing, but just something I would expect. However, Table 1 suggests the opposite, and also, it appears that ProtoPairNet even outperforms black-box PPO. I was wondering if the authors have an explanation about this? Would including additional benchmarks help here?\n\nQ. Have the authors tried using any pre-trained backbones?\n\nQ. If the perpendicular drop from the input latent to the prototype axis lies outside the segment formed by the prototype pair, how is this case handled?\n\n## Suggestions for Improvement\n\nAn ablation of the three “metrics” and the various loss terms would be informative.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Pros\n\n1. The paper is well-written and easy to follow.\n2. The paper is appropriately scoped.\n3. The idea of using prototype pairs is novel.\n\n## Cons\n\n1. The motivation for using pairs versus single prototypes is somewhat unclear. The issue of interference from irrelevant prototypes could be addressed using Gumbel-Softmax alone. So, what necessitates the use of *pairs* of prototypes?\n2. If the intent is to increase expressiveness, it would be nice to have an ablation that varies the tuple size from 1 to 2 and to perhaps even higher-order tuples. Additionally, the following aspects are unclear to me:\n    \n    (1) How are the pairs chosen to form the prototypes?\n    \n    (2) Are these prototypes decided during training, or are they provided at test time?\n    \n    (3) Does the method explicitly enforce the “linearity” of the target function with respect to the latent representations of the prototype pairs?\n    \n\nSee also the “Questions”.", "was_revised": true, "final_justification": "My main initial concerns were around things that were unclear to me (mainly because of their writing quality in my opinion), e.g., it was not quite clear if and how prototypes were decided/learned. Even though I understand these points better now after reading the rebuttal response, and therefore I am increasing the score, I am broadly concerned that the quality of writing in the paper could be made cleaner if the community has to derive the full value out of this paper.", "rebuttal": "- Motivation for using pairs (Cons 1):\n\nWhile Gumbel-softmax helps with sparse prototype selection, it does not address the core limitation of using single prototypes for regression. Specifically, a one-to-one comparison between an input and a single prototype only tells us that the input is “similar” to that prototype, and therefore its label should be similar. However, in regression, this is not sufficient – we often need to know in which direction the output should differ (i.e., should it be greater or smaller than the prototype's label). As shown in Figure 1 of our paper, reasoning along a prototype pair provides this directional context: the model can predict where along the axis between two known values the input lies. This cannot be achieved with reasoning using single prototypes.\n\n- Ablation study for varying number of prototype pairs (Cons 2):\n\nBelow, we include an ablation study on the number of prototype pairs used in ProtoPairNet. Results for 5 and more pairs are also provided in Appendix E. As shown in the table, performance differences are minimal for 5 pairs or fewer, but we chose 5 pairs to allow greater expressiveness. Compared to using only 1 or 2 pairs, having 5 allows the model to represent a wider range of prototypical variations (e.g., across age groups or demographics), making the reasoning behind predictions more accessible to human understanding.\n\n| Number of prototype pairs | MAE           | R² score      |\n|-----------------------------|---------------|---------------|\n| 1 pair  | 4.50 ± 0.02  | 0.877 ± 0.002 |\n| 2 pairs | 4.51 ± 0.01  | 0.878 ± 0.0005|\n| 5 pairs | 4.52 ± 0.01  | 0.876 ± 0.0009|\n| 10 pairs | 4.85 ± 0.18  | 0.86 ± 0.004|\n| 20 pairs | 4.89 ± 0.15  | 0.86 ± 0.007|\n| 40 pairs | 5.24 ± 0.04  | 0.84 ± 0.001|\n| 50 pairs | 5.58 ± 0.50  | 0.83 ± 0.019|\n\n- How are the prototype pairs chosen? Are they chosen during training? How is the pairing done? (Cons 2(1), Cons 2(2), Question 2)\n\nPrototype pairs are selected during Stage 2 of our training algorithm (see pages 6–7 of the paper). Before this stage, prototype pairs exist as abstract vectors in the latent space and are not tied to specific training examples. In Stage 2, we replace each abstract prototype pair with a pair of training images chosen based on the following three criteria:\n\n(i) Cosine similarity – the candidate pair should define an axis closely aligned with the original abstract prototype pair;\n\n(ii) Diversity – the two images in the pair should be sufficiently far apart in the latent space to capture meaningful variation;\n\n(iii) Average axis distance – the candidate pair’s axis should not deviate too much from the current prototype pair axis.\n\nFor each prototype pair, the algorithm searches the training set for a pair of images that maximizes cosine similarity (alignment) while satisfying minimum diversity and maximum average axis distance constraints. Once selected, we update the prototype pair to the latent representations of the chosen images and assign their ground-truth labels as the new prototypical labels. More details can be found in Section 2.4 of the paper.\n\n- Does the method enforce “linearity” of the target function with respect to the latent representations of the prototype pairs? (Cons 2(3))\n\nYes, our method explicitly enforces the linearity with respect to the latent representations of the prototype pairs. This is achieved by the design of the prediction branch of our model (described in Section 2.2 of our paper), and the minimization of the mean absolute/square error on a training dataset, which ensures that the latent representations of training instances have to vary linearly with respect to the axis defined by their closest prototype pairs in the latent space.\n\n- Why is now a timely moment for this paper? (Question 1)\n\nWhile it is true that the core ingredients of prototype-based learning have been available for several years, their application has remained largely restricted to classification problems [1–5] or reinforcement learning problems with discrete action spaces [6].\n\nHowever, a noticeable and important gap remains: prototype-based reasoning has not been meaningfully extended to regression tasks. Unlike classification, where the decision boundaries between classes can be cleanly interpreted via proximity to class prototypes, regression requires a continuous output space and thus demands a fundamentally different formulation of how prototypes are used for prediction. To our knowledge, very few (if any) prior works have addressed this challenge, and those that do have shown weak performance or lacked interpretability (see Table 2 of our paper).\n\nOur work closes this gap by introducing ProtoPairNet, which formulates prototype-based reasoning as comparative rather than absolute—shifting from matching to a single prototype to reasoning based on the relative position between pairs of prototypes. This approach is naturally suited to regression, where inputs can be interpreted as \"more like this than that\" along a continuous axis. In this sense, now is a timely moment for this work: the community has matured in its understanding of prototype-based interpretability for classification, and the need to extend these benefits to regression is both technically open and practically valuable.\n\n- Do the authors have an explanation for how prototype models performed similarly or even better than the black-box baselines? Would including additional benchmarks help here? (Question 3)\n\nWhile it is commonly assumed that interpretability comes at the cost of predictive performance, recent research suggests that this tradeoff is not inevitable. As argued in [7], interpretable models can perform on par with black-box models for many real-world tasks. This is because many problems admit a Rashomon set – a large collection of diverse models that achieve similarly high accuracy [8]. Black-box models often find one such solution, but interpretable models can also lie within this set; the difficulty lies not in their existence, but in finding them. Empirical evidence supporting this claim appears in existing prototype-based models [1–6], all of which achieved competitive accuracy while maintaining interpretability.\n\nTo further evaluate the generalizability of ProtoPairNet, we have expanded our benchmarking to include additional tasks across diverse domains:\n\nMedical imaging: We applied ProtoPairNet to the APTOS dataset [9] for predicting the severity of diabetic retinopathy from retinal fundus images. This benchmark presents both high-dimensional input (3×224×224) and clinically relevant labels. For the baseline model we used ResNet50.\n\nTabular regression: We added the California housing prices dataset [10], a classic tabular regression task with continuous outputs and heterogeneous input features. For the black-box baseline, we used a simple fully-connected network with three hidden layers (128, 64, 32 units), each followed by ReLU and 10% dropout. The final layer outputs a single value for predicting house prices.\n\n| Dataset            | Model               | MAE               | R² score         |\n|--------------------|---------------------|-------------------|------------------|\n| **APTOS**          | Baseline (No prototypes) | 0.301 ± 0.008      | 0.825 ± 0.009     |\n|                    | HPN                  | 0.360 ± 0.007      | 0.728 ± 0.006     |\n|                    | ProtoPairNet (ours)        | 0.278 ± 0.028      | 0.837 ± 0.010     |\n| **California housing** | Baseline (No prototypes) | 0.320 ± 0.001      | 0.811 ± 0.004     |\n|                    | HPN                  | 0.275 ± 0.0004     | 0.816 ± 0.001     |\n|                    | ProtoPairNet (ours)        | 0.289 ± 0.003      | 0.796 ± 0.004     |\n\nAs shown above, ProtoPairNet performs comparably to baselines in these new benchmarks. We hope these new results address the request for more benchmarks.\n\n- Have the authors tried using any pre-trained backbones? (Question 4)\n\nYes, our backbone models were pre-trained on ImageNet.\n\n- If the perpendicular drop from the input latent to the prototype axis lies outside the segment formed by the prototype pair, how is this case handled? (Question 5)\n\nEquation (4), which defines prediction from a prototype pair, remains valid even when the perpendicular projection from the input latent to the prototype axis falls outside the prototype segment. As shown in Figures 17 and 18 (Appendix I), ProtoPairNet can predict ages beyond those of the most relevant prototype pair. For example, in Figure 17 (bottom-left), the model predicts age 11.49 for an input of age 12, despite the closest prototype pair having ages 16 and 35. Here, the projected input lies to the left of the prototype segment, which our ProtoPairNet handles elegantly.\n\n[1] Chen, C. et al. (2019). This looks like that: deep learning for interpretable image recognition. NeurIPS.\n\n[2] Nauta, M. et al. (2021). Neural prototype trees for interpretable fine-grained image recognition. CVPR.\n\n[3] Rymarczyk, D. et al. (2022). Interpretable image classification with differentiable prototypes assignment. ECCV.\n\n[4] Ma, C. et al. (2023). This looks like those: Illuminating prototypical concepts using multiple visualizations. NeurIPS.\n\n[5] Ma, C. et al. (2024). Interpretable image classification with adaptive prototype-based vision transformers. NeurIPS.\n\n[6] Ragodos, R. et al. (2022). ProtoX: Explaining a reinforcement learning agent via prototyping. NeurIPS.\n\n[7] Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206-215.\n\n[8] Breiman, L. (2001). Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16(3), 199-231.\n\n[9] Kaggle (2019). APTOS 2019 blindness detection. Kaggle.\n\n[10] Pace, R. K., & Barry, R. (1997). Sparse spatial autoregressions. Statistics & Probability Letters, 33(3), 291-297."}, {"review_id": "lSVIgOU07S", "reviewer": "Reviewer_pNkw", "summary": "The authors propose ProtoPairNet as an interpretable approach to regression through prototypical pair reasoning. Experiments show the utility of the proposed approach in both computer vision and reinforcement learning settings. Several human studies are conducted that further highlight the benefits of the pair based approach.", "questions": "- Can you provide some more clarity on the assumptions of the prediction branch? It would be nice to know of scenarios when these assumptions would break so I can better understand the assumptions themselves. Would the labels not always vary linearly along the k-th ProtoPair axis?\n\n- Is one prototype from the pair always on either side of the projected input image? \n\n- What is the motivation for the diversity criterion? Do you not want the prototypes to be always similar to the input image? Was training quite difficult given the number of terms in the loss function?\n\n- What were the limitations of the user study? \n\n- Why do you not use a part-based framework, given the success of this approach in other works.", "limitations": "- Authors could go into more detail about the computational advantages or drawbacks of the proposed approach. There is related work on prototypes and criticisms, I also wonder if using maximum mean discrepancy would have advantages over the k-means based approach used here?\n\n- Not using a part-based framework is a notable weakness. The authors mention this as a weakness in the limitations but I think some rationale is needed here given that most developments in the field recognise the benefits of part-based approaches. \n\n- Authors mention code upon acceptance but I think it would actually have been useful to provide the anonymised code in the supplement or on one of the anonymous archives as this would give further clarity on methodological approaches. \n\n- User study design (see weaknesses)", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Prototype based approaches are uncommon in regression scenarios and solutions would be a welcome and useful contribution. Modernising case-based solutions are one of Rudin’s 10 Grand Challenges for interpretable machine learning.\n\nThe approaches and comparisons are reasonably well grounded in the literature. However, there is no discussion of seminal work on prototypes and criticisms [1]. This is especially relevant as the user study in this prior work also shows the benefits of using a prototype and another example (a criticism in the case of [1]). \n\nThere is a theoretical contribution for both prediction and pair relevance. The design is (for the most part) intuitive and I liked the geometric interpretation. \n\nWhile the inclusion of a user study is important, there are several and design questions and flaws. How was the number of participants determined? It does not seem like there was a power analysis conducted to determine this? It does not seem like the experiment was controlled for the number of items presented to users. Would it not have made more sense to compare a prototypical pair against either/both (i) a single prototype and a random prototype (ii) two random prototypes. The problem is that with the current design, the number of items presented to users is not fixed and could therefore be considered a confounding variable. Were there attention checks? This is typically considered best practice in user study design. The figures in the appendix that show the set-up (e.g., figure 11) should probably go in the main text for clarity. In fact, the appendix was quite a big document but it was rarely referred to in the main text. \n\n\n[1] Kim, B., Khanna, R. and Koyejo, O.O., 2016. Examples are not enough, learn to criticize! criticism for interpretability. Advances in neural information processing systems, 29.", "was_revised": true, "final_justification": "From the rebuttal the authors have improved the quality and clarity of the manuscript. I think this is a good paper offering a novel solution to a problem with limited coverage in the literature (prototypes for regression). One notable weakness in my opinion is in the user study.", "rebuttal": "- No discussion of Kim et al., 2016 (Strengths and Weaknesses paragraph 2):\n\nThank you for highlighting this important prior work. While Kim et al. (2016) [1] shares our motivation of improving interpretability through comparison, their prototype + criticism framework is designed for classification, not regression. In contrast, ProtoPairNet introduces prototype pairs to define semantic axes for continuous-valued prediction, enabling structured reasoning in regression tasks. We agree the work is related in spirit and will include a discussion of it in the final version.\n\n- User study design (Strengths and Weaknesses paragraph 4, Question 4, Limitation 4):\n\nWe chose a sample size of 30 participants based on practical considerations and methodological precedent. A sample of 30 is widely accepted as a minimum for the Central Limit Theorem to apply [2, 3].\n\nNumber of items shown to users: You are right that controlling the number of items shown is important to avoid confounds. However, our primary goal was to compare prototype pair reasoning with individual prototype-based explanations, which naturally led to different numbers of items being shown.\n\nThat said, in Study 1, we conducted a controlled comparison where users were shown two prototype pairs—one used by the model and one distractor—and asked to guess the target image’s age and choose the more helpful pair. Users largely agreed with the model’s choice, supporting the effectiveness of prototype-pair reasoning even under controlled conditions.\n\nAttention checks: Although we did not include explicit attention checks, we flagged and removed responses from users with abnormally short completion times and replaced them with new participants. In future studies, we plan to add dedicated attention checks to improve data quality.\n\n​​We agree that including study setup figures like Figure 11 in the main text would improve clarity, and we will try to incorporate them in the final version within page limits. We will also ensure better referencing of relevant appendix sections.\n\n- Can you provide more clarity on the assumptions of the prediction branch? (Question 1)\n\nThe prediction branch of ProtoPairNet relies on two key assumptions:\n\n1. The label is assumed to vary linearly along the axis defined by a prototype pair in latent space.\n\n2. Inputs with the same target value are assumed to lie on a common hyperplane orthogonal to the prototype pair axis.\n\nThese assumptions are directly built into Equation (4) of our paper, which defines the prediction made by a prototype pair. Importantly, the two assumptions are not imposed directly on the raw data, but rather on the latent space produced by the model’s backbone. A well-trained ProtoPairNet is optimized to shape the latent space such that these assumptions hold approximately true.\n\nIf the assumptions were violated – e.g., if labels were not linearly aligned with prototype axes – prediction accuracy would suffer, resulting in high MAE and poor R². Since ProtoPairNet achieves strong test performance, the assumptions appear to be satisfied in practice.\n\n- Is one prototype from the pair always on either side of the projected input image? (Question 2)\n\nNo. The input image's latent representation need not fall between both prototypes. In the bottom-left example of Figure 17 in Appendix I, the model predicts an age of 11.49 for an individual of age 12, using a pair of prototypes aged 16 and 35. Here, the input lies to the left of both prototypes in latent space. More such cases appear in Figures 17 and 18.\n\n- What is the motivation for the diversity criterion? Do you not want the prototypes to be always similar to the input image? Was training quite difficult given the number of terms in the loss function? (Question 3)\n\nThe diversity criterion ensures that each prototype pair captures a meaningful direction in the latent space. While some prototypes should resemble the input (via cluster loss), the diversity criterion prevents pairs from being too similar, thereby avoiding poorly defined axes and supporting a more meaningful interpolation. Despite multiple loss terms, training was stable and converged reliably using our multi-stage procedure (Section 2.4).\n\n- Why not use a part-based framework? Not using a part-based framework is a notable weakness (Question 5, Limitation 2)\n\nPrior work on prototype-based deep neural networks spans two major approaches: using prototypes defined at the global (entire image) level [4, 5] and those defined at the local (patch or part) level [6]. Our choice to adopt image-level prototypes aligns with the former line of work and is a deliberate design decision, primarily due to the nature of our data and labels. Specifically, for age predictions, learning semantically meaningful part prototypes can be challenging, since a 50-year-old may have facial features that resemble someone younger, making part-based prototypes less reliable.\n\nThat said, we fully agree with the concern about not using a part-based framework. Fortunately, this limitation can be addressed by post-hoc attribution methods. For instance, occlusion-based sensitivity analysis [7] or input perturbation methods [8] can help localize which input regions most influence the similarity (or dissimilarity) to a given prototype pair. This approach has precedent: Ragodos et al. [5] also used global prototypes (entire game frames) but applied post-hoc attribution to identify salient regions that drive prototype similarity.\n\nIn our setting, to identify which input region a prototype pair attends to for a given input image, we sequentially masked input patches and measured the change in its prediction as an importance score. This patch occlusion method, though not built into the core architecture, provides spatial explanations without requiring unreliable part-level supervision. We analyzed the top-10 most important patches across 5 randomly chosen test images and found that high-impact regions consistently fall within rows 2–6 and columns 2–6 of the 8×8 patch grid—corresponding to key facial landmarks such as the eyes, nose, lips, and jawline. This suggests the model consistently focuses on semantically meaningful features.\n\nQuantitatively, as shown in the table below, the average change in prediction across the occluded patches ranged from 0.38 to 1.07, with maximum change for each image between 1.82 and 4.40, confirming that these regions exert a significantly greater influence on the model’s output. In all cases, these regions corresponded to one of the facial landmarks.\n\n| Test image   | Average prediction change | Max prediction change |\n|---------------|----------------|-------------|\n| Test image 1  | 0.525  | 2.737 |\n| Test image 2  | 0.773  | 4.153 |\n| Test image 3  | 0.378  | 1.820 |\n| Test image 4  | 1.071  | 4.400 |\n| Test image 5  | 0.884  | 3.438 |\n\n- Computational advantages or drawbacks (Limitation 1):\n\nWhile ProtoPairNet does introduce some computational overhead due to the need to learn and maintain prototype pairs in latent space and the multi-stage training procedure, in practice, we found that its training time is shorter than, or similar to, that of other existing models (Appendix M). In particular, we have found that ProtoPairNet training is much faster than HPN [9] training. \n\nIn addition to enhanced interpretability, ProtoPairNet offers several computational advantages:\nProtoPairNet’s reasoning based on Equations (4) and (6) can be implemented efficiently using existing deep learning frameworks (e.g., PyTorch).\nProtoPairNet uses a fixed set of prototype pairs, rather than relying on large nearest-neighbor databases or exemplar memory, which reduces memory requirements and lookup time.\nThe use of Gumbel-softmax allows differentiable, sparse, and low-cost identification of the most relevant prototype pairs.\n\n- Maximum mean discrepancy (MMD) versus k-means based approach (Limitation 1):\n\nMMD [1] offers distributional matching but is computationally heavier and less aligned with our geometrical setup. Our k-means-based approach allows explicit control over prototype diversity and easily grounds prototype pair axes in real data pairs. It also scales efficiently with latent dimensions and dataset size.\n\n- Code availability (Limitation 3):\n\nAnonymized code was uploaded to OpenReview as a supplementary material (in a .zip file) prior to the submission deadline. Please download the supplementary material of our paper.\n\n[1] Kim, B., Khanna, R. and Koyejo, O.O., 2016. Examples are not enough, learn to criticize! criticism for interpretability. Advances in neural information processing systems, 29.\n\n[2] Zhang, Xijuan, et al. \"How to think clearly about the central limit theorem.\" Psychological Methods 28.6 (2023): 1427.\n\n[3] Matz, David C., and Emily L. Hause. \"“Dealing” with the central limit theorem.\" Teaching of Psychology 35.3 (2008): 198-200.\n\n[4] Kenny, E. M., Tucker, M., & Shah, J. (2023). Towards interpretable deep reinforcement learning with human-friendly prototypes. In The Eleventh International Conference on Learning Representations.\n\n[5] Ragodos, R., Wang, T., Lin, Q., & Zhou, X. (2022). Protox: Explaining a reinforcement learning agent via prototyping. Advances in Neural Information Processing Systems, 35, 27239-27252.\n\n\n[6] Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., & Su, J. K. (2019). This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32.\n\n[7] Zeiler, M. D., & Fergus, R. (2014, September). Visualizing and understanding convolutional networks. In European conference on computer vision (pp. 818-833). Cham: Springer International Publishing.\n\n[8] Petsiuk, V., Das, A., & Saenko, K. (2018) RISE: Randomized Input Sampling for Explanation of Black-box Models. British Machine Vision Conference.\n\n[9] Mettes, P., Van der Pol, E., & Snoek, C. (2019). Hyperspherical prototype networks. Advances in neural information processing systems, 32."}]

Author rebuttal:

Meta-review: This paper introduces ProtoPairNet, a novel architecture that extends interpretable, prototype-based reasoning to regression and continuous control tasks. The core idea is to move beyond the traditional one-to-one comparisons used in prototype-based classification and instead use "prototypical pairs". These pairs define axes in the latent space, and the model makes predictions based on an input's geometric projection onto the most relevant axis. This is an interesting solution to a key challenge in interpretable regression: capturing the directional information needed for continuous predictions. The method is validated on multiple tasks including age prediction and a car racing behavioral cloning task, where it achieves performance competitive with black-box models while providing transparent, case-based explanations.

The primary strength of this paper is its novel and elegant conceptual contribution. As noted by multiple reviewers (tFYK, gJBu), extending prototype-based methods to regression is an important and underexplored area. ProtoPairNet's use of paired prototypes is a simple yet powerful idea that is well-grounded mathematically and provides an intuitive geometric interpretation for its reasoning process. Crucially, the paper demonstrates that this interpretability does not come at a significant cost to performance; the model achieves results on par with, and in some cases exceeding, its black-box counterparts. The interpretability claims are further supported by extensive user studies, which confirm that the model's reasoning aligns well with human intuition and is preferred over single-prototype explanations.

The main weakness, shared by several reviewers (uVMS, tFYK), is the limited scope of the experimental validation. The evaluation is primarily focused on two domains, and a broader set of benchmarks, including non-visual tasks, would strengthen the claims of generality. Another valid point, raised by reviewer tFYK, is that the interpretability is based on global prototypes (entire images) and lacks the fine-grained, part-based attribution of some other prototype models. However, the authors made a great effort to address these concerns during the rebuttal. They added new experiments on tabular and medical imaging datasets, showing competitive performance and broadening the evaluation scope. They also conducted a post-hoc occlusion analysis to provide more evidence that the model focuses on semantically relevant regions.

Overall, this is a strong paper that introduces a novel, well-motivated, and thoroughly evaluated method for a challenging problem. While the evaluation could be broader, the core contribution is significant enough to warrant acceptance. The authors' proactive and comprehensive rebuttal further strengthened the submission by addressing the main reviewer concerns.
Decision: Accept (poster)