Hardware Resilience Properties of Text-Guided Image Classifiers
Authors: Syed Talal Wasim
Kabila Haile Soboka
Abdulrahman Mahmoud
Salman Khan
David Brooks
Gu-Yeon Wei
Venue: NeurIPS 2023 poster (2023)
Keywords: Hardware Resilience
Reliability
Image Classification
CLIP
Vision-Language
Multimodal
TL;DR: Utilizing enriched text embeddings to enhance hardware reliability of image classification models, achieving upto 14-fold increase in hardware reliability with minimal accuracy drop.

Abstract: This paper presents a novel method to enhance the reliability of image classification models during deployment in the face of transient hardware errors. By utilizing enriched text embeddings derived from GPT-3 with question prompts per class and CLIP pretrained text encoder, we investigate their impact as an initialization for the classification layer. Our approach achieves a remarkable $5.5\times$ average increase in hardware reliability (and up to $14\times$) across various architectures in the most critical layer, with minimal accuracy drop ($0.3\%$ on average) compared to baseline PyTorch models. Furthermore, our method seamlessly integrates with any image classification backbone, showcases results across various network architectures, decreases parameter and FLOPs overhead, and follows a consistent training recipe. This research offers a practical and efficient solution to bolster the robustness of image classification models against hardware failures, with potential implications for future studies in this domain. Our code and models are released at https://github.com/TalalWasim/TextGuidedResilience.

Hardware Resilience Properties of Text-Guided Image Classifiers
Syed Talal Wasim Mohamed bin Zayed University of AI
Kabila Haile Soboka Independent
Abdulrahman Mahmoud Harvard University
Salman Khan
David Brooks
Gu-Yeon Wei Harvard University
Mohamed bin Zayed University of AI
Harvard University
Abstract
This paper presents a novel method to enhance the reliability of image classification models during deployment in the face of transient hardware errors. By utilizing enriched text embeddings derived from GPT-3 with question prompts per class and CLIP pretrained text encoder, we investigate their impact for the classification layer. Our approach achieves a remarkable 5.5x average increase in hardware reliability (and up to 14x) across various architectures in the most critical layer, with minimal accuracy drop (0.3% on average) compared to baseline PyTorch models. Furthermore, our method seamlessly integrates with any image classification backbone, showcases results across various network architectures, decreases parameter and FLOPs overhead, and follows consistent training recipe. This research offers practical and efficient solution to bolster the robustness of image classification models against hardware failures, with potential implications for future studies in this domain. Our code and models are released at/htps://gthhub.com www.eooooooaaaaaaaao Wwwimm/eextCuidddRees.o..
initialization
Introduction
As transistors in hardware shrink in size, they become susceptible random bit-flips from environmental factors such as cosmic particle strikes 4.] voltage droops [53], manufacturing defects (55), and/or aging effects [36]. This is particularly noticeable scale, where even small error rates in hardware components can cause data corruptions that affect large-scale corporations such Google (2 Meta [3], prompting these corporations deploy major resources to address the issue [56). The problem of silent data corruptions, or SDCs, is further exacerbated in safety-critical domains (such as in autonomous vehicles, medical devices, or robotics), where even a few errors can lead to fatal and undesirable consequences.
At the same time, the rise of ML. algorithms proliferating data centers and safety-critical domains means that hardware robustness to this particular domain is extremely important. Recent work has shown that little single adversarrsa)i bit flip during DNN's inference can cause wrong predictions by the model [34]. In the context of a self-driving car, this can potentially lead to consequential downstream decision making which be fatal, such as accelerating instead of braking (e.g.. by classifying a truck as bird for instance) (34). Thus, is imperative to understand and mitigate the effect of hardware bit flips on executing softwrre application, where focus here is vision classification for its broad applicability and importance.
The current state-of the-art-tt technique to mitigate hardware errors is full modular hardware redundancy. This is the approach taken recently by Tesla in their Full Self- Driving (FSD) chip, where they employ a fully redundant co- co-processor with additional wiring, logic, and packaging
parallel inferences
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

for comparison (and rerunning on a mismatch) [64]. While this approach is effective in identifying and mitigating errors during inference, the associated 2x overhead is excessive and potentially unscalable for many domains which may need to operate under stricter hardware, energy, and cost budgets.
While memory errors be protected by traditional error-coreecingg code (ECC) additional parity bits at fraction of the cost of full redundancy, errors that occur during computation are more difficult to address at low cost. Further, they are also exceptionally difficult to detect, since they are many times silent and do not cause an application to crash but still result in incorrect outcomes. Recent research at the intersection of ML and silent data corruption (SDC) detection has explored the use of low-cost dynamic range detectors during deployment [7]. selective feature- feature-map duplication [40], and inference re-execution [41] to detect and mitigate single-bit flips at run time in order avoid full modular redundancy while targeting high enror coverage. While these techniques have shown promise, they are more reactive in that they target inference, with the objective of hardening pre-existing model to function in faulty environment. Instead, in this work, we introduce what we believe is the first training-side technique, with the objective of developing out-of-the-boe models that are more resilient against transient bit-flips in hardware.
In this paper, we present a novel software-driven solution to improve hardware reliability in neural net- works. Our approach combines textual information from the Contrastive Language-Iman Pre-training (CLIP)
49 model with visual information from a traditional classification neural network to strongly attenuate the effect of single-bit hardware errors in the computational components of a model. The proposed method based on the observation that textual information can often provide useful context to interpret visual data, thereby enhancing the accuracy of error detection and correction. Our experiments show that the combination of textual and visual information can improve the reliability of a neural network's is classification layer by up to 14x compared to traditional error detection and correction tech- niques, with minimal changes to pre-existing training recipes and their corresponding training accuracy.
The primary contributions of this paper are:
1. Contribution 1:
We propose simple training methodology combining textual and visual information about an image to improve model's robustness to hardware-based. based. transient computational errors which can occur during model deployment
2. Contribution 2:
We rigorously evaluate our proposed methodology using both traditional accuracy-based metrics from the ML community, as well as reliability- bassed metrics from the hardware resiliency community [5] Our results provide favorable tradeoff, where, on average, a 0.32% validation accuracy loss on the ImageNet dataset translates to a hardware reliability improvement of up to 14x ($6). Furthermore, we show that the 0.32% average accuracy loss statistically insignificant by analyzing the statistical distribution of predictions across both the original, unhardened model and our novel, robust model ($7).
3. Contribution 3:
We provide a thorough discussion based on of visualization techniques and empirical data to explain why our method performs better, with ablation studies, intuitive explanations, and statistical validation ($7).
2 Scope and Limitations
This work not about adversarial robustness, but rather focuses on hardware-based fault mitigation and analysis. Two ligh- distinctions between these two are that (1) we do not assume a malicious adver- sary, but rather environmental effects which cause faults to occur in the hardware during the exccution 6 5) and (2) adversarial attacks typically corrupt the input of model, while we focus on computational errors (i.e., neuron corruptions), which may occur in multiply-and accumulate (MAC) operations during execution due to environmental- manufacturing- based effects.
of model [54] 33 12 58 28
In the context of to identify and mitigate to avoid data corruption at- scale, or fatally worse outcomes in real-time systems. While we believe the concept of resilience may be similar to adversarial robustness or Out-of-Domain (OOD) reliability (and in fact, our idea to use stems focus our evaluation on improving hardware reliability. Exploring the correlation between hardware reliability and these other domain-speciffi reliability concepts is of particular interest for future work.
critical systems and/or large-scale systems, these hardware errors are important
stems from this similarity), we

---

3 Related Work
Unimodal Vision Models: The AlexNet [3] model, introduced in 2012, was Convolutional Neural Network (CNN) that gained popularity by winning the ImageNet [T]] competition. It was followed by other models like VGG [57], which emphasized smaller filter sizes and deeper networks, and ResNet 200, which addressed the vanishing gradient problem with skip connections, enabling the training of very deep networks. More recent models such as EfficientNet [59], and MobileNet [24] have further improved efficiency and accuracy by utilizing compound scaling and lightweight architectures for mobile devices. However, CNNs have limitations such as limited receptive field and spatial inductive biases. To overcome these limitations, transformer-based approaches have emerged in computer vision. Inspired by the Transformer [6]] architecture in natural language processing, the Vision Transformer (ViT) [44 model was proposed. It processes image patches as sequences and achieves competitive performance
various benchmarks. Other studies, like the Swin Transformer [38 37 and Max ViT [60], have built upon the success of ViTs, focusing on improving accuracy and computational efficiency. Additionally, there are hybrid works that take inspiration from both Transformers and CNNs, such as FocalNets [65], which propose an efficient alternative to the self-attention operator, focal modulation, based on Convolutions. These models are typically trained using a cross-entropy objective. However, they have shown high susceptibility and unreliability to hard- ware errors [45 51 34 35 39), such as bit flips in the weights and activations. To ensure trustworthy deployment for real applications, it is crucial to establish strong resilience and reliability.
Multi-Modal Vision- LlsguunLanguage Models: Advances in Natural Language Processing (NLP) has led to the development of vision-language models like CLIP [49]. Aligm [26), and Florence [66]. These models consist of image and text encoders and are trained using a contrastive approach with extensive image-text pairs. The goal is to establish a shared feature space between visual and textual features, allowing models like CLIP [4] to gain nuanced understanding of visual concepts. This approach benefits various downstream tasks such as "zero-shot" image classification, semantic segmentation [50 68), object detection (15), point cloud classification [67], and video recognition [46). Additionally, CLIP has demonstrated impressive generalization capabilities on dut-of-distribution tasks, including evaluations on datasets like ImageNet-A (22. ImageNet-R (2]], ImageNet-Sketel) [622] and ImageNet (52). However, training a CLIP model from scratch is prohibitively expensive. To address this, researchers have employed techniques like using enriched text prompts from large language models [48] such as GPT-3 4 or employing prompting/linctuni
methods 77 69
63 to enhance performance on -utoo--distribution tasks.
The impact of such text-guided classification on hardware resilience and reliability is an unexplored topic in literature. The strong generalization capabilities demonstrated by rext-guided classification models like CLIP suggest the potential for improved resilience to hardware errors. By leveraging the semantic supervision provided by text, these models can acquire nuanced understanding of visual concepts, which may help them to better handle and adapt to errors or inconsistencies in hardware.
Hardware Resilience and Reliability: As NN-ba image classification models begin to take off in vision-basse based tasks (such as for autonomous driving, or AV, systems), their robustness to hardware perturbations has become of paramount importance for practical deployment and government certification (25.. For example, Tesla's FSD uses the simplistic full duplication method to achieve high resilience, effectively allocating double the silicon to detect and correct errors [64]. However, due to the high associated costs of full modular duplication, an open call for cheaper yet equally accurate techniques has surfaced in recent years 455.
Rather than relying on hardware solutions for hardware faults, software-based solutions have risen to prominence due to their comparatively lower overhead in this fast-moving field. Proposed techniques leverage unique insights about the application domain (namely, neural networks) to systematically detect and recover from errors. Selective duplication of feature maps in CNNs 44), value attenuation in the form of range-detectors [7]. and temporal re-execution of inferences 41 have all shown to be adept at identifying errors in a low-cost manner, with reasonable guarantees error coverage.
However, all prior research in the field assumes a model already trained and ready for deployment, and only then does the task of making it more resilient to hardware errors come into play (by using some of the aforementioned techniques above). contrast to prior work, our focus in this paper is to provide a training routine that generates robust models directly using textual-visual information (. Effectively, our technique is a training- based method for designing robust image classification models, evaluated for its robustness single-bit, transient hardware errors at run-time.
3

---

Feulures Cless
(C x
Averaged
Fealures
DesctoN piiclu af ALLASS
Ustaned oeaopgoneppoonnn HuLASS
Questions
Descriptions
Per Class
GPT
Per Class
CUP Text Enseder
pfetorule <GLASS
+GLASS
Initialization
Gotdieie jeir
Frozen
B: Baten Size
Moculcs
C: Number of Clesses
Image Model
Parameters
Leered
E: Latent Embedding Sbse
D: Number of Desorizions
Imagge Inguut
Lalent Representation
(B
Projection Layer
(Bx
(B x 724 224)
Figure 1: The proposed Architecture: D questions for each class (total C) are hand-crafted. These fed to GPT-3 4 model, to obtain D detailed descriptions per class. A CLIP text encoder is used to produce text embeddings, which are averaged across descriptions. The text embeddings initialize a projection layer which is then trained alongside the randomly initialized backbone.
4 Our Approach
Multimodal pretrained models like CLIP 49 which are used for image classification, have demonstrated the ability learn generalized representations. These models are trained vast datasets of language-image image pairs in a contrastive manner, resulting in impressive zero-shot capabilities and effective transfer learning to various downstream tasks.
Given this strong generalization, we the following question: can the generalized representations of these Vision-Languas models help improve hardware reliability?. Our method augments the standard training image classification models by utilizing textual context from the CLIP text encoder, allowing to improve hardware resilience with minimal train and test time overhead.
We start by providing brief overview of vision- langual pre-training, specifically focusing on CLIP in 14.1 However, our methodology can be applied to other vision-languar models that share similarities with CLIP, such as ALIGN and Florence, Following the overview, we provide detailed explanation of our text-guided classification scheme in $4.2
4.1 Overview of the CLIP Model
Conventional methods of image classification have traditionally used the common Cross-Enin Pross-yn
loss-based training for a closed-set classification problem [57 20 38 37). However, recently there has been a trend to employ text supervision for image classification rather than one-hot labels such as major works on contrastive language-image pretraining like CL.IP 49). The CLIP model is composed of two encoders that encode the visual content of images and their corresponding text descriptions, respectively. These encoded representations are then compared using cosine similarity objective.
Proposed Text-Guided Classification
Consider an input image e # x3 of spatial size H x W with 3 channels Red, Green, and Blue (R, G, and B). A standard image classification model [57 20 37 maps the input image to the classification domain C where is the number of classes. However, has been shown that such model is unreliable and susceptible to bit errors 8 41] especially in the classification layer 43
To counter this problem we propose our text-guided image classifier which modifies the last layer of the image classification model to incorporate text features. Hence, this modification applies to any image underlying architecture. Given an input image / H X x3 we first map it to latent dimension E where E is the embedding length of the CLIP Encoder. We then apply a classification projection Plass which maps the latent dimension R RC. We initialize the projection layer P class using features obtained from the CLIP text encoder for each class.
classification model, regardless

---

A naive way to obtain the text features would be to simply pass each class name through the text encoder. However, this is less robust to distribution shifts 77].. and we argue that would therefore be less re- liable. Instead, we follow [48] and augment the class labels using a large question-answering language model like GPT-3 4 We ask GPT-3 a total of 1 questions for each class the total number of classes C, making a total of C' D questions. For each question, GPT-3 outputs detailed text descriptions, form- ing D number of descriptions per class, which we then pass through the CLIP text encoder to produce embeddings of shape C' D x E. We then average over the descriptions per class, to form the final em- bedding tensor of shape C' x E, where each class CE I...,, has an embedding vector in R This tensor is then used to initialize the projections layer P classs Figure
summarizes our proposed approach.
5 Evaluation Methodology
We evaluate our approach on two fronts: first, the impact of our technique on the classification accuracy of the model; and second, the hardware reliability impact of our new technique.
5.1 Evaluation Infrastructure
For each backbone, we train a baseline model (the default architecture), and our modified classification model on the ImageNet training set. For both methods, we follow the standard Py PyTorch [47] training recipe. We then report accuracy and resilience on the ImageNet validation set and compare across mul- tiple architecture families. We train our models on A GPUs and run the training routine for both the original and our modified models to the same number of epochs and with the same hyperparameters as described in the Py Torch Github repository for fair comparison. Our results are presented in For the ablation study presented in Table 3
N6 in Table
we train each model on 8 VVVV GPUs.
5.2 Hardware Reliability Evaluation Methodology
To evaluate the reliability of the proposed model compared to the original model, we use the GoldenEye [3] testbed for error analysis. We describe how this testbed works in more detail in this section. Due to the exponentially large number of potential hardware error sites (e.g.. a single bit flipping in a random register during any dynamic cycle for any image during inference at deployment time), it is impractical to explore all possible error locations and values for a given error model to perform exhaustive evaluation of DNN. Instead, error injection mechanisms used statistically evaluate the likelihood of errors propagating and corrupting an application' 's output 33 g 39 40)
In this work, we transient single-bit flip error model for evaluation, commonly used abstraction
for modeling hardware faults 334 18
35 2 19]. In particular, we focus on errors that occur in
activation values (i.e., neurons), during inference. We assume that memory is protected via ECC or parity (which is common in commercial and safety-critical systems
42) ), allowing us to focus on
computational errors in hardware (i.e., MAC operations).
An error injection experiment involves flipping a single bit in the entire network (i.c., from 0++1 or 1 1->0), and then comparing the final classification of the network with respect to the original, baseline correct output. We use Py Py 33 to perform the random bit flip, and we perform 4096 unique error injection experiments per layer, totaling more than 4.3 million experiments across all our models and corresponding to a 99% confidence level with less than 0.23% confidence interval [32].
To measure reliability, we calculate the rate of all errors which led to an image misclassification as function of all the injections performed. Recent work (4) has proposed a more accurate metric called ALoss, which captures the same information as mismatches but converges asymptotically faster, Conceptually, the ALoss metric calculates the difference cross entropy (CE) between a freet-free inference and an erroncous inference, rather than purely looking at a binary mismatch. Consequently. it provides more granular information per error injection experiment. We use the ALoss metric for comparing the reliability of each layer in the network for the original, baseline training routine and our proposed,
technique. To gather overall network-level reliability improvement, we average the ALoss information gathered from each layer, producing a singular value to compare the baseline model and our proposed text-guided model. We note that it is simple mapping to go
5

---

back to the mismatch- metric and ground this in hardware-centric FIT-rate, while this work leverages the Aosss metric for it's drastically faster speed and accuracy
Prior work has shown that the activation values of the final layer of a network are typically the most vulnerable single-bit perturbation, this layer directly impacts the final classification 4 For this reason, we target our technique and analysis on the last layer, in an effort to mitigate errors at this stage. To ensure fair comparison, we compare the last last layer of the baseline model with the weighted average of the last two layers of our proposed model. This is because our proposed technique technically splits the original last layer into two fully connected layers: the latent representation (B ExE and projection layer (ExC). We combine these last two layers into single layer (B XCC for efficiency and a fair head-to-heed evaluation during inference (we keep them separate during training for initializing the projection layer).
We further show the necessity of the rich GPT-3 initialization of our method through an ablative study in infbbl 3 Finally, we provide qualitative analysis using Ablation-CAM [ as well quantitative analysis for per-image inferences in to further validate the benefits and trade-offs of our textual- visual-based approach for improved hardware reliability. We use the concept of the Top2Diff from the hardware reliability literature [4]] to build intuitive arguments on the reliability of our model. The Top2Diff metric is simply the difference in classification accuracy between the top inferred class and the second-highen class. A large Top2Diff directly translates to better reliability, as the catalyst required by a single bit flip overcome and flip the classification from the correct class to different class is larger.
Results
Our main results across various backbones are summarized in Table 1 trained a baseline version using the training recipe provided by Py PyTorc, followed by our own method trained using the text-guided initialization via GPT-3, again with the same recipe. We used the same set of hyperparameters for both models (detailed hyperparameters are reported in Appendix A). We report the accuracy of the bascline model, the accuracy of proposed approach, the difference in the number of parameters of the two versions, the difference in FLOPs, and the improvement in reliability on the last layer and across the entire backbone. We observe multiple takeaways in our results, which are described below.
For each model backbone,
Table 1: The tables present results across various backbones, reporting top-1 accuracy the ImageNet (10) validation set for both the baseline and our method. Additionally, we report the change total parameters and FLOPs (a negative sign indicates a decrease), the improvement in last-layer and overall model reliability, and percentage increase in Top2Diff.
Additional
Additional FLOPS
Improvement Reliahility Layer)
Improvement in Reliability (Overall)
Aoc.
Acc. Ours
Improvement Top2Diff
Back bone
Buseline
Params
baseline)
baseling
Alexnct 31
G6.
57. 07.28%
1.491
1.50M
7.92x
4.67
2.83%
VGG-16-BN 57
73.45%
72.96%
1.49M
1.50M
14.43x
9.64x
1.62%
VGG-19-BN
57
74.10%
71.01%
-149M
1.50M
13.29x
8.67
1.13%
ResNet-18
69.60%
69.68%
0.26.M
0.26M
2.87x
1.91x
3.07%
ResNet-34
20
73.25%
72.62%
0.26M
0.26M
3.89%
2.53x
2.08%
ResNet-50 200
75.61%
74.81%
-0.99M
-0.99M
4.48x
2.96x
3.35%
ResNet-101 20
77.25%
75.52%
0.49M
0.9MM
4.33x
2.77x
3.13%
ResNet-152 20
77.98%
76.18%
0.49A
0.50M
4.7XX
2,85x
3.09%
MobileNet-V2 V2 24
71.87%
71.83%
-0.11M
0.09M
3.92x
2.43x
5.36%
MaxViT-T 60
82.00 22.988
83.08%
0.26M
0.28M
3.38x
2.63x
2.62%
Swin-V2-T 37
80.97%
80.02%
0.13.M
0.15AL
1.65x
1.07x
2.85%
Swin-V2-S 37
82.71% 7-
82, 86%
0.13M
0.15M
3,51x
2.60x
3.04%
FocalNet-T
65
80.23%
80. 77%
0.13M
0.14M
3.87x
2.61x
2.61%
FocalNet-S
65
82.52%
0.18M
0.14M
4.73x
3.50x
3.10%
Our proposed model has small accuracy reduction on the backbone compared to the baseline, ranging from -1.77% to +0.52% for an average decrease of .3%. Despite the reduction, we find that our proposed model is in fact more confident its accurate predictions based on the difference between the top two classes (the Tof2Diff), For ResNet50, the Top2 for the baseline model 70.32%, while the Top2Diff of our model is 73.67%, a +3.35% improvement. A similar phenomenon is observed across all models, where the average Top2Diff increases by 2.50%. Further, we perform an additional study in $7.4 where we empirically show that this accuracy impact is indeed minimal, especially compared to the upside observed in reliability improvement.
Accuracy impact:

---

Table 2: The tables present results across various datasets (CIFARIO 22
CIFARIO0 [0]
FOOD101 [3] and STLIO ( for two backbones (ResNet-50 [0] and FocalNet-T T 655 D. reporting top-
1 accuracy on the respective validation set for both the baseline and our method. Additionally, we report the improvement in last-layer and overall model reliability, and a percentage increase in Top2Diff.
Improvement
Improvement
Acc. Baseline
Acc. Ours
Improvement in Top2Diff
Dataset
Backbone
in Reliability in Reliability (Overall)
(Last Layer)
CIFARIO
29
ResNet-50
20
95.07% 94.76%
95.29%
2.04x
1.71x
6.70%
CIFARIO
29
FocalNet-T
65
94.94%
2.47x
1.30x
3.58%
CIFARI00
ResNet- 50
20
78.23%
78.53%
2.19x
1.65x
3.69%
CIFAR100 30
FocalNet-T
65
77.06%
79.21 %
3,21x
1.58x
2.90%
FOOD101
ResNet-50
20
83.13%
83.97%
2.66x
2.15x
2.78%
FOODI01
FocalNet-T
65
85.64%
85.91%
3.28x
2.85x
1.70%
STLI0 9
ResNet-50
20
47.73%
52.68%
2.10x
1.91x
2.45%
STL10 9
FocalNet-T
65
62.74%
63.78%
2.23x
1.72x
1.96%
Model Size and Runtime Impact: Our proposed method marginally increases the total number of parameters, on average, by 0.18M compared to the baseline. This reduction is nodel-dependent, as it depends on the second-to-last layer feeding into the latent representation (B x E) before moving onto the projection layer (E x C*) for the final prediction. A few models (such as deeper ResNet's 's and VGG) actually observe slight decrease in total model parameters, which is topology dependent. This parameter difference translates to a small increase/decrease of FLOPs during inference, respectively, as show in column 5. Overall, our proposed technique produces models with similar size and runtime to the baseline on average.
Evaluation on Additional Datasets: We evaluate our method on additional datasets (CIFAR10 29 CIFAR 30, Food101 3 and STLIO (9) for two networks: ResNet-50 (2) and FocalNet-T 66]...
Our results, shown below in Table 2 validate that our technique is general and can work across an array of model types and datasets. Furthermore, we did not have to modify any hyperparameters in the process, suggesting the ease of our technique well the increased benefit from reliability point of view. Additionally, adding these new datasets further support our claims that our technique has negligible impact model training accuracy, whilst still providing with large upside in resilience.
Reliability Evaluation: Most importantly, our proposed technique significantly improves the hardware reliability of the model, as this was the intention behind the method. The most significant change occurs on the last layer, where the average reduction is model-family specific. In other words, the ResNet family observes average 4.01 x hardware reliability improvement and the VGG family observes a 13.68x improvement on the final layer, using the áoss metric as explained in S This difference is related to the baseline backbone, where in general the ResNet family baseline can be considered more reliable than the VGG family to hardware errors 41], resulting in a larger improvement with our proposed technique for the less robust model family (VGG). Overall, observe improvements across the board for all models studied, signifying the benefits of our technique. Similarly, looking at model's end-to-ent hardware reliability, for the VGG family, and 2.61 x for the ResNet family. While this value is strongly influenced by the last layer, we observe that most layers in the network do get modest hardware resilience improvement, captured by the averages listed in the table and additional results in Appendix C.
find the average to be 9.16x better
7 Discussion and Analysis
We perform a series of additional studies to validate and better understand the insights of our proposed method. First, we describe an ablation study on the initialization of the projection layer in $7.1 Second, we provide qualitative explanation of the impact of errors on the baseline versus our proposed method using the ttat--of-the-an A Ablation-CAn -CMM visualization in $7.2 Third, we analyze the impact of the baseline training versus our method's 's training on the activation values produced by each model, and use this to provide an intuitive ex explanation for the improved hardware reliability in $7.3 Finally, in $7.4 we further discuss the tradeoff between the small accuracy drop on the validation set compared with the large improvement in hardware reliability by studying the output classification accuracy of images.
7

---

7.1 Ablation Study
Our ablation study (Table 3) measures the improvement in hardware reliability different projection initialization techniques on ResNet50. We compare 1) random initialization, 2) a CLIP-based initialization ("a photo of [CLASS]" prompt), and 3) our CLIP+GPT-3 initialization.
Table 3: Ablation for type of initialization on the projection layer. CLIP refers to a simple hand-crafted prompt "a photo of [CLASS]" while CLIP+GPT refers to the proposed method in $4
Improvement in Reliability (Last Layer)
Improvement in Reliability (Across Backbone)
Projection
Backbone
Initialization
ResNet-50
random
1.74x
1.09
ResNet-50
CLIP
5.06x
3.28
ResNet-50
CLIP+GPT-3
6.09x
3.93x
In general, we find that any text-based projection initialization helps improve reliability, observed via the "random" experiment which gives 74% last layer improvement, and an overall 9% improvement across the network compared to the bascline. However, more intelligent projection initialization via CLIP improves the hardware reliability up to 3.28 across the network (5.06x for the final layer) and good prompting via GPT-3 "describe [CLASS]" further improves it to 3.93x (6.09x for the final layer). summarize, our ablation study validates that our good hardware reliability improvements indeed come from proposed initialization.
7.2 Ablation-CAl CAM Visualization
Raw
Baseline Without Error
Baseline With Error
Ours Without Error
Ours With Error
Image
(a) Class Label: "Admiral"
(b) Class Label: "Welsh springer spaniel"
(c) Class Label: "African hunting dog"
Figure 2: Comparative Visualization of Baseline and Our version of ResNet-50 Before and After Single Bit Flip Error Injection. Each subfigure presents the original image, alongside the Class Activation Mapping (CAM) visualizations for both the baseline and model before and after error injection. Prior to error injection, both models concentrate on key features of the images for classification. Post error injection, the baseline model's focus diverges, for instance, from the African hunting dog to the surrounding foliage (Figure 2c), whereas our model maintains its original focus, demonstrating its robustness against the induced error.
Ablation-C CAM is a visualization technique developed for DNNs that employs gradient-free approach [II. A departure from conventional gradient- methods, Ablation-CAM systematically removes parts of the network, allowing a deeper understanding of the individual feature map units

---

contributing to the model's decisions. This ablation process generates a coarse localization map, highlighting regions in the network's input image that are critical for predictions.
In our study, we chose Ablation-CAM to visualize the decision-making process of our models (original versus proposed technique). Ablation-CAM's gradient-free nature offers way of comprchending the focus areas within deep learning models, addressing the limitations of gradient-based methods, such susceptibility noise the gradient and the inability capture the entire network's is collective decision-making process [. Furthermore, Ablation-
robust
ability evaluate model trustworthiness [T] was critical in understanding the robustness of our models to error injection. By observing how the models' focus shifts in response to error injection, we could make judgments about their resilience and reliability. This unique combination of features made Ablation Ablation-CAM CAM an ideal tool for our study.
Figure depicts results for three images the ResNet50 backbone (Additional visualizations presented in Appendix B). We inject 2000 random errors weights across the network (for visualization purposes) and project the impact on the imput image, to see how the model responds to the same exact per- turbations. Our results highlight the fact that despite the many errors, our proposed technique maintains focus the important features which correspond to better hardware-level reliability as discussed in
7.3 Value Ranges
Another angle we study in this work is the impact our proposed technique on the value ranges for ac- tivation values (i.e., neurons) of model. Prior work has shown that smaller values during computations typically are more robust to single-bit perturbations, as their impact does not propagate as far (except for errors in the exponent fields of floating point value, which range detectors quantization can help mit- igate). fact, Google previously proposed a ReLU6 (24] activation function to clip values above the ar- bitrary value of 6, later used range detector for reliability purposes [I6]. Similarly, organic reduc- tion in values is beneficial from hardware reliability perspective, which we target with our approach.
Max Abso Absolute lute
Neuron Value
ResNet-50 baseline
40
ResNet-50 ours
20
6
12
24
30
36
42
48
54
Model Layer
(a) ResNet-50 Max Absolute Value
ResNet-50 baseline
Mean Absolute
Neuron Value
1.5
ResNet-50 ours
0.0
Model Layer
(b) RcsNet-50 Mean Absolute Value
Figure 3: Observed Neuron values for ResNet50. The Y-axis shows the max absolute value
(Figure 3a) and mean absolute value (Figure 35) observed by profiling the ImageNet dataset on the baseline and our model per-layer basis. It can be seen that both max and mean viable choices for profiling network neuron value ranges, and result in similar trends.
Figure 3 shows the absolute maximum and absolute mean observed neuron values per layer for ResNet50 across the ImageNet dataset. We find that our proposed technique strongly attenuates the values in the last layer for both measurems. This result helps explain why our technique is more reliable, and why particularly beneficial the final layer. The fault-free values are smaller to begin with, which in turn enable smaller range detectors 7 and also are less likely to change into negative impactful error on the classification result. We observe similar trend across network studies in experiments. We provide additional results for different networks in Appendix C.

---

The number representation in hardware also plays large role in the reliability of a model, which our proposed technique directly influences. To better understand this effect, we direct the reader the hardware implementation of numbers, which typically use the IEEE-754 floating point format, consist- ing of sign bit, exponent bits, and mantissa bits. Intuitively, bit flips in the exponent bit are the most egregious, and having range detectors in place helps detect these types of errors. More subtle, however, is that depending on the original exponent value, certain bit flips can transform a number to become much larger or much smaller. In this case, a bit flip in small number (which we identify as smaller than 2) has a very high probability of changing to another small value, regardless of which bit is flipped. In particular,
long as the sign bit or the most significant bit of the exponent (bit 31 and 30) are not the ones flipped. then the IEEE- 754 format guarantees that the new, erroneous number stays under the value 2. As such, the small magnitude change has little impact the end-to-end inference of a classification, and masks such errors. This is why it is crucial and advantageous to have smaller neuron values in neural network, which various techniques such as hatch normalization and newly proposed technique help organically enforce (unlike an artificial enforcer such as Rel.U6). Thus, our new training routine helps accomplish this through the use of the projection layer at the tail end of the neural network.
7.4 Understanding the Accuracy Degradation in the Context of Hardware Reliability
To better understand the ~0.3% accuracy loss of our technique, we wanted to see how the baseline and proposed models matched up if we excluded low-confidence images (i.e., images "at the border" during classification). In practice, low-confidence images would not be relied upon for safety-critical decision-making. hence wanted to measure the "true" accuracy of the models.
Model Accuracy
0.74
ResNet-50 baseline
ResNet-50 ours
0.72
0.70
0.68
0.66
20
35
Delta [Top2Diff cutoff for image exclusion)
Figure 4: Model accuracy as function of Top2Diff deltas for ResNet50. This figure shows that as we exclude images with low Top2Diff for classification accuracy measurements, our proposed model recoups accuracy faster than the original baseline model, indicating that many correctly classified images by the baseline model are borderline correct, begin with.
We perform a sweep of different Top2Diff values (where "Delta" goes from 1 1% to 40%) and exclude the "correct" images that have a Top2Diff value below each sweep point. We measure the new network accuracy in light of these delta values and found that many images that were classified correctly by the original model "fell off" as we increased the delta. On the other hand, our proposed model did not lose its classification accuracy fast; delta of Top2Diff=15, the inflection point occurs where our method has the same accuracy (i.e., 0% accuracy degradation between models) as the original model, and improves beyond this point. That said, a 0.3% accuracy loss itself is reasonable in and of itself for the large hardware reliability gains we observe, yet this discussion point presents a trade-off' opportunity (as function of Top2Diff) that can enable model designer tune their model for their desired accuracy and hardware reliability targets. To further validate this claim, we find that for different datasets (Table 2). our technique marginally improves accuracy across the board.
8 Conclusion
In conclusion, our paper presents a software-driven solution to enhance hardware reliability in neural net- works. By combining textual and visual information, we mitigate the impact transient bit-flips during computation. Our approach improves neural network reliability of the most critical layer by up to 14x compared to the baseline, with minimal changes to training. We contribute a simple training method- ology, rigorous evaluation using accuracy and reliability metrics, and comprchensive discussion sup- ported by visualization techniques. Our work highlights the significance of addressing hardware errors during training, offering a promising direction for developing robust models against transient bit- bit-flips.
10

---

Acknowledgments
This work supported in part by the National Science Foundation (NSF) grant CCF- 1704844. We also thank the Fatima Fellowship for their GPU grant and Hugging Face for organizing and sponsoring the Fatima Research Fellowship program.
References
[1] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-basee attribution methods for deep neural networks, 2018.
[2] R. A. Ashraf, R. Gioiosa, G. Kestor, R. F. DeMara, C. Cher, and Bose. Understanding the propagation of transient errors in hpc applications. In SC, pages 1-12, 2015.
[3] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 - mining discriminative components with random forests. In ECCV, 2014.
[4] Tom Brown Language models are few-shot learners. In NeurlPS, 2020.
[5] Franck Cappello, Geist AI, William Gropp, Sanjay Kale, Bill Kramer, and Marc Snir. Toward Exascale Resilience: 2014 Update. Supercomput. Front. Innow: Int. J, 2014.
[6] Franck Cappello, Al Geist, Bill Gropp. Laxmikant Kale, Bill Kramer, and Mare Snir. Toward exascale resilience. I/HPCA, 23(4):374-388, November 2009.
[7] Zitao Chen, Guanpeng Li, and Karthik Pattahiraman. Ranger: Boosting Error Resilience of Deep Neural Networks through Range Restriction. CoRR, abs/2003. 13874, 2020.
[8] Zitao Chen, Niranjhana Narayanan, Bo Fang, Guanpeng Li, Karthik Pattabiraman. and Nathan DeBardeleben. TensorFI: A Flexible Fault Injection Framework for TensorFlow Applications. CoRR, abs/2004.01743, 2020.
[9] Adam Coates, Honglak Lee, and Andrew Y. Ng. An analysis of single layer networks in unsupervised feature learning. AISTAAS, 2021.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[11] Saurabh Desai and Harish G. Ramaswamy, Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization. In WACV, 2020.
[12] S. Di, H. Guo, R. Gupta, E. R. Pershey, M. Snir, and F. Cappello. Exploring properties and correlations of fatal events in large-scale hpe system. TPDS, 2019.
[13] Harish Dattatraya Dixit, Sneha Pendharkar, Matt Beadon, Chris Mason, Tejasvi Chakravarthy, Bharath Muthiah, and Sriram Sankar. Silent data corruptions scale. arXiv preprini a0X / /245, 2021.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[15] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- voccbuacyyylary object detection with lanionnl model. In CVPR, 2022.
[16] Florian Geissler, Syed Qutub. Sayanta Roychowdhury, Ali Asgari, Yang Peng, Akash Dhamasia. Ralf Graefe, Karthik Pattabiraman, and Michael Paulitsch. Towards a safety case for hardware fault tolerance in convolutional neural networks using activation range supervision, 2021.
[17] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Lin. Open-vocabulary image segmentation. arXiv preprint arXt::2112.121433 2021.

---

[18] Hui Guan, Lin Ning, Z. Lin, Xipeng Shen, Huiyang Zhou, and Seung- Seung-Hwan Hwan Lim. In-Place Zero-Space Memory Protection for CNN. ArXiv, (/bs//910.11/. 14479. 2019.
[19] Siva Kumar Sastry Hari, Michael Sullivan, Timothy Tsai, and Stephen W Keckler. Making Convolutions Resilient via Algorithm-Based Error Detection Techniques. TDSC, 2021.
[20] Kaiming He, Xiangyu Zhang. Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,
[21] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribttion generalization. In ICCV, 2021.
[22] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
[23] Peter H Hochschild, Paul Turner, Jeffrey C Mogul, Rama Govindaraju, Parthasarathy Ranganathan, David E Culler, and Amin Vahdat. Cores that don't 't count. In HotOS, 2021.
[24] Andrew G. Howard, Menglong Zhu amd Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arxiv preprint, 3XXi::7744.0861/ 2017.
[25] International Organization for Standardization. Road vehicles - Functional safety.
https: :/wp:://wwww... is0. org/ /ttg///aandaa//444444.4. hhml 2011.
[26] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuar Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arxiv preprint, arXiv:2102.0550 2021.
[27]
Muhammad Uzair khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023.
(28) Peter Kogge, S. Borkar, Dan Campbell, William Carlson, William Dally, Monty Denneau, Paul Franzon, William Harrod, Jon Hiller, Stephen Keekler, Dean Klein, and Robert Lucas. Exascale computing study: Technology challenges in achieving exascale systems. DARPA IPTO, 15, 01 2008.
[29] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 Dataset. Technical report, Canadian Institute for Advanced Research, 2009.
[30] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-100 Dataset. Technical report,
Canadian Institute for Advanced Research, 2009.
[31] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012. 2012..
[32] R. Leveugle, A. Calvez, P. Maistri, and P. Vanhauwaert. Statistical fault injection: Quantified error and confidence. In DATE, pages 502-506, 2009.
[33] S. Levy, K. B. Ferreira, N. DeBardeleben, T. Siddiqua, V. Sridharan, and F. Baseman. Lessons learned from memory errors observed over the lifetime of ciclo. In SC, pages 554-565, 2018.
[34] Guanpeng Li, Siva Kumar Sastry Hari, Michael Sullivan, Timothy Tsai, Karthik Pattabiraman, Joel Emer. and Stephen W. Keckler. Understanding Error Propagation in Deep Learning Neural Network (DNN) Accelerators and Applications. In SC, SC 17. New York, NY, USA, 2017. Association for Computing Machinery.
[35] Guanpeng Li, Karthik Pattabiraman, Siva Kumar Sastry Hari, Michael Sullivan, and Timothy Tsai. Modeling Soft-Error Propagation in Programs. DSN, 2018.
[36] Chang Liu. Improvement of hardware reliability with aging monitors. PhD thesis, 05 Fakultät Informatik, Elektrotechnik und Informationssechnik, 2017.
[37] Ze Liu et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, 2022.
12

---

[38]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[39] Abdulrahman Mahmoud, Neeraj Aggarwal, Alex Nobbe, Jose Rodrigo Sanchez Vicarte, Sarita Adve, Christopher W. Fletcher, Iuri Frosio, and Siva Kumar Sastry Hari. Py PyTorchFI: A Runtime Perturbation Tool for DNNs. In DSN-W W, 2020.
[40] Abdulrahman Mahmoud, Siva Kumar Sastry Hari, Christopher W. Fletcher, Sarita V. Adve, Charbel Sakr, Naresh Shanbhag, Pavlo Molchanov, Michael B. Sullivan, Timothy Tsai, and Stephen W. Keckler. Hardnn: Feature map vulnerability evaluation in cnns, 2020.
[41]
Abdulrahman Mahmoud, Siva Kumar Sastry Hari, Christopher W. Fletcher, Sarita V. Adve, Charbel Sakr, Naresh R. Shanbhag, Pavlo Molchanov, Michael Sullivan, Timothy Tsai, and Stephen W. Keckler. Optimizing Selective Protection for CNN Resilience. ISSRE, 2021.
[42] Abdulrahman Mahmoud, Siva Kumar Sastry Hari, Michael B. Sullivan, Timothy Tsai, and Stephen W. Keckler. Optimizing software-directer instruction replication for gpu error detection. In SC, SC 18, 2018.
[43] Abdulrahman Mahmoud, Thiery Tambe, Tarek Aloui, David Brooks, and Gu Yeon-Wei. Gold- enEye: A Platform for Evaluating Emerging Data Formats in DNN Accelerators. In DSN, 2022.
[44] Sarah Michalak, Andrew J DuBois, Curtis B Storlie, Heather M Quinn, William N Rust, David H DuBois, David Modl, Andrea Manuzzato, and Sean Blanchard. Assessment of the impact of rossiic-ray-nnduee neutrons on hardware in the roadrunner supercomputer. TDMR, 2012.
[45] Sparsh Mittal. A survey modeling and improving reliability of DNN algorithms and accelerators. JSA, 104:101689, 2020.
[46] Bolin Ni, Houwen Peng. Minghao Chen, Songyang Zhang, Gaofeng Meng. Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding taguuge-imagg pretrained models for general video recognition. In ECCV, 2022.
[47] Adam Paszke et al. Pytorch: An imperative style, hegh-peeffrrmacc deep learning library. In NeurIPS, 2019.
[48] Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does platypus look like? generating customized prompts for zero-shot image classification. arXiv preprint arXiv:2209.033220 2022.
[49] Alec Radford et al. Learning transferable visual models from natural language supervision. In (CML,2021.
[50] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang. Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, pages 18082-18091, 2022.
[51] Brandon Reagen, Udit Gupta, Lillian Pentecost, Paul Whatmough, Sae Kyu Lee, Niamh Mulholland, David Brooks, and Gu- Yeon Wei. Ares: Framework for Quantifying the Resilience of Deep Neural Networks. In DAC, DAC 118 pages 17:1-17:6, New York, NY, USA, 2018. ACM.
[52] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? ICML, 2019.
[53] Vijay Janapa Reddi, Meeta Gupta, Glenn Holloway, Michael D. Smith, Gu-Yeon Wei, and David Brooks. Predicting voltage droops using recurring program and microarchitectural event activity. MICRO, 2010.
[54] Safety Research and Strategies. Inc Toyota unintended acceleration and the big bowl of 'spaghetti" code. "http://wwal http://www..ffrrreeeeeeeeeeeeeee http:///weeaeacateeee/////rrttta toyota-uma 9D- code 2013.
[55] Igor Schagaev and Thomas Kaegi-Traches Hardware Faults. Springer International Publishing,
02 2016.
13

---

[56] Kostya Serebryany. Maxim Lifantsev, Konstantin Shtoyk, Doug Kwan, and Peter Hochschild. Silifuzz: Fuzzing epus by proxy, 2021.
[57] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
[58] Li Tan and Nathan DeBardeleben. Failure analysis and quantification for contemporary and future supercomputers. ArXiv, abs/1911.00211 2019.
[59] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML. 2019.
[60] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeuriPS, 2017.
[62] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurlPS, 2019.
[63] Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. In CVPR, 2023.
[64] WikiChip. Fsd chip tesla wikichip. https://on wtw.ciip...rgr..............o.o. wttpichppp.rr//iii//sssssaaaaaaa (car
company)/fsddchip 2019.
[65] Jianwei Yang. Chunyuan Li, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Focal modulation networks. In NeurlPS, 2022.
[66] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In CVPR, 2022.
[67] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In CVPR, 2022.
[68] Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip. arXiv preprint arXiv:2112.01000 2021.
[69] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. CVPR, 2022.
[70] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. I/CV,2022
14


Reviews:
[{"review_id": "14gsyV9xk5", "reviewer": "Reviewer_4rtm", "summary": "This paper studies how CLIP can be used to mitigate the effect of hardware failures on image classification models. The authors do this by incorporating embeddings from the clip text encoder of class based inputs (queried through GPT) to initialize the classification layer. The authors evaluate this technique by comparing how it affects accuracy on the downstream task, as well as some hardware reliability evaluation.", "questions": "See weaknesses - mostly concerned about evaluation and comparison", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Presentation is clear and explains why this is an important problem to work on for someone not familiar with the area\n\n- Method is simple and generalizes to standard image classification models\n\n- Evaluation metrics make sense (maintaining accuracy, investigating reliability)\n\n- Ablations are interesting and cover initial questions from reviewing the results section", "weaknesses": "- Since you are using the CLIP text encoder, I wish there was more exploration in that space. I think a natural question that is in-scope is how robust are CLIP models both for zeroshot and fine-tuning to hardware failure.\n- Evaluation is limited to just ImageNet\n- Confidence (top2diff) might not be an informative metric and doesn't necessarily make a better model - perhaps something related to calibration could be helpful", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**q1: How robust are CLIP models to hardware failure?**\n\n**a1:** We perform error injection on the CLIP model in the zero-shot setting below and compare it against the baseline (a standard model trained on ImageNet in a supervised manner). We use the resnet50 backbone version of CLIP.\n\n|Backbone|Acc. Baseline|Acc. CLIP ZeroShot|Improvement in Reliability (Last Layer)|Improvement in Reliability (Overall)|Improvement in Top2Diff|\n|-|:-:|:-:|:-:|:-:|:-:|\n|ResNet-50|75.64|58.18|-5.38x|-3.71x|-2.85%|\n\nWe see that zero-shot CLIP has much lower resilience than the supervised trained model on ImageNet. We believe that two reasons contribute to this. Firstly, the CLIP has not seen the exact dataset distribution like the supervised baseline. And Secondly, the CLIP model uses both a vision and text encoder during inference. So it has a larger number of parameters that are susceptible to bit flips that can cause a misclassification.\n\n**q2: Beyond ImageNet evaluation.**\n\n**a2:** We thank the reviewer for highlighting this point and have included additional evaluation spanning multiple datasets (CIFAR10, CIFAR100, Food101, and STL10) for two networks: ResNet-50 [1] and FocalNet-T [2]. Our results, shown below in the table, validate that our technique is general and can work across an array of model types and datasets. Furthermore, as we have shown in the paper, we did not have to modify any hyperparameters in the process, suggesting the ease of our technique as well as the increased benefit from a reliability point of view. Additionally, adding these new datasets further support our claims made in Section 7.4 of the paper, that our technique has negligible impact on model training accuracy, whilst still providing us with a large upside in resilience.\n\n|Dataset|Backbone|Acc. Baseline|Acc. Ours|Improvement in Reliability (Last Layer)|Improvement in Reliability (Overall)|Improvement in Top2Diff|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|CIFAR10|ResNet-50|95.07|95.29|2.04x|1.71x|6.70%|\n|CIFAR10|FocalNet-T|94.76|94.94|2.47x|1.30x|3.58%|\n|CIFAR100|ResNet-50|78.23|78.53|2.19x|1.65x|3.69%|\n|CIFAR100|FocalNet-T|77.06|79.21|3.21x|1.58x|2.90%|\n|FOOD101|ResNet-50|83.13|83.97|2.66x|2.15x|2.78%|\n|FOOD101|FocalNet-T|85.64|85.91|3.28x|2.85x|1.70%|\n|STL10|ResNet-50|47.73|52.68|2.10x|1.91x|2.45%|\n|STL10|FocalNet-T|62.74|63.78|2.23x|1.72x|1.96%|\n\n**q3: Confidence (Top2Diff) as an informative metric for improving a model?**\n\n**a3:** To clarify: Top2Diff is an *analysis* metric, and it is measured post-training (as the authors of [39] suggest), as opposed to an optimization/cost metric *for* training.  The metric itself points to the propensity that an error could sufficiently cause a misclassification at the output, and as such, increasing the gap between the first and second-class confidence, a hardware error manifestation is less likely to corrupt the output. Our insight that Top2Diff can be used to understand the relationship between resilience and accuracy is strongly supported by our original experiments in Section 7.4, plus the addition of the new datasets which we thank the reviewer for suggesting we include in our paper.\n\nAdditionally, with regard to model calibration, we compare and contrast Model Calibration with the Top2Diff Metric in the table below:\n\n|Aspect|Model Calibration|Top2Diff|\n|-|-|-|\n|Nature|Technique to adjust predicted probabilities to match true probabilities.|Analysis metric quantifying the difference between the top two predicted class probabilities.|\n|Purpose|Ensure well-calibrated predicted probabilities for accurate confidence estimates.|Indicate susceptibility of predictions to errors and assess the resilience-accuracy relationship.|\n|Timing|Performed post-training to refine predicted probabilities.|Measured post-training to analyze model behavior.|\n|Impact|Refines predicted probabilities to align with true probabilities.|Quantifies potential impact of errors on predictions and resilience-accuracy trade-off.|\n\nWhile both calibration and Top2Diff address the reliability and accuracy of machine learning models, calibration focuses on refining the probability estimates to align with reality, while Top2Diff serves as a post-training metric to measure the potential impact of errors on the model's predictions and understand its resilience-accuracy trade-off. We would like to clarify that while calibration could potentially be used as an inference-based technique to improve model resilience, our goal as outlined in Lines 43-47 was to introduce a low-entry and entirely training-based routine to reduce the inherent and underlying vulnerability of a model significantly, after which many inference-side techniques could potentially be appended for even stronger resilience (including calibration, as proposed by this reviewer, or other selective protection mechanisms as outlined in Lines 39-43).\n\n**References:**\n\n[1] K. He, X. Zhang, S. Ren, and J. Sun, ‘Deep Residual Learning for Image Recognition’, in CVPR, 2016.\n\n[2] J. Yang, C. Li, X. Dai, and J. Gao, ‘Focal Modulation Networks’, in NeurIPS, 2022.\n\n[39] A. Mahmoud et. al., ‘Optimizing Selective Protection for CNN Resilience’, in ISSRE, 2021."}, {"review_id": "upF6qfFxCT", "reviewer": "Reviewer_DM9r", "summary": "This paper provides a method to enhance the reliability of image classification models against hardware errors. For this purpose, the authors propose to combine textual and visual information to improve the reliability of neural networks by up to $14\\times$, in comparison with traditional error detection and correction techniques. The authors verify their method on ImageNet classification with several models.", "questions": "Could the author explain how to apply the proposed method for cases when the image contains information outside the pretrained classes such as ImageNet in the paper?", "limitations": "The authors did not talk about limitations, and the question asked above can be one limitation of the proposed method.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method to utilize texture information to enhance reliability of vision models is interesting and the author conducted some experiments on ImageNet classification with several models.", "weaknesses": "The proposed method requires textural information during inference, and thus the set of textural information collections would determine the final performance. For example, if the input image contains information outside those classes in ImageNet dataset, it is unclear how to use the proposed method during inference, except for retraining the models. Also, the authors only verify on early CNN architectures, not including efficient models or attention models, so the effectiveness of the proposed method might be questionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**q1: Does the proposed method require textual information during inference?**\n\n**a1:** No, the model only uses the textual information during *training*, to make it more resilient. Please note that in this paper we focus on a *closed-set* classification setting. We only use the text information to provide a rich initialization to the additional projection head. During inference, the model *only* uses the input image to perform its classification. As such, our technique produces an entirely drop-in solution where legacy models can be replaced with our new model, without breaking any previous abstractions of a system at inference time.\n\n**q2: Evaluation on newer models**\n\n**a2:** We sincerely thank the reviewer for their suggestion to help improve this work. Included below, we have evaluated our proposed method on recent and state-of-the-art image classification models, to complement our CNN-based evaluation. Additionally, adding these models to our work strongly supports our original claims in the paper that our technique is general and can support any vision classification model type where we have the training recipe available. We find that the new models agree with our original assessments from the paper, that our technique not only helps improve the reliability of the last layer and the overall model but also has a negligible impact on the accuracy (and in some cases actually improves it), as was a primary goal in this research paper.\n\n|Backbone|Acc. Baseline|Acc. Ours|Improvement in Reliability (Last Layer)|Improvement in Reliability (Overall)|Improvement in Top2Diff|\n|-|:-:|:-:|:-:|:-:|:-:|\n|FocalNet-T [1] (NeurIPS'22)|80.23|80.77|3.87x|2.61x|2.61%|\n|FocalNet-S [1] (NeurIPS'22)|82.01|82.52|4.73x|3.50x|3.10%|\n|Swin-V2-T [2] (CVPR'22)|80.97|80.02|1.65x|1.07x|2.85%|\n|Swin-V2-S [2] (CVPR'22)|82.71|82.86|3.51x|2.60x|3.04%|\n|MaxVit-T [3] (ECCV'22)|82.98|83.08|3.38x|2.63x|2.62%|\n|MobileNet-V2 [4] (CVPR'18)|71.87|71.83|3.92x|2.43x|5.36%|\n\n**q3: How does the proposed method work when images contain information outside pre-trained classes?**\n\n**a3:** To clarify, we do not work in an *open-set/zero-shot classification* setting. The proposed technique aims at improving the resilience of image classification models in the *closed-set classification* supervised setting. We only use the textual features to provide a rich initialization to the new projection head added to the network which improves the overall resilience of the network. Afterward, we follow a standard supervised training and testing setting.\n\n**References:**\n\n[1] J. Yang, C. Li, X. Dai, and J. Gao, ‘Focal Modulation Networks’, in NeurIPS, 2022.\n\n[2] Z. Liu et al., ‘Swin Transformer V2: Scaling Up Capacity and Resolution’, in CVPR, 2022.\n\n[3] Z. Tu et al., ‘MaxViT: Multi-Axis Vision Transformer’, in ECCV, 2022.\n\n[4] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, ‘MobileNetV2: Inverted Residuals and Linear Bottlenecks’, in CVPR, 2018."}, {"review_id": "fPgmrxqFcQ", "reviewer": "Reviewer_eFF3", "summary": "This paper presents a software-based approach to enhance classifier resilience to hardware errors. It leverages GPT-3 to enhance text descriptions for target classes, followed by utilizing the CLIP text encoder to generate text embeddings. These embeddings initialize the classifier head, enabling it to learn robust representations by leveraging the generalization abilities of CLIP and GPT-3.", "questions": "1. Do you train the classifier head, i.e., the projection layer, during training? If yes, do the head and the backbone use the same learning rate? If the head is updated significantly during training, would it lose the ability to guide the backbone learning representations?\n\n2. In Table 1, why do some backbones (ResNet-18 and ResNet-34) have increased parameters, whereas others' parameters reduce?\n\n3. For Figure 3, do you use the training set or validation set of ImageNet? Which set makes more sense for this purpose? Why use the maximum absolute value of one neuron rather than the average of absolute values of all neurons of one layer? The maximum value may favor more outliers.", "limitations": "It would be better to use some more recent vision backbones such as BEiT, DINO-v2, MAE, and Swin Transformer.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Using vision-language models to improve model resistance to hardware errors is an interesting idea.\n\n2. The method is simple and effective. Experiments on ImageNet with VGG and ResNet show the model reliability increases significantly.\n\n3. The paper is well-written and easy to follow. Figure 1 provides the data shapes at each step, making it easy to understand.\n\n4. The experiment section includes rich ablation studies and intuitive explanations.", "weaknesses": "1. In Line 216, the delta isn't converted to a math symbol.\n\n2. It seems that Top2Diff is not used correctly. The reference paper \"Optimizing Selective Protection for CNN Resilience\" defines Top2Diff as the difference between the top two class confidences. However, Line 231 assumes the top-1 class is a correct prediction. We don't know ground-truth labels during model deployment, so we can't tell whether the top-1 prediction is correct. But we still can compute Top2Diff, right? This is also related to the ablation study in Section 7.4. When removing images with certain Top2Diff,  do you remove only the images with correct predictions or all the images, no matter whether the predictions are correct?\n\n3. The experiments don't use more recent backbones such as BEiT and Swin Transformer. ResNet and VGG are kind of out-of-dated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**q1: Top2Diff Clarifications**\n\n**a1:** We thank the reviewer for highlighting this point. To clarify, we use the same exact procedure for hardware resiliency evaluation as the referenced paper [39]. It is true that we do not have a “ground truth” at runtime, but [39] claims that a strong indication of the probability an error occurs is the difference between the top 2 classes. During the evaluation, we focus only on the “correct” subset of images to perform error injections in, because it does not make sense to evaluate whether an error changed an incorrect class into a correct class. In deployment time, where we do not have the ground truth available, we would rely on our analysis of the model’s robustness, but would also need correction mechanisms. While our proposed technique does not propose a detection-and-recovery mechanism like [39], we introduce a technique that reduces the baseline probability of errors occurring in the first place, which would subsequently help lower the cost of techniques such as ILR and FLR proposed by [39].\n\n**q2: Evaluation on newer models**\n\n**a2:** We sincerely thank the reviewer for their suggestion to help improve this work. Included below, we have evaluated our proposed method on recent and state-of-the-art image classification models, to complement our CNN-based evaluation. Additionally, adding these models to our work strongly supports our original claims in the paper that our technique is general and can support any vision classification model type where we have the training recipe available. We find that the new models agree with our original assessments from the paper, that our technique not only helps improve the reliability of the last layer and the overall model but also has a negligible impact on the accuracy (and in some cases actually improves it), as was a primary goal in this research paper.\n\n|Backbone|Acc. Baseline|Acc. Ours|Improvement in Reliability (Last Layer)|Improvement in Reliability (Overall)|Improvement in Top2Diff|\n|-|:-:|:-:|:-:|:-:|:-:|\n|FocalNet-T [1] (NeurIPS'22)|80.23|80.77|3.87x|2.61x|2.61%|\n|FocalNet-S [1] (NeurIPS'22)|82.01|82.52|4.73x|3.50x|3.10%|\n|Swin-V2-T [2] (CVPR'22)|80.97|80.02|1.65x|1.07x|2.85%|\n|Swin-V2-S [2] (CVPR'22)|82.71|82.86|3.51x|2.60x|3.04%|\n|MaxVit-T [3] (ECCV'22)|82.98|83.08|3.38x|2.63x|2.62%|\n|MobileNet-V2 [4] (CVPR'18)|71.87|71.83|3.92x|2.43x|5.36%|\n\n**q3: Do you train the classifier head during training?**\n\n**a3:** Yes, we do train the classifier head during training, and yes, we use the same learning rate for both the head and the backbone to ensure we stay consistent with the overall training recipe.\n\n**q4: Why does ResNet-18,34 increase params, while ResNet50 decrease (Table 1)?**\n\n**a4:** This goes back to the DNN architecture design for different ResNet versions. As explained in lines 251-256, the second-to-last layer feeding into the projection layer impacts the change in the number of parameters for our model. If we take a look at Table 1 of the ResNet paper [5], we see that ResNet18 and ResNet34 end on a 3x3 512 layer, while the other ResNets end on a 1x1 2048 layer. More specifically:\n\nIn case of the baseline Resnet18/34 last layer parameters are 512x1000 (num classes)  = 512000 \\\nIn case of our Resnet18/34 last layer parameters are 512x512 (embed size) + 512x1000 (proj layer)  = 262144 + 512000 \\\nHence params are increased by 0.26M\n\nIn case of the baseline Resnet50/101/152 last layer parameters are 2048x1000 (num classes)  = 2048000 \\\nIn case of our Resnet18/34 last layer parameters are 2048x512 (embed size) + 512x1000 (proj layer)  = 1048576 + 512000 \\\nHence params are decreased by 0.49M\n\n**q5: Fig-3 elaboration:**\n\n**a5:** We use the validation set, as we are evaluating the network’s reliability outside the training regime. We conducted an analysis of the mean magnitude of neuron values for each layer, as per the reviewer's request. Complimenting Figure 3 in the main paper which uses absolute max value, we added a Figure in the rebuttal PDF showing the difference between the mean absolute value per layer for the baseline and our model. We see a similar trend between both figures, with the baseline model having a large value, especially in the final layer. This rationale is expounded upon in detail in Appendix D.\n\n**References:**\n\n[1] J. Yang, C. Li, X. Dai, and J. Gao, ‘Focal Modulation Networks’, in NeurIPS, 2022.\n\n[2] Z. Liu et al., ‘Swin Transformer V2: Scaling Up Capacity and Resolution’, in CVPR, 2022.\n\n[3] Z. Tu et al., ‘MaxViT: Multi-Axis Vision Transformer’, in ECCV, 2022.\n\n[4] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, ‘MobileNetV2: Inverted Residuals and Linear Bottlenecks’, in CVPR, 2018.\n\n[5] K. He, X. Zhang, S. Ren, and J. Sun, ‘Deep Residual Learning for Image Recognition’, in CVPR, 2016.\n\n[39] A. Mahmoud et. al., ‘Optimizing Selective Protection for CNN Resilience’, in ISSRE, 2021."}, {"review_id": "5pOTpF8Ztg", "reviewer": "Reviewer_8rdZ", "summary": "The paper studies the hardware resiliency of image classifiers, that is misclassification rates under random-bit flipping. They show that initializing the last classification layer using CLIP embeddings can greatly improve hardware resiliency.\n\nTo obtain CLIP embeddings, for every class, GPT3 produces C different text prompts which are then averaged to produce a single embedding per class. In specific, the original classification layer of dim $B \\times C$ is replaced by a latent layer of dim $B \\times E$ and a projection layer of dim $E \\times C$. The projection layer is initialized with embeddings from a CLIP text encoder. Results are shown on VGG and ResNet on 2 metrics (Top2Diff and $\\Delta$ Loss which is difference in cross-entropy between the original prediction and misclassified prediction). Post hoc, the authors see that their proposed method has better saliency properties and lower last layer activations. ", "questions": "Metrics\n--------\n* It is a bit unclear to me what the \"GoldenEye\" benchmark actually is. The authors say \"We use PyTorchFI [36] to perform the random bit flip, and we perform 4096 unique error injection experiments per layer, totaling more than 3.6 million experiments across all our models\". Does the GoldenEye benchmark prescribe the exact way this error injection is done?\n* I suggest that the authors report the classification flip rate as well in Table 1. The 14x number is interesting but is a bit unintuitive. Measuring how many decisions are flipped when the bit is flipped may be more intuitive.\n* The Top2Diff metric is unclear. \"Top2Diff metric is simply the difference in classification accuracy between the top (correct) class and the\nsecond-highest class\". I'm unsure what the classification accuracy of the second highest class means. Did you mean logits?\n\nExperiments\n---------------\n* Major: What is the difference between RandomInit in Table 2 and the baseline in Table 1? If the projection layer is initialized randomly, then  does this not default to a baseline classifier?\n*  Is layer 53 in Figure 3, the final classification layer meaning that the ResNet-50 baseline is highly confident? If yes, can the authors provide some intution on why small perturbations at the output layer changes its decision drastically?\n* According to Figure 3, can an explicit additional L2 loss term at the outputs of the ResNet-50 baseline have the same effect as clip initialization?\n* L305: The authors say that they inject 2000 random errors in weights across the network. Is the output saliency map averaged across these 2000 different maps?\n* It might be nice to show some experiments on Vision Transformers, but this is more of a nice to have.", "limitations": "No negative societal impact.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper proposes a simple change with low overhead that improves hardware reliability rates.\n* Since the clip embedding dimension is generally lower than classifier bottleneck dimension, a side effect is that the number of parameters are also reduced.\n* Results are shown on a number of different convolutional architectures.", "weaknesses": "The positioning of the paper is a bit confusing. While finetuning pretrained clip models on ImageNet is not new in some sense, the paper seems to make the claim that for eg, in Figure 1 and Section 4.2 that this work proposes a new approach. The paper may be better positioned as an investigation of CLIP finetuning for hardware reliability. I would like to discuss with other reviewers about this aspect.\n\nI have a few more questions about the clarity of some metrics and experiments used in this work, which may be useful to the broader computer vision community. Please see the questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**q1:Positioning of the paper**\n\n**a1:** We would like to clarify that we don't exactly *finetune* CLIP models. We augment a standard, randomly initialized image model, with an additional projection head initialized with rich textual features. The rest of the model is randomly initialized as with any standard image classification training setup on the ImageNet dataset. Our major contribution is to the analysis that this simple additional projection + initialization to any image classification architecture can immediately improve the model resilience to hardware errors.\n\nWe thank the reviewer for pointing this out. We will further clarify in the final version of the paper our major contribution focusing on the analysis of inherent resilience gains obtained by utilizing rich text features.\n\n**q2: GoldenEye clarifications**\n\n**a2:** Yes, we use the GoldenEye benchmark as provided in their repository for our evaluation. At a high level, GoldenEye is a wrapper around the PyTorchFI error injection framework plus providing us with an analysis of the results in terms of DeltaLoss resilience. To that end, we configure GoldenEye to perform 4096 unique error injection experiments per layer and use it to compare a baseline model with our proposed technique in terms of reliability. By using GoldenEye, the 3.6 million error injection experiments are performed under the hood to provide us with strong statistical guarantees on the model's resilience to hardware errors.\n\n**q3: Classification Bit Rate vs DeltaLoss**\n\n**a3:** The GoldenEye benchmark provides us with both the classification bit rate and the DeltaLoss metric as well. We report the DeltaLoss metric because the authors of [39] show that it converges asymptotically faster with fewer error injections compared to the classification bit rate, signifying that it is a stronger and more preferable metric for hardware resiliency analysis. In terms of the 14x number: it is the same whether reported through DeltaLoss or classical flip rate - the only difference is that the statistical guarantees are stronger via DeltaLoss at fewer error injection experiments (Figure 3 of [39] helps highlight this point).\n\n**q4: Top2Diff Metric is unclear**\n\n**a4:** The Top2Diff metric is computed for the *softmax*, not the logits. We apologize for the unclarity and will fix it.\n\n**q5: Difference between RandomInit in Table 2 and baseline in Table 1?**\n\n**a5:** The baseline in Table 1 is the default model as defined in the pytorch/torchvision library. The RandomInit model in Table 2 is our design with the additional projection, but no textual initialization to ablate whether the textual initialization helps or not.\n\n**q6: Figure 3, last layer Clarifications**\n\n**a6:** To clarify: Figure 3 does not refer to confidence. It shows empirical data for the maximum neuron value across the dataset, which we correlate with the single-bit flip error model as a reason for a high probability of a new (erroneous) value corrupting the output. We provide more detail in Appendix D of the Supplementary material.\n\n**q7:** Can an explicit L2 loss term be used at the output instead?\n\n**a7:** To see if adding an explicit L2 loss on the output helps, we run an experiment with this additional loss on the baseline ResNet-50 output. We show the results for this in the table below:\n\n|Backbone|Acc. Baseline|Acc. Last Layer L2 Norm|Improvement in Reliability (Last Layer)|Improvement in Reliability (Overall)| Improvement in Top2Diff|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|ResNet-50|75.64|75.13|-1.14x|-0.82x|-0.26%|\n\nWe see that the explicit loss term reduces accuracy/resilience instead of improving it. We believe that the reason for this is that the training scheme already uses weight decay, which is also an implicit penalty on the model weights. Hence, the additional penalty does not really help much in terms of resilience.\n\n**q8: Figure 2 Clarifications**\n\n**a8:** For this experiment, there is only one saliency map generated by performing 2000 perturbations in the inference of an image. We chose this as a “large” number to show visual differences in the saliency between the two techniques. No averaging is needed. For error injection experiments, we perform a single bit flip in the entire network, and show that it can, in fact, alter the final classification.\n\n**q9: Additional Models**\n\n**a9:** We sincerely thank the reviewer for their suggestion to help improve this work. Included below, we have evaluated our proposed method on recent and state-of-the-art image classification models, to complement our CNN-based evaluation. Additionally, adding these models to our work strongly supports our original claims in the paper that our technique is general and can support any vision classification model type where we have the training recipe available.\n\n|Backbone|Acc. Baseline|Acc. Ours|Improvement in Reliability (Last Layer)|Improvement in Reliability (Overall)|Improvement in Top2Diff|\n|-|:-:|:-:|:-:|:-:|:-:|\n|FocalNet-T [1] (NeurIPS'22)|80.23|80.77|3.87x|2.61x|2.61%|\n|FocalNet-S [1] (NeurIPS'22)|82.01|82.52|4.73x|3.50x|3.10%|\n|Swin-V2-T [2] (CVPR'22)|80.97|80.02|1.65x|1.07x|2.85%|\n|Swin-V2-S [2] (CVPR'22)|82.71|82.86|3.51x|2.60x|3.04%|\n|MaxVit-T [3] (ECCV'22)|82.98|83.08|3.38x|2.63x|2.62%|\n|MobileNet-V2 [4] (CVPR'18)|71.87|71.83|3.92x|2.43x|5.36%|\n\n**References**\n\n[1] J. Yang, C. Li, X. Dai, and J. Gao, ‘Focal Modulation Networks’, in NeurIPS, 2022.\n\n[2] Z. Liu et al., ‘Swin Transformer V2: Scaling Up Capacity and Resolution’, in CVPR, 2022.\n\n[3] Z. Tu et al., ‘MaxViT: Multi-Axis Vision Transformer’, in ECCV, 2022.\n\n[4] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, ‘MobileNetV2: Inverted Residuals and Linear Bottlenecks’, in CVPR, 2018.\n\n[39] A. Mahmoud et. al., ‘Optimizing Selective Protection for CNN Resilience’, in ISSRE, 2021."}]

Author rebuttal:
We thank the reviewers for all their feedback and comments. We respond to common themes across all reviewers in this global rebuttal, and then answer further specific questions per reviewer individually.

**Q1. Vision model evaluation for hardware resilience, and the use of CLIP-text features**

One of the fundamental points of this work is that we use textual features to enhance the resilience of *closed-set image classification models*. One common misconception we gathered was about the use of textual information during inference. Rather, we operate entirely within a closed-set regime in this paper and only use the textual information from CLIP only to initialize the projection layer during the *training* of our model. Thus, at deployment time, there is no need for explicit textual information, as that has already been incorporated into the model via our technique of pre-training the final layer of the model. To that end, our proposed model can directly replace any classical model and would have the same pros and cons (in the context of in-distribution and out-of-distribution performance), but operate at a higher level of resilience to single-bit perturbations and hardware errors.

In summary, our approach involves refraining from fine-tuning CLIP models directly. Instead, we enhance a conventional image model, which starts as a randomly initialized framework, by incorporating an extra projection head initialized with textually enriched features. The remaining segments of the model are also subject to random initialization, following the conventional image classification training process using the ImageNet dataset. Notably, our primary contribution lies in demonstrating that this uncomplicated augmentation through an extra projection and initialization step, when applied to any image classification architecture, can swiftly enhance the model's ability to withstand hardware errors.

**Q2. Top2Diff Discussion**

We credit the authors of [39] for the introduction of the Top2Diff metric in the context of hardware reliability and would like to further clarify its use here. Top2Diff is a measurement between the top softmax value and the 2nd highest softmax value. In Section IV-B of [39], the authors show that this metric is good for hardware reliability, intuitively because it indicates a higher threshold that an error needs to overcome for the output to change its classification. For example, a Truck with 52% confidence and Bird with 48% confidence indicates a Top2Diff of 4%, while a 90% Truck to 10% Bird indicates an 80% Top2Diff. The idea is that “overcoming” a 4% difference is a much lower threshold than an 80% difference, implying that a single bit flip during the first Truck example is more likely to produce an output misclassification.

To further clarify, we do not use Top2Diff during training at all, and simply measure it as a proxy to understand the reliability of a model to hardware errors. The error injection experiments produced can be considered the primary metric for resilience (which is measured by DeltaLoss), and Top2Diff is helpful to gather insight into the model only.

**Q3. Additional Results**

We included additional results as requested by multiple reviewers, including:
- Adding more models including modern transformers (FocalNet, Swin-V2, MaxVit, and MobileNet-V2)
- Adding additional datasets (CIFAR10/100, STL10, and Food101)
- Including an additional metric (absolute mean) to complement our original metric (absolute max) in Figure 3 as requested by one of the reviewers. This is presented in the rebuttal PDF file.

|Backbone|Acc. Baseline|Acc. Ours|Improvement in Reliability (Last Layer)|Improvement in Reliability (Overall)|Improvement in Top2Diff|
|-|:-:|:-:|:-:|:-:|:-:|
|FocalNet-T [1] (NeurIPS'22)|80.23|80.77|3.87x|2.61x|2.61%|
|FocalNet-S [1] (NeurIPS'22)|82.01|82.52|4.73x|3.50x|3.10%|
|Swin-V2-T [2] (CVPR'22)|80.97|80.02|1.65x|1.07x|2.85%|
|Swin-V2-S [2] (CVPR'22)|82.71|82.86|3.51x|2.60x|3.04%|
|MaxVit-T [3] (ECCV'22)|82.98|83.08|3.38x|2.63x|2.62%|
|MobileNet-V2 [4] (CVPR'18)|71.87|71.83|3.92x|2.43x|5.36%|


|Dataset|Backbone|Acc. Baseline|Acc. Ours|Improvement in Reliability (Last Layer)|Improvement in Reliability (Overall)|Improvement in Top2Diff|
|-|:-:|:-:|:-:|:-:|:-:|:-:|
|CIFAR10|ResNet-50|95.07|95.29|2.04x|1.71x|6.70%|
|CIFAR10|FocalNet-T|94.76|94.94|2.47x|1.30x|3.58%|
|CIFAR100|ResNet-50|78.23|78.53|2.19x|1.65x|3.69%|
|CIFAR100|FocalNet-T|77.06|79.21|3.21x|1.58x|2.90%|
|FOOD101|ResNet-50|83.13|83.97|2.66x|2.15x|2.78%|
|FOOD101|FocalNet-T|85.64|85.91|3.28x|2.85x|1.70%|
|STL10|ResNet-50|47.73|52.68|2.10x|1.91x|2.45%|
|STL10|FocalNet-T|62.74|63.78|2.23x|1.72x|1.96%|

Meta-review: This paper introduces a method to enhance the reliability of image classification models against hardware errors. They specifically study how they can mitigate random bit flips in image classification models. The reviewers agree this is an important and interesting problem to study. The reviewers agree that this solution seems simple and effective. There is an added bonus that this model has less flops. The authors do a good job setting up the problem of reliability and provide + motivate a metric to study reliability.
Decision: Accept (poster)