Do Image Classifiers Generalize Across Time?
Authors: Vaishaal Shankar
Achal Dave
Rebecca Roelofs
Deva Ramanan
Ben Recht
Ludwig Schmidt
Venue:  (2020)
Keywords: robustness
image classification
distribution shift

Abstract: We study the robustness of image classifiers to temporal perturbations derived from videos. As part of this study, we construct ImageNet-Vid-Robust and YTBB-Robust, containing a total 57,897 images grouped into 3,139 sets of perceptually similar images. Our datasets were derived from ImageNet-Vid and Youtube-BB respectively and thoroughly re-annotated by human experts for image similarity. We evaluate a diverse array of classifiers pre-trained on ImageNet and show a median classification accuracy drop of 16 and 10 percent on our two datasets. Additionally, we evaluate three detection models and show that natural perturbations induce both classification as well as localization errors, leading to a median drop in detection mAP of 14 points. Our analysis demonstrates that perturbations occurring naturally in videos pose a substantial and realistic challenge to deploying convolutional neural networks in environments that require both reliable and low-latency predictions.

Under review as a conference paper at ICLR 2020
Do IMAGE CLASSIFIERS GENERALIZE ACROSS TIME?
Anonymous authors
Paper under double-blind review
ABSTRACT
We study the robustness of image classifiers to temporal perturbations derived from vidcos. As part of this study, we construct ImageNet Moniltt-Vice -Vis-Robust containing total 57,897 images grouped into 3,139 sets of perceptually similar images. Our datasets were derived from ImageNet- Vid and Youtube-BB BB respectively and thoroughly re-annotated by human experts for image similarity. We evaluate a diverse array of classifiers pre- pre-trained on ImageNet and show a median classification accuracy drop of 16 and 10 percent on our two datasets. Additionally, we evaluate three detection models and show that natural perturbations induce both classification as well as localization errors, leading to median drop in detection mAP of 14 points. Our analysis demonstrates that perturbations occurring naturally in videos pose a substantial and realistic challenge to deploying convolutional neural networks in environments that require both reliable and low-latency predictions.
and
INTRODUCTION
Convolutional neural networks (CNNs) still exhibit many troubling failure modes. At one extreme, fp-adversarial examples cause large drops in accuracy for of-te-of models while relying only on visually imperceptible changes to the input image (Goodfellow et al.] 2014) [Bigiio and Roli 2018). However, this failure mode usually does not pose a problem outside fully adversarial context because carefully crafted Ey-perturbations are unlikely to occur naturally in the real world.
To study more realistic failure modes, researchers have investigated benign image perturbations such as rotations& & translations, colorspace changes, and various image corruptions (Fawzi and Frossard
2013; Engstrom et al., 2017] Fawzi and Frossard 2015] Hendrycks and Dietterich] (019). However, is still unclear whether these perturbations reflect the robustness challenges arising real data since the perturbations also rely on synthetic image modifications.
Recent work has therefore turned to videos as source of naturally occurring perturbations of images (Zheng al.) [0166] Azulay and Weiss 2018; Gu et al. 2201). In contrast to other failure modes, the perturbed images are taken from existing image data without further modifications that make the task more difficult. As result, robustness to such perturbations directly corresponds to performance improvements on real data.
However, it is currently unclear to what extent such video perturbations pose significant robustness challenge. Azulay and Weiss (2018) 8) and Zheng et al. (2016) only provide anecdotal evidence from a small number of videos. (uu et al.] (2019) go beyond individual videos and utilize a large video dataset (Real et al. 2017) ) in order to measure the effect of video perturbations more quantitatively. In their evaluation, the best image classifiers lose about 3% accuracy for video frames up to 0.3 seconds away. However, the authors did not employ humans to review the frames in their videos. Hence the accuracy drop could also be caused by significant changes in the video frames (c.g., due to fast camera or object motion). Since the 3% accuracy drop is small begin with, it remains unclear whether video perturbations are a robustness challenge for current image classifiers.
We address these issues by conducting a thorough evaluation of robustness to natural perturbations arising in videos. As a comerstone of our investigation, we introduce two test sets for evaluating model robustness: ImageNet-ve -Vid-Robusttt and YT3B-Robust, carefully curated from the ImageNet- Vid and Youtube-BB -B datasets, respectively (Russakovsky et al.) 2015 Real et al. (017). All images in the two datasets were screened by a set of expert labelers to ensure high annotation quality and minimize selection biases that arise when filtering dataset with CNNs. To the best of

---

Under review as a conference paper at ICLR 2020
knowledge these are the first datasets of their kind, containing tens of thousands of images that are human reviewed and grouped into thousands of perceptually similar sets. In total, our datasets contain 3,139 sets of temporally adjacent and visually similar images (57,897 images total).
We then utilize these datasets to measure the accuracy of current CNNs to small, naturally oc- curring perturbations. Our testbed contains over 45 different models, varying both architecture and training methodology (adversarial training, data augmentation, etc.). To better understand the drop in accuracy due natural perturbations, we also introduce a robustness metric that more stringent than those employed in prior work. Under this metric, we find that natural perturba- tions from SaaageNee-Vid--eee dotd-tt and YTBB-Robust induce median accuracy drop of 16% and 10% respectively for classification tasks and a median 14 point drop in mAP for detection Even for the pest-performing classification models, we observe an accuracy drop of 14% for and 8% for YTBB YTBB-Robust.
tasks D Ima -eNet-Vid-Rollle
Our results show that robustness natural perturbations videos is indeed a significant challenge for current CNNs. As these models are increasingly deployed in safety-critical environments that require both high accuracy and low latency (e.g., autonomous vehicles), ensuring reliable predictions on every frame of video is an important direction for future work.
Predicted: domestic
Predicted monkey
Predicted: bird
Predicted domestic
Predicted: turtie
Predicted linard
Classiter
0.5
0.0
bear
rabdor
monkay
domustic oat
dog
betr
squiraal
domastic, cat
rabbi
mankey
0.0
deg
rabdal
nartl
bird
radder
contle
Bird
deg
sputree
anity
anake
liard
turtle
entay
soutral
anaiee
turtle
Hoarr
domextic
domestic
Figure I: Three examples of natural perturbations from nearby video frames and resulting classifier confidences from a ResNet-152 model fine-tuned on ImageNet- Vid. While the images appear almost identical to the human eyc, the classifier confidence changes substantially.
CONSTRUCTING A TEST SET FOR ROBUSTNESS
ImageNet holidd-Rouusttoo and YTBB Robust sourced from videos in the ImageNet- Vid and Youtube-BB datasets (Russakovsky et al. 2015 Real et al. (017). All object classes in ImageNet- Vid and Youtube-Bi are from the WordNet hierarchy (Miller 1995) and direct ancestors of ILSVRC-2012 classes. Using the WordNet hierarchy, we construct a canonical mapping from ILSVRC-2012 classes ImageNet- Vid and Youtube-Bi BB classes, which allows to evaluate off the-the-shelf ILSVRC-2012 models on ImageNet-Vic (iiiidd-Ro Robust and Robus.. We provide more background on the source datsets in Appendix A
2.1 CONSTRUCTING IMAGRNET-VION
AND YTHB-RORUST
Next, describe how extracted sets of naturally perturbed frames from ImageNet- Vid and Youtube-BB to create imageNiett-iid-Robbe would be select set of anchor frames and use temporally adjacent frames in the video with the assumption that such frames contain only small perturbations from the anchor. However, as Fig. 2 illustrates, this assumption frequently violated, especially due fast camera or object motion.
and YTRE-Robust. A straightforward approach
Instead, we first collect preliminary datasets of natural perturbations following the same approach, and then manually review each of the frame sets. For each video, we randomly sample an anchor frame and take k = 10 frames before and after the anchor frame as candidate perturbation images This results in two datasets containing one anchor frame each from 3,139 videos, with approximately 20 candidate perturbation per anchor frame
Wee only evaluated detection ImageNet w-w--vidoooooo Robust bounding- box annotations in Youtube-BB were only pra-eond and dense enough for evaluation.
"For YT'83- Robust we use subset of the anchor frames used by Gu et al. 2019)
"Anchor frames the start end of the video may have less than 20 candidate frames.
2

---

Under review as a conference paper at ICLR 2020
Anchor frame
Discarded frame
Anchor frame
Discarded frame
Anchor frame
Discarded frame
Figure Temporally adjacent frames may not be visually similar. We show three randomly sampled frame pairs where the nearby frame was marked as "dissimilar" to the anchor frame during human review and then discarded from our dataset.
ImageNett-idd-Robus.
Reviewed Accepted
1,314
2,467
Anchor frames
1,109 (84%)
2,030 (82%)
Labels updated
834 (41%)
Reviewed
26,029
45,631
Frame pairs
Accepted
21,070 (80.9%)
36,827 (80.7%)
Table 1: Statistics of MmageNet-vid-Rolle and YTBB- -ebrsst......... For YTBB-Robust, we updated the labels from for 41% % (834) of the accepted anchors due to labeling errors in Youtube-BB.
Next, we curate the dataset with the help of four expert human annotators. The goal of the curation step is to ensure that each anchor frame and its nearby frames are correctly labeled with the same ground truth class, and that the anchor frame and the nearby frames are visually similar.
Denser labels for Youtube-BB, As Youtube- BB contains only single category label per frame at 1 frame per second, annotators first viewed each anchor frame individually and marked any missing labels. In total, annotators corrected the labels for 834 frames, adding an average of 0.5 labels per anchor frame. These labels are then propagated to nearby, unlabeled frames at the native frame rate and verified in the next step. ImageNet- Vid densely labels all classes per frame,
skip this step.
Frame pairs review. Next, for each pair of anchor and candidate perturbation frames, a human annotates (i) whether the pair is correctly labeled in the dataset, and (ii) whether the pair is similar. We took several steps to mitigate the subjectivity of this task and ensure high annotation quality. First, we trained reviewers mark frames as dissimilar if the scene undergoes any of the following transformations: significant motion, significant background change, or significant blur change. We asked reviewers to mark each dissimilar frame with one of these transformations, or "other", and to mark pair of images as dissimilar if a distinctive feature of the object is only visible in one of the two frames (such as the face of a dog). If annotator was unsure about the correct label, she could mark the pair as "unsure". Second, we present only single pair of frames at a time to reviewers because presenting vidcos or groups frames could cause them miss large changes due to the phenomenon of change blindness (Pashler 1988).
Verification. In the previous stage, all annotators were given identical labeling instructions and individually reviewed total of 71,660 images pairs. To increase consistency in annotation, annotators jointly reviewed all frames marked as dissimilar, incorrectly labeled, or "unsure". A frame was only considered similar to its anchor if striet majority of the annotators marked the pair such.
After the reviewing was complete, we discarded all anchor frames and candidate perturbations that annotators marked as dissimilar or incorrectly labeled. The final datasets contain combined total of 3,139 anchor frames with a median of 20 similar frames each.
2.2 THE PM-K EVALUATION METRIC
Given the datasets introduced above, propose a metric to measure model's robustness natural perturbations. In particular, let {ai,.... an) be the set of valid anchor frames our dataset. Let Y (81,-.. ..., yn be the set of labels for A. We let Na((;; be the set of frames marked as similar to anchor frame a, our setting, N₁g is a subset of the 2/ temporally adjacent frames (plus/minus frames from the anchor).
3

---

Under review as a conference paper at ICLR 2020
Classification. Classification accuracy is defined as acCori 1- Zo/1 the standard 0-1 loss function. We define the pm-k analog of accuracy as
Σ
where
pcCp
max (a))
Lo/1
(1)
N
which corresponds to picking the worst frame from each set (a:) before computing accuracy.
Defection. The standard metric for detection is mean average precision (mAP) of the predictions at fixed intersection-over-niis (loU) threshold Lin et al. (2014). We define the pm-k k metric analogous to that for classification: We replace each anchor frame with the nearest frame that minimizes the average precision (AP, averaged over recall thresholds) of the predictions, and compute pm-k as the mAP on these worst-case neighboring frames.
MAIN RESULTS
YTBB-Robust
70
Perturbed Test Accuracy
Perturbed Test Accuracy
60
50
40
50
40
30
40
50
60
70
80
50
60
70
80
90
Original Test Accuracy
Original Test Accuracy
No Accuracy Drap
ILSVRC
noise augmentation
ILSVRC finetuned an ILSVRC-VID-DE ILSVRC finetuned
Linear
adversarial training finetuned an ILSVRC-VID
ILSVRC
ILSVRC
Figure 3: Model accuracy original vs. perturbed images. Each data point corresponds to one model our testbed (shown with 95% Clopper-Pearse confidence intervals). Each perturbed frame taken from ten frame neighborhood of the original frame (approximately 0.3 seconds). All frames were reviewed by humans to confirm visual similarity to the original frames.
We evaluate a testbed of 45 classification and three detection models on gmageeet-iico -iie-eet-id--oubus and YTBB-Robust. We first discuss the various types of classification models evaluated with the pm-k classification metric. Second, we evaluate the performance of detection models on ImageNet- -Vid -Rbuust using use the bounding box annotations inherited from ImageNet- Vid using a variant of pm-k for detection. We then analyze the errors made the detection adversarial examples to isolate the effects of localization errors vs. classification errors.
3.1 CLASSIFICATION
The classification robustness metric is aCCpmk defined in Equation ( For frames with multiple labels, we count a prediction as correct the model predicts any of the correct classes for a frame. In 3 plot the benign accuracy, acCorig. versus the robust accuracy, aCCpmk. for all classification models in our test bed and find that the relationship between acCorig and aCCpmk is approximately linear. This relationship indicates that improvements in the benign accuracy result improvements in the worst-case accuracy, but do not suffice to resolve the accuracy drop due to natural perturbations.
Our test bed consists of five model types with increasing levels of supervision. We present results for representative models from each model type in Table and defer the full classification results table to Appendix B.2
4

---

Under review as a conference paper at ICLR 2020
Table 2: Accuracies of five different model types and the best performing model. The model
architecture is ResNet-50 unless noted otherwise. 'FT" is 'fine-tuning." See Section 3.1 for details.
Accuracy Original
Accuracy Perturbed
Model Type
A
LmageNett-Vid-Robub
Trained on ILSVRC
67.5 [64.7, 70.3]
52.5 [49.5, 55.5]
15.0
+ Noise Augmentation
68.8 [66.0, 71.5]
53.2 [50.2, 56.2]
15.6
fo robustness (ResNext- 101)
54.3 [51.3, 57.2]
40.8 [39.0, 43.7]
12.4
+ FT on ImageNet- Vid
80.8 [78.3, 83.1]
65.7 [62.9, 68.5]
15.1
FT on ImageNet-Vin (ResNet-152)
84.8 [82.5, 86.8]
70.2 [67.4, 72.8]
14.6
+ FT on ImageNet-Vid-Det
77.6 [75.1, 80.0]
65.4 [62.5, 68.1]
12.3
YTBB Robust
Trained on ILSVRC
57.0 [54.9, 59.2]
43.8
[41.7, 46.01
13.2
+ Noise Augmentation
62.3 [60.2, 64.4]
45.7 [43.5, 47.91
16.6
Loe robustness (ResNext-101)
53.6 [51.4, 55.8]
43.2 [41.0, 45.3]
10.4
+ FT on Youtube-BB
91.4 [90.1, 92.6]
82.0 [80.3, 83.7]
9.4
FT on Youtube-BB (ResNet-152)
92.9 [91.6, 93.9]
84.7 [83.0, 86.2]
8.2
ILSVRC Trained The WordNet hierarchy enables us to repurpose models trained for the 1,000 class ILSVRC dataset ImageNet -ie-eeeeet-ii-obuus- and YTBB-Robust (see Appendix [AII) We evaluate a wide array of ILSVRC-2012 models (available from Cadene) against our natural perturbations. Since these datasets present a substantial distribution shift from the original ILSVRC- 2012 validation, we expect the benign accuracy acCorig to be lower than the comparable accuracy on the ILSVRC-2012 validation set. However, our main interest here is in the difference between the original and perturbed accuracies acCorig pm A small drop in accuracy would indicate that the model is robust to small changes that occur naturally in videos. Instead, we find significant drops of 15.0% and 13.2% in accuracy on our two datasets, indicating sensitivity to such changes.
Noise augmentation One hypothesis for the accuracy drop from original to perturbed accuracy is that subtle artifacts and corruptions introduced by video compression schemes could degrade performance when evaluating on these corrupted frames. The worst-case nature of the pm-k metric could then be focusing on these corrupted frames. One model for these corruptions are the perturba- tions introduced in [Hendrycks and Dietterich (2019). To test this hypothesis, evaluate models augmented with a subset of the perturbations (exactly one of: Gaussian noise, Gaussian blur, shot noise, contrast change, impulse noise, or JPEG compression). We found that these augmentation schemes did not improve robustness against our perturbations substantially, and still result in accuracy drop of 15.6% and 16.6% on the two datasets.
Ex robustness. We evaluate the model from Xie et al. (2018), which currently performs best against lxx attacks on ImageNet. We find that this model has a smaller accuracy drop than the two aforementioned model types on both datasets. However, we note that the robust model achieves significantly lower original and perturbed accuracy than either of the two model types above, and the robustness gain is modest (3% compared to models of similar benign accuracy).
Fine-tuning on video frames. To adapt to the new class vocabulary and the video domain, we fine-tune several network architectures the ImageNet- Vid and Youtube-BB training sets. For Youtube-BB, BBB we train on the anchor frames used for training in Gu et (201)), and for ImageNet- Vid we use all frames in the training set. We provide hyperparameters for all models in Appendix K The resulting models significantly improve in accuracy over their ILSVRC pre-trained counterparts (e.g., 13% on Seiigeeee--ii-R-ooee provement in accuracy results in modest improvement in the accuracy drop for YTBB-Robust. but a finetuned ResNet-50 still suffers from significant 9.4% drop. On ImageNet-va vineeet--iiieeeee Robbusst there is almost no change in the accuracy drop from 15.0% to 15.1 %.
and 34% on YTBB-Robust for ResNet-50). This im-
Fine-tuning for detection on video frames. We further analyze whether additional supervision in the form of bounding box annotations improves robustness. To this end, we train the Faster R-CNN detection model Ren et al. (2015) with a ResNet-50 backbone on ImageNet- Vid. Following standard practice, the detection backhone is pre-trained on I To evaluate this detector
5

---

Under review as a conference paper at ICLR 2020
for classification, assign the class with the most confident bounding box as label to the image. We find that this transformation reduces accuracy compared to the model trained for classification (77.6% vs. 80.8%). While there is a slight reduction in the accuracy drop caused by natural perturbations, the reduction well within the erorr bars for this test set.
3.2 DETECTION
We further study the impact of natural perturbations object detection. Specifically, we report results for two related tasks: object localization and detection. Object detection is the standard computer vision task of correctly classifying object and finding the coordinates of tight bounding box containing the object. "Object localization", meanwhile, refers to only the subtask of finding the bounding box, without attempting to correctly classify the object.
We present results ImageNet- di--d-RRoo -o-obsst, which contains dense bounding box labels unlike Youtube- BB, which only labels boxes at frame per second. We use the popular Faster R-CNN Ren al. (2015) and R-FCN Dai al. (2016); Xiao and Jae Lee (2018) architectures for object detection and localization and report results in Table 3 For the R-FCN architecture, we use the model from Xiao and Jae Lee (2018 2218 We first the significant drop in mAP of 12- 15 points for object detection due to perturbed frames for both the Faster R-CNN and R-FCN architectures. Next, we show that localization is indeed easier than detection, as the mAP higher for localization than for detection (e.g.. 76.6 62.8 for Faster R-CNN with ResNet-50 backhone). Perhaps surprisingly, however, switching to the localization task does not improve the drop between original and perturbed frames, indicating that natural perturbations induce both classification and localization errors. We show examples of detection failures Figure
airplane 0.99
matoreycla
0.81
airplane
horse 0.53
lizard 0.65 motoroyale
Sheop 0.88
Figure Naturally perturhed examples for detection. Red boxes indicate false positives; green boxes indicate true positives, white boxes are ground truth. Classification enrors are common failures, such as the fox on the left, which classified correctly in the anchor frame, and misclassified as sheep in nearby frame. However, detection models also have localization errors, where the object of interest not correctly localized in addition to being misclassified, such as the airplane (middle) and the motorcycle (right). All visualizations show predictions with confidence greater than 0.5.
3.3 IMPACT OF DATASET REVIEW
We analyze the impact of our human review, described in Section 2.1 on the classifiers in our test bed. First, we compare the original and perturbed accuracies representative classifier (ResNet- 152 finetuned) with and without review in Table E Our review improves the original accuracy by 3-4% by throwing away mislabeled or blurry anchor frames, and improves perturbed accuracy by 5-6% by discarding pairs of dissimilar frames. Our review reduces the accuracy drop by 1.8%
"This model was originally trained on the 2015 subset of ImageNet-Vir Vid. We evaluated this model on the 2015 validation set because the method requires access to pre-computed bounding box proposals which are available only for the 2015 subset of ImageNet-V Vid.

---

Under review as a conference paper at ICLR 2020
Table 3: Detection and localization mAP for two Faster R-CNN backbones. Both detection and localization suffer from significant drops in mAP due to the perturbations. (*Model trained on ILSVRC Det and VID 2015 datasets, and evaluated on the 2015 subset of ILSVRC- VID 2017.)
mAP Original
mAP Perturbed
mAP
Task
Model
A
FRCNN, ResNet 50
62.8
48.8
14.0
Detection
FRCNN, ResNet 101
63.1
50.6
12.5
R-FCN, ResNet 101
Xiao and Jae Lee (2018)
79.4*
63.7*
15.7*
FRCNN, ResNet 50
76.6
64.2
12.4
Localization
FRCNN, ResNet 101
77.8
66.3
11.5
R-FCN, ResNet 101*
80.9
70,3*
10,6
ImageNet- -iiieet--id-- -iid--Vidudssssse- Robust and 1.1% on TTBB-Robuste but still results in large accuracy drops. These results indicate that the changes in model predictions are indeed due to a lack of robustness, rather than due to significant differences between adjacent frames.
To further analyze the impact of our review on model errors, we plot how frequently each offset distance from the anchor frame results in model error across all model types in Figure For both datasets, larger offsets (indicating pairs of frames further apart in time) lead to more frequent model errors. Our review reduces the fraction of errors across offsets, and especially for large offsets, which are more likely to display large changes from the anchor frame.
Impact of review, Youtube- B8-Robust
Impact review, ImageNNet-iid-Ronne Rid--ssst
Y788 YT38
Imagakee-Vid
Imagehee-Vidnne racis
% AA0 Sareea
Office
Offiet
We plot the fraction of times each offset caused error, across all evaluated models, for frames with and without review. Frames further away more frequently cause classifiers to misfire. Our review process reduces the number of errors, especially for frames further in time, by removing dissimilar frames.
Figure 5:
Table 4: Impact of human review on original and perturbed accuracies for - -Vi-Robust and Youtube-Bii respectively.
Robust and YTBB-Robust, using ResNet-155 fine-tuned on ImageNet-Vid
Accuracy
Reviewed Original
Perturbed
Drop
80.3
64.1
16.2
ImageNet -Vinedet---ii---e -od-Rooust
84.8
70.2
14.4
88.1
78.1
10.0
YTBB-Robust
/
92.9
84.7
8.9
4 RELATED WORK
Adversarial examples. While various forms of adversarial examples have been studied, the major- ity of research focuses L, robustness Goodfellow et al. (2014); Biggio and Roli (2018). However, is unclear whether adversarial examples pose problem for classifter robustness outside of a truly worst context. It is open question whether perfect robustness against a E, adversary will induce robustness to realistic image distortions such as those studied in this paper. Recent work has proposed more realistic image modifications such small rotations & translations Engstrom et al.
7

---

Under review as a conference paper at ICLR 2020
(2017): Azulay and Weiss (2018); Fawzi and Frossard (2015); Kanbak al. (2017), hue and color changes Hosseini and Poovendran (2018), image stylization Geirhos et al. (2018g 8a) and synthetic image corruptions such as Gaussian blur and JPEG compression Hendrycks and Dietterich (2019); Geirhos et al. (201 Even though the above examples are more realistic than the l model, they still syntherically modify the input images to generate perturbed versions. In contrast, work
performs no synthetic modification and instead uses images that naturally occur in videos.
Utilizing videos to study robustness. In work concurrent to ours, Gu et al. (019)) exploit the temporal structure in videos study robustness. However, their experiments suggest substantially smaller drop in classification accuracy. The primary reason for this is less stringent metric used in Gu et al. (2019). By contrast, our "pm-k" metric is inspired by the "worst-of-k k metric used in prior work Engstromm et al.] (2017). highlighting the sensitivity of models to natural perturbations. In Appendix [F we study the differences between the two metrics in more detail. Furthermore, the lack of human review and the high label error-rate we discovered in Youtube- Youtube-BB(Table I presents troubling confounding factor that we resolve in our work.
Distribution shift. Small, benign changes in the test distribution are often referred to as distribution shift. Recht al. (2019) explore this phenomenon by constructing new test sets for CIFAR- 10 and ImageNet and observe performance drops for a large suite of models on the newly constructed test sets. Similar our Figure 3 the relationship between original and approximately linear. However, the images in their test set bear little visual similarity to images in the original test set, while all of our failure cases in ImageNee -Vi d- Robust and YTBB-Robust, on perceptually similar images. In a similar vein of study, Torralba et al. (2011) studies distribution shift across different computer vision data sets such as Caltech-101, PASCAL, and ImageNet.
accuracy is also
Computer vision. A common issue when applying image based models to videos is flickering, where object detectors spuriously produce false-port poisitpossiives or dalse-negatives in isolated frames or groups of frames. Ji et al. (2018) explicitly identify such failures and use a technique reminiscent of adversarially robust training to improve image-based models. A similar line of work focuses on improving object detection videos as objects become occluded or move quickly Kang et al. (2017); Feichtenhofer et al. (2017); Zhu et al. (2017); : Xiao and Jac Lee 2018). The focus in this line of work has generally been on improving object detection when objects transform in a way that makes recognition difficult from single frame, such fast motion occlusion. In this work, document a broader set of failure cases for image- based classifiers and detectors and show that failures occur when the neighboring frames imperceptibly different.
5 CONCLUSION
Our study quantifies the sensitivity of image classifiers to naturally occuring temporal perturbations. We show that these perturbations cause significant drops in accuracy for wide range of models for both classification and detection. Our work on analyzing this failure mode opens multiple avenues for future research:
Building more robust models. Our mmageNet-vid-Robute and YTBB-Robust. datasets provide a standard measure for robustness that can be applied to any classification or detection model. In Table 2] we evaluated several commonly used models and found that all of them suffer from substantial accuracy drops due to natural perturbations. In particular, we found that model improvements with respect artificial perturbations (such as image corruptions or Exo adversaries) induce at best modest improvements in robustness. We hope that our standardized datasets and evaluation metric will enable future work to quantify improvements natural robustness directly.
Further natural perturbations. Videos provide a straightforward method for collecting natural perturbations of images, admitting the study of realistic forms of robustness for machine learning methods. Other methods for generating these natural perturbations are likely to provide additional insights into model robustness. As example, photo sharing websites contain a large number of near-duplicate images: pairs of images of the same scene captured at different times, viewpoints, or from a different camera Recht et al. (2019). More generally, devising similar, domain-specifii strategies to collect, verify, and measure robustness to natural perturbations in domains such as natural language processing or speech recognition is promising direction for future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint arXiv: 12177, 2018.
Luca Bertinetto, Jack Valmadre, Joao Henriques, Andrea Vedaldi, and Philip HS Torr. Fully- convolutional siamese networks for object tracking. In European conference computer vision, pages 850-865. Springer, 2016.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 2018. Batess//arxx2..c////////1....0
Remi Cadene, Pretrained models for
rad-oodels 1.--mmeess.yytortt Accessed: 2019-05-20.
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fen: Object detection via region-based fully convolutional networks. In Advances in neural information processing systems, pages 379-387, 2016.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. rotation and a translation suffice: Fooling cnns with simple transformations. arXiv preprint arXiv:1712.02279, 2017.
Alhussein Fawzi and Pascal Frossard. Manitest: Are classifiers really invariant? In British Machine Vision Conference (BMVC), 2015.
Christoph Feichtenhofer, Axel Pinz. and Andrew Zisserman. Detect track and track to detect. In Proceedings of the IEEE International Conference on Computer Vision, pages 3038-3046, 2017.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. smagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: /8/ 1. 12231, 20.8a.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schüit, Matthias Bethge, and Felix Wichmann. Generalisation in humans and deep neural networks. In Advances in Neural Information Processing Systems, pages 7538-7550, 2018b.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 2014.
Keren Gu, Brandon Yang, Jiquan Ngiam, Quoe Le, and Jonathan Shlens. Using videos evaluate image model robustness. arXiv preprint arXiv: 100,, 2019.
Wei Han, Pooya Khorrami, Tom Le Paine, Prajit Ramachandran, Mohammad Babaeizadeh, Honghui Shi, Jianan Li, Shuicheng Yan, and Thomas S Huang. Seq-nms for video object detection. arXiv preprint
2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint 17Xiv:1903.12266 2019.
Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1614-1619, 2018.
Sou Young Jin, Aruni RoyChowdhury, Huaizu Jiang, Ashish Singh, Aditya Prasad, Deep Chakraborty, and Erik Learned-Miller. Unsupervised hard example mining from videos for improved object detection. In ECCV, 2018
Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Geometric robustness of deep networks: analysis and improvement. arXiv preprint arXiv: /7/ 1177111000 1,09/15, 2017.
Kai Kang, Hongsheng Li. Tong Xiao, Wanli Ouyang, Junjie Yan, Xihui Liu, and Xiaogang Wang. Object detection videos with tubelet proposal networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 727-735, 2017.

---

Under review as a conference paper at ICLR 2020
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. MS COCO detection evaluation. htttp://cocoaaaseet. g/ //Adetection-eval. Accessed: 2019-05-16.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 755-755. Springer, 2014.
Francisco Massa and Ross Girshick. maskrenn maskrem--benchmate benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch. |ht /ttp:://githuii
om/ htt//aacbookkesss.ccch/////aaeee betc////mccccoobbbmaaaarrrrr 2018. Accessed: 2019-05-20.
benchmark
George A Miller. Wordnet: a lexical database for english. Communications of the 38(1)):
39 39,41, 1995.
Harold Pashler. Familiarity and visual change detection. Perception & psychophysics, 44(4):369-378, 1988.
Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtube- boundinghoxes: A large high-precision human-annotated data set for object detection video. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5296-5305, 2017.
Benjamin Recht, Rehecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint (rXiv:/902. /0811, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211-22,, 2015. doi: 10.1007/s1 126-0015011--11-
Antonio Torralba, Alexei A Efros, al. Unbiased look dataset bias. In CVPR, volume page Citeseer, 2011.
Fanyi Xiao and Yong Jae Lee. Video object detection with an aligned spatial-temporal memory. In Proceedings of the European Conference on Computer Vision (ECCV), pages 485-501, 2018.
Cihang Xic, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. arXiv preprint arXiv: 18/2.03411 2018.
Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016. doi: 10. 20.11099.4p:20116.888, URL http://dxx org/10 1109/ cwpp//vpr... 2016.44 485)
Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 408 2017. 2017.
10

---

Under review as a conference paper at ICLR 2020
A SOURCE DATASET OVERVIEW
A.1 IMAGENET-VID
The 2015 ImageNet- Vid dataset is widely used for training video object detectors Han et al. (2016) as well as trackers [Bertinttto et al. (2016). We chose work with the 2017 ImageNet-Vin dataset because is superset the 2015 dataset. 1,181,113 training frames from 4,000 videos and 512,360 validation frames from 1,314 videos. The videos have frame rates ranging from 9 to 59 frames per second (fps), with a median fps of 29. The videos range from 0.44 to 96 seconds in duration with median duration of 12 seconds. Each frame annotated with labels indicating the presence or absence of 30 object classes and corresponding bounding boxes for any label present in the frame. The 30 classes ancestors of 293 of the 1 1,000 ILSVRC-2012 classes.
total, the 2017 ImageNer- Vid dataset consists of
A.2 YOUTUBE-BB
The 2017 Youtube-BB is a large scale dataset with 8,146, 143 annotated training frames 253,569 unique videos and with 1,013,246 validation frames from 31,829 videos. The video segments approximately 19 seconds long on average. Each frame is annotated with exactly one label indicating the presence of 22 object classes, all of which are ancestors of 229 out of the I1SSVRC-2012 classes.
B FULL ORIGINAL VS PERTURBED ACCURACIES
B. IMAGENET-VEL ROBUST
Accuracy Original
Accuracy Perturbed
Model
A
resnet152 finetuned
84.8 [82.5, 86.8]
70.2 [67.4, 72.8]
14.6
Essne500fineturee
80.8 [78.3, 83.1]
65.7 [62.9, 68.5]
15.1
vggl6bn_finetuned finetuned
78.0 [75.4, 80.4]
61.0 [58.1, 63.9]
17.0
nasnetalarge.
nasnetalargejimagnee___o
77.6 [75.1, 80.1]
62.1 [59.2, 65.0]
15.5
resnet50_detection
77.6 [75.1, 80.1]
65.0 [62.1, 67.8)
12.6
inceptionresnetv2 imagenet
umageeeetppetraineed pretrained
75.7 [73.1, 78.2]
58.7 [55.7, 61.6]
17.0
dpn107 10000l0imaaeee___m imwgemettppetraineee pretrained
75.6 [72.9, 78.1]
59.1 [56.1, 62.0]
16.5
inceptionv4_imagnnee__ pretrained
p
75.3 [72.6, 77.8]
59.0 [56.0, 61.9]
16.3
74.4 [71.7, 76.91
56.8 [53.8, 59.7]
17.6
dpm131_imagenne_pretailnn
74.0 [71.3, 76.6]
59.9 [56.9, 62.8]
14.1
htt_jjiaane___eeee pretreattdeerreeedeeeeee
73.7 [71.0, 76.2]
54.0 [51.0, 57.0]
19.7
resnext101_3240 gamagenet_pretraanng
73.3 [70.6, 75.9]
57.2 [54.2, 60.1]
16.1
resnext101 64x4d
pretrained
72.9 [70.1, 75.5]
56.6 [53.7, 59.6]
16.3
resnet152 www.immetppretraaeeeeeeee
72.8 [70.0, 75.4]
57.0 [54.0, 59.9]
15.8
resnet101, imaggnntt_pretrained
71.5 [68.7, 74.1]
53.7 [50.8, 56.7]
17.8
fbresnet 52_ [2__imagenet_prrrrinneee
71.5 [68.7, 74.11
54.5 [51.5, 57.4]
17.0
densenet d ___maagenet__retrrinneedd
71.4 [68.7, 74.1]
55.1 (52.1,
16.3
densenet 169 Lamsenec_l__immaanee__eprtet
70.2 [67.5,
53.1 [50.1, 56.1]
17.1
(ensener20l_imagnnet_ppo _tr_j_ggnee_prrrrinnee
70.2 [67.5, 72.9]
53,4 [50.4, 56.4]
16.8
dpn68
1ace0___maeee______ Lengenettppretriieed
69.4 [66.6, 72.1]
53.3 [50.3, 56.3]
16.1
bninception iaroooonnmaaeeeem_m jia_ggnnnt_prerrriieeedd
69.0 [66.2, 71.7]
49.0 [46.0, 51.9]
20.0
densenet121 Jinbenttt2L_imggeeeepprtttttnn
69.0 [66.2, 71.7]
50.9 [47.9, 53.8)
18.1
nasnetamobile. http:/oobieeeiimeeeeeeerrrrrrnne
68.8 [66.0, 71.5]
48.4 [45.4, 51.4]
20.4
1aar550__aummet___ina jpeg_coopeession
68.8 [66.0, 71.5]
53.2 [50.2, 56.2]
15.6
resnet34 i04444etaaeetttc imagenet_pretrained
68.0 [65.2, 70.7]
48.0 [45.0, 51.0]
20.0
resnet50 augment impulse noise
67.7 [64.9, 70.5]
50.2 [47.2,
17.5
resnet50_and 200or500_augmnntt__m gaussian_blin
67.7 [64.9, 70.5]
52.5 [49.5, 55.5]
15.2
disnet50_imagenee sttrggnnet_pettaineed
67.5 [64.7, 70.3]
52.5
(49,5,55.5)
15.0
resnet50 200550t_umnnntt_.m gaussian__ooe noise
67,4 [64.5, 70.1]
50.6 (47.6, 53.6]
16.8
resnet50 2a1550__aumen_____ shot_nois _oise
66.5 [63.6, 69.2]
51.1
[48.1, 54.11
15.4
perugenet_peetraineed resnet50 augment defocus blur
vgg16 bn imagenet pretrained
66.4 [63.5, 69.1]
47.4 [44.5, 50.4]
19.0
66.3 [63.4, 69.1]
47.6 [44.6, 50.6]
18.7
vgg19 bn imagenet pretrained
65.6 [62.7, 68.4]
46.6 [43.6, 49.6]
19.0

---

Under review as a conference paper at ICLR 2020
vgg19 tongenee__prerainne
63.2 [60.3, 66.1]
45.4
(42.4,48.11
17.8
resnet18 imagenet_pretrained
61.9 (59.0, 64.8]
41.5
[38.6, 44.4]
20.4
vgg13 bn imagenet_pretrreeeeeeeee
pretrained
61.9 [59.0, 64.8]
43.3
(40.3,46.3)
18.6
61.4 [58.5, 64.3]
43.1
[40.2, 46.1]
18.3
Bagll_bn_imageet__a
60.9 [57.9, 63.8]
43.2
[40.3, 46.2]
17.7
59.6 [56.6, 62.5]
41.1
[38.2, 44.1]
18.5
vggll_imagneee_peerriinee
57.3 [54.4, 60.3]
41.3 [38.4, 44.3]
16.0
57.3 [54.3, 60.2]
43.6 [40.7, 46.6]
13.7
ResNeXtDeenosseAll- robust pgd
54.3 [51.3, 57.2]
40.8 [37.8, 43.7]
13.5
squeezenerl_ _ta___mmaenee__prrteeee
49.8 [46.8, 52.8]
31.7 [28.9, 34.5]
18.1
ttagl_mmagennt__eeeee Resnet50_augmentt ltt____ooottttarrneeee
49.4 [46.4, 52.41
32.0 [29.3, 34.8]
17.4
38.3 [35.5, 41.3]
23.3 [20.8, 25.9]
15.0
Table 5:
Classification model perturbed and original accuracies for all models in our test bed evaluated on the ImageNet-Vin Vide--iiduu-ss dataset.
B.2 VTBB-ROBUSE
Accuracy Original
Accuracy Perturbed
Model
A
resnet152 resnet152_finttnned finetuned
92.9 [91.2, 94.3]
84.7 [82.4, 86.8)
8.2
resnet50 finetuned
91.4 [89.6, 93.0]
82.0 [79.6, 84.2]
9.4
nueptionresnetv22fnneeee finetuned
91.3 [89.5, 92.9]
79.0 [76.4. 81.3]
12.3
vgg19 finetuned
90.5 [88.6, 92.2]
79.1 [76.5, 81.4]
11.4
vggl6 ffi_ffffuneeddd
89.1 [87.1, 90.8]
78.0 [75.4, 80.4]
11.1
inceptionv4finettume finetuned
88.5 [86.5, 90.3]
76.3 [73.6, 78.7]
12.2
resnet18 finctuned
88.0 [85.9, 89.8]
76.2 [73.6. 78.7]
11.8
fecxnet_finetunn
80.6 [78.2, 82.9]
64.4 [61.5, 67.3]
16.2
pnasnet5large
paaggene__peetrineed
65.2 [62.3, 68.0]
51.0 [48.0, 54.0]
14.2
nasnetalarge aasnetalarge_imagenee__e immgenettppeerrinee
64.9 [62.0, 67.7]
51.4 [48.4, 54.4]
13.5
64.5 [61.6, 67.4]
50.4 [47.5, 53.4]
14.1
mom9__imagenet_prtrraae
64.1 [61.2, 66.91
49.0 [46.0, 52.0)
15.1
dpn dpmll0__imagenee_perraainee
64.1 [61.2, 66.91
50.1 [47.2, 53.11
14.0
1onl3l_imagenete ppretraine_eeedd
64.0 [61.1, 66.8]
49.9 [46.9. 52.9]
14.1
inceptionv4 prangenet_pretrainedd
0ingr/net_meeet.._ pretrained
63.6 [60.7, 66.4]
48.8 [45.8. 51.8]
14.8
Creption_imagene htt___iaeaee_eeerrrineeeee
63.2 [60.2, 66.0]
47.6 [44.6, 50.6]
15.6
62.3 [59.3, 65.1]
47.7 [44.8, 50.7]
14.6
Sanee550auggment http__ppeeeessssseeeee_
62.3 [59.4, 65.2]
45.7 [42.8, 48.7]
16.6
jolynee_inagenee_peerzzii
61.4 [58.4, 64.3)
47.3 [44.4, 50.3]
14.1
nasnetamobile _icllleetmeeeeet_e imagenet_pretrained
61.4 [58.4, 64.3]
43.0 [40.1, 46.0]
18.4
resnet50 augment shot noise
61.3 [58.3, 64.2]
46.4 [43.4. 49.3]
14.9
dpn68
61.2 [58.3, 64.1]
44.2 [41.2.
17.0
Leassnet_S__imaageee__eeeett
61.1 [58.1, 64.0)
45.9
[42.9, 48.8]
15.2
60.8 [57.8, 63.7]
46.5 [43.5, 49.5]
14.3
60.8 [57.8, 63.71
45.2 [42.2, 48.2]
15.6
senet154_imagenet
60.7 (57.7, 63.6]
47.2 [44.3, 50.2]
13.5
1ainet50aaugnaee
Bestnnent__mmuseee_e____ _ noise
60.6 [57.7, 63.5]
45.5 [42.6, 48.5]
15.1
se ee_resnet101 -magenett_peetriined
60.5 [57.6, 63.4]
45.6 [42.6. 48.6]
14.9
pniineptioo_immageeet_peeee
60.4 (57.4, 63.3
41.8 [38.9, 44.71
18.6
Lomnenntt6___mmageee__ereeeee
60.2 [57.3, 63.11
46.4 [43.4, 49.4]
13.8
Srret50_aaugeet.. aaw.aaaaennnaaausiiaaaaaaaa gwwssian_nnnnnn _oiiee
60.2 [57.3, 63.1
45.7 [42.8, 48.7]
14.5
se reseenct500 32x4d imagenet
imagenettpretraiied pretrained
59.9 [56.9, 62.8]
45.7 (42.7, 48.6]
14.2
dpn68b baabbbimmgeeetttte cecragenet_prttaineed pretrained
59.7 [56.7, 62.6)
45.9 [42.9, 48.8)
13.8
inceptionv3_ Eicettions3immageettt betrsseettppetraineddd
59.6 [56.6, 62.5]
43.8
[40.8, 46.8]
15.8
densenet121 imagenet_preerreneee
59.5 [56.5, 62.4]
43.1
[40.1, 46.0]
16.4
se 1e_rsnext101_32x444 Le____xx4A__mmaeeee_errrrraaaaae
59.2 [56.3, 62.1]
45.2
[42.3, 48.2)
14.0
densenet201 Laapreeeeo__mmmaeeeeeeerrrrraaee
59.2 [56.2, 62.11
44.8
[41.8, 47.81
14.4
densenet 169 (ee_iiaaeene__perrriinneee
59.2 [56.2, 62.11
44.6
[41.7,
14.6
12

---

Under review as a conference paper at ICLR 2020
resnet50 augment brightness change
58.9 [56.0, 61.8]
42.6
[39.6. 45.5]
16.3
se resenet5 seangenet_preraiiie
58.8 [55.9, 61.7]
44.1
47.11
14.7
resnet152 imaagenet_peeraaaeee
pretrained
58.8 [55.9, 61.7]
44.8
[41...7.8
14.0
Laarreseeetiiiinnnneeeeeeeeee LamnettO_augment____
58.2 [55.2, 61.1]
44.3
[41.3, 47.31
13.9
58.0 (5..1,61.00
42.9
[39.9, 45.8]
15.1
57.9 [55.0, 60.9]
42.8 [39.8, 45.7]
15.1
57.5 [54.6, 60.5]
40.1 (37.2, 43.1]
17.4
#ssnt55_augmmens gaussian blur
57.5 [54.5, 60.4]
41.8 [38.9, 44.7]
15.7
vgg bn aon__mmageet__perraiiii
57.2 [54.2, 60.1]
39.6 [36.7. 42.6]
17.6
resnet5_jimagene__peeee
57.0 [54.1,
43.8 [40.9, 46.8]
13.2
vgg
56.8 [53.9, 59.81
40.6 [37.7, 43.5
16.2
vggl6_ing Lggl6_imagenee_preeraee
55.4 [52.4, 58.4]
40.1 [37.2, 43.1]
15.3
vgg bn imagenet parptraiee_trreeeedeeeee
54.8 [51.8, 57.7]
38.6 [35.7. 41.6]
16.2
54.8 [51.8, 57.7]
38.8 [35.9, 41.8]
16.0
aggll_imagenet sarageene_pretrainedd
54.7 (51.7,57.66
38.4 [35.5, 41.3]
16.3
resnet18 htr___aannee__eerrrriiee
54.4 (51.4, 57.4]
38.1 [35.2, 41.0]
16.3
vgg 13 Lagglliiagnn___prrrraieee
54.2 [51.3, 57.21
37.7 [34.9, 40.7]
16.5
ResNextDeenissell--0011. 1ttll-0001rros___PPe___
53.6 150.7, 56.61
43.2
[40.2, 46.11
10.4
squeezenetl 0 imagenet pretrained
51.1 (4..1,
33.1
[30.3. 36.0)
18.0
squeezenetl 1 imagenet pprtttiieeeteeeee
48.6 [45.6, 51.6]
31.3
[28.6. 34.2]
17.3
resnet50 augment defocus blur
48.4 [45.4, 51.4]
29.1
[26.4. 3188
19.3
alexnet_ taaitet_iinanne_eeetrriinn
45.3 [42.4, 48.3]
30.5 [27.8, 33.3]
14.8
Table Classification model perturbed and original accuracies for all models our test bed
evaluated the YTBB-robust dataset..
C MODEL INDEPENDENT DISTRIBUTION SHIFT
Though the distribution shift we induced in our study were model dependent because we found the worst neighbor frame for each model, could study the same problem but impose static set perturbed frames across all models. In Figure 66 we study this static set of perturbations across all models and substantial (but smaller) drop in accuracy for both models. The static set of perturbations were chosen by choosing the neighbor frame that the largest number of models classified incorrectly.
ImageNet-Vid-Robust Accuracy
YTBB-Robust Accuracy
Perturbed Accuracy
70
Accuracy
80
70
60
40
Perturbed
50
30
30
40
50
60
80
50
60
70
90
Original Accuracy
Original Accuracy
No Accuracy Drop
ILSVRC noise augmentation
ILSVRC finetuned on ILSVRC-VID-DET finetuned
Linear
adversarial training
ILSVAC
ILSVRC finetuned on ILSVRC-ID
Per model perturbations
Figure 6: Model accuracy original perturbed images for static set of perturbed frames across all models. The grey points and grey linear fit correspond to the perturbed accuracies of models evaluated on per model perturbations studied in Figure
D PER CLASS ANALYSIS
We study the effect of our perturbations on the 30 classes in ImageNet-Vid-Ree and YTBB VTBB-Robust. determine whether the performance drop concentrated in few "hard" classes.
13

---

Under review as a conference paper at ICLR 2020
ImageNet-VidRRouutt Per-Class
1.0
Accuracy
0.6
0,0
momee the
yous
Basubbe-B-Robbuus Per-Class
1.0
0.6
0.4
0.2
0.0
pland
dlg
ell
Car
Figure 7: Per-class accuracy statistics for our best performing classification model (fine-tuned ResNet152) on -iiieNeetvi--obbutt and YTBB-Robust For Youtube-BB. note that 'zebra' is the least common label, present in only 24 anchor frames sampled by Gu et al. (011),, of which included in dataset.
by/Gu
Figure 7 shows the original and perturbed accuracies across classes for our best performing model (a fine-tuned ResNet- 152).. Although there are a few particularly difficult classes for perturbed accuracy (c.g., lion or monkey on most classes. On -maaeeeet-iid-Robbae and perturbed images and a median drop of 1 14.0% in per-class accuracy. On YTBB Robuss, the total drop was 8.9% and the median drop was 6.7%.
the accuracy drop is spread across this model saw total drop of 14.4% between original
PER- FRAMF CONDITIONAL ROBUSTNESS METRIC INTRODUCED IN GU E
ET AL. (0119)
ShageNet-Vid-RRbaee (Gu, et al. 2019) Metric
-TBB-Robust (Gu, et al 2019) Metric
1.00
1.00
0.98
0.98
Rabustness
Rabustness
0.96
0.96
0.94
0.92
0.92
2
4
8
10
2
8
10
Frame Perturbation Distance (k)
Frame Perturbation Distance (k)
(LVVIC Prerained
Riieuaed
1.99RC Fretsined
Rheeaned
Nesp cugmenittee
12 Aeversatss Kanuot (155
Detectiee
L2 Advorienaal Prssutt tes/txt-tt0
Figure 8: Conditional robustness metric from Gu et al. (019) on perturbed frames as function of perturbation distance ImageNet -iid --iidtRobust and YTBB Robust. Model accuracies from five different model types and the best performing model are shown. The model architecture is ResNet-50 unless otherwise mentioned.
14

---

Under review as a conference paper at ICLR 2020
In concurrent work, the authors of Gu et al. 2019)) considered different metric of robustness. In this section, we compute this metric on all models in our test bed to compare our findings to Gu et al.
(2019). There are two main differences between PM-k and the robustness metric in Gu et al. (2019).
1. For visually similar "neighbor" frames Io and I with true label Y and classifier f. Gu
et al. (2019) studies the conditional probability P(f(I,) = vlf = y)
2. While PM-k looks for errors in all neighbor frames in a neighborhood of l: frames away from the anchor frame (so this would include frames 1. .... k frames away). Gu al.
(2019) only considers enrors from exactly k frames away.
In Fig. we illustrate simple example where two videos can have the same behavior for the metric introduced by Gu et al. (2019) but drastically different behavior for the PM-kmetric.
Dataset
Dataset Ancho Yoom
Fais praccionn
Wileo mormples
Timz stop
Time
113-
Timestepp
Accuracy (.55
PH accuracy
Figure 9: For the two example videos above the from Gu et al. 2019) metric (Accuracy @ K) identical, but the PM-k metric behaves substantially differently when the errors are spread across many independent videos, shown in the right example
F foc DISTANCE VS PM-K ACCURACY
to adversarial examples are well studied in the robustness community, yet the connection between tx and other forms of more "natural" robustness is unclear. Here, we plot the cumulative distribution of the t.x distance between pairs of nearby frames in our datasets. In Figure T0 we show the CDF of l. distance for all pairs, all reviewed pairs, and mistakes made by 3 indicative models. Note the fbrobust model trained specifically to robust to loo adversaries.
All pairs without reviews
All pairs with review
resnet152 mistakes
0.8
fbrobust mistakes
resnet152-fineturee mistakes
Frequency
0.6
0.4
0.2
0
50
100
150
200
250
L. norm
Figure 10: CDF showing the f.r distance between pairs of frames from different distributions.
15

---

Under review as a conference paper at ICLR 2020
Table Analyzing results based on frame-type in video compression. See Appendix H.
for details.
Original Acc.
Perturbed Acc.
A
anchor frames
All frames
84.8
70.2
14.6
1109
w/o ii-frames'
84.7
70.3
14.4
1104
w/o "p-frames'
83.9
73.7
10.2
415
w/o 'b-frames'
85.4
73.2
12.2
699
G PM-K ACCURACY WITH VARYING K
G.1 IMAGENET-VID-RORII
Perturbed Test Accuracy
80
trained ILSVRC
70
ILSVRC noise augmentation
60
ILSVRC + 12 adversarial training (ResNext-107)
ILSVRC fine-tune on ILSVRC-VID
50
ILSVRC fine-tune on ILSVRC-VID (ResNet-152)
ILSVRC fine-tune ILSVRC-VID-DET
40
2
10
Frame Perturbation Distance (k)
Figure 11: Model classification accuracy perturbed frames as function of perturbation distance (shown with 95% Clopper-Pearson confidence intervals). Model accuracies from five different model types and the best performing model are shown. The model architecture is ResNet-50 unless otherwise mentioned.
In Figure we plot the relationship between acCpmk and perturbation distance (i.e., the k in the pm-k metric). The entire x-axis in Figure 11 corresponds temporal distance of at most 0.3 seconds between the original and perturbed frames.
H I-FRAMES AND P-FRAMES
H.1 INAGENET-VID-RSSE
One possible concern with analyzing performance on video frames is the impact of video compression model robustness. In partieular, the videos in -iigeeeet-ii--oooe frame types: 'i-frames', 'p-frames', and 'b-frames'. "p-frames' are compressed by referencing pixel content from previous frames, while "b-frames' compressed via references previous and future frames. 'i-frames' are stored without references to other frames.
contain 3 different
We compute the original and perturbed accuracies, and the drop in accuracy for subset of the dataset without 'i-frames' subset without 'p-frames', and subset without 'b-frames' in Table While there are modest differences in accuracy due to compression, this analysis suggests that the sensitivity of models is not significantly due the differences in quality of frames due to video compression.
I FPS ANALYSIS
I.1 INAGENET-VIO-ROGEE
To analyze the impact of frame-rate accuracy, show results subsets of videos with fixed fps (25, 29, and 30, which cover 89% of the dataset) using a fine-tuned ResNet-152 model in Table g The accuracy drop is similar across the subsets, and similar the drop for the whole dataset.
16

---

Under review as a conference paper at ICLR 2020
FPS
Acc. Orig.
Acc. Perturbed
Drop
Videos
25
87.3 [83.0, 90.9]
73.3 [67.8, 78.3]
14.0
292
29
87.7 [84.0, 90.8]
74.9 [70.3, 79.2]
12.8
383
30
78.3 [73.3, 82.7]
61.7 [56.0, 67.1]
16.6
313
Table 8: Results on subsets of ImageNet-Vidd Vid-Robust with fixed FPS.
J ILSVRC TRAINING WITH MAAGENET-VID-RON
CLASSES
We trained ResNet-50 from scratch on ILSVRC using the 30 ImageNet- Vid classes. We also fine- tuned the model on ImageNet- Vid. In Table we show the accuracy drops are consistent with models in our submission. We hypothesize that the lower accuracy is due coarser supervision ILSVRC.
Model
Acc. Orig.
Acc. Perturbed Drop
ILSVRC-30
61.0
44.9
15.1
ILSVRC-30 FT
77.8
59.9
17.9
Table 9: Results of training ResNet-50 on ILSVRC with classes from ImageNet-Vid-RRohe
K EXPERIMENTAL DETAILS & HYPERPARAMETERS
All classification experiments were carried out using PyTorch version 1.0.1 AWS p3.2xlarge with the NVIDIA V100 GPU. All pretrained models were downloaded from Cadenee at commit hash 021d97899 7 6ec75 9de ff 3d34 c4 fd4 5d 7ba. Evaluations Table ?? all use the default settings for evaluation. The hyperparameters for the fine-tuned models are presented in Table 10 We searched for learning rates between 10-3 and 10-5 for all models.
We additionally detail hyperparameters for detection models in Table []] Detection experiments were conducted with Py PyTorrc version 1.0.1 on a machine with 4 Titan X GPUs, using the Mask R- CNN benchmark repositoryMassa and Girshick (2018). We used the default learning rate provided
in Massa and Girshick (2018). For R R-FCN, we used the model trained by Xiao and Jae Lee (2018)
Table 10: Hyperparameters for models finetuned on ImageNet-Vin
Model
Base Learning Rate
Learning Rate Schedule
Batch Size
Epochs
resnet152
10
Reduce LR Plateau
32
10
resnet50
10
Reduce LR On Plateau
32
10
alexnet
10
Reduce LR On Plateau
32
vgg16
10 10-5
Reduce
Plateau
32
10
Table 11: Hyperparameters for detection models.
Model
Base Learning Rate
Learning Rate Schedule Step 20k, 30k
Batch Size
Iterations
F-RCNN ResNet-50
10-2
40k
F-RCNN ResNet-101
10-2
Step 20k, 30k
40k
L DETECTION PM-K
We briefly introduce the mAP metric for detection here and refer the reader to Lin et al. for further details. The standard detection metric proceeds by first determining whether each predicted bounding box in an image true false positive, based the intersection over union (IoU) of the predicted and ground truth bounding boxes. The metric then computes the per-category average precision (AP, averaged over recall thresholds) of the predictions across all images. The final metric is reported as the mean of these per-category APs (mAP).
We define the pm-k analog of mAP by replacing each anchor frame in the dataset with nearby frame that minimizes the per-image average precision. Since the category-specinf average precision is undefined for categories not present in an image, we minimize the average precision across categories present each frame rather than mAP.
17


Reviews:
[{"review_id": "rkxcWM73FH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents new datasets based on ImageNet and Youtube-BB to assert networks performance consistency across time. Compared to previous work, it uses human labeler to further validate the dataset and discard frames that are deemed too different from the reference one. It provides results on image classification and detection using popular network architectures. Based on these results, the paper claims an accuracy drop of 10 to 16%.\n\nThe main contribution of this paper is to introduce a new, human annotated dataset for robustness assessment of image classifiers. In itself, it is valuable work, but it is not clear if the contribution is important enough for ICLR. However, I would still be ok with accepting the paper (better datasets are always useful) if it was not for the way the results are reported. I do not specifically have issues with \"more stringent robustness metric\" but it should not be used to claim incredible results (like an accuracy drop of 10 to 16% instead of 3% for previous work (Real et al. 2017)).\n\nThere is one thing for sure: using \"accuracy drop\" in this context is just misleading. The underlying concept to which \"accuracy\" refers is _not_ the \"maximum error made by the network over the whole set of images\". By this definition of accuracy, if the number of images around the reference frame were 100, missing a _single one_ each time (that is 99% of actual accuracy) would result, according to this peculiar redefinition, to a _0%_ accuracy. This is actually highlighted in Appendix G.1: the \"accuracy\" trend can only go down, since every supplemental frame brings one more chance to fail and obtain a 0% accuracy for this set of perturbed images.\nSame thing goes for the detection, where the only frame that matters among all is the one _minimizing_ the AP. Same thing in Table 4, where the \"accuracy\" of the Original column means one thing (the amount of correctly identified images over the total number of images) while the \"accuracy\" of the Perturbed column right next to it means something completely different. Same thing in Table 2, which even provides a \"delta\" between two unrelated metrics.\nI cannot see how this can be justified. Sure, there could be some usage for such strict metric, but again, this is _not_ accuracy and cannot be compared to any previous results. Having a more stringent metric is one thing, but in this case it just seems like a justification to get high drop numbers.\n\nKeeping that in mind, these are the actual conclusions we can make from the paper:\n1) Human reviewers removed or changed about 20% of the frames\n2) This resulted in a relative accuracy improvement of about 4% for the reference frame (Table 4, column Original). The improvement for the perturbed frames are not actually provided.\n3) The comparison (and improvements) to previous work due to the dataset cleaning remains unclear.\n4) Comparison between different networks and training procedures\n\nOverall, the paper presents impressive numbers but does not actually back them up. I am open to eventually consider acceptance given the value of the datasets, but the paper would then require a significant overhaul to remove all confusing aspects.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Skx8zfYntr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper targets on the evaluation of model robustness on similar video frames. The authors build two carefully labeled video datasets, and extensive experiments are conducted to show that the state-of-the-art classification and detection models are not robust enough when dealing with very similar video frames.  The results are similar to my intuitive feelings.\n\nThe authors propose acc_orig (the average acc) and acc_pmk (which chooses the worse one in nearest 2k frames) to amplify the gap. I personally think acc_pmk is too stringent. I wonder if there is still large gaps if we choose a random frame in the nearest 2k frames.\n\nThe authors have tried fine-tuning and data augmentation techniques to improve the robustness, although the performance is improved, the gap between acc_orig and acc_pmk does not change much.\n\nThe paper has done many work to analyze the robustness of image classification and detection models, however, the results are expected and no effective methods are proposed to improve the results. Overall, the contribution is limited. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1eNC80O5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nIn this paper, the authors curated two datasets: ImageNet-Vid and Youtube-BB in order to create human-reviewed perceptibly similar sets (Imagenet-Vid-Robust and YTBB-Robust). The obtained datasets are evaluated over 45 different models pre-trained on ImageNet in order to see their drop in accuracy on natural perturbations. Three detection models are also evaluated and show that not only classification models are sensitive to these perturbations, but that it also yields to localization errors.\n\nComments\nThe paper is clear, well organized, well written and easy to follow.\nThe authors present two novel datasets grouped in sets of perceptibly similar images and answer to the following hypothesis: Can the perturbations occurring naturally in videos be a realistic robustness challenge?\nThe thorough evaluation over the curated datasets shows pretty well that the changes in the model prediction are indeed due to a lack of robustness of the models themselves rather than the difference occurring from one frame to the other (occlusion etc).\nThe authors mention the curation was done with the help of expert human annotators. Details could be added as to how the annotators are considered experts and what process they went through (mturk? Handmade application to select the frames?).\nOverall I think the paper adds an interesting contribution, with the datasets themselves which can be used for image similarity tasks for example\nAlthough the contribution of the paper is important, it seems limited for the conference with no novel method proposed. \n\nTypo\nSection 3, l 4: using use -> using\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed to evaluate the robustness of CNN models on similar video frames. The authors construct two carefully labeled video databases. Based on extensive experiments, they conclude that the state of the art classification and detection models are not robust when testing on very similar video frames. While Reviewer #1 is overall positive about this work, Reviewer #2 and #3 rated weak reject with various concerns. Reviewer #2 concerns limited contribution since the results are similar to our intuition. Reviewer #3 appreciates the value of the databases, but concerns that the defined metrics make the contribution look huge. The authors and Reviewer #3 have in-depth discussion on the metric, and Reviewer #3 is not convinced. Given the concerns raised by the reviewers, the ACs agree that this paper can not be accepted at its current state.
Decision: Reject