Beyond the Pixels: Exploring the Effects of Bit-Level Network and File Corruptions on Video Model Robustness
Authors: Trenton Chang
Daniel Yang Fu
Yixuan Li
Venue:  (2021)
Keywords: robustness
machine learning
file corruption
network corruption
video

Abstract: We investigate the robustness of video machine learning models to bit-level network and file corruptions, which can arise from network transmission failures or hardware errors, and explore defenses against such corruptions. We simulate network and file corruptions at multiple corruption levels, and find that bit-level corruptions can cause substantial performance drops on common action recognition and multi-object tracking tasks. We explore two types of defenses against bit-level corruptions: corruption-agnostic and corruption-aware defenses. We find that corruption-agnostic defenses such as adversarial training have limited effectiveness, performing up to 11.3 accuracy points worse than a no-defense baseline. In response, we propose Bit-corruption Augmented Training (BAT), a corruption-aware baseline that exploits knowledge of bit-level corruptions to enforce model invariance to such corruptions. BAT outperforms corruption-agnostic defenses, recovering up to 7.1 accuracy points over a no-defense baseline on highly-corrupted videos while maintaining competitive performance on clean/near-clean data.

Under review conference paper ICLR 2021
BEYOND THE PIXELS: EXPLORING THE EFFECTS OF BIT-LEVEL NETWORK AND FILE CORRUPTIONS ON VIDEO MODEL ROBUSTNESS
Anonymous authors Paper under double-blind review
ABSTRACT
We investigate the robustness of video machine learning models to bit-level net- work and file comuptions, which can arise from network transmission failures or hardware errors, and explore defenses against such corruptions. We simulate network and file corruptions at multiple corruption levels, and find that bit-level corruptions can cause substantial performance drops common action recogni- tion and multi-object tracking tasks. We explore two types of defenses against bit-level corruptions: corruption-anoosiii and corruption -wware aware defenses. We find that corruttio--annosiic defenses such adversarial training have limited effec- tiveness, performing up to 11. accuracy points worse than a no-defense baseline. In response, we propose Bit-corruption Augmented Training (BAT), a corruption- aware bascline that exploits knowledge of bit-level corruptions to enforce model invariance such corruptions. BAT outperforms covering up to 7.1 accuracy points over a no-defense baseline on highly-corrupte videos while maintaining competitive performance clean/near-clens data.
defenses, re-
INTRODUCTION
Video becoming an increasingly common data modality, with applications in online conferenc- ing (Jansen et al. 2018)). autonomous vehicles (Chen et al., 2018; Bojarski et al. (016)) action recognition (Kuehne et al. 2011 Soomro et al.] 2012 Carreira & Zisserman, 2017). and event detection (Fu et al., 2019; Gaidon et al.] 2013). Recent work in computer vision has studied the robustness of maching hine learning (ML) models to pixel-space corruptions such as adversarial exam-
ples (Madry et al.) 2017 Carlini & Wagner 2016 Goodfellow al. 2014 Moosavi-Dezzoor et al.
2016 (Szeedyy et al.] 2003)). However, in the real world, vidcos are susceptible to corruptions beyond the pixels, such as bit-level network and file corruptions (Fig. [
Bit-level corruptions are generally non-adversarial and arise in the real world from network conges- tion (Mushtaq & Mellouk] (2017). i.e., transmission problems in video confereneing, hardware errors during storage (Sivathanu et al., 2005; Zhang et al. 2010). These corruptions spuriously modify bits in video file, leading to data loss or visual distortion (e.g. object duplication, noisy patches. freeze-frane frames). Although some previous work in computer vision robustness has been compresssion-an aware, e.g. studying the effects of JPEG compression (Aydemir et all. 2018 Dziu- gaite et al., 2016; Das et al.] 2018)) or changing bitrates (Guo et al. 2017b), direct corruptions to the bit representations of videos remain underexplored.
In this work, we take a first step in understanding video model robustness to naturally-occurring levels of network and file corruptions, and explore baseline defenses to these corruptions. To this end, we simulate wide range of corruption severity levels consistent with empirical studies of real-world corruptions (Schroeder al.) 2016 Hu & Zhang, 2018), and explore their effects on video artifacts and model performance. We evaluate model performance on two action recognition datasets (Kuehne et al., 2011] Soommo et al. 2012) and one multi-object tracking dataset (Leal- Taixé et al., 2015) under simulated network packet loss and random bit-flip file corruptions. Our experiments demonstrate that model performance begins to degrade when packet loss rates exceed 1 16, or when the proportion of randomly- flipped bits in a file exceeds 10-5 and ultimately drops by up 77. % under the most severe corruption levels.

---

Under review conference paper ICLR 2021
Aoruuption-ggnostic Defenses
Bit-Level Corruptions
(to))
A
Out-of distribution detection
Network corruptions
Corrupted Videos
corruptions
Adversarial Training
Corruption-Aware Defenses
Video
File Storage
Network Transmission
Video
Recording
Frames
Bit Sit-CCoruption Augmented Training
Figure 1: An overview of the video machine learning pipeline under level corruptions, which can arise due to file storage and network transmission. We explore various defenses against network and file comuptions.
We then focus on exploring two categories of defenses against network and file corruptions: corruption-agnoxtic methods without knowledge of bit-level corruptions, and corruption-ani methods that exploit knowledge bit-level corruptions enforce model invariance network and file corruptions. For corruption-agnosti defenses, we evaluate out-of-distribution (OOD) de- tection (Hendrycks & Gimpel| 2016 [Liang al.] (0188) and adversarial training (Goodfellow al.) 2014) as baselines. Our findings suggest that corruption-agnotti defenses have limited effective- ness, especially at low corruption levels; for example, under adversarial training, model perfor- mance drops by up 11.3 points corrupted data, and 8.6 points clean data, compared to the no-defense baseline.
In response, we propose Bit-corruption Augmented Training (BAT), a corruption-awand baseline. BAT augments data with corrupted video files from randomly chosen bit-level corruptions during training. Experiments show that BAT recovers up 7.1 points over no-defense baseline on highly- corrupted videos, while maintaining competitive performance on dlean/near-clean data. This sug- gests that the noise structure of bit-level corruptions is fundamentally different from that of adver- sarial noise, and that incorporating knowledge of bit-level corruptions is crucial to model robustness against network and file corruptions. Lastly, to better understand the effect of augmenting with dif- ferent corruptions, we explore variants of BAT that sample augmentations from different subsets of simulated comuptions. Interestingly,
find that augmenting with high levels of network and file corruption is key to improving model robustness, resulting in an accuracy gain of up to 10.1 points over the no-defense baseline. We conclude that BAT is a promising starting point for robustness to bit-level corruptions video machine learning. Our results motivate future studies understand and defend against bit-level corruption in ML robustness.
BIT-LEVEL VIDEO CORRUPTIONS
In this section, we first provide brief overview video encoding to provide intuition about the effects of network and file corruptions [Section[..] and then describe how simulate naturally- occurring range network and file corruptions (Section 2.2b
2.1 VIDEO ENCODING
present a simplified explanation video encoding, using the H.264 codec as an exam- ple (Richardson 2003). Let X #x XCXT be the space of possible videos. An uncompressed video X' consists of sequence of frames. These frames often redundant (i.e., two neighhor- ing frames in a video are likely to look similar), so the H.264 codec stores only a small fraction frames (known I- I-frames) and encodes the remaining frames differences from I-f rames. To further save space, the frames go through series of compression steps like frequency space conversion, quantization, and entropy encoding, much like JPEG compression. We notate the en- coding process as the function (nc(x), which is mapping from pixel-space to bit-space. The corresponding decoding function is denoted Dec(-).

---

Under review conference paper ICLR 2021
The H.264 video codec is also equipped with error resilience and concealment techniques for repair- ing damaged data. For example, the codec might use uncorrupted portions of a frame to decode a corrupted frame (Arbitrary Slice Order/Flexible Macroblock Order), or store redundant portions of a frame for the same purpose. However, bit-level network/file corruptions can destroy particular in- formation such that these methods are insufficient to restore visual quality; for example, I-framm frames are important references for decoding other frames, so corruptions to them often result in propagat- ing noise patches or smearing artifacts throughout the video.
2.2 SIMULATING BIT-LEVEL CORRUPTIONS
Network corruptions. We simulate packet loss, network corruption where data packets are perma- nently lost during transmission (Fig. P which can result in visual quality drops in video streaming applications (Hu & Zhang 2018, Adeyemi-Eiii
et al. (017). We simulate packet loss by streaming videos using the Real-Tiine Time Streaming Protocol (RTSP) with UDP, dropping video packets randomly using the Mahimahi network emulator (Netravali et al, 2015). This setup targets real-time applica- tions like video-conferencing that use UDP due to latency requirements.
File corruptions. We simulate random corruptions, file corruption where random bits in a video file are er- roneously flipped (Fig. 2) Such bit-flips occur as the result of bardware integrity issues like bus errors (Zhang
Packet Loss (Network Corruption)
ct al. 2010), malicious software (Sivathanu et al. 2005). or cosmic rays OoGGmmnn et al., [996].. Despite re- dundancy mechanisms in modern file systems, many such corruptions are unrecoverable due to unsafe fault- handling. or can persist undetected (Ganesan et al.) 2017j. In Appendix B we additionally explore contigu- ous corruptions, another real-world bit-corruption pat- tern in which a contiguous segment of file bitstream is replaced with random bits.
lost packet
(pellow)
Random File Corruption
bistream index
Appped
lipped
Depiction of corrupted bits (or- ange) affected by packet loss (top) and ran- dom file corruption (bottom).
Figure 2:
Corruption Levels. Our primary interest is in simulating naturally-occurce levels of bit-level corruptions. Thus, we choose ranges for p based on real-world empirical studies of packet loss and file corruption. Notationally, we describe corruptions with a parameter p E J0. 1]. the expected proportion of the video file corrupted. For packet loss, each packet is dropped with probability p. For random corruptions, we flip each bit with probability p. Real-wor world network traces have yielded packet loss rates from 0.31 31% % to 12.38% (Hu & Zhang 2018), so we simulate packet loss rates from 0.01% (p 10 up to 20% (p- 0.2). On the file corruptions side, studies of flash memory have found median bit-error rates ranging from 6.0 10-10
10 to 1.0 10-5, with 99th percentile error rate up to 1.2 x 10 (Schroeder et al. 2016), so we simulate file corruption rates from 10-8 up p 10 We provide more discussion of our chosen parameter settings in Appendix
3 BASELINE DEFENSES FOR BIT-LEVEL CORRUPTIONS
We explore three defenses as baselines for mitigating the impact of network and file corruptions. Broadly, these methods can be categorized as vyruption-agnostie which make no assumptions about network and file corruptions, or corruption-wware which explicitly incorporate knowledge of network and file corruptions. As corruption-agnosti baselines, evaluate distrriibbttio (OOD) detection and adversarial training (AT), We propose bit-corruption augmented training (BAT) as a corruption-aware baseline.
3.1 CORRUPTION- AGNOSTIC DEFENSES
Network and file corruptions often result in severe visual artifacts (see Section ples). As result, these corrupted videos can be viewed as for large amounts of corruption. In such cases, a natural defense for pre-trained video models is OOD detection, a test-time technique that attempts to identify out-of-distribution videos and filters them out. This method requires no model retraining, and makes zero assumptions about the noise structure.
4.2 for some exam- examples, particularly

---

Under review conference paper ICLR 2021
As an overview of OOD detection, we consider ML model f(x: 0) RB*WWXXXT R with parameters 0, which are optimized under categorical cross-entropy loss:
min
Σ
((Fo(x).y).
(x
where Fx(x) is the softmax output. We use ODIN (Liang et al., 2018). standard OOD detection method that uses calibrated softmax score So(x) as indicator score for flagging OOD examples:
e (x:8)7
So(x) max Σ. e
2017a). and fi(x; 0) is the logit value for class i. The calibrated score Sa(x) is then compared against a preset threshold (often 95% TPR) computed on in in-distribution data; x is flagged as OOD if it falls below the threshold and is marked in-distribution otherwise. The idea that corrupted videos yield lower average calibrated scores than clean videos.
where T is temperature scaling parameter (Guo
Adversarial Training (AT). Adversarial training (Goodfellow et al. 2014) a common technique that makes models robust to small adversarial perturbations, Adversarial training optimizes
min Σ max [(Fa(x 8).
[16] [
(x.y)~
where is the perturbation budget, or maximum permissible Lac -oom of perturhation 6. We use the training scheme proposed in Wong al. (2020), where initial 8 is randomly chosen from video x. Then, the above objective is iteratively optimized for one step. This method requires model retraining, and assumes that the noise structure of test-time input is adversarial.
the (Lo,,) neighbooooo
3.2 CORRUPTION- AWARE DEFENSES
Bit-corruption Augmented Training (BAT). We consider a corruption-aware baseline defense that exploits knowledge of bit-level corruptions enforce invariance to network and file corruptions. particular, we propose Bil-corruption Augmented Training (BAT), which regularizes the video model by augmenting training data with examples from randomly chosen bit-level corruptions:
min Σ Ze(Fo((Dec Enc)(x)), (1 F))(F((x).))
(x.
where is the set possible network and corruptions, is a randomly chosen corrup- tion for a particular video x applied with probability 0.5 (shown with indicator variable Z e Bernoull 1i (0.0(.55)) )..5)) Enc and Dec denote the video encoding and decoding transformations, respectively, o denotes function composition. This method assumes prior knowledge about the nature of network and file corruptions, but not the exact corruption parameters for test-time videos. To better understand the effect of augmenting with different corruptions, also consider variants that augment with subset of corruption parameters.
4 EFFECT OF BIT-LEVEL CORRUPTIONS ON VIDEO MODELS
report the impact bit-level corruptions on video model performance. We then report the performance of pre- existing cormutiionagmoosic agnostic defenses (Section 4.3), as well as Bit-corruption Augmented Training (BAT) corruption-aware defense (Section 4.4). finding that BAT outperforms the corruption- agnostic defenses. We also discuss variants BAT study the effect of training with particular subsets of the available corruptions.
We provide overview of our datasets and models in Section 4.1 In Section
4.1 DATASETS AND MODELS
HMDB51. The Human Motion DataBase (HMDB51) (Kuchne et al., 2011) is 51-class action recognition benchmark comprised of movie scenes and web videos. We use pre-trained 3D- Resnet18 18 architecture fine-tuned HMDB51 split (3520 train, 1530 test videos), using standard

---

Under review conference paper ICLR 2021
HMDB51 Acc.
UGF101 Acc.
MOT Detect. Rate
HMDB51 L2
UCF101
0.20
0..0
comuptinn
Network
0.9
0.8
0.8
06
Eudkdean
0.15
0.10
10.10
spacket
0.3
0,9
03-
0.05
0.05
0.0
0.0
0.00
0.00
10
1
10
10a
10
10
10a
10
10"
0.9
Exdhoos dtt
Accuaay
Random Bio ccmuption
0.8
0.6
0:16
0.15
0:10
0.5
0,5
0.3
0.05
0.00
0.0
0.00
0.
104
16
10t
104 10
104
107
10
10
10
109
-0
10
10 10+
10
107
10
10-
10
10
109
10
Proporton corupted
Propartion corrussed
Proportion comupted
Propartion comppted
Prapertion comupted
Legend
Euckdean dist. (overal)
Euclidcan
Euciddan
Figure 3:
The impact of various file corruption types model performance (left); average pixel-space Ea- clidcan distance measurements (right) incorrectly classified examples (red), correctly classified examples (green), the entire dataset (yellow) organized by corruption proportion and type (right).
training setup (Kataoka al. 2020). During evaluation, we split the input clip into contiguous 16- frame segments, outputting the action class with the highest probability averaged over segments. We report top-1 accuracy.
UCF101. UCFI01 (Soomro et al. 2012) is 101-class action recognition benchmark comprised of web videos. Our setup identical that of HMDB5 fine-tune a pre-trained 3D-Resnet18 architecture on UCF101 split (9537 train, 3783 test videos) using the setup of Kataoka et al. (2020). We report top-1 accuracy.
MOT15. MOT15 TTcal-Taixé al. 2015) is multi-objeci tracking dataset. We use DLANet-34
(Deep Layer Aggregation) architecture (Zhagg et al. (2020) pre-trained on MOT20 (Dendorfer et al., 2020 We evaluate on a subset of the train split of MOTI5 (6 clips, 12885 objects) following the setup in Zhang et (2020). We report object detection rate (1 FPR), representing the proportion of objects correctly detected.
4.2 EFFECTS OF BIT-LEVEL CORRUPTIONS
Effects of bit-level corruptions video model performance. We evaluate model performance on playable") videos under bit-level corruptions, finding that model accuracy drops substantially as corruption level increases (Fig. 33) On HMDB51 and 101, for random file corruptions, accuracy drops under packet loss for p 0.01 and for random corruptions for loss, the performance drops reach 46.5% on HMDB51 and 32.6% on UCF101.
10-6 Under packet
random file corruptions, the drop reaches 68. 9% on HMDB51 and 77.1% 1% on UCF101, On multi-object tracking, object detection rate drops near 0 for all types of corruption. Notably, performance declines start lower corruption levels for this task than for action recognition, starting packet loss greater than p = 10-3 and random corruption rates exceeding 10 More details on multi-object tracking performance provided in Appendix
Pixel-space effects of bit-level corruptions. We visualize some corrupted videos better un- derstand the qualitative effects of bit-level coruptiions. In general, misclassified videos are more visually distorted than correctly classified videos under corruption. We see this in Fig. | the model prediction changes as visual artifacts worsen and corruption level increases. Furthermore, under the same corruption type and proportion, visual artifacts correlate with erroncous model predictions, as seen in Fig. 5 In the leftmost pair of clips in Fig. 5 under packet loss (p 0.1), we see noisy patches in the Kayaking ning clip (incorrect), while the sky, grass, and person in the folffwing example (correct) are not distorted, Similar patterns hold for random corruption (Fig. [ right). On MOT15, more distorted videos yield more false positive and false negative object detections. For visualizations of corrupted videos MOTI5, see Appendix
Video files become unplayable if file metadata is corrupted. Metrics calculated playable videos only: if a video unplayable, the model does output a prediction at all. We evaluate accuracy purely based outputted model predictions. Appendix gives probubilistic overview of the effect of bit-level corruptions video playability.

---

Under review conference paper ICLR 2021
Packet Loss
Random Corruption
p=0.0001
p=10
p=0.01
p=10
p-0.2
p-10
Figure 4: Frames from clip (class VableCernisShof al varying corrupiion levels. Color shows classification. Videos become visually more distorted as the corruption level increases. Packet Loss (p=0.1)
correct
Vs.
incorrect
Random Corruption (p=10*) ApplyEyeMakeup
GolfSwing
Clean
Corrupt
Kayaking
WalkingDog
Clean
Corrupt
Figure 5: Frames from correctly and incorrectly classified examples, with both clean (Ist and 3rd rows) and corrupted (2nd 4th rows) versions shown. Each pair comrupted clips were subjected the same corruption proportion and mode (packet loss and random corruption).
To quantify this visual distortion, we compute pixel-space Fuclidean distance (l22 norm between pixels in corrupted vs. clean video clips). Fig. (isows the average L2 distance between corrupted and clean clips the entire dataset (yellow), correctly classified examples (green), and incorrectly classified examples (red). Consistent with our qualitative analysis, incorrect examples have higher Le distance on average than correct examples. Under packet loss. incorrect examples are up to 1.52 times more perturbed than correct examples average (HMDB51, network corruption, top left). random corruptions, incorrect examples are up 1.57 more perturbed on average (UCF101, random corruption, bottom right). This strongly suggests that the visual severity of a network and file-corrupted video adversely affects model performance, seen in Fig
4.3 IERRUPTION-AGNOSTIO DEFENSES FOR BIT-LEVEL CORRUPTIONS
ODIN (Out-of-distributiin Detection). As observed in Section 4.2 videos with larger pixel- space distortions are more likely to misclassified, motivating OOD detection baseline for filtering out corrupted videos. This defense does not require modifying model parameters, and be applied to any pre-trained model. Specifically, distribution data, and a corrupted version of the same split as OOD data. During inference, we apply ODIN (Liang et al. 2018) with T = 100 to corrupted datasets as filter, setting the score threshold the 5th percentile of calibrated scores on the in-distribution (95% TPR).
HMDB51 clean test
in-
We evaluate the effectiveness of ODIN based its ability successfully filter corrupted videos. Figure shows AUROC for whether ODIN can successfully filter out corrupted videos, as well as model accuracy on the videos labeled as in-distribution (clean). We observe that ODIN is best able detect corrupted videos high levels of corruption, with AUROC reaching 98.5% for the highest levels of random corruption. ODIN is less effective at discriminating packet loss, with AUROC only reaching 76.6% at the highest levels of corruption. At low levels of corruption, ODIN discards many clean/near-cled examples, AUROC degrades 0.5. This consistent with Fig. 3 which

---

Under review conference paper ICLR 2021
Table 1: Robust accuracy on readable videos in the HMDB51 action recognition classification task (%) of no defense, adversarial training (AT), and bit-coruption augmented training (BAT) as defenses against network corruptions (top) and random file corruption (middle), with aggregate statisties (bottom).
Corruption type
No defense
AT
BAT
BAT (Low)
BAT (High)
BAT (Oracle)
2/255)
32.8
27.0
32.4
32.0
47.1
35.6
44.0
48.5
Packet loss
0.01
48.3
58.3
57.4
58.4
10
60.6
49.3
58.2
57.9
58.1
57.6
10
60.0
49.4
58,9
58.4
58.7
58.0
10
14.1
13.2
15.0
13.0
24.2
27.3
5 x 10
27.6
28.4
33.7
25.7
36.8
40.8
10-5
54.5
48.0
55.6
52.8
55.7
54.1
Random
5 x0100 10-6
56.3
56.3
57.7
file corruption
10
53.3
58.9
58.4
59.1
57.5
5 x 10-7
62.0'
53.5
59.1
58.4
59.1
57.7
10
61.9
53.5
59.1
58.5
59,0
58.5
10
61.9
53.5
59.1
58.4
58.9
58.6
10
61.9
53.5
59.1
58.4
58.9
58.2
None (Standard Acc.)
61.6
53.0
59.5
59.0
59.0
N/A
Accuracy on the no-defense baseline appears to surpass buseline at some corruption levels as the num- ber of unplayable videos is non-zero at all levels of corruption tested.
suggests that the pixel-space gap between correctly classified and incorrectly classified videos nar- rows at low levels corruption, meaning that corrupted and clean videos are more distributionally similar. In conclusion, ODIN shows some promise for filtering out severely corrupted videos, but cannot effectively discriminate clean from corrupted video at low levels of corruption. Thus, ap- plying ODIN for OOD detection could result good performance filtering out highly corrupted data at the expense discarding clean/lightly corrupted data. We additionally run evaluation on energy-based OOD (Liu et al. 2020), with almost identical results. Full results OOD detection can be found in Appendix D
Adversarial Training. Given the pixel-space proximity between clean and corrupted videos at low levels of corruption (Fig. 3 right), we explore adversarial training (AT), family methods for increasing model robustness to small adversarial perturbations (Goodfellow et al. 2014:) Madry al.)
AUROC
In dist. Accuracy
Networks
0.9
comuptiinn
2017). We follow the method Wong al. (2020). The best results (E 2/255) are summarized in Ta- ble [ larger e 4/255, 8/255 yield worse perfor- mance (Appendix C.. Interestingly, it appears that AT offers little to no utility in increasing robust- ness bit-level corruptions, underperforming defense baseline both on corrupted data by up 11.3 points (packet loss,
Ramdem fi compptoon
7.75
50
10t
10tl 10rk 10s 10 Proportion compted
Proportion cormupted
Logend
Baseine (no comuppton)
AUROC
Acouraay
10-3), and on clean data by 8.6 points. This suggests that the noise structure bit-level corruptions is fundamentally different from that of adversarial noise, and that this form of AT may not be viable defense against bit-level corruptions.
Figure 6: AUROC (left), in-distribution accuracy (right) of OOD detection under packet loss (lop) and random file corruption (bottom).
4.4 BIT-CORRUPTION AUGMENTED TRAINING
We propose Bit-corruption Augmented Training (BAT) baseline corruption-awar defense that injects knowledge of file/network corruptions into video ML models. This technique augments training data with examples subjected to randomly chosen network and file corruptions. We report the results of BAT, as well as few variants, in Table [ Overall, BAT shows promise, providing up 7.1% % lift coghly-e data (packet loss, p 0.2) no-defense baseline. On

---

Under review as conference paper ICLR 2021
clean/near-clean data, BAT does only slightly worse than a no-defense baseline. This slight drop is likely attributable to the noise that BAT introduces into the model, since under BAT, we are now optimizing for accuracy not only on clean data, but also on corrupted data.
We study few variants of BAT to determine which certain network and file corruptions are more important for increasing model robustness. First, we augment using two subsets of corruption levels: low levels of corruption, i.c. those that did not cause a significant performance drop on the baseline model, and high levels of corruption. i.e., corruption levels sufficient to cause significant perfor- mance drops in the baseline modef Training on low levels of corruption (BAT-Low) decreases model robustness across almost all levels of coruuptions, but training with high levels of corruption (BAT- results in accuracy gain of up to 10.1% (random file corrupiion, p 10-") over the no-defense haseline-and outperforms vanilla BAT at almost all levels of corruption. As with vanilla BAT, the lift over the no-defense baseline is correlated with corruption proportion. This suggests that augmenting with highly corrupted videos in particular is key to increased robustness.
Lastly, we explore the effect of training and testing on the same coruptiion type and level (BAT- Oracle) as an oracle model with perfect knowledge of the corruption parameters the model will encounter. Though this assumption is unrealistic, BAT-Oracle provides insight into the utility of incorporating particular corruption levels within the BAT augmentation scheme. Interestingly, this technique lags behind BAT-High at high levels of corruption, and performs worse than BAT, BAT- High, and the no-defense baseline at low levels of corruption. This suggests that training on a variety of different corruptions is important for increasing robustness. Thus, BAT is promising starting point for improving robustness to bit-level corruptions in video machine learning. especially for more severe corruptions. Our results motivate future studies of understanding and defending against bit-level corruptions.
5 RELATED WORK
Robustness in computer vision. Imagenet-C and -P (Hendrycks & Dietterich) 2019) are Ima- geNet (Deng et al. (2009) extensions for benchmarking image classifiers with -oversaerallial cor- ruptions. Hendrycks et al. (2020) studies model robustness to distributional shifts like occlusion and perspective shift. The effect of frequency-domain features on convolutional networks has also been studied (Yin et al. 2019: Tsuzuku & Sato 2018). On the adversarial side, (Szegedy et al. (2013) in- troduce the idea of adversarial examples in ML, while Goodfellow et al. (2014) Zheng et al. (2016): Madry et al.) (2017) propose methods for creating and defending against adversarial input. Recent works also extend adversarial attacks to the video domain (Wang Cherian 2018 Wei al. 2018 Inkawhich et al.] 2018 Jiang et al.] 22019; Nach et al. 2020). Our work is not only an extension of previous image MI. robustness studies to video, but also study of network and file corruptions, which remain underexplored in the robustness literature.
Defenses against corrupted data. Augmenting data with speckle noise (Rusak et al. 2020 style transfer (Geirhos et al. 2018), mixtures of augmentations (AugMix) (Hendrycks et al. 22199) and distortions constructed from image-to-image networks (DeepAugment) (Hendrycks al. 2020
have been shown to increase robustness to common corruptions like Imagenet-C. Other defenses include certificate-based methods (Raghunathan et al. [20 b) or self-supervision (Carmon et al, 2019). Non-adversarial perturbations can be interpreted as OOD data, for which supervised (Lee 2018; [Bahat Shakhnarovich 2018, Geiffman & El-iv 2019) Gorbett & Blanchard 2020 and semi-supervised OOD detection techniques (Liu et al. 2018) have been proposed. Video anomaly detection techniques also explored in (Ben-Ari & Shwartz Ziv) 2018 Gutoski et al.) 2017]. Adversarial training (Goodfellow et al. 2014] Madry et 2017) a common defense to adversarial corruptions. We evaluate bascline non-adversarial and adversarial robustness techniques, and extend the augmentation-as-dent
al.
paradigm to let-l corruptions.
Bit-space aware robustness studies. Previous work has studied the robustness of computer vision models under various aspects of image and video compression, such as compression rates or encod- ing schemes (Srinivasan et al. 2016, (Semouur et al.) 2007). On the network side, deep learning- based packet loss concealment (Lee & Chang 2016 Lotfidereshii & Gournay 2018 Mohamed & Schuller 2020) techniques aim to recover from packet loss by predicting lost information. We view
Fxxcct corruption parameters low/high levels of corruption are described in Appendix

---

Under review conference paper ICLR 2021
our work complementary previous efforts, extending robustness studies to explicitly examine the effect of network and file corruptions at the bit-level on video model performance.
CONCLUSION
In this paper, we take first step investigating the effect of network and file corruptions video model robustness, finding that these corruptions significantly decrease performance on benchmark tasks like action recognition and multi-object tracking. We propose corruption-awy baseline, Bit-corruption Augmented Training (BAT), which incorporates knowledge of network and file corruptions directly. We show that BAT can improve performance by up to 7.1 % accuracy, and outperforms corruption-agnoste levels while perserving performance augmenting with high levels of corruption particular. This suggests that exploiting knowledge network and file corruptions in model training is important for improving robustness, making BAT compelling starting point for future studies of network and file corruptions and model robustness. hope this work motivates future studies defenses against network and corruptions, as well as other real-world non-ad an--adverraiial corruptions.
defense
baselines (OOD detection and AT) at wide range of corruption
clean data. Further experiments highlight the utility of
REFERENCES
A.O. Adeyemi-Fjeye, M. Alreshoodi, I.. Al-Jobouri, M. Fleury, and J. Woods. Packet loss visi- bility across sd, hd, 3d, and uhd video streams. Journal of Visual Communication and Image Representation. 45:95 106, 2017.
Ayse Elvan Aydemir, Alptekin Temizel, and Tugba Taskaya Temizel. The effects of jpeg and jpeg2000 compression on attacks using adversarial examples, 2018.
Yuval Bahat and Gregory Shakhnarovich. Confidence from invariance to image transformations, 2018.
Itamar Ben-Ari and Ravid Shwartz-Ziv. Attentioned convolutional LSTM inpaintingnetwork for anomaly detection in videos. CoRR, 2018.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski. Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks, 2016.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled
data improves adversarial robustness, 2019.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset, 2017.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset, 2019.
Yiping Chen, Jingkang Wang, Jonathan Li, Cewu Lu, Zhipeng Luo. Han Xue, and Cheng Wang. Lidar-video driving dataset: Learning driving policies effectively. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5870-5878. 2018.
Nilaksh Das, Madhuri Shanbhogue, Shang-Tse Chen, Fred Hohman, Siwei Li, Li Chen, Michael E. Kounavis, and Duen Horng Chau. Shield: Fast, practical defense and vaccination for deep learn- ing using jpeg compression, 2018.
Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixé, Mot20: A benchmark for multi object tracking in crowded scenes, 2020.

---

Under review conference paper ICLR 2021
J. Deng, W. Dong, Socher, Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M. Roy. A study of the effect of jpg compression on adversarial images, 2016.
Daniel Y. Fu, Will Crichton, James Hong, Xinwei Yao, Haotian Zhang, Anh Truong, Avanika Narayan, Maneesh Agrawala, Christopher Ré, and Kayvon Fatahalian. Rekall: Specifying video events using compositions of spatiotemporal labels. 2019.
Adrien Gaidon, Zaid Harchaoui, and Cordelia Schmid. Temporal localization of actions with ac- toms. IEEE transactions on pattern amalysis and machine intelligence, 35(11):2782 279222222279 2013.
Aishwarya Ganesan, Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci- Dusseau. Redundancy does not imply fault tolerance: Analysis of distributed storage reactions to file-system faults. ACM Trans. Storage, 13(3), September 2017.
Yonatan Geifman and Ran Yaniv.. Selectivenet: A deep neural network with an integrated reject option, 2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge. Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness, 2018.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2014.
Matt Gorbett and Nathaniel Blanchard. Utilizing network properties to detect erroneous inputs, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning- Voluue 70, pp. 1321-1330. JMLR. org, 2017a
Chuan Guo, Mayank Rana, Moustapha Cissé, and Laurens der Maaten. Countering adversarial images using input transformations. CoRR, abs/1711.001 2017b.
Matheus Gutoski, Nelson Marcelo Romero Aquino, Manassés Ribeiro, André Lazzaretti, and Heitor Lopes. Detection of video anomalies using convolutional autoencoders and one-class support vector machines. 11 2017.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2016.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayaman. Augmix: A simple data processing method to improve robustness and uncertainty, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer, The many faces of robustness: A critical analysis of out-of-distribution generalization, 2020.
Zhiguo Hu and Qiqiang Zhang. A new approach for packet loss measurement of video streaming and its application. Multimedia Tools Appl., 77(10): 17(10):11589) 1119:11999111111 1608, May 2018.
Nathan Inkawhich, Matthew Inkawhich, Yiran Chen, and Hai Li. Adversarial attacks for optical flow-hased action recognition classifiers, 2018
Bart Jansen, Timothy Goodwin, Varun Gupta, Fernando Kuipers, and Gil Zussman. Performance evaluation of webrtc-based video conferencing. SIGMETRICS Perform. Eval, Rev,, 45(3):56-66 68, March 2018
10

---

Under review conference paper ICLR 2021
Linxi Jiang, Xingjun Ma, Shaoxiang Chen, James Bailey, and Yu-Gang Jiang. Black-box adversarial attacks on video recognition models. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 6644872,44 2019.
Hirokatsu Kataoka, Tenga Wakamiya, Kensho Hara, and Yutaka Satoh. Would mega-scale datasets further enhance spatiotemporal 3d cnns?, 2020.
H. Kuehne, H. Jhuang, Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recognition. In Proceedings of the International Conference on Computer Vision (ICCV), 2011.
Laura Leal-Taixé, Anton Milan, Ian Reid, Stefan Roth, and Konrad Schindler. Motchallenge 2015: Towards benchmark for multi-target tracking, 2015.
B. Lee and J. Chang. Packet loss concealment based on deep neural networks for digital speech transmission. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 24(2):378- 387, 2016.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks, 2018
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of oft-of- distribution image detec- tion in neural networks. International Conference on Learning Representations (ICIR), 2018.
Si Liu, Risheck Garreaali, Thomas G. Dietterich, Alan Fern, and Dan Hendrycks. Open category detection with pac guarantees, 2018.
Weitang Liu, Xiaoyun Wang. John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection, 2020.
R. Lotfidereshgi and P. Gournay. Speech prediction using an adaptive recurrent neural network with application to packet loss concealment. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5394-5398, 2018
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2017.
Mostafa M. Mohamed and Björ W. Schuller. Concealnet: An end-to-end neural network for packet loss concealment in deep speech emotion recognition. 2020.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, and Aude Oliva, Moments in time dataset: one million videos for event understanding. 2018.
Seyed-Mohsen Moosavi-Dezfooll, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations, 2016.
Muhammad -ajjd Mushtaq and Abdelhamid Mellouk. 2 methodologies for subjective video streaming qoe assessment. In Muhammad-Sajid Mushtaq and Abdelhamid Mellouk (eds.), Qual- ity of Experience Paradigm Multimedia Services, pp. 27 57. Elsevier, 2017.
Itay Nach, Roi Pony, and Shie Mannor. Flickering adversarial attacks against video recognition networks, 2020.
Ravi Netravali, Anirudh Sivaraman, Somak Das, Ameesh Goyal, Keith Winstein, James Mickens, and Hari Balakrishnan. Mahimahi: Accurate acord-andd-relly for HTTP. In 2015 USENIX An- nual Technical Conference (USENIX ATC 15), pp. 417-429, Santa Clara, CA, July 2015. USENIX Association.
J. O'Gorman, J. M. Ross, A. H. Taber, J. F. Ziegler, H. P. Muhlfeld, C. J. Montrose, H. W. Curtis, and J. I.. Walsh. Field testing for cosmic ray soft errors in semiconductor memories. IBM Journal of Research and Development, 40( 40(1):4 :-41500, 1996.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang, Certified defenses against adversarial exam- ples, 2018a.

---

Under review conference paper ICLR 2021
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Semidefinite relaxations for certifying ro- bustness to adversarial examples, 2018b.
lain E. Richardson. H1264 and MPEG-4 Video Compression. Wiley, 2003.
Evgenia Rusak, Lukas Schott, Roland S. Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions, 2020.
Bianca Schroeder, Raghav Lagisetty, and Arif Merchant. Flash reliability production: The ex- pected and the unexpected. In 14th USENIX Conference File and Storage Technologies (FAST 16), pp. 67-80, Santa Clara, CA. February 2016. USENIX Association.
Rowan Seymour, Darryl Stewart, and Ji Ming. Comparison of image transform-based features for visual speech recognition in clean and corrupted videos. EURASIP Journal on Image and Video Processing, 2007.
Gopalan Sivathanu, Charles Wright, and Erez Zadok. Ensuring data integrity in storage: Tech- niques and applications. In Proceedings of the 2005 ACM Workshop on Storage Security and Survivabiliry, pp. 26-36, 2005.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012.
V. Srinivasan, S. Gul, S. Bosse, J. T. Meyer, T. Schierl, C. Hellge, and W. Samek. On the robustness of action recognition methods in compressed and pixel domain. In 2016 6th European Workshop on Visual Information Processing (EUVIP), pp. 1-6, 2016.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2013.
Yusuke Tsuzuku and Issei Sato. On the structural sensitivity of deep convolutional networks to the directions of fourier basis functions, 2018.
Jue Wang and Anoop Cheriam. Contrastive video representation learning via adversarial perturba- tions, 2018.
Xingxing Wei, Jun Zhu, and Hang Su. Sparse adversarial perturbations for videos, 2018.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training, 2020.
Dong Yin, Raphael Gontijo Lopes. Jonathon Shlens, Ekin Cubuk, and Justin Gilmer. fourier perspective on model robustness in computer vision, 2019.
Yifu Zhang, Chunyu Wang. Xinggang Wang, Wenjun Zeng, and Wenyu Liu. simple baseline for multi- tracking, 2020.
Yupu Zhang, Abhishck Rajimwale, Andrea C. Arpaci--Dsssent and Remzi H. Arpaci-Dusseed End- to-end data integrity for file systems: A zfs case study. In Proceedings of the 8th USENIX Con- ference File and Storage Technologies, pp. 3, 2010.
Stephan Zheng, Yang Song. Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the IEEE Conference on Computer Vision and Patern Recognition JCVPR),Jnn 2016.
12

---

Under review conference paper ICLR 2021
Table 2: Overall accuracy on all videos in the HMDB51 action recognition classification task (%) of no defense, adversarial training (AT), and bit-coruption augmented training (BAT) as defenses against network corruptions (top) and random file corruption (middle), with aggregate statisties (bottom).
Cormuption type
No defense
AT
BAT
BAT (Low)
BAT (High)
BAT (Oracle)
2/255)
25.6
32.4
25.0
36.1
45.2
44.0
Packet loss
0.01
51.8
57.3
57.1
57.4
56.3
10 a 10
59.0
52.4
58.0
57.5
57.6
57.4
59.3
52.7
58.6
58.2
58.5
57.6
10
1.3
1.2
1.6
1.4
2.5
1.5
5 x 10-
5.0
4.6
6.4
4.8
6.9
5.0
10-5
31.0
26.4
33,0
32.5
34.3
32.8
Random
5 10-6
35.4
43.3
43.5
file corruption
10
45.2
55.9
55.3
55.9
54.4
5 x 10-7
59.9
46.5
57.6
56.8
57.7
56.1
10
61.4
47.1
58,8
58.3
58.8
58.3
10
61.6
47.6
59.1
58.4
58.8
58.0
10
61.6
47.6
59.1
58.4
58.9
58.0
None (Standard Acc.)
61.6
53.0
59.5
59.0
59.0
N/A
Table 3: Model performance HMDB51 with various defenses under contiguous file corruptions.
Corruption lype
P
AT
BAT
BAT (Low)
BAT (Highh)
BAT (Oracle)
defense
(c 2/255)
0.9
25.6
25.8
32.2
29.2
34.9
38.1
0.75
41.4
37.2
44.1
41.2
46.0
44.6
Contiguous
0.5
49.5
45.0
51.6
49.6
51.7
51.6
file corruption
0,25
55.0
47.8
54.9
52.9
54.6
53.9
0.1
57.9
51.0
56.3
56.8
57.4
55.1
0,01
61.2
$2.0
59.5
57.8
59.2
58.5
A METRICS INCLUDING READABLE FILES
For completeness, we provide performance metrics here for accuracy on all videos, including un- playable videos. Unplayable videos are logged as incorrect. We notice similar trends as described in Table in that BAT outperforms the no-defense baseline at high levels of corruption, and is close the no-defense baseline at low corruption of accuracy. We also similar results the variants: BAT-High outperforms BAT high levels corruption, which suggests that augmenting with high levels of corruption important for model robustness, while BAT-Oracle underperforms BAT general, suggesting that augmenting with wide range of corruptions positively influences model robustness. These the same conclusions drawn in Section 4.4 As observed earlier, adversarial training generally hurts model performance.
B A STUDY ON CONTIGUOUS FILE CORRUPTIONS
We present study contiguous file corruptions. In contiguous corruptions, segment of video bitstream is replaced with random bits. These errors occur due to sudden temperature changes (ther- mal asperity), malware (Sivathanu et al. (2055), misdirected data writes (Zhang et al.) (010)). While these corruptions are plausible in the real world, corruptions do not yield significant perfor- mance drops or artifacts until over 10% of the file bitstream all files are corrupted. For complete- ness, we include results on this type of corruption in Table 2 as well as in the remaining results in the Appendix.

---

Under review conference paper ICLR 2021
C FULL RESULTS OF ADVERSARIAL TRAINING
summary, adversarial training offers little to no utility in defending against network corruptions. In Table E we see that at all levels of corruttion except for random corruption, p 5e - and contiguous corruption, p 0.9, adversarially trained models sigaificantly less accurate than the normally trained model. training accuracy tends to get closer to baseline accuracy. Interestingly, the gap between an adver- sarially trained model and the baseline model narrows corruption proportion increases. Thus, this mode of adversarial training is an insufficient defense against network and file corruptions.
notice that ε decreases (i.e. adversary weakens), adversarial
Table 4: Accuracy of adversarially trained models (s 2/255, 4/255, 8/255) against random file (top). contiguous file (middle), and network corruptions (bottom). AT adversarial training via FGSM random initialization.
p
defenses
(T 2/255)
AT (5 1/255)
AT (E 8/255)
0.2
33.8
30.1
27.0
26.7
0.1
45.6
38.6
35.6
34.7
Packet loss
0.01
59.5
52.1
48.3
46.0
10-3
61.4
52.8
49.3
46.9
10
60.5
52.9
49.4
47.3
10-4
14.1
13.2
9.7
11.8
10
28.4
23.2
10.4
10
48.0
43.8
Random file corruptions
5 x 10
58.7
51.3
47.4
44.4
10-6
61.5
53.3
49.2
46.5
10
62.0
53.5
49.4
46.8
10-7
61.9
53.5
49.5
46.7
5 x 10-8
61.9
53.5
49.3
46.7
10
61.9
53.5
49.3
46.7
0.9
25.6
25.8
26.5
20.4
0.75
41.4
37.2
37.4
35.9
Contiguous file
0.5
49.5
45.0
43.4
40.3
corruptions
0.25
55.0
47.8
44.2
44.4
0.1
$7.9
51.0
47.1
45.7
0.01
61.2
52.0
48.1
46.9
corruption
53.0
49.5
47.5
D FULL RESULTS OF OUT-OF-DISTRIBUTION DETECTION
MACHINE LEARNING- BASSD SOLUTIONS
Table 5 provides full results for our study of ODIN OOD detection as a defense for network and
file corruptions. We report in-distribution accuracy (left) and AUROC (right). As in Fig. 6 AUROC relatively high the maximum levels of corruption (76.6% on packet loss, 98.5% on random file corruptions). but drops down to essentially random at low levels of corruption.
Table 6 provides full results for the energy-based OOD detection (Liu et al. 2020) framework defense for network and file corruptions. As brief overview, similarly ODIN, energy-based OOD detection produces a score given by the Helmholtz free energy function, which
So(x) -T Σ eoxx00/r
-T
with and (x(x as defined for ODIN.
We report in-distribution accuracy (left) and AUROC (right). The setup is identical to ODIN, except we have T = 1 following the authors' specification. Results for energy-basee OOD detection are very similar that of ODIN, with AUROC as high 77.0% and 98.7% packet loss and random

---

Under review conference paper ICLR 2021
Table 5: In-distribution accuracy and AUROC for ODIN OOD detection (T 100), HMDB51.
Corruption type
In-dist. Accuracy
AUROC
0.2
37.4
76.6
0.1
48.5
65.8
Packet
0.01
62.3
52.2
10-3
62.5
50.5
10-4
62.1
50.5
10-4
19.8
x 10-5
32.8
95.2
10-5
57.0
73,9
Random
5 10-6
60.0
63.9
file corruption
10
62.9
5 x 10-7
63.5
51.7
10
63.8
50.4
10
63.7
50.2
10
63.7
50.2
0.9
35.5
96.2
0.75
53.2
83.6
Contiguous
0.5
52.9
71.1
file corruption
0.25
57.2
62.7
0.1
60.1
58.7
0.01
61.6
55.6
None
0
63.8
50.0
Table 6: In-distribution accuracy and AUROC for energy-based OOD detection, HMDB51.
Corruption type
P
In-dist. Accuracy
AUROC
0.2
36.6
77.0
0.1
47.3
66.3
Packet loss
0.01
62.7
52.5
10-3
62.4
50.7
10-4
62.6
50.5
10-4
9.1
98.7
10-8 10
31.8
95.8
54.5
Random
5 x 10-6
60.2
64.7
file corruption
10-6
63.5
53,5
10
63.4
51.8
10-7
63.6
50.5
x 10
63.6
50.2
10
62.0
50.3
0.9
36.6
0.75
49.4
Contiguous
0.5
52.9
file corruption
0.25
57.1
58.3
58.8
62.1
None
0
63.6
50.0
file corruptions, respectively. However, both techniques are near random at low levels of corruption. We provide side-by-side comparison of ODIN and energy-base OOD detection in Figure ??, with plots AUROC and in-distribution accuracy under packet loss and random file corruptions.
15

---

Under review conference paper ICLR 2021
Accuracy
AUROC
In-dist. Accuracy
1.00
1.00
Network
Natwork comuuption
comuptiin
0.75
0.75
(packat losa)
0.50
0.3
(packol losa)
0.50
03
10
10
10 10
0.8
09
Random file
0.76
Random fie comupption
0.75
corruption
50
0.50
10 10
10 10 10
10 10
10 10 10-
10 10 10 10 10
10 10
10 10 10*
Proportion comupted
Proportion comppted
Proportion comupted
Proportin corupled
Legend
Legend
.....: Basaino (no compption)
- AUROC
Accuracy
Baseline (no corruptio)
-
AUROC
Accuracy
(a) AUROC and in-distribution accuracy of ODIN OOD detection, HMDB51.
(b) AUROC and in-distribution accuracy of energy-
based OOD detection, HMDB51.
Figure 7: Comparison of ODIN and energy-based OOD detection methods.
Table 7: In-distribution accuracy and support for baaedd corrupted video detection, HMDB51.
Corruption type
p
In-dist. Accuracy
Support not corrupted)
0.2
N/A
0.1
N/A
Packet loss
0.01
N/A
10
N/A
10
N/A
10
N/A
10-5
N/A
10-8
Random
5 x 10
66.7
42
file corruption
10-6
63.1
620
5 x 10
62.7
944
10-7
5 x 10
61.9
1508
10
61.9
1508
None
61.6
1530
D.2 CHECKSUM BASED FILE INTEGRITY CHECKS
In Table 7 we provide results testing the effectiveness of checksum-based file integrity checks, a model-agnostic, machine learning-free technique for detecting file corruption. We flag video corupteddouttoo-disttee distiitiibttion if the checksums the corrupted vs. original video fail match. We use the MD5, SHA1, and SHA256 checksums using the corresponding Linux utilities; however, all checksums produced identical detections.
Ultimately, this method discards large quantity of clean near-clean videos, meaning that its effectiveness is limited. In fact, even re-streaming video through link with zero packet loss results in checksums no longer being equal, which can happen if there are differences in the uplink and downlink codec parameters. Since checksums merely binary indicators of corruption, and have no notion of how severe or recoverable corruption is, this a core limitation of this technique in the setting of model robustness to bit-level network/filc corruptions.
16

---

Under review conference paper ICLR 2021
E MULTI-OBJECT TRACKING
We provide full results for our robustness study of multi-object tracking, with multi-object tracking. false positive count, and false negative count.
E. ROBUSTNESS STUDY RESULTS
On the MOT task, for all modes of corruption, multi-object tracking accuracy (MOTA) drops (Fig. 9.. For random and network corruptions, performance drops while for contiguous corruptions, performance drops by 160% to -0.453. The negative value is possible due to the form of the MOTA calculation:
MOTA E, FN, SWITCH, EO
(I)
where FP: is the number of false positive detections at frame/timestep t, FN, is the number of missed/false negative detections at timestep 1, SW SWICHH, is the number of "identity switches" at time & (i.e. one object is now classified as another object), and OBJ, is the total number objects at time t. The negativity comes from the fact that FP, and SW W, are not bounded above. This is likely attributable to the false positive rate being much higher on the contiguous corruption experiments; empirically, the maximum average false positive count in these experiments is 7.41x greater than the maximum average number (8030, contiguous corruption, p = 0.75) of false positives in any other experiment (1083, packet loss, 0.01).
We also comment on the trends in the false positive and negative counts (center, right columns of Fig. 9 On each type of corruption, the number of false positives first increases, then decreases as corruption proportion increases. Similarly, the number of false negatives consistently increases. On average, there are overwhelmingly more false negatives than false positives.
To understand the patterns false positive and false negative counts, we provide an illustrative visualization for packet loss in Fig, X In the ground truth example (Fig, Bap. the model detects three individuals, but in the example with packet loss p 0.01 (Fig. (b). duplication artifacts, resulting a double detection of the rightmost individual (false positive). This signals partial non-I frame corruption, as the motion information fails to erase some parts of the old image, while the new location of the individuals is not updated correctly, resulting in the double images. However. as corruption proportion continues to increase, the video becomes so damaged that the model makes very few detections, hence the continual increase in the number of false negatives. This is shown in Fig. [c] showing example packet loss 0.1, the video becomes so distorted that two out of three the individuals detected in the clean image are not detected at all. This signals T- frame destruction, as the image background is entirely grayed-out. The same visual trends hold contiguous and random corruption.
multiple object
(a) Clean, p 0
(b) Packet loss, p 0.01
(c) Packet loss, p 0.1
Figure Frame #8 from the PETS09-SS S2LI sequence, showing model predictions on clean data (left), p 0.01 packet loss (center), and p 0.1 packet loss (right).
17

---

Under review conference paper ICLR 2021
False positives
negatives
1000
Random computtion
MOTA
0.5
Count
Gount
10000
0.3
-0.5
5000
0
10"
10
10-
1.0
Contiguous file
MOTA
0.5
Count
7500
Count
10000
corupption
0.0
5000
5000
-1.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.00
0.25
0.50
0.75
1.0
1500
Network corruption
0.5
Count
Count
(packet loss)
-0.5
500
-1.0 10
10
10
10
10
10
10
10
10
10
10
10"
Corruption proportion
Coruption proportion
Corruption proportion
Legend
accuracy
Acouracy
false positives
nagatives
Figure 9: The impact of various file corruption types model accuracy. false positive and false negative for the multi-object tracking task. Error bars denote standard deviation over random seeds.
F
ADDITIONAL EXPERIMENTAL DETAILS
F.1 CORRUPTION SIMULATION PARAMETERS
Random file corruptions. We tested the effect of random corrpptions semi- semi-logarithmic scale, experimenting with = [le-8, 8, Se 8, 1e-1,,7,,0 76- -ele--0 5e 6, le -5-5, 5e-5, 5e 14- We chose this scale since corruption proportions greater than p le 4 yielded too few playable examples get stable estimations of the accuracy and average pixel-space Euclidean distance. To randomly comrupt video, we convert the video file into a raw bytearray and flip each bit independently with probability For BAT variants, we define our cutoff for low levels of corruption as 10 10-6, and corruption as high otherwise.
Contiguous file corruptions. We tested the effect of contiguous corruptions at [0.01,0.1, 0.25, 0.5, 0.75, 0.9]. We chose this scale since the effect of corruption proportion on model accuracy and average pixel-space Euclidean distance was more visually apparent on this convert the video file into a
scale than on a logarithmic scale. To apply contiguous corruption, [(1 p) L.J. where f. is the bytearray and choose random starting bit from between 0 and length of the video. For BAT variants, we define our cutoff for low levels of corruption as p & 0.1, and corruptions as high otherwise.
Packet loss. We tested the effect of packet loss rates = [0.0001, 0.001, 0.01, 0.1, 0.2]. This parameter range yielded wide variety of artifacts and allowed us observe the accuracy decline of pre-trained video models. We stream videos using fmpegg using the Real-Time Streaming Protocol (RTSP) over the User Datagram Protocol (UDP), and simulate packet loss by streaming within LossShell 11 in the mahimah: network simulation program (Netravali et al. 2013). Each packet is dropped independently with probability p. For BAT variants, define cutoff for low levels corruption as & 0.01, and corrppiion as high otherwise.
Probabilistic perspective on bit-level corruptions. We provide a brief probabilistic perspective on the chance of corruptions making the video unplayable by hitting critical section of the video file bitstream. For a video of /' packets, the probability video becomes unreadable is -1 MM corruption random corruption is 1 (1 M) B.p where the number of
where M is the fraction of video composed of critical metadata. The probability
hitting critical section due bits in the video (P & B). For contiguous corruptions, making the simplifying assumption that critical sections of the video are contiguous, the expression is [M/(1 p)J. which is close to linear outside the neighhorhood of I for sufficiently small M. In practice, M varies by video container

---

Under review conference paper ICLR 2021
format (e.g. AVI, MP4) and with the size of the video file. Replacing M with the proportion bits in the video that results in distortions when corrupted, we can derive analogous expressions for the probability of visible corruption.
F.2 MODEL ARCHITECTURE AND CONFIGURATION
Training. For action recognition, follow the fine-tuning setup Kataoka et al. (2020). We obtained the pre-trained model from the 3D-ResNets-PyTorce GitHub repository thttps://githes wwwtttuuu..omeeeeeeee comm/ensshhaaa.... -ew----eeesseyPeeeeeeee
using the model trained for 200 epochs on Kinetics-700 (Carreira et al., 2019) and MiT (Montort et al.] 2018) (33d1__MM200
pth). We fine-tune on split I of HMDB51 and UCFI01, sampling a ran- dom 16-frame segment from each video. If the video is shorter than 16 frames, loop the video to the required length. All videos are resized to 112 pixel frames. We also apply with a probability 0.5 a random crop across all frames of the segment with crop scales {1, -.1 T
2 2). where a crop scale of means that the crop has the same width and height, a crop scale of means that the shorter side is 2 the length of the longer side, and likewise for the other crop scales. We flip each clip with probability 0.5 and subtract the mean values of ActivityNet from each channel (RGB: [114.7748, 107. 7354, 19.4755).. We train for 50 epochs using stochastic gradient descent with a learning rate le - 3, momentum 0.9, and weight decay of lc 5. We do not use early stopping, we reduce the learning rate by factor of 10 if validation loss does not improve for 10 epochs, training for maximum of 50 epochs. We use the same training parameters for adversarial training and coruuptio-augmeented training.
27
We do not perform further training on multi-object tracking.
Evaluation. For action recognition, we split the clip into contiguous non-overlapping 16-frame segments, matching the length of segments used during training, and evaluate the model on each segment. We also subtract the mean values of ActivityNet from each channel (RGB: [114.7748, 107.7354, 99.4755] during evaluation for consistency with training. If the final video segment is shorter than 16 frames, we loop until it reaches the requisite 16 frames. For our predic- tion, output the class with the highest probability averaged over all segments.
For multi-object tracking, we simply pass in the comrupted versions of MOTI5- train to the preset evaluation script for MOTI5, which can be found at https: //g.tbu.....gg///////gggg....... wtc.ub.ccom/aa.. coom/ffzang/// Faai rMOT.
Hardware. Training and evaluation was done on one NVIDIA Tesla P100 GPU with 16GB VRAM.
F.3 CALCULATING PIXEL-SPACE EUCLIDEAN DISTANCE
For completeness, we explicitly formulate our calculation of pixel-space Euclidean distance. video can be interpreted as four-dimensional tensor, with time, channel (representing colors), height, and width dimensions, which and UCF101 datasets, the channel dimension has size 3, corresponding to RGB values of each pixel, which we represent as a float between O and inclusive.
notate as C, T, W, H, respectively. In the HMDB51
Average pixel-space Euclidean distance measures the amount each pixel at point in time and space changes on average. This is calculated as the pixel-wise L2 norm in the spatial and temporal di- mensions averaged over all pixels in the video tensor. As an example, suppose that we have two video tensors, Vo and Vi, each with dimensions (C' x T x W x H). Consistent with Section F.2 our experiments, C
and W / 1 12. The number of frames in video T varies by video. In the case that durations To and T, for videos Vi and V1, respectively, are different, we take length 7 min(To. Ti) and truncate each video in the time dimension to have dimension (C's T" x W H). Let D Vo V1, the element-wise difference; then the pixel-space Euclidean distance is
H D 2
1
(2)
v8 W.
19

---

Under review conference paper ICLR 2021
where Dewh is the pixel at the location specified by (w, h) at time Note the v in the denominator of the left-most term of Eq. 2 This is a normalization constant that ensures the pixel-space Euclidean distance between two video tensors stays between 0 and inclusive.
20


Reviews:
[{"review_id": "icrC4nTbJYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work investigates the problem of building robust video prediction models in the presence of signal corruption. The problem itself is not widely studied and experimental work like this one certainly opens some possibilities. The solution on the other hand is surprisingly simple and easy to implement. It serves the purpose of introducing the problem to a wider audience, and shed some light in different types of remedies. \n\nClarification:\n\n1. What is the importance of solving this type of problem in the general video application setting? One would assume there are existing methods in signal processing without machine learning to tackle the issues well, especially when knowing the type of the noise and their corruption ratio in range. One would hope a section of related work on it and discuss the further reliance of a data driven approach.\n2. The \"No defense\" baseline at some corruption levels are much better in accuracy. I'm not sure I understand the explanation given in Table 1. \n3. What models are used to perform action recognition and tracking? Does the performance of BAT have anything to do with choice of models?\n\nIn  all, the problem itself seems worth a pursuit if stated with more context. The concern is that the solution is fairly specialized on the known noise types and their general range. One would argue that adding input noise has already been used in numerous previous work to improve the robustness of models.  The major issue, which is completely ignored, is that adding input noise could also hurt the model performance on clean inputs (assuming not all videos are corrupted). This is the aspect that is worth a discussion, a trade off that needs some illustration.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NDLE6jzkVaS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\n\nThe authors evaluate the effect of bit-level corruption, including network packet losses and bit corruptions, on video models such as action recognition and multi-object tracking. They found that the model performances drop significantly under severe corruption levels. To overcome this issue, they propose a defense method named Bit-corruption Augmented Training (BAT) to enhance the robustness of the model by embedding corrupted video samples in the training process. Results show that BAT is able to improve the model robustness over other methods such as Out-Of-Distribution (OOD) detection and Adversarial Training (AT). \n\n**Strength:**\n\n- Explores an interesting and new problem space for video model robustness.\n- A range of realistic corruption rates was evaluated.\n\n**Weakness:**\n- The threat model for the defense is unclear.\n- No comparison with naive fault tolerance methods at the network and video file level.\n- The proposed BAT method is trivial and the performance is not good.\n- Packet loss should not be categorized into data corruption.\n\n**Detailed comments:**\n\nThis paper explores a topic that is both realistic and interesting for the increasingly popular video model applications. I appreciate that the authors evaluated their approach under realistic packet loss and bit-corruption rates that would occur in the physical world. Yet the paper has several points which I think significantly weaken its value. I will detail them as follows.\n\n- The authors claim the proposed BAT method as well as OOD detection and AT as *defenses*. However, I can hardly find any evidence that the authors evaluated these methods under an adversarial setting, i.e., the paper does not assume the existence of an attacker. It seems to me that this paper is more on the side of fault tolerance than defenses for attacks.\n- For detecting or correcting network packet losses and bit-corruptions in the files, a straightforward solution is applying existing network and memory/file-level fault-tolerance methods. For example, checksum or error-correction codes are two easily applicable solutions. The benefit of applying these methods is that they are model-agnostic and would not affect model performance under normal conditions.\n- The BAT method proposed by the authors seems rather trivial to me. How is this different from data augmentation? In addition, the reported performance of BAT is not good. For example, would a 27.3% accuracy under 10^-4 random file corruption bring enough utility for the model?\n- Lastly, I do not think it is a good idea to categorize packet losses as bit-level corruption since data loss is fundamentally different from data corruption. In other words, data corruption considers cases when data got changed but data loss only considers when data is missing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Irf2mqioNRz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-rebuttal: The rebuttal partly addresses my concerns, so I would like to change my score to 4.\n\nThis paper simulates network and file corruptions at multiple corruption levels, and explore corruption-agnostic and corruption-aware defenses. The presented Bit-corruption Augmented Training enhances the robustness of the video machine learning models. Experimental results show the effectiveness.\n \nPros: \nThe experiments are very extensive. The effectiveness of the presented method compared with the corruption-agnostic is clearly validated.\n\nCons:\n1.      The novelty is limited. Considering the attack in the encoding and decoding process have been studied, and sophisticated methods [1] have been proposed. Compared with [1], I wonder about the advantage of the proposed method.\n2.      The experimental results are not convincing. Baselines such as [1] is not included in the experimental results for fair comparison. I think the baselines in the manuscript is too weak.\n3.      I think the idea is too ad-hoc, which just generates adversarial samples in the space of encoder and decoder instead of the space of the classifier. I think the difference is not significant.\n \n[1] Rakin, A. S., He, Z., & Fan, D. (2019). Bit-flip attack: Crushing neural network with progressive bit search. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1211-1220).", "extra_scores": {}, "extra_text": {}}, {"review_id": "1biktfSjxZ-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors explored the robustness of video machine learning models to bit-level corruption. They investigated previous methods such as Out-Of-Distribution (OOD) detection and adversarial training and found that they are not effective enough to defense against the bit-level corruption.  Accordingly, this paper proposed a new framework, Bit-corruption Augmented Training (BAT), which utilizes the knowledge about corruption by bit-level data augmentation at the training stage. Also, the authors argue that the proposed method outperforms the previous methods in handling the bit-level corrupted dataset.\n\nWhile the proposed method seems simple and more effective than the previous studies, the authors do not currently provide a sufficient amount of evidence to support their claim.\n\nPros\n-\tThe proposed defense technique is effective on the bit-level corruption of videos. Also, it is simple to apply in real-world deployment.\n-\tThis is the first work that addresses the robustness against bit-level corruption of videos.\n\nCons\n-\tThe technical novelty of this paper is not significant. They proposed a new framework for robustness to video bit-level corruption. However, data augmentation method is the main technical contribution this paper proposes. I think adopting existing technique to bit-level corruption problem is not significantly novel.\n-\tThere is insufficient evidence that the proposed method is better than previous studies. I think additional experiments on OOD detection are needed. The authors implemented only one method as a baseline, ODIN [1]. Since there exist many other state-of-the-art methods than ODIN, such as Mahalanobis [2] and Outlier exposure [3], they should also be compared. In addition, there is no explanation of an input preprocessing method proposed in ODIN. The authors need to articulate why they omit the preprocessing. Also, additional experiments on UCF101 by using corruption-agnostic and corruption-aware defenses would make the paper more convincing.\n-\tFurthermore, in Section 4.3, the rationale behind the importance of detecting low-level corrupted samples by OOD is not elaborated. As the authors mentioned, largely corrupted videos are likely to be misclassified, while the accuracy for videos with low-level corruption does not decrease.\n-\tFor clarity, I recommend the authors to correct the minor typos in the paper. The adversarial training in Section 4.3 seems to be worse compared to the no-defense baseline by 8.6 points (not 8.1 points) on clean data. \n\n[1] Liang et al., \"Enhancing the reliability of out-of-distribution image detection in neural networks.\", ICLR'18\n\n[2] Lee et al., \"A simple unified framework for detecting out-of-distribution samples and adversarial attacks.\", NIPS'18\n\n[3] Hendrycks et al., \"Deep anomaly detection with outlier exposure.\", ICLR'19\n\n----------------------------------\nAfter rebuttal:\n\nI appreciate the authors for thoughtful response and additional experimental results, which are helpful for further understanding of the manuscript. Especially, the additional experiment on the recent OOD detection method addresses my concern about the evidence that the previous OOD studies are not sufficient for defending the bit-level corruption.\n\nUnfortunately, I am still not sure about the technical novelty of this paper. I agree that the paper proposed a new problem setting, but I do not think that the technical novelty is significant, given the proposed approach of just applying the data augmentation simply at a bit level, rather than at a pixel level.\n\nDue to this concern, I want to keep my rating of \"4. Ok but not good enough - rejection\" as it is.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates robustness of the neural networks under bit-level network and file corruptions, and proposes corruption-agnostic and corruption-aware defense approaches. The Bit-corruption Augmented Training is introduced, which is about applying the data augmentation at a bit level.

The majority of the reviewers are against the acceptance of the paper. R1 gave the rating of "marginally above acceptance threshold", finding the problem interesting but not the proposed solution. The other reviewers gave rejections and a clear reject rating.

The main concern raised by all of the reviewers is regarding the technical novelty of the proposed approach in the paper. While some reviewers appreciate the importance of the problem and the thoroughness of the experiments more than the others, none of the reviewers find the proposed solution novel and interesting.

The AC agrees with the reviewers that the technical contribution of the paper is not significant despite that it focuses on an interesting problem. We do not recommend this paper to ICLR.
Decision: Reject