Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors explored the robustness of video machine learning models to bit-level corruption. They investigated previous methods such as Out-Of-Distribution (OOD) detection and adversarial training and found that they are not effective enough to defense against the bit-level corruption.  Accordingly, this paper proposed a new framework, Bit-corruption Augmented Training (BAT), which utilizes the knowledge about corruption by bit-level data augmentation at the training stage. Also, the authors argue that the proposed method outperforms the previous methods in handling the bit-level corrupted dataset.

While the proposed method seems simple and more effective than the previous studies, the authors do not currently provide a sufficient amount of evidence to support their claim.

Pros
-	The proposed defense technique is effective on the bit-level corruption of videos. Also, it is simple to apply in real-world deployment.
-	This is the first work that addresses the robustness against bit-level corruption of videos.

Cons
-	The technical novelty of this paper is not significant. They proposed a new framework for robustness to video bit-level corruption. However, data augmentation method is the main technical contribution this paper proposes. I think adopting existing technique to bit-level corruption problem is not significantly novel.
-	There is insufficient evidence that the proposed method is better than previous studies. I think additional experiments on OOD detection are needed. The authors implemented only one method as a baseline, ODIN [1]. Since there exist many other state-of-the-art methods than ODIN, such as Mahalanobis [2] and Outlier exposure [3], they should also be compared. In addition, there is no explanation of an input preprocessing method proposed in ODIN. The authors need to articulate why they omit the preprocessing. Also, additional experiments on UCF101 by using corruption-agnostic and corruption-aware defenses would make the paper more convincing.
-	Furthermore, in Section 4.3, the rationale behind the importance of detecting low-level corrupted samples by OOD is not elaborated. As the authors mentioned, largely corrupted videos are likely to be misclassified, while the accuracy for videos with low-level corruption does not decrease.
-	For clarity, I recommend the authors to correct the minor typos in the paper. The adversarial training in Section 4.3 seems to be worse compared to the no-defense baseline by 8.6 points (not 8.1 points) on clean data. 

[1] Liang et al., "Enhancing the reliability of out-of-distribution image detection in neural networks.", ICLR'18

[2] Lee et al., "A simple unified framework for detecting out-of-distribution samples and adversarial attacks.", NIPS'18

[3] Hendrycks et al., "Deep anomaly detection with outlier exposure.", ICLR'19

----------------------------------
After rebuttal:

I appreciate the authors for thoughtful response and additional experimental results, which are helpful for further understanding of the manuscript. Especially, the additional experiment on the recent OOD detection method addresses my concern about the evidence that the previous OOD studies are not sufficient for defending the bit-level corruption.

Unfortunately, I am still not sure about the technical novelty of this paper. I agree that the paper proposed a new problem setting, but I do not think that the technical novelty is significant, given the proposed approach of just applying the data augmentation simply at a bit level, rather than at a pixel level.

Due to this concern, I want to keep my rating of "4. Ok but not good enough - rejection" as it is.

review_title: The topic this paper tackled may be important for the real-world application, and the method presented in the paper might be effective. However, the paper lacks experiments and evidence to properly support the authors’ claim. Also, the technical novelty is not significant. Therefore, I chose "rejection" as an evaluation of this paper.  If all the issues below are fully addressed, I may reconsider my assessment of this paper.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This work investigates the problem of building robust video prediction models in the presence of signal corruption. The problem itself is not widely studied and experimental work like this one certainly opens some possibilities. The solution on the other hand is surprisingly simple and easy to implement. It serves the purpose of introducing the problem to a wider audience, and shed some light in different types of remedies. 

Clarification:

1. What is the importance of solving this type of problem in the general video application setting? One would assume there are existing methods in signal processing without machine learning to tackle the issues well, especially when knowing the type of the noise and their corruption ratio in range. One would hope a section of related work on it and discuss the further reliance of a data driven approach.
2. The "No defense" baseline at some corruption levels are much better in accuracy. I'm not sure I understand the explanation given in Table 1. 
3. What models are used to perform action recognition and tracking? Does the performance of BAT have anything to do with choice of models?

In  all, the problem itself seems worth a pursuit if stated with more context. The concern is that the solution is fairly specialized on the known noise types and their general range. One would argue that adding input noise has already been used in numerous previous work to improve the robustness of models.  The major issue, which is completely ignored, is that adding input noise could also hurt the model performance on clean inputs (assuming not all videos are corrupted). This is the aspect that is worth a discussion, a trade off that needs some illustration.
review_title: an unfamiliar problem with familiar solutions 

final_score: rating: 3: Clear rejection
review_content: **Summary:**

The authors evaluate the effect of bit-level corruption, including network packet losses and bit corruptions, on video models such as action recognition and multi-object tracking. They found that the model performances drop significantly under severe corruption levels. To overcome this issue, they propose a defense method named Bit-corruption Augmented Training (BAT) to enhance the robustness of the model by embedding corrupted video samples in the training process. Results show that BAT is able to improve the model robustness over other methods such as Out-Of-Distribution (OOD) detection and Adversarial Training (AT). 

**Strength:**

- Explores an interesting and new problem space for video model robustness.
- A range of realistic corruption rates was evaluated.

**Weakness:**
- The threat model for the defense is unclear.
- No comparison with naive fault tolerance methods at the network and video file level.
- The proposed BAT method is trivial and the performance is not good.
- Packet loss should not be categorized into data corruption.

**Detailed comments:**

This paper explores a topic that is both realistic and interesting for the increasingly popular video model applications. I appreciate that the authors evaluated their approach under realistic packet loss and bit-corruption rates that would occur in the physical world. Yet the paper has several points which I think significantly weaken its value. I will detail them as follows.

- The authors claim the proposed BAT method as well as OOD detection and AT as *defenses*. However, I can hardly find any evidence that the authors evaluated these methods under an adversarial setting, i.e., the paper does not assume the existence of an attacker. It seems to me that this paper is more on the side of fault tolerance than defenses for attacks.
- For detecting or correcting network packet losses and bit-corruptions in the files, a straightforward solution is applying existing network and memory/file-level fault-tolerance methods. For example, checksum or error-correction codes are two easily applicable solutions. The benefit of applying these methods is that they are model-agnostic and would not affect model performance under normal conditions.
- The BAT method proposed by the authors seems rather trivial to me. How is this different from data augmentation? In addition, the reported performance of BAT is not good. For example, would a 27.3% accuracy under 10^-4 random file corruption bring enough utility for the model?
- Lastly, I do not think it is a good idea to categorize packet losses as bit-level corruption since data loss is fundamentally different from data corruption. In other words, data corruption considers cases when data got changed but data loss only considers when data is missing.
review_title: The paper needs to clearly define the threat model and compare with fault-tolerance methods

final_score: rating: 4: Ok but not good enough - rejection
review_content: Post-rebuttal: The rebuttal partly addresses my concerns, so I would like to change my score to 4.

This paper simulates network and file corruptions at multiple corruption levels, and explore corruption-agnostic and corruption-aware defenses. The presented Bit-corruption Augmented Training enhances the robustness of the video machine learning models. Experimental results show the effectiveness.
 
Pros: 
The experiments are very extensive. The effectiveness of the presented method compared with the corruption-agnostic is clearly validated.

Cons:
1.      The novelty is limited. Considering the attack in the encoding and decoding process have been studied, and sophisticated methods [1] have been proposed. Compared with [1], I wonder about the advantage of the proposed method.
2.      The experimental results are not convincing. Baselines such as [1] is not included in the experimental results for fair comparison. I think the baselines in the manuscript is too weak.
3.      I think the idea is too ad-hoc, which just generates adversarial samples in the space of encoder and decoder instead of the space of the classifier. I think the difference is not significant.
 
[1] Rakin, A. S., He, Z., & Fan, D. (2019). Bit-flip attack: Crushing neural network with progressive bit search. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1211-1220).
review_title: Extensive experiments but limited novelty

Meta-review: This paper investigates robustness of the neural networks under bit-level network and file corruptions, and proposes corruption-agnostic and corruption-aware defense approaches. The Bit-corruption Augmented Training is introduced, which is about applying the data augmentation at a bit level.

The majority of the reviewers are against the acceptance of the paper. R1 gave the rating of "marginally above acceptance threshold", finding the problem interesting but not the proposed solution. The other reviewers gave rejections and a clear reject rating.

The main concern raised by all of the reviewers is regarding the technical novelty of the proposed approach in the paper. While some reviewers appreciate the importance of the problem and the thoroughness of the experiments more than the others, none of the reviewers find the proposed solution novel and interesting.

The AC agrees with the reviewers that the technical contribution of the paper is not significant despite that it focuses on an interesting problem. We do not recommend this paper to ICLR.
Decision: Reject