Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a context-agnostic learning approach that combines an object area and a context image (s.t. background image) to generate input synthetic image and train the model as context-independent. The proposed method is made more efficient by including this generation process in the training loop, compared to a exhaustive sampling method that randomly selects from a set of combinations of object areas and context images. When applying this method on the task of traffic sign recognition, character recognition, it provides enhanced performance in a setting where model is trained using synthetic data and evaluated on real-world dataset.

My concern is the presentation.
"Context" and "context-agnostic" are properly defined. However, it is very difficult to understand how to achieve "context-agnostic" in algorithm 1 and 2. In my understanding, context bias is corrected by forcing a model trained using the opposite (or other) object class in the same context in the next training iteration. If it is correct, how can the bias be corrected just by training with other objects? Does it affect to reduce ||B(h,c)|| in Definition 3.3? Please clarify it.

In addition, the effectiveness of the proposed method can be verified with such simple tasks, but it is advisable to add a discussion about more complex tasks such as object detection that requires more diverse contexts. This will be more interesting for many.

I can recognize some technical contributions that can appeal to many ICLR researchers.

As a minor correction,
In Section 3.2, the object error o^ is defined but not used anywhere.

----------------------------------------------------------------------------------------------------------
I have read the revised manuscript and the comments of all reviewers. For concerns that the experiments are not sufficient to validate the proposed method, I am leaning to the author's rebuttals that the datasets have been chosen to meet the heretical assumptions of context-agnostic learning. The given datasets seem to be sufficient to demonstrate the effectiveness of the proposed learning method.

Therefore, I will not change the rating.

review_title: How to achieve "context-agnostic" characteristic in algorithm 1 and 2?

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper defines the task of context-agnostic learning and proposes an algorithm to solve the problem while assuming the ability to sample objects and contexts independently. They propose to decompose factors contributing to the risk into two, context bias and object error. Based on this interpretation, an algorithm is designed to 'greedily correct bias' while employing adversarial training (or robustness training) for 'local refinement'. The method achieves high accuracy on two synthetic visual tasks, digits and traffic sign classification, when a model is trained using one sample per class from the source domain and tested on an unseen target domain.

+) Theorem 3.1 provides a new view on risk. Risk is decomposed into two factors, context bias and object error. I think this gives new insight to consider the effect of context bias and object modeling on risk separately.

+) The experimental results are impressive because the model is trained using a very limited number of samples (one sample per class from the source domain) but showed high generalization performance on an unseen domain. The proposed method achieves promising performance on two synthetic classification tasks compared to other existing methods for few-shot learning and domain adaptation, which requires more labeled or unlabeled data during training.

-) Their underlying assumption for greedy bias correction is that a classifier learns a strong bias on recent training inputs when taken as contexts. However, if stochastic gradient descent is used for optimization, I think it is unlikely because the model changes continuously. Therefore, it is uncertain how effective this greedy selection strategy can sample contexts with large bias.

-) Relating to the above point, I also have a concern about the experimental validation. In all the experiments, gamma is defined as a function that takes object and context images and outputs their overlap. It is not guaranteed that the proposed heuristic sampling strategy generalizes to other gamma functions.

-) Also, all the experiments are performed for a relatively small number of classes (up to 50), and synthetic images are small iconic images with objects in the center. Although the method shows promising results under this specific setting, it is hard to conclude that the proposed heuristics will generalize other settings, such as when there are more classes, image resolutions are higher, and objects have a larger variation in their appearance. I think evaluation on additional datasets with different characteristics (such as CIFAR-100, Caltech-256, CUB-200) would be necessary.

-) The assumption that one can sample objects and contexts independently may restrict its application.

Though I found this paper proposes an interesting view on risk, I would recommend 'reject' due to concerns stated above.

---
Thanks for the detailed response. I can agree that the observation function gamma in the form of addition can model many noisy signals. However, the argument in the paper that the proposed method works for an arbitrary gamma still lacks experimental validation. So I would like to keep my recommendation. My other concerns have been addressed.

review_title: Interesting paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a theoretical setting for context-agnostic few-shot learning. While I do acknowledge that the background signals should not be involved or impact the predictions, I am concerned why this should have a new setting specific for this case? Can the authors elaborate more on the motivation for this theoretical setting and how this setting can benefit the application in the real world? To me, I appreciate that the synthetic data can provide more supervision for few-shot image classification tasks, while I don't understand why this should set up a new setting here.

In the experiments, the authors show the comparison to previous methods in Appendix Table 2, 3 , 4, and the proposed Context Agnostic method seems not outperforms previous baselines and sometimes even perform much worse than baselines. For example, 7% lower than GCR on Omniglot. Can the authors explain the  benefits of the proposed model please? Also, I notice the authors mainly use two simple types datasets, traffic signs and hand-written characters, while I am expecting to see the performance on more realistic datasets, such as ImageNet. Can the authors explain why they choose these two types of datasets? Are they easier to separate the context / background ?

As this paper seems more related to synthetic data, I am curious how synthetic data-based few-shot learning methods perform, such as [a,b]

[a]Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Abhishek Kumar,
Rogerio Feris, Raja Giryes, and Alex Bronstein. Delta-encoder: an effective sample synthesis
method for few-shot object recognition. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman,
N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31,
pp. 2845–2855. 2018.

[b]Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot
learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),
October 2019.

After rebuttal:
Thanks very much for the detailed response! I do agree with the AC's comment that *I don't see a drawback the use of only one sample and no info about the target, rather I'd like to know if the proposed method could use more than one sample in order to make the comparison with SoA methods fair and still be competitive or outperforming them.* Also,  I would recommend the authors to have more experimental validation and resubmission. Thus, I keep my score.



review_title: More comparison in experiments and motivation for this theoretical setting

final_score: rating: 6: Marginally above acceptance threshold
review_content: Strengths:

1. This paper studies a theoretical setting for learning models whose predictions are independent of background signals. There are indeed some practical applications of this setting.

2. The proposed approach is very simple, yet effective. This method is able to learn a context-agnostic model by minimizing a formally defined notion of context bias. 

3. On the theorical side, the authors try to explain their goal of classification as learning to extract reliable signals.

4. The paper is well written and easy to read. Also, the figures in supplementary materials help understand the paper, especially the context learning part.

Weaknesses/concerns:

1. The proposed method has a strict restriction: The training sets must have a single synthetic image for each object class with no additional information about the target domain. This may limit the training of the model in some hard conditions.

2. The quality of the paper will be upgraded if the authors further investigate more data augmentation or style transfer methods in related work and ablation study. Since the context and object concepts are similar with the style and content in some GANs and transferring works.

3. The experiments seem a little bit weak. The experiments are only conducted on two simple datasets, i.e. GTSRB and MINST.

4. The comparison with existing methods is insufficient. There is no representative baseline comparison. Though this setting is novel, we can conduct task by slighting changing existing methods.

review_title: The paper proposes an interesting setting for learning, which relies on the ability to sample from the object and context spaces independently. This paper introduces a novel idea showing promising results in several benchmarks. The authors give a theoretical analysis for their method which is convincing. Extensive empirical results show that the method produces good results both in domain adaptation and few-shot learning settings.

Meta-review: The paper addresses the task of context-agnostic learning and presents an algorithm to solve the problem while assuming the ability to sample objects and contexts independently. It is reported a theoretical ground proposing to decompose factors contributing to the classification risk in context bias and object error. The method makes use of only one synthetic sample in training, still being able to generalize well.

The paper received contrasting reviews, 2 positive (7 and 6) and 2 below threshold (5 and 5). R2, R3 and R4 raised similar issues, especially regarding the experimental validation, which is the main shortcoming of the work: addressing "simple" datasets only, no comprehensive comparative analysis only in relation to baselines (vanilla SGD) but not in relation to state-of-the-art methods, possibly slightly revised to accomplish the experimental protocol proposed in this work (authors claim that the originality of the work do not allow a proper comparison with SoA method as-is). Indeed, I deem R3's rating (6) a bit overestimated given the provided comments. 
AC does not see an issue the use of only one sample and no info about target, rather, it'd be interesting to know if the proposed method could use more than one sample in order to make the comparison with SoA methods fair, while assessing performance in comparative terms with SoA, to give value to the method also in relation to performance. 

Unfortunately the rebuttal did not lead an increase of the ratings, nor to better comments. 
After the rebuttal, R2 and R4 still remained below threshold; R1 was also not changing idea, remaining positive, and R3 did not react after rebuttal.

Overall, the AC deems this paper containing interesting contributions, but it is not sufficiently ready to be accepted at ICLR mainly because the experiental validation is not showing a fully convincing evaluation of the proposed approach (see above). 

Decision: Reject