Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an active learning-base model for scene graph generation, in order to reduce the prohibitively expensive annotation cost.
strength_and_weaknesses: Strength:
(1) Adopting evidential uncertainty and diversity guided deep active learning can alleviate the issue of open-set relationships sounds.
(2) The proposed method can save human annotation costs by approaching the performance of the fully supervised model seems promising.


Weakness:
(1) How to evaluate the robustness of proposed uncertainty method? Please offer more details.
(2) How to handle the long-tail problem with the proposed model? Please examine the effect of the method.
clarity,_quality,_novelty_and_reproducibility: -Clarity: The writing could be improved to make this paper more clear. -Quality: The overall of this paper is good. -Novelty: The novelty is incremental and limited. -Reproducibility: It seems to be reproducible but some details are missing.
summary_of_the_review: Regarding the limited contribution and unclear details of this paper, I recommed to weakly reject the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an evidential uncertainty and diversity guided deep active learning framework for the scene graph generation task. In particular, uncertainty is estimated by coupling evidential deep learning (EDL) and global relationship mining. Also, a context blocking module (CBM) and image blocking module (IBM) are designed to explore the diversity existing with context-level and image-level bias. The experiments on the VG150 dataset seem to validate the effectiveness of the proposed method.
strength_and_weaknesses: #### Strengths
- Overall the paper is well written and easy to follow.
- It is interesting to see the authors apply active learning in the task of scene graph generation, which is not well explored so far.

#### Weaknesses
- Is the proposed method able to handle the label noise? Usually we cannot guarantee that the human annotations are 100% accurate and therefore a solid active learning method should also be able to handle the label noise to some extent.
- Are the orders of the uncertainty based module and the diversity based module exchangeable?
- Is the proposed task-specific active learning method able to outperform the task-agnostic active learning works like: 

[1] Yoo, Donggeun, and In So Kweon. "Learning loss for active learning." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 93-102. 2019.




clarity,_quality,_novelty_and_reproducibility: - Clarity: overall this article is well-organized and easy to follow. 
- Quality and novelty: the paper is solid and it is an application-level novelty to apply the idea of active learning to the scene graph generation task.
- Reproducibility: The authors promise to release  the code.
summary_of_the_review: Based on the above statements, I would like to weakly accept the paper at the initial stage. After carefully reading the authors' response, I am happy to see most of my concerns have been well addressed and therefore would like to keep the initial rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: As a structured representation, the scene graph bridges computer vision and natural language processing, and the scene graph generation task is vital for the CV community. However, The annotation process of the scene graph is very complex including bounding boxes, labels, and relationships, so it is necessary to explore how to ensure the robustness of the scene graph model with few annotations. The authors propose a new method based on active learning, which trains a robust SGG model with fewer data by fully exploiting informative samples in the dataset.
strength_and_weaknesses: The positive aspects of the paper include:

1. Using the active learning method to save the annotation cost is a novel direction in scene graph generation and is important for further SGG study.

2. The designs of the proposed modules, i.e., EDL, RPG, CBM, and IBM, are convincing and fully consider the inherent problems of SGG, such as data bias and sparsity of the dataset.

3. The final performance of the proposed method is good, which approaches the performance of a fully supervised SGG model with only about 10% annotation cost for training the SGG model. The abundant ablation studies and detailed comparisons verified the effectiveness of the proposed modules.

There are some main concerns:

1. As stated in the abstract, the authors aim to reduce the annotation cost. However, the proposed method only reduces the training set for training models, and the method also needs a large labeled dataset to sample an informative training set. Does this approach really save annotation costs throughout data collecting and model training processes?

2. In section 3.1.1, the authors mentioned that “the uncertainty of xi can be estimated via evidential uncertainty ui = k/Si with a maximum value of 1.” Why is the maximum value 1? Is eij greater than or equal to 0? The definition of eij is unclear.

3. The authors should explain the qualitative results in Figure 4 in Section 4.2. What the figure illustrates? 

Some minor issues:

1. Inconsistent symbol in Eq.(1) and following text. “S” (standardized) in Eq.(1) and “S” (italic) in text.

2. Inconsistent information in the caption and the image in Figure 4, “3-rd and 10-th AL rounds” in the caption and “5-th/10-th AL round” in the image.
 
clarity,_quality,_novelty_and_reproducibility: Overall, this article is well-organized and easy to follow. The authors have also provided the code in the supplementary material to ensure the reproducibility of the paper to some extent.
summary_of_the_review: Based on the above considerations, I believe that this paper has some novelty and will have some positive impact on the field. In the rebuttal stage, the authors should solve my main concerns to polish the final version.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method, EDAL, for scene graph generation with active learning. The paper combines an uncertainty-based and diversity-based active learning. The former utilizes evidential deep learning to estimate the relationship uncertainties and the diversity-based sampling is driven by image-level biases. The main idea is to utilize the evidential probability based cross entropy loss as a regularizer for relationship classification. This evidential probability is the second-order probability derived from the parameterized Dirichlet distribution. Additionally, diversity-based on context and image-level bias is incorporated.  
strength_and_weaknesses: The paper aims to develop a hybrid approach utilizing both the uncertainty-based and diversity-based active learning for scene graph generation tasks. The paper utilizes evidential deep learning for its known property of being reliable specially in open-set problems. 
They took inspiration from EDL originally aimed for open-set action recognition and modified it for relationship recognition. Essentially that is utilizing the evidential probability based cross entropy loss as a regularizer. This evidential probability is the second-order probability derived from the parameterized Dirichlet distribution. The hypothesis is that the cross-entropy loss based on evidential-probability prevents the overconfident predictions in open-set relationships without affecting the uncertainty estimates. This is an intended behavior specially because of sparse relationship triplet labels present in a scene. The experimental designs including baselines corresponding different sampling strategies are convincing. Fig 3 depicting mean Recall values with varying labeling budget is a valuable addition in this paper. It also shows that the improvement of mR values don't come at the cost of Recall values. 

From the visualizations, it also looks like the method generates more fine-grained (in the case of VG, they are tail classes) relations : e.g. On vs sitting on in Fig 4. I am curious about how the relative performance trends are for a labeling budget over 10% such as 20%-25%.  
clarity,_quality,_novelty_and_reproducibility: The figure captions need to be more descriptive, such as Fig 2. 
Minor things: Fig 4 title and subtitle aren't consistent. Not clear if the visualization outputs are from 3-rd or 5-th AL round of AL in the top-right section for each. 

The main contribution is to aptly applying the EDL for uncertainty based AL and combining with diversity-based AL for SGG. 

They have provided the code based on masked RCNN repo to substantiate reproducibility. Although I didn't try to run the code, but the structure of the code appears to be in accordance with the main contributions. 

summary_of_the_review: The paper develops an hybrid active learning framework for learning SGG. The hybrid method utilizes both the diversity-based and EDL for uncertainty-based AL. One of the major contributions is modifying the regularization for the relationship recognition task using EDL. The results show their method archives higher mean Recall for PredCls, SGCls and SGDet, without compromising the performance of head classes as measured by Recall metrics within a relationship labeling budget of 10%.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents an evidential uncertainty and diversity guided deep active learning framework for scene graph generation. The key idea of this work is to utilize the evidential probability based cross entropy loss as a regularizer for relationship classification. Experimental results on the VG150 dataset are provided and discussed. 

Overall, this paper is well organized and clearly written. Using active learning for scene graph generation is an interesting direction. The motivation of the proposed method is clear, and experiments demonstrate its effectiveness. 

Meanwhile, reviewers raised some concerns such as technical details, experimental settings, and analysis of the qualitative analysis. The authors provided detailed responses and additional experimental results, which can well address most of the concerns from reviewers. The authors are strongly encouraged to incorporate the suggestions from reviewers as well as the new results to the final version.
justification_for_why_not_higher_score: Considering the overall contribution of this work and the quality of this paper, this paper is marginally above the acceptance threshold. 
justification_for_why_not_lower_score: This paper studies how to leverage active learning for scene graph generation, which would inspire researchers in this field.
Decision: Accept: poster