Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a computer vision pipeline for computing semantic regions in a scene and predicting where the car will go in terms of those regions, in the context of autonomous vehicles. The method is evaluated using publicly available datasets and compared to other methods for the same or similar problems. 

strengths_and_weaknesses: 
As this is a computer vision paper, it is difficult for me to judge particular strengths. 

The main weakness with respect to the venue is that this is a computer vision paper being submitted to a robotics conference. 

In addition, the authors need to motivate better why predicting the future in terms of discrete semantic regions is better than just predicting a continuous trajectory, which is what many other works do. 


robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: 
My main justification is that this paper belong in a computer vision venue or at a venue that solely focuses on autonomous driving. 
issues: 
There are a number of clarity issues in the paper.

The sentence "While promising results are demonstrated, the learned representations are effective to the trained task" is not clear, do you mean to say ineffective? Or something else?

The term "semantic region" is defined very narrowly and hence, it would be better to use a different, more specific term that relates to a region that the car will be in the future. 

Also, is there a particular reason why map information is not used as an additional input to the model?

Some references are messed up, for example [3]
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies driver intent prediction and risk object identification from egocentric images/videos, which are crucial tasks for autonomous driving. This paper investigate a major problem: what's the ideal representation for these tasks? This paper proposes a novel scene representation, which is called semantic region. Such representation considers both higher-level semantic and geometric representations of roads. To learn this representation, this paper also introduces a novel semantic region prediction task. On the risk object identification task, this new representation achieves 6% improvements over existing best-performing methods. This paper studies an interesting problem for autonomous driving and proposes a novel scene representation. This novel representation can be potentially used for other tasks in autonomous driving and robotics. 
strengths_and_weaknesses: Overall, I am in favor of this paper because it proposes a novel representation for several important tasks in autonomous driving. Also, this paper shows significant improvements with this new representation on both driver intent prediction and risk object identification tasks. However, I do have several questions regarding this semantic region representation and some experiment details. I list strengths and weaknesses as follows. 

Strengths:

1. This paper selects two important autonomous driving tasks to study. For these two tasks, this paper proposes a novel scene representation -- semantic region, which considers both higher-level semantics and geometrics of roads. Such representation is technically sound and will potentially benefit more tasks in autonomous driving/robotics. 

2. This new representation is indeed effective. On the risk object identification task, this paper improves over best-performing methods by 6%. Also, on the driver intent prediction task, this new representation also exhibits very strong performance. 

Weaknesses:

1. My major concern is the fairness of experiments. In data collection and annotation, this method uses a ResNet50 pre-trained on Mapillary Vistas dataset. I am worried that this actually introduces additional supervision/information for downstream applications studied in this paper. 

2. I also find this paper is not easy to follow. This paper talks about semantic region as a new scene representation, however, it doesn't present what semantic region is in details. I would encourage authors to add more details of semantic region as it is the key idea of this paper. 

3. In addition, the architecture shown in Figure 4 doesn't make sense to me. What are the roles of these two SR classifiers? Why are two classifiers are needed here? 

4. In Section 3.2 (supplementary materials), pre-training on the in-domain data (Mapillary Vistas) is very important. Why is this method pre-trained on Mapillary Vistas instead of more popular autonomous driving datasets such as nuScenes/Waymo Open Dataset? 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think this paper presents interesting ideas and shows significant improvements with the new representation proposed in this paper, although there are certain weaknesses/problems with current draft. That being said, I am leaning towards weak acceptance if my concerns are addressed in the next revision. 
issues: 1. Since pre-training is very important as shown in multiple experiments, can authors provide more discussions of whether the comparisons to existing methods are fair? 

2.  I encourage authors to add more details of semantic region since it's the major idea of this paper. 

3. Can authors provide more ablation of pre-training on different datasets (nuScenes & Waymo Open Dataset)?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a new method for predicting driver intent and risk objects from egocentric driving videos. Their approach relies on semantic region prediction based on road topologies. This approach improves performance on existing datasets.
strengths_and_weaknesses: Strengths:

The problem is important and valuable.

The authors approach is interesting and makes sense.

The paper is easy to follow and understand.

The results show improvement in real world scenarios.

The authors use an automatic labelling scheme that may be of interest to the community.

Weaknesses:

The method only works with certain fixed road topologies, and does not handle unusual intersections and other edge cases that come up in the real world. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: If the impact section had a 3.5: "possibly substantial impact in a paricular subfield" I would give this paper that. The results are good, the approach is interesting, if not revolutionary. Most importantly though, the content seems valuable to others working in this area.
issues: Long list of citations on line 20 contains duplicates.

In fig 1. The orange path is A1, B1, C1, T1, while the blue is A2, B2, B3, T2 and the green is A3, C2, C3, T3. So one is ABCT, the other two are ABBT and ACCT.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper designs a novel scene-level represent learning framework by predicting the current and future semantic region. The authors utilize the hidden state in the decoding process for two important downstream tasks: driver intent prediction and risk object identification. The experimental results demonstrates the effectiveness of the framework.
strengths_and_weaknesses: Strengths:
1.The paper designs a novel scene-level represent learning framework a by predicting the current and future semantic region.
2.The authors design an automatic algorithm to label the semantic regions.
3.The proposed framework could handle different types of road topology.
4.The representation of the road could improve the precision on downstream tasks. 

Weaknesses:
1.The framework separately study the representation of the intersection and non-intersection road topology, which leads to higher cost and parameter quantity in complicated scenario where exists many different types of road topology.
2.The quality of the representation is only verified on two simple downstream tasks, the authors could test it on some driving simulator or real world scenario.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper designs a novel scene-level represent learning framework by predicting the current and future semantic region. The authors design an automatic algorithm to label the semantic regions and train the represent learning framework. The proposed framework only takes the egocentric frames as input, and could handle different types of road topology. The performance improvement of the two downstream tasks (driver intent prediction and risk object identification) demonstrates the quality of the scene-level representation.  However, there are some minor defects of the paper: The framework separately study the representation of the intersection and non-intersection road topology, which leads to higher cost and parameter quantity in complicated scenario where exists many different types of road topology; The quality of the representation is only verified on two simple downstream tasks, the authors could test it on some driving simulator or real world scenario.
issues: The definition of the semantic region should be described clearer.
review_title: null

Meta-review: metareview: Reviewers like the novel formulation of the method, but are concerned about the fairness of the experiments, the generalization to unseen topologies, and the presentation.  One reviewer also argued that the topic is not closely relevant to robotics, which is required by CoRL.  Please consider addressing these points if possible. 

The authors did a good job during rebuttal, after which three of the four reviewers were in favor of acceptance.  The main concern of the negative reviewer was the submission's alignment with CoRL.  The AC agrees that this is a reasonable concern, but also well addressed by the authors and other reviewers.  In this case, the AC is recommending acceptance.  The authors should revise the submission accordingly based on the reviews.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)