Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In general, this paper addresses an important problem in shape segmentation: the irregular cases. Previously I have tried to push many papers to provide results on irregular input shapes, but most of them never took my suggestions seriously, and I get bored of seeing chairs, tables, and airplanes. So I'm pretty happy that finally this problem is being researched. 

Overall I like this submission, but I do have concerns as listed below. So I decide to weak accept. 



strengths: Strength of this paper:
1. The proposed dataset looks good, and can be used as an additional benchmark for all future instance segmentation methods. 
2. The authors tested multiple benchmarks on their dataset, and showed that existing methods do not work well. This verifies my concern on previous papers too.
3. The proposed ASIS framework uses a more effective affinity kernel approach and it improves the performance quite drastically.
weaknesses: Concerns that I hope authors can help me here:

(1) For the dataset:

1. Most of the data (or all of them) are string- or bar- like objects. I know the authors trying to convey the message that this is irregular and challenging, but I'm not sure if this covers the definition of 'irregular shapes'. For example, you can have irregular surfaces like leafs with different shapes and appearances, and so on. I would suggest tuning the topic and title to a more specific one, if the dataset cannot cover all major irregular cases. 
2. It looks to me most of the data are synthetic, maybe rendered by graphics. Do you think this may be a problem when transferring the trained model like ASIS on iShape to real world data?

(2) For the ASIS:

3. The affinity kernel comparison in Figure 4 is confusing me. Based on my understanding the affinity modeling capability is affected by both the kernel shape and resolution. And 4(a) ASIS seems to be more fine-grained than 4(b) GMIS. Is this fair? Also, the ASIS kernel is semicircle since authors believe the other half is redundant. But do you really guarantee that the missed connections can be recovered when the kernel is moving to another location especially when you only have a sparse kernel? And the kernel parameters r_k and g must be pre-defined before training (not learnable)?
4. Can the sparse graph contain loops or not? Details are quite missing on the graph part. 
4. Paper structure: the ASIS section spends quite many texts on the loss. But I don't think that's the correct focus. I would suggest adding more description on how you build the sparse graph, and how you do the graph merge. In addition, I suggest adding two or three sentences (or one formula) introducing the affinity kernels for general readers. 

(3) For experiments:

5. Maybe I missed this detail, but did you re-train all the previous methods on your iShape data? This needs to be explicitly stated otherwise it's unfair. Also, could you compare the cost such as memory (or #params) and time? This can help readers rule out the gain from the increased network capacity.
6. Since the authors were focusing on affinity kernels, I would like to see some kernel or graph visualizations in the results compared to other affinity kernels. But unfortunately there is not. This is critical because it verifies the design.
correctness: Most of the claims in the paper are correct and sound. Some claims such as 'irregular shapes' may need further consideration as mentioned in the weakness. And in the experimental section, authors need to explicitly state how they re-train the baselines and make sure the comparison was completed on similar total network capacity. 
clarity: The paper is well written in most parts. The ASIS (section 4) needs some adjustment, like describing more on graph construction and merging, and maybe give a short introduction on affinity kernels. 
relation to prior work: Yes I believe so. Studying shape segmentation on irregular shapes like this is well explained and differs from previous works clearly.
additional feedback: Please check the weakness comments for details.

[Post rebuttal comments]
I'm ok to accept this now, but please fix the comments I added in my response.
review_title: Nice paper and dataset but with some concerns

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper proposes a mostly synthetic image dataset for instance segmentation of irregularly-shaped objects. The authors also propose a strong baseline method for irregular shape segmentation and reason why popular segmentation methods fail. 
strengths: The paper is well written and easy to follow. The key statistics of the dataset are well explained. The addition of a strong baseline model provides a good starting point for future research.

The reasoning of why common segmentation methods such as Mask RCNN fail in the proposed tasks is convincing.

weaknesses: Major:

1. A formal definition of “irregular shape” is not explicitly given. While the paper describes a range of statistics for several popular datasets and points out the differences, it does not provide a clear definition of shape irregularity. One could argue that the logs in the logs dataset are regularly shaped given their stereotypical color and rectangular shape. In contrast, many classes in the iSAID such as harbors have irregular shapes.

2. The motivation of an image dataset with only irregularly shaped objects is not well explained. It is unclear why having a dataset with only a few classes of seemingly arbitrarily chosen classes of irregularly shaped objects is helpful. The Antenna class and Log class seem to have similar long rod-like shapes, it seems redundant to have these two. Also the objects seem to have more or less homogeneous colors. One could provide concrete examples of potential real-world use cases to justify the usefulness of such a task. 

3. In many datasets such as COCO and iSAID, multiple classes are often present in the same image. In contrast, the proposed dataset limits the number of classes in an image to one, which is unrealistic in most real-world vision tasks. The authors fail to justify the decision of not allowing more than one class in an image.

4. 5 of 6 sub datasets are synthetic, however this is not highlighted as there is only one mention of “synthetic” in the entire paper. Even among synthetic images, this is visually simplistic compared with some of the more sophisticated synthetic datasets such as GTA dataset (https://arxiv.org/pdf/1608.02192.pdf). It is well known that there is usually a generalization gap between synthetic and real images. As a result, it is unclear how much of what we learned with this dataset will generalize to real images.

Minor:

5. Standard deviation is missing from results.

6. GitHub missing instructions and code to reproduce results.

correctness: As discussed above, several choices of the construction of the dataset are questionable.

The evaluation methods are mostly appropriate.

clarity: The paper is overall well written and easy to follow.
relation to prior work: The paper has detailed discussions of prior work. 
additional feedback: Since the authors claim that the proposed dataset is “a complement to existing datasets”, it seems reasonable to clarify whether the proposed ASIS method only works for the proposed dataset or it also generalizes to common segmentation datasets like COCO.
review_title: Interesting dataset, not well motivated 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper presents iShape, a benchmark of 2D instance segmentation for irregular shaped objects (e.g., antenna, branch, fence) and benchmarks existing algorithms on the new data. The authors also propose ASIS, their own instance segmentation algorithm. Segmenting instances of irregular shapes is an important problem in many applications of machine learning and computer vision, such as medical imaging, astronomy, etc. Thus, releasing the benchmark would spur additional interest in development of novel methods for these fields. 

strengths: The paper proposed a novel dataset for evaluating instance segmentation algorithms, and focuses on six challenging aspects of irregular objects. The dataset is released, well-documented, and easy to access. 
An evaluation of several existing algorithms on the new dataset is presented, and a novel instance segmentation method is proposed.

weaknesses: I found the paper easy to read and understand, however, it could be organized better. For example, the details of the data creation are currently in supplementary material, and some of them need to be in the main paper. It appears that part of the data is generated synthetically (using Blender), which part is collected from real scenes. This process, as well as main details of data generation need to be added to the main manuscript. Also, given that there is both synthetic and real data, how does the source of data affect performance of existing techniques? How realistic is the synthetic data?

Also, the authors propose a novel instance segmentation algorithm. How is is similar and different from existing works? Based on what properties are the other baseline methods selected for evaluation? In Table 2, the proposed method outperforms on all data subsets, except for branch. Why does this happen?

correctness: It appears that the data is constructed synthetically using blender and collected from other datasets. While the details of the synthetic data are outlined in supplementary material, the details of manually annotated data subsets ( supplementary material, 3.1) are missing. Who performed these annotations, and how was this process organized?

clarity: In general, the paper is well, written. However, important details are missing. Please see list below:

- What does ASIS, the name of the proposed method, stand for?
- Why is aspect ratio an important object property that needs to be discussed? Is it not sensitive to image rotation, whereas modern algorithms often use random rotations as a form of data augmentation. 
- Table 1: for the proposed dataset, please also include summary statistics (e.g., average), to be more comparable to other datasets. Please also add reference to other datasets’ papers
- Figure 3 is hard to read, please increase font size
- Table 2: why is evaluation performed only using ResNet-50 backbone? Will results change with a different backbone?
- Figure 6: can you please add zoom ins, or otherwise highlight the main differences of the methods in the figure? The differences are described in text, but difficult to extract from the figure.
- In the data release, when I try to access instance maps in shape/ishape_dataset/antenna/train/instance_map/***.png, I see black images only.  Could you please provide some source code to explain how to load the instance maps and/or add visualizations of the instance-segmented images into the data repository?

Typos:
- Line 151: “segemtation” -> segmentation
- Line 153: “As is shown in the table” -> add link to table
- Line 170: “stage, We apply” -> stage, we apply
- Line 233: “lost the view of thin objects” -> bad grammar
- Line 245: “SOLO V2” -> which method is that? Please add a reference.
- Line 248: “PolarMask” -> which method is that? Please add a reference.
- Line 249: “That’s because” -> That is because
relation to prior work: Related work is well-written, but please explain what are the current state-of-the-art methods.

additional feedback: None
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes a new benchmark for tackling irregular objects in instance segmentation.

Prior arts fail to achieve promising performance on the proposed benchmark, and the authors propose a new method called ASIS that leverages the semantic segmentation and affinity predictions for yielding the final accurate instance segmentation results.

strengths: + Good paper writing and formatting.

+ The proposed dataset is interesting. It focuses on the real scenarios in industrial applications where prior state-of-the-art methods are shown to fall short.

+ The rationale behind the method design is clearly stated and the proposed components have been empirically shown effective with adequate analysis and explanations.

weaknesses: 1. The proposed framework leverages the affinity groudtruth for training guidance while the other methods do not. The author can try to apply the affinity loss with OHEM strategy to other state-of-the-art methods to show the effectiveness.

2. The model seems to be a general framework for instance segmentation. The author could apply the method to other popular datasets for instance segmentation to have a comprehensive comparison and show its generalization ability.

correctness: Yes.
clarity: Yes, the paper is with good formats and contents. 
relation to prior work: Yes.
additional feedback: [Post Rebuttal] The authors' rebuttal does not address some of my concerns.
OHEM is a popular trick that could bring considerable improvements, which has been widely verified in the vision community, and OHEM is not one of the contributions in this paper. Only applying OHEM on the proposed method causes an unfair comparison with the others. Also, it is not proper to use the default configurations to training other methods in your setting because even the same algorithm performs very differently on different benchmarks. Directly applying the default setting to another dataset usually leads to inferior performance, according to the reviewer's experience. The unfair comparison does not tell the superiority of the method and the necessity of the benchmark.

Considering other issues raised by Reviewer#2, the reviewer believes that this dataset is somewhat unrealistic since there are several inherent constraints listed by Reviewer#2. Also, as claimed by the authors, the dataset is a complement to the existing dataset. It would be better to verify the proposed method on other popular benchmarks.
review_title: Review of Irregular Shape Instance Segmentation

Meta-review: This paper received conflicting reviews. AC and PC discussed the paper. We agree that the proposed dataset is filling in an interesting gap in instance segmentation study. However, the model study is not comprehensive enough yet to convince all the reviewers the problem can not be solved by the existing methods. We suggest the authors spend some more time polishing the experiments and paper for later submissions.
Decision: Reject