Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: =========
Summary:

This paper proposes a homogeneous network structure for semantic segmentation, which optimizes for prediction accuracy, latency as well as memory footprint.  The paper studies anytime prediction setting and designs a re-usable single building block to reduce the memory footprint. Experimental results on CamVid data shows that it's possible to use a homogeneous network architecture to achieve competitive mIoU compared to previous work at the cost of increased MACs.  Experimental evaluation on larger datasets such as Cityscapes is prohibited due to memory constraints of the available GPUs. 

=========
Pros:
*  The idea of using a homogeneous network is interesting and re-usable blocks are reasonable for the anytime prediction setting. 
*  The paper is easy to follow. 

=========
Concerns:

1.  Is the parameters of the re-usable building block updated iteratively during inference? The paper described two scenarios where the parameters of the re-usable building block can be updated or shared across multiple iterations.  My first question is,  is this for training time only or for both training and inference? 

 a) At inference time if the parameter updates are allowed at each iteration, what is the mechanism to decide the new weights?  If an independent block was used and only the structure of the building block is reused, there would not be any benefit for latency or memory footprint. 
 
 b) If the parameters of the building blocks are fixed during inference,  then it falls into the weight-sharing scenario where the model performance is much lower according to Figure 1 c). 

2. Lack of anytime prediction experiments.  The paper claims it studies anytime prediction setting, however, I couldn't find explicit descriptions about it in the experiment section and descriptions about how the model choose the number of iterations during inference.  

3. Missing Cityscapes experiments.  The paper claims in the abstract that the method was evaluated on cityscapes but in the discussion section, it says they couldn't provide cityscapes experiments due to memory issues.  The inconsistency in the paper should be fixed.  

4. What is the exact benefit of the homogenous network?  The paper shows this structure requires at least 3 times larger MACs in order to achieve competitive performance. It also says this network has potential for novel, massively-parallel hardware accelerators.  This is a bit vague to me, could you provide a concrete example? 

5. Related work on image pyramid and feature pyramid.  In this paper,  the data block preprocesses the images into multiple scales and the re-usable building block concatenates the features from multiple scales.  A discussion/revisit on the literature about the image pyramid and feature pyramid is recommended here and it might give some inspirations to address the memory issues of training the model on Cityscapes. 

========

Overall, I think the idea of building a homogenous network is reasonable to me and may lead to a better trade-off between latency, accuracy, and memory. However, the current manuscript still needs much improvement. 
























review_title: Interesting idea; need more experimental justification

final_score: rating: 3: Clear rejection
review_content: This paper studies homogenious networks, which is defined by the paper as networks that reuse building blocks with shared or different weights multiple times during the inference of the network. During the inference, the network iteratively use the same set of blocks to process input feature maps with different resolutions, and each step, the output feature map can be used by the prediction head to generate the output. This paper studies the cost of the network, in terms of MACs, parameters, memory footprints, and the accuracy vs. the number of iterations. The author noted that for the studied network, they need to increase the MACs by 3x in order to match the performance of regular networks. Despite this, this kind of homogeneous networks can be useful for novel hardware architectures with limited memory bandwidth. 

Strength of the paper: this paper presents an interesting study over a novel type of network and studied its accuracy-cost trade-off. 

Weaknesses ofthe paper: 
1/ The organization of the paper should be improved significantly. For example, the architecture of the homogeneous network is a central part of the paper, yet the introduction to it is only provided in two figures, and not in the method description. 
2/ It is not clear what should be counted as the contribution of the paper. I am assuming a few possibilities: a) proposing the new homogeneous networks? However, there are already several previous papers mentioning this type of networks. [cite] b) Better performance achieved by using homogenous networks? However, the experiments of the paper show that the network under study is not as competitive as previous baselines cited in this paper. Also, though the authors suggest this type of network can be useful for future hardware processors, but without explicitly mentioning the processor, it is hard to justify this. c) The analysis of the homogeneous network and their accuracy/cost trade-off. This might be counted as a contribution, but it is not clear if this analysis can, say, help us improve the performance of homogeneous networks. 

Overall, I think this paper is not complete and has not met the standard of acceptance. 
review_title: Review opinion

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a homogeneous network structure, where a single building block is iteratively executed to refine the outputs, in order to achieve a good trade-off between latency and accuracy. Empirical studies are conducted to show the superiority of such homogeneous networks over ENet and CGNet on CamVid and Cityscapes datasets.

Concerns: 

- The paper is not well organized which makes it hard to follow. The authors should revise the writing and clearly introduce the challenges, the contributions, the proposed architecture structure, the design insights, the experiment settings, and so on.

- The key concern about the paper is the lack of novelty and design insights. Why are homogeneous networks more beneficial for massively-parallel hardware systems? It would better to see the empirical evaluation and comparison against those non-homogeneous networks, such as multi-scale DenseNet, on a real hardware platform. 

review_title: Reivew

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper studies the influence of the use of shared vs independent parameters in re-used blocks of neural networks. This is achieved for the task of semantic segmentation, with a network architecture that iteratively refines its prediction.

Strengths:
- Studying the effect of parameter sharing vs the use of independent parameters in recurrent types of architecture is interesting and could lead to a better understanding of these architectures
- The paper is clearly written, and the methodology would be reproducible


Weaknesses:

Contribution:
- While I do believe that such a study could provide the community with a better understanding of architectures with re-used blocks, the scale of the study performed in this paper is too small to draw conclusions. The paper tackles a single task (semantic segmentation), and, more critically, evaluates a single architecture. There is therefore no evidence that the conclusions drawn here will generalize to other architectures/tasks, which significantly limits the potential impact of this paper on the community.

Related work:
- While I am not aware of similar studies, several important references tackling the task of recurrent semantic segmentation are missing, e.g., Pinheiro & Collobert, ICML 2014; Wang et al., ICCV 2019. These methods rely on different architectures, and studying the effect of parameter sharing in their frameworks would broaden the scope of this work.
- Similarly, video segmentation has also been tackled with recurrent architectures, e.g., Ballas et al., ICLR 2016; Valipour et al., WACV 2017. Studying the use of parameter sharing in this context would thus also increase the potential impact of this work.

Empirical results:
- While the results indeed evaluate the effect of parameter sharing on the chosen architecture, they seem to be somewhat disappointing in terms of absolute performance. In particular, on CityScapes (Fig. 1(b)), CNMM yields significantly higher mIoUs than the proposed method for the same number of MACs. Wouldn't it be possible to also extend this study to the CNMM architecture?

Minor comments:
- In the caption of Fig. 1, the authors mention that the parameters of the classification block are never shared. Does this mean that there is one classification block for each recurrent iteration?
- In Section 2.2, when explaining the normalization of the weight factors, I suggest using a different notation for the weights before and after normalization, e.g., w_n = w_n'/(\sum_i w_i').
- The title of Section 3.1 (Neural Architecture Search) is a bit misleading, as the authors simply perform a grid search of a few hyper-parameters, not a proper search over a large space of different architectures as in the NAS sub-field.
- The meaning of the different colors in Fig. 4 is not always explicitly defined.

review_title: A potentially interesting topic but studied on a too small scale

Meta-review: All reviewers agree that the paper is well written and some of the experiments are interesting. However, the paper did not clearly highlight how this work fits in with prior research, neither did it show what the advantages of the presented homogeneous network are. The authors addressed some of these concerns in the rebuttal, but not enough to sway the reviewers. In the end all reviewers recommend rejection, and the AC sees no evidence to overturn this recommendation.
Decision: Reject