Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a normalization method for monocular-based depth estimation before computing the regression loss. It is based on scale-and-shift invariant loss. Instead of computing the median and doing the normalization globally, it split the depth map into multiple size patches and compute scale-and-shift invariant loss for each patch. They propose two strategies to generate the patches consisting of spatial domain splitting and depth domain splitting. In the zero-shot setting, the results show the model with the proposed normalization can outperform all baselines. 
strengths_and_weaknesses: 1. This paper is well written. It is easy to read.
2. The proposed idea is clean and simple. We can easily apply it to other depth estimation models.
3. The improvement on the zero-shot monocular depth estimation task is significant. 

Weakness:
1. Table 1 does not show the ablation when combining the different domains. It is unclear if combing all of them can get the best performance.
2. The proposed method can be applied to all monocular-based models. Have the authors tested the loss on fully supervised monocular/stereo depth models? This paper only presents the results in a zero-shot setting.
3. The proposed method is very simple, so the contribution might be limited. To improve the contribution, it is important to show the generalization of different models and datasets.
questions: 1. Because it will compute multiscale scale-and-shift invariant loss, will the proposed loss make the training significantly slower?
limitations: None
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a novel normalization technique for learning monocular depth estimation networks with various sources of datasets that may have different depth scales. Unlike the previous approach that learns with the image-level normalization, which often disregards the fine-grained depth difference, this paper presents a multi-scale depth normalization that hierarchically normalizes the depth representations based on both spatial information and depth distributions. Experimental results have shown the superiority of this method even though the technique is relatively simple.
strengths_and_weaknesses: + The idea of using multi-scale normalization, inspired by conventional normalization methods such as AlexNet, SIFT, and HOG, is very simple, but effective.
+ Dividing the normalization candidates as spatial and depth axis is interesting and makes sense.
+ The state-of-the-art performance is attained.
questions: - Using a hierarchical normalization has been already used in many other fields, as the authors also described, and applying this for monocular depth estimation is the first attempt in our knowledge, so it would be fine. However, due to its simplicity, I think more through experiments should be conducted. For instance, why not dividing the depth representation irregularly? or is there any way to consider additional semantic information, e.g., object boundaries, and divide the depth representation with respect to this semantic information? Without this kinds of through experiments, it would be hard to say this regular hierarchical representation is optimal. 
-  Fig. 2 is hard to follow. In the left, what the authors tries to say? Maybe different colors are coming from the different spatial location? It would be better if it is clarified.
- It would be interesting if we only used a single dataset for learning monocular depth estimation networks which does not suffer from depth scale or shift problem, e.g., KITTI itself. Maybe performance is degraded with this additional local normalization layers? 
- To overcome depth scale and shift problem, other loss functions may be used, for instance, AbsRel. It would be better if the other loss functions are compared. 
- Computation complex may be evaluated as well.

Minor comments:
- It would be great if the authors clarify which datasets are used for each evaluation of Table 2. 
- Other backbone models such as Monodepth2 can also be used? In many applications, the CNN-based models are still popularly used, so this experiment would be very interesting.

Overall I like such a simple idea, but more through evaluations would be required to argue this is optimal. I really want to see the rebuttal.
limitations: The authors well mentioned the limitations of this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors introduce a novel strategy for monocular depth estimation that focuses on both global structure and fine-grained details. Their implementation is effective at preserving fine-grained details via the spatial and depth domains using Hierarchical Normalization. Ultimately, performing well in terms of accuracy, even against state-of Art benchmarks from before our work was published.
strengths_and_weaknesses: An interesting solution to a renowned problem in monocular depth estimation.

Clearly motivated by the shortcomings of existing SOTA.

The presentation is nice, for the most part; the language is okay but should be further improved if accepted for publication.

Weaknesses:
Seems the authors ran out of space, with many means to improve figures and tables to increase the data-to-ink ratio.

Figure 3-4 : why not share x and y axes labels and make graphs easier to see in less space.

Add future work to the Limitations section.

Be consistent with style: different tables use different schemes (some use arrows, others use colors).

Avoid orphans (single words taking up entire lines, like line 205).

Lack of depth in analysis for the many ablations conducted.
questions: Why the median in (1)? More elaboration on the insights could be worthwhile.
limitations: A lack of limitations in the ability to be extended to MVS; lack of demonstrating the final reconstruction results (just depth images depicted).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies zero-shot monocular depth estimation. One way to improve zero-shot performance is to train on a combination of datasets. The problem with this strategy is that different datasets has different scales and shifts. One way to address this issue is to train with a scale and translation invariant loss (SSI [22]). This paper argues that the SSI loss is normalized over an entire image. A more effective way is to normalize over a small local window or a certain depth range. This paper thus propose a hierarchical depth normalization training loss, which helps their model to achieve state-of-the-art performance in zero-shot depth estimation on five major benchmarks.
strengths_and_weaknesses: Strength:
+ This paper proposes a simple yet very effective version of the scale-shift-invariant loss for training depth estimation network. This loss proves to help in improving zero-shot depth estimation (Table 1). I like this idea.

Weakness:
- It is not very clear if the improvement really comes from the proposed hierarchical depth loss (Table 2). The "Ours" model differs from SOTAs in the training data, the number of training samples, and the network architecture. The conclusion would be more solid if only one thing is changed at a time.  For example it is unclear if it is fair to claim that "our method outperforms previous methods by a large margin on multiple benchmarks with fewer training data than recent state-of-the-art methods, such as Leres [33] and Midas [22].", because it could be a result of using higher quality training data.


questions: 1. The formulation in equation (1) seems to be different from the original one in the SSI [22] paper. SSI solves for an optimal scale and shift between pred and gt depth, while equation (1) is normalizing using simple global stats such as median and mean. Why?
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper addresses the problem of training a monocular depth estimation network from variable sources of data. As opposed to only using a single scaling factor as in existing work, the authors propose local schemes for normalising. While the proposed approaches are conceptually simple, they result in a non-trivial boost in performance (both qualitatively and quantitatively) and will likely be of interest in the field of monocular depth estimation. 

The reviewers were broadly in support of this paper. This area-chair agress, and recommends acceptance. However, the authors are strongly encouraged to incorporate the valuable comments and suggestions from the reviewers into the revised text. 

Minor comments:
* Fig 2 (a) is not clear and should be revised to make it clearer what it is trying to communicate. 
* Re-title section 5.1. To “Limitations”
* Add the new results for NYUv2
* The two supplementary videos are not very informative. Should consider using different examples


confidence: Certain
Decision: Accept