Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper authors propose to do saliency object detection with weak supervisions. To train the saliency network, pseudo labels are generated and refined step-by-step, by involving the depth map and the caption completion net. Authors relabeled NJUD and NLPR datasets with off-the-shelf image captioning model. Experiments are carried out on four SOD benchmark datasets and demonstrate the effectiveness of the proposed method.
main_review: This paper proposes to use a weakly-supervised manner to train saliency object detection network, which is an interesting and useful topic. It avoid extra test-time cost so the efficiency of inference is obtained.

The main comparison is made to MSW considering the framework and supervision used in the proposed method. However, this paper and MSW seem to evaluate on different testing datasets and the authors of this paper retrain the MSW model on the new datasets. It would be interesting to have a better aligned evaluation and comparison, if it is possible to reuse some of the datasets used in the previous paper.

The proposed method is also adapted to the fully-supervised settings, and it is interesting to see that the proposed method can obtain higher performance than other state-of-the-arts. More experiments and discussions are necessary to justify why the updated Eq 1 can help with the model training even if we already have the ground-truth supervision.
limitations_and_societal_impact: Authors have adequately addressed the limitations and potential negative societal impact of their work, or there are no obvious issues regarding that direction.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Training saliency detection models with weak supervisions, e.g., image-level tags or captions, is appealing as it removes the costly demand of per-pixel annotations. The manuscript proposed a novel joint semantic mining for weakly supervised RGB-D salient object detection. Specifically, the proposed framework maintain per-pixel pseudo-labels with iterative refinements by reconciling the multimodal input signals. Moreover, the authors also prepared a new dataset that is constructed by augmenting existing benchmark training set with additional image tags and captions. Sufficient experiments on four benchmarks show that the proposed method surpass the state-of-the-art methods. The shortcomings lie in the following aspects: the explanation of the overall framework is not clear enough, take figure 2 as an example, there are too many internal inputs, and the data flow is chaotic. 
Please check the following details.
main_review: a) The main idea is reasonable and novel.
The manuscript proposed a novel joint semantic mining for weakly supervised RGB-D salient object detection. Specifically, the proposed framework maintain per-pixel pseudo-labels with iterative refinements by reconciling the multimodal input signals. Moreover, the authors also prepared a new dataset that is constructed by augmenting existing benchmark training set with additional image tags and captions.
b) Extensive experiments.
In the experiment section, the evaluation on 4 benchmarks demonstrates that the proposed framework attains superior performance to previous models.
c) The overall pipeline is well described and easy to follow.
limitations_and_societal_impact: a) The experiments were extensive, but the manuscript placed the ablation experiment at the front of the Experiment section, which was not very friendly for quick understanding of the paper. I suggest the authors analyze the model performance (Sec. 4.4) at the beginning of the Experiments section.
 
b) The manuscript is well-written, but the explanation of the overall framework is not clear enough, take figure 2 as an example, there are too many internal inputs, and the data flow is chaotic. 
 
c) I wonder how to prevent the model from going down in the wrong direction.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This article addresses salient object detection using depth maps and image captions as supervised signals. The proposed network includes a SSM for spatial semantic modelling and a TSM for cross-modal semantic mining. Additionally, a CapS dataset is built to extend existing datasets with extra weak annotations.
main_review: PROS:
- The paper is well written and easy to follow, with clear, complete and concise explanations.
- The paper provides a good and detailed description of the proposed network.
- The ablation study is comprehensive.

CONS:
- My major concern goes to the design of network architecture, especially the necessity of the depth network. How about using $D_\text{mask}$ directly for $D_s$? Please clarify the motivation to predict a depth map rather than directly using the true depth map?
- The approach exploits a combination of multiple sources (\ie, depth maps, captions, tags)，which leads to unfair comparisons with some leading methods that only rely on one or two types of knowledge. This make the results somewhat unconvinced.
- L339 claims the model to be end-to-end learnable, however, the modules are "independently trained" (L207). I am guessing whether the network can be end-to-end trained in practise. It will be good to study the performance difference.
- Additionally, is CRF applied during every update? How is it implemented to be end-to-end trainable?
- How does the proposed model perform in the tailed categories? The long-tail problem seems to be obvious in CapS (as shown in Fig. 2 in the supplement)
- It is not that obvious why does the model achieves the best results in the fully supervised setting, since the model will degrade to a simple two-branch neural network. More discussions are preferred.
- Only RGB-D datasets are taken into account in the experiments. What's the reason for this, especially considering that the network does not rely on depth information during inference? What's the model performance on RGB only benchmarks?
limitations_and_societal_impact: Yes. The supplement has provided some discussions of the limitations of the work.
review_title: null

Meta-review: The paper finally receives mixed reviews with slight preference to being accepted. We thoroughly check the review comments of Reviewer n5it (the only reviewer who has ‘reject’ opinion) and the authors’ responses to them in the rebuttal. Although the n5it submitted no post-rebuttal review, from our perspective, his or her comments mostly fall into technical questions (not some fundamental ones for this work), and the authors’ rebuttal seems to be mostly satisfactory to clarify the concerns. This is the ground for our recommendation, and the authors may need to reflect their own responses about the novelty and additional experiments in the final draft along with the source code. 
Decision: Accept (Poster)