Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a deep hashing model with only a single learning objective. It avoids the problem that existing deep hashing models are difficult to train due to a large number (>4) of losses. Specifically, it maximizes the cosine similarity between the continuous codes and their corresponding binary orthogonal codes to ensure both the discriminative capability of hash codes and the quantization error minimization. Besides, it adopts a Batch Normalization layer to ensure code balance and leverages the Label Smoothing strategy to modify the Cross-Entropy loss to tackle multi-labels classification. Extensive experiments show that the proposed method achieves better performance compared with the state-of-the-art multi-loss hashing methods on several benchmark datasets.
main_review: The paper proposes a deep hashing model with only a single learning objective. It avoids the problem that existing deep hashing models are difficult to train due to a large number (>4) of losses. Specifically, it maximizes the cosine similarity between the continuous codes and their corresponding binary orthogonal codes to ensure both the discriminative capability of hash codes and the quantization error minimization. Besides, it adopts a Batch Normalization layer to ensure code balance and leverages the Label Smoothing strategy to modify the Cross-Entropy loss to tackle multi-labels classification. Extensive experiments show that the proposed method achieves better performance compared with the state-of-the-art multi-loss hashing methods on several benchmark datasets.

Strengths: 
1. This work is meaningful for effectively optimizing the deep hashing models. It unifies the training objectives of deep hashing under a single classification objective.
2. The motivation is well presented by the authors in the manuscript.
3. The paper carries out extensive experiments and the supplementary material is rich in content.

Weaknesses:
1. According to the experimental results in Table 1, the proposed method does not consistently outperform the baseline methods, which is inconsistent with the sentence"outperforming the state-of-the-art multi-loss hashing models on three large-scale instance retrieval benchmarks, often by significant margins" mentioned in Abstract. In addition, the performance improvement of the proposed method is not significant.

2. As the main contribution of this work is to propose a unified objective function, to verify its superiority, the authors should compare the results of the following two experiments:
a) Baseline methods without any change 
b) The variant baselines which replace the multiple loss functions with the proposed single loss 

3. There are several issues that the authors may want to explain.
a) In the second paragraph of section 3 OrthoHash: One Loss for All, the authors write "o_i denotes a column vector belongs to i-th class". Since the dimension of the binary orthogonal targets O is C×K, I think o_i is a row vector.
b) The proposed method is supervised. Please expain the reason why using the unsupervised Bihalf as the baseline?
c) Why mAP@1k is used for ImageNet100 and mAP@5k for MS COCO for performance evaluation, especially when the two datasets are close in size. Besides, the metrics of performance evaluation for instance-level benchmark datasets are also different.

4. For experimental evaluation with SOTA methods, the authors should compare more newly deep hashing methods in the experiments.
limitations_and_societal_impact: 1. According to the experimental results in Table 1, the proposed method does not consistently outperform the baseline methods, which is inconsistent with the sentence"outperforming the state-of-the-art multi-loss hashing models on three large-scale instance retrieval benchmarks, often by significant margins" mentioned in Abstract. In addition, the performance improvement of the proposed method is not significant.

2. As the main contribution of this work is to propose a unified objective function, to verify its superiority, the authors should compare the results of the following two experiments:
a) Baseline methods without any change 
b) The variant baselines which replace the multiple loss functions with the proposed single loss 

3. There are several issues that the authors may want to explain.
a) In the second paragraph of section 3 OrthoHash: One Loss for All, the authors write "o_i denotes a column vector belongs to i-th class". Since the dimension of the binary orthogonal targets O is C×K, I think o_i is a row vector.
b) The proposed method is supervised. Please expain the reason why using the unsupervised Bihalf as the baseline?
c) Why mAP@1k is used for ImageNet100 and mAP@5k for MS COCO for performance evaluation, especially when the two datasets are close in size. Besides, the metrics of performance evaluation for instance-level benchmark datasets are also different.

4. For experimental evaluation with SOTA methods, the authors should compare more newly deep hashing methods in the experiments.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, authors propose a novel deep hashing model with only a single learning objective and claim that state of the art papers generally use a large number (>4) of losses. Specifically, authors show that maximizing the cosine similarity between the continuous codes and their corresponding binary orthogonal codes can ensure both hashcode discriminativeness and quantization error minimization. Authors show significance of their method using extensive experiments. 

main_review: 1) Main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. Proposed method is simple but novel. The submission is technically sound and well supported by experimental results. 
2) Under what conditions the proposed method would fail? Can authors comment more on this?
3) Using orthogonal targets is a simple but clever idea. Can authors comment on how can modify this method when one doesn’t have categorical labels but have some pairwise constraints and the goal is to learn quantized representation learning for better retrieval? 
4) In Table 1, proposed method is not always the best. From the discussion it was not clear why? Can authors comment more on why? Is it because the state of the art methods were already close to the best possible results or there is some drawback of the method that does not improve the results on some datasets? 
5) Angular margin used in the paper seems very related to semantic hashing [1] where angular similarity was used to generate representations better for hashing retrieval. Can authors compare and comment? 
6) In Figure 6, not only the distance between two distributions (intra class and inter class) matter but also the overlap between them matters. Can authors calculate and comment on the overlap between two distributions?  

[1] https://openreview.net/pdf?id=sFDJNhwz7S 
limitations_and_societal_impact: I don't see any potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents OrthoHash, a new deep hashing model that is trained by maximizing the cosine similarity between the continuous codes and the corresponding binary orthogonal target. In addition, code balancing is achieved by using batch normalization. Different from prior works that typically involves multiple loss functions, the training objective in this paper is simpler and easy to optimize. 
main_review: Strength:
- Neat idea and good novelty: the proposed training objective is well designed with the idea of cosine similarity. The final formulation with softmax-like loss function is very interesting, and it is simpler than the existing multi-loss approaches. Additionally, the use of BN for balancing code is elegant. 
- Strong results on multiple datasets. It is highly competitive compared to multi-loss approaches. 
- Good ablation studies to demonstrate the performance impact of different factors.
- The paper is well-written and easy to understand.

Weakness:
- One possible weakness is that it may have large performance degrade when the bit length is short. That is because in the target code generation, the target is sampled from Bern(0.5). One may expect larger information loss with short code.
- Evaluation: For ROxf and RPar, the authors use multi-scale features for instance retrieval. It is unclear to me if the compared exiting deep hashing methods also use such a multi-scale scheme. Does it a fair comparison?
- Evaluation: The authors use mAP@1K for ImageNet100, but mAP@5k for NUS-WISE and COCO. In instance retrieval, the authors use mAP@100 and mAP@all instead. They are inconsistent and not easy to interpret the performance. The authors may want to provide some intuitions/discussions about their design choices. For example, how many returned images are required to better reflect the  retrieval performance?  What's the effect of evaluating all returned images?  
- In current presentation, the proposed method performs only on par with prior works in category-level retrieval. This gives an impression that only ordinary improvements are achieved. I believe this paper has many advantages that are not well demonstrated in current form. One shining point should be its scalability to very large dataset. I was hopping to see at least one experiment at a large scale, to clearly differentiate itself from pair-wise and triplet-wise methods. This would strengthen the paper significantly. 


Overall:
- Overall this is a nice paper. My main concerns are in their evaluation. But this paper has merits that outweigh the flaws.
- This paper presents a simpler learning objective that can reduce the difficulty of training deep hash models. Given its effectiveness and the simplicity, I think this paper could serve as a strong baseline to facilitate future research. In my opinion, it would bring insightful contributions to our community. My initial recommendation is Accept.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: Authors propose a novel deep hashing model with only a single learning objective which is a simplification from most state of the art papers generally use lots of losses and regularizer. Two of the three reviewers liked its effectiveness and the simplicity, they believe it would bring insightful contributions to our community. Reviewers liked the final formulation with softmax-like loss function. They also found the use of Batch normalization for balancing code elegant.

One of the reviewers raised certain concerns.  The authors in rebuttal clarified that most of the negative argument raised by reviewers are misunderstanding and are not valid. It is clear that the concerns raised were already addressed in the main draft. 
.
Overall, the merits of the paper justifies the publication.  
Decision: Accept (Poster)