Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the performance of multi-task learning in universal dense retrieval systems. The authors show that the network capacity is not the limiting factor for the universal retrieval accuracy. Instead due to insufficient optimization, large portion of parameters show low sensitivity to each task. A method, called TACO-DR, is proposed that optimizes the task-specialty of the parameters. 
strength_and_weaknesses: The paper is generally well-written and easy to follow. I have the following comments and questions:
- If I understand correctly, the results in table 2 of the paper should be comparable to table 3 of Chen et al. (2022). It seems that their best results are slightly different from what is reported in table 2. For instance, for T-REx and TQA Chen et al. (2022) are reporting 85.03 and 71.71 while the results in table 2 are 77.62 and 68.78. If possible and in order to make the results more comparable, can you use a similar training strategy to theirs? 
- For the sake of comparing the models on all 11 KILT tasks, can you also add the results for the 3 remaining tasks? 
- Can you provide the confidence intervals in Figure 3(a)
clarity,_quality,_novelty_and_reproducibility: The paper is clear and the idea seems novel to me (but I'm not an expert in this area). 
summary_of_the_review: Overall, I found the idea of having task-specific adaptive learning rate interesting. However, I believe the empirical results can be further improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposess a multi-task universal retrieval model that is based on task-specific prompt tuning and adaptive learning are based on task sensitivity. Experiments on the KILT benchmark dataset over existing models, and ablation on the task sensitivity, show that although the model capacity of existing works is not sufficiently utilized, universal retrieval models can capture task-specific signals. The proposed method tries to address the limitation of model capacity utilization.
strength_and_weaknesses: [+] The analysis of task sensitivity provides a clearly articulated motivation for the proposed work.

[-] In many cases, the performance difference between TACO-DR and the next best baseline is marginal.

[-] The proposed work seems to be a straightforward application of prompt tuning and task sensitivity adaptive learning rate (similar to momentum-based optimizers in the literature). If there exist novel unique contributions, these could be better highlighted in the introduction and related work.

[-] More relevant baselines are needed, e.g., other formulations of task-specific learning rates, e.g., a self-paced version that the task-specific hyper-parameter is weighted based on a loss-based task difficulty score, or an entropy-based task-specific learning rate. If baselines are trained with vanilla gradient descent, it would be interesting to see if standard Adagrad produces the same results. 
clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written. Related work could better describe the key differences between this paper and existing literature. 


What is "special fraction" in Fig. 3 (b)?

Fig. 3 (a) needs to increase such that the x-axis ticks are more readable. Also, it seems that the ablation study shows there is not much difference between the proposed models and MT-T5-ANCE.

Please further explain the task commitment in Table 4, in particular how the task commitment rate is computed. Wouldn't make sense to computer this task commitment rate for the best-performing baselines and present a comparison among models?
summary_of_the_review: The analysis of existing models is promising and could lead to new universal retrieval methods. However, the current proposed method seems to be a direct application of existing methods in the literature. The introduction and related work could be enhanced to clearly showcase any technical contributions. The proposed work could be further improved by comparing the task-sensitivity adaptive learning rate method with other adaptive learning methods in the literature.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a multi-task learning method for universal retrieval. They identify that the reason existing approaches fail is because they fail to capture task-specific signals and therefore propose a method that learns task-specific parameter-specific learning rates to better capture task-specific signals. 
strength_and_weaknesses: Strengths:
* The authors extend sensitivity guided adaptive learning (Liang et al., 2022) to the multi-task setting.
* Empirically, the authors show that they are able to outperform multi-task and single-task baselines on the KILT benchmark.
* The authors also show that adding task labels as a prompt improves multi-task retrieval performance. 

Weakness:
* The sensitivity analysis graphs in  figure 1 and figure 2 are unclear and need more explanations. The x-axis in unlabelled in the graph and it is unclear what the axes are and what range of values they show. In figure 2, is there a tail that is not shown. How come the peak on the left is lower for the TACO methods but the tail is the same for all the methods?
* The percentage ranges for the values in figure 3a and table 4 don't seem to be consistent with each other. Is the commitment rate calculated differently for these?
* It's not clear whether the current framework can handle the case when a parameter is important for a subset of the tasks but not all the tasks. 
* This method doesn't seem specific to retrieval. Can other multitask settings also benefit from using task-specific parameter-specific learning rates?
clarity,_quality,_novelty_and_reproducibility: The figures can be improved by adding labels to axes and adding additional explanation about what the figures are showing. 

I would also encourage the authors to release their code to facilitate reproducibility. 
summary_of_the_review: The presented work does achieve state-of-the-art results for multitask retrieval. TACO-DR is an extension of an existing method (Liang et al., 2022). Although it works well for retrieval, it's not clear whether this method is useful just for this one setting or is more broadly useful. It would be interesting to investigate whether this method provides benefits for other multi-task learning applications.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The submission concerns universal retrieval, or multi-task dense retrieval, where queries and documents from different sources are mapped to the same space. The paper first conducts a study using T5 on the KILT dataset and empirically shows that there is some gap between single task performance vs multi-task. Then parameter importance analysis is done using existing techniques, showing that it is mainly due to parameters under capturing task specific signals. Then an optimization with adaptive learning rates is applied. Experiments on the KILT shows the proposed framework can achieve better average performance than alternatives, including BM25, naive multi-task learning, and two multi-task methods can focus on mitigating gradient conflicts.
strength_and_weaknesses: Strength
The paper is generally well written, though some details can be provided more properly.
The studied problem is relatively new.

Weakness
Novelty. The analysis and method used in this paper are applications of existing methods. Though it is a good effort, this is unlikely to be sufficient for a top-tier ML venue.
Given the rich literature of multi-task learning, there are methods that do not focus on gradient conflicts. The chosen related baselines are likely to be used in the wrong context.  

clarity,_quality,_novelty_and_reproducibility: The paper is generally clearly written. 

The major concern is novelty. First, the paper mainly applies an existing adaptive method to the universal retrieval task, which is one kind of multi-task learning in general. It is good to show that the phenomenon for retrieval is mainly due to “under captured task specific signals”, but it is one kind of problem in general multi-task learning. 

Though the proposed method is shown to perform better than few multi-task learning optimization methods that focus on gradient conflicts, there is a rich literature of multitask learning that deals with other concerns, and the paper does not discuss why the proposed method works better (again, the optimization used in this paper is an application, not really a brand new algorithm). For example, the classical GradNorm (GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. ICML2018) paper is not cited or discussed, which shares similar spirits by adjusting gradient magnitudes/learning rates. There are many follow-ups of GradNorm. 

The other major technique section, the prompt trick, is commonly used in the literature so they do not contribute much to novelty. 

Reproducibility should be fair but not very clear. The authors claim “code and model checkpoints will be open-sourced” but not yet. 

summary_of_the_review: Overall the paper deals with a relatively new problem and is clearly written. The novelty is a major concern for a top-tier ML venue (an NLP venue may make more sense). Please see the detailed summary and discussions above. 
Some other comments:

“The results indicate that model capacity may not be the bottleneck of universal retrieval” - what size of T5 is used (this should be clarified anyway - looks like BASE is used from experiments)? It looks like the gap is smaller in Table 1. What if larger T5 models are used?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Strength
* Techniques for improving multi-task learning in universal retrieval are proposed.
* Experiments are conducted.

Weakness
* The novelty is limited. The method is a combination of existing techniques.
* The experimental results need to be more convincing. The work is not significant enough.
* There are places that need to be better explained.

The authors have added one more baseline during the rebuttal. However, the concerns on the novelty and significance remain. The authors have clarified some of the problems. However, it is unclear whether they can fix all the issues if the submission is accepted.
justification_for_why_not_higher_score: The submission is not of ICLR quality.
justification_for_why_not_lower_score: The submission is not of ICLR quality.
Decision: Reject