Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a joint learning framework for document ranking and query suggestion. It introduces the session embeddings to capture the connections between queries in a session, and potential impact of previous queries in a session to the document ranking of the current query. I like the idea in general. 

However, I have a few comments as follows:

- Multi-task Match Tensor model, which is important in the experiments (best results), is only briefly introduced in Section 3.4. It is not very clear how to extend from match tensor model to a multi-task match tensor model. This makes me feel like this paper is not self-contained. The setting for this model is not introduced either in Section 4.2. 

- Section 3 is written mostly about what has been done but not why doing this. More intuition should be added to better explain the idea. 

- I like the analysis about testing the impact of the different model components in Section 4.4, especially analyzing the impact of the session. It would be nice to have some real examples to see the impact of session embeddings on document ranking. One more related question is how the clicked documents of a previous query in the same session influence the document ranking of this current query? Would that be feasible to consider in this proposed framework?

- Session seems to play an important role in this multi task learning framework. This paper used the fixed 30 minute window of idle time to define a session. It would be nice to know how sensitive this model is to the definition / segmentation of sessions. 

review_title: I like the idea of a joint learning framework for document ranking and query suggestion, but the writing and analysis can be improved.

final_score: rating: 7: Good paper, accept
review_content: The work is interesting and novel. The novelty lies not in the methods used (existing methods are used), but in the way these methods are combined to solve two problems (that so far have been treated separately in IR) simultaneously. The fitness of the proposed architecture and methodological choices to the task at hand is sufficiently argued. 

The experimental evaluation is not the strongest, in terms of datasets and evaluation measures. While I understand why the AOL dataset was used, the document ranking experiments should also include runs on any of the conventional TREC datasets of documents, queries and actual (not simulated) relevance assessments. Simulating document relevance from clicks is a good enough approximation, but why not also use datasets with real human relevance assessments, especially since so many of them exist and are so easy to access?

When evaluating ranking, MAP and NDCG are indeed two popular measures. But the choice of NDCG@1,3,10 seems a bit adhoc. Why not NDCG@5? Furthermore, as the aim seems to be to assess early precision, why not also report MRR? 

The paper reports that the M-NSRF query suggestion method outperforms all baselines. This is not true. Table 2 shoes that M-NSRF is best for BLEU-1/2, but not for BLEU-3/4. 

Three final points:

- Out of the contributions enumerated at the end of Section 1, only the novel model and the code & data release are contributions. The rigorous comparison to soa and its detailed analysis are the necessary evaluation parts of any empirical paper.  
- The conclusion states that this work provides useful intuitions about the advantages of multi-task learning involving deep neural networks for IR tasks. What are these? Where were they discussed? They should be outlined here, or referred to somehow.
- Although the writing is coherent, there are a couple of recurrent English language mistakes (e.g. missing articles). The paper should be proofread and corrected.
review_title: The paper presents a multi-task learning architecture for document ranking and query suggestion, based on (predicted) user clicks.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Novelty: It looks quite straightforward to combine document ranking and query suggestion.  For the model architecture, it is a standard multi-task learning framework. For the “session encoder”, it is also proposed (at least, used) in (Sordoni et al., CIKM 2015). Therefore, I think the technical novelty of the work is limited. 

Clarify: The paper is in general well written. One minor suggestion is to replace Figure 1 with Figure 3, which is more intuitive. 

Experiments: 
1.	Why don’t you try deep LSTM models and attention mechanisms (although you mentioned them as future work)? There are many open-source tools for deep LSTM/GRU and attention models, and I see no obstacle to implement your algorithms on their top. 
2.	In Table 2, M-NSRF with regularization significantly outperforms the version without regularization. This indicates that it might be the regularization that works rather than multi-task learning. For fair comparison, the regularization trick should also be applied to the baselines. 
3.	For the evaluation metric of query suggestion, why not using BLEU score? At least, you should compare with the metrics used in (Sordoni et al., 2015) for fairness. 
4.	The experiments are not very comprehensive – currently, there is only one experiment in the paper, from which one can hardly draw convincing conclusions.
5.	How many words are there in your documents? What is the average length of each document? You only mention that “our goal is to rank candidate documents titles……” in Page 6, 2nd paragraph. It might be quite different for long document retrieval vs. short document retrievel. 
6.	How did you split the dataset into training, validation and test sets?  It seems that you used a different splitting rule from (Sordoni et al., 2015), why? 
review_title: This work proposes a multi-task learning framework, M-NSRF, that can jointly learn document ranking and query suggestion. A session recurrence is introduced and modeled by an extra LSTM. The authors carry out extensive experiments to verify their algorithm. But I think this work is not of enough novelty and the experiment design can be improved. 

Meta-review: Overall, the committee finds this paper to be interesting, well written and proposes an end to end model for a very relevant task.  The comparisons are also interesting and well rounded.  Reviewer 2 is critical of the paper, but the committee finds the answers to the criticisms to be satisfactory.  The paper will bring value to the conference.
Decision: Accept (Poster)