Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a strategy to automatically adjust the temperature scaling based on the context of words in a sentence for NLP. Experiments demonstrate that this approach can significantly improve perplexity scores on several datasets popular for NLP.

NLP is not an area of research I'm very familiar with so this review is limited to my understanding of temperature scaling as a general technique to improve learning. As described in the paper, temperature scaling is a type of hyper-parameter estimation that adjusts the sensitivity of the softmax function as training evolves. The paper proposes to learn a function that given context, adjust the temperature automatically. This can be seen as a meta-learning method. 

I believe this can be a useful technique but before considering such an approach as a general strategy, more theoretical insights should be provided. The authors report on ablation studies that demonstrate some empirical benefits. However, until I see more theoretical analysis on how the method improves convergence or lead to better losses by smoothing out the output of the objective function, I remain skeptical of the usefulness of this as a general training method.  
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposed a contextual temperature scaling to improve language modeling. The temperature model is parameterized using a deep neural network. Experiments on the language modeling datasets show some effects of the method. 

The idea of dynamic temperature scaling has been tried in other works and tasks (e.g., attended temperature scaling). The paper parameterizes this mechanism with DNNs for the language model.  Though the idea looks interesting, it fails to explain why the scaling is better than other dynamic temperature scaling frameworks. 

The experiments are not solid. The baseline only includes Mos, which is not very strong. To validate whether this approach works with other LM of high-order attention or self-attention, a better baseline model is required (e.g., transformer, GPT). I would like to see this technique can help either NLU or NLG tasks, instead of just pure modeling. The case analysis section needs more examples instead of just cherry-picking few. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This work proposes a learned and context dependent way to calculate the temperatures for the softmaxes. More specifically, a low-rank affine-transformation, taking the hidden state at the current step as input, is used to calculate scalar weighting for every token in the vocabulary. The method is very general, and can be used in combination with other techniques in tasks such as language modeling and text generation. Experiments on language modeling with Penn TreeBank and WikiText-2 show that the proposed method yields strong performance.

Overall I found the paper well-motivated and easy to follow. The empirical results are solid and strong. The analysis is also interesting. I vote for an acceptance, if the authors can polish the writing.

Details:

- Eq. 5. The temperature scalar for each token competes with each other, since they are calculated with a softmax (and then rescaled). Another way is to use, e.g., a sigmoid function. Can the authors explain the motivation behind the use of softmax?

- Another view of the proposed method is that it learns a context-dependent weighting of the tokens in the vocabulary, such that "important" tokens (those with smaller \tau) receive more gradient updates. Can the authors comment on this? Also, I don't see the thermodynamics connection and find calling the proposed method `temperature` a bit misleading. 

- Adding onto above. [1] discusses the low-rank bottleneck of using a single softmax. Since elementwise matrix product can blow up the rank, how do the authors think the proposed method can serve as a more efficient way to deal with the softmax bottleneck?

- Last but not least, the paper can be improved a lot if the authors can thoroughly polish the writing.


[1] Breaking the Softmax Bottleneck: A High-Rank RNN Language Model. https://arxiv.org/abs/1711.03953
review_title: Official Blind Review #1

Meta-review: With an average post author response score of 4 - two weak rejects and one weak accept, it is just not possible for the AC to recommend acceptance. The author response was not able to shift the scores and general opinions of the reviewers and the reviewers have outlined their reasoning why their final scores remain unchanged during the discussion period.
Decision: Reject