Contextual Temperature for Language Modeling
Authors: Pei-Hsin Wang
Sheng-Iou Hsieh
Shieh-Chieh Chang
Jia-Yu Pan
Yu-Ting Chen
Wei Wei
Da-Cheng Juan
Venue:  (2020)
Keywords: natural language processing
language modeling
sequence modeling
temperature scaling

Abstract: Temperature scaling has been widely used to improve performance for NLP tasks that utilize Softmax decision layer. Current practices in using temperature either assume a ﬁxed value or a dynamically changing temperature but with a ﬁxed schedule. Little has been known on an optimal trajectory of temperature that can change with the context. In this paper, we propose contextual temperature, a mechanism that allows temperatures to change over the context for each vocabulary, and to co-adopt with model parameters during training. Experimental results illustrated that contextual temperature improves over state-of-the-art language models signiﬁcantly. Our model CT-MoS achieved a perplexity of 55.31 in the test set of Penn Treebank and a perplexity of 62.89 in the test set of WikiText-2. The in-depth analysis showed that the behavior of temperature schedule varies dramatically by vocabulary. The optimal temperature trajectory drops as the context becomes longer to suppress uncertainties in language modeling. These evidence further justiﬁed the need for contextual temperature and explained its performance advantage over ﬁxed temperature or scheduling.

Under review as a conference paper at ICLR 2020
CONTEXTUAL TEMPERATURE FOR LANGUAGE MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents generalized approach to provide an individual optimal tem- perature trajectory to each class. Temperature scaling is an effective technique control the smoothness of the distribution. Current implementations of tem- perature scaling assume either a fixed or manually-crafeer dynamically changing schedule. This schedule is used identify the shared optimal trajectory. which is applied to every class. However. our studies indicate that under the change of context, we can identify the corresponding individual optimal trajectory for each class. To this end, we propose contextual temperature, mechanism that allows the temperature to be learned during training along with the remaining model pa- rameters, over the context for each vocabulary. Experimental results confirm that contextual temperature significantly improves state-of-the-wee language models. The proposed method achieves a perplexity of 55.31 and 62.89 on the test of Penn Treebank and WikiText-2, respectively. Additionally, in-depth analyses in- dicate that (a) every vocabulary possesses its individual schedule of temperature, and (b) due to the increased context from every time step, the optimal trajectory drops in order to suppress the uncertainties.
INTRODUCTION
Due to the discrete nature of human language, the Softmax layer is mandatory to convert learned representations into a sequence of linguistic vocabulary. Temperature scaling often used along with the Softmax layer in natural language processing. Generally, the temperature is applied as denominator to output logits of the Softmax layer (Krizhevsky et al.] 2012) Bahdanau et al., 2014] Hu et al. 2017 Caccia et al. 2018). The scale of temperature controls the smoothness of the output distribution. That is, given a temperature т, when T - oo, the distribution becomes more uniform, thus increasing the uncertainty. Contrarily, when - O, the distribution collapses to point mass.
Although temperature scaling has been justified to achieve great success, existing implementations limited. For instance, the temperature is assumed to be constant throughout training (Norouzi) jet al. 2016: Ma et al., 2017, Chen et al. 2019), or to be a fixed schedule (Hu et al., 20.7). Most importantly, none of the existing works studies the effects
different word tokens when the tem- perature changes. In reality, however, the temperature of each token can be dramatically differentt Figure [(a) shows the optimized temperature for each word token during the course of training. As shown in the figure, some certain words have a distinct heating-up temperature scaling, while the majority of words have a scaling that gradually cooling down the temperature. We argue that existing methods limit themselves to some fixed schedules, and thus have great difficulty to generalize.
In addition, another example can be observed Figure [(b] which indicates that the average tem- perature drops, as the length of the context increases. This suggests that the temperature mechanism helps promote stochasticity
the beginning of sentence, then gradually suppressing the uncer- tainty until the end. All of these suggest the use of more generalized temperature mechanism with the advantage of being able to deal with these phenomena.
We propose contextual temperature, generalized temperature scaling method which takes both the history a sequence and a particular vocabulary into consideration. Through optimizing the use of temperature scaling by the change of contexts, contextual temperature has the ability to generate unique temperature for each vocabulary. As we parameterize contextual temperature using a deep neural network, its parameters co-evolve with the rest of model parameters, making the temperature

---

Under review as a conference paper at ICLR 2020
adapt to the training procedure. Experiment results on language modeling demonstrate significant improvements on both Penn Treebank and WikiText-2 datasets, with and without the fine-tune set- shown on both validation and testing splits. In addition, contextual
ting. Consistent improvements
conduct comprehensive analyses and ablation studies to confirm the improvements temperature. We observe that method is capable of controlling the uncertainties as the patterns of contexts change, allowing language models to achieve much better performances.
To the best of knowledge, this is the first systematic work that studies the role of per-token temperature changing over context for training language models, The experimental results suggest new way of training sequential models with discrete outputs, that is, using parameterized tem- peratures. We have also established the link between the control of model uncertainty and the use of temperatures, paying the way for extensions on tasks that require such long-term control, for instance summarization, translation, and dialogue generation.
5000>
0.30020
perrrrotttae Iverraee terr por var (+2)
and
di
0.30015
10-2
0.00010
10-3
<unks
0.30005
10
0.00000
200
400
600
800
1000
2
12
14
position sequence
(a)
(b)
Figure 1: (a) Temperature of each word using the proposed method over training epochs. Each line in the figure represents a distinct token ranked by the frequency. The vertical axis shows the actual temperature value with minus of 2. (b) Mean and confidence interval of the temperature over the position in the sentence. We can see that the average temperature high at the beginning of the sentence and gradually decreases time.
RELATED WORK
2.1 LANGUAGE MODELING
Given a sequence of history tokens Z1..t11. language modeling aims at predicting the next token a, in a sentence. In other words, the goal to model the probability of choosing the 4thh word, P(z, k|a1..t-1), by using an encoding function and Softmax layer f is parameterized by 0 to compress the history
W.
P( k| 0) o(z)
Here W E Rd is matrix that converts a d dimensional output of f into a K dimensional vector. The state-of-the-an language models adopt deep neural networks as the encoding function, including the earlier feed-forward networks (Bengio et al. (2003), more recent recurrent-based ones (Mikolov al. 2010, Hochreiter Schmidhuber 1997) and attention-based models (Devlin et al. 2018).
2.2
SOFTMAX LAYER
Sofumax layer (2) normalizes a K dimension, real-valued vector z to make it sum to
o(2):
Σ5 e
2

---

Under review as a conference paper at ICLR 2020
Recent progress in language modeling suggests that Mixture of Softmaxes (MoS) Yang al. 2018)
would significantly improve the performance by computing multiple Softmax distributions and use set of weights to sum them up the final probability distribution. To achieve this, set of M matrices W., is applied to the output generated by f. That is, Zm f 0)" W m. where Wm E IRdx K has the dimension of the embedding d and the dimension of the vocabulary K. The probability distribution of the next word 24 under the MoS model is thus a mixture of M Softmaxes weighted by æ. Here θ U Wm U O.
PMo (T+ = lo 11111-1100 =
Σ Tm a (Zm) )
(3)
2.3 TEMPERATURE SCALING
Temperature scaling is an approach used to control the smoothness of the distribution. Instead of applying the Softmax function as suggested in Equation logits here are divided by the temperature before passing through the Softmax layer.
P( 1 = o(z/T)x
(4)
Constant Temperature. Earlier works that adopt the constant temperature can be traced back to Model Distillation (Hinton et al. 2014). Several works have been taking advantage of a fixed temperature during training (Norouzi et al., 2016 Ma et al., 2017 Chen al. 2019). For instance, (Norouzi et al. [2016] Ma et al. 2017) optimize log-likelihood on augmented outputs, which are sam- pled proportionally to their exponential scaled rewards, where the temperature controls the degree of augmentation. Other works incorporate the temperature to enhance calibration during inference (Guo et al. 2017), or to achieve great trade-off between quality and diversity (Caccia et al. 2018).
Dynamic Temperature Over Training Iterations. A wide range of works adopt manually- crafted schedule to tune the temperature during training. Notably in (Hu et al.] 2017), a new text generation architecture is introduced that combines VAE and discriminator. Since text samples discrete and non-differentiable, a continuous approximation based on the Softmax with a decreasing temperature is used to enable gradient propagation from the discriminator to the generator. Similar techniques are adopted in the gumbel-softmax trick (Jang et al. 2017). which also allows gradients to pass through discrete sampling objectives. Studies also show that temperature with a heating up schedule makes the embedding vectors more compact (Zhang et al. 2018).
Dynamic Temperature Over Word Position. Another work that is closely related our work, is the adaptive temperature over an attention model (Lin et al., 2018). We note that contextual temperature further learns the temperature for each vocabulary in the output distribution.
METHODS
3.1 CONTEXTUAL TEMPERATURE
Contextual temperature is a mechanism that chooses the optimal temperature by considering the "context" of word x,, A context of a word includes not only the history #1...-1, but also the specific vocabulary & that we calculated the probability on. Such a mechanism allowed us to parameterize the temperature vector 7 using a deep neural network and adapted the softness of the Softmax layer.
Our temperature vector r E RK was generated from the mapping function f as discussed before. Although can be any sequential models such as RNN or LSTM, we chose parameterize it by the AWD-LSTM (Merity et al., 2018). We omitted the details of its architecture due to the limited space in this paper. The output of AWD-LSTM was a vector with dimension D. We multiplied this vector by two matrices Wr, E IR D and Wi, E RROK Please note that one can potentially use a single matrix to represent these two. However, doing so can significantly increase the number of parameters and thus in practice factorized them into two. Finally, scaled temperatures using a Softmax function over the dimension of the vocabulary and its range was bounded in a 1+a J.
[ 0
T [[(6 W., W²2) +
(5)
3

---

Under review as a conference paper at ICLR 2020
3.2 CONTEXTUAL TEMPERATURE MOS
We then used the Contextual Temperature Mixture of Softmaxes architecture for language modeling (CT-MoS). The CT-MoS model extends Equation adding contextual temperature Equation Here @ represents the element- wise division between 2m and the temperature vector The results after division are then sent to the Mixture Softmaxes model. Since new parameters are added to the model, the CT-MoS now has total of parameters @
1 Wm U U Wr, U Wr..
M Σ m
PoT-Mos(1: 21.. 0)
o ozm r))
(6)
One thing worth noticing is that comparing to prior works, the proposed contextual temperature model has the ability adopt different temperature for (a) different vocabulary the prediction, (b) different position of the same vocabulary given the history and (c) tunable parameter that is capable of changing as the training progresses. The detailed architecture illustrated in Figure which highlights the difference between the proposed CT-MoS model and the MoS model.
x
Softmax
Softmax
CT-Mo8
CT-MoS
~..00
aeu prior
FC Layer; K
4' logits
AWD-LSTM
AWD-
FC Layer, 0
Emhedding Lookup
Embedding Lookeup
AWD-LSTM
X
Embedding Lookup
Figure 2:
The architecture of the proposed CT-MoS model. Black components are those the same the MoS model, while the blue ones are the newly added ones in approach.
3.3 TRAINING OBJECTIVES AND LOSS SCALING
We adopt the same regularization techniques MoS (Yang et al., 2018) and AWD-LSTM (Merity) et al. (2088). Our loss function thus consists of four terms: Cross Entropy (H).. Activation Regu- larization (AR), Temporal Activation Regularization (TAR), and Weight Decay (WD). AR used to penalize high values of outputs, and TAR is used to prevent outputs from changing too much between timesteps. WD prevents the model from overfitting. Here 71, 72 and 73 are constants to scale regularization terms. And m is the dropout mask (See in Merity et al. (2018)).
Σ T K
((0)
H(f.y) +m Ta(m @ f(11.111;0) +72 I=( f=1..):) +79 T2((O) (7)
LS
The uniqueness in our setting is the Loss Scaling term (LS). The use of temperature scaling makes gradients of H disproportional to that the other three terms, so the scale of needs to adjusted accordingly. A similar phenomenon is reported in (Hinton et al. 2014). In our case, only gradients of H will be influenced by temperatures. As for the other regularization terms, their gradients are not effected since they do regularization on parameters before temperature scaling, This difference leads to the unbalance between four objectives. Therefore, we scale H. in order to redress the balance. We empirically find that scaling H with the average of temperatures works well in our setting.
4

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
4.1 DATASETS
We evaluate contextual temperature on two widely-used datasets for language modeling: Penn Tree- bank (PTB) (Marcus et al. 1993] Mikolov et al. 201T) and WikiText-2 (WT2) (Merity et al.) 2017) The PTB dataset contains 929k training. 73k validation and 82k test tokens. The vocabulary size capped at 10,000 most frequent unique tokens, while the rest of tokens are replaced with the <unk>>
token. We follow common practices to pre-process the dataset (Mikolov et al., 2011): (a) words are lower-cased, (b) numbers are replaced with "N", (c) newlines are replaced with <eos>> and (d) punctuation removed.
WikiText-2 is derived from Wikipedia articles and released as popular option to train language models. WT2 contains 2M training tokens and a vocabulary of around 33k tokens. Compared to PTB, WT2 is roughly two times larger in sample size and three times larger in vocabulary size.
4.2 EXPERIMENTAL SETUPS
We conduct experiments on PTB and WT2 using one and four 1080 Ti GPUs, respectively. The environment we use PyTorch (Paszke et al.] 2017). We follow the training configurations as reported in the MoS paper and their githut For both PTB and WT2, we use the same number of parameters as MoS. We use three layers of LSTM with embedding sizes of 860-960-620 in PTB experiments, that is, the number of embedding for functional mapping is D 620. The embedding size here is Q = 280. For WT2, we use three layers LSTM with embedding sizes of 1150-1 150- 650. In both experiments, 71, 72, 73 for the regularization terms are 2.0, 1.0 and 1.2e respectively. The number of Softmaxes to be mixed is M = 15. Furthermore, we perform normalization as shown in Equation to ensure that temperatures have positive values. We have tried several different values for a and B, and find (a, B) ) works best in all experiments.
We use MoS as our baseline model, and AWD-LSTM as additional baseline for comparison. Since MoS is the state-of-the -fahe-of-the-art model on PTB and WT2 datasets for language modeling, attention based models, such Transformer-XL (Dai et al. 2019) and GPT-2 (Radford al.) (019). not used here to be the baseline. However, the performances of CT-MoS, MoS, Transformer-XI. and GPT-2 is listed in Appendix
4.3 RESULTS
First, we show experimental results on the PTB dataset in Table The original MoS model (Yang jet al., 2018) has a model size of 22M parameters. To make a fair comparison, we augment the number of parameters of the MoS model to have 24M parameters and name it Mos This is done by increasing the size of word embedding from 280 to 410. We show that our CT-MoS model outperforms AWD-LSTM, MoS and MoS models both validation and test sets with and without fine-tuning (Merity et al. 2018) and dynamic evaluation (Krause et al. 2017). Our best model achieves 48. 12 perplexity MoS model with a significant margin. Table provides results for WT2, a much larger language modeling dataset. We see similar pattern the PTB dataset that CT-MoS outperforms the state- of-the-art models. When using dynamic evaluation, our model also achieves comparable results to MoS and AWD-LSTM.
the validation set and 47. 47.42 the test set, beating the state-of-the-me
4.4 ABLATION STUDIES
Fixed Model Size Comparison. As mentioned in the previous section, we increase the number of parameters of the MoS model and name it MoS+ to provide a fair comparison under the same number of parameters. To construct MoS+, we increase the embedding size from 280 to 410. In Table we notice that MoS+ has a higher perplexity compared to MoS and CT-MoS, indicating that directly increasing model parameters cannot improve the performance in this case. Similar observation and results are also reported by (Yang et al. 2018). This ablation study shows that the improvements brought by CT-MoS more than the mere grow of parameters.
httts::/ggitheee comp hihaggaai
/ /mos
5

---

Under review as a conference paper at ICLR 2020
Table 1: Performance Comparison the Penn Treebank (PTB) dataset
Model
#Param
Validation
Test
AWD-LSTM w/o finetune
24M
60.7
58.8
AWD-LSTM
24M
60.0
57.3
AWD-LSTM dynamic evaluation
24M
51.6
51.1
MoS w/o finetune
22M
58.08
55.97
MoS
22M
56.54
54,44
MoS dynamic evaluation
22M
48.33
47.69
MoS w/o finetune
24M
59.72
57,43
MoS+
24M
58.54
56,36
MoS dynamic evaluation
24M
50.49
49.81
CT-MoS w/o finetune
24M
56.95
54.69
CT-MoS
24M
55.31
53.2
CT-Mos + dynamic evaluation
24M
48.12
47.42
Table Perfomrance Comparison on the WikiText-2 (WT2) dataset
Model
#Param
Validation
Test
AWD-LSTM w/o finetune
33M
69.1
66.0
AWD-I.STM
33M
68.6
65.8
AWD-ISTMM dynamic evaluation
33M
46.4
44.3
MoS w/o finetune
35M
66.01
63,33
MoS
35M
63.88
61,45
MoS dynamic evaluation
35M
42.41
40.68
CT-MoS w/o finetune
45M
65.25
62.21
CT-MoS
45M
62.89
60.13
CT-MoS + dynamic evaluation
45M
42.88
40.96
Temperature Normalization Methods. addition to using the Softmax function in Equation to normalize the temperature between [
14-0 J. consider several alternative normalization meth
ods: (a) T'nnhau), provided by (Lin et al. 2018). making the range of the temperature to be (+) A), (b) Tanh(w)+ + A, which generates range of 1, 1)1,, and (c) out method (a(n) + / where f(i..(111 ()" W W,, is defined for simplicity. Results are shown in Table 3 For the con- ciseness, the three methods evaluated on the PTB dataset without either fine-tuming or dynamic evaluation. The experimental results show that our proposed method outperforms other temperature normalization methods.
Table 3: Different methods for temperature normalization on the PTB dataset
Model
hyper- parametr
range
#Param
Validation
Test
Tanh(()) (Lin al. 2018) Tanh(u)+ +
A
[1/4, 4]
24M
65.11
62.21
=
[2,4]
24M
61.35
58.89
(a(1) + a
a= 1=10=00
[2.4]
24M
56.95
54.69
Effects of Loss Scaling. As discussed in Section 3.3 the loss value may need to be scaled since applying the temperature may lead to smaller gradients. Here, compare the results of whether applying loss scaling or not, the results shown in Table Note that using loss scaling leads to better results, i.e. lower perplexity.

---

Under review as a conference paper at ICLR 2020
Table 4: Perplexity PTB w/o finetune
Model
#Param
Validation
Test
CT-MoS w/o loss scaling
24M
$7.13 56.95
55.28
CT-MoS
24M
54.69
4.5 ANALYSIS
Temperature Change During Training. Going back to Figure [(a] in the Introduction section, have shown that contextual temperature capable of generating changing temperature as the training proceeds on a per vocabulary basis. This allows each vocabulary to have a flexible tempera- ture schedule that is optimal the model. Tokens that have higher temperatures are common words that don't I convey much information. This suggests that the model is more confident about these words. Our method has the advantage of determining the schedule automatically, which difficult to achieve in the traditionally fixed temperature scheduling method.
Average Temperature Over Word Positions. We also want to dive deeper into Figure [(b) which shows the mean and confidence interval of the temperature over different positions. We group sen- tences whose length ranges from 15 to 25. For sentences whose length are greater than 15, we pick first 5 tokens, middle 5 tokens, and last 5 tokens to form new sentence. At the beginning of a sentence, temperatures are usually high to smooth probabilities of the Softmax. This is the region where the model has little confidence since there is little information in the history. As the history builds up over time. the model becomes more confident and the temperature begins to cool down, making the probability distribution spiky and the model more confident. The confidence intervals of the temperature also become tighter as the length of the history increases.
Word Frequencies And Temperature Change. Evidence from Figure I(a) suggests that only a small fraction of tokens have significantly larger temperatures. However, even small changes might deliver large impacts to the performance due to the facts that these tokens might be used more often than others. To further analyze, we present analysis the weighted temperature in Figure ((a) that is, the temperature multiplied by its frequency. Here we see that for the majority words, small changes in temperatures might have great effects many of them occur fairly frequently.
244
1.2
tomperature
0.8
0.4
0.0
2000
4000
6300
8000
10000
Figure 3: (a) The weighted temperature over the token index ranked by frequency. Here the weighted temperature refers to the absolute temperature multiplied by the frequency of the word in the corpus.
Case Studies. We conduct several case studies on PTB to visualize the effect of contextual tem- perature. Table Sihows shows the comparison of CT-MoS v.s. MoS. We highlight two differences between CT-MoS and MoS. annotated using red and blue colors. In the first spot, we see that both CT-MoS and MoS fail predict the correct answer "single-a-1", which refers to rating for securities. The

---

Under review as a conference paper at ICLR 2020
CT-MoS model predicts "triple-a", which is not the same the reference but is much closer to the answer, since "triple-a" refers to the highest rating for securities. MoS, on the other hand, predicts <unk>> which deviates too much from the ground truth. Taking close look at the temperature, the word "triple-a" has a temperature of 2 + 8.34 x 10-6, which is a bit smaller than that of the <unk>> whose temperature is 2 8.6 x 10-5. This contributes to the factor that the model chooses "triple-a" over <unk>> Another example is illustrated by the prediction of the word "standard". Here the temperature of "standard" is smaller than that of "s&p", making the model more likely to predict the prior word. We refer readers to Appendix B for more samples.
Table Analysis of Model Performance on Sample from the PTB Dataset
rated single-a-1 by moody 's investors service inc. and single-a by standard & poor 's corp....
Reference
rated triple-a by moody and service inc, and <unk>> by standard & poor 's corp.
CT-MoS
rated <unk> by moody 's and service inc. and <unk>> by s&p & poor corp.
MoS
CT-MoS top-4
triple-a 0.34
single-a-2 0.2
single-a-1 0.15
single-a-3 0.11
MoS top-4
<unk>> 0.28
triple-a 0.27
single-a-2 0.24
single-a-1 0.1
7(le 2)
triple-a 8.34
<unk>> 8.60
CT-MoS top-4
standard 0.53
s&p 0.22
moody 0.17
dow 0.02
MoS top-4
s&p 0.4
moody 0.23
standard 0.19
<unk>> 0.03
7(ee + 2)
standard 11.2
s&p 11.4
Another indicator of how contextual temperature works is to look at the change of the tempera- ture across different positions in sentence. In Table 6 visualize the occurrence of the word "mortage" and its temperature. Here we see that as the position changes, contextual temperature chooses different value for each of the position, adjusting its confidence of model's belief. As analyze before, a general trend is that words appearing early in a sentence get larger temperatures while those approaching the end of the sentence get smaller temperatures.
Table 6: Analysis of Temperature for Same Words at Different Positions
loan mortgage(1) corp freddie mac posted posted yields on 30-year mort- gage(2) commitments for delivery within N days <eos>> N N standard conven- tional fixed-rate mortgages(3) N N N rate rate capped one-ycar adjustable rate mortgages(4) 4) <cos>> source telerate systems inc <<cos>> federal national mort- gage(5) association fannic mac posted posted yields on N year mortgage(6) commitments for delivery within N days priced at par N N N standard conven- tional fixed-rate mortgages(7) N
CT-MoS
5(le-5 +2) (1) 18.9 (2) 19.3
(3) 20.1
(4) 19.2
(5) 18.9
(6) 19.2
(7) 18.2
CONCLUSION AND FUTURE WORK
In this paper, we have proposed contextual temperature, a generalized and effective approach able to assigning individual optimal temperature of each class, by changing temperature based on the history of the context. Contextual temperature parameterized using deep neural network, and generates a unique schedule for each vocabulary to compute the corresponding optimal temperature. Experiments on the language modeling datasets achieve significantly better performances. In the future, our work opens up potential new research directions along the line of fully automated tem- perature mechanism to explore the implementation of contextual temperature various NLP tasks such summarization, machine translation, and dialogue generation.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint urXiv:1409.0477 2014.
Yoshua Bengio, Réjean Ducharme. Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 137- 2003.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. In NIPS Workshop, 2018
Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong. Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In CVPR, 2019.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In ACL, 2019.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural netowrks. ICML, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. In NIPS, 2014.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780) 1997.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Toward con- trolled generation of text. In ICML, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztiion with gumbel-softmax. In ICLR, 2017.
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. arXiv preprint arXiv:/709.0r0 07432, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Junyang Lin, Xu Sun, Xuancheng Ren, Muyu Li, and Qi Su. Learning when to concentrate or divert attention: Self-adaptive attention temperature for neural machine translation. In EMNLP, 2018.
Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, Graham Neubig, and Eduard Hovy. Softmax q-
distribution estimation for structured prediction: A theoretical interpretation for raml. arXiv preprint arXiv:1705.07136 2017.
Mitchell P Mareus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. In Computational linguistics, 1993.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm lan- guage models. In ICLR, 2018.
Tomáx Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
Tomáš Mikolov, Anoop Deoras, Stefan Kombrink, Lukáš Burget, and Jan Černocký. Empirical eval-
uation and combination of advanced language modeling techniques. In INTERSPEECH, 2011.

---

Under review as a conference paper at ICLR 2020
Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural structured prediction. In NIPS, 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In ICLR, 2018.
Xu Zhang, Felix Ximnan Yu, Svebor Karaman, Wei Zhang, and Shih-Fu Chang. Heated-up softmax embedding. arXiv preprint 1atp:.88890011577 2018.
A EXPERIMENT RESULTS OF CT-MoS, MoS, TRANSFORMER- XL AND GPT-2
Below shows the experiment results of CT-MoS, MoS, Transformer-XI and GPT-2 on the Penn Treebank dataset. Lower perplexity represents better performance. Please note that the GPT model works on a very different setting than the ones found in the mainstream language model research. In the GPT paper, it utilizes a large dataset that is collected outside the domain of the language modeling.
Table 7: Performance Comparison on the PTB dataset
Model
#Param
Validation
Test
CT-MoS w/o finetune
24M
56.95
54.69
CT-MoS
24M
55.31
53.2
CT-MoS dynamic evaluation
24M
48.12
47.42
MoS w/o finetune
22M
58.08
55.97
MoS
22M
56.54
$4.44
MoS dynamic evaluation
22M
48.33
47.69
Tansformer-Xi XL
24M
56.72
54.52
GPT-2
117M
65.85
GPT-2
345M
47.33
GPT-2
762M
40.31
GPT-2
1542M
35.76
B MORE SAMPLES FROM PTB
10

---

Under review as a conference paper at ICLR 2020
Table 8: More samples from PTB.
these rate indications are n't directly comparable lending practices vary widely by location <cos>> treasury bills <eos>> results of the tuesday ...
Reference
these rate indications are n't directly comparable lending practices vary widely by location <cos>> treasury bills results results of the monday ...
CT-MoS
these rate indications n't directly comparable lending practices vary widely by location <<os> treasury bills results treasury of the monday ...
MoS
CT-MoS top-4
results 0.81
treasury 0.07
0.01
bonds 0.01
MoS top-4
treasury 0.64
results 0.09
0.04
N 0.02
7(le 6 + 2)
results 8.11
treasury 8.58
corporate loans large u.s. money center commercial banks <eos>> federal funds <eos>> NN N
Reference
corporate loans at large u.s. money center commercial banks <cos>> federal funds N N N
CT-MoS
corporate loans at large u.s. money center commercial banks cos>> federal funds N N high
MoS
CT-MoS top-4
0,45
high 0.41
<eos>> 0.05
low 0.04
MoS top-4
high 0.46
N 0.41
<cos>> 0.06
and 0.02
7(le** + 2)
N 1.56
high 2.01
share compared with net loss of $ N million last year after loss from discontinued operations
Reference
share <cos>> with $ loss loss of N million year <coo>> the loss of the operations....
CT-MoS
ssare <cos>> with $ loss of N million year <cos>> the of the operations
MoS
CT-MoS top-4
loss 0.24
0.20
N 0.06
<unk> 0.06
MoS top-4
0.11
loss 0.09
one-time 0.07
N 0.07
7(1e +
loss 1.40
1.62
Reference
the nine months <unk>> net rose N N to N million ...
CT-MoS
the the nine months the said net income to N million...
MoS
the the first months the said net income N N N million
CT-MoS top-4
nine 0.17
third 0.10
ycar-ago 0.09
carricrrlier 0.09
MoS top-4
first 0.14
nine 0.12
third 0.12
year-ago 0.09
7(e-- 5
nine 7.37
first 7.94
11


Reviews:
[{"review_id": "rJlIaog6YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a learned and context dependent way to calculate the temperatures for the softmaxes. More specifically, a low-rank affine-transformation, taking the hidden state at the current step as input, is used to calculate scalar weighting for every token in the vocabulary. The method is very general, and can be used in combination with other techniques in tasks such as language modeling and text generation. Experiments on language modeling with Penn TreeBank and WikiText-2 show that the proposed method yields strong performance.\n\nOverall I found the paper well-motivated and easy to follow. The empirical results are solid and strong. The analysis is also interesting. I vote for an acceptance, if the authors can polish the writing.\n\nDetails:\n\n- Eq. 5. The temperature scalar for each token competes with each other, since they are calculated with a softmax (and then rescaled). Another way is to use, e.g., a sigmoid function. Can the authors explain the motivation behind the use of softmax?\n\n- Another view of the proposed method is that it learns a context-dependent weighting of the tokens in the vocabulary, such that \"important\" tokens (those with smaller \\tau) receive more gradient updates. Can the authors comment on this? Also, I don't see the thermodynamics connection and find calling the proposed method `temperature` a bit misleading. \n\n- Adding onto above. [1] discusses the low-rank bottleneck of using a single softmax. Since elementwise matrix product can blow up the rank, how do the authors think the proposed method can serve as a more efficient way to deal with the softmax bottleneck?\n\n- Last but not least, the paper can be improved a lot if the authors can thoroughly polish the writing.\n\n\n[1] Breaking the Softmax Bottleneck: A High-Rank RNN Language Model. https://arxiv.org/abs/1711.03953", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Ske6l8RJjS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a strategy to automatically adjust the temperature scaling based on the context of words in a sentence for NLP. Experiments demonstrate that this approach can significantly improve perplexity scores on several datasets popular for NLP.\n\nNLP is not an area of research I'm very familiar with so this review is limited to my understanding of temperature scaling as a general technique to improve learning. As described in the paper, temperature scaling is a type of hyper-parameter estimation that adjusts the sensitivity of the softmax function as training evolves. The paper proposes to learn a function that given context, adjust the temperature automatically. This can be seen as a meta-learning method. \n\nI believe this can be a useful technique but before considering such an approach as a general strategy, more theoretical insights should be provided. The authors report on ablation studies that demonstrate some empirical benefits. However, until I see more theoretical analysis on how the method improves convergence or lead to better losses by smoothing out the output of the objective function, I remain skeptical of the usefulness of this as a general training method.  ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1em6VUatr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a contextual temperature scaling to improve language modeling. The temperature model is parameterized using a deep neural network. Experiments on the language modeling datasets show some effects of the method. \n\nThe idea of dynamic temperature scaling has been tried in other works and tasks (e.g., attended temperature scaling). The paper parameterizes this mechanism with DNNs for the language model.  Though the idea looks interesting, it fails to explain why the scaling is better than other dynamic temperature scaling frameworks. \n\nThe experiments are not solid. The baseline only includes Mos, which is not very strong. To validate whether this approach works with other LM of high-order attention or self-attention, a better baseline model is required (e.g., transformer, GPT). I would like to see this technique can help either NLU or NLG tasks, instead of just pure modeling. The case analysis section needs more examples instead of just cherry-picking few. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: With an average post author response score of 4 - two weak rejects and one weak accept, it is just not possible for the AC to recommend acceptance. The author response was not able to shift the scores and general opinions of the reviewers and the reviewers have outlined their reasoning why their final scores remain unchanged during the discussion period.
Decision: Reject