Year: 2019

Abstract:
This paper presents a generalized approach to provide an individual optimal temperature trajectory to each class. Temperature scaling is an effective technique to control the smoothness of the distribution. Current implementations of temperature scaling assume either a fixed or manually-crafted dynamically changing schedule. This schedule is used to identify the shared optimal trajectory, which is applied to every class. However, our studies indicate that under the change of context, we can identify the corresponding individual optimal trajectory for each class. To this end, we propose contextual temperature, a mechanism that allows the temperature to be learned during training along with the remaining model parameters, over the context for each vocabulary. Experimental results confirm that contextual temperature significantly improves state-of-the-art language models. The proposed method achieves a perplexity of 55.31 and 62.89 on the test set of Penn Treebank and WikiText-2, respectively. Additionally, in-depth analyses indicate that (a) every vocabulary possesses its individual schedule of temperature, and (b) due to the increased context from every time step, the optimal trajectory drops in order to suppress the uncertainties.

heading: 1 INTRODUCTION
text: Due to the discrete nature of human language, the Softmax layer is mandatory to convert learned representations into a sequence of linguistic vocabulary. Temperature scaling is often used along with the Softmax layer in natural language processing. Generally, the temperature is applied as a denominator to output logits of the Softmax layer (Krizhevsky et al., 2012; Bahdanau et al., 2014; Hu et al., 2017; Caccia et al., 2018). The scale of temperature controls the smoothness of the output distribution. That is, given a temperature τ , when τ → ∞, the distribution becomes more uniform, thus increasing the uncertainty. Contrarily, when τ → 0, the distribution collapses to a point mass. Although temperature scaling has been justified to achieve great success, existing implementations are limited. For instance, the temperature is assumed to be constant throughout training (Norouzi et al., 2016; Ma et al., 2017; Chen et al., 2019), or to be a fixed schedule (Hu et al., 2017). Most importantly, none of the existing works studies the effects on different word tokens when the temperature changes. In reality, however, the temperature of each token can be dramatically different. Figure 1(a) shows the optimized temperature for each word token during the course of training. As shown in the figure, some certain words have a distinct heating-up temperature scaling, while the majority of words have a scaling that gradually cooling down the temperature. We argue that existing methods limit themselves to some fixed schedules, and thus have great difficulty to generalize.
In addition, another example can be observed in Figure 1(b), which indicates that the average temperature drops, as the length of the context increases. This suggests that the temperature mechanism helps promote stochasticity in the beginning of a sentence, then gradually suppressing the uncertainty until the end. All of these suggest the use of a more generalized temperature mechanism with the advantage of being able to deal with these phenomena.
We propose contextual temperature, a generalized temperature scaling method which takes both the history of a sequence and a particular vocabulary into consideration. Through optimizing the use of temperature scaling by the change of contexts, contextual temperature has the ability to generate a unique temperature for each vocabulary. As we parameterize contextual temperature using a deep neural network, its parameters co-evolve with the rest of model parameters, making the temperature
adapt to the training procedure. Experiment results on language modeling demonstrate significant improvements on both Penn Treebank and WikiText-2 datasets, with and without the fine-tune setting. Consistent improvements are shown on both validation and testing splits. In addition, we conduct comprehensive analyses and ablation studies to confirm the improvements of contextual temperature. We observe that our method is capable of controlling the uncertainties as the patterns of contexts change, allowing language models to achieve much better performances.
To the best of our knowledge, this is the first systematic work that studies the role of per-token temperature changing over context for training language models. The experimental results suggest a new way of training sequential models with discrete outputs, that is, using parameterized temperatures. We have also established the link between the control of model uncertainty and the use of temperatures, paying the way for extensions on tasks that require such long-term control, for instance summarization, translation, and dialogue generation.

heading: 2 RELATED WORK

heading: 2.1 LANGUAGE MODELING
text: Given a sequence of history tokens x1..t−1, language modeling aims at predicting the next token xt in a sentence. In other words, the goal is to model the probability of choosing the kth word, P (xt = k|x1..t−1), by using an encoding function f and a Softmax layer σ. f is parameterized by θ to compress the history z = f(x1..t−1; θ) T ·W .
P (xt = k|x1..t−1; θ) = σ(z)k (1)
Here W ∈ Rd×K is a matrix that converts a d dimensional output of f into a K dimensional vector. The state-of-the-art language models adopt deep neural networks as the encoding function, including the earlier feed-forward networks (Bengio et al., 2003), more recent recurrent-based ones (Mikolov et al., 2010; Hochreiter & Schmidhuber, 1997) and attention-based models (Devlin et al., 2018).

heading: 2.2 SOFTMAX LAYER
text: A Softmax layer σ(z) normalizes a K dimension, real-valued vector z to make it sum to 1.
σ(z)i = zi∑K j e zj (2)
Recent progress in language modeling suggests that Mixture of Softmaxes (MoS) (Yang et al., 2018) would significantly improve the performance by computing multiple Softmax distributions and use a set of weights to sum them up as the final probability distribution. To achieve this, a set of M matrices Wm is applied to the output generated by f . That is, zm = f(x1..t−1; θ)
T ·Wm, where Wm ∈ Rd×K has the dimension of the embedding d and the dimension of the vocabulary K. The probability distribution of the next word xt under the MoS model is thus a mixture of M Softmaxes weighted by π. Here Θ = ∪Mm=1Wm ∪ θ.
PMoS(xt = k|x1..t−1; Θ) = M∑ m πm · σ(zm)k (3)

heading: 2.3 TEMPERATURE SCALING
text: Temperature scaling is an approach used to control the smoothness of the distribution. Instead of applying the Softmax function as suggested in Equation 1, logits here are divided by the temperature τ before passing through the Softmax layer.
P (xt = k|x1..t−1; θ, τ) = σ(z/τ)k (4)
Constant Temperature. Earlier works that adopt the constant temperature can be traced back to Model Distillation (Hinton et al., 2014). Several works have been taking advantage of a fixed temperature during training (Norouzi et al., 2016; Ma et al., 2017; Chen et al., 2019). For instance, (Norouzi et al., 2016; Ma et al., 2017) optimize log-likelihood on augmented outputs, which are sampled proportionally to their exponential scaled rewards, where the temperature controls the degree of augmentation. Other works incorporate the temperature to enhance calibration during inference (Guo et al., 2017), or to achieve a great trade-off between quality and diversity (Caccia et al., 2018).
Dynamic Temperature Over Training Iterations. A wide range of works adopt a manuallycrafted schedule to tune the temperature during training. Notably in (Hu et al., 2017), a new text generation architecture is introduced that combines VAE and a discriminator. Since text samples are discrete and non-differentiable, a continuous approximation based on the Softmax with a decreasing temperature is used to enable gradient propagation from the discriminator to the generator. Similar techniques are adopted in the gumbel-softmax trick (Jang et al., 2017), which also allows gradients to pass through discrete sampling objectives. Studies also show that temperature with a heating up schedule makes the embedding vectors more compact (Zhang et al., 2018).
Dynamic Temperature Over Word Position. Another work that is closely related to our work, is the adaptive temperature over an attention model (Lin et al., 2018). We note that contextual temperature further learns the temperature for each vocabulary in the output distribution.

heading: 3 METHODS

heading: 3.1 CONTEXTUAL TEMPERATURE
text: Contextual temperature is a mechanism that chooses the optimal temperature by considering the “context” of a word xt. A context of a word includes not only the history x1..t−1, but also the specific vocabulary k that we calculated the probability on. Such a mechanism allowed us to parameterize the temperature vector τ using a deep neural network and adapted the softness of the Softmax layer.
Our temperature vector τ ∈ RK was generated from the mapping function f as discussed before. Although f can be any sequential models such as RNN or LSTM, we chose to parameterize it by the AWD-LSTM (Merity et al., 2018). We omitted the details of its architecture due to the limited space in this paper. The output of AWD-LSTM was a vector with dimension D. We multiplied this vector by two matrices Wτ1 ∈ RD×Q and Wτ2 ∈ RQ×K . Please note that one can potentially use a single matrix to represent these two. However, doing so can significantly increase the number of parameters and thus in practice we factorized them into two. Finally, we scaled temperatures using a Softmax function over the dimension of the vocabulary and its range was bounded in [αβ , 1+α β ].
τ = σ(f(x1..t−1; θ) T ·Wτ1 ·Wτ2) + α β
(5)

heading: 3.2 CONTEXTUAL TEMPERATURE MOS
text: We then used the Contextual Temperature Mixture of Softmaxes architecture for language modeling (CT-MoS). The CT-MoS model extends Equation 3 by adding contextual temperature in Equation 5. Here represents the element-wise division between zm and the temperature vector τ . The results after division are then sent to the Mixture of Softmaxes model. Since new parameters are added to the model, the CT-MoS now has a total of parameters Θ = ∪Mm=1Wm ∪ θ ∪Wτ1 ∪Wτ2 .
PCT−MoS(xt = k|x1..t−1; Θ) = M∑ m πm · σ(zm τ)) (6)
One thing worth noticing is that comparing to prior works, the proposed contextual temperature model has the ability to adopt a different temperature for (a) different vocabulary in the prediction, (b) different position of the same vocabulary given the history and (c) a tunable parameter that is capable of changing as the training progresses. The detailed architecture is illustrated in Figure 2, which highlights the difference between the proposed CT-MoS model and the MoS model.

heading: 3.3 TRAINING OBJECTIVES AND LOSS SCALING
text: We adopt the same regularization techniques in MoS (Yang et al., 2018) and AWD-LSTM (Merity et al., 2018). Our loss function thus consists of four terms: Cross Entropy (H), Activation Regularization (AR), Temporal Activation Regularization (TAR), and Weight Decay (WD). AR is used to penalize high values of outputs, and TAR is used to prevent outputs from changing too much between timesteps. WD prevents the model from overfitting. Here γ1, γ2 and γ3 are constants to scale regularization terms. And m is the dropout mask (See in Merity et al. (2018)).
L(Θ) = ∑K i τi
K︸ ︷︷ ︸ LS H(ŷ,y) + γ1 L2(m f(x1..t−1; θ))︸ ︷︷ ︸ AR +γ2 L2(f(x1..t−1; θ)− f(x1..t; θ))︸ ︷︷ ︸ TAR +γ3 L2(Θ)︸ ︷︷ ︸ WD (7)
The uniqueness in our setting is the Loss Scaling term (LS). The use of temperature scaling makes gradients ofH disproportional to that of the other three terms, so the scale ofH needs to be adjusted accordingly. A similar phenomenon is reported in (Hinton et al., 2014). In our case, only gradients ofH will be influenced by temperatures. As for the other regularization terms, their gradients are not effected since they do regularization on parameters before temperature scaling. This difference leads to the unbalance between four objectives. Therefore, we scaleH in order to redress the balance. We empirically find that scalingH with the average of temperatures works well in our setting.

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS
text: We evaluate contextual temperature on two widely-used datasets for language modeling: Penn Treebank (PTB) (Marcus et al., 1993; Mikolov et al., 2011) and WikiText-2 (WT2) (Merity et al., 2017). The PTB dataset contains 929k training, 73k validation and 82k test tokens. The vocabulary size is capped at 10,000 most frequent unique tokens, while the rest of tokens are replaced with the<unk> token. We follow common practices to pre-process the dataset (Mikolov et al., 2011): (a) words are lower-cased, (b) numbers are replaced with “N”, (c) newlines are replaced with <eos> and (d) punctuation is removed.
WikiText-2 is derived from Wikipedia articles and released as a popular option to train language models. WT2 contains 2M training tokens and a vocabulary of around 33k tokens. Compared to PTB, WT2 is roughly two times larger in sample size and three times larger in vocabulary size.

heading: 4.2 EXPERIMENTAL SETUPS
text: We conduct experiments on PTB and WT2 using one and four 1080 Ti GPUs, respectively. The environment we use is PyTorch (Paszke et al., 2017). We follow the training configurations as reported in the MoS paper and their github1. For both PTB and WT2, we use the same number of parameters as MoS. We use three layers of LSTM with embedding sizes of 960-960-620 in PTB experiments, that is, the number of embedding for functional mapping is D = 620. The embedding size here is Q = 280. For WT2, we use three layers of LSTM with embedding sizes of 1150-1150- 650. In both experiments, γ1, γ2, γ3 for the regularization terms are 2.0, 1.0 and 1.2e−6, respectively. The number of Softmaxes to be mixed isM = 15. Furthermore, we perform normalization as shown in Equation 5 to ensure that temperatures have positive values. We have tried several different values for α and β, and find (α, β) = (1, 12 ) works best in all experiments.
We use MoS as our baseline model, and AWD-LSTM as an additional baseline for comparison. Since MoS is the state-of-the-art model on PTB and WT2 datasets for language modeling, attention based models, such as Transformer-XL (Dai et al., 2019) and GPT-2 (Radford et al., 2019), are not used here to be the baseline. However, the performances of CT-MoS, MoS, Transformer-XL and GPT-2 is listed in Appendix A.

heading: 4.3 RESULTS
text: First, we show experimental results on the PTB dataset in Table 1. The original MoS model (Yang et al., 2018) has a model size of 22M parameters. To make a fair comparison, we augment the number of parameters of the MoS model to have 24M parameters and name it Mos+. This is done by increasing the size of word embedding from 280 to 410. We show that our CT-MoS model outperforms AWD-LSTM, MoS and MoS+ models on both validation and test sets with and without fine-tuning (Merity et al., 2018) and dynamic evaluation (Krause et al., 2017). Our best model achieves 48.12 perplexity on the validation set and 47.42 on the test set, beating the state-of-the-art MoS model with a significant margin. Table 2 provides results for WT2, a much larger language modeling dataset. We see a similar pattern to the PTB dataset that CT-MoS outperforms the stateof-the-art models. When using dynamic evaluation, our model also achieves comparable results to MoS and AWD-LSTM.

heading: 4.4 ABLATION STUDIES
text: Fixed Model Size Comparison. As mentioned in the previous section, we increase the number of parameters of the MoS model and name it MoS+ to provide a fair comparison under the same number of parameters. To construct MoS+, we increase the embedding size from 280 to 410. In Table 1, we notice that MoS+ has a higher perplexity compared to MoS and CT-MoS, indicating that directly increasing model parameters cannot improve the performance in this case. Similar observation and results are also reported by (Yang et al., 2018). This ablation study shows that the improvements brought by CT-MoS are more than the mere grow of parameters.
1https://github.com/zihangdai/mos
Temperature Normalization Methods. In addition to using the Softmax function in Equation 5 to normalize the temperature between [αβ , 1+α β ], we consider several alternative normalization methods: (a) λTanh(µ), provided by (Lin et al., 2018), making the range of the temperature to be ( 1λ , λ), (b) Tanh(µ) + λ, which generates a range of (λ−1, λ+1), and (c) out method (σ(µ)+α)/β, where µ = f(x1..t−1; θ)
T ·Wτ1 ·Wτ2 is defined for simplicity. Results are shown in Table 3. For the conciseness, the three methods are evaluated on the PTB dataset without either fine-tuning or dynamic evaluation. The experimental results show that our proposed method outperforms other temperature normalization methods.
Effects of Loss Scaling. As discussed in Section 3.3, the loss value may need to be scaled since applying the temperature may lead to smaller gradients. Here, we compare the results of whether applying loss scaling or not, the results shown in Table 4. Note that using loss scaling leads to better results, i.e. lower perplexity.

heading: 4.5 ANALYSIS
text: Temperature Change During Training. Going back to Figure 1(a) in the Introduction section, we have shown that contextual temperature is capable of generating a changing temperature as the training proceeds on a per vocabulary basis. This allows each vocabulary to have a flexible temperature schedule that is optimal to the model. Tokens that have higher temperatures are common words that don’t convey much information. This suggests that the model is more confident about these words. Our method has the advantage of determining the schedule automatically, which is difficult to achieve in the traditionally fixed temperature scheduling method.
Average Temperature Over Word Positions. We also want to dive deeper into Figure 1(b), which shows the mean and confidence interval of the temperature over different positions. We group sentences whose length ranges from 15 to 25. For sentences whose length are greater than 15, we pick first 5 tokens, middle 5 tokens, and last 5 tokens to form a new sentence. At the beginning of a sentence, temperatures are usually high to smooth probabilities of the Softmax. This is the region where the model has little confidence since there is too little information in the history. As the history builds up over time, the model becomes more confident and the temperature begins to cool down, making the probability distribution spiky and the model more confident. The confidence intervals of the temperature also become tighter as the length of the history increases.
Word Frequencies And Temperature Change. Evidence from Figure 1(a) suggests that only a small fraction of tokens have significantly larger temperatures. However, even small changes might deliver large impacts to the performance due to the facts that these tokens might be used more often than others. To further analyze, we present an analysis on the weighted temperature in Figure 3(a), that is, the temperature is multiplied by its frequency. Here we see that for the majority of words, even small changes in temperatures might have great effects as many of them occur fairly frequently.
Case Studies. We conduct several case studies on PTB to visualize the effect of contextual temperature. Table 5 shows the comparison of CT-MoS v.s. MoS. We highlight two differences between CT-MoS and MoS, annotated using red and blue colors. In the first spot, we see that both CT-MoS and MoS fail to predict the correct answer “single-a-1”, which refers to a rating for securities. The
CT-MoS model predicts “triple-a”, which is not the same as the reference but is much closer to the answer, since “triple-a” refers to the highest rating for securities. MoS, on the other hand, predicts <unk>, which deviates too much from the ground truth. Taking a close look at the temperature, the word “triple-a” has a temperature of 2 + 8.34 × 10−5, which is a bit smaller than that of the <unk>, whose temperature is 2 + 8.6× 10−5. This contributes to the factor that the model chooses “triple-a” over <unk>. Another example is illustrated by the prediction of the word “standard”. Here the temperature of “standard” is smaller than that of “s&p”, making the model more likely to predict the prior word. We refer readers to Appendix B for more samples.
Another indicator of how contextual temperature works is to look at the change of the temperature across different positions in a sentence. In Table 6, we visualize the occurrence of the word “mortage” and its temperature. Here we see that as the position changes, contextual temperature chooses a different value for each of the position, adjusting its confidence of model’s belief. As we analyze before, a general trend is that words appearing early in a sentence get larger temperatures while those approaching the end of the sentence get smaller temperatures.

heading: 5 CONCLUSION AND FUTURE WORK
text: In this paper, we have proposed contextual temperature, a generalized and effective approach able to assigning individual optimal temperature of each class, by changing temperature based on the history of the context. Contextual temperature is parameterized using a deep neural network, and generates a unique schedule for each vocabulary to compute the corresponding optimal temperature. Experiments on the language modeling datasets achieve significantly better performances. In the future, our work opens up potential new research directions along the line of fully automated temperature mechanism to explore the implementation of contextual temperature in various NLP tasks such as summarization, machine translation, and dialogue generation.

heading: A EXPERIMENT RESULTS OF CT-MOS, MOS, TRANSFORMER-XL AND
text: GPT-2
Below shows the experiment results of CT-MoS, MoS, Transformer-XL and GPT-2 on the Penn Treebank dataset. Lower perplexity represents better performance. Please note that the GPT model works on a very different setting than the ones found in the mainstream language model research. In the GPT paper, it utilizes a large dataset that is collected outside the domain of the language modeling.
B MORE SAMPLES FROM PTB

Review summaries:
This work proposes a learned and context dependent way to calculate the temperatures for the softmaxes. More specifically, a low-rank affine-transformation, taking the hidden state at the current step as input, is used to calculate scalar weighting for every token in the vocabulary. The method is very general, and can be used in combination with other techniques in tasks such as language modeling and text generation. Experiments on language modeling with Penn TreeBank and WikiText-2 show that the proposed method yields strong performance.