Ruminating Word Representations with Random Noise Masking
Authors: Hwiyeol Jo
Byoung-Tak Zhang
Venue:  (2021)
Keywords: representation learning for natural language processing
pretrained word embeddings
iterative training method
model regularization

Abstract: We introduce a training method for better word representation and performance, which we call \textbf{GraVeR} (\textbf{Gra}dual \textbf{Ve}ctor \textbf{R}umination). The method is to gradually and iteratively add random noises and bias to word embeddings after training a model, and re-train the model from scratch but initialize with the noised word embeddings. Through the re-training process, some noises can be compensated and other noises can be utilized to learn better representations. As a result, we can get word representations further fine-tuned and specialized in the task. On six text classification tasks, our method improves model performances with a large gap. When GraVeR is combined with other regularization techniques, it shows further improvements. Lastly, we investigate the usefulness of GraVeR.

Under review as a conference paper at ICLR 2021
RUMINATING WORD REPRESENTATIONS WITH RANDOM NOISE MASKING
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce a training method for better word representation and performance, which we call GraVeR (Gradual Vector Rumination). The method is to gradually and iteratively add random noises and bias to word embeddings after training a model, and re-train the model from scratch but initialize with the noised word embeddings. Through the re-training process, some noises can be compensated and other noises can be utilized to learn better representations. As result, we get word representations further fine-tuned and specialized in the task. On six text classification tasks, our method improves model performances with large gap. When VeR is combined with other regularization techniques. shows further improvements. Lastly, investigate the usefulness of Gra Vek
INTRODUCTION
Most machine learning methodologies can be formulated to get computational representations from real-life objects (e.g.. images, languages, and sounds) and then get high-level representations using model architectures. Therefore, there have been two main approaches to improve model perfor- mances: (I) starting with better representations (Melamud et al. 2016 Petees et al., 2018), and (2) building more sophisticated architectures that can extract important features and generate higher- level representations (Vaswani et al. 2017 Conneau et al. (017)).
For better initial representations, many NLP researchers have used pretrained word vectors trained on substantially large corpus through unsupervised algorithms, such as word2vee (Mikolov et al., 2013a), GloVe (Pennington et al. 2014), and fastText (Bojanowski et al.] 2016). The pretrained word vectors represent the general meaning of words and increase the model performances most NLP tasks (Turian et al. 2010). In addition to the algorithms, word vector post-processing re- search (Faruqui et al. [201 Vulié et al. 2017 Mrk3ić al.) 2017] Jo & Choil 2018) have attempted to enrich the pretrained representations using external resources. They simply modified the values of the vector representations in some way, and showed improved performance. It implies that can get further improvement through better initial representations.
When training NLP models, we first initialize word representations with pretrained word vectors and then update both the model parameters and the word representations. However, in the training process, the model performance can be limited due the initial word vectors. For example, the pre- trained word representations have general meanings of words, but, in some tasks, the words might not be used for the general meaning. Although the gap between meanings can learned through the training process, it could fail. Since the pretrained representations are trained from a huge dataset, and their objective functions are based on language modeling, the word vectors are naturally biased to general and frequently used meaning. Besides, the word vectors are updated through gradient descent algorithms, so the values are changed slightly. The word vectors are thus easy to converge on local minima.
Therefore, our method starts with an idea-using the word representations fine-tuned by a training process as pretrained word vectors in the next re-training process. Then, word vectors can be trained to learn more appropriate representations to the task. However, the model must be overfitted, and then the word representation would be stuck in local minima. Thus, we add random noise and bias to the word representations before the re-training processes, in order to prevent the model from over- fitting and take the word representations far from the local minima.

---

Under review as a conference paper at ICLR 2021
In this paper, we propose simple training framework to find better representations by adding ran- dom noise and bias on the word vectors during iterative training processes, which we call Gra VeR (Gradual Vector Rumination). We expect that the model makes good uses of the re-training pro- cesses with noises, both for learning better representation and for model regularization.
2 RELATED WORKS
The representations fine-tuned by GraVeR can be considered as pretrained representations from the previous training process. Also, GraVeR utilizes word-level noises, which used for model regularization.
2.1 PRETRAINED REPRESENTATIONS
Pretrained Embedding Vector is also called pretrained word representation. According to the distributional representation hypothesis (Mikolov et al. 2013b), pretrained embedding vectors are composed of pairs of (token, n-dimensional float vector). Unsupervised algorithms (e.g., word2vec (Mikolov et al. (0) GloVe (Pennington et al. 2014). fastText (Bojanowski et al.) 2016)) learn the word vectors on substantial corpora to represent general meanings of words. The pretrained embedding vectors widely used to initialize the word vectors in models.
Pretrained Embedding Model is suggested to get deep representation of each word in the con- text. Previous research McCann et al. (2017): ; Peters et al. (2018); Devlin et al. (20T8) trained deep architecture models and then utilized the model weights to represent words by using the outputs of the models. Although recent advanced pretrained representations (Peters al.) 20188] Devlin al.) 2018) show good performances, we take the pretrained embedding vector approach because (1) re- training processes in the pretrained embedding models are very expensive and (2) we use word-level noises whereas the embedding models use token-level embeddings combined with position embed- dings.
2.2 WORD-LEVEL NOISES
Adding noises to input data is an old idea (Plaut et al. 1986). However, small number of studies were on word-level noises, since the noises on words can distort words' meaning. Word Dropping. NLP tasks that utilize the text each word as features. However, too many features can lead models to be overfitted to the training data due the curse of dimensionality. Therefore, the easiest way reduce the number of features is to drop words in the sentence at random.
form of sentence and phrase have considered
Word Embedding Perturbation. |Myytto et al. (2016) tried to perturb word vectors and used them in the adversarial training framework for model regularization. Cheng et al.) (2018) utilized the noises to build a robust machine translation model. Also, Zhang Yang (0118) considered the perturbation as data augmentation method. The previous works added the noises to all word em- beddings. It can regularize the model weights, but they ignored the change of word representations and its re-usability.
On the other hand, our method gradually adds noises to word embeddings by controlling the amount of noise. Also, iterative training processes that re-use fine-tuned word representation as pretrained word vectors for the next training process can benefit from the noises and make better word repre- sentations.
2.3 REGULARIZATION TECHNIQUES
Some research explained that the normalization could be used for model regularization
van
Laarhoven 2017 Luo et 2018 Hoffer et al., 2018).
Dropout (Srivastava et al. 2014) is applied to neural network models, masking random neurons with O. Dropout randomly and temporarily removes the neural activations during training, so the masked weights are not updated. As a result, the model is prevented from over-tuning on specific
features, which involves regularization.
Batch Normalization (BN) (loff & Szegedy 2015) normalizes the features according to mini- batch statistics. Batch normalization enables the features to avoid covariate shift-the weight gradi- ents are highly dependent the previous layers' gradients. Besides, batch normalization speeds up
2

---

Under review as a conference paper at ICLR 2021
High
Train Set
Validation Set
(?) Re-train Model initialized with
Freq.
(1)
(2) Validatioo
W
Miskiess
filled
+ Maskers
Model
Emb. Table (W]
Modal meters (9)
Low Freq.
Fine-tuned Embedding Table (W)
Maskers
Stopping
Extract
-Weighted
(4) Change MMuskkrr Accor dling to Valld. Performance
Figure 1: The flow of GraVeR. (1) We first train classifier using the training set, (2-3) do early- stopping using the validation set, (3) extract fine-tuned word embeddings, (4) change the number of random maskers according to the validation performance, (5) add weighted bias to fine-tuned embedding W', (6) add the maskers to portion W', and lastly, (7) re-train the classifier with from the very first step. We repeat the process until all the vocabularies are masked.
the training process by reshaping loss function. Layer Normalization (LN) (Ba et al. 2016) also utilizes mini-batch statistics to normalize the fea- tures. The difference with batch normalization is that layer normalization normalizes the inputs across the features. The statistics are computed across each feature, which is the same for all the feature dimensions.
PROPOSED METHOD
3.1
OVERALL PROCESS
The overall process of VeR is illustrated in Figure 1 GraVeR is applied to conventional train- ing framework, but needs a meta-level approach that trains the model again. The iterative training process will also be denoted as meta-epoch.
When a training process finishes, we extract the fine-tuned word embeddings (W") and add bias (W" - W) weighted by 1/ [Kaskinngatte]. The additive bias is optional but it takes the word embed- dings far from the minima in which the fine-tuned embeddings converged. Next, we add maskers filled with random values a portion of W", and then re-train the model from scratch with the noised word embeddings. Observing the validation performance in the training process, we select the best-performing model and the fine-tuned embeddings. The additional details Gra VeR will be described. In short, Gra VeR is training framework that adds random noises (and bias) to the fine-tuned word embeddings and then repeats the training process with the word embeddings.
3.2 GRAVER DETAILS
The details in GraVeR are based on the idea that moderate noises are required to change word vector distribution, but the noise scale should not be too large to distort the word vector distribution. Random Noises and Bias. We have two hyperparameters related to the noise maskers; Masking Rate and Noise Scale. The masking rate denotes a portion of word vectors masked in every (re- training process. We the masking are filled with random values in a range, which is denoted as noise scale. The default sampled from uniform distribution (-1,1) since word vectors in the range are widely used. Well-defined perturbation methods like Gaussian kemels (Vilnis & McC alllum 2014) can be an extension.
to 20% of the vocabulary size default. The maskers
Next, we add bias to the fine-tuned word embeddings (W') order to take W' the same em- bedding space with better initial values. Intuitively, since we initialize with the word embeddings fine-tuned the previous training process, the embedding space the next training process the same the space in the previous training process. And, the difference of fine-tuned embedding and initial embedding (IW' - -W) in the space means the representations learned by training pro- cess, which improved the performance in the previous training process. So, add the bias W" order to give large bias when a small
weighted by 1/ [MaskingRate]] where <MaskingRates! number of word vectors noised; 1/ [MMaskingRate]] (W' WW'-W) W). When the validation performance

---

Under review as a conference paper at ICLR 2021
decreases, the bias is [MaskingRate]] (W- - W") for small noise.
When selecting a portion of words to noise, we use word frequency driven from training data. Intu- itively, if the representations of frequently used words are changed, which means the input data changed a lot, the high-level representation such as sentence representation are largely affected. Gradualness. While following the aforementioned process, change the number of random maskers according to validation performance. If the re-trained model's validation performance in- creases, we move the maskers the next frequently used words. Otherwise, we gradually increase the number of random maskers without moving them so that the maskers make noises on the words again in the next re-training process. As result, GraVeR can process both words in the previous training process and words in the current training process. This gradualness lets the re-training pro- cesses benefit from newly generated noises again, which makes VeR dynamic. In the end, overall algorithms of VeR are summarized as follows:
Algorithm Training Framework with GraVeR
AcCual <- O, MaxACCool f 0
Train set (Train), Validation set (Val), A classifier (M), Word embeddings (W). Word frequency information counted from training data
-Train M with W
Geet trained M' and fine-tuned
Meta-level validation; Acc,al - M' (Vaaa W")
if MaxAcc,,l < Acc then
MaxAcc.a! c Acc,al
MaskedWords - NextFrequently UssterreeuedyyyysWWWWrrr
W < W + 1 1/MaskingRate (W" W)
else
MaskedWords + MaskedWords NextFrequenttyy W W MaskingRate x (W"- W)
end if
Words] < W[MaskedWordss + U)) Repeat the training process until all the words are masked
GraVeR can be applied to any conventional training frameworks, the method is independent of model architectures that most NLP models use word-level emheddings. The random noises might disturb the representation, but some of the noises which harm the performance during the re-training processes. In contrast, other noises are used to update the word vectors over the initial values. Besides, the additive bias makes the use of random noises stable.
compensated
By using this re-training with the noise process, we expect that the model is prevented from overfit- ting. Meanwhile, the model incrementally fitted to the validation through early-stopping in training process and meta-level carly-stopping in re-training processes. Therefore, the model keeps being fitted to the validation set with regularization, showing better performance on the test set since the model performance on the validation set is normally correlated to the performance on the test set.
4 EXPERIMENT
4.1 DATASETS
We prepare three topic classification datasets: DBpedia ontology (Lehmann et. 2015), YahooAn- swerf] swerf (Chang et al. 2008). AGNews. We also prepare two sentiment classification datasets: Yelp reviews (Zhang et al. 2015). IMDB (Maas et al. (011]. YahooAnswer dataset is used for two dif- ferent tasks: classify upper-level categories and classify lower-level categories. The data statistics are presented in Appendix. We assign 15% of each train set to validation set, and each dataset has its own test set. The validation set is used for early- storpigg both every epoch and every meta-epoch. We use all words tokenized by space and all symbols using 300 dimensional embedding space.
hnttss//cogcomppsea www.ssaw..............ee www.ssueeenenueaeeeeeeeeeee www.e//ae/eesourree.ooo -view/89
Note that Chang al, (2008) said the dataset has 20 top-level categories, but it has three duplicated top-level categories because of typos.
4

---

Under review as a conference paper at ICLR 2021
4.2 CLASSIFIER
We use TexICNN (Kim 2014) classífiers. The model consists of 2 convolutional layers with 32 channels and 16 channels, respectively. We also adopt multiple sizes of kernels-2, lowed by ReLU activation (Hahnloser et al. 20000) and max-pooling. The kernels are concatenated after every max-pooling layer. Finally, the features are classified into the classes using fully con- nected layer.
and fol-
Although this model has few parameters (136K) compared with recent high- perrrrreffrmmece models like BERT (Devlin et al.] 2018), we use this model to utilize multiple re-training processes. Also, we employ simple techniques for model improvements such as Dropout and Layer Normalization to get similar performances to recent models and justify the use of simple model basic classifier. The vanilla classifier and the tuned classifier are denoted as TextCNNease and TeXtCNN(une, respec- tively. Additional comparison with very deep models is described in Appendix.
We optimize the model using Adam (Kingma & Ba 2014) with 1e-3 learning rate and early- stopping. If the validation accuracy does not increase Initial word embeddings are random if we do not mention explicitly.
five epochs, we stop model training.
4.3 BASELINE IMPLEMENTATION
We cannot fully compare method with Word Embedding Perturbation (Miyato et al. 2016) be- cause we do not use adversarial training framework. Instead, random noises are added to all word embeddings, as other word embedding perturbation methods did (Cheng al. 2018 Zhang & Yang 2018).
In order to compare the effect of regularization, we implement five regularization (including normal- ization) methods. Word dropping is implemented in the pre-processing part, which removes random words in the text. We set the random probability p as 0.1. Dropout (Srivastava et al. 2014) is added to the final fully connected layer with dropout probability 0.1, which performs the best in our exper- iments. Batch Normalization (loffe & Szegedy 2015) located between every convolutional layer and activation function, used in the original paper. Layer Normalization (Ba al., (2066) is implemented in the same position. We report the performance averaged over five runs.
RESULTS
5.1
PERFORMANCE
Table 1: The performance of TexICNNaase classifiers and the standard deviation with GraVeR ap-
plied to various pretrained word vectors.
DBpedia
Yah(Up)
Yah(Low)
AGNews
Yelp
IMDB
Rand 9..0
67.412.07
44.69
88.87+.01
61.75+.33
86.42+.35
GraVeR
+0.51(.02)
+5.29(.17)
+4.76(.07)
+2.38(.39)
+1.34(.25)
+2.62(.27)
w/ w2v
98.20+.22
67.63+.31
43.18+.74
90.28+.84
62.26+.16
86.53+.36
GraVeR
+0.26(.02)
+4.92(.20)
+6.25(.75)
+0.99(.14)
+0.69(.47)
+0.433.37)
w/ glv
98.63+.04
72.97=.57
48.46+.41
91.88+.11
63.04+.18
88.564.76
GraVeR
+0.01(.03)
+0.18(.33)
+3.00(.52)
0.00(.11)
-0.03(.10)
+0.50(.58)
w/ ftt
97.80+.32
63.13+.54
38.62+.38
19.07+1.56
61.78+.18
75.98+3.32
GraVeR
+0.69(.02)
+9.01(.32)
(10.31(.87)
+2.21(.10)
+0.91(.31)
+9.71(.53)
Firstly, we present the effect of GraVeR on major pretrained word embeddings, as pre- sented in Table [ We use three pretrained word embeddings: word2vec (w2v) (Mikolov al.] (013a) GoogleNews- secnkeews-eetorrs www.veeetteeeeeee negtorrvengtt bin, GloVe (glv) (Pennington al. 2014) glove. 42B. 300d. txt, fastText (fttt) (Bojanowski et al. 2016) wiki-news 300d- 1M www-subordddcmm vec, and random embedding.
GraVeR improves the model performance most of the datasets, and the largest performance gain in random embeddings. Random embeddings with GraVeR even perform better than pretrained embeddings in few datasets. The result implies that we learn better word representations through GraVeR in any embedding space since we train a model from scratch except for word em- beddings. However, with GloVe, GraVeR is not effective the model performance because the
5

---

Under review as a conference paper at ICLR 2021
Table 2: The performance of TextCNNNaas classifiers and the standard deviation with different reg- ularization techniques. We can observe that GraVeR positively matches with the classifier and with other regularization techniques. Bold indicates the largest performance gain.
DBpedia
Yah(Up)
Yah(Low)
AGNews
Yelp
IMDB
TNNt
98.01
67.41..07
44.69+.15
88.87+.01
61.75=.33
86.42+.35
WordDrop
+0.05(.03)
+0.16(884
-0.40((35)
+0.33(.34)
+0.51(.16)
-0.40(.11)
+ DO (p=0.1)
-0.01(.08)
-0.41((44)
+0,10(.2)
+0.06(.33)
+0.43(.12)
-0.03(.34)
BN
-0.16(.02)
+..27(.31)
+0.03(.29)
-0.00.30)
+0.49(.10)
-0.89(.45)
+ LN
+0.17(.04)
-0.11(255
+0.47(68)
-0.23(.08)
+1.14(29)
-1.23(.38)
GraVeR
+0.51(.02)
+5.29(.17)
+4.76(00)
+2.38(.39)
+1.34(.25)
+2.62(.27)
DBpedia
Yah(Up)
Yah(Low)
AGNews
Yelp
IMDB
TextCNMaase
98.01+.03
67.41
44.694.15
88.87+.01
61.75+.33
86,424.35
GraVeR
+0,51(.02)
+5.29(17)
+4.76(.07)
+2.38(339
+1.34(225)
+2.62((27)
DO (p=0.1)
98.00+.08
67.01=.44
44.79#.42
88.93+.33
62.18=.12
86.39+.34
DO&GraVeR
+0.48(.03)
+5.33(221)
+5.11(.17)
+2.44(.13)
+1.14(.15)
+2.55(.09)
+ BN
97.86+.02
67.68=.31
44.72*.29
88.07
62.24..10
85.53
BN&GraVeR
+0.66(.01)
+5.00(44)
+5.17(.07
+2.71(.49)
+1.29(.27)
+2.18(.11)
+ LN
98.18+.04
27.10
45.15
88.644.08
62.89
85.182.38
LN&GraVeR
+0.430.03
+6.50(.16)
+6.51(.16
+2.39(.10)
+0.80(.10)
(2.99(.51)
DO&BN
97.91+.03
67.87=.04
45.10
88.324.27
62.51
84.894.36
DO&BN&GraVeR
+0.58(00)
+4.49(.1)
+4.53(.06
+2.71(.12)
+0.75(.21)
+2.71(.46)
DO8LM(TextCCN....)
98,494.06
69.26a.31
46.854.18
90.064.35
63.241.34
86.11+1.17
DO&LN&GraVeR
+0.24(.07)
+5.83(226
+5.95(.38)
+ +1.33(255
+0.70(.17)
+2.67(.36)
DBpedia
YahootUpp
YahoolLow))
98.7
Accuracy
Accuracy
50
72
88.5
Val
70
Val
Val Test
Test
Test
2
0
2
2 MelaaEpochs
Eetchs
MetaEpochs
AGNewws
Yelp
IMDB
93.5
63.8
Accuaae
93.0
63.6
88
6..44
90.5
87
90.0
63.0
06
Test
Test
Test
MetaFpochs
MetaFparhs
MetaFpochs
Figure 2: Training curves the classification datasets. As GraVeR makes the model fit to the validation set, the performance the test set increases.
distribution of word vectors in the pretrained embeddings is already good enough for the tasks. In contrast, GraVeR makes substantial improvements when using relatively poor embeddings for the tasks.
The comparison with regularization techniques is presented Table 2 (Top). The result shows that the performance gain by GraVeR is larger than other regularization methods. We also present the results when our method is combined with other regularization methods Table P (Bottom). The results show that GraVeR positively matches with the other regularization techniques, further im- proving the model performance. Figure also reports that our method definitely increases validation performance, which results in improvements in the test set.
Comparisons of a recent model (c.g., BERT) without extra data resources and word embedding perturbation methods will be discussed in Further Analysis ($

---

Under review as a conference paper at ICLR 2021
Table 3: List of top-20 nearest words of cue word (love) in initial embedding (Initial), after fine-tuned once (nineTuned), and our method GraVeR in DBpedia dataset. The differences between initial embedding and the other methods are marked in underlined. The differences between fine- tuned once and VeR are marked in bold. Gra further changes the distribution of word vectors.
Method
Top-20 Nearest Words with Iove(Cosine Similarity)
loved(. loved(.77).ic
Initial
know httpa//..w.aoao/////o/oca//////a
Fine Tune
so bu..6...eemmbbbrr((3335vaav....
know
GraVeR
(un. (um(()).hate((.6),
wanted waateee....elllyyyyyyyaveeyysyyr hoppe..))..
5.2 WORD DISTRIBUTIONS
To analyze VeR with respect word representation, we extract the word representations updated on DBpedia dataset and present the list of top-20 nearest words of a cue word in T In order to reduce the effect of randomness, we use GloVe for initial embedding. We can see that the word vectors are further fine-tuned and even find other similar words not shown in the embedding fine- tuned once. These results imply that our method can change the word vector distribution to further fine-tuned to the tasks. We also present another cue word results and visualization of top-100 nearest word vector distribution using I-SNE (Maaten Hinton 2008) in Appendix.
FURTHER ANALYSIS
From this Section, we mainly use TextCNN show the effect of GraVeR on the model whose performance is similar to state-of-the-art models.
6.1 RANDOM NOISE AND BIAS
Table 4: The performance of TexICNN and the standard deviation with Gra according to the random noises and bias. indicates our default setting noise scale 1, which means the random values sampled from uniform distribution the range between -1 and 1.
DBpedia
Yah(Up)
Yah(Low)
AGNews
Yelp
IMDB
Only Re train.
98.47=.05
68.73
45.55=.42
89.85+.22
63.21
85.43+
Gaussian Noise
98.71=.03
74.12
51.27=.34
90.91
63.642.08
88.55 19
No Addi tive Bias
98.65..02
73.41a.10
49.57=.81
90.73+.38
63.63+.12
88.54+.27
NoiseScale 0.
98.732.01
74.78a.10
52.17=.49
90.93+.21
63.832.16
88.55+.33
> NoiseScale 1.
98.732.07
75.09:.26
52.80
91.40:.25
63.942.17
88.77
NoiseScale 5.
98.72+.03
74.30+
51.27=.25
90.72+.52
63.34=.27
88.10+.49
Emb Perturbation 98.461.03
68.934.33
45.994.23
89.394.38
63.034.11
86.27..53
Noises. The range of random values filled in the maskers also affects the performance. We first re-train the model without any modification to observe the effect of the re-training processes. The model shows slightly better performance within a few meta-epochs, but it becomes overfitting, as we expected (see Table 4). The performance when we use Gaussian noise instead of the uniform distribution is also presented. It shows comparable but slightly worse result than our default settings. When the noise range is O, the noise only consists of the additive bias, which also shows marginally worse performance. The word perturbation method performs slightly better than just re- re-training, but there are large gaps between variations of Gra VeR.
7

---

Under review as a conference paper at ICLR 2021
6.2 HYPERPARAMETERS
Table 5: The performance of TextCNN and the standard deviation with GraVeR according to the masking rate, which determines how much the random maskers move to the next frequently used words. b indicates our default setting that masks 20% of vocabulary at once.
DBpedia
Yah(Up)
Yah(Low)
AGNews
Yelp
IMDB
MaskRate 0.1
98.78
74.86
52.45+.21
91.064..0
64.06:.22
88.50
MaskkRat 0.2
98.73..07
75.09:.26
52.80=.38
91.40:.25
63.942.17
88.77=.36
MaskRate 0.5
98.602.02
73.062.33
49.87=.42
91.02+.28
63.60=.14
88. 88.44+.20
MaskRate 1.0
98.65a.02
70.89..28
47.17=1.31
90.994.18
63.75=.02
87.83+.34
Masking Rate The amount of noise added by GraVeR is an important factor in that some noises should be small enough to be corrected during the re-training processes, while other noises should be large enough to change the word vector distribution. We first change the masking rate of how much random maskers move in every re-training process. The larger the masking rate becomes, the more words are masked in a re-training process, so the noise increases. Conversely, the amount of noise decreases the masking rate becomes small. The effect of the masking rate is presented Table Masking Rate=0.1 1 also shows good performance, but needs 2x re-training processes more than Masking Rate=0.2 2. We thus 0.2 default.
Table 6: The performance of TexICNN classifiers and the standard deviation with GraVeR accord- ing to the gradualness policy. The default method is to increase the number of maskers when the validation performance decreases.
DBpedia
Yah(Up)
Yah(Low)
AGNews
Yelp
IMDB
Proposed
98.73+.07
75.09:.26
52.80=.38
91.40+.25
63.94..17
88.77+.36
No Gradualness
98.68+.08
74.17+.24
52.23=.56
91.28+.11
63.74+.18
88.47
Reversed Grad. 98.694.06
73.97+.40
52.07+.23
91.00+.34
63.844.14
88.961.47
Both Grad.
98.66+.02
74.23a..8
51.87
91.09
63.55
88.79
Gradualness Policy Our proposed method is increase the number of maskers when the valida- tion performance decreases in order to make noises on the words masked in the previous re-training process again. Otherwise, we simply move the next frequently used words. We try changing the gradualness that (I) do not have gradualness, (2) have gradualness only when the validation performance increase, which is reverse to our proposed method, and (3) always have gradualness regardless of the validation performance. The result is presented in Table 6 Among the ablations, our proposed approach performs the best.
6.3 ON OTHER MODEL ARCHITECTURE
Table 7: The performance of Transformer- based classifier and the standard deviation.
DBpedia
Yah(Up)
Yah(Low)
AGNews
Yelp
IMDB
forme
98.42+.00
76.11+.02
57.68=.08
89,54+.31
60.07+.25
84.49+.93
+GraVeR
98.50+.02
76.
57.92+.25
90.40+.28
60.26+.25
85.75
We opt our method to the transformer-based classifier. The transformer classifier has 300 embedding dimension with positional-embeeddngg 32 batch size, 512 sequence length. also has 10 multi-heads but only encoder layer. Stacking more than 2 encoder layers harms the performance; We guess that the number of training set is not enough to train the model parameters. We do average-pooling to the encoded sentence vectors and use linear layer in order for classification. The other parameters follow the default setting of Pytorch nn. www.rnnfoomeerEnoodeeeayyyrrrr Table ssoows that Gra VeR works well in other model architectures.
8

---

Under review as a conference paper at ICLR 2021
Table 8: The performance of TextCNN tune classifiers and the standard deviation with pretrained word vectors trained from only training data and comparison with pretrained word vector trained from external resources.
DBpedia
Yah(Up)
Yah(Low)
AGNews
Yelp
IMDB
(Sp)
98.73
75.09
52.80+.38
91.40=.25
63.94..17
88.77=.36
w2v (Gen)
98.614.03
70.45+1.02
46.74..66
91.27
62.73+.23
85.07
w2v (Sp)
+0.08(.04)
+3.72(.43)
+4.25(.51)
-0.32(34)
+0.52(.33)
-0,02(1.18)
glv (Gen)
98.79+.07
7..77+.41
52.83+.57
91.37 1.10
63.59+.32
87.67=.60
glv (Sp)
-0.54(.34)
-2.47(1.01)
.4.72(1844)
-2.41(77))
-0.59(443)
-5.19(1.99
ftt (Gen)
93.43
64.56+.31
40.91
89.43
62.75..11
ftt (Sp)
+0.17(.11)
+9.46(.56
+9.52(.58)
+1.49(.14)
+0.41(.23)
+4.57(1.53)
BERT (Gen)
99.20+.01
81.75+.41
65.83+.37
93.58+.23
69.25..15
93.24
BERT (Sp)
98.57+.13
77.19+.22
58.11+.58
90.35+.21
60.99+.22
87.17+.34
6.4
PRETRAINING TRAINING DATA.
Table shoow the classification performance according to the usage of pretrained word emheddings. Gen means general representation trained on a large corpus, which is the pretrained embeddings what mentioned in $5.1 and bert-base -ancaee from huggingface (Wolf et 2019). Sp denotes specialized representation trained using only training data. We set the hyperparameter for training as default settings used in their API.
Despite a small number of parameters, TexICNN tune with GraVeR shows comparable performance with BERT (Sp). Vea's improvement is even similar glv (Gen), which means GraVeR can complement the information from external resources, in this case, Common Crawl (42B tokens, 1.9M vocab). Although there performance gap with pretrained BERT, the advantages of Gra VeR (1) do not need any extra data resources, and (2) do not need to very deep model architectures consisting of a bunch of parameters. Therefore, GraVeR must be an attractive option when using pretrained word embeddings.
7 DISCUSSION
Re-training Cost. Although GraVeR shows strong performance, its re-training processes take 1/ 3/[MaskingRate] more times than conventional training process. However, we showed that a small model with GraVeR even performs on par with recent huge models. When considering the parameter size, for example, TextCNNtume has 136K parameters and it needs five times re-training process in our default setting MaskingRate=0. .2 while BERT has 110M parameters with one fine-tuning process. Then, the parameters need to be trained are 780K and 10M, respectively; i.e. 141 times cheaper. Furthermore, the training time for small model must be faster since it can be trained with larger mini bani-batchhae size. As a representation learning.
GraVeR's representations are learned from given training set only. In Table g general word embeddings trained from large data shows worse performance than domain (or data) specialized word embeddings. That is, order to solve the task, the word representation should be specialized in given data. By using GraVeR, we can easily get specialized (further fine-tuned) representation only with the random noise including bias and the iterative training processes. Besides, believe that using sophisticated noising trick instead of simple random makes further improvement in Gra VeR.
8 CONCLUSION
We propose GraVeR, which adds random noises and bias to word embeddings in order to change the word vector distribution and regularize a model. Through the re-training process, we can make the of noises to learn better representations. In the experiments, the model incrementally fits the validation set, GraVeR largely improves model performances. We expect that our general training approach can be used to various models to improve model performances.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: (00.:0607.5555 2016.
arXiv preprint
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. arXiv preprint wxXiv:160.004606 2016.
Ming-Wei Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar, Importance of semantic repre- sentation: Dataless classification. In AAAI, volume 2. pp. 8300-35,10 2008.
Yong Cheng, Zhaopeng Tu, Fandong Meng. Junjie Zhai, and Yang Liu. Towards robust neural ma- chine translation. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 1756-1766, 2018.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. Very deep convolutional net- works for text classification. European Chapter of the Association for Computational Linguis- fics EACL' 17, 2017.
Jacob Devlin, Ming-Wes Heig-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0488 2018.
Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1606-1615, 2015.
Richard HR Hahnloser, Rahul Sarpeshkar, Misha A Mahowald, Rodney J Douglas, and H Sebastian Scung. Digital selection and analogue amplification coexist cortex-inspired silicon circuit. Nature, 405(6789):947, 2000.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, pp. 2164-2174, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, In International Conference on Machine Learning, pp. 448 448-456, 2015.
Hwiyeol Jo and Stanley Jungkyu Choi. Extrofitting: Enriching word representation and its vector space with semantic lexicons. ACL 2018, pp. 24, 2018.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746-1751, 2014.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 2014.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sehastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia-a large- scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 177) 2015.
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. 2018.
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies- 1, pp. 142-150. Association for Computational Linguisties. 2011.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2779 2008
10

---

Under review as a conference paper at ICLR 2021
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pp. 6294 6305, 2017.
Oren Melamud, Jacob Goldberger, and Ido Dagan. contex(2vec: Learning generic context embed- ding with bidirectional Istm. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 51-61, 2016.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 1301.3888 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 31 1111-3119, 2013b.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. arXiv preprint 1605:0770 2016.
Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. Semantic specialization of distributional word vector spaces using monolingual and gross-lingual constraints. Transactions of the Association for Computational Linguistics, 5:309-324, 2017.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. I Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers), volume 1, pp. 2227-2237, 2018
David C Plaut et al. Experiments on learning by back propagation. 1986.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Iya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-13)) 1958, 2014.
Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: simple and general method for semi sumi-supereiee learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pp. 384 394. Association for Computational Linguisties, 2010.
Twan van Laarhoven. 1.2 regularization versus batch and weight normalization. arXiv preprint
arXiv:1706.053500 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. arXiv preprint 1rXi::142..6623, 2014.
Ivan Vulić, Nikola Mrkšić, and Anna Korhonen. Cross-lingual induction and transfer of verb classes based word vector space specialisation. arXiv preprint arXiv: 1707.06977...... 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing. ArXiv, abs/1910.03771, 2019.
Dongxu Zhang and Zhichao Yang. Word embedding perturbation for sentence classification. arXiv preprint
2018
Xiang Zhang, Junbo Zhao and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.


Reviews:
[{"review_id": "aeF6n5Nmoev", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to improve word embedding by iteratively 1) adding noises to the trained word embeddings, and 2) retraining the model with the noised embeddings.\n\nWhile the paper has some interesting points, I feel the paper lacks novelty because adding noises as regularization is a well-known, and the iterative incurs significant costs. Besides, I found the experiment comparison unconvincing. Thus, I give this paper the overall rating \"rejection\".\n\nComment:\n1) In the introduction, the author claims the embeddings will often stuck in local minimums during training as one motivation for this work. It would be better to have some experimental demonstration of this claim.\n2) The iterative approach requires an task-specific validation set, and consults the validation accuracy multiple times during the whole training process. This provide an unfair advantage for the proposed method, compared to the baselines which does not involves any task-specific signals.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "X81QeUPqp2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n##########################################################################\n\nSummary:\n\nThis paper presents a simple meta-learning method for iteratively fine-tuning\nword embeddings. The method works by adding noise and bias terms to the learned\nembeddings after each training session, and initializing the same model on the\nsame task with the resulting representations for further fine-tuning. Noise is\nadded to a subset of the vocabulary, while bias is added to every element of it.\nThe process is repeated until noise has been added to all the elements of the\nvocabulary.\n\nResults show that the proposed method consistently improves results in the set\nof classification tasks in which it was tested.\n\n\n##########################################################################\n\nReasons for score:\n\nHowever, the paper is not very thorough in its exposition and leaves too many\nquestions without answer. In all, it is difficult to assess the real\ncontribution of the suggested method.\n\n\n##########################################################################\n\nPros:\n\n- Method could be useful to the community working on non-contextualized word\n  representations.\n\n##########################################################################\n\nCons:\n\n- The proposed method is unclear.\n- Analyses are lacking\n\n#########################################################################\n\nComments, suggestions and questions for the authors:\n\n- p. 1, sec. 1, par. 1: I do not understand the difference between\n  \"starting with better representations\" and \"building more sophisticated\n  architectures\". For the former you cite ELMo as an example, but ELMo was both\n  a new architecture and training methodology, hence my confusion with\n  qualifying it as \"starting with better representations\". For the latter you\n  cite the Transformers, which makes more sense as a new architecture. But then,\n  how would you classify BERT, being both a new architecture sharing transformer\n  components, and a new training methodology? (This is more of a rhetorical\n  question; I suggest clarifying the points above directly in the paper.)\n\n- p. 1, sec. 1, par. 4: Did you take into consideration phenomena such as\n  catastrophic forgetting (https://arxiv.org/abs/1312.6211), when re-training?\n  Even though you are not changing the training task, I think it would be\n  valuable to at least mention this.\n\n- p. 2, sec. 2.1, par. 1: Mikolov et al. 2013b make no mention of the\n  distributional hypothesis, and even if they did, there is no implication\n  between this hypothesis and the fact that pretrained embedding vectors are\n  composed of the pairs you mention.\n\n- p. 2, sec. 2.1, par. 2: A reader not deeply familiarized with the differences\n  between contextualized word representations, and pre-trained word-vectors will\n  probably have a hard time understanding this paragraph. I suggest making it\n  clear that contextualized representations in the context of ELMo and BERT are\n  actually pre-trained models, and not only pre-trained word representations\n  such as GloVe, word2vec, and fasttext.\n\n- p. 2, sec. 2.2, par. 2: References to previous works doing \"Word Dropping\"\n  would be a welcome addition to your paper. Is this paragraph an allusion to\n  stopword removal, or something else?\n\n- p. 3, sec. 3.2, par. 2: `MaskingRate` is a hyperparameter of the noise\n  maskers, but in sec. 3.1, par. 2 it says that the bias is weighted by\n  1/[`MaskingRate`]. I suggest clarifying the differences between noises and\n  biases, and their relationship to maskers.\n\n- p. 3, sec. 3.2, par. 3: You mention you add a bias to the fine-tuned word\n  embeddings, in order to take them to the same embedding space. What do you\n  mean exactly with this? Addition does not change the dimensionality of the\n  summands, therefore adding something to W' will only shift its values and not\n  modify their embedding space.\n\n- p. 5, sec. 4.2, par. 3: How do you initialize word embeddings randomly? Is it\n  also by sampling from the uniform distribution between -1 and 1?\n\n- p. 5, sec. 5.1, par. 2: It is unclear what you mean with \"since we train a\n  model from scratch except for word embeddings.\"\n\n- p. 6, sec. 5.1, par. 2: You mention that GraVeR makes substantial improvements\n  when using relatively poor embeddings, but I can see that GraVeR has a\n  significant impact on fasttext embeddings (Table 1), which I would not call\n  poor.\n\n- p. 6, Table 2: I suggest specifying what the values between parentheses and\n  after the $\\pm$ are; standard deviation? confidence interval? if so, with\n  which confidence level?. Same suggestion for Table 3.\n\n- p. 7, sec 5.2: Why did you decide to show cherry-picked results for GloVe,\n  despite your method not working well with these pre-trained embeddings, as\n  shown in Table 1? Why not show results for fasttext embeddings instead?\n\n- p. 7, Table 4: If NoiseScale 1 means that the values from the masker were\n  sampled from Uniform(-1; 1), as explained in section 3.2, I assume that\n  NoiseScale 5 implies Uniform(-5; 5). If this is true then what does NoiseScale\n  0 imply?\n\n- p. 7, sec. 6.1, par. 1: How can you tell from Table 4 that the model\n  overfits?\n\n- p. 8, Table 5: As mentioned previously, I am confused by the choice of\n  parenthesis versus plus-minus symbols, and the meaning of the numbers they\n  refer to. Why do you use $\\pm$ for every Gen variant, but parentheses for Sp?\n  why do you use $\\pm$ for BERT (Sp), despite the previous? Also, why did you\n  decide to invert the Gen-Sp row order in the BERT entry?\n\n- I am a bit confused about the role Maskers play in your model. You mention\n  them for the first time in the caption of Figure 1, without much context. Then\n  you mention that you \"add maskers filled with random values to a portion of\n  W'\", which makes me think that maskers are nothing other than random vectors,\n  but I am not 100% sure. The paper goes on assuming that the reader has a clear\n  understanding of what maskers are, but I personally found this not to be the\n  case. Further, maskers are not mentioned in Algorithm 1, only `MaskedWords`\n  and `NextFrequentlyUsedWords`, which I guess are somehow related to maskers,\n  but again, I can't be sure exactly how just by reading the paper.\n\n- Another pain point related to the above is that you superficially mention (in\n  Algorithm 1, and the two paragraphs before it), that word frequency somehow\n  conditions which words are masked, but no details are given as to how this\n  happens.\n\n- I found the tables to be a bit overcrowded and difficult to parse in general.\n  I suggest trying to simplify them to better get your point across.\n\n- I think the Table 1 mentioned in the Appendix should be Table 6.\n\n\n#########################################################################\n\nSome typos:\n\n- Abstract: \"some of noises\" -> \"some noises\"\n- Abstract: What do you mean with the last sentence?\n- p. 1, sec. 1, par. 2: \"the general and meaning of words\" -> \"the general\n  meaning\"\n- p. 1, sec. 1, par. 4: \"to the next re-training process\" -> \"in the next\n  re-training process\"\n- p. 1, sec. 1, par. 4: What do you mean with \"the model must be overfitted\"?\n- p. 2, sec. 2.2, par. 3: \"It can regularize the models,[...]re-usability.\" is\n  not a grammatically correct sentence.\n- p. 2, sec. 2.3, par. 2: In \"Also, dropout discourses the weights\" I think you\n  meant to write something other than \"discourses\".\n- p. 3, sec. 3.1, par. 2: \"best-performed\" -> \"best-performing\"\n- p. 4, sec. 3.2, par. 4: I think the sentence \"Although this approach cannot\n  warrant moderate noise, we take a safe approach.\" is not clear enough.\n- p. 4, sec. 4.1, par. 1: \"in Appendix\" -> \"in the Appendix\"\n- p. 5, sec. 4.2, par. 1: \"with the 32 channels\" -> \"with 32 channels\"\n- p. 5, sec. 4.2, par. 1: \"max-pooled\" -> \"max-pooling\"\n- p. 5, sec. 4.2, par. 2: \"small parameters\" -> \"few parameters\"\n- p. 5, Table 1 caption: \"vector\" -> \"vectors\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "MrxYRsTZXV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "It's already known that embeddings like word2vec and glove are biased [1] and needs postprocessing for better performance. This paper designed a novel approach to do embedding normalizations. After each round of training, noise is intentionally introduced to perturbate the finetuned parameters. Afterwards, another round of training starting from perturbated local optima could in potential converge to a better one. This method is validated via CNN-based text classification.\n\nPros:\n+ The idea to iterate the training process is interesting and the proposed approach shows reasonable consistent improvement in different domains, comparing to existing regularization methods.\n\nCons:\n- Text classification is a standard NLP task and CNN is not the only solution. The performance boost on top of RNN or transformer-based approach could make the regularization more convincing.\n- The power of word2vec and glove comes from the fact that they're trained using a large unsupervised corpus. The regularization method proposed in this paper, however, requires supervised labels on the training and validation set. The proposed approach is more stronger if the idea can be further generalized to pretraining a better set of word embeddings.\n- Retraining cost. Even though the model size of CNN is smaller than BERT so the cost on retraining seems to be reasonable, there is still a performance gap between multi-round and one-shot BERT. \n\nMinor comment: the notation is not clear in the paper. In the algorithm, one perturbation is added to W' but the other one is added to W, is this a typo?\n\n[1] Mu, Jiaqi, Suma Bhat, and Pramod Viswanath. \"All-but-the-top: Simple and effective postprocessing for word representations.\" arXiv preprint arXiv:1702.01417 (2017).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes adding noise regularization, iteratively during training to word embeddings.
The method is evaluated on CNN-based text classification.
Overall, there is novelty in the proposed method, however there are concerns about the experiments and analysis of the proposed approach.
Decision: Reject