prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Why masking words in the decoder is helpful?
Context 1: Introduction
Text summarization generates summaries from input documents while keeping salient information  It is an important task and can be applied to several real-world applications  Many methods have been proposed to solve the text summarization problem BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   There are two main text summarization techniques: extractive and abstractive  Extractive summarization generates summary by selecting salient sentences or phrases from the source text, while abstractive methods paraphrase and restructure sentences to compose the summary  We focus on abstractive summarization in this work as it is more flexible and thus can generate more diverse summaries 
Recently, many abstractive approaches are introduced based on neural sequence-to-sequence framework BIBREF4 , BIBREF0 , BIBREF3 , BIBREF5   Based on the sequence-to-sequence model with copy mechanism BIBREF6 , BIBREF0 incorporates a coverage vector to track and control attention scores on source text  BIBREF4 introduce intra-temporal attention processes in the encoder and decoder to address the repetition and incoherent problem 
There are two issues in previous abstractive methods: 1) these methods use left-context-only decoder, thus do not have complete context when predicting each word  2) they do not utilize the pre-trained contextualized language models on the decoder side, so it is more difficult for the decoder to learn summary representations, context interactions and language modeling together 
Recently, BERT has been successfully used in various natural language processing tasks, such as textual entailment, name entity recognition and machine reading comprehensions  In this paper, we present a novel natural language generation model based on pre-trained language models (we use BERT in this work)  As far as we know, this is the first work to extend BERT to the sequence generation task  To address the above issues of previous abstractive methods, in our model, we design a two-stage decoding process to make good use of BERT's context modeling ability  On the first stage, we generate the summary using a left-context-only-decoder  On the second stage, we mask each word of the summary and predict the refined word one-by-one using a refine decoder  To further improve the naturalness of the generated sequence, we cooperate reinforcement objective with the refine decoder 
The main contributions of this work are:
1  We propose a natural language generation model based on BERT, making good use of the pre-trained language model in the encoder and decoder process, and the model can be trained end-to-end without handcrafted features 
2  We design a two-stage decoder process  In this architecture, our model can generate each word of the summary considering both sides' context information 
3  We conduct experiments on the benchmark datasets CNN/Daily Mail and New York Times  Our model achieves a 33 33 average of ROUGE-1, ROUGE-2 and ROUGE-L on the CNN/Daily Mail, which is state-of-the-art  On the New York Times dataset, our model achieves about 5 6% relative improvement over ROUGE-1 
Text Summarization
In this paper, we focus on single-document multi-sentence summarization and propose a supervised abstractive model based on the neural attentive sequence-to-sequence framework which consists of two parts: a neural network for the encoder and another network for the decoder  The encoder encodes the input sequence to intermediate representation and the decoder predicts one word at a time step given the input sequence representation vector and previous decoded output  The goal of the model is to maximize the probability of generating the correct target sequences  In the encoding and generation process, the attention mechanism is used to concentrate on the most important positions of text  The learning objective of most sequence-to-sequence models is to minimize the negative log likelihood of the generated sequence as following equation shows, where $y^*_i$ is the i-th ground-truth summary token 
$$Loss = - \log \sum _{t=1}^N P(y_t^*|y_{<t}^*, X)$$   (Eq  3)
However, with this objective, traditional sequence generation models consider only one direction context in the decoding process, which could cause performance degradation since complete context of one token contains preceding and following tokens, thus feeding only preceded decoded words to the decoder so that the model may generate unnatural sequences  For example, attentive sequence-to-sequence models often generate sequences with repeated phrases which harm the naturalness  Some previous works mitigate this problem by improving the attention calculation process, but in this paper we show that feeding bi-directional context instead of left-only-context can better alleviate this problem 
Text summarization models are usually classified to abstractive and extractive ones  Recently, extractive models like DeepChannel BIBREF8 , rnn-ext+RL BIBREF9 and NeuSUM BIBREF2 achieve higher performances using well-designed structures  For example, DeepChannel propose a salience estimation network and iteratively extract salient sentences  BIBREF16 train a sentence compression model to teach another latent variable extractive model 
Also, several recent works focus on improving abstractive methods  BIBREF3 design a content selector to over-determine phrases in a source document that should be part of the summary  BIBREF11 introduce inconsistency loss to force words in less attended sentences(which determined by extractive model) to have lower generation probabilities  BIBREF5 extend seq2seq model with an information selection network to generate more informative summaries 
Bi-Directional Pre-Trained Context Encoders
Recently, context encoders such as ELMo, GPT, and BERT have been widely used in many NLP tasks  These models are pre-trained on a huge unlabeled corpus and can generate better contextualized token embeddings, thus the approaches built on top of them can achieve better performance 
Since our method is based on BERT, we illustrate the process briefly here  BERT consists of several layers  In each layer there is first a multi-head self-attention sub-layer and then a linear affine sub-layer with the residual connection  In each self-attention sub-layer the attention scores $e_{ij}$ are first calculated as Eq  ( 5 ) () shows, in which $d_e$ is output dimension, and $W^Q, W^K, W^V$ are parameter matrices 
$$&a_{ij} = \cfrac{(h_iW_Q)(h_jW_K)^T}{\sqrt{d_e}}  \\ &e_{ij} = \cfrac{\exp {e_{ij}}}{\sum _{k=1}^N\exp {e_{ik}}} $$   (Eq  5)
Then the output is calculated as Eq  ( 6 ) shows, which is the weighted sum of previous outputs $h$ added by previous output $h_i$   The last layer outputs is context encoding of input sequence 
$$o_i = h_i + \sum _{j=1}^{N} e_{ij}(h_j W_V) $$   (Eq  6)
Despite the wide usage and huge success, there is also a mismatch problem between these pre-trained context encoders and sequence-to-sequence models  The issue is that while using a pre-trained context encoder like GPT or BERT, they model token-level representations by conditioning on both direction context  During pre-training, they are fed with complete sequences  However, with a left-context-only decoder, these pre-trained language models will suffer from incomplete and inconsistent context and thus cannot generate good enough context-aware word representations, especially during the inference process 
Model
In this section, we describe the structure of our model, which learns to generate an abstractive multi-sentence summary from a given source document 
Based on the sequence-to-sequence framework built on top of BERT, we first design a refine decoder at word-level to tackle the two problems described in the above section  We also introduce a discrete objective for the refine decoders to reduce the exposure bias problem  The overall structure of our model is illustrated in Figure 1  
Problem Formulation
We denote the input document as $X = \lbrace x_1, \ldots , x_m\rbrace $ where $x_i \in \mathcal {X}$ represents one source token  The corresponding summary is denoted as $Y = \lbrace y_1, \ldots , y_L\rbrace $ , $L$ represents the summary length 
Given input document $X$ , we first predict the summary draft by a left-context-only decoder, and then using the generated summary draft we can condition on both context sides and refine the content of the summary  The draft will guide and constrain the refine process of summary 
Summary Draft Generation
The summary draft is based on the sequence-to-sequence model  On the encoder side the input document $X$ is encoded into representation vectors $H = \lbrace h_1, \ldots , h_m\rbrace $ , and then fed to the decoder to generate the summary draft $A = \lbrace a_1, \ldots , a_{|a|}\rbrace $  
We simply use BERT as the encoder  It first maps the input sequence to word embeddings and then computes document embeddings as the encoder's output, denoted by following equation 
$$H = BERT(x_1, \ldots , x_m)$$   (Eq  10)
In the draft decoder, we first introduce BERT's word embedding matrix to map the previous summary draft outputs $\lbrace y_1, \ldots , y_{t-1}\rbrace $ into embeddings vectors $\lbrace q_1, \ldots , q_{t-1}\rbrace $ at t-th time step  Note that as the input sequence of the decoder is not complete, we do not use the BERT network to predict the context vectors here 
Then we introduce an $N$ layer Transformer decoder to learn the conditional probability $P(A|H)$   Transformer's decoder-encoder multi-head attention helps the decoder learn soft alignments between summary and source document  At the t-th time step, the draft decoder predicts output probability conditioned on previous outputs and encoder hidden representations as Eq  ( 13 ) shows, in which $q_{<t} = \lbrace q_1, \ldots , q_{t-1}\rbrace $   Each generated sequence will be truncated in the first position of a special token '[PAD]' 
$$&P^{vocab}_t(w) = f_{dec}(q_{<t}, H)  \\ &L_{dec} = \sum _{i=1}^{|a|} -\log P(a_i = y_i^*|a_{< i}, H) $$   (Eq  13)
As Eq  () shows, the decoder's learning objective is to minimize negative likelihood of conditional probability, in which $y_i^*$ is the i-th ground truth word of summary 
However a decoder with this structure is not sufficient enough: if we use the BERT network in this decoder, then during training and inference, in-complete context(part of sentence) is fed into the BERT module, and although we can fine-tune BERT's parameters, the input distribution is quite different from the pre-train process, and thus harms the quality of generated context representations 
If we just use the embedding matrix here, it will be more difficult for the decoder with fresh parameters to learn to model representations as well as vocabulary probabilities, from a relative small corpus compared to BERT's huge pre-training corpus  In a word, the decoder cannot utilize BERT's ability to generate high quality context vectors, which will also harm performance 
This issue exists when using any other contextualized word representations, so we design a refine process to mitigate it in our approach which will be described in the next sub-section 
As some summary tokens are out-of-vocabulary words and occurs in input document, we incorporate copy mechanism BIBREF6 based on the Transformer decoder, we will describe it briefly 
At decoder time step $t$ , we first calculate the attention probability distribution over source document $X$ using the bi-linear dot product of the last layer decoder output of Transformer $o_t$ and the encoder output $h_j$ , as Eq  ( 15 ) () shows 
$$u_t^j =& o_t W_c h_j \\ \alpha _t^j =& \cfrac{\exp {u_t^j}}{\sum _{k=1}^N\exp {u_t^k}} $$   (Eq  15)
We then calculate copying gate $g_t\in [0, 1]$ , which makes a soft choice between selecting from source and generating from vocabulary, $W_c, W_g, b_g$ are parameters:
$$g_t = sigmoid(W_g \cdot [o_t, h] + b_g) $$   (Eq  16)
Using $g_t$ we calculate the weighted sum of copy probability and generation probability to get the final predicted probability of extended vocabulary $\mathcal {V} + \mathcal {X}$ , where $\mathcal {X}$ is the set of out of vocabulary words from the source document  The final probability is calculated as follow:
$$P_t(w) = (1-g_t)P_t^{vocab}(w) + g_t\sum _{i:w_i=w} \alpha _t^i$$   (Eq  17)
Summary Refine Process
The main reason to introduce the refine process is to enhance the decoder using BERT's contextualized representations, so we do not modify the encoder and reuse it during this process 
On the decoder side, we propose a new word-level refine decoder  The refine decoder receives a generated summary draft as input, and outputs a refined summary  It first masks each word in the summary draft one by one, then feeds the draft to BERT to generate context vectors  Finally it predicts a refined summary word using an $N$ layer Transformer decoder which is the same as the draft decoder  At t-th time step the n-th word of input summary is masked, and the decoder predicts the n-th refined word given other words of the summary 
The learning objective of this process is shown in Eq  ( 19 ), $y_i$ is the i-th summary word and $y_{i}^*$ for the ground-truth summary word, and $a_{\ne i} = \lbrace a_1, \ldots , a_{i-1}, a_{i+1}, \ldots , a_{|y|}\rbrace $  
$$L_{refine} = \sum _{i=1}^{|y|} -\log P(y_i = y_i^*|a_{\ne i}, H) $$   (Eq  19)
From the view of BERT or other contextualized embeddings, the refine decoding process provides a more complete input sequence which is consistent with their pre-training processes  Intuitively, this process works as follows: first the draft decoder writes a summary draft based on a document, and then the refine decoder edits the draft  It concentrates on one word at a time, based on the source document as well as other words 
We design the word-level refine decoder because this process is similar to the cloze task in BERT's pre-train process, therefore by using the ability of the contextual language model the decoder can generate more fluent and natural sequences 
The parameters are shared between the draft decoder and refine decoder, as we find that using individual parameters the model's performance degrades a lot  The reason may be that we use teach-forcing during training, and thus the word-level refine decoder learns to predict words given all the other ground-truth words of summary  This objective is similar to the language model's pre-train objective, and is probably not enough for the decoder to learn to generate refined summaries  So in our model all decoders share the same parameters 
Researchers usually use ROUGE as the evaluation metric for summarization, however during sequence-to-sequence model training, the objective is to maximize the log likelihood of generated sequences  This mis-match harms the model's performance, so we add a discrete objective to the model, and optimize it by introducing the policy gradient method  For example, the discrete objective for the summary draft process is as Eq  ( 21 ) shows, where $a^s$ is the draft summary sampled from predicted distribution, and $R(a^s)$ is the reward score compared with the ground-truth summary, we use ROUGE-L in our experiment  To balance between optimizing the discrete objective and generating readable sequences, we mix the discrete objective with maximum-likelihood objective  As Eq  () shows, minimizing $\hat{L}_{dec}$ is the final objective for the draft process, note here $L_{dec}$ is $-logP(a|x)$   In the refine process we introduce similar objectives 
$$L^{rl}_{dec} = R(a^s)\cdot [-\log (P(a^s|x))]  \\ \hat{L}_{dec} = \gamma * L^{rl}_{dec} + (1 - \gamma ) * L_{dec} $$   (Eq  21)
Learning and Inference
During model training, the objective of our model is sum of the two processes, jointly trained using "teacher-forcing" algorithm  During training we feed the ground-truth summary to each decoder and minimize the objective 
$$L_{model} = \hat{L}_{dec} + \hat{L}_{refine}$$   (Eq  23)
At test time, each time step we choose the predicted word by $\hat{y} = argmax_{y^{\prime }} P(y^{\prime }|x)$ , use beam search to generate the draft summaries, and use greedy search to generate the refined summaries 
Settings
In this work, all of our models are built on $BERT_{BASE}$ , although another larger pre-trained model with better performance ( $BERT_{LARGE}$ ) has published but it costs too much time and GPU memory  We use WordPiece embeddings with a 30,000 vocabulary which is the same as BERT  We set the layer of transformer decoders to 12(8 on NYT50), and set the attention heads number to 12(8 on NYT50), set fully-connected sub-layer hidden size to 3072  We train the model using an Adam optimizer with learning rate of $3e-4$ , $\beta _1=0 9$ , $\beta _2=0 999$ and $\epsilon =10^{-9}$ and use a dynamic learning rate during the training process  For regularization, we use dropout BIBREF13 and label smoothing BIBREF14 in our models and set the dropout rate to 0 15, and the label smoothing value to 0 1  We set the RL objective factor $\gamma $ to 0 99 
During training, we set the batch size to 36, and train for 4 epochs(8 epochs for NYT50 since it has many fewer training samples), after training the best model are selected from last 10 models based on development set performance  Due to GPU memory limit, we use gradient accumulation, set accumulate step to 12 and feed 3 samples at each step  We use beam size 4 and length penalty of 1 0 to generate logical form sequences 
We filter repeated tri-grams in beam-search process by setting word probability to zero if it will generate an tri-gram which exists in the existing summary  It is a nice method to avoid phrase repetition since the two datasets seldom contains repeated tri-grams in one summary  We also fine tune the generated sequences using another two simple rules  When there are multi summary sentences with exactly the same content, we keep the first one and remove the other sentences; we also remove sentences with less than 3 words from the result 
To evaluate the performance of our model, we conduct experiments on CNN/Daily Mail dataset, which is a large collection of news articles and modified for summarization  Following BIBREF0 we choose the non-anonymized version of the dataset, which consists of more than 280,000 training samples and 11490 test set samples 
We also conduct experiments on the New York Times(NYT) dataset which also consists of many news articles  The original dataset can be applied here  In our experiment, we follow the dataset splits and other pre-process settings of BIBREF15   We first filter all samples without a full article text or abstract and then remove all samples with summaries shorter than 50 words  Then we choose the test set based on the date of publication(all examples published after January 1, 2007)  The final dataset contains 22,000 training samples and 3,452 test samples and is called NYT50 since all summaries are longer than 50 words 
We tokenize all sequences of the two datasets using the WordPiece tokenizer  After tokenizing, the average article length and summary length of CNN/Daily Mail are 691 and 51, and NYT50's average article length and summary length are 1152 and 75  We truncate the article length to 512, and the summary length to 100 in our experiment(max summary length is set to 150 on NYT50 as its average golden summary length is longer) 
On CNN/Daily Mail dataset, we report the full-length F-1 score of the ROUGE-1, ROUGE-2 and ROUGE-L metrics, calculated using PyRouge package and the Porter stemmer option  On NYT50, following BIBREF4 we evaluate limited length ROUGE recall score(limit the generated summary length to the ground truth length)  We split NYT50 summaries into sentences by semicolons to calculate the ROUGE scores 
Results and Analysis
Table 1 shows the results on CNN/Daily Mail dataset, we compare the performance of many recent approaches with our model  We classify them to two groups based on whether they are extractive or abstractive models  As the last line of the table shows, the ROUGE-1 and ROUGE-2 score of our full model is comparable with DCA, and outperforms on ROUGE-L  Also, compared to extractive models NeuSUM and MASK- $LM^{global}$ , we achieve slight higher ROUGE-1  Except the four scores, our model outperforms these models on all the other scores, and since we have 95% confidence interval of at most $\pm $ 0 20, these improvements are statistically significant 
As the last four lines of Table 1 show, we conduct an ablation study on our model variants to analyze the importance of each component  We use three ablation models for the experiments  One-Stage: A sequence-to-sequence model with copy mechanism based on BERT; Two-Stage: Adding the word-refine decoder to the One-Stage model; Two-Stage + RL: Full model with refine process cooperated with RL objective 
First, we compare the Two-Stage+RL model with Two-Stage ablation, we observe that the full model outperforms by 0 30 on average ROUGE, suggesting that the reinforcement objective helps the model effectively  Then we analyze the effect of refine process by removing word-level refine from the Two-Stage model, we observe that without the refine process the average ROUGE score drops by 1 69  The ablation study shows that each module is necessary for our full model, and the improvements are statistically significant on all metrics 
To evaluate the impact of summary length on model performance, we compare the average rouge score improvements of our model with different length of ground-truth summaries  As the above sub-figure of Figure 2 shows, compared to Pointer-Generator with Coverage, on length interval 40-80(occupies about 70% of test set) the improvements of our model are higher than shorter samples, confirms that with better context representations, in longer documents our model can achieve higher performance 
As the below sub-figure of Figure 2 shows, compared to extractive baseline: Lead-3 BIBREF0 , the advantage of our model will fall when golden summary length is greater than 80  This probably because that we truncate the long documents and golden summaries and cannot get full information, it could also because that the training data in these intervals is too few to train an abstractive model, so simple extractive method will not fall too far behind 
Additional Results on NYT50
Table 2 shows experiments on the NYT50 corpus  Since the short summary samples are filtered, NYT50 has average longer summaries than CNN/Daily Mail  So the model needs to catch long-term dependency of the sequences to generate good summaries 
The first two lines of Table 2 show results of the two baselines introduced by BIBREF15 : these baselines select first n sentences, or select the first k words from the original document  Also we compare performance of our model with two recent models, we see 2 39 ROUGE-1 improvements compared to the ML+RL with intra-attn approach(previous SOTA) carries over to this dataset, which is a large margin  On ROUGE-2, our model also get an improvement of 0 51  The experiment proves that our approach can outperform competitive methods on different data distributions 
Pre-trained language models
Pre-trained word vectors BIBREF17 , BIBREF18 , BIBREF19 have been widely used in many NLP tasks  More recently, pre-trained language models (ELMo, GPT and BERT), have also achieved great success on several NLP problems such as textual entailment, semantic similarity, reading comprehension, and question answering BIBREF20 , BIBREF21 , BIBREF22  
Some recent works also focus on leveraging pre-trained language models in summarization  BIBREF23 pretrain a language model and use it as the sentiment analyser when generating reviews of goods  BIBREF24 train a language model on golden summaries, and then use it on the decoder side to incorporate prior knowledge 
In this work, we use BERT(which is a pre-trained language model using large scale unlabeled data) on the encoder and decoder of a seq2seq model, and by designing a two stage decoding structure we build a competitive model for abstractive text summarization 
Conclusion and Future Work
In this work, we propose a two-stage model based on sequence-to-sequence paradigm  Our model utilize BERT on both encoder and decoder sides, and introduce reinforce objective in learning process  We evaluate our model on two benchmark datasets CNN/Daily Mail and New York Times, the experimental results show that compared to previous systems our approach effectively improves performance 
Although our experiments are conducted on summarization task, our model can be used in most natural language generation tasks, such as machine translation, question generation and paraphrasing  The refine decoder and mixed objective can also be applied on other sequence generation tasks, and we will investigate on them in future work 
Context 2: Introduction
There have been many implementations of the word2vec model in either of the two architectures it provides: continuous skipgram and CBoW (BIBREF0)  Similar distributed models of word or subword embeddings (or vector representations) find usage in sota, deep neural networks like BERT and its successors (BIBREF1, BIBREF2, BIBREF3)  These deep networks generate contextual representations of words after been trained for extended periods on large corpora, unsupervised, using the attention mechanisms (BIBREF4) 
It has been observed that various hyper-parameter combinations have been used in different research involving word2vec with the possibility of many of them being sub-optimal (BIBREF5, BIBREF6, BIBREF7)  Therefore, the authors seek to address the research question: what is the optimal combination of word2vec hyper-parameters for intrinsic and extrinsic NLP purposes? There are astronomically high numbers of combinations of hyper-parameters possible for neural networks, even with just a few layers  Hence, the scope of our extensive work over three corpora is on dimension size, training epochs, window size and vocabulary size for the training algorithms (hierarchical softmax and negative sampling) of both skipgram and CBoW  The corpora used for word embeddings are English Wiki News Abstract by BIBREF8 of about 15MB, English Wiki Simple (SW) Articles by BIBREF9 of about 711MB and the Billion Word (BW) of 3 9GB by BIBREF10  The corpus used for sentiment analysis is the IMDb dataset of movie reviews by BIBREF11 while that for NER is Groningen Meaning Bank (GMB) by BIBREF12, containing 47,959 sentence samples  The IMDb dataset used has a total of 25,000 sentences with half being positive sentiments and the other half being negative sentiments  The GMB dataset has 17 labels, with 9 main labels and 2 context tags  It is however unbalanced due to the high percentage of tokens with the label 'O'  This skew in the GMB dataset is typical with NER datasets 
The objective of this work is to determine the optimal combinations of word2vec hyper-parameters for intrinsic evaluation (semantic and syntactic analogies) and extrinsic evaluation tasks (BIBREF13, BIBREF14), like SA and NER  It is not our objective in this work to record sota results  Some of the main contributions of this research are the empirical establishment of optimal combinations of word2vec hyper-parameters for NLP tasks, discovering the behaviour of quality of vectors viz-a-viz increasing dimensions and the confirmation of embeddings being task-specific for the downstream  The rest of this paper is organised as follows: the literature review that briefly surveys distributed representation of words, particularly word2vec; the methodology employed in this research work; the results obtained and the conclusion 
Literature Review
Breaking away from the non-distributed (high-dimensional, sparse) representations of words, typical of traditional bag-of-words or one-hot-encoding (BIBREF15), BIBREF0 created word2vec  Word2Vec consists of two shallow neural network architectures: continuous skipgram and CBoW  It uses distributed (low-dimensional, dense) representations of words that group similar words  This new model traded the complexity of deep neural network architectures, by other researchers, for more efficient training over large corpora  Its architectures have two training algorithms: negative sampling and hierarchical softmax (BIBREF16)  The released model was trained on Google news dataset of 100 billion words  Implementations of the model have been undertaken by researchers in the programming languages Python and C++, though the original was done in C (BIBREF17) 
Continuous skipgram predicts (by maximizing classification of) words before and after the center word, for a given range  Since distant words are less connected to a center word in a sentence, less weight is assigned to such distant words in training  CBoW, on the other hand, uses words from the history and future in a sequence, with the objective of correctly classifying the target word in the middle  It works by projecting all history or future words within a chosen window into the same position, averaging their vectors  Hence, the order of words in the history or future does not influence the averaged vector  This is similar to the traditional bag-of-words, which is oblivious of the order of words in its sequence  A log-linear classifier is used in both architectures (BIBREF0)  In further work, they extended the model to be able to do phrase representations and subsample frequent words (BIBREF16)  Being a NNLM, word2vec assigns probabilities to words in a sequence, like other NNLMs such as feedforward networks or recurrent neural networks (BIBREF15)  Earlier models like latent dirichlet allocation (LDA) and latent semantic analysis (LSA) exist and effectively achieve low dimensional vectors by matrix factorization (BIBREF18, BIBREF19) 
It's been shown that word vectors are beneficial for NLP tasks (BIBREF15), such as sentiment analysis and named entity recognition  Besides, BIBREF0 showed with vector space algebra that relationships among words can be evaluated, expressing the quality of vectors produced from the model  The famous, semantic example: vector("King") - vector("Man") + vector("Woman") $\approx $ vector("Queen") can be verified using cosine distance  Another type of semantic meaning is the relationship between a capital city and its corresponding country  Syntactic relationship examples include plural verbs and past tense, among others  Combination of both syntactic and semantic analyses is possible and provided (totaling over 19,000 questions) as Google analogy test set by BIBREF0  WordSimilarity-353 test set is another analysis tool for word vectors (BIBREF20)  Unlike Google analogy score, which is based on vector space algebra, WordSimilarity is based on human expert-assigned semantic similarity on two sets of English word pairs  Both tools rank from 0 (totally dissimilar) to 1 (very much similar or exact, in Google analogy case) 
A typical artificial neural network (ANN) has very many hyper-parameters which may be tuned  Hyper-parameters are values which may be manually adjusted and include vector dimension size, type of algorithm and learning rate (BIBREF19)  BIBREF0 tried various hyper-parameters with both architectures of their model, ranging from 50 to 1,000 dimensions, 30,000 to 3,000,000 vocabulary sizes, 1 to 3 epochs, among others  In our work, we extended research to 3,000 dimensions  Different observations were noted from the many trials  They observed diminishing returns after a certain point, despite additional dimensions or larger, unstructured training data  However, quality increased when both dimensions and data size were increased together  Although BIBREF16 pointed out that choice of optimal hyper-parameter configurations depends on the NLP problem at hand, they identified the most important factors are architecture, dimension size, subsampling rate, and the window size  In addition, it has been observed that variables like size of datasets improve the quality of word vectors and, potentially, performance on downstream tasks (BIBREF21, BIBREF0) 
Methodology
The models were generated in a shared cluster running Ubuntu 16 with 32 CPUs of 32x Intel Xeon 4110 at 2 1GHz  Gensim (BIBREF17) python library implementation of word2vec was used with parallelization to utilize all 32 CPUs  The downstream experiments were run on a Tesla GPU on a shared DGX cluster running Ubuntu 18  Pytorch deep learning framework was used  Gensim was chosen because of its relative stability, popular support and to minimize the time required in writing and testing a new implementation in python from scratch 
To form the vocabulary, words occurring less than 5 times in the corpora were dropped, stop words removed using the natural language toolkit (NLTK) (BIBREF22) and data pre-processing carried out
Context 3: Introduction
Neural networks have been successfully used to describe images with text using sequence-to-sequence models BIBREF0  However, the results are simple and dry captions which are one or two phrases long  Humans looking at a painting see more than just objects  Paintings stimulate sentiments, metaphors and stories as well  Therefore, our goal is to have a neural network describe the painting artistically in a style of choice  As a proof of concept, we present a model which generates Shakespearean prose for a given painting as shown in Figure FIGREF1  Accomplishing this task is difficult with traditional sequence to sequence models since there does not exist a large collection of Shakespearean prose which describes paintings: Shakespeare's works describes a single painting shown in Figure FIGREF3  Fortunately we have a dataset of modern English poems which describe images BIBREF1 and line-by-line modern paraphrases of Shakespeare's plays BIBREF2  Our solution is therefore to combine two separately trained models to synthesize Shakespearean prose for a given painting 
Introduction ::: Related work
A general end-to-end approach to sequence learning BIBREF3 makes minimal assumptions on the sequence structure  This model is widely used in tasks such as machine translation, text summarization, conversational modeling, and image captioning  A generative model using a deep recurrent architecture BIBREF0 has also beeen used for generating phrases describing an image  The task of synthesizing multiple lines of poetry for a given image BIBREF1 is accomplished by extracting poetic clues from images  Given the context image, the network associates image attributes with poetic descriptions using a convolutional neural net  The poem is generated using a recurrent neural net which is trained using multi-adversarial training via policy gradient 
Transforming text from modern English to Shakespearean English using text "style transfer" is challenging  An end to end approach using a sequence-to-sequence model over a parallel text corpus BIBREF2 has been proposed based on machine translation  In the absence of a parallel text corpus, generative adversarial networks (GANs) have been used, which simultaneously train two models: a generative model which captures the data distribution, and a discriminative model which evaluates the performance of the generator  Using a target domain language model as a discriminator has also been employed BIBREF4, providing richer and more stable token-level feedback during the learning process  A key challenge in both image and text style transfer is separating content from style BIBREF5, BIBREF6, BIBREF7  Cross-aligned auto-encoder models have focused on style transfer using non-parallel text BIBREF7  Recently, a fine grained model for text style transfer has been proposed BIBREF8 which controls several factors of variation in textual data by using back-translation  This allows control over multiple attributes, such as gender and sentiment, and fine-grained control over the trade-off between content and style 
Methods
We use a total three datasets: two datasets for generating an English poem from an image, and Shakespeare plays and their English translations for text style transfer 
We train a model for generating poems from images based on two datasets BIBREF1  The first dataset consists of image and poem pairs, namely a multi-modal poem dataset (MultiM-Poem), and the second dataset is a large poem corpus, namely a uni-modal poem dataset (UniM-Poem)  The image and poem pairs are extended by adding the nearest three neighbor poems from the poem corpus without redundancy, and an extended image and poem pair dataset is constructed and denoted as MultiM-Poem(Ex)BIBREF1

ground_truth: ecause this process is similar to the cloze task in BERT's pre-train process, therefore by using the ability of the contextual language model the decoder can generate more fluent and natural sequences.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What sentiment analysis dataset is used?
Context 2:  Table TABREF2 describes most hyper-parameters explored for each dataset  In all, 80 runs (of about 160 minutes) were conducted for the 15MB Wiki Abstract dataset with 80 serialized models totaling 15 136GB while 80 runs (for over 320 hours) were conducted for the 711MB SW dataset, with 80 serialized models totaling over 145GB  Experiments for all combinations for 300 dimensions were conducted on the 3 9GB training set of the BW corpus and additional runs for other dimensions for the window 8 + skipgram + heirarchical softmax combination to verify the trend of quality of word vectors as dimensions are increased 
Google (semantic and syntactic) analogy tests and WordSimilarity-353 (with Spearman correlation) by BIBREF20 were chosen for intrinsic evaluations  They measure the quality of word vectors  The analogy scores are averages of both semantic and syntactic tests  NER and SA were chosen for extrinsic evaluations  The GMB dataset for NER was trained in an LSTM network, which had an embedding layer for input  The network diagram is shown in fig  FIGREF4  The IMDb dataset for SA was trained in a BiLSTM network, which also used an embedding layer for input  Its network diagram is given in fig  FIGREF4  It includes an additional hidden linear layer  Hyper-parameter details of the two networks for the downstream tasks are given in table TABREF3  The metrics for extrinsic evaluation include F1, precision, recall and accuracy scores  In both tasks, the default pytorch embedding was tested before being replaced by pre-trained embeddings released by BIBREF0 and ours  In each case, the dataset was shuffled before training and split in the ratio 70:15:15 for training, validation (dev) and test sets  Batch size of 64 was used  For each task, experiments for each embedding was conducted four times and an average value calculated and reported in the next section
Results and Discussion
Table TABREF5 summarizes key results from the intrinsic evaluations for 300 dimensions  Table TABREF6 reveals the training time (in hours) and average embedding loading time (in seconds) representative of the various models used  Tables TABREF11 and TABREF12 summarize key results for the extrinsic evaluations  Figures FIGREF7, FIGREF9, FIGREF10, FIGREF13 and FIGREF14 present line graph of the eight combinations for different dimension sizes for Simple Wiki, trend of Simple Wiki and Billion Word corpora over several dimension sizes, analogy score comparison for models across datasets, NER mean F1 scores on the GMB dataset and SA mean F1 scores on the IMDb dataset, respectively  Combination of the skipgram using hierarchical softmax and window size of 8 for 300 dimensions outperformed others in analogy scores for the Wiki Abstract  However, its results are so poor, because of the tiny file size, they're not worth reporting here  Hence, we'll focus on results from the Simple Wiki and Billion Word corpora 
Best combination changes when corpus size increases, as will be noticed from table TABREF5  In terms of analogy score, for 10 epochs, w8s0h0 performs best while w8s1h0 performs best in terms of WordSim and corresponding Spearman correlation  Meanwhile, increasing the corpus size to BW, w4s1h0 performs best in terms of analogy score while w8s1h0 maintains its position as the best in terms of WordSim and Spearman correlation  Besides considering quality metrics, it can be observed from table TABREF6 that comparative ratio of values between the models is not commensurate with the results in intrinsic or extrinsic values, especially when we consider the amount of time and energy spent, since more training time results in more energy consumption (BIBREF23) 
Information on the length of training time for the released Mikolov model is not readily available  However, it's interesting to note that their presumed best model, which was released is also s1h0  Its analogy score, which we tested and report, is confirmed in their paper  It beats our best models in only analogy score (even for Simple Wiki), performing worse in others  This is inspite of using a much bigger corpus of 3,000,000 vocabulary size and 100 billion words while Simple Wiki had vocabulary size of 367,811 and is 711MB  It is very likely our analogy scores will improve when we use a much larger corpus, as can be observed from table TABREF5, which involves just one billion words 
Although the two best combinations in analogy (w8s0h0 & w4s0h0) for SW, as shown in fig  FIGREF7, decreased only slightly compared to others with increasing dimensions, the increased training time and much larger serialized model size render any possible minimal score advantage over higher dimensions undesirable  As can be observed in fig  FIGREF9, from 100 dimensions, scores improve but start to drop after over 300 dimensions for SW and after over 400 dimensions for BW  More becomes worse! This trend is true for all combinations for all tests  Polynomial interpolation may be used to determine the optimal dimension in both corpora  Our models are available for confirmation and source codes are available on github 
With regards to NER, most pretrained embeddings outperformed the default pytorch embedding, with our BW w4s1h0 model (which is best in BW analogy score) performing best in F1 score and closely followed by BIBREF0 model  On the other hand, with regards to SA, pytorch embedding outperformed the pretrained embeddings but was closely followed by our SW w8s0h0 model (which also had the best SW analogy score)  BIBREF0 performed second worst of all, despite originating from a very huge corpus  The combinations w8s0h0 & w4s0h0 of SW performed reasonably well in both extrinsic tasks, just as the default pytorch embedding did 
Conclusion
This work analyses, empirically, optimal combinations of hyper-parameters for embeddings, specifically for word2vec  It further shows that for downstream tasks, like NER and SA, there's no silver bullet! However, some combinations show strong performance across tasks  Performance of embeddings is task-specific and high analogy scores do not necessarily correlate positively with performance on downstream tasks  This point on correlation is somewhat similar to results by BIBREF24 and BIBREF14  It was discovered that increasing dimension size depreciates performance after a point  If strong considerations of saving time, energy and the environment are made, then reasonably smaller corpora may suffice or even be better in some cases  The on-going drive by many researchers to use ever-growing data to train deep neural networks can benefit from the findings of this work  Indeed, hyper-parameter choices are very important in neural network systems (BIBREF19) 
Future work that may be investigated are performance of other architectures of word or sub-word embeddings, the performance and comparison of embeddings applied to languages other than English and how embeddings perform in other downstream tasks  In addition, since the actual reason for the changes in best model as corpus size increases is not clear, this will also be suitable for further research 
The work on this project is partially funded by Vinnova under the project number 2019-02996 "Språkmodeller för svenska myndigheter"
Acronyms
Context 3: 
We use a collection of line-by-line modern paraphrases for 16 of Shakespeare’s plays BIBREF2, for training a style transfer network from English poems to Shakespearean prose  We use 18,395 sentences from the training data split  We keep 1,218 sentences in the validation data set and 1,462 sentences in our test set 
Methods ::: Image To Poem Actor-Critic Model
For generating a poem from images we use an existing actor-critic architecture BIBREF1  This involves 3 parallel CNNs: an object CNN, sentiment CNN, and scene CNN, for feature extraction  These features are combined with a skip-thought model which provides poetic clues, which are then fed into a sequence-to-sequence model trained by policy gradient with 2 discriminator networks for rewards  This as a whole forms a pipeline that takes in an image and outputs a poem as shown on the top left of Figure FIGREF4  A CNN-RNN generative model acts as an agent  The parameters of this agent define a policy whose execution determines which word is selected as an action  When the agent selects all words in a poem, it receives a reward  Two discriminative networks, shown on the top right of Figure FIGREF4, are defined to serve as rewards concerning whether the generated poem properly describes the input image and whether the generated poem is poetic  The goal of the poem generation model is to generate a sequence of words as a poem for an image to maximize the expected return 
Methods ::: Shakespearizing Poetic Captions
For Shakespearizing modern English texts, we experimented with various types of sequence to sequence models  Since the size of the parallel translation data available is small, we leverage a dictionary providing a mapping between Shakespearean words and modern English words to retrofit pre-trained word embeddings  Incorporating this extra information improves the translation task  The large number of shared word types between the source and target sentences indicates that sharing the representation between them is beneficial 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with Attention
We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences  We experimented with two different types of attention: global attention BIBREF9, in which the model makes use of the output from the encoder and decoder for the current time step only, and Bahdanau attention BIBREF10, where computing attention requires the output of the decoder from the prior time step  We found that global attention performs better in practice for our task of text style transfer 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with a Pointer Network
Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input  Moreover, there are lot of proper nouns and rare words which might not be predicted by a vanilla sequence to sequence model 
Methods ::: Shakespearizing Poetic Captions ::: Prediction
For both seq2seq models, we use the attention matrices returned at each decoder time step during inference, to compute the next word in the translated sequence if the decoder output at the current time step is the UNK token  We replace the UNKs in the target output with the highest aligned, maximum attention, source word  The seq2seq model with global attention gives the best results with an average target BLEU score of 29 65 on the style transfer dataset, compared with an average target BLEU score of 26 97 using the seq2seq model with pointer networks 
Results
We perform a qualitative analysis of the Shakespearean prose generated for the input paintings

ground_truth: IMDb dataset of movie reviews

IMDb

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What models are used for painting embedding and what for language style transfer?
Context 3:  We conducted a survey, in which we presented famous paintings including those shown in Figures FIGREF1 and FIGREF10 and the corresponding Shakespearean prose generated by the model, and asked 32 students to rate them on the basis of content, creativity and similarity to Shakespearean style on a Likert scale of 1-5  Figure FIGREF12 shows the result of our human evaluation 
The average content score across the paintings is 3 7 which demonstrates that the prose generated is relevant to the painting  The average creativity score is 3 9 which demonstrates that the model captures more than basic objects in the painting successfully using poetic clues in the scene  The average style score is 3 9 which demonstrates that the prose generated is perceived to be in the style of Shakespeare 
We also perform a quantitative analysis of style transfer by generating BLEU scores for the model output using the style transfer dataset  The variation of the BLEU scores with the source sentence lengths is shown in Figure FIGREF11  As expected, the BLEU scores decrease with increase in source sentence lengths 
Results ::: Implementation
All models were trained on Google Colab with a single GPU using Python 3 6 and Tensorflow 2 0  The number of hidden units for the encoder and decoder is 1,576 and 256 for seq2seq with global attention and seq2seq with pointer networks respectively  Adam optimizer was used with the default learning rate of 0 001  The model was trained for 25 epochs  We use pre-trained retrofitted word embeddings of dimension 192 
Results ::: Limitations
Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score  This happens when the style transfer dataset does not have similar words in the training set of sentences  A solution would be to expand the style transfer dataset, for a better representation of the poem data 
Results ::: Conclusions and Future Work
In conclusion, combining two pipelines with an intermediate representation works well in practice  We observe that a CNN-RNN based image-to-poem net combined with a seq2seq model with parallel text corpus for text style transfer synthesizes Shakespeare-style prose for a given painting  For the seq2seq model used, we observe that it performs better in practice using global attention as compared with local attention  We make our models and code publicly available BIBREF12  In future work we would like to experiment with GANs in the absence of non-parallel datasets, so that we can use varied styles for text style transfer  We would also like to experiment with cross aligned auto-encoders, which form a latent content representation, to efficiently separate style and content 

ground_truth: generating a poem from images we use an existing actor-critic architecture, various types of sequence to sequence models