Introduction
There has been significant research on style transfer, with the goal of changing the style of text while preserving its semantic content. The alternative where semantics are adjusted while keeping style intact, which we call semantic text exchange (STE), has not been investigated to the best of our knowledge. Consider the following example, where the replacement entity defines the new semantic context:
Original Text: It is sunny outside! Ugh, that means I must wear sunscreen. I hate being sweaty and sticky all over. Replacement Entity: weather = rainy Desired Text: It is rainy outside! Ugh, that means I must bring an umbrella. I hate being wet and having to carry it around.
The weather within the original text is sunny, whereas the actual weather may be rainy. Not only is the word sunny replaced with rainy, but the rest of the text's content is changed while preserving its negative sentiment and fluency. With the rise of natural language processing (NLP) has come an increased demand for massive amounts of text data. Manually collecting and scraping data requires a significant amount of time and effort, and data augmentation techniques for NLP are limited compared to fields such as computer vision. STE can be used for text data augmentation by producing various modifications of a piece of text that differ in semantic content.
Another use of STE is in building emotionally aligned chatbots and virtual assistants. This is useful for reasons such as marketing, overall enjoyment of interaction, and mental health therapy. However, due to limited data with emotional content in specific semantic contexts, the generated text may contain incorrect semantic content. STE can adjust text semantics (e.g. to align with reality or a specific task) while preserving emotions.
One specific example is the development of virtual assistants with adjustable socio-emotional personalities in the effort to construct assistive technologies for persons with cognitive disabilities. Adjusting the emotional delivery of text in subtle ways can have a strong effect on the adoption of the technologies BIBREF0. It is challenging to transfer style this subtly due to lack of datasets on specific topics with consistent emotions. Instead, large datasets of emotionally consistent interactions not confined to specific topics exist. Hence, it is effective to generate text with a particular emotion and then adjust its semantics.
We propose a pipeline called SMERTI (pronounced `smarty') for STE. Combining entity replacement (ER), similarity masking (SM), and text infilling (TI), SMERTI can modify the semantic content of text. We define a metric called the Semantic Text Exchange Score (STES) that evaluates the overall ability of a model to perform STE, and an adjustable parameter masking (replacement) rate threshold (MRT/RRT) that can be used to control the amount of semantic change.
We evaluate on three datasets: Yelp and Amazon reviews BIBREF1, and Kaggle news headlines BIBREF2. We implement three baseline models for comparison: Noun WordNet Semantic Text Exchange Model (NWN-STEM), General WordNet Semantic Text Exchange Model (GWN-STEM), and Word2Vec Semantic Text Exchange Model (W2V-STEM).
We illustrate the STE performance of two SMERTI variations on the datasets, demonstrating outperformance of the baselines and pipeline stability. We also run a human evaluation supporting our results. We analyze the results in detail and investigate relationships between the semantic change, fluency, sentiment, and MRT/RRT. Our major contributions can be summarized as:
We define a new task called semantic text exchange (STE) with increasing importance in NLP applications that modifies text semantics while preserving other aspects such as sentiment.
We propose a pipeline SMERTI capable of multi-word entity replacement and text infilling, and demonstrate its outperformance of baselines.
We define an evaluation metric for overall performance on semantic text exchange called the Semantic Text Exchange Score (STES).
Related Work ::: Word and Sentence-level Embeddings
Word2Vec BIBREF3, BIBREF4 allows for analogy representation through vector arithmetic. We implement a baseline (W2V-STEM) using this technique. The Universal Sentence Encoder (USE) BIBREF5 encodes sentences and is trained on a variety of web sources and the Stanford Natural Language Inference corpus BIBREF6. Flair embeddings BIBREF7 are based on architectures such as BERT BIBREF8. We use USE for SMERTI as it is designed for transfer learning and shows higher performance on textual similarity tasks compared to other models BIBREF9.
Related Work ::: Text Infilling
Text infilling is the task of filling in missing parts of sentences called masks. MaskGAN BIBREF10 is restricted to a single word per mask token, while SMERTI is capable of variable length infilling for more flexible output. BIBREF11 uses a transformer-based architecture. They fill in random masks, while SMERTI fills in masks guided by semantic similarity, resulting in more natural infilling and fulfillment of the STE task.
Related Work ::: Style and Sentiment Transfer
Notable works in style/sentiment transfer include BIBREF12, BIBREF13, BIBREF14, BIBREF15. They attempt to learn latent representations of various text aspects such as its context and attributes, or separate style from content and encode them into hidden representations. They then use an RNN decoder to generate a new sentence given a targeted sentiment attribute.
Related Work ::: Review Generation
BIBREF16 generates fake reviews from scratch using language models. BIBREF17, BIBREF18, BIBREF19 generate reviews from scratch given auxiliary information (e.g. the item category and star rating). BIBREF20 generates reviews using RNNs with two components: generation from scratch and review customization (Algorithm 2 in BIBREF20). They define review customization as modifying the generated review to fit a new topic or context, such as from a Japanese restaurant to an Italian one. They condition on a keyword identifying the desired context, and replace similar nouns with others using WordNet BIBREF21. They require a “reference dataset" (required to be “on topic"; easy enough for restaurant reviews, but less so for arbitrary conversational agents). As noted by BIBREF19, the method of BIBREF20 may also replace words independently of context. We implement their review customization algorithm (NWN-STEM) and a modified version (GWN-STEM) as baseline models.
SMERTI ::: Overview
The task is to transform a corpus $C$ of lines of text $S_i$ and associated replacement entities $RE_i:C = \lbrace (S_1,RE_1),(S_2,RE_2),\ldots , (S_n, RE_n)\rbrace $ to a modified corpus $\hat{C} = \lbrace \hat{S}_1,\hat{S}_2,\ldots ,\hat{S}_n\rbrace $, where $\hat{S}_i$ are the original text lines $S_i$ replaced with $RE_i$ and overall semantics adjusted. SMERTI consists of the following modules, shown in Figure FIGREF15:
Entity Replacement Module (ERM): Identify which word(s) within the original text are best replaced with the $RE$, which we call the Original Entity ($OE$). We replace $OE$ in $S$ with $RE$. We call this modified text $S^{\prime }$.
Similarity Masking Module (SMM): Identify words/phrases in $S^{\prime }$ similar to $OE$ and replace them with a [mask]. Group adjacent [mask]s into a single one so we can fill a variable length of text into each. We call this masked text $S^{\prime \prime }$.
Text Infilling Module (TIM): Fill in [mask] tokens with text that better suits the $RE$. This will modify semantics in the rest of the text. This final output text is called $\hat{S}$.
SMERTI ::: Entity Replacement Module (ERM)
For entity replacement, we use a combination of the Universal Sentence Encoder BIBREF5 and Stanford Parser BIBREF22.
SMERTI ::: Entity Replacement Module (ERM) ::: Stanford Parser
The Stanford Parser is a constituency parser that determines the grammatical structure of sentences, including phrases and part-of-speech (POS) labelling. By feeding our $RE$ through the parser, we are able to determine its parse-tree. Iterating through the parse-tree and its sub-trees, we can obtain a list of constituent tags for the $RE$. We then feed our input text $S$ through the parser, and through a similar process, we can obtain a list of leaves (where leaves under a single label are concatenated) that are equal or similar to any of the $RE$ constituent tags. This generates a list of entities having the same (or similar) grammatical structure as the $RE$, and are likely candidates for the $OE$. We then feed these entities along with the $RE$ into the Universal Sentence Encoder (USE).
SMERTI ::: Entity Replacement Module (ERM) ::: Universal Sentence Encoder (USE)
The USE is a sentence-level embedding model that comes with a deep averaging network (DAN) and transformer model BIBREF5. We choose the transformer model as these embeddings take context into account, and the exact same word/phrase will have a different embedding depending on its context and surrounding words.
We compute the semantic similarity between two embeddings $u$ and $v$: $sim(u,v)$, using the angular (cosine) distance, defined as: $\cos (\theta _{u,v}) = (u\cdot v)/(||u|| ||v||)$, such that $sim(u,v) = 1-\frac{1}{\pi }arccos(\cos (\theta _{u,v}))$. Results are in $[0,1]$, with higher values representing greater similarity.
Using USE and the above equation, we can identify words/phrases within the input text $S$ which are most similar to $RE$. To assist with this, we use the Stanford Parser as described above to obtain a list of candidate entities. In the rare case that this list is empty, we feed in each word of $S$ into USE, and identify which word is the most similar to $RE$. We then replace the most similar entity or word ($OE$) with the $RE$ and generate $S^{\prime }$.
An example of this entity replacement process is in Figure FIGREF18. Two parse-trees are shown: for $RE$ (a) and $S$ (b) and (c). Figure FIGREF18(d) is a semantic similarity heat-map generated from the USE embeddings of the candidate $OE$s and $RE$, where values are similarity scores in the range $[0,1]$.
As seen in Figure FIGREF18(d), we calculate semantic similarities between $RE$ and entities within $S$ which have noun constituency tags. Looking at the row for our $RE$ restaurant, the most similar entity (excluding itself) is hotel. We can then generate:
$S^{\prime }$ = i love this restaurant ! the beds are comfortable and the service is great !
SMERTI ::: Similarity Masking Module (SMM)
Next, we mask words similar to $OE$ to generate $S^{\prime \prime }$ using USE. We look at semantic similarities between every word in $S$ and $OE$, along with semantic similarities between $OE$ and the candidate entities determined in the previous ERM step to broaden the range of phrases our module can mask. We ignore $RE$, $OE$, and any entities or phrases containing $OE$ (for example, `this hotel').
After determining words similar to the $OE$ (discussed below), we replace each of them with a [mask] token. Next, we replace [mask] tokens adjacent to each other with a single [mask].
We set a base similarity threshold (ST) that selects a subset of words to mask. We compare the actual fraction of masked words to the masking rate threshold (MRT), as defined by the user, and increase ST in intervals of $0.05$ until the actual masking rate falls below the MRT. Some sample masked outputs ($S^{\prime \prime }$) using various MRT-ST combinations for the previous example are shown in Table TABREF21 (more examples in Appendix A).
The MRT is similar to the temperature parameter used to control the “novelty” of generated text in works such as BIBREF20. A high MRT means the user wants to generate text very semantically dissimilar to the original, and may be desired in cases such as creating a lively chatbot or correcting text that is heavily incorrect semantically. A low MRT means the user wants to generate text semantically similar to the original, and may be desired in cases such as text recovery, grammar correction, or correcting a minor semantic error in text. By varying the MRT, various pieces of text that differ semantically in subtle ways can be generated, assisting greatly with text data augmentation. The MRT also affects sentiment and fluency, as we show in Section SECREF59.
SMERTI ::: Text Infilling Module (TIM)
We use two seq2seq models for our TIM: an RNN (recurrent neural network) model BIBREF23 (called SMERTI-RNN), and a transformer model (called SMERTI-Transformer).
SMERTI ::: Text Infilling Module (TIM) ::: Bidirectional RNN with Attention
We use a bidirectional variant of the GRU BIBREF24, and hence two RNNs for the encoder: one reads the input sequence in standard sequential order, and the other is fed this sequence in reverse. The outputs are summed at each time step, giving us the ability to encode information from both past and future context.
The decoder generates the output in a sequential token-by-token manner. To combat information loss, we implement the attention mechanism BIBREF25. We use a Luong attention layer BIBREF26 which uses global attention, where all the encoder's hidden states are considered, and use the decoder's current time-step hidden state to calculate attention weights. We use the dot score function for attention, where $h_t$ is the current target decoder state and $\bar{h}_s$ is all encoder states: $score(h_t,\bar{h}_s)=h_t^T\bar{h}_s$.
SMERTI ::: Text Infilling Module (TIM) ::: Transformer
Our second model makes use of the transformer architecture, and our implementation replicates BIBREF27. We use an encoder-decoder structure with a multi-head self-attention token decoder to condition on information from both past and future context. It maps a query and set of key-value pairs to an output. The queries and keys are of dimension $d_k$, and values of dimension $d_v$. To compute the attention, we pack a set of queries, keys, and values into matrices $Q$, $K$, and $V$, respectively. The matrix of outputs is computed as:
Multi-head attention allows the model to jointly attend to information from different positions. The decoder can make use of both local and global semantic information while filling in each [mask].
Experiment ::: Datasets
We train our two TIMs on the three datasets. The Amazon dataset BIBREF1 contains over 83 million user reviews on products, with duplicate reviews removed. The Yelp dataset includes over six million user reviews on businesses. The news headlines dataset from Kaggle contains approximately $200,000$ news headlines from 2012 to 2018 obtained from HuffPost BIBREF2.
We filter the text to obtain reviews and headlines which are English, do not contain hyperlinks and other obvious noise, and are less than 20 words long. We found that many longer than twenty words ramble on and are too verbose for our purposes. Rather than filtering by individual sentences we keep each text in its entirety so SMERTI can learn to generate multiple sentences at once. We preprocess the text by lowercasing and removing rare/duplicate punctuation and space.
For Amazon and Yelp, we treat reviews greater than three stars as containing positive sentiment, equal to three stars as neutral, and less than three stars as negative. For each training and testing set, we include an equal number of randomly selected positive and negative reviews, and half as many neutral reviews. This is because neutral reviews only occupy one out of five stars compared to positive and negative which occupy two each. Our dataset statistics can be found in Appendix B.
Experiment ::: Experiment Details
To set up our training and testing data for text infilling, we mask the text. We use a tiered masking approach: for each dataset, we randomly mask 15% of the words in one-third of the lines, 30% of the words in another one-third, and 45% in the remaining one-third. These masked texts serve as the inputs, while the original texts serve as the ground-truth. This allows our TIM models to learn relationships between masked words and relationships between masked and unmasked words.
The bidirectional RNN decoder fills in blanks one by one, with the objective of minimizing the cross entropy loss between its output and the ground-truth. We use a hidden size of 500, two layers for the encoder and decoder, teacher-forcing ratio of 1.0, learning rate of 0.0001, dropout of 0.1, batch size of 64, and train for up to 40 epochs.
For the transformer, we use scaled dot-product attention and the same hyperparameters as BIBREF27. We use the Adam optimizer BIBREF28 with $\beta _1 = 0.9, \beta _2 = 0.98$, and $\epsilon = 10^{-9}$. As in BIBREF27, we increase the $learning\_rate$ linearly for the first $warmup\_steps$ training steps, and then decrease the $learning\_rate$ proportionally to the inverse square root of the step number. We set $factor=1$ and use $warmup\_steps = 2000$. We use a batch size of 4096, and we train for up to 40 epochs.
Experiment ::: Baseline Models
We implement three models to benchmark against. First is NWN-STEM (Algorithm 2 from BIBREF20). We use the training sets as the “reference review sets" to extract similar nouns to the $RE$ (using MINsim = 0.1). We then replace nouns in the text similar to the $RE$ with nouns extracted from the associated reference review set.
Secondly, we modify NWN-STEM to work for verbs and adjectives, and call this GWN-STEM. From the reference review sets, we extract similar nouns, verbs, and adjectives to the $RE$ (using MINsim = 0.1), where the $RE$ is now not restricted to being a noun. We replace nouns, verbs, and adjectives in the text similar to the $RE$ with those extracted from the associated reference review set.
Lastly, we implement W2V-STEM using Gensim BIBREF29. We train uni-gram Word2Vec models for single word $RE$s, and four-gram models for phrases. Models are trained on the training sets. We use cosine similarity to determine the most similar word/phrase in the input text to $RE$, which is the replaced $OE$. For all other words/phrases, we calculate $w_{i}^{\prime } = w_{i} - w_{OE} + w_{RE}$, where $w_{i}$ is the original word/phrase's embedding vector, $w_{OE}$ is the $OE$'s, $w_{RE}$ is the $RE$'s, and $w_{i}^{\prime }$ is the resulting embedding vector. The replacement word/phrase is $w_{i}^{\prime }$'s nearest neighbour. We use similarity thresholds to adjust replacement rates (RR) and produce text under various replacement rate thresholds (RRT).
Evaluation ::: Evaluation Setup
We manually select 10 nouns, 10 verbs, 10 adjectives, and 5 phrases from the top 10% most frequent words/phrases in each test set as our evaluation $RE$s. We filter the verbs and adjectives through a list of sentiment words BIBREF30 to ensure we do not choose $RE$s that would obviously significantly alter the text's sentiment.
For each evaluation $RE$, we choose one-hundred lines from the corresponding test set that does not already contain $RE$. We choose lines with at least five words, as many with less carry little semantic meaning (e.g. `Great!', `It is okay'). For Amazon and Yelp, we choose 50 positive and 50 negative lines per $RE$. We repeat this process three times, resulting in three sets of 1000 lines per dataset per POS (excluding phrases), and three sets of 500 lines per dataset for phrases. Our final results are averaged metrics over these three sets.
For SMERTI-Transformer, SMERTI-RNN, and W2V-STEM, we generate four outputs per text for MRT/RRT of 20%, 40%, 60%, and 80%, which represent upper-bounds on the percentage of the input that can be masked and/or replaced. Note that NWN-STEM and GWN-STEM can only evaluate on limited POS and their maximum replacement rates are limited. We select MINsim values of 0.075 and 0 for nouns and 0.1 and 0 for verbs, as these result in replacement rates approximately equal to the actual MR/RR of the other models' outputs for 20% and 40% MRT/RRT, respectively.
Evaluation ::: Key Evaluation Metrics
Fluency (SLOR) We use syntactic log-odds ratio (SLOR) BIBREF31 for sentence level fluency and modify from their word-level formula to character-level ($SLOR_{c}$). We use Flair perplexity values from a language model trained on the One Billion Words corpus BIBREF32:
where $|S|$ and $|w|$ are the character lengths of the input text $S$ and the word $w$, respectively, $p_M(S)$ and $p_M(w)$ are the probabilities of $S$ and $w$ under the language model $M$, respectively, and $PPL_S$ and $PPL_w$ are the character-level perplexities of $S$ and $w$, respectively. SLOR (from hereon we refer to character-level SLOR as simply SLOR) measures aspects of text fluency such as grammaticality. Higher values represent higher fluency.
We rescale resulting SLOR values to the interval [0,1] by first fitting and normalizing a Gaussian distribution. We then truncate normalized data points outside [-3,3], which shifts approximately 0.69% of total data. Finally, we divide each data point by six and add 0.5 to each result.
Sentiment Preservation Accuracy (SPA) is defined as the percentage of outputs that carry the same sentiment as the input. We use VADER BIBREF33 to evaluate sentiment as positive, negative, or neutral. It handles typos, emojis, and other aspects of online text. Content Similarity Score (CSS) ranges from 0 to 1 and indicates the semantic similarity between generated text and the $RE$. A value closer to 1 indicates stronger semantic exchange, as the output is closer in semantic content to the $RE$. We also use the USE for this due to its design and strong performance as previously mentioned.
Evaluation ::: Semantic Text Exchange Score (STES)
We come up with a single score to evaluate overall performance of a model on STE that combines the key evaluation metrics. It uses the harmonic mean, similar to the F1 score (or F-score) BIBREF34, BIBREF35, and we call it the Semantic Text Exchange Score (STES):
where $A$ is SPA, $B$ is SLOR, and $C$ is CSS. STES ranges between 0 and 1, with scores closer to 1 representing higher overall performance. Like the F1 score, STES penalizes models which perform very poorly in one or more metrics, and favors balanced models achieving strong results in all three.
Evaluation ::: Automatic Evaluation Results
Table TABREF38 shows overall average results by model. Table TABREF41 shows outputs for a Yelp example.
As observed from Table TABREF41 (see also Appendix F), SMERTI is able to generate high quality output text similar to the $RE$ while flowing better than other models' outputs. It can replace entire phrases and sentences due to its variable length infilling. Note that for nouns, the outputs from GWN-STEM and NWN-STEM are equivalent.
Evaluation ::: Human Evaluation Setup
We conduct a human evaluation with eight participants, 6 males and 2 females, that are affiliated project researchers aged 20-39 at the University of Waterloo. We randomly choose one evaluation line for a randomly selected word or phrase for each POS per dataset. The input text and each model's output (for 40% MRT/RRT - chosen as a good middle ground) for each line is presented to participants, resulting in a total of 54 pieces of text, and rated on the following criteria from 1-5:
RE Match: “How related is the entire text to the concept of [X]", where [X] is a word or phrase (1 - not at all related, 3 - somewhat related, 5 - very related). Note here that [X] is a given $RE$.
Fluency: “Does the text make sense and flow well?" (1 - not at all, 3 - somewhat, 5 - very)
Sentiment: “How do you think the author of the text was feeling?" (1 - very negative, 3 - neutral, 5 - very positive)
Each participant evaluates every piece of text. They are presented with a single piece of text at a time, with the order of models, POS, and datasets completely randomized.
Evaluation ::: Human Evaluation Results
Average human evaluation scores are displayed in Table TABREF50. Sentiment Preservation (between 0 and 1) is calculated by comparing the average Sentiment rating for each model's output text to the Sentiment rating of the input text, and if both are less than 2.5 (negative), between 2.5 and 3.5 inclusive (neutral), or greater than 3.5 (positive), this is counted as a valid case of Sentiment Preservation. We repeat this for every evaluation line to calculate the final values per model. Harmonic means of all three metrics (using rescaled 0-1 values of RE Match and Fluency) are also displayed.
Analysis ::: Performance by Model
As seen in Table TABREF38, both SMERTI variations achieve higher STES and outperform the other models overall, with the WordNet models performing the worst. SMERTI excels especially on fluency and content similarity. The transformer variation achieves slightly higher SLOR, while the RNN variation achieves slightly higher CSS. The WordNet models perform strongest in sentiment preservation (SPA), likely because they modify little of the text and only verbs and nouns. They achieve by far the lowest CSS, likely in part due to this limited text replacement. They also do not account for context, and many words (e.g. proper nouns) do not exist in WordNet. Overall, the WordNet models are not very effective at STE.
W2V-STEM achieves the lowest SLOR, especially for higher RRT, as supported by the example in Table TABREF41 (see also Appendix F). W2V-STEM and WordNet models output grammatically incorrect text that flows poorly. In many cases, words are repeated multiple times. We analyze the average Type Token Ratio (TTR) values of each model's outputs, which is the ratio of unique divided by total words. As shown in Table TABREF52, the SMERTI variations achieve the highest TTR, while W2V-STEM and NWN-STEM the lowest.
Note that while W2V-STEM achieves lower CSS than SMERTI, it performs comparably in this aspect. This is likely due to its vector arithmetic operations algorithm, which replaces each word with one more similar to the RE. This is also supported by the lower TTR, as W2V-STEM frequently outputs the same words multiple times.
Analysis ::: Performance By Model - Human Results
As seen in Table TABREF50, the SMERTI variations outperform all baseline models overall, particularly in RE Match. SMERTI-Transformer performs the best, with SMERTI-RNN second. The WordNet models achieve high Sentiment Preservation, but much lower on RE Match. W2V-STEM achieves comparably high RE Match, but lowest Fluency.
These results correspond well with our automatic evaluation results in Table TABREF38. We look at the Pearson correlation values between RE Match, Fluency, and Sentiment Preservation with CSS, SLOR, and SPA, respectively. These are 0.9952, 0.9327, and 0.8768, respectively, demonstrating that our automatic metrics are highly effective and correspond well with human ratings.
Analysis ::: SMERTI's Performance By POS
As seen from Table TABREF55 , SMERTI's SPA values are highest for nouns, likely because they typically carry little sentiment, and lowest for adjectives, likely because they typically carry the most.
SLOR is lowest for adjectives and highest for phrases and nouns. Adjectives typically carry less semantic meaning and SMERTI likely has more trouble figuring out how best to infill the text. In contrast, nouns typically carry more, and phrases the most (since they consist of multiple words).
SMERTI's CSS is highest for phrases then nouns, likely due to phrases and nouns carrying more semantic meaning, making it easier to generate semantically similar text. Both SMERTI's and the input text's CSS are lowest for adjectives, likely because they carry little semantic meaning.
Overall, SMERTI appears to be more effective on nouns and phrases than verbs and adjectives.
Analysis ::: SMERTI's Performance By Dataset
As seen in Table TABREF58, SMERTI's SPA is lowest for news headlines. Amazon and Yelp reviews naturally carry stronger sentiment, likely making it easier to generate text with similar sentiment.
Both SMERTI's and the input text's SLOR appear to be lower for Yelp reviews. This may be due to many reasons, such as more typos and emojis within the original reviews, and so forth.
SMERTI's CSS values are slightly higher for news headlines. This may be due to them typically being shorter and carrying more semantic meaning as they are designed to be attention grabbers.
Overall, it seems that using datasets which inherently carry more sentiment will lead to better sentiment preservation. Further, the quality of the dataset's original text, unsurprisingly, influences the ability of SMERTI to generate fluent text.
Analysis ::: SMERTI's Performance By MRT/RRT
From Table TABREF60, it can be seen that as MRT/RRT increases, SMERTI's SPA and SLOR decrease while CSS increases. These relationships are very strong as supported by the Pearson correlation values of -0.9972, -0.9183, and 0.9078, respectively. When SMERTI can alter more text, it has the opportunity to replace more related to sentiment while producing more of semantic similarity to the $RE$.
Further, SMERTI generates more of the text itself, becoming less similar to the human-written input, resulting in lower fluency. To further demonstrate this, we look at average SMERTI BLEU BIBREF36 scores against MRT/RRT, shown in Table TABREF60. BLEU generally indicates how close two pieces of text are in content and structure, with higher values indicating greater similarity. We report our final BLEU scores as the average scores of 1 to 4-grams. As expected, BLEU decreases as MRT/RRT increases, and this relationship is very strong as supported by the Pearson correlation value of -0.9960.
It is clear that MRT/RRT represents a trade-off between CSS against SPA and SLOR. It is thus an adjustable parameter that can be used to control the generated text, and balance semantic exchange against fluency and sentiment preservation.
Conclusion and Future Work
We introduced the task of semantic text exchange (STE), demonstrated that our pipeline SMERTI performs well on STE, and proposed an STES metric for evaluating overall STE performance. SMERTI outperformed other models and was the most balanced overall. We also showed a trade-off between semantic exchange against fluency and sentiment preservation, which can be controlled by the masking (replacement) rate threshold.
Potential directions for future work include adding specific methods to control sentiment, and fine-tuning SMERTI for preservation of persona or personality. Experimenting with other text infilling models (e.g. fine-tuning BERT BIBREF8) is also an area of exploration. Lastly, our human evaluation is limited in size and a larger and more diverse participant pool is needed.
We conclude by addressing potential ethical misuses of STE, including assisting in the generation of spam and fake-reviews/news. These risks come with any intelligent chatbot work, but we feel that the benefits, including usage in the detection of misuse such as fake-news, greatly outweigh the risks and help progress NLP and AI research.
Acknowledgments
We thank our anonymous reviewers, study participants, and Huawei Technologies Co., Ltd. for financial support.

Question: Does the model proposed beat the baseline models for all the values of the masking parameter tested?
Answer: No

Question: Has STES been previously used in the literature to evaluate similar tasks?
Answer: No

Question: What are the baseline models mentioned in the paper?
Answer: Noun WordNet Semantic Text Exchange Model (NWN-STEM), General WordNet Semantic Text Exchange Model (GWN-STEM), Word2Vec Semantic Text Exchange Model (W2V-STEM)