# Mask Transformer: Unpaired Text Style Transfer Based on Masked Language
## 4 Experiment
### Experimental Results  
Table 2 shows the automatic evaluation results of the model on the two datasets. In the Yelp dataset, although RetrieveOnly achieved the highest accuracy rate of 92.6%, and the perplexity is the lowest of 7. While in terms of semantic retention, the self-BLEU of RetrieveOnly was only 0.7, and the ref-BLEU was only 0.4. Almost no information about the original sentence was retained. This is because RetrieveOnly uses a retrieve method to find the most suitable template in the target style dataset, and directly use the template as the generated sentence. Therefore, the style conversion degree of this method is very high and the fluency of the template sentence almost as natural language. However, since the content of the generated sentence is completely changed compared to the original sentence, this method has no practical value. In terms of IMDB, the CycleRL method suffers from the same problem. Although it achieves an accuracy of 97.8% in style conversion, it only has a self-BLEU of 4.9 and a perplexity of 177 in terms of semantic retention and language fluency. We can see that our model has achieved the best results in style accuracy and content reservation for both datasets, and it also has better perplexity.  
For the manual evaluation, we select two representative models: Delete And Retrieve [16] and the Style Transformer [18]. For our method, we chose the model based on the multiclass discriminator. We randomly sample 100 sentences for each style in the dataset, and the transfer sentences were scored by three reviewers after being scrambled. The results are shown in Table 3. It can be seen that our model performs significantly better than other methods in style transfer accuracy as well as content remaining and fluency of the target sentence. To better understand the characteristics of our model, we sampled some sentences in the Yelp dataset, as shown in Table 4.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Yelp**} & \multicolumn{4}{c}{**IMDb**} \\ \cline{2-7}  & **ACC** & **Self-BLEU** & **ref-BLEU** & **PPL** & **ACC** & **Self-BLEU** & **PPL** \\ \hline ControlledGen [21] & 88.8 & 45.7 & 14.3 & 219 & 94.6 & 62.1 & 143 \\ \hline DeleteOnly [16] & 85.7 & 28.6 & 9.7 & 72 & N/A & N/A & N/A \\ RetrieveOnly [16] & 92.6 & 0.7 & 0.4 & 7 & N/A & N/A & N/A \\ TemplateBased [16] & 84.3 & 44.1 & 13.7 & 117 & N/A & N/A & N/A \\ DeleteAndRetrieve [16] & 87.7 & 29.1 & 10.4 & 60 & 58.8 & 55.4 & 57 \\ \hline CycleRL [7] & 88.0 & 7.2 & 2.8 & 107 & 97.8 & 4.9 & 177 \\ StyleTransformer [18] & 89.5 & 54.5 & 18.7 & 76 & 82.3 & 67.5 & 108 \\ \hline Conditional MaskTransformer & 91.8 & 54.6 & 19.3 & 81 & 95.7 & 63.9 & 92 \\ Multi-class MaskTransformer & 88.3 & 55.4 & 20.1 & 75 & 91.0 & 66.2 & 86 \\ \hline \hline \end{tabular} ACC and PPL denote accuracy and perplexity, respectively.  
\end{table}
Table 2: The automatic evaluation results of the Yelp and IMDb datasets.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Yelp**} & \multicolumn{3}{c}{**IMDb**} \\ \cline{2-7}  & **Style** & **Content** & **Fluency** & **Style** & **Content** & **Fluency** \\ \hline DeleteAndRetrieve & 3.67 & 2.61 & 2.84 & 3.75 & 2.33 & 2.26 \\ StyleTransformer & 3.41 & 3.34 & 3.64 & 3.36 & 3.58 & 3.52 \\ Ours (conditional) & 4.12 & 4.08 & 3.67 & 3.98 & 3.87 & 3.31 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The manual evaluation results of the Yelp and IMDb datasets.