SlowGuess/NAACL_2021_OCR huggingface dataset:
We utilized Amazon Mechanical Turk (AMT) to get annotators. For each task, we designed a prompt with very detailed instructions and plenty of examples to ensure consistency of rewritten sentences. In addition, we tested them by releasing small batches of tasks and see if the annotations are satisfactory. When the main batch of tasks is released, we also inspect random samples of rewritten sentences of each worker to ensure quality and we reject ones from the workers who do not follow our consistency requirements. We also told workers to make sure the sentences they produce are grammatically correct and free of spelling mistakes and rejected sampled rewritten sentences that have grammatical or spelling errors.
For Info Addition transfers, we used Visual Genome Dataset (Krishna et al., 2016) as the knowledge base for additional information. We first made a dictionary mapping each word to attributes and relations
Figure 5: The Amazon Mechanical Turk prompt page for information addition task.
Figure 6: The Amazon Mechanical Turk instruction page for information addition task.
in Visual Genome that contains the word, ordered by frequency of appearance in Visual Genome, and then for each noun in the sentence, we select the most frequent attribute and relation from Visual Genome that contain the noun (if any) as additional information to be added to the sentence. Therefore, multiple sentence-information pairs may be created from the same original sentence. We ended up with 4, 412 total pairs to be annotated. Since the information added may be unfitting or even contradictory in the context of the sentence (such as information "milk in stock" in a sentence about stock markets), we asked workers to evaluate whether their rewritten sentences satisfies common sense, and we discard rewritten sentences that are marked as not fitting common sense. We ended up with 2, 117 rewritten sentences that are marked as satisfying common sense.
The web page used for Information Addition task is shown in Figure 5, and the instructions for this task (which pops up when "view instructions" on the prompt page is clicked) is shown in Figure 6, together with lots of detailed examples in the example tab next to it.
For adjective emphasis and verb emphasis tasks, we use information from the parse trees to identify adjectives and verbs to be emphasized, and we filter out words that shouldn't be emphasized (such as “‘be’ for verb emphasis). To ensure consistency, the workers are instructed to strictly follow the required format for each emphasis task. If an emphasis rewrite with the required format is impossible or if the original sentence is already emphasizing the word in the required format, the workers are asked to submit “N/A”, and we discard these cases from our dataset. We started with 808 adjective emphasis tasks and 1,373
Figure 7: The Amazon Mechanical Turk prompt page for adjective emphasis task.
Figure 8: The Amazon Mechanical Turk instruction page for adjective emphasis task.
Figure 9: The Amazon Mechanical Turk prompt page for verb/action emphasis task.
verb emphasis tasks, and after discarding "N/A" results we still have 696 rewritten sentences for adjective emphasis task and 1201 rewritten sentences for verb emphasis task.
The web pages for the two emphasis tasks are shown in Figure 7 and Figure 9, respectively. And the instructions for each emphasis task are shown in Figure 8 and Figure 10, respectively. Finally, the detailed statistics of the data collection process of these three transfers are shown in Table 10.
Rewriting sentences such that it emphasizes the specific verb or action.
In this sentence verb/action emphasis task, you are given a sentence and a verb in the sentence to emphasize. Your task is to rewrite each sentence to emphasize the given verb action. If the rewrite is impossible or the given emphasis isn't a verb or action, write capitalized "N/A" in the answer line.
The verb action emphasis should be done by making the verb a noun and move it to the front. For example, "Big brown fox jumps over the sleazy dog" when "jumps" needs to be emphasized, the rewritten sentence should be "Jumping over the sleazy dog is done by the big brown fox" (See examples for more details)
Note that the original sentences will be without punctuations, and the re-written sentences should also not have punctuations. Other than punctuation, the rewritten sentence should be grammatically correct and free of misspelled words. Do not alter the meaning of the sentence. Do not change the tense of the sentence.
If the verb or action is already emphasized or such rewrite is impossible, write "N/A" as your answer. For example, in "Introducing pool by Riley was effective" and emphasize "Introducing", "N/A" should be written because "Introducing" is already emphasized.
If the verb to emphasize occurs multiple times in a sentence, emphasize the first occurrence of the verb in the original sentence.
If the verb to emphasize is inside a quote, just rewrite the quote part. (e.g. "He said I bought a red bird" goes to "He said the buying of a red bird was done by me")
There are a lot more examples of what should be done and what should not be done in the "examples" tab. Please view them before starting the task.
Figure 10: The Amazon Mechanical Turk instruction page for verb/action emphasis task.
Clarity	Content	Style
Active To Passive	4.93	5.00	4.87
Passive To Active	5.00	5.00	5.00
To Future	4.87	5.00	5.00
To Present	5.00	5.00	5.00
To Past	5.00	5.00	5.00
PP Front To Back	5.00	5.00	5.00
PP Back To Front	5.00	4.83	5.00
ADJ/ADV Removal	4.97	5.00	5.00
PP Removal	5.00	4.97	5.00
Substatement Removal	5.00	5.00	5.00
Table 11: Human evaluations of randomly sampled automatically generated sentence transfers. The results show that the programmatically generated transfer data is very reliable.
A.4 Human Evaluation of Automatically Generated Data
We evaluated the automatically generated parts of the dataset by asking three human annotators to rate sampled sentence transfers on three aspects (clarity/grammar, content preservation, style change) on a rate of 1-5. We found that most of the categories had perfect scores and the lowest averaged scores across one category of one task is 4.83. The full results are shown in Table 11.
A.5 Transfer Difficulty with Semantics Distance
To measure the semantic distance between original and transferred sentences in each transfer, we used BERT pre-trained models (Devlin et al., 2019) to compute the contextual representations of each sentence, and measured the average  $\ell_2$  distance as well as cosine similarity between representations of original and transferred sentences. The results are shown in Table 12. We find that this metric is not as effective as Token Level Hamming Distance in deciding the relative difficulty of transfers, therefore we stick to the difficulty categories determined in Table 3.
Transfer	MSE ↓	Cosine ↑
To Future Tense	0.015	0.978
To Past Tense	0.019	0.971
To Present Tense	0.012	0.982
Active To Passive	0.018	0.973
Passive To Active	0.029	0.960
PP Front to Back	0.021	0.969
PP Back To Front	0.016	0.977
ADJ or ADV Removal	0.013	0.981
PP Removal	0.032	0.953
Substatement Removal	0.045	0.934
Information Addition	0.012	0.981
Adjective Emphasis	0.031	0.952
Verb/Action Emphasis	0.035	0.948
Table 12: Average  $\ell_2$  distance and cosine similarity between BERT pooled output vectors of original and transferred sentences of the syntax, semantic and thematic transfers.
A.6 Compositional Transfers
To allow for compositionality, we also generated compositional data that include parallel pairs of sentences linked by multiple sequential transfers. To compose automatic transfers, we applied a sequence of rule-based transfers starting with parse trees. We use prefix labels to indicate the sequence of transfers undertaken. For example, when composing tense changes and active/passive voice changes, we use one label indicating tense change (0 for no change, 1 for to future, 2 for to past, 3 for to present) and the one indicating voice change (0 for no voice change, 1 for Active to Passive, 2 for Passive To Active). Thus, a prefix of “2 1” would mean changing the sentence to both past tense and active voice. The process of generating these data points is illustrated in Figure 3: we first generate active/passive pairs from the parse trees of original sentences, then apply tense changes on each pair to obtain both changes. Final statistics are shown in Table 4.
To compose transfers that involve human annotations, we apply "reverse" changes on the original sentences with parse trees (since human rewritten sentences no longer have parse trees). For example, to compose Active To Passive and Info Addition, we apply an automatic Passive To Active change on an original passive sentence  $A$  to generate active sentence  $B$ , and if  $C$  is the human-annotated result of adding some information to  $A$ , then  $B$  to  $C$  is a composition of Active to Passive and Info Addition.
B.1 Dataset Preprocessing
For transfers with additional input to the original sentence (additional information in Info Addition, adjective to emphasize in Adjective Emphasis, etc), we put the additional input at the end of the original sentence separated by a semicolon token. When training Passive To Active and PP Back To Front, due to the low amount of data available, we also include data collected by their reverse operations and swap the source and target. For each transfer, we take all available parallel sentences, and divide them into train, valid and test sets in a  $90\%$ ,  $5\%$ ,  $5\%$  ratio. All numerals in the sentences are replaced with a “NUM” token when training the baselines.
The hyperparameters used for all models trained in all experiments is shown in Table 13.
Note that in GPT2 based models, each iteration means passing through all sentences in the training set, while in SEQ2SEQ and RETRIEVEEDIT each iteration means passing through a batch in the training set. Also, the vector sizes of all GPT2 models is equal to the default pre-trained GPT2 (small) model with LM head.
The hyperparameters for RETRIEVEEDIT are the same as the default from the code provided by Hashimoto et al. (2018)<sup>1</sup>. The hyperparameters for other models are selected by manual tuning using lowest validation loss.
<https://worksheets.codalab.org/worksheets/0x1ad3f387005c492ea913cf0f20c9bb89/
Model	Parameter	Value
GPT	pretrained model	GPT2 (small) with LM head
pretrained encoder/decoder	GPT2 (small)
batchsize	20
optimizer	RMSprop
initial learning rate	2e-5
#turns to half learning rate	15
evaluate every #iterations	1
weight decay	0.015
teacher force ratio	1.0
max iterations	60
Model	Parameter	Value
CS-GPT and CS-GPT-ZERO	pretrained model	GPT2 (small) with LM head
pretrained encoder/decoder	GPT2 (small)
batchsize	20
optimizer	RMSprop
initial learning rate	2e-5
#turns to half learning rate	5
evaluate every #iterations	1
weight decay	0.015
teacher force ratio	1.0
max iterations	30
Model	Parameter	Value
SEQ2SEQ	encoder GRU hidden size	256
decoder GRU hidden size	256
attention size	256
word embedding size	256
batchsize	1
optimizer	SGD
initial learning rate	1e-2
#turns to half learning rate	5000
evaluate every #iterations	1000
weight decay	0.015
teacher force ratio	0.9
max iterations	185000
Model	Parameter	Value
RETRIEVEEDIT	encoder layers	2
decoder layers	4
hidden size	256
agenda size	256
attention size	256
word embedding size	300
batchsize	16
VAE-kappa	500
ident_pr	0.1
optimizer	Adam
learning rate	1e-3
max iterations	1000
evaluate every #iterations	100
Table 13: Table of hyperparameters for all models in all experiments respectively. Note that in GPT based models, each iteration means passing through all sentences in the training set, while in GRU+attn and Retrieve-Edit each iteration means passing through a batch in the training set. Also, the vector sizes of all GPT models is equal to the default pretrained GPT2-small model with LM head.
Transfer	Baseline Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE_L	CiDER
To Future Tense	GPT2	0.895	0.852	0.813	0.778	0.540	0.899	7.709
SEQ2SEQ	0.527	0.368	0.261	0.188	0.173	0.531	1.525
RETRIEVEEDIT	0.899	0.854	0.815	0.778	0.531	0.901	7.731
HUMAN	0.954	0.915	0.884	0.855	0.636	0.964	9.174
To Past Tense	GPT2	0.836	0.776	0.722	0.674	0.484	0.842	6.700
SEQ2SEQ	0.478	0.313	0.204	0.133	0.155	0.490	1.374
RETRIEVEEDIT	0.935	0.903	0.873	0.847	0.606	0.933	8.358
HUMAN	0.974	0.957	0.939	0.916	0.709	0.982	9.549
To Present Tense	GPT2	0.754	0.663	0.586	0.524	0.412	0.772	5.293
SEQ2SEQ	0.516	0.361	0.267	0.210	0.190	0.518	1.819
RETRIEVEEDIT	0.909	0.870	0.830	0.793	0.599	0.916	7.987
HUMAN	0.969	0.952	0.936	0.918	0.745	0.979	9.501
ADJ or ADV Removal	GPT2	0.647	0.508	0.394	0.308	0.313	0.652	3.259
SEQ2SEQ	0.450	0.274	0.172	0.112	0.140	0.469	1.171
RETRIEVEEDIT	0.897	0.841	0.786	0.731	0.511	0.919	7.461
HUMAN	0.933	0.894	0.870	0.847	0.591	0.965	8.924
Table 14: Evaluation results on easy transfers.
Transfer	Baseline Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE_L	CiDER
PP Front to Back	GPT2	0.398	0.210	0.081	0.001	0.184	0.406	0.886
SEQ2SEQ	0.393	0.280	0.207	0.161	0.162	0.391	1.492
RETRIEVEEDIT	0.541	0.423	0.301	0.176	0.247	0.547	2.536
HUMAN	0.965	0.959	0.952	0.945	0.690	0.970	9.671
PP Back to Front	GPT2	0.407	0.241	0.091	0.001	0.166	0.406	0.931
SEQ2SEQ	0.298	0.157	0.090	0.060	0.112	0.284	0.606
RETRIEVEEDIT	0.649	0.584	0.535	0.491	0.333	0.656	4.667
HUMAN	1.000	1.000	1.000	1.000	1.000	1.000	10.000
PP Removal	GPT2	0.763	0.700	0.645	0.593	0.419	0.787	6.012
SEQ2SEQ	0.330	0.195	0.121	0.081	0.112	0.363	1.004
RETRIEVEEDIT	0.798	0.770	0.739	0.712	0.478	0.846	7.111
HUMAN	0.957	0.944	0.931	0.919	0.681	0.976	9.207
Substatement Removal	GPT2	0.430	0.332	0.247	0.176	0.250	0.588	3.090
SEQ2SEQ	0.317	0.192	0.110	0.001	0.100	0.368	1.041
RETRIEVEEDIT	0.706	0.678	0.647	0.607	0.405	0.767	6.183
HUMAN	0.731	0.720	0.705	0.685	0.607	0.788	7.691
Information Addition	GPT2	0.479	0.305	0.189	0.121	0.207	0.475	1.359
SEQ2SEQ	0.345	0.180	0.094	0.053	0.098	0.335	0.632
RETRIEVEEDIT	0.493	0.396	0.328	0.275	0.284	0.603	3.401
HUMAN	0.846	0.762	0.690	0.624	0.521	0.892	6.863
Table 15: Evaluation results on medium transfers. INFO ADDITION is especially hard for current models.
Transfer	Baseline Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE_L	CiDER
Active To Passive	GPT2	0.476	0.329	0.238	0.189	0.216	0.464	1.820
SEQ2SEQ	0.373	0.220	0.141	0.103	0.131	0.345	0.845
RETRIEVEEDIT	0.681	0.598	0.503	0.427	0.383	0.663	4.535
HUMAN	0.931	0.881	0.835	0.795	0.587	0.905	8.603
Passive To Active	GPT2	0.433	0.271	0.167	0.120	0.191	0.434	1.329
SEQ2SEQ	0.339	0.214	0.160	0.132	0.126	0.331	1.062
RETRIEVEEDIT	0.714	0.659	0.559	0.474	0.397	0.732	5.024
HUMAN	0.977	0.962	0.942	0.919	0.685	0.973	9.409
Adjective Emphasis	GPT2	0.263	0.079	0.028	0.000	0.112	0.188	0.386
SEQ2SEQ	0.187	0.058	0.018	0.000	0.059	0.179	0.141
RETRIEVEEDIT	0.387	0.276	0.211	0.164	0.193	0.369	1.679
HUMAN	0.834	0.753	0.679	0.611	0.522	0.811	6.796
Verb/Action Emphasis	GPT2	0.309	0.170	0.095	0.041	0.140	0.292	0.593
SEQ2SEQ	0.289	0.127	0.066	0.038	0.098	0.275	0.300
RETRIEVEEDIT	0.416	0.284	0.209	0.148	0.223	0.423	1.778
HUMAN	0.649	0.569	0.493	0.421	0.433	0.693	5.668
Table 16: Results on hard transfers. Thematic transfers are especially difficult for current models.
B.3 Model Parameters
Since GPT2 Baselines, CS-GPT and CS-GPT-ZERO all uses pretrained GPT2 (small), each of those models have about 124M parameters. Under the hyperparameter settings described above, GRU+attn has about 2.4M parameters. Retrieve-Edit has 51.8M parameters.
B.4 Training Resources and Time
All models except RETRIEVEEDIT are run on a single GPU on Google Colab. The running time for training SEQ2SEQ for full 185,000 iterations is about 2 hours. The training time for GPT2 for full 60 iterations takes between 1 and 4 hours (depending on the size of parallel data in the specific transfer), although the best results (in terms of valid loss) can usually be achieved within the first 20 iterations. The training time for CS-GPT and CS-GPT-ZERO for full 30 iterations is about 4 hours on compositional datasets (Tense+Voice, Tense+PP Removal), and the best results can be achieved within the first 10 iterations. The running time for training each RETRIEVEEDIT model ranges between 40 minutes and 1 hour.
C.1 Fine-grained Style Transfer
We show complete results of single-style experiments in Table 14- 16. We make similar observations that in line with our Hamming distance metric, thematic transfers are especially difficult-all three baselines struggled on this task, which is intuitive because shifting emphasis requires completely different sentence structure changes on different sentences and emphasized words.
Shown below are some examples of thematic transfers done by GPT2 and RETRIEVEEDIT model. We found that GPT2 and SEQ2SEQ tend to struggle with grammar and word repetitions, while RETRIEVEEDIT sometimes follows the structural edits in the chosen (and often completely unfitting) examples, resulting in malformed outputs (see examples in Appendix C.1). Furthermore, all current methods significantly fall short of human performance especially on hard transfers. Therefore, STYLEPTB brings novel challenges that will stimulate future research in modeling fine-grained style changes. Note: in the input, along with the original sentence, the word to emphasize is in red):
Adjective Emphasis
Original Sentence several other banks have similar applications pending; similar Human Annotation: several other banks have applications pending which are similar GPT2: other applications applications applications applications applications applications pending SEQ2SEQ: the bank that the the the the the that was RETRIEVEEDIT: several applications pending is similar application pending that is
Verb Emphasis
Original Sentence: i much prefer money i can put my hands on ; put Human Annotation: putting my hands on money is something i much prefer GPT2: putting my my my on on on i do do SEQ2SEQ: the saying that is what we is not to do RETRIEVEEDIT: the handing of my hands was by something that my hands on it
RETRIEVEEDIT performed equally well compared to GPT2 in some transfers such as To Future Tense and performs significantly better compared to GPT2 in most transfers. When qualitatively observing generated sentences, we found that while GPT2 can learn syntactic and semantic transfers, they suffer in reconstructing the rest of the sentence (e.g. making word repetitions). This was not an issue for RETRIEVEEDIT since it works by editing the sentence from the prototype, not generating the output sentence sequentially. Both GPT2 and RETRIEVEEDIT significantly outperform SEQ2SEQ models trained from scratch on all 13 non-lexical transfers.
Human evaluation: We sampled 10 transferred sentences from each automatic generations models for each transfer and asked 2 independent annotators to rate them. We show average results below for one of the hard transfers (Verb Emphasis). From Table 17, we found that all approaches fall far short of human performance, which was judged by a separate human as having almost perfect clarity, content, and
Clarity	Content	Style
Active To Passive	GPT2	2.50	2.95	2.70
SEQ2SEQ	1.95	1.75	2.00
RETRIEVEEDIT	4.05	3.65	4.10
HUMAN	5.00	5.00	4.55
To Future	GPT2	4.65	4.85	4.80
SEQ2SEQ	2.05	2.20	3.25
RETRIEVEEDIT	4.70	4.70	4.35
HUMAN	5.00	5.00	5.00
ADJ/ADV Removal	GPT2	2.65	3.50	4.40
SEQ2SEQ	2.50	1.45	3.10
RETRIEVEEDIT	4.65	4.45	4.25
HUMAN	4.95	4.95	4.25
Substatement Removal	GPT2	3.05	3.15	3.95
SEQ2SEQ	3.30	2.05	3.75
RETRIEVEEDIT	4.30	3.65	4.20
HUMAN	5.00	5.00	3.55
Info Add	GPT2	2.15	2.55	3.05
SEQ2SEQ	2.70	1.35	1.60
RETRIEVEEDIT	4.00	2.55	2.75
HUMAN	5.00	4.80	4.70
ADJ emph	GPT2	1.30	2.65	2.85
SEQ2SEQ	2.65	1.05	1.00
RETRIEVEEDIT	4.15	3.05	3.00
HUMAN	4.55	4.75	4.75
VB emph	GPT2	1.60	2.20	4.05
SEQ2SEQ	3.85	1.45	1.25
RETRIEVEEDIT	4.15	2.65	2.20
HUMAN	4.70	4.45	5.00
Table 17: Human evaluation for single atomic style transfer on 7 selected transfers (the 7 transfers with BLEU scores appearing in main part of paper). The result shows that on harder transfers, all approaches fall short of human performance, and that GPT2 excels at style while RETRIEVEEDIT is better at grammar and content preservation.
Transfer	To Future + Passive To Active	To Past + PP Removal
Source Sentence	NUM % was risen by sales to NUM billion from NUM billion	the bond market was unmoved by the economic statistics
Target Sentence	sales will rise NUM % to NUM billion from NUM billion	the bond market is unmoved
SEQGPT	willalesalesales sales to billion from NUM billion	the bond market is is
CS-GPT-ZERO	NUM % % % risen risen sales sales NUM NUM from NUM billion	the bond market is unmoved by the economic statistics
CS-GPT	sales will rise NUM % to NUM billion from NUM billion	the bond market is unmoved
Table 18: 2 examples of composition transfers generated by CS-GPT, SEQGPT and CS-GPT-ZERO. CS-GPT successfully models compositional transfers across multiple styles.
style metrics. Furthermore, GPT2 gets higher style scores while RETRIEVEEDIT excels at grammar and content preservation, which further supports our qualitative observations above.
C.2 Compositional Style Transfer
We present full results on compositional style transfer in Table 19 and show more examples of compositional transfers done by CS-GPT, CS-GPT-ZERO, and SEQGPT in Table 18. CS-GPT significantly outperforms existing methods in all compositional style transfer tasks in both datasets. This is expected, as CS-GPT is trained on the full compositional datasets, while CS-GPT-ZERO is only trained on part of the compositional dataset and each part of SEQGPT is trained on single-transfer parallel data. Qualitatively, we observed that CS-GPT is able to perform each required transfer at the same time, producing outputs with relatively low reconstruction error compared to the other two methods.
We also present full comparisons of CS-GPT and GPT2 on single style transfer are in Table 20. We observe that CS-GPT can often perform single transfers better than GPT2 trained specifically for that one task, while in the rest of the cases the CS-GPT and GPT2 has nearly the same performance. Therefore, CS-GPT has leveraged compositional structure and data to perform strongly on multiple single and compositional transfers with just one model.
Dataset	Transfers	Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE_L	CiDER
Tense + Voice	ToPast+ ActiveToPassive	SEQGPT	0.332	0.155	0.057	0.024	0.144	0.300	0.636
CS-GPT-ZERO	0.337	0.163	0.075	0.029	0.154	0.283	0.760
CS-GPT	0.409	0.238	0.133	0.064	0.180	0.378	1.029
ToFuture+ ActiveToPassive	SEQGPT	0.391	0.222	0.120	0.065	0.167	0.373	0.866
CS-GPT-ZERO	0.419	0.243	0.114	0.047	0.209	0.325	1.238
CS-GPT	0.496	0.340	0.240	0.185	0.217	0.479	1.800
ToFuture+ PassiveToActive	SEQGPT	0.401	0.212	0.097	0.048	0.163	0.385	0.888
CS-GPT-ZERO	0.399	0.245	0.123	0.047	0.212	0.349	1.075
CS-GPT	0.528	0.364	0.259	0.197	0.234	0.524	2.020
ToPast+ PassiveToActive	SEQGPT	0.381	0.210	0.098	0.045	0.156	0.368	0.876
CS-GPT-ZERO	0.365	0.181	0.073	0.025	0.156	0.343	0.752
CS-GPT	0.474	0.297	0.175	0.099	0.206	0.473	1.513
ToPresent+ PassiveToActive	SEQGPT	0.348	0.189	0.085	0.037	0.142	0.343	0.745
CS-GPT-ZERO	0.424	0.257	0.118	0.046	0.208	0.389	1.025
CS-GPT	0.523	0.366	0.264	0.210	0.243	0.522	2.118
ToPresent+ ActiveToPassive	SEQGPT	0.396	0.256	0.177	0.136	0.179	0.384	1.209
CS-GPT-ZERO	0.445	0.254	0.120	0.059	0.212	0.348	1.271
CS-GPT	0.503	0.358	0.271	0.223	0.233	0.491	2.118
Tense + PP Removal	ToFuture+ PPRemoval	SEQGPT	0.722	0.644	0.581	0.524	0.385	0.755	5.562
CS-GPT-ZERO	0.465	0.335	0.221	0.137	0.313	0.496	1.907
CS-GPT	0.738	0.652	0.578	0.518	0.393	0.755	5.289
ToPast+ PPRemoval	SEQGPT	0.714	0.640	0.573	0.510	0.374	0.724	5.152
CS-GPT-ZERO	0.542	0.389	0.268	0.182	0.314	0.535	2.103
CS-GPT	0.772	0.695	0.624	0.564	0.421	0.775	5.585
ToPresent+ PPRemoval	SEQGPT	0.618	0.518	0.435	0.368	0.338	0.663	4.119
CS-GPT-ZERO	0.545	0.393	0.269	0.184	0.323	0.539	2.017
CS-GPT	0.709	0.609	0.523	0.446	0.718	0.718	4.588