SlowGuess/NAACL_2021_OCR huggingface dataset:
J.S. Chall and E. Dale. 1995. Readability revisited: the new Dale-Chall readability formula. Brookline Books.
Muthu Kumar Chandrasekaran, Guy Feigenblat, Dayne Freitag, Tirthankar Ghosal, Eduard Hovy, Philipp Mayr, Michal Shmueli-Scheuer, and Anita de Waard. 2020. Overview of the first workshop on scholarly document processing (SDP). In Proceedings of the First Workshop on Scholarly Document Processing, pages 1-6, Online. Association for Computational Linguistics.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 13063-13075. Curran Associates, Inc.
Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45-54, Melbourne, Australia. Association for Computational Linguistics.
Goran Glavaš and Sanja Štajner. 2015. Simplifying lexical simplification: Do we need simplified corpora? In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 63-68, Beijing, China. Association for Computational Linguistics.
Kazuma Hashimoto and Yoshimasa Tsuruoka. 2019. Accelerated reinforcement learning for sentence generation by vocabulary prediction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3115-3125, Minneapolis, Minnesota. Association for Computational Linguistics.
Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020. A probabilistic formulation of unsupervised text style transfer. In International Conference on Learning Representations.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on
computer vision and pattern recognition, pages 770-778.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1693-1701. Curran Associates, Inc.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1587-1596, International Convention Centre, Sydney, Australia. PMLR.
Xinyu Hua and Lu Wang. 2019. Sentence-level content planning and style specification for neural text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 591-602, Hong Kong, China. Association for Computational Linguistics.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, Lisa Orii, and Peter Szolovits. 2020. Hooks in the headline: Learning to generate headlines with controlled styles. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5082-5093, Online. Association for Computational Linguistics.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval 2019 Task 4: Hyperpartisan News Detection. In 12th International Workshop on Semantic Evaluation (SemEval 2019). Association for Computational Linguistics.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020. Gedi: Generative discriminator guided sequence generation.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In International Conference on Learning Representations.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Courtney Naples, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated Gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX), pages 95–100, Montreal, Canada. Association for Computational Linguistics.
Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics, 6:373-389.
James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of liwc2015. Technical report, Austin, TX: University of Texas at Austin.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Mingyue Shang, Piji Li, Zhenxin Fu, Lidong Bing, Dongyan Zhao, Shuming Shi, and Rui Yan. 2019. Semi-supervised text style transfer: Cross projection in latent space. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4937-4946, Hong Kong, China. Association for Computational Linguistics.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5926-5936, Long Beach, California, USA. PMLR.
Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training.
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. 2018. Unsupervised text style transfer using language models as discriminators. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 7287-7298. Curran Associates, Inc.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In International Conference on Learning Representations.
Ye Zhang, Nan Ding, and Radu Soricut. 2018. SHAPED: Shared-private encoder-decoder for text style adaptation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1528-1538, New Orleans, Louisiana. Association for Computational Linguistics.
Training. We train our simplicity style scorer and ideology style scorers for 10 epochs. The peak learning rate is  $1 \times 10^{-5}$  with a batch size of 32.
The class-conditional language models for simplicity and ideology are trained with a peak learning rate of  $5 \times 10^{-4}$  until the perplexity stops dropping on the validation set. We limit the number of tokens in each batch to 2, 048.
All word unit predictors are trained with a peak learning rate of  $1 \times 10^{-4}$  until the loss on the validation set no longer drops. We use a batch size of 32 for training.
Decoding. We use beam search for decoding. A beam size of 5 is used for all models except for the decoder state adjustment having a beam size 1 (greedy decoding) to maintain a reasonable running time. Repeated trigrams are disabled for generation in all experiments. As suggested by Lewis et al. (2020) and Yan et al. (2020), length penalties are set to 2.0 and 1.0 for summarization and headline generation, respectively. The minimum and maximum lengths are set for decoding at 55 and 140 for summarization, 0 and 75 for headline generation.
B Statistics on SemEval and Allsides
Each article in the SemEval dataset is labeled with a stance: left, leaning left, neutral, leaning right, or right. Here we combine left and leaning-left
Split	Left	Neutral	Right
Training	122,449	86,472	138,064
Validation	10,000	10,000	10,000
Test	10,000	10,000	10,000
articles into one bucket, and similarly for right and leaning-right articles. The ideology distribution for training, validation and test splits are in Table 6.
In our human evaluation of ideology-controlled headline generation, we use data collected from Allsides. The Allsides news clusters are curated by editors. The stance labels for different publishers are provided by Allsides, which are synthesized from blind surveys, editorial reviews, third-party analyses, independent reviews, and community feedback. We collect all the Allsides news clusters by April 26, 2020. After removing empty clusters, the total number of news clusters is 4,422. Among them, 2,985 clusters contain articles written in all three stances. For each article in the cluster, we keep the first paragraph and pair it with the headline. We remove the bylines in the first paragraphs.
C Additional Results for Headline Generation
In Table 7, we show the results of ideology-controlled headline generation on SemEval with BART fine-tuned on Gigaword (Napoles et al., 2012). Our methods are still effective, especially by using decoder states adjustment with style scorers.
Table 6: Ideology distribution for training, validation and test set splits of SemEval.
Model	Left	Right
Ideol.	BERT	Ideol.	BERT
BART	21.77	88.81	20.72	88.81
Ours w/ Decoder State Adjustment
IDEOL. SCORER	39.61	87.96	34.14	87.89
IDEOL. CC-LM	27.38	87.79	22.21	87.76
Ours w/ Word Unit Prediction
WORDU	22.98	88.35	21.09	88.40
DYNAMIC WORDU	22.84	88.32	21.08	88.47
Table 7: Ideological headline generation results with BART fine-tuned on the Gigaword dataset.
D Human Evaluation Guidelines
We include the evaluation guidelines for summarization and headline generation in Figures 4 and 5.
E Sample Outputs
Additional outputs are in Figures 6 and 7.
There was no special treatment for Lewis Ferguson at Paul Nicholls's yard on Thursday morning. The 18-year-old was mucking out the stables as usual, just a cut on the nose to show for the fall which has made him an internet sensation. Ferguson's spectacular double somersault fall from the favourite Merrion Square in the  $4.20\mathrm{pm}$  at Wincanton has been watched hundreds of thousands of times online. But he was back riding out and is undeterred from getting back in the saddle. Amateur jockey Lee Lewis Ferguson has just a cut on his nose to show for his ordeal . Teenager Ferguson was flung from his horse in spectacular fashion at Wincanton . 'It was just a blur,' he said. 'I couldn't work out what had happened until I got back to the weighing room and watched the replay. All the other jockeys asked me if I was all right and stuff, they all watched with me and looked away in horror. (...)
1	Not relevant to the article
e.g., “Paul Nicholl's yard will start its expansion in December. The expansion plan was carried out six months ago.”
3	Relevant, but misses the main point
e.g., “Amateur jockey Lee Lewis Ferguson has just a cut on his nose to show for his ordeal. ‘It was just a blur,’ he said.”
5	Successfully captures the main point and most of the important points.
e.g., “Lewis Ferguson was mucking out the stables as usual on Thursday. Favourite Merrion Square threw jockey in a freak fall on Wednesday.”
1	Summary is full of garbage fragments and is hard to understand
e.g., “18 year old nose. to cut show nose. the horse fashion, as to”
2	Summary contains fragments, missing components but has some fluent segments
e.g., “Lewis Ferguson out on Thursday. threw jockey on Wednesday.”
3	Summary contains some grammar errors but is in general fluent
e.g., “Lewis Ferguson was muck out the stables as usual on Thursday. The Merrion Square threw jockey jockey in a freak fall on Wednesday. His spectacular doublesomersault fall made him internetsensation.”
4	Summary has relatively minor grammatical errors
e.g., “Lewis Ferguson was mucking out the stables as usual on in Thursday. Favourite Merrion Square threw jockey ina freak fall on Wednesday. His spectacular double somersault fall made him internet sensation.”
5	Fluent Summary
e.g., “Lewis Ferguson was mucking out the stables as usual on Thursday. Favourite Merrion Square threw jockey in a freak fall on Wednesday. His spectacular double somersault fall made him internet sensation.”
Simplicity:
Bad	The summary uses complex words that can be replaced with simpler ones in almost all sentences and complex syntax structures (e.g., two or more clauses in a sentence)
e.g., “Lewis Ferguson was thrown by Merrion Square and made a spectacular double somersault fall which gathered millions of views online, making him internet sensation. But he was back riding out and is undeterred from getting back in the saddle, just a cut on the nose to show for the fall.”
Moderate	The summary uses at most one complex words that can be replaced with simpler ones per sentence, and uses syntax structures with at most one clause in a sentence
e.g., “Lewis Ferguson fell from Merrion Square. His spectacular double somersault fall made him internet sensation. But he was back riding out and is not afraid of getting back in the saddle.”
Good	The summary almost always uses simple and common words and simple syntax structures (e.g., no clause or at most one clause in the whole summary)
e.g., “Lewis Ferguson fell from his horse on Wednesday. His eye-catching double flip fall made him famous on the Internet. He was back to the yard. He is not afraid of getting back in the saddle.”
Figure 4: Sample summaries with explanations on human evaluation aspect scales and examples of summaries at different simplicity levels.
Paragraph
US President Donald Trump has said he is going to halt funding to the World Health Organization (WHO) because it has "failed in its basic duty" in its response to the coronavirus outbreak.
Relevance:
1	The headline does not contain any information related to the input
e.g., “‘a hateful act’: what we know about the ft. lauderdale airport shooting”
2	The headline contains some relevant event or person in the paragraph, but the topic is largely irrelevant
e.g., “trump: i don’t take questions from cnn”
3	The headline includes the main point of the paragraph, but have a different focus
e.g., “health experts condemn donald trump’s who funding freeze: ‘crime against humanity’”
4	The headline captures the main point of the paragraph, but contains some information that cannot be inferred from the paragraph
e.g., “trump cuts off u.s. funding to who, pending review”
5	The content of the headline and the paragraph are well aligned
e.g., “coronavirus: us to halt funding to who, says trump”
Figure 5: Sample headlines with explanations on human evaluation aspect scales.
Example A
Article: [ No Tiger Woods and no Rory McIlroy in the field is proving no problem in Houston, where record numbers have descended on the small suburb of Humble for the Shell Houston Open. ] ... It's 18 months now since Spieth attracted criticism for giving up on university and joining the PGA Tour as the 809th ranked player in the world. Spieth, only 21, has since risen to number four in the world. Funnily enough, no-one's criticising him anymore. Spectators have flocked to see Texas local boy Jordan Spieth in action at the Shell Houston Open. Spieth's fellow American Phil Mickelson has also impressed in the run up to the Masters . . .
Reference: Record numbers have flocked to see local boy Jordan Spieth in action . Spieth and Phil Mickelson have been gearing up for the Masters . Texas-born Spieth quit university in order to concentrate on golf career .
Decoder State Adjustment: [Tiger Woods and Rory McIlroy are not in the field at the Shell Houston Open. Record numbers have descended on the small suburb of Humble.] Phil Mickelson and local boy Jordan Spieth have impressed in the run up to the Masters. Spieth, only 21, has risen to number four in the world. (reorganize sentences)
Word Unit Prediction: The Shell Houston Open is the last tournament before the Masters. Phil Mickelson and Jordan Spieth are both playing well at the tournament. Spieth, only 21, is now number four in the world after 18 months on the PGA Tour. The American has finished in the top seven only twice this season.
Example B
Article: ... [Raikkonen's contract finishes at the end of the current Formula One season, although there is an option for 2016 providing both parties are in agreement.] The Finn stated this week he has never been happier working with a team in his entire F1 career, although his form to date has not matched that of team-mate Sebastian Vettel. [ Kimi Raikkonen has been urged to improve his performances if he wants to stay at Ferrari.] ...
Reference: Kimi Raikkonen's deal with Ferrari expires at the end of the season. Team principal Maurizio Arrivabene wants Finn to improve performances . Maranello driver has struggled to replicate team-mate Sebastian Vettel .
Decoder State Adjustment: [ Kimi Raikkonen's contract with Ferrari ends at the end of the season. There is an option for him to stay with the Maranello marque in 2016.] Raikk Finnish has said he has never been happier working with a team. [ But team principal Maurizio Arrivabene has urged him to improve. ]
Word Unit Prediction: Kimi Raikkonen is out of contract with Ferrari at the end of the season. The Finn has said he has never been better at a team in his F1 career. But team-mate Sebastian Vettel has been in better form so far this season. Maurizio Arrivabene has said that results will have to go up if he wants to be at Ferrari for another season.
Article: Three British citizens arrested in the United Arab Emirates after they were found plane spotting near Fujairah airport are to be released Monday, their lawyer, Nasser al-Hashem, tells CNN. The three have been in jail since February 22. . . . No charges were filed, there will be no travel ban, and the men will not face deportation, he said. [Conrad Clitheroes, 54, and Gary Cooper, 45] were on a five-day visit to the UAE from Manchester when they were arrested. The third man, [Neil Munro], is a British national who lives in the UAE. As a hobby, plane spotters view and photograph aircraft around the world.
Reference: Three British men won't be charged or deported, their lawyer says. They were arrested after plane spotting near Fujairah airport and have been in jail since February 22.
Decoder State Adjustment: Three Britons were arrested in the United Arab Emirates in February. They were found plane spotting near Fujairah airport. The men will be released Monday, their lawyer says. No charges were filed, and the men will not face deportation.. The three men were on a five-day visit to the UAE from Manchester. (omit the name)
Word Unit Prediction: The three have been in jail since February 22. They were arrested near Fujairah airport in the United Arab Emirates. The three will be released on Monday. No charges will be brought against them, and they will not be sent back to the U.K. or other countries.
Figure 6: Examples of document summarization on CNN/DM dataset. We highlight lexical simplifications with blue colors. Texts removed for simplification by decoder state adjustment are in gray or explained in italics.
Example A
Paragraph: Acting chief of staff Mick Mulvaney says President Trump willing to accept a barrier made of steel
REFERENCE	mulvaney: Saturday shutdown meeting ‘did not make much progress’	mick mulvaney: trump willing to take concrete wall ‘off the table’
RERANKING	mick mulvaney says trump willing to accept a bar-rier made of steel	mick mulvaney: trump willing to accept steel bar-rier
LBLCTRL	mick mulvaney: trump willing to accept barrier made of steel	mick mulvaney: trump willing to accept barrier made of steel
IDEOL.SCORER	trump's budget proposal would increase the number of military contractors in the us	trump wants to build a wall, and he's willing to pay for it
DYNAMIC WORDU	trump wants a border wall, but it's not all about the wall	mick mulvaney: trump willing to accept ‘steel’ border wall
Example B
Paragraph: Rep. Paul Ryan accused President Barack Obama of emboldening Iran and those storming U.S. embassies abroad while curtailing individual freedoms at home, during a speech here to a gathering of religious conservatives.
REFERENCE	paul ryan hits obama on national security: if we project weakness, they come	ryan to values voters: “american foreign policy needs moral clarity”
RERANKING	paul ryan accuses obama of emboldening iran, Protesters	paul ryan: obama emboldens iran healthcare bill
LBLCTRL	paul ryan: obama emboldening iran	ryan: obama emboldened iran, embassy protesters
IDEOL.SCORER	paul ryan accuses obama of emboldening iran, Protesters at religious conservatives' gathering	ryan: obama emboldening iran, protesters while curtailing freedoms at home
DYNAMIC WORDU	paul ryan to religious conservatives: obama has ‘emboldened’ iran	paul ryan: obama has ‘emboldened’ iran, protesters
Example C
Paragraph: The FBI on Wednesday issued an extraordinary public statement condemning the Republican push to release a classified memo that alleges surveillance abuses at the Department of Justice.
REFERENCE	opinion: why trump is so eager to release the nunes memo	trump to declassify infamous fisa memo
RERANKING	the fbi just responded to the gop's push to release the memo	fbi condemns gop push to release classified memo
LBLCTRL	the fbi just issued a public statement condemning the release of the republican memo	fbi condemns gop push to release classified memo
IDEOL.SCORER	the fbi just released a statement condemning the release of the republican memo	fbi releases statement condemning release of russia memo
DYNAMIC WORDU	fbi condemns gop push to release classified memo on russia	fbi condemns gop push to release memo on surveil-lance abuses
Figure 7: Examples of ideology-controlled headline generation. Best viewed in color. The left panel (shaded in blue) shows headlines generated with control toward the left stance. The right panel (red) shows headlines generated with control toward the right. We highlight words that are commonly used with the corresponding stances in **bold**.