SlowGuess/ACL_2020_OCR huggingface dataset:
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2019a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019b. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563-578, Hong Kong, China. Association for Computational Linguistics.
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654-663, Melbourne, Australia. Association for Computational Linguistics.
Markus Zopf, Eneldo Loza Mencia, and Johannes FURNKranz. 2018. Which scores to predict in sentence regression for text summarization? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1782-1791, New Orleans, Louisiana. Association for Computational Linguistics.
4951
Figure 4: Comparison of summarization metrics. Support sentences are marked in the same color as their corresponding facets. SCUs have to be annotated for each extracted summary during evaluation, while facet-aware evaluation can be conducted automatically by comparing sentence indices.
A Practical Notes on CNN/Daily Mail
We note several issues of the CNN/Daily Mail dataset in the hope that the researchers working on this dataset are better aware of these issues.
One issue is that sometimes the titles and image captions are introduced in the main body of the document by mistake (usually captured by "-lrb-pictured -rrb-" or colons), which may lead to bias or label leaking for model training since the reference summaries are observed to be similar to the titles and image captions (Narayan et al., 2018a). For example, we found that if there is a sentence in the main body that is almost the same as one of the captions, then that sentence is very likely to be used in the reference summary. Many such cases can be found in our annotated data.
We also found that in many documents, the 4-th sentence is "scroll down for video". And if this sentence appears in one document, it is often the case that the first three sentences are good enough to summarize the whole document. This finding provides yet another evidence why a simple Lead-3 baseline could be rather strong on CNN/Daily Mail. In addition, sentences similar to the first three sentences can often be found afterward, which suggests that the first three sentences may not even belong to the main body of the document.
B Additional Illustration
In Fig. 4, we show the comparison of ROUGE, FAR, and Pyramid. In Fig. 5, we show the ground-truth FAR scores, the FAR scores estimated
by various sentence regression approaches, and the prediction of FAR scores by linear regression.
C Detailed Examples
We list below the full documents, reference summaries, and the corresponding FAMs of several examples shown in Table 2. In particular, Table 10 shows an example of several support groups covering the same facet. We release all of the annotated data to facilitate facet-aware evaluation and follow-up studies along this direction.
Figure 5: The first three figures show the ground-truth and estimated FAR scores via human-annotated FAMs and machine-created FAMs. The fourth figure shows the fitting of linear regression on the human-annotated samples (LR-Small) and the prediction on the whole test set of CNN/Daily Mail (LR-Large). Systems are sorted in an ascending order by the ground-truth FAR on the human-annotated samples.
ID: 1b2cc634e2bfc6f2595260e7ed9b42f77ecbb0ce Category: Noise
Document:
-LRB- CNN -RRB- Paul Walker is hardly the first actor to die during a production .
But Walker's death in November 2013 at the age of 40 after a car crash was especially eerie given his rise to fame in the "Fast and Furious" film franchise. The release of "Furious 7" on Friday (this is the only mention of "Friday" in the whole document) offers the opportunity for fans to remember – and possibly grieve again – the man that so many have praised as one of the nicest guys in Hollywood.
"He was a person of humility , integrity , and compassion , ” military veteran Kyle Upham said in an email to CNN . Walker secretly paid for the engagement ring Upham shopped for with his bride .
"We did n't know him personally but this was apparent in the short time we spent with him. I know that we will never forget him and he will always be someone very special to us," said Upham.
The actor was on break from filming “ Furious 7 ” at the time of the fiery accident , which also claimed the life of the car ’ s driver , Roger Rodas . Producers said early on that they would not kill off Walker ’ s character , Brian O’Connor , a former cop turned road racing .
Instead , the script was rewritten and special effects were used to finish scenes , with Walker ’ s brothers , Cody and Caleb , serving as body doubles .
There are scenes that will resonate with the audience – including the ending, in which the filmmakers figured out a touching way to pay tribute to Walker while “retiring” his character.
At the premiere Wednesday night in Hollywood , Walker ’ s co-star and close friend Vin Diesel gave a tearful speech before the screening , saying “This movie is more than a movie . ” (random quotation, may use other quotes as well)
"You'll feel it when you see it," Diesel said. "There's something emotional that happens to you, where you walk out of this movie and you appreciate everyone you love because you just never know when the last day is you're gon na see them."
There have been multiple tributes to Walker leading up to the release . Diesel revealed in an interview with the “ Today ” show that he had named his newborn daughter after Walker . Social media has also been paying homage to the late actor .
A week after Walker ’ s death , about 5,000 people attended an outdoor memorial to him in Los Angeles . Most had never met him .
Marcus Coleman told CNN he spent almost $ 1,000 to truck in a banner from Bakersfield for people to sign at the memorial
It's like losing a friend or a really close family member... even though he is an actor and we never really met face to face," Coleman said. "Sitting there, bringing his movies into your house or watching on TV, it's like getting to know somebody. It really, really hurts."
Walker ’ s younger brother Cody told People magazine that he was initially nervous about how “ Furious 7 ” would turn out , but he is happy with the film .
"It's bittersweet, but I think Paul would be proud," he said.
CNN's Paul Vercammen contributed to this report.
Reference Summary:
"Furious 7" pays tribute to star Paul Walker, who died during filming
Vin Diesel : “ This movie is more than a movie ” (random quotation)
"Furious 7" opens Friday (unimportant detail)
FAMs: N/A
Table 9: Full document, reference summary, and the FAMs presented in Table 2.
ID: d58bf9387cd76f34bbb95fe25f8036015e5cc90a
Category: Low Abstraction
Dover police say a man they believe to be the so-called ‘ rat burglar ’ who cut holes to tunnel into buildings has been arrested in Maryland .
Authorities said in a news release Thursday that 49-year-old Thomas K. Jenkins of Capitol Heights , Maryland , was arrested last month by deputies with the Prince George ’ s County Sheriff ’ s Office .
‘ Rat burglar ’ : Thomas K. Jenkins , pictured is accused of robbing 18 Dover businesses
From September 2014 to February 2015 , Jenkins allegedly carried out 18 commercial robberies in Dover , Delaware , authorities there said .
During the investigation it was learned that the Prince George ’ s County Sheriff ’ s Department had a series of burglaries that were similar in nature to the eighteen committed in Dover , ’ the release said .
Thomas Jenkins has been accused by the Dover Police Department of robbing multiple businesses .
They are :
Maple Dale Country Club
Manlove Auto Parts
Sovereign Properties
Morgan Properties
U and I Builders
AMCO Check Cashing
Colonial Investment
1st Capital Mortgage
Advantage Travel
Ancient Way Massage
Tranquil Spirit Massage/Spa
Christopher Asay Massage
Morgan Communities
Vincenzo's Restaurant
Happy Fortune Chinese Restaurant
Happy 13 Liquors
Del-One Credit Union
Pizza Time
Melvin's Auto Service
Source : Dover Police Department/The News Journal
A car was found behind a building where a robbery took place and led deputies in Maryland to consider Jenkins as a suspect , authorities said .
Law enforcement later found Jenkin ’ s car and tracked where he went , Dover police said
Police say Jenkins had cut a hole in the roof of a commercial business in Maryland on March 9 and deputies arrested him as he fled .
According to Dover police , ‘ Jenkins was found in possession of .45 - caliber handgun that was stolen from a business in Delaware State Police Troop 9
jurisdiction . A search of Jenkins vehicle revealed an additional .45 - caliber handgun stolen from the same business
Jenkins is being held in Maryland and will face 72 charges involving the 18 burglaries in Dover when he is returned to Delaware .
The charges he is facing break down to : four counts of wearing a disguise during the commission of a felony , eighteen counts of third-degree burglary, eighteen counts of possession of burglary tools , fourteen counts of theft under $ 1,500 , and eighteen counts of criminal mischief , two of which are felonies , authorities said .
Cpl. Mark Hoffman with the Dover Police Department told the News Journal that Delaware State Police are planning to file charges over a 19th robbery at Melvin ’ s Auto Service , which reportedly occurred in a part of Dover where jurisdiction is held by state police .
Sharon Hutchison, who works at one of the businesses Jenkins allegedly robbed, told the newspaper 'He cut through two layers of drywall, studs and insulation.'
The Prince George ’ s County Sheriff ’ s Department did not immediately return a request for information on what charges Jenkins is facing there .
FAMs:
- thomas k. jenkins , 49 , was arrested last month by deputies with the prince george ’ s county sheriff ’ s office , authorities said .
[Support Group0][Sent0]: authorities said in a news release thsday that 49-year-old thomas k. jenkins of capitol heights , maryland , was arrested last month by deputies with the prince george ’ s county sheriff ’ s office .
police say jenkins had cut a hole in the roof of a commercial business in maryland on march 9 and deputies arrested him as he fled .
[Support Group0][Sent0]: police say jenkins had cut a hole in the roof of a commercial business in maryland on march 9 and deputies arrested him as he fled.
- jenkins is accused of carrying out multiple robberies in dover, delaware.
[Support Group0][Sent0]: jenkins is being held in maryland and will face 72 charges involving the 18 burglaries in dover when he is returned to delaware .
[Support Group1][Sent0]: ‘rat burglar’ : thomas k. jenkins , pictured is accused of robbing 18 dover businesses .
[Support Group2][Sent0]: thomas jenkins has been accused by the dover police department of robbing multiple businesses.
- he is facing 72 charges from the dover police department for 18 robberies.
the delaware state police is planning to file charges over a 19th robbery , which occurred in a part of dover where jurisdiction is held by state police .
[Support Group0][Sent0]: mark hoffman with the dover police department told the news journal that delaware state police are planning to file charges over a 19th robbery at melvin ’ s auto service , which reportedly occurred in a part of dover where jurisdiction is held by state police .
Table 10: Full document, reference summary, and the FAMs presented in Table 2.
ID: d1fa0db909ce45fe1ee32d6cb546e9d784bcf74
- LRB- CNN 
-RRB- You probably never knew her name , but you were familiar with her work .Betty Whitehead Willis , the designer of the iconic “ Welcome to Fabulous Las Vegas ” sign , died over the weekend . She was 91 .
Willis played a major role in creating some of the most memorable neon work in the city .
The Neon Museum also credits her with designing the signs for Moulin Rouge Hotel and Blue Angel Motel
Willis visited the Neon Museum in 2013 to celebrate her 90th birthday
Born about 50 miles outside of Las Vegas in Overton , she attended art school in Pasadena , California , before returning home .
She retired at age 77
Willis never trademarked her most-famous work , calling it “ my gift to the city . ”
Today it can be found on everything from T-shirts to refrigerator magnets .
People we've lost in 2015
willis never trademarked her most-famous work , calling it “ my gift to the city ”
[Support Group0][Sent0]: willis never trademarked her most-famous work, calling it "my gift to the city."
- she created some of the city's most famous neon work.
[Support Group0][Sent0]: willis played a major role in creating some of the most memorable neon work in the city.
Table 11: Full document, reference summary, and the FAMs presented in Table 2.
ID: dc833f8b55e381011ce23f89ea909b9a141b5a66
Category: High Abstraction
-LRB- CNN -RRB- As goes Walmart , so goes the nation ?
Everyone from Apple CEO Tim Cook to the head of the NCAA slammed religious freedom laws being considered in several states this week , warning thatthey would open the door to discrimination against gay and lesbian customers .
But it was the opposition from Walmart , the ubiquitous retailer that dots the American landscape , that perhaps resonated most deeply , providing the latest evidence of growing support for gay rights in the heartland .
Walmart ’ s staunch criticism of a religious freedom law in its home state of Arkansas came after the company said in February it would boost pay for about 500,000 workers well above the federal minimum wage . Taken together , the company is emerging as a bellwether for shifting public opinion on hot-button political issues that divide conservatives and liberals .
And some prominent Republicans are urging the party to take notice.
Former Minnesota Gov. Tim Pawlenty, who famously called on the GOP to “be the party of Sam’s Club, not just the country club,” told CNN that Walmart’s actions “foreshadow where the Republican Party will need to move.”
The Republican Party will have to better stand for” ideas on helping the middle class, said Pawlenty, the head of the Financial Services Roundtable, a Washington lobbying group for the finance industry . The party ’ s leaders must be “ willing to put forward ideas that will help modest income workers , such as a reasonable increase in the minimum wage , and prohibit discrimination in things such as jobs , housing , public accommodation against gays and lesbians . ”
Walmart , which employs more than 50,000 people in Arkansas , emerged victorious on Wednesday . Hours after the company ' s CEO , Doug McMillon , called on Republican Gov. Asa Hutchinson to veto the bill , the governor held a news conference and announced he would not sign the legislation unless its language was fixed .
Walmart ’ s opposition to the religious freedom law once again puts the company at odds with many in the Republican Party , which the company ’ s political action committee has tended to support .
In 2004 , the Walmart PAC gave around  $2 million to Republicans versus less than$  500,000 to Democrats , according to data from the Center for Responsive Politics . That gap has grown less pronounced in recent years . In 2014 , the PAC spent about $ 1.3 million to support Republicans and around$ 970,000 for Democrats .
It has been a gradual transformation for Walmart
In 2011 , the company bulked up its nondiscrimination policies by adding protections for gender identity . Two years later , the company announced that it would start offering health insurance benefits to same-sex partners of employees starting in 2014 .
Retail experts say Walmart ’ s evolution on these issues over the years is partly a reflection of its diverse consumer base , as well as a recognition of the country ’ s increasingly progressive views of gay equality - LRB- support for same-sex marriage is at a new high of 59 % , according to a recent Wall Street Journal/NBC News poll - RRRB .
It's easy for someone like a Chick-fil-A to take a really polarizing position , ” said Dwight Hill , a partner at the retail consulting firm McMillanDoolittle . “ But in the world of the largest retailer in the world , that ’ s very different . ”
Hill added : Same-sex marriage , “ while divisive , it ’ s becoming more common place here within the U.S . , and the businesses by definition have to follow the trend of their customer . ”
The backlash over the religious freedom measures in Indiana and Arkansas this week is shining a bright light on the broader business community ’ s overwhelming support for workplace policies that promote gay equality .
After Indiana Gov. Mike Pence , a Republican , signed his state ’ s religious freedom bill into law , CEOs of companies big and small across the country threatened to pull out of the Hoosier state .
The resistance came from business leaders of all political persuasions , including Bill Oesterle , CEO of the business-rating website Angie ' s List and a one-time campaign manager for former Indiana Gov. Mitch Daniels . Oesterle announced that his company would put plans on hold to expand its footprint in Indianapolis in light of the state ’ s passage of the religious freedom act .
NASCAR , scheduled to hold a race in Indianapolis this summer , also spoke out against the Indiana law
What we're seeing over the past week is a tremendous amount of support from the business community who are standing up and are sending that equality is good for business and discrimination is bad for business , ” said Jason Rahlan , spokesman for the Human Rights Campaign .
The debate has reached presidential politics
National Republicans are being forced to walk the fine line of protecting religious liberties and supporting nondiscrimination
Likely GOP presidential candidate Jeb Bush initially backed Indiana 's religious freedom law and Pence , but moderated his tone a few days later . The former Florida governor said Wednesday that Indiana could have taken a “ better ” and “ more consensus-oriented approach . ”
"By the end of the week , Indiana will be in the right place , " Bush said , a reference to Pence ’ s promise this week to fix his state ’ s law in light of the widespread backlash .
Others in the GOP field are digging in . Sen. Ted Cruz of Texas , the only officially declared Republican presidential candidate , said Wednesday that he had no interest in second-guessing Pence and lashed out at the business community for opposing the law .
"I think it is unfortunate that large companies today are listening to the extreme left wing agenda that is driven by an aggressive gay marriage agenda , " Cruz said .
Meanwhile , former Secretary of State Hillary Clinton , who previously served on Walmart ’ s board of directors , called on Hutchinson to veto the Arkansas bill , saying it would “ permit unfair discrimination ” against the LGBT community .
Jay Chesshir , CEO of the Little Rock Regional Chamber of Commerce in Arkansas , welcomed Hutchinson ’ s pledge on Wednesday to seek changes to hisstate ’ s bill . He said businesses are not afraid to wade into a politically controversial debate to ensure inclusive workplace policies .
“When it comes to culture and quality of life, businesses are extremely interested in engaging in debate simply because it impacts its more precious resource – and that’s its people,” Chesshir said. “Therefore, when issues arise that have negative or positive impact on those things, then the business community will again speak and speak loudly.”
While Republican Gov. Asa Hutchinson was weighing an Arkansas religious freedom bill, Walmart voiced its opposition (highly abstractive, hard to obtain by rephrasing original sentences)
Walmart and other high-profile businesses are showing their support for gay and lesbian rights
Their stance puts them in conflict with socially conservative Republicans , traditionally seen as allies
Table 12: Full document, reference summary, and the FAMs presented in Table 2.
ID: 1b2cc634e2bfc6f2595260e7ed9b42f77ecbb0ce
- LRB- CNN 
-RRB- He ’ s a blue chip college basketball recruit . She ’ s a high school freshman with Down syndrome
At first glance Trey Moses and Ellie Meredith could n't be more different . But all that changed Thursday when Trey asked Ellie to be his prom date .
Trey – a star on Eastern High School ’ s basketball team in Louisville , Kentucky , who ’ s headed to play college ball next year at Ball State – was originallygoing to take his girlfriend to Eastern ’ s prom .
So why is he taking Ellie instead? " She's great ... she listens and she's easy to talk to" he said.
Trey made the prom-posal -LRB- yes , that ’ s what they are calling invites to prom these days -RRB- in the gym during Ellie ’ s P.E. class .
Trina Helson , a teacher at Eastern , alerted the school ’ s newspaper staff to the prom- posal and posted photos of Trey and Ellie on Twitter that have goneviral . She was n’ t surprised by Trey ’ s actions .
"That's the kind of person Trey is," she said.
To help make sure she said yes , Trey entered the gym armed with flowers and a poster that read “ Let ’ s Party Like it ’ s 1989 , ” a reference to the latestalbum by Taylor Swift , Ellie ’ s favorite singer .
Trey also got the OK from Ellie ’ s parents the night before via text . They were thrilled .
overprotect . ”
Darla Meredith said Ellie has struggled with friendships since elementary school, but a special program at Eastern called Best Buddies had made things easier for her.
She said Best Buddies cultivates friendships between students with and without developmental disabilities and prevents students like Ellie from feeling isolated and left out of social functions.
"I guess around middle school is when kids started to care about what others thought , ” she said , but “ this school , this year has been a relief . ”
Trey ’ s future coach at Ball State , James Whitford , said he felt great about the prom- posal , noting that Trey , whom he ’ s known for a long time , often works with other kids
Trey's mother, Shelly Moses, was also proud of her son.
"It 's exciting to bring awareness to a good cause , ” she said . “ Trey has worked pretty hard , and he ’ s a good son . ”
Both Trey and Ellie have a lot of planning to do . Trey is looking to take up special education as a college major , in addition to playing basketball in the
As for Ellie, she can't stop thinking about prom.
"Ellie can't wait to go dress shopping" her mother said.
"Because I've only told about a million people!" Ellie interjected.
College-bound basketball star asks girl with down syndrome to high school prom. (highly abstractive, hard to obtain by rephrasing original sentences) Pictures of the two during the "prom-posal" have gone viral.
Table 13: Full document, reference summary, and the FAMs presented in Table 2.