Transformer-based Live Update Generation for Soccer Matches from Microblog Posts
Authors: Masashi Oshika
Kosuke Yamada
Ryohei Sasano
Koichi Takeda
Venue: EMNLP 2023 Main (2023)
Keywords: Timeline Summarization
Social Media
TL;DR: This paper proposes a system to generate live sports updates from tweets, which are based on a large pre-trained language model and incorporate a mechanism to control the number of updates and a mechanism to mitigate redundancy.

Abstract: It has been known to be difficult to generate adequate sports updates from a sequence of vast amounts of diverse live tweets, although the live sports viewing experience with tweets is gaining the popularity. In this paper, we focus on soccer matches and work on building a system to generate live updates for soccer matches from tweets so that users can instantly grasp a match’s progress and enjoy the excitement of the match from raw tweets. Our proposed system is based on a large pre-trained language model and incorporates a mechanism to control the number of updates and a mechanism to reduce the redundancy of duplicate and similar updates.

Transformer-based Live Update Generation for Soccer Matches from Microblog Posts
Masashi Oshika
Kosuke Yamada
Ryohei Sasano
Koichi Takeda
Graduate School of Informatics, Nagoya University, Japan
(oshika.masasae f6, yamada. kosuke v1 @s mail. nagoya-u. aca. sasano, takedasu a... jc
Abstract
Tweets
Generated Live Updates
Doan's shot awfull
It has been known be difficult generate ad- equate sports updates from sequence of vast amounts of diverse live tweets, although the live sports viewing experience with tweets is gaining the popularity. In this paper, we focus on soccer matches and work on building sys- tem to generate live updates for soccer matches from tweets so that users can instantly grasp a match's progress and enjoy the excitement of the match from raw tweets. Our proposed system is based on a large pre-trained language model and incorporates mechanism to control the number of updates and a mechanism to re- duce the redundancy of duplicate and similar updates.
Goal spilled
Doan gets an equalizer.
equalizer Vesessssal
Generator
What a goal Doan!!
Equalizer amazing
14 Mussiara OUT +> Goetze
German subsiiuttinn
Mussiara Goetao
t+2
Doon scores goal off ball played by the keeper!
NaN
is Germany in hurry. isn't It?
Figure 1: Overview of live sports update generation. Tweets in the figure show the timestamped sequence of tweets containing hashtags associated with a match such #WorldCup #UCL. Hashtags are excluded when tweets are input into the generator.
1 Introduction
2022). However, these methods are not easy to apply to this task because they do not account for time series data. Time-series-aware update genera- tion methods can be divided into two approaches depending on the generation timing. One approach determines the timing according to the number of posts per unit time (Nichols et al., 2012: Kubo al., 2013; Tagawa and Shimada, 2016). While this ap- proach can accurately capture main events, it tends to miss less significant events that have fewer re- lated tweets. Another approach, which generates updates at regular intervals (Dusart et al., 2021), can generate exhaustive updates but tends pro- duce redundancies by mentioning the same event at different times after it occurs.
When a sports match is broadcast, Twitter users often enjoy sharing the status of the match their opinions. For example, during televised soccer match, many users post tweets that include informa- tion about how goals were scored or their thoughts on a certain play, and it is possible to roughly under- stand a match's 's progress by reading these tweets. However, because of the diverse nature of tweets, ranging from informative to purely emotive content, it can be challenging to quickly grasp a match's progress. In this study, we focus soccer matches and work on building system to generate live sports updates from tweets so that users can in- stantly grasp match's progress.
Generation of live sports updates from tweets can be considered a type of multi-document sum- marization (McKeown and Radev, 1995). Much research exists on multi nolti-document summarization, including recent studies that aim to generate high- quality summarization by capturing the hierarchi- cal structure of documents (Fabbri et al., 2019; Jin et al., 2020). More recently, pre-trained lan- guage models such as BART and PRIMERA have also been used (Pasunuru et al., 2021; Xiao et al.,
In this paper, we build a pre-trained language model-based system to generate live updates for soccer matches. In particular, we propose to in- corporate a mechanism to control the number of updates and mechanism to mitigate redundancy in the system. Figure shows overview of our sys- tem. It uses the pre-trained language model Text- to-Text Transfer Transformer (T5) (Raffel et al., 2020), for which the inputs are tweets related to a specific match and the outputs are updates at cer-

---

t
Kickofft First half begins.
Kickoff! First half begins.
Kickoffi First half begins.
Kickoffi First half begins.
H+1
Kickofff First half begins.
NaN
NaN
NaN
t+2
NaN
Doan gets an equalizer.
Doan gets an equalizer.
Doan gets an equalizer.
Generator
Generator
Yes
No
Yes
No
Generator
Generator
Classifier
Classifier
Tweets
Tweets
Tweets
Preceding Updates
Tweets
Preceding Updates
(a) Maase
(b) M
(c) M
(d) Mclfviext
Figure
Architecture of baseline and proposed models.
tain times. If there is no appropriate update, "NaN" is output. To control the number of updates, it uses binary classifier to determine whether an update should be generated each time, and to mitigate redundancy, it leverages preceding updates.
shown in Figure (b). Specifically, before generat- ing update, this model performs binary classifi- cation of whether to generate update (yes) not (no) from input tweets. If the decision is yes, then an update is generated by the generator; otherwise, "NaN" is output. We use T5 for the classifier so that we can construct the model using only a pre-trained T5 although we can use pre-trained models based on Transformer encoder, such as BERT (Devlin et al., 2019) RoBERTa (Liu et al., 2019). Note that the weights of these T5 models not shared.
2 Proposed Method
Our system is based on abstractive model using T5. We also apply the approach that generates the update every minute. As a preliminary experiment regarding the generation timing, we tried another approach, which is detecting events such as goals or substitutions by monitoring the increasing number of tweets and generating updates at those detected confirmed that some events
2.2 Incorporation of Preceding Updates
To avoid generating redundant updates, intro- duce model in which the preceding updates added as input in addition tweets as context infor- mation (M+) as shown in Figure (c). Previ- ous studies have shown considcrable improvement of performance on NLP tasks by using prompts that combine input text with its related text (Liu and Chen, 2021; Liu et al., 2022; Wang et al., 2022). These studies led to the idea of incorporating the preceding context for generating updates fol- lows. The preceding updates combined using the T5 special token and then input to the model in the form of "tweets <ext ra_id__ preced- ing updates." Since the classifier and this genera- tion mechanism are independent, we can define the fourth model that combines them (M+elf+txe) as shown in Figure 2 (d).
points. However, such as close shot or good defensive plays could not be detected. For this reason, our models typically generate one update per minute and output "NaN" if they determine that better not to output any update.
Figure 2 (a) shows the architecture of our base- line model (Mbase). It generates updates every minute without referring to preceding updates; thus, updates may be redundant if event is mentioned repeatedly over long period. In addition, "NaN" is more frequently generated unless a highly proba- ble word predicted at the beginning of the output, which may result too many "NaN." To address these problems, in this study, we propose model that incorporates a classifier to determine whether to output "NaN" and a mechanism to mitigate re- dundancy by incorporating preceding updates.
2.3 Fine-tuning Setup
Both the classifier and the generator are constructed by fine-tuning pre-trained T5. The classifiers used
2.1 Classifier
To limit the number of "NaN" outputs, we intro- duce model that separates the classifier of whether to output "NaN" from the generator (M as
We experimented with pre-trained language model other than T5 as classifiers, such as BERT, but could not observe major diffcrences in performance.

---

in M+err and M+elf+ext are fine-tuned output either yes or no by inputting only tweets, or tweets and their previous updates, respectively. The gener- ators used in Muse and M+ext are fine-tuned so that the generator can generate an update or "NaN."
In addition to the proposed models, we built two oracle models. The first model is Orclextr. which an oracle extractive model that measured the similarity between the reference update and all the tweets at the corresponding time and extracted the tweets with the highest percentage of matching words. This method is oracle version of Dusart et al. (2021)'s method, and we used it to investi- gate the upper limit of the achievable performance of the extractive approach. The second model is Orcl clf which uses the reference updates as the preceding context in the generation phase. The model is based on M+ It enables us to in- vestigate the oracle performance that the proposed method could achieve. These models only gener- ated updates at times when the reference updates exist to ensure that the numbers of generated up- dates matched those of reference updates.
Models
The generators used in M and M+dl++ext are fine-tuned to always generate updates, using only the time periods when updates are present. In : ext that use preceding updates, M+ext and jclf
updates included in the training data are used as the preceding updates for fine-tuning, and the au- tomatically generated updates are used during the generation.
3 Experiment
To confirm the proposed method's 's effectiveness, we compared the performance of the baseline model, three proposed models, and two oracle models.
M and Mellftext which incorporate pre- ceding updates, used the updates of the last min- utes before each time, and all methods used tweets posted up 3 minutes after each time. All models are based on Japanese pre-trained T5* on Hugging Face Hub. All models were fine-tuned with three different random seeds to build three versions of the models, and their average was used as the eval- uation score.
3.1 Settings
Dataset We first collected data for 68 weeks of the J-League, the Japanese professional soccer league, from 2021 and 2022, comprising total of 612 matches. Of these, we used 83 matches with more than 3200 tweets between one hour before the match and one hour after the match. In total, we collected 8,502 updates and 324,835 tweets. We then created five subsets based on match and performed five-fold cross-validation with train- ing:developmenttesst = 3:1:1. We used the training set to fine-tune the model, the development set to find the best model, and the test set to evaluate the model. We used the Twitter 2 to collect tweets, We obtained information on game times and op- posing teams from the official J-League website, and we created the query for the API consisting of the game time to be collected, and manually deter- mined hashtags such as #grampus and #fmarinos related to that match. We collected tweets using the queries and pre-processed the tweets to remove URLs and hashtags in the tweets.
We evaluated the methods by compar- ing the generated updates with the reference up- dates. Because similar events could occur multiple times, an evaluation that assesses word agreement throughout the match could result an unfairly high score. To address this, we evaluated the simi- larity of strings after mapping them on the timeline. ROUGE (Lin, 2004) is commonly used eval- uation metric for text generation, and alignment- based ROUGE (Martschat and Markert, 2017) is an extension of ROUGE that was designed for time- line summarization. In this study, although we accounted for time differences for the generated updates, we did not apply weights to the time dif- ferences. Our evaluation ber of overlapping n-grams, defined aligned n-grams, between the reference and generated up- dates. Specifically, dynamic programming is ap- plied to maximize the aligned n-grams while allow- ing for a one-minute difference. More details are provided in Appendix A.
Metrics
We used the text updates published by the official J-Lcague website as the reference updates. Some updates were related to results up to the previous week or lifetime achievements. Since it is consid- ered to be out of scope to generate such updates from live tweets, we excluded these updates from reference updates using manually created rules.
based the num-
Phttps://develope switter.com //wp::///wwww.aaa
'tttps: /ttp:s///ugeee aace www.aaag..oooga.oo..cocooo labs/ t5- base- japenese-anneeeeeeee

---

of reference unigrams
of generated unigrams
of aligned unigrams
Precision
Recall
Fl-score
Mbase
164,245
49,088
19,585
0,401
0.118
0.182
M4ar
164,245
129,387
42.199
0.326
0.255
0.286
+exs
164,245
107,049
41,639
0.389
0.251
0.305
Maelitoxs
164,245
137,072
43.074
0,318
0.260
0.284
Orclacte
164,245
109,046
33,159
0,304
0.200
0.241
164,245
134,598
51,487
0.382
0.311
0.343
Table 1: Experimental results when unigrams are used for evaluation.
Reference updates
Generated updates (Miww))
Generated updates (M-oot)
NaN
minute extra time.
minute extra time.
The first half ended with the game tied at 0-0.
[+1
I minute of extra time.
minute of extra time.
The first half ended with the game tied at (-D.
The first half ended with
Second-harll kickoff, Yokohamal 'MM ball. No halftime substitutions for either team.
c+2
the game tied at 0-0.
Second-hare -ooond-harlfPa kickoff, Yokohamal'M
t+3
ball. 10 MJunio OUT + 16 Fujita
NaN
NaN
IN. 19 Ogashiwa OUT->2 Koroki IN
Table Examples reference and generated updates.
3.2 Main Results
Lastly, we compared the two oracle models with the proposed models. All of the proposed models scored higher than Orclextr- This suggests that the extractive approach has a clear limitation for this task and that is preferable apply the abstractive approach for generating live sports updates from tweets. The Orcl+cif+ext achieved a higher recall while maintaining precision compared to M cxt- which achieved the highest Fl-score among the proposed methods. However, the recall is only 0,311, which suggests the existence of reference updates that very difficult generate from the of tweets posted in real time.
Table 1 shows the results when unigrams are used for evaluation. As expected, MMbase generated many "NaN," because it generated fewer unigrams compared to the reference unigrams. In contrast, for M++ar,, the number of generated unigrams was close to the number of the reference unigrams, sug- gesting that the classifier enabled more appropriate decisions on whether to generate updates. Compar- ing Mass and M+ we confirmed that the re- call was improved considerably while the precision remained the same. This shows that by incorporat- ing the preceding context, the number of updates to be generated can be controlled some extent, while successfully suppressing redundant output.
3.3 Analysis
Table lists examples of reference updates and up- dates generated by Mhuse and the best proposed model | Note that the actual updates are in Japanese, but their English translations are pro- vided here for legibility. We can see that while the baseline model generated the same updates peatedly, redundant updates were suppressed by M+xxt. Regarding the overall output, however, the generation of redundant updates was not com- pletely suppressed, which remains to be a chal- lenge. In addition, the comparison of the reference updates with those generated by M+ shows that updates mentioning the same event were output one minute off. Such time discrepancies are not con- sidered a major practical problem, and this result confirms the effectiveness of the evaluation metries, which allowed for small time differences.
Next, by comparing M- and M+el++xxt. we can confirm that the introduction of the classifier resulted in slight improvement in recall, but a significant decrease in precision and consequently in F-score. We speculate that this is due to the following: the model with the classifier is forced generate updates if the decision of the classifier is yes, even when the corresponding set of tweets contains almost no information necessary to gen- erate the reference updates, which significantly de- creases the precision and thus the Fl-score as well. Overall, M achieved the best scores among the proposed models. We performed paired permu- tation tests the differences in F1-scores between M ext and the other models and found that all dif- ferences were significant at the significance level of 0.001.
We further conducted manual evaluations to in- vestigate how accurately each model was able to
*Results with bigrams are provided in Appendix B.

---

Event
Meinics
Minse
M
M
Orclaxte
Orcl Lol-
precision
0.71 70.43
0.66 /0,42
0.75 / 00.4
/0.27
0.766
0.5470.25
Goals
recall
0.53 / 0.32
0.66/ 0,44
0.87 / 0.50
0,84 00.32
0.79/0.68
0,87 0.45
Fl-score
0.61 / 0.36
0.66 0.42
0.80/ / 0.46
0.78 / 00.29
0.83 0.72
0.67 / 0,34
precision
0.71 /0.3
0.80 70.33
0.86 / 0.29
0.75 0.38
0.97 0.82
0.76 0.30
Player
recall
0.11 / 0.05
0.13 0.05
0.19/ 0.06
0.26 / 0.13
0.74 0.63
0.37 0.15
substitutions
Fl-score
0.09/0.09
0.22 / 0.09
0.31 / 0.10
0.38 0.19
0.84 0.71
0.50 0.20
precision
0,4070.20
0.427
0.50 0.38
0.4270.32
0.8370.67
0.13
Penalty cards
recall
0.15 0.07
0.41 0.22
0.30/ / 0.22
0.30 0.22
0.19
0.30/ 0.15
Fi-score
0.22 0.11
0.42 0.23
0.37 0.28
0.35 00.26
0.30 0.24
0.31 0.16
precision
0.65 /..37
0.61 70,34
0.73 0.38
0.67 0.33
0.94/0.79
0.587 0.27
0.27
Total
recall
0.21 0.112
0.30 /..17
0.37 0.19
0.40 0.19
0.66 0.56
0.48 0.22
Fl-score
0.32/0.18
0.40 /.22
0.49 / 0.26
0.50 0.24
0.77 00.66
0.52 0.24
Table 3:
Manual evaluation results. Each cell lists the results of lenient / strict evaluation.
detect key events in the matches. Specifically, we considered three types of key events: goals, player substitutions, and penalty cards, and manually eval- uated whether these events were detected correctly. The evaluation was performed using two criteria: lenient and striet. In the case of lenient evaluation, each event was evaluated using precision, recall, and Fl-score, where each event was considered to correctly detected if the reference updates and generated updates contained the same type of event within minutes. In the case of strict eval- uation, each event was considered to be correctly detected only when the scorer was identical for goals, when the type of card and the name of the target player were identical for penalty cards, and when the names of the target players were identical for substitutions.
a classifier to control the number of updates and preceding updates to reduce redundancy. We con- firmed that our method can achieve high perfor- mance by considering preceding updates. However, the generation of redundant updates was not com- pletely suppressed, and this remains challenge for implementing an update generation system with higher performance. In the future, we would like to investigate update generation for other sports.
Limitations
Our experiment has two limitations. First, we have not been able to conduct experiments using mod- els other than T5 in the generator. It is possible that performance could be improved by using different model. Secondly, the experiment was con- ducted only on soccer matches in Japanese. The effectiveness towards other languages and other sports is unverified.
Table shows the results of the human evalu- ation on 10 randomly selected matches. Overall, Orclextr achieved a high score, which is natural since Orclextr is model that selects tweets that are most consistent with the reference updates. Other than the oracle models, M+ex and +elf achieved relatively high scores. appears that by considering the context, redundant output can be avoided, resulting higher key event detection performance. However, the score for the strict eval- uation is not high enough, especially when com- pared to Orclextr- This indicates that the gener- ated updates do not include detailed information on key events such as player names, even though the source tweet in most cases includes such information, and more accurate detection of key events remains as future work.
Acknowledgements
This work partly supported by JSPS KAK- ENHI Grant Number 21K12012.
References
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL). pages 4171-4186.
Alexis Dusart, Karen Pinel-Sauvagnat, and Gilles Hu- bert. 2021. Tssubert: Tweet stream summarization using BERT. CoRR, abs/2106.08770)
4 Conclusion
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstrac- tive hierarchical model. In Proceedings of the 57th
In this paper, we have investigated the update gener- ation of soccer matches from tweets. We presented models based on T5 and attempted to incorporate

---

Annual Meeting of the Association for Computational Linguistics (ACL), pages 1084.1084.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 2111:5485-5553
Hanqi Jin, Tianming Wang. and Xiaojun Wan. 2020. Multi-gran aultiigraaullrity interaction network for extractive and abstractive multi-document summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics (ACL), pages
Yuuki Tagawa and Kazutaka Shimada. 2016. Gener- ating abstractive summaries of sports games from japanese tweets. In 2016 5th IAI International Congress on Advanced Applied Informatics (IIAI- AAI), pages 82-87.
Mitsumasa Kubo, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2013. Generating live sports updates from twitter by finding good reporters. In 2013 IEEE/WIC/ACM International Joint Confer- ences Web Intelligence (WI), pages 527-534.
Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022. Training data is more valuable than you think: simple and effective method by retrieving from training data. In Pruceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (ACL), pages 3170-3179.
Chin-Yew Lin. 2004. ROUGE: package automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81.
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022); The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114.
Wen Xiao, Iz Beltagy. Giuseppe Carenini, and Arman Cohan. 2022. PRIMERA: Pyramid-basse masked sentence pre-training for multi-document summariza- tion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL.) pages
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa:
Robustly Optimized BERT Pretraining Approach. vrXiv:1977.1169
Zhengyuan Liu and Nancy Chen. 2021. Controllable neural dialogue summarization with personal named entity planning. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pages 92-106.
Sebastian Martschat and Katja Markert. 2017. Improv- ing ROUGE for timeline summarization. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics (EACL), pages 285-290.
Kathleen McKcown and Dragomir R. Radev. 1995. Generating summaries of multiple news articles. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, page 74-82.
Jeffrey Nichols, Jalal Mahmud, and Clemens Drews. 2012. Summarizing sporting events using twitter In Proceedings of the 2012 ACM International Con- ference on Intelligent User Interfaces (IUT), page 189-198.
Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Su- jith Ravi, and Markus Dreyer. 2021. Efficiently sum- marizing text and graph encodings of multi- dotument clusters. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (NAACL), pages 4768-4779.

---

f1
82
93
9
gs
Reference Updates
Generated Updates
2
0
0
T1 The first half is about start.
01 NaN
F2
0
0
T22 NaN
02 First half kick off.
33
T3 NaN
Ø3 Argentina gets corner.
0
4
2
T4 Argentina gets corner.
4+ Argentina scores goal.
Messi gets score.
0s Messi gets a bcautiful goal.
0
0
Example of evaluation procedure. matrix stores the number of overlapping n-grams between the 9i and rj. After applying dynamic programming, circles are placed the reference updates and generated updates, and their paths are connected by arrows. On the right side, examples of reference updates and generated updates are shown, and aligning ones are connected.
Figure 3:
# of reference bigrams
of generated bigrams
of aligned bigrams
Precision
Recall
Fi-score
Mison
159,605
46,505
8,825
0.192
0,055
0,085
M+cle
159,605
123,680
16,160
0.130
0.101
0.114
M+ax
159,605
102,739
16,309
0.156
0.100
0.122
Maclf+oxs
159,605
131,441
15,752
0.121
0.098
0.108
Orcloxte
159.605
103,406
3,447
0.033
0.021
0.026
159.605
128,958
20,764
0.161
0.130
0.143
Table 4: Experimental results when bigrams are used for evaluation.
A Evaluation Details
similar. M had the highest Fl-score among the proposed models, while OrcI+elf+ext scored the highest among all six models. Orclextr had the lowest score, unlike the case of unigrams. It is considered difficult for extractive methods to generate updates that are very close to the reference updates.
The following is the evaluation procedure used in this study and Figure 3 illustrates the evaluation procedure with example.
1. Define arrays g and r store the generate updates and reference updates, respectively, and matrix S to store the mapping scores between updates, = [sty]-
2. Assign to Sij the number of overlapping grams between the 9; and ry, where i are in- dexes of the match time. To prevent mapping between updates with large time differences, assign 0 when 1 - >
3. Apply dynamic programming to to map a generated update to reference update so as to maximize the number of overlapping n-grams in the entire match, under the constraint that each update is mapped to at most one refer- ence update. Denote this maximum number as aligned n-gram. Evaluate model in terms of three indices for aligned n-gram: the pre- cision, recall, and -s-score.
B Experimental Results with Bigrams
Table lists the results an experiment using bigrams. The scores were generally lower when using unigrams, but the overall tendencies were


Reviews:
[{"review_id": "WDrxcNwi8R", "reviewer": "Reviewer_hkC3", "summary": "", "questions": "1. In line 105-106, the authors said, “We use T5 for the classifier so that we can construct the model using only a pre-trained T5”. However, in line 110, the authors said, “The weights of these T5 models are not shared”. Then what’s the benefit of only using one architecture?\n2. Maybe out of the scope of this short paper, what’s the performance by applying Dusart et al. 2021 to this dataset?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank you for your time and efforts in reviewing our paper.\n> In line 105-106, the authors said, \"We use T5 for the classifier so that we can construct the model using only a pre-trained T5\". However, in line 110, the authors said, \"The weights of these T5 models are not shared\". Then what's the benefit of only using one architecture?\n\nIt is true that there is no advantage in using the same pre-trained model for both generators and classifiers in terms of either computational cost or required memory size, but we chose to use the same pre-trained model for both generators and classifiers because of the advantage of having only one language model to be prepared in advance.\n\n> Maybe out of the scope of this short paper, what's the performance by applying Dusart et al. 2021 to this dataset?\n\nWe have not checked the performance of applying Dusart et al.'s model, but we can say that Dusart et al.'s model is an extractive model and therefore will have lower performance than Orcl_extr, an extractive oracle model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is easy to understand and the application is quite interesting. \n2. Data construction method is useful for the community. The evaluation method is comprehensive. The significance test is conducted.\n", "reasons_to_reject": "This work is an application-oriented task where the pretrained language models are applied to a specific task that is underexplored. I will challenge the model contribution for this work but in terms of new application, it is a neat short paper. \n", "questions_to_authors": "1. In line 105-106, the authors said, “We use T5 for the classifier so that we can construct the model using only a pre-trained T5”. However, in line 110, the authors said, “The weights of these T5 models are not shared”. Then what’s the benefit of only using one architecture?\n2. Maybe out of the scope of this short paper, what’s the performance by applying Dusart et al. 2021 to this dataset?\n", "paper_topic_and_main_contributions": "In this work, the authors focus on the application of generating live sports from tweets by leveraging powerful transformer-based text-to-text models such as T5 models. Strategies are proposed to control the number of updates and mitigate the redundancy in the system. More specifically, a binary classifier is used to determine whether an update should be generated. By considering the preceding updates, the system can further alleviate the redundancy issue. Experimental results on J-League related tweets have shown the effectiveness of the proposed method. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TqViHCWhPh", "reviewer": "Reviewer_Yzg3", "summary": "", "questions": "- The reviewer did not understand how reference updates look like. There should be gap between reference collected from J-League official site and tweets. So, ROUGE may not the best evaluation metric.\n- The reviewer is not sure what　\"preceding update\" is at final.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors would like to thank the invaluable questions and comments.\n> The performance on the best method is still low and the difficulty of the task is unclear because there is not an explanation of an upper bound.\n\nWe agree that we do not have a very detailed analysis of the difficulty of this task. However, we believe that the low performance of 0.241 for the F1-score of Orcl_extr, which is to select the tweets that will give the highest score, indicates that this is a difficult task and that the score of around 0.3 achieved by the proposed model is relatively high.\nWe also believe that experiments based on the Orcl+clf+cxt model, which uses the correct information about whether to generate an update at that time and past updates, are experiments that provide some sort of upper bound.\n> The reviewer did not understand how reference updates look like. There should be gap between reference collected from J-League official site and tweets. So, ROUGE may not the best evaluation metric.\n\nThe \"reference updates\" column in Table 2 shows the English translation of the actual reference updates. In addition, you can find the actual reference update from https://www.jleague.jp, although the data is in Japanese, e.g., https://www.jleague.jp/match/j1/2023/082602/live/#livetxt.\nIt is true that there is a gap between references collected from the official J-League website and tweets, so when employing an extractive summarization strategy, ROUGE may not be an appropriate metric for evaluation, but since this study employs an abstractive method, we believe that ROUGE is one of the best evaluation metrics.\n> The reviewer is not sure what　\"preceding update\" is at final.\n\nIn this study, we define \"preceding updates\" as updates in the three minutes immediately preceding the time at which the update is to be generated. For example, considering the timing of t+3 in Table 2, the reference update from t to t+2, are the preceding updates. When performing fine-tuning, or generating updates with the Orcl+clf+cxt model, this would specifically be \"NaN\", \"1 minute of extra time.\", and \"The first half ended with the game tied at 0-0.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The topic \"Live update generation\" is interesting and useful, and it can be expanded to other topics.\n- The paper shows tweets can be good candidates as an information source to replicate the reference.", "reasons_to_reject": "- The authors did not prepare other baselines in the experiments, so it is difficult to compare the proposed systems' performance.\n- The performance on the best method is still low and the difficulty of the task is unclear because there is not an explanation of an upper bound.", "questions_to_authors": "- The reviewer did not understand how reference updates look like. There should be gap between reference collected from J-League official site and tweets. So, ROUGE may not the best evaluation metric.\n- The reviewer is not sure what　\"preceding update\" is at final.", "paper_topic_and_main_contributions": "In this paper, the authors built a pre-trained language model-based system to generate live updates for soccer matches.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rhTAU0xX8q", "reviewer": "Reviewer_23Lj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for reviewing our paper.\n>In the experiments, this paper adopts the number of overlapping unigrams, which lacks persuasiveness.\n\nThank you for your important remarks. In the case of a soccer match update, for example, if both the reference and the generated update contain the same person's name and the word \"scored,\" the generated update is considered the appropriate update in most cases. On the basis of this thinking, we employed an evaluation based on unigrams, and bigrams, as shown in Table 3 in the Appendix. However, we agree with the point that these evaluations are not sufficient. We would like to add a human evaluation of agreement rates for events that are definitely included in the updates, such as scoring, player changes, etc.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of this work is simple and interesting.", "reasons_to_reject": "1. In the experiments, this paper adopts the number of overlapping unigrams, which lacks persuasiveness.\n2. This article is more suitable for demo session.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, authors propose a Transformer-based live update generation for soccer matches based on tweets. Specifically, it adopts the T5 model as the backbone to generate live updates. To control the number of updates, it uses a binary classifier to determine whether an update should be generated at each time, and to mitigate redundancy, it leverages preceding updates.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the task of generating live sports updates from tweets and proposes a series of T5-based models that controls the number of updates to be generated to mitigate redundancy. Overall, reviewers are rather positive about this submission and agree that the proposed task / models are interesting and that the content is a good fit for a short paper. On the negative side, reviewers point out that the experiments could be strengthened (more baselines, oracle / upper bound performance evaluation metrics).
Decision: Accept-Main