CS2W: A Chinese Spoken-to-Written Style Conversion Dataset with Multiple Conversion Types
Authors: Zishan Guo
Linhao Yu
Minghui Xu
Renren Jin
Deyi Xiong
Venue: EMNLP 2023 Main (2023)
Keywords: Spoken-to-Written Style Conversion
Disfluency Detection
Grammatical Error Correction
ASR
TL;DR: A Fine-grained Chinese Spoken-to-Written Style Conversion Dataset with Multiple Conversion Types

Abstract: Spoken texts (either manual or automatic transcriptions from automatic speech recognition (ASR)) often contain disfluencies and grammatical errors, which pose tremendous challenges to downstream tasks. Converting spoken into written language is hence desirable.  Unfortunately, the availability of datasets for this is limited. To address this issue, we present CS2W, a Chinese Spoken-to-Written style conversion dataset comprising 7,237 spoken sentences extracted from transcribed conversational texts. Four types of conversion problems are covered in CS2W: disfluencies, grammatical errors, ASR transcription errors, and colloquial words. Our annotation convention, data, and code are publicly available at https://github.com/guozishan/CS2W.

CS2W: A Chinese Spoien-to-Written Style Conversion Dataset with Multiple Conversion Types
Zishan Guo, Linhao Yu, Minghui Xu, Renren Jin, Deyi Xiong"
College of Intelligence and Computing, Tianjin University, Tianjin, China {guozishan, linhaoyu, xuminghuti rrjin, dyxiong)@tju. (ew.oonnn@euueuuu..... edu. cn
Abstract
Category
Example
28 4 E S
Uh. you eat this morning?
Disfluency
Spoken texts (either manual automatic tran- scriptions from automatic speech recognition (ASR)) often contain disfluencies and gram- matical errors, which pose tremendous chal- lenges to downstream tasks. Converting spo- ken into written language hence desirable. Unfortunately. the availability of datasets for this is limited. To address this issue, we present CS2W, Chinese Spoken-to-Writ -Writtte style conversion dataset comprising 7,237 spo- ken sentences extracted from transcribed con- versational texts. Four types of conversion problems are covered in CS2W: disfluencies, grammatical errors, ASR transcription errors, and colloquial words. Our annotation conven- tion, data, and code are publicly available at https://gihub..co/uuuuishaaa
f # Li J 247
Did you cat this morning?
2 A 12. He can kick rubber hall.
ASR
Transcription Errors
fis ② 1 & 28 can perform banana kick.'
H 33
Grammatical Errors
have soen this electric."
# f it i #3 tu $ have seen this movic.
3 $ 18- n K kept paddling the game. $ fie fl the
Colloquial Words
He has been lazy in the game.
Table I: Examples of the four conversion types. In each category, the first pair of sentences is the original spoken style text and its machine translation output while the second is the normalized text converted from the original spoken style text and machine translation output.
1 Introduction
Automatic speech recognition (ASR) plays a vital role in a wide range of NLP application scenarios, such simultaneous interpretation, where verbal utterances are transcribed into spoken style texts. These transcriptions serve fundamental inputs plenty of downstream tasks. However, they of- ten inherently contain disfluencies, grammatical er- rors, and colloquial words, which pose tremendous challenges on downstream tasks. Automatically correcting errors and editing spoken into written language would significantly benefit downstream tasks that are usually trained on canonical texts. De- veloping such spoken-to to-ttitten style conversion models usually requires labeled data that cover dif- spoken style
To bridge this gap, we propose CS2W, a large- scale fine-grained Chinese Mandarin spoken-to- written style conversion dataset, developed on the Real Spontaneous Dialogue Speech dataset MagicData-RAMMC (Yang et al., 2022). CS2W con- sists of 7.237 annotated instances, covering four major conversion problems corresponding to the majority of spoken phenomena. For each conver- sion instance, we manually annotate the spans and types of the corresponding conversion problems.
We conduct thorough and in-depth analysis Chinese spoken texts and summarize four conver- sion problems: disfluencies. ASR transcription er- rors, grammatical errors, and colloquial words. The
ferent phenomena texts.
'In Chinese, "rubber ball" pronounced the same "ba- mana kick".
Unfortunately, existing datasets usually focus on single type of spoken style, such as disfluencies. Consequently, models trained on these datasets can- not address all spoken style issues.
PIn Chinese, the word " 3 (movie) is pronouneed "dian ying." and the individual character "ø" (electric) is pronounced "dian." 'The error in this sentence that only half the word i $ is mentioned.
'In Chinese, the colloquial word "#] *" refers to the act of slacking off during work or study.
*Corresponding author

---

four conversion problems are common in Chinese spoken texts and cover the major tasks in spoken- to-written style conversion (i.e., grammatical and ASR error correction, and simplification..
A flight to um Boston mean Denver
Tuesday
FW
RM
IM
RP
Filler Words
R-type
Figure 1: A sentence with disfluencies annotated in the style of Shriberg (1996) and the SWITCHBOARD corpus (Godfrey et al., 1992). FW=Filler Words, RM-Reparandum, IM=Interregnum, RI=Repair.
Disfluency:
Repetitions, restarts, or repairs in spontaneous communication.
ASR transcription errors: Occasional homo- phone mistakes in ASR transcriptions.
spoken-to-we to-wiitten language conversion. Ex- periment results demonstrate that the conver- sion from spoken written language effec- tively improves the performance of down- stream tasks.
Grammatical errors: Missing words, incor- rect syntax structures, etc., similar to those occurring in written style texts.
Colloquial words: Problems related to collo- quial words that differ from written language.
Related Work
Previous studies have treated disfluency detection and grammatical error correction as separate tasks. We hence review their progress separately. A comprehensive comparison of GEC, disfluency detection, and spoken-to-written style conversion datasets is presented in Table 2.
Table 1 shows examples of the four types of conversion problems and their impact on machine translation.
In comparison to existing datasets that focus on grammatical errors or disfluencies, our dataset con- tains more conversion types and closely aligns with the distribution of real-world spoken data. For ample, in the commonly used SWITCHBOARD corpus (Godfrey et al., 1992), instances usually contain few disfluent words (Charniak and John- son, 2001), and more than half those disfluencies consist repetitions (Shriberg, 1996). In contrast, our dataset frequently contains various types of grammatical errors and disfluencies within a single sentence.
Disfluency Detection and Related Datasets Disfluency common phenomenon in spoken language and is generally divided into two cate- gories: R-types and Filler words. Filler words have no meaning and are often used to indicate pauses and hesitations of the speaker. They are enumer- able and easy to detect, for example, in English, Filler words include "uh", "you know". "well" and so R-types include repeat, restart, and repair. As shown in Figure 1, a repair type disfluency in- cludes a reparandum ("Boston") and an interreg- num ("I mean"), followed by its repair ("Denver "). A repetition type has the same reparandum and interregnum. restart type has only a reparandum without interregnum, meaning the speaker starts a new topic. The most prevalent approach to disflu- ency correction involves treating it as sequence tagging task, aiming to produce fluent discourse by identifying and removing disfluent segments (Zayats al., 2016). Traditional models of disflu- ency detection use syntactic features (Honnibal and Johnson, 2014), language models (Johnson et al., 2004), or rhyme-based learning features (Zayats and Ostendorf, 2019), while more recent disfluency detection models mainly make use of pre-trained neural representations (Jamshid Lou et al., 201 8). Most of these models rely on manually annotated data.
Our contributions are as follows:
1.
We curate and release CS2W, the first open- source Chinese dataset for spoken-to- toritett style conversion. The dataset is derived from real-world spontaneous conversations. We provide fine-grained annotation along with written style manually normalized texts. Ad- ditionally, we establish a comprehensive set of criteria for spoken-to-wrine style conversion classification and annotation.
2. We conduct an in-depth analysis on the distri- bution of spoken-to- to- style conversion problems and identify new types of disfluen- cies.
3. We conduct benchmark evaluation experi- ments on CS2W to evaluate the performance of cutting-edge large language models on
Unfortunately, few disfluent detection datasets

---

Type
Annotation
Dataset
Task
#Sentences
Domain
Language
Conversion Covered
Paradigm
FCE
GEC
GEC
Error-ooded
34K
Essay
EN
AESW
GEC
GEC/ S2W
Error-coded
1.2M
Journal Articles
EN
JFLEG
GFC
GFC
Direct Rewriting
1.5K
TOFFL Fxam
EN
Language-teaming
WI-LOCNESS
GEC
GEC
Direct Rewriting
3.7K
EN
Website
Language-learinn
NLPCC18-task
GEC
GEC
Direct Rewriting
717K
CH
Website
CGED
GEC
GEC
Error-coded
7.2K
I1SK Exam
CH
Telephone
SWITCHBOARD
DC
Error-coded
265K
EN
Conversation
Telephone
PhoDisfluency
DC
Error-coded
5.8K
Conversation
Telephone
Japanese S2W
S2W
GEC/ GEC/DC/ S2W
Direct Rewriting
52K
JA
Conversation
Direct Rewriting
Telephone
CS2W
S2W
GEC DC/ S2W
7.2K
CH
Error-coded
Conversation
Table 2: Comparison between our dataset and other datasets. DC: Disfluency Correction. GEC:
Grammatical Error
Correction. S2W: Spoken-to to-Written Style Conversion.
are publicly available. The SWITCHBOARD cor- pus (Godfrey et al., 1992) consists of transcribed telephone conversations between two individuals discussing specific topic, which include disfluen- cics. Compared to this dataset, CS2W contains higher density of disfluent sentences.
FCE (Vougiouklis et al., 2018) is an early large- scale English GEC dataset using the error-coded paradigm, comprising raw text produced by En- glish learners who complete their First Certificate in English. Another example AESW (Daudar- avicius al., 2016), sourced from a professional editing company, which focuses on assessing the level technical writing and includes questions on not only writing style conversion but grammati- cal errors. Additionally, the NLPCC18-task (Zhao et al., 2018) dataset originates from Lang-8t, language learning website where native speakers can freely select essays by learners for revision. Conversely, the direct rewriting paradigm instructs annotators to rephrase the input sentence directly, providing a corresponding grammatically correct version without altering the original meaning. An example is JFLEG (Napoles et al., 2017), devel- oped with reference to the TOEFL exam. It places emphasis preserving the overall fluency of the rewritten text. Moreover, WI-LOCNESS (Bryant et al., 2019) encompasses rived from essays written by students with English as either their first or second language. Finally, CGED (Rao et al., 2018, Rao et al., 2020) is de- signed to diagnose grammatical errors in Chinese, based on the HSK (Hanyu Shuiping Kaoshi, Test of Chinese Level) examination.
Due to the sparsity of disfluency data, Dao et al. (2022) construct a Vietnamese disfluency dataset PhoDisfluency by manually introducing disfluency perturbations to the fluent sentences. In contrast, CS2W is collected from natural, spontancous con- versations, and is the first open-source Chinese dataset containing disfluency issues.
Grammatical Error Correction and Related Datasets Common errors in Chinese grammatical error correction (GEC) datasets include spelling errors, missing words, redundant words, incorrect word order, collocation errors, etc. Seq2Seq mod- els, based on RNN/CNN Transformer, are usu- ally used for Chinese GEC tasks. Hinson al. (2020) first propose Seq2Edit model for Chinese GEC, which achieves comparable performance with the Seq2Seq counterparts. Most Seq2Edit- based models use PLMs like BERT (Devlin et al., 2019) to initialize their encoders. Li and Shi (2021) apply a non-auuoregresssive neural machine transla- tion model Chinese GEC.
distinct datasets, de-
Numerous datasets have been proposed for GEC. These datasets primarily use two main paradigms label the data: error-coded and direct rewrit- ing. In the error-coded paradigm, annotators are tasked with explicitly identifying erroneous spans in the original sentence, specifying the error type, and subsequently making corrections. For instance,
In contrast the existing GEC datasets, CS2W dataset places stronger emphasis on gram- matical errors occurring in the spoken domain rather than the written domain. Notably, while all previously established datasets feature a single
"htps://lang-S.co/

---

annotation paradigm, the CS2W dataset stands out by offering two distinct annotation paradigms. Spoken-to- -rrittn Style Conversion and Re- lated Datasets Spoken-to -porinnto-written style conversion can be formulated as monolingual translation task (Wubben et al., 2010), mapping from spo- ken style text to normalized written style text. A variety of approaches have been proposed to ad- dress these monolingual translation challenges, in- cluding noisy channel models and hidden Markov models (Johnson and Charniak, 2004; Ferguson et al., 2015; Matusov et al., 2006) used earlier. In recent years, neural sequence conversion models have demonstrated superior performance (See et al.. 2017). Nevertheless, these models require a sizable parallel corpus for training since they learn the relationship between input and output sequences directly an end-to-end manner. Therefore, is essential to curate a large-scale parallel corpus of spoken and written language.
3.2 Annotation Guidelines
The spoken-to-written style conversion normally involves lexical and syntactical editing and style transfer. The former is similar grammatical error correction, dealing with disfluencies, ASR tran- scription errors, and grammatical errors while the latter is for the translation of colloquial words into canonicalized words with the same meaning used in written style texts.
To annotate the selected sentences, we employ two established annotation paradigms: error-coded and direct rewriting, commonly used in grammati- cal error correction datasets (Zhao et al., 2018; Rao et al., 2020), which have been described in Section 2.
Initially, for each sentence, use the error- coded paradigm meticulously identify spoken- to-written style conversion problems and determine the types of these problems (i.e., E disfluency, ASR transcription error, grammatical error, colloquial word). We also pinpoint the specific spans within the sentences where these problems manifest. It's essential to recognize that a sentence may exhibit multiple conversion problems simultancously.
Publicly available spoken-to-written style con- version data scarce, with the majority of datasets being private. Ihori al. (2020) curate the Parallel Corpus for Japanese Spoken-to- Written Style Conversion, which annotates not only gram- matical errors but also special conversion types in Japanese. These types include restoring postposi- tional particle expressions, hiragana symbols, etc. CS2W is spoken-written style conversion dataset, where the conversion problems are categorized ac- cording to Chinese linguistic characteristics.
However, it worth noting that (Sakaguchi al., 2016) highlight certain challenges associated with the error-coded paradigm, particularly concerning consensus among annotators regarding the identi- fication of spans and problem types. Furthermore, complex annotation paradigms sometimes lead an- notators to neglect the aspect of sentence fluency, resulting in unnatural expressions. Therefore, we also incorporate the direct rewriting paradigm into our annotation. This involves rewriting the spoken style language directly into a written form, pro- ducing references that are not only grammatically correct but also fluent and in a written style. Ad- ditionally, the labeled conversion problems in the error-coded paradigm need to be consistent with those corrected in the written style reference.
3 Dataset Curation
In this section, we elaborate on the creation of the
CS2W dataset.
3.1 Data Source
CS2W is built upon the existing MagicData-RAMCC dataset (Yang al., 2022), which consists of 351 sets of spontaneous conversations in Chinese Man- darin. Each set features natural conversations be- tween two speakers on a single topic, and it in- cludes audio files and transcribed texts that retain real-world disfluencies, grammatical errors, and ASR transcription errors. We manually select sen- tences from the ASR transcriptions, which are self- contained in meaning but with conversion problems for annotation. In total, collect 7,237 sentences for further annotation. comprehensive descrip- tion the data extraction process can be found in Appendix A.
Table provides a breakdown of the labeling for different conversion types. To enhance anno- tation consistency among different annotators, we develop comprehensive guidelines in Appendix B, which offer detailed descriptions of our annotation convention along with illustrative annotation exam- ples.
3.3 Annotation Process
To cnsure the consistency and quality of our an- notations, we implement a two-round annotation

---

Myen
Sub-Type
SouceSentence
Reference
$ Th HE 1
R- T-yee
Disfuency
Some apps, uh, some apps require payment.
Some apps require payment.
Filler
1 I it # bi E
* R I & 15
Words
He has, I mean, headaches be gets older.
He has headaches be gets older.
ASR
4: 26 12 He can kick rubber ball.
16 * KE,A
Transcription
He can perform a banana kick.
Errors
Missing Words
f5 AB7f 2 ty?
$5 to 18 18: 2 5/?
Grammatical Errors
you know what he?
Do you know what he said?
SEiF, " (L AA A2 #
E &
E B5 A & 1674
Redundant
"I TT A R d4 &
Words
Their für is shiny and can be
Their für is shiny and can be
hardly seen with the naked eyc.
seen with the naked eyc.
Incorrect
# Y 31 3 -- 35 & W..
# & 1.
-
Word
Yesterday watched newly
Yesterday watched
Order
purchased movie on TV.
newly purchased movic on TV.
P
#
T # - #8
1 $
Colloquial
This is obviously your credit, but your coworkers took it away. You're re such an unlucky guy.
This is obviously your credit, but your coworkers took it away. You'l 'e so unlucky.
Words
Table Example source sentences and references for the four main types and subtypes of spoken to to--ttt---tt written style conversion. Underlined phrases are spans with conversion problems.
process.
they submit their annotations the senior annota- tors. The senior annotators then perform a compar- ative analysis of the results from both rounds and issue the final annotation verdict. Detailed guide- lines for resolving inconsistencies are provided in Appendix B.
In the first round of annotation, we enlist the expertise of eight Chinese native speakers as part- time annotators after are-annotation training. We require that each conversion problem within a given spoken style text be annotated in accordance with established conventions, assuring the validity and uniformity of the annotations.
Dataset Analysis
In the second round, we conduct a manual eval- uation and re-a re-annotaioon, with the authors of this paper serving as senior annotators. Each annotated instance is scrutinized according to the answers to the following three core questions:
We provide data statistics and analyses on conver- sion type distribution, sentences with multiple con- version problems, and new subtype of disfluency existing in dataset, which is absent in previous studies.
1. Are the type and span annotations correct? This step ensures the accuracy of annotations under the error-coded paradigm.
4.1 Overall Statistics
Table 4 presents the overall statistics of our dataset. We randomly shuffle all the annotated texts and par- tition them into training, development, and test sets with proportion of 8:1:1. The training set consists of 5,789 texts, while both the development and test sets contain 724 texts each. Not surprisingly, most erroneous texts contain only one error. After Chinese word segmentation via the PKUseg tool (Luo et al., 2019), we obtain a total of 143.738 to- kens. On average, each text contains approximately 12.82 tokens. Additionally, the average length of each span is 2.02, and each sentence contains an average of 1.35 spans.
2. Does the written style reference faithfully re- tain the meaning of the original spoken style text? This step ensures the accuracy of anno- tations under the direct rewriting paradigm.
3. Is the modification in the written language reference consistent with the conversion prob- lems being labeled? This ensures the align- ment between the annotations under the error- coded paradigm and those under the direct rewriting paradigm.
the annotators the second round encounter any inconsistencies with the first-round annotations,
*In Chinese, "A 2 # is an Internet phrase used to de- scribe people who are aggrieved hut have no way to complain.

---

Train
Dev
Test
All
of conversion problems. This observation under- scores the prevalence of disfluency problems, as a substantial number of sentences with other types of conversion problems also include disfluencies. Surprisingly, over 50% of sentences featuring gram- matical errors or colloquial words also include dis- fluency problems. This suggests likely connec- tion between disfluencies and the emergence of grammatical errors and colloquial words, as disflu- encies often arise from pauses in thought during spontaneous language production, potentially con- tributing to these issues.
ASentences
5,789
724
724
7,237
w/ I ernor
4,211
596
597
5,404
1,288
w/ 3 errors
317
15
11
343
2:4
ASpans
8,022
889
881
9,792
Avg.
1.386
1.228
1.217
1.353
Avg. Jen
2,003
1,883
2,069
2,020
RCharacters
121,0077
10,844
10,987
148,738
Avg. #churacters
21,058
15.033
15.175
19.862
#Tokens
78,549
7,120
7,116
92,785
Avg.
13,509
9.834
9.829
Table 4: Data statistics of CS2W. "w/n" means that a sentence contains errors.
2
Collequial Words Grammatical Errors Disfluency ASR Transciptionn Errors
34.80%
7
7.22%
4.4 A New sub-type of Disfluency
62.40%
8.37%
Our analysis of the most prevalent conversion prob- lem, disfluency, in the context of spoken-to-written style conversion, has revealed a novel disfluency pattern. In the conventional R-type disfluency, the reparandum (the incorrect portion) typically pre- cedes the repair (the corrected portion). This pat- tern aligns with the common observation that speak- often correct themselves upon realizing an error. However, in this distinct disfluency pattern, the repair precedes the reparandum, as illustrated in Figure In other words, an utterance fragment is originally said correctly but immediately said incorrectly when it is repeated.
Missing Words
83.71%
Redundant Words
23.80%
Incorrect Word Order
R-Type Filler Words
77.20%
Figure 2: Distributions of conversion types and their subtypes.
4.2 Conversion Type Distribution
To assess the distribution over different conver- sion types, we calculate the proportions of each conversion type present in the dataset. Results are illustrated in Figure 2. Since sentences in our dataset often contain multiple conversion types, the percentage of each conversion type is obtained by dividing the number of occurrences of the conver- sion type by the total number of conversions in the source sentence. As shown in Figure 2, disfluency dominates in conversion problems, and notably, the percentage of R-type conversion problems, which is often challenging to rectify, is substantially high. Grammatical errors constitute a relatively minor portion, accounting for 6.9% of the dataset. Within this category, the principal error types include word missing and word redundancy.
Traditional disfluency correction models com- monly delete the portion preceding the disfluency to generate grammatically correct sentence. How- ever, when confronted with this specific pattern, such an approach could potentially compromise the accuracy the correction process.
Experiment
We conducted experiments on the curated dataset to evaluate the performance of the advanced open- source large language models (LLMs)) Chinese spoken-to-wee toeiitet style conversion.
5.1 Dataset
4.3 Analysis on Sentences with Multiple Conversion Problems
We ensured randomness in data selection pro- cess by shuffling all the annotated texts. Subse- quently, we partitioned these texts into training, development, and test sets, maintaining a distribu- tion ratio of 8:1:1. The training set encompasses 5,789 texts, while both the development and test sets consist of 724 texts each.
In contrast to previous disfluency detection and grammatical error correction datasets, where each sentence typically has only problem, as de- picted in Figure 3, our CS2W dataset presents notable departure. Specifically, we find that 25.34% of the single sentences within CS2W exhibit multi- ple conversion problems, with a significant 31.86% these sentences displaying more than two types
5.2 Baselines
We used the following baseline models:

---

Number of Conversion Problems
9
GE ATE
Multiple Types of Conversion Problems Single Type Conversion Problems
8
CW & ATE
7
& CW
6
30
D ATE
20
5
[42
4
124
& GE
234
&
305
3
343
CW
439
2
1,288
1
5,404
D
5,700
10
100
1000
10000
10
100
1 1000
Number of Sentences
Number of Sentences
(a) Statistics on number of conversion problems contained (h) Statistics on the types of conversion problems contained in single sentence.
in single sentence.
Figure 3: Statistics the number and type of conversion problems contained in single sentence. The horizontal axis uses logarithmic coordinates. D: Disfluency. ATE: ASR Transcription Errors. GE: Grammatical Errors. CW; Colloquial Words.
+ Barnhse
x
T
BELLE (Yunjie Ji, 2023): a model based on BLOOM and finetuned with Chinese data combined with 50,000 pieces of English data, resulting in good Chinese instruction follow- ing and response generation capabilities.
only
pandas, but also Iato fomittra.
RP
RM
# &R STUNTARE Bumber
furaitur.
H f 7F- At Rt to
iE # AK $$ a X
Bombeon is not hed for pandon, but can alon made inmo forsttars.
ChatGLMP: an open bilingual language model based on General Language Model (GLM) (Du et al., 2022) framework, trained IT tokens of Chinese and English corpus, supple- mented by supervised fine-tuning, feedback bootstrap, and reinforcement learning with hu- man feedback.
A new type disfluency example, RM=Rpparandum, RP=Repair; Generally, RM is before RP.
Figure 4:
BART (Lewis et al., 2020): an encoder- decoder language model where bidirectional encoder is used to encode inputs and left-to- right decoder is used generate outputs. We used the Chinese BART proposed by Shao et (2021).
GPT3.5-turbo" model trained diverse data sources, which an advanced language model based on the GPT-3.5 architecture.
We fine-tuned BART and CPT with the training set of CS2W and tested BLOOM. BELLE, ChatGLM, and GPP.355turbi under the zero- and 5-shot set- tings. The demonstrations used for the 5-shot set- ting and the method of selecting them are described in detail in Appendix C. The numbers of parame- ters of the models, as well the hyperparameters, described in detail in Appendix D.
CPT (Shao et al., 2021): a language model that shares knowledge between natural lan- guage understanding (NLU) and natural lan- guage generation (NLG) to boost performance. We used CPT-base? and CPT-largee
BLOOM (Seao al., 2022): a decoder-only transformer language model that trained on the ROOTS corpus (Laurençon et al., 2022), which contains huge amount of data for 46 natural languages and 13 programming lan- guages.
5.3 Metries
We evaluated models with BLEU (Papineni et al., 2002) and ROUGE-L (Lin, 2004). Referring most grammatical error correction tasks, we also used the word-based MaxMatch scorer to calculate
http://nnnnnenec/////bbbbbbbabaa "htpss:/hugginggaace
http:/gggaaacaac////////p/// co/fnlp/cpt-buss http://hwwggnnaaaaoooooo/o/o//ap
"https//ggttuu..om/TTUUUUDDAD wtt.e/ooo//////eee COw/THHLDMCCCCHGGLM.... 10 "hhnps//oppeaai.commopoooee

---

Model
Setting
Fb.s
R-2
B-3
B-4
Rouge-I.
BAKT-base
0.3049
0.3521
0.2641
0.1391
0.0434
0.0113
0.0020
0.4259
BaRe-large
0,4486
0,5025
0.3351
0.3051
0.1476
0.0696
0.0323
0,5129
fine-tuning
CPT-base
0,8421
0,8503
0.8255
0.7348
0.6598
0.6026
0.5470
0.8539
CPT-large
0.8469
0.8526
0.8377
0.7376
0.6669
0.6117
0.5577
0.8589
BLOOM-7B
0,4400
0.4097
0.7140
0.2783
0.2203
0.1808
0.1441
0,4319
BELLE-7B-002M
0,5583
0.5537
0.6013
0,4390
0.3076
0.2249
0.1631
0,5215
BELLE-7B-2M
0-shot
0.6262
0.6232
0.6651
0.4829
0,3574
0.2731
0,2040
0.5735
ChatGLM-6B
0.6414
0.6295
0.7227
0.5765
0,4741
0.4134
0.3627
0.6892
GPT3.5-turbo
0,7285
0.7260
0.7673
0.5846
0.4802
0,4042
0.3373
0.7023
BLOOM-7B
0,4520
0.4180
0,7480
0.2982
0.2409
0.2000
0.1609
0,4601
BELLE-7B-002M
0,5376
0,5289
0.6276
0.2903
0.2186
0.1733
0.1347
0,5135
BELLE-7B-2M
5-shot
0.6308
0.6282
0.6662
0.4885
0.3519
0.2790
0.2167
0,5860
ChatGLM-6B
0.6648
0.6593
0.7326
0.5924
0,4872
0.4261
0.3785
0.6943
GPT3.5-turbo
0,7778
0.7777
0.7954
0.6882
0,5862
0.5087
0.4393
0.7466
Table 5: Automatic evaluation results for models on the CS2W dataset.
P//F values. The method for calculating these values is described in Appendix E.
CPT llarge
GPT3.5
Type
Human
-turbo
Disfueecc
87.99
80.53
75.72
ASR
5.4 Main Results
Transcription
77.77
71.21
90.47
Table presents the performance of benchmark models on CS2W. Notably, the fine-tuned CPT- large delivers outstanding results across all met- rics. Importantly, significantly outperforms the advanced LLMs under both the zero-shot and few- shot (5-shot) settings. This underscores the contin- ued significance of CS2W in the context of Chinese Spoken-to -pokennto-Written Written Style Conversion, even in the era of LLMs.
Errors
Grammatical Errors
83.59
80.73
61.54
Colloquial Words
57.96
54.58
86.73
Mixed Type
74.21
62.93
84.12
Table 6: for different conversion types.
Fo.s scores for CPT-large and GPT3.5-turbo
Under the zero-shot setting, ChatGLM exhibits a slight advantage over GPT3.5-turbo in terms of BLEU-3 and BLEU-4 scores, while GPT3.5-turbo excels the remaining metrics. contrast, under the 5-shot setting, GPT3.5-turbo emerges as the frontrunner, achieving the highest scores across all metrics. It's worth noting that, except for BELLE- 7B-0.2M, all models demonstrate improved perfor- mance under the 5-shot setting when compared to their shot setting performance.
5.6 Contribution of Spoken-to- Mooken-to-Written Language Conversion to Downstream Tasks
To assess the impact of spoken-to-written language conversion on downstream tasks, we employed Chinese-to-Enng tonEnlisshh machine translation as a repre- sentative task. We randomly selected 100 normal- ized text references from the test set and then man- ually translated them into English to serve as refer- ences for the Chinese-to-Englissh machine transla- tion task.
5.5 Performance on Four Conversion Types
We utilized OPUS-MT'! (Tiedemann and Thot- tingal, 2020) to translate the source sentences these 100 references, along with the corresponding outputs of CPT-base and CPT-large. This approach allowed to quantitatively evaluate the effect of booken-to-writter style conversion the Chinese- to-English machine translation task. The results, presented in Table underscore the substantial positive impact of spoken-to totn-tttww-ttter language con- version as a preprocessing step for spoken language.
Table 6 presents the benchmark model's perfor- mance across various types of conversions. The results from human evaluation are based on the ac- curacy of the first round of annotation. The results show that CPT-large and GPT3.5-ture performed similarly across different conversion types. They achieve the highest accuracy in correcting disflu- encies while encountering more challenges in ad- dressing colloquial words, which aligns with the distribution of problems observed in CS2W.

---

BLEU-1
BLEU-2
BLEU-4
Conclusion
Source
0.587
0.511
0.410
In this paper, we have presented a dataset CS2W. which is the first open-source Chinese spoken-to- written conversion dataset. The dataset covers four types of conversion problems commonly occurring in Chinese spoken texts. We manually annotate the type and span for cach conversion problem and provide high-quality written style normalized texts. The dataset is used benchmark testbed to evaluate the performance of advanced L.MMs on spoken-to-writee style conversion and would promote future research on this underexplored di- rection.
sentence
CPT-base 0.711
0.635
0.524
CPT-large
0.727
0.645
0.536
Table BLEU-I, BLEU-2, and BLEU-4 of the source sentence and the corresponding outputs of CPs-base and CPT-large on the Chinese-to-Engllin translation task.
0.07
0.065
0.06
0.05
0.045
0.04
Limitations
0.03
For all zero-shot and few-shot experiments, we used the same prompt for all models. However, prompt selection important for large language models. We plan to use more prompts and prompt engineering methods to conduct experiments on the curated dataset in the future.
0,025
0.02
1.1B
1.7B
3B
7.1B
Figure 5: BLEU--2 scores for BLOOM series models with different numbers of demonstrations.
Ethics Statement
Notably, the BLEU-4 score shows an improvement 0.126 after the conversion done by CPT-large.
All data used in this study freely available to the public. The raw data are from public dataset built in previous work. We follow the policy of using these data without infringing any copyright issues. The curated dataset this study is for academic research purposes only. All annotators are well paid according to the number of their annotations.
5.7 Results with different numbers of demonstrations provided to LLMs
We conducted further experiments on the BLOOM series models to investigate the impact of the num- ber of demonstrations in the prompts on these LLMs. Specifically, we selected BLOOM series models with parameter sizes of 12 13 1B14, and 7. and performed experiments across range of demonstration numbers, from 0 to 5. The outcomes of these experiments are graphically represented in Figure 5. Most models achieve significant improvement under the 1-shot setting, but performance drops and then increases with more demonstrations. We conjecture that this is due to model sensitivity to prompts. From zero to one demonstration, the model performance im- proves significantly due to the relevant prompts. In experiments with two three demonstrations, limited prompt diversity leads to performance decrease. With more demonstrations and diversity, the model performance gradually increases again.
Acknowledgements
The present research was supported by the Key Research and Development Program of Yunnan Province (No. 202203AA080004) and Zhejiang Lab (No. 2022KHOAB01). We would like thank the anonymous reviewers for their insightful com- ments.
References
Christopher Bryant, Mariano Felice, Øistein E. Ander- sen, and Ted Briscoe. 2019, The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52-75, Florence, Italy. Association for Computational Lin- guistics.
12 http::///wugiiggffee.oo/ggeeeeee lb1
Eugene Charniak and Mark Johnson. 2001. Edit detec- tion and parsing for transcribed speech. In Second Meeting of the North American Chapter of the Asso- ciation for Computational Linguistics.
13 "htps://huggingface htt.://ggggggggg.ooo///cceeeee/e co/bigscience/lloom-l 1b7
1 "https:/hugginggaace co/hisscience/llomm331
#htps://huggingtacc.

---

Mai Hoang Dao, Thinh Hung Truong, and Dat Quoe Nguyen. 2022. Disfluency detection for Vietnamese. In Proceedings of the Fighth Workshop Noisy User-generated Text (W-NUT 2022), pages 194-200, Gyeongju, Republic of Korea. Association for Com- putational Linguistics.
Paria Jamshid Lou, Peter Anderson, and Mark Johnson. 2018. Disfluency detection using auto nuto-correlational neural networks. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 4610-4619, Brussels, Belgium. Association for Computational Linguisties.
Vidas Daudaravicius, Rafael E. Banchs, Elena Volod- ina, and Courtney Napoles. 2016. A report on the automatic evaluation of scientific writing shared task. In Proceedings of the 1 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 53-62, San Diego, CA. Association for Com- putational Linguistics.
Mark Johnson and Eugene Charniak. 2004. A TAG- based noisy-channel model of speech repairs. In Proceedings of the 42nd Anmual Meeting of the As- sociation for Computational Linguistics (ACL-04), pages 33-39, Barcelona, Spain.
Mark Johnson, Eugene Charmiak, and Matthew Lease. 2004. An improved model for recognizing disfluen- cies in conversational speech. In Proceedings of Rich Transcription Workshop.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Hugo Laurençon, Lucile Saulnier, Thomas Wang. Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Ed- uardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 th composite mul- tilingual dataset. Advances in Neural Information Processing Systems, 35:31809 35:31809-31826
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding. Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 320-335.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising eequence-to-sequenee pre-training for natural language generation, tramslation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880 Online. Association for Computa- tional Linguistics.
James Ferguson, Greg Durrett, and Dan Klein, 2015. Disfluency detection with a semi-Markov model and prosodic features. In Proceedings of the 2015 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 257-262, Denver, Col- orado. Association for Computational Linguistics.
Piji Li and Shuming Shi. 2021. Tail-to-tail non- autoregressive sequence prediction for Chinese gram- matical error correction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Confer- ence on Natural Language Processing (Volume 1:
J.J. Godfrey, E.C. Holliman, and J. McDaniel. 1992. Switchboard: telephone speech corpus for research and development. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517- 520 v.
Long Papers), pages 4973-4984, Online. Association for Computational Linguistics.
Chin-Yes Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- lion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Charles Hinson, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Heterogeneous recycle generation for Chi- nese grammatical error correction. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2191-2201, Barcelona, Spain (On- line). International Committee Computational Lin- guistics.
Ruixuan Luo, Jingjing Xu, Yi Zhang, Zhiyuan Zhang, Xuancheng Ren, and Xu Sun. 2019. Pkuseg: A toolkit for multi-domain chinese word segmentation. arXiv preprint arXiv: 1455.
Matthew Hounibal and Mark Johnson. 2014. Joint incre- mental disfluency detection and dependency parsing. Transactions of the Association for Computational Linguistics. 2:131-142.
Evgeny Matusov, Ame Mauser, and Hermann Ney. 2006. Automatic sentence segmentation and puncu- ation prediction for spoken language translation. In Proceedings of the Third International Workshop on Spoken Language Translation: Papers, Kyoto, Japan.
Mana Ihori, Akihiko Takashima, and Ryo Masumura. 2020. Parallel corpus for Japanese spoken-to- toeen-tt-warr to-written style conversion. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 6346-6353, Marseille, France. European Language Resources Association.
Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and bench- mark for grammatical error correction. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics:

---

Volume Short Papers. pages 229-234, Valencia, Spain. Association for Computational Linguistics.
Pavlos Vougiouklis, Hady Elsahar, Lucie-Aimé Aiméé Kaffee, Christophe Gravier, Frédérique Laforest, Jonathon Hare, and Elena Simperl. 2018. Neural wikipedian: Generating textual summaries from knowledge base triples. Journal of Web Semantics, 52:1-15.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Sander Wubben, Antal van den Bosch, and Emiel Krah- mer. 2010. Paraphrase generation as monolingual translation: Data and evaluation. Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguis- tics.
Gaoqi Rao, Qi Gong, Baolin Zhang, and Endong Xun. 2018. Overview of NLPTEA-2018 share task Chi- nese grammatical error diagnosis. In Proceedings of the 5th Workshop on Natural Language Process- ing Techniques for Educational Applications, pages 42-51, Melbourne, Australia, Association for Com- putational Linguistics.
Zehui Yang. Yifan Chen, Lei Luo, Runyan Yang. Lingx- uan Ye, Gaoleng Cheng, Ji Xu, Yaohui Jin, Qingqing Zhang, Pengyuan Zhang, et al. 2022. Open source magiedata-rame: rich annotated mandarin con- versational (ramc) speech dataset. arXiv preprint arXiv:2203
Gaoqi Rao, Erhong Yang, and Baolin Zhang. 2020. Overview of NLPTTA-2000 shared task for Chinese grammatical error diagnosis. In Proceedings of the 6th Workshop on Natural Language Processing Tech-
Yan Gong Yiping Peng Qiang Niu Baochang Ma Xian- gang Li Yunjie Ji. Yong Deng. 2023. Belle: Bloom- enhanced large language model engine. https: tigbthub.ccm wwt.m//Laanjaaaeec/BLLLLL
niques for Educational Applications, pages 25-35, Suzhou, China. Association for Computational Lin- guistics.
Vicky Zayats and Mari Ostendorf. 2019. Giving atten- tion to the unexpected: Using prosody innovations in disfluency detection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 86-95, Minneapolis, Minnesota. As- sociation for Computational Linguistics.
Keisuke Sakaguchi, Courtney Napoles, Matt Post, and Joel Tetreault. 2016. Reassessing the goals of gram- matical error correction: Fluency instead of grammat- icality. Transactions of the Association for Computa- tional Linguisties. 4:169-188
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvo,, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.00
Vicky Zayats, Mari Ostendorf, and Hannaneh Hajishirzi. 2016. Disfluency detection using bidirectional Istm. Interspeech 2016.
Yuanyuan Zhao, Nan Jiang, Weiwei Sun, and Xiaojun Wan. 2018, Overview of the nlpce 2018 shared task: Grammatical error correction. In Natural Language Processing and Chinese Computing.
Abigail See, Peter J. I Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguisties.
Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. 2021. Cpt: A pre-trained unbalanced transformer for both chinese language understanding and genera- tion. arXiv preprini arXiv:2109.05729.
Elizabeth Shriberg, 1996. Disfluencies in switchboard. In Proceedings of international conference on spo- ken language processing, volume 96, pages 11-4. Citeseer.
Jörg Tiedemann and Santhosh Thottingal. 2020. Opus- mt-building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. European Association for Machine Translation.

---

A Data Extraction
First Level
Second Level
R-type
MagicData-RAMC is ASR transcription dataset that comprises 351 sets of spontaneous conversa- tions in Chinese Mandarin. Each set features nat- ural dialogues between two speakers discussing a single topic, and it includes both audio files and transcribed texts that preserve disfluencies, gram- matical errors, and ASR transcription errors. We manually select sentences from the ASR transcrip- tions, ensuring they are self-contained in meaning but with conversion problems for annotation.
Disfluency
Filler words
ASR Transcription Errors
Missing Words
Grammatical Errors
Redundant Words
Incorrect Word Order
Colloquial Words
Table 8: Two-level classification system of conversion problem types.
In transcribed text, we do not extract the follow- ing three types of sentences: Incomplete Sentences, Sentences that require context, and Sentences that are too short.
In conclusion, we select complete sentences with conversion problems whose lengths are appropri- ate.
A.1 Incomplete sentences
spontaneous conversations, a speaker often breaks off abruptly or is interrupted by another speaker, resulting in many incomplete sentences in transcriptions.
B
Annotation Guidelines
B.1 Task Statement
Automatic speech recognition (ASR) plays vital role in a wide range of NLP application scenarios. Spoken language, which serves as a fundamental input for plenty of downstream tasks, is transcribed into written text in spoken style. However, they often inherently contain disfluencies, grammatical errors, and colloquial words. This dataset consists of transcribed texts with conversion problems for annotation. For each sentence, the annotator needs to annotate the type and the range of conversion problems and write the corresponding written lan- guage.
$ (He and I were just getting ready hang out but.)
e.g. # fé RI Rll & t4 3E.
This indicates that the speaker stops speaking at the word "but" or is interrupted by someone else. Incomplete sentences cannot be easily understood by annotators and, as result, may not be accurately annotated.
A.2 Sentences that require context
Spontancous conversations have continuity and many sentences need to be understood in context.
e.g. Speaker 1: $ A X $ # $ # T $ + 3 (Speaker 1: What grade did you get on this
B.2 Conversion Type Definitions and Examples
math test?) )
We further categorize the conversion problem with a two-level classification system shown in Figure 8. Each conversion problem is described in detail next.
Speaker (Speaker 2: little over a hundred, no more than you. )
3 h
Speaker 2's speech is response to Speaker I's question, which may not be fully comprehensible without considering Speaker I's speech. Sentences that require context can also be challenging to an- notate accurately.
B.2.1 Disfluency
The elements that make a sentence not fluent are referred to as "disfluency", which can be catego- rized into R-type and Filler Words based on their structures.
A.3 Sentences that are too short
Filler Words Filler Words, such as "uh" and "ah," have no specific meaning and are often used to indicate pauses and hesitations in the speaker's discourse. Additionally, common words like "yeah" and "okay" are also sometimes classi- fied as filler words.
In spoken language, phrases such "no problem," "yes." and "okay" are frequently employed, and these are considered too general. Therefore, we refrain from selecting sentences with fewer than
tokens.

---

e.g. BE. 3 8 - (Well, this is, you know, good plan.) In this sentence, the phrases "well" and "you know" lack specific meaning and should be anno- tated as "Filler words."
conversations often lack careful thinking. Com- mon grammatical errors in spoken language in- clude Missing Words, Redundant Words, and In- correct Word Order.
Missing Words Missing Words include miss- ing subjects, missing predicates, missing objects, missing function words, and missing modifiers. e.g. sps # # A
R-type The standard structure of R-type disflu- ency encompasses three elements: the reparandum, an optional interregnum, and the associated repair. The reparandum consists of words that the speaker initially intends to discard, representing an unin- tended inclusion in the utterance. This section typically comprises one or more words slated for repetition correction. The interregnum, often comprising fixed phrases like "uh" or "you know," serves as a non-lexicalized component, contribut- ing filler words without specific meaning. Lastly, the repair phase involves correcting or repeating words from the reparandum, thereby refining the overall coherence of the utterance.
f #
* t. 2 (We had chance to equalize, but we didn't it.)
This sentence is missing a verb. The correct sentence is "We had chance to equalize, but we didn't i take it".
Redundant Words Redundant Words include dundant subjects, redundant predicates, redundant objects, redundant function words, and redundant modifiers.
e.g. ET A H 33 # * (Their fur is shiny and can be hardly seen with the naked eyc.)
* 45 i it * de i A [ (Let us, I mean, let me work on the problem.)
e.g.
it t I1]
The modifier in this sentence is redundant. The fur is shiny it should be visible to the naked eye. The word "hardly" should be deleted.
In the provided example, the sentence "Let us" functions as the reparandum, embodying the words originally unintended for inclusion. The subse- quent phrase "let me" constitutes the repair, correct- ing the preceding reparandum. The interregnum in this instance is "I mean," " non-essential filler phrase devoid of substantive meaning.
Incorrect Word Order Incorrect Word Order is also common in spoken transcribed texts because of the frequent inversions in spoken language. e.g. # R J $ & 69 - ap è $3 & & ve 1. (Yesterday watched a newly purchased movie I on TV.)
In Chinese, the correct sentence is "Yesterday watched a newly purchased movie on TV."
B.2.2 ASR Transcription Errors
ASR Transcription Errors occasional homo- phone mistakes in ASR transcriptionss CS2W is built upon the existing MagicData-RAMMC dataset (Yang et al., 2022), which comprises 351 sets of spontancous conversation speech in Chinese Man- darin and their ASR transcriptions. Consequently, there are occasional homophone mistakes in some sentences.
B.2.4 Colloquial Words
Spoken language often contains informal expres- sions, such as some popular Internet phrases, which are called " "Colloquial Words."
e.g. 3 nd # $ - credit, but your coworkers took it away. You're such an unlucky guy.)
&f 46 $ (This is obviously your
#
e.g. 1 Π cauliflower.! 16)
(The artist is very
We need to replace all informal expressions with formal ones. In Chinese, the Internet phrase * # is used to describe people who are aggrieved but have no way complain. Therefore, we should replace it with the more formal expression "You" unlucky."
According to the intended meaning of the sen- tence, the correct version should be "The artist is very talented." " Therefore, "cauliflower" needs to be annotated an "ASR Transcription Error."
B.2.3 Grammatical Errors
B.2.5 Mixed Type
The transcription text of spoken language often includes grammatical errors because speakers in
In real spontaneous conversations. single sen- tence often contains multiple conversion problems.
c.g. A म 1 * & +
"In Chinese, "cauliflower" and "talented" have the same pronunciation.
क #
(The establish-

---

ment of the bison, mean, buffalo research center in the country will help to adapt to an artificial environment.)
B.4 Inconsistent Treatment
To ensure annotation consisteney and quality, we implement two-round annotation process. In the first round, we enlist eight native Chinese speak- ers as part-time annotators after providing pre- annotation training. the second round, we con- duct manual evaluation and re-annotation, with the authors of this paper serving as senior annotators. In the event of anter-annotator disagreements, the annotator in the second round reannotates the sen- tence and submits the results of both rounds to the senior annotator. There are two scenarios of inter-annotator disagreements.
This sentence contains both disfluency and gram- matical errors. The corrected version should be: "The establishment of the buffalo research center in the country will help them adapt to an artificial environment."
B.3 Annotation Rules
We built an annotation platform to accelerate our annotation progress. When using it, the annotator needs to select the conversion type and annotation range of the current sentence. Then, the annotator provides the written language corresponding to this spoken language. Next, we will present the annota- tion rules for different conversions. Please note that on the annotation platform, each sentence is word- segmented into individual words, each of which can be selected make it casier for the annotator to annotate the conversion range.
The first scenario is when the annotator from either the first or second round makes an incorrect annotation.
1t e.g.
말 +
(When he returned to the ga-, garage, had changed markedly.)
The sentence exhibits two conversion problems, namely Disfluency and Missing Words. The an- notator in the first round accurately annotates the disfluency but overlooks the grammatical errors. In the second round, the annotator correctly annotates both conversion problems. The senior annotator will then determine the correct annotation.
Disfluency The annotator selects the Disfluency button on the annotation platform. For Filler Words, the annotator should annotate their range. For R- type, both the reparandum and the interregnum should be annotated, but the repair does not need to be annotated. This is because the sentence can be corrected by simply deleting the reparandum and the interregnum.
The second case is that the sentence can be cor- rected in multiple ways, which is common in gram- matical errors.
e.g. Source sentence: 1 JJ nt E AL $ स. people watch TV programs for four or five hours in a row, will feel very tired.)
के (If
ASR Transeription Errors The annotator se- lects the ASR Transcription Errors button on the annotation platform and annotates the range of the ASR Transcription Errors.
Target sentence 1: A t +) of in t $ H S $ #
(Peo-
The annotator selects the Grammatical Errors button on the annotation plat- form. For Missing Words, the annotator needs to annotate the two words before and after the missing part. For Redundant Words, the annotator needs to annotate the redundant part. As for Incorrect Word Order, the entire sentence has to be annotated.
Grammatical Errors
ple in case of watching TV programs for four or five hours in a row will feel very tired.)
Target sentence 2: to * # 1 #f 69 t $ 11 # (If people watch TV programs for four five hours in a row, they will feel very tired.)
This sentence lacks a subject and has two po- tential solutions. First, considering the conversion as an "Incorrect Word Order," the corresponding written language is target sentence 1. Second, con- sidering the conversion written language is target sentence 2. In this sce- nario, the senior annotator selects the solution with smaller edit distance. If the edit distances are equal, the senior annotator opts for the first-round solution.
Colloquial Words The annotator selects the Col- loquial Words button on the annotation platform and annotates the range of the colloquial words.
Mixed Type First, the annotator selects the but- ton for the initial conversion type and annotates the range of that conversion. Subsequently, the annotator sequentially selects the buttons for the other conversion types and annotates their respec-
"Missing Word," the
tive ranges.

---

C Prompt Templates
In Section 5.4 and 5.7, the demonstrations used in 5-shot experiments are shown in Figure 11.
C.1 Instructions Used in Main Results
C.3 Impact of Demonstrations Diversity
Regarding the prompts used for zero- and few- shot settings, we tried two different prompts on BLOOM-7B under the zero-shot and 5-shot set- tings.
While our initial intuition suggested that including a higher diversity of conversion types in demonstra- tions would enable LLMs to address more types of conversions and subsequently improve results, our findings have led us to reconsider this notion. To test this hypothesis, we randomly selected five demonstrations from the validation set and repeated the selection process three times, resulting in three distinct prompts used for the 5-shot setting. The specific demonstrations included in each prompt are detailed in Table 12. Notably, Prompt in- corporates demonstrations with three different con- version types, offering the highest diversity, while Prompt 2 exclusively includes demonstrations fea- turing disfluency problems, resulting in the lowest diversity.
Prompt 1:
M5. 6] $ i (Here's spoken-to to-wiitten style conver- sion task, please rewrite the spoken language into the written language: spoken: [source sentence] written:).
t t $ i # is D
Prompt 2: "F fi - - $,
1 to $ x 2 05 i [8
(Here's a grammatical error cor- rection task, please correct the wrong text into the right text: wrong text: {source sentence) correct text:
)) D
Prompt 2 outperforms Prompt 1 on all metrics. We speculate that the model may struggle to com- prehend the definition of the spoken-to-we torritn lan- guage conversion task. However, grammatical error correction a widely-used task, and the four con- version problems, except for colloquial words, can be considered either simple or complex grammati- cal errors, allowing it to perform well. Hence, we adopt Prompt 2 for the rest of our experiments.
We tested DELLE-7B-2MM ChatGLM-6B, and GPT3.5-turbo under the 5-shot settings with these three prompts. The results, in comparison to those presented in Section 5.4, are summarized in Table 13.
In summary, our experimental findings challenge our initial hypothesis that greater diversity in ex- amples would consistently enhance the ability of LLMs to address a wider range of conversion types and lead to improved results. Surprisingly, it is not always the case. The prompt with the most diver- sity, Prompt 1, displays the weakest performance. In contrast, Prompt 2 and Prompt 3, which feature fewer types of conversions but higher concentra- tion of demonstrations with disfluencies, delivered more favorable results.
C.2 Demonstrations Used in Main Results
Under the zero-shot setting, the prompt is "F to -
# ff $, क th of # x JE 4 # c x
6) JE अक $
(Here's a grammatical error correction task, please correct the wrong text into the right text: wrong text: (source sentence) correct text:).
We attribute this phenomenon to the prevalence of disfluency within the CS2W dataset. When the number of demonstrations with disfluencies sur- passes certain threshold, the overall performance tends to improve. However, it's worth noting that the dataset's 's distribution of conversion types may play a pivotal role in these results. In a scenario where the four conversion types were more bal- anced, prompts with greater diversity might have exhibited improved performance.
We expect LLMS to be capable of correcting all types of conversion problems. Therefore, it is crucial ensure the diversity of demonstrations provided LLMs. Given that CS2W encompasses four types of conversion problems, we incremen- tally add demonstrations of different types as the number of demonstrations increases. The demon- strations for each conversion problem are randomly selected from the validation set. These demonstra- tions differ from the input sentences in the test set and contain only one conversion problem. Addi- tionally, since CS2W is dominated by disfluency as the primary conversion problem, we specifically select two demonstrations, one for R-type and an- other for Filler Words, under the 5-shot setting.
D Model Parameter Sizes and Hyperparameters
The numbers of parameters of the models used in the experiments are shown in Table

---

Model
Parameter Count
BART-base
139M
BART-large
406M
CPT-base
121M
CPT-large
393M
BLOOM
7B
BELLE
7B
ChatGLM 6B
Table 9: The number of parameters of the models used
the experiments.
Model
Decoding Temperature
BLOOM-7B
1.0
BELLE-7B-0.2M
0.35
BELLE-7B-2M
1.0
ChatGLM-6B 0.95
GPT3.5-turbo
0.7
Table 10: The decoding temperature of LLMs used in the experiments.
The decoding temperature of LLMs used in the experiments is shown in Table 10.
E P/R/F Calculation Method
Firstly, the gold sentence and the models' outputs are word-seemmented using the PKUNLP word seg- mentation (WS) tool (Luo et al., 2019), and then we calculate the number of maximal matches ac- cording to the WS results. The P. R. F0.5 measure the different rates between the set of the model's output edits e1, .... e2 and the set of gold sentence edits 91, ... 92 for all sentences:
P
i
ga
[es]
-= 0
Σm= [9:]
1.25 P
R
Fo.5 0.25
+ R
Where we define the intersection between e and gi as:
- f fe Bg (match((eg))

---

Demonstration
Demonstration
Number
Conversion Type
(Wrong Text)
(Correct Text)
5 X $ 38 #x
& 3 #
Disfiuency
I don't I don't like this type of music.
I don't t like this type of music.
R-type
46% 18 Aa 1 65 t &
46 ia 1h 66 tt & J
Disfluency
2
You, uh have been drinking lot.
You have been drinking lot.
Filler words
& & 2 to &t A
& & & if té @t A
3
Grammatical Errors
My dad can sweet and sour dumplings.
My dad
make sweet and sour dumplings.
& a2 th 28 n a.
4., t iif 18, 8.
4
Colloquial Words
want to spend money on the game.
want to recharge in the game.
i3 14 $ 1/ A t $
33 34 $ ti & Æ 9 $
ASR
5
Transcription
I have nothing report in this matter
I have nothing to say in this matter.
Errors
Table 11: Five demonstrations used in main experiments.
Demonstration
Demonstration
Conversion
Number
(Wrong Text)
(Correct Text)
Type
Prompt
# 2 $ A
# & & A
Colloquial
Demonstration
Why do try to trap other people?
Why do try to lie to other people?
Word
Prompt
Ab tb il: # & lē E.
At il 4i #H & as %
Disfluency
Demonstration
He he asked me to pretend to sleep.
He asked me to pretend to sleep.
X + & RL 6b E i. Bj - of Ni Ms. The male protagonist main male cha- racter of this drama is named Jin Ran. AE 3 V 65 $1 (R $3 $ 0 F $1 8 1t # The leg can reach lot places your arm can't. 16 7 * A
it + e RL. ii 3 - + 4 M.
Prompt
The main male character of this dra- named Jin Ran.
Disfluency
Demonstration
BE o E At $1 &R $5 N x $1 6h N # The leg can touch lot places your arm can't. f -
Prompt
Colloquial Word
Demonstration 4
Disfluency and Grammatical
98
f 18 9- #
Prompt Demonstration
He disappeared for 28 years and he finally developed the like the, the hydrogen bomhs and atomic bombs.
He disappeared for 28 years and he finally developed weapons like hyd- rogen hombs and atomic hombs.
Error
A x #E - X # 48 #
Prompt Demonstration
A 4 l * + * They will cook whole table of food.
They will cook, uh, whole ta. table of food.
Disfluency
38 M. SE AR dl & # #7 #3
13 st. At IR dl: # * 85 # 22 These are some the benefits that come with career.
Prompt 2 Demonstration
These are some the be,, benefits that come with career.
Disflueney
4 & 4'1 11 tt. & as is & a ( If you, we analyze him rationally, we will find that he is very clever. 4/ f at X & iE i & x You you will still be hungry even after eating.
# (1 t a li is 4 fe
Prompt 2 Demonstration
If we analyze rationally, we will find that he is very clever. #t & J AE i # You will still be hungry even after eating.
Disfluency
Prompt
Disfluency
Demonstration
dà de f6 Af At $ J t # #
i8 J6 f X St f $ t # T AL* Her master won the championship with real strength.
Prompt
Her, her master won the championship with real strength.
Disfluency
Demonstration
3 B 1 3
# TT f # i
Prompt # i
Disfluency
Demonstration
In addition, in addition to work. many people continue their studies. # 4 7 & # it & $ of r ? Do you like. uh traveling not? $ # T क # FK.) Zhang Sanfeng has historical mate- rials trap find.
In addition to work, many people continue their studies.
Prompt 3 Demonstration
# 4 7 a # Do you like traveling?
Disfluency
$ # A म #
Prompt Demonstration
Grammatical Error
Zhang Sanfeng has historical mate- rials to find.
i $
& x
i $ i8
Prompt 3
One, one the most impressive part- time jobs I've ever had was courier. courier sorting.
One of the most impressive part- time jobs I've ever had was courier sorting.
Disfluency
Demonstration 4
# 3J 1E 2) tit a I'm getting used to, used to going
f T a
Prompt
I'm used to going to bed early and getting early.
Disfluency
Demonstration
to bed early and getting up early.
Table 12: Demonstrations used in the three prompts.

---

Prompt
Model
BLEU-1
BLEU-2
BLEU-3
BLEU-4
BELLE-7B-2M
0.4885
0.3519
0.2790
0.2167
ChatGLM-6B
0.5924
0.4872
0.4261
0.3785
GPT3.5-turbo
0.6882
0.5862
0.5087
0.4393
BELLE-7B-22
0.4593
0.3465
0.2688
0.2042
ChatGLM-6B
0.6600
0.5445
0.4596
0.3847
GPT3.5-turbo 0.6260
0.5215
0.4442
0.3748
BELLE-7B-2M 0.4558
0.3532
0,2790
0,2166
2
ChatGLM-6B 0.6932
0.5977
0.5225
0.4530
GPT3.5-turbo 0.6918
0.5971
0.5249
0,4586
BELLE-7B-2M 0.4559
0.3598
0.2910
0.2312
3
ChatGLM-6B 0.6842
0.5822
0.5043
0.4324
GPT3.5-turbo 0.7331
0.6131
0.5333
0.4644
Table 13: The BLEU-I, BLEU-2, BLEU-3, and BLEU-4 scores using BEL.FE-B--M,, ChaiGLM-6B, and GPT3.5- turbo, each with the three prompts. To facilitate comparison, we include the results from Section 5,4 in this table, and we refer to the prompt used in Section 5.4 as "Prompt 0". The results "Prompt 0" are considered the baseline, and any improvements over Prompt 0 are highlighted in orange, while decreases marked in blue.


Reviews:
[{"review_id": "SpIbW0FJ9Y", "reviewer": "Reviewer_7Dmy", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments and suggestions.\n\n**Q1:** The downstream task performance is not convincing of general gains on any downstream task. It's hard to tell if the improvements only show on the specific setting of translation.\n\n**A1:** Thank you for your question. For the machine translation tasks, we randomly selected 100 source sentences in the test set and manually translated their corresponding gold written sentences into English as references. Then we use the OPUS Chinese to English translation model to translate the source sentences and the output from the fine-tuning model (CPT-base, CPT-large) into English, and calculate BLEU. \n\nIn order to demonstrate the enhancement of the spoken-to-written style conversion on the machine translation task, we increase the number of test cases. We randomly selected 200 new source sentences in addition to the 100 sentences already selected in section 5.6 and repeat the experiments. The results are shown in below table.\n\n|                 | BLEU-1 | BLEU-2 | BLEU-3 |\n| :-------------: | :----: | :----: | :----: |\n| Source sentence | 0.7672 | 0.7272 | 0.6565 |\n|    CPT-base     | 0.8293 | 0.7773 | 0.7011 |\n|    CPT-large    | 0.8520 | 0.8064 | 0.7354 |\n\nThe results show that that spoken-to-written language con version can still significantly enhance the performance of machine translation tasks when the test cases are increased to 300. BLEU-4 can increase 0.0789 with CPT-large.\n\n**Q2:** Disfluency is the primary conversion type in CS2W, it is not clear if this is expected, or an artifact of CS2W. Comparison of such stats with other datasets would be useful, if possible.\n\n**A2:** Thank you for your question.\nIn the field of spoken-to-written style conversion, besides CS2W, publicly available datasets include Japanese S2W [1] , which is mentioned in related work section (Line 219). Similar to our dataset, S2W is extracted from transcribed texts of spontaneous conversations. Chinese and Japanese have different linguistic conventions, thus making the type of conversions different. S2W contains conversion types: notation correction, postpositional particle expressions restoration, disfluency, and simplification. Unfortunately, the authors do not calculate the percentage of each conversion type although disfluency is also considered.\n\nIn addition to S2W, a parallel corpus of formal-informal sentences in Persian [2] has recently been posted on arXiv (on Aug. 10, 2023). In this corpus, informal sentences are spoken language or in a written style but used in informal conversations and social media. Raw data for this corpus comes from social networking sites, books, and movie scripts, while our data source is transcribed text from spontaneous conversations. It is demonstrated that disfluencies are more prevalent in spontaneous speech [3]. However, in the Informal-Formal Persian Corpus, the authors define only two types of transformations, lexical and syntactic, without focusing on disfluencies.\n\nIn the field of disfluency detection, Switchboard is the popular dataset, which is an English ASR telephone spontaneous conversations dataset, labeled with laughter, pause, disfluency, etc. The sentences with disfluency comprise for 5.9% of all sentences. However, as a spoken-to-written style conversion dataset, each sentence of CS2W has conversion problems, while the vast majority of sentences in the Switchboard are normal sentences that have no conversion problems. Therefore, the two statistics cannot be directly compared.  \n\nIn the field of dialog and Q&A, there are dialog and Q&A datasets annotated with disfluency [4][5]. However, they generally only annotate disfluency without considering other problems. This also reflects the fact that disfluency is a major problem in spontaneous speech. \n\nIn addition, there are several large spoken transcription datasets similar to Switchboard  annotated with pauses, laughter but without disfluency.\n\nIn summary, unfortunately, there are no other spoken-to-written style conversion dataset or spoken transcription datasets that have provided the proportion of sentences with disfluency. Our work is the first spoken-to-written style conversion dataset in Chinese, and we are the only one that has explored the conversion problem in spoken Chinese and computed the proportion of each conversion. Disfluency is a common problem in spontaneous speech, which is why disfluency detection has always been widely studied. Therefore, it is also intuitive to believe that disfluency dominates in spoken language that needs to be converted to written language.\n\n[1] Mana Ihori, Akihiko Takashima, and Ryo Masumura. 2020. Parallel corpus for japanese spoken-to-written style conversion. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 6346–6353. European Language Resources Association.\n\n[2] Vahide Tajalli, Fateme Kalantari, and Mehrnoush Shamsfard. 2023. Developing an informal-formal persian corpus. CoRR, abs/2308.05336\n\n[3] Elizabeth Ellen Shriberg. 1994. Preliminaries to a theory of speech disfluencies. Ph.D. thesis, Citeseer.\n\n[4] Julian Hough, Ye Tian, Laura E. de Ruiter, Simon Betz, Spyros Kousidis, David Schlangen, and Jonathan Ginzburg. 2016. DUEL: A multi-lingual multimodal dialogue corpus for disfluency, exclamations and laughter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC 2016, Portorož, Slovenia, May 23-28, 2016. European Language Resources Association (ELRA).\n\n[5] Aditya Gupta, Jiacheng Xu, Shyam Upadhyay, Diyi Yang, and Manaal Faruqui. 2021. Disfl-qa: A benchmark dataset for understanding disfluencies in question answering. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP2021 of Findings of ACL, pages 3309–3319. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. First Chinese language spoken to written style conversion dataset with more than 7000 examples focusing on 4 conversion problems, viz., disfluency, ASR transcription error, grammatical error and colloquial words. These conversion problems are of practical importance, and have significant impact on downstream tasks based on ASR transcripts.\n2. Good presentation of background work, and well-motivated research that explain the need for a dataset like CS2SW comparing to existing datasets in English, Chinese and Japanese.\n3. Well presented annotation methodology as well as sound analysis of the dataset statistics and characteristics.\n4. Benchmark model performance comparing five baselines trained with different pretraining methods, viz, encoder-decoder (seq2seq), decoder only, and decoder only with instruction tuning & RLHF.", "reasons_to_reject": "1. The downstream task performance study is not convincing of general gains on any downstream task. It's hard to tell if the improvements only show on the specific setting of translation shown in the paper.\n2. The dataset is dominated by disfluency (87.4%) as the primary conversion problem. It's not clear if this is expected, or an artifact of CS2W. Comparison of such stats with other datasets would be useful, if possible.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Main contributions can be summarized as:\n\n1. A new curated dataset CS2W, a Chinese Spoken-to-Written style conversion dataset comprising 7,237 spoken sentences extracted from transcribed conversational texts with four types of conversion problems.\n2. Thorough analysis of the dataset, and benchmark evaluation of the on spoken-to-written language conversion with five SoTA baseline models. The analysis led to the discovery of a new type of disfluency.\n3. Annotation guidelines and methodology behind the dataset preparation are shared with satisfactory detail.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r8Fz6R4khc", "reviewer": "Reviewer_3uWx", "summary": "", "questions": "1) In section 4.4, you describe a new sub-type of disfluency. Can you quantify how common this type of disfluency is?\n\n2) Do I understand correctly that you do not use the originally created training, development and test sets (described on lines 323f.) in your experiments (because you use a different split described on lines 400f.)? If so, what is the benefit of the original splits?\n\n3) In Figure 4, the English translation of the example is confusing: In the first line, you translate 不单 as \"is not only\" and 不单独 as \"is not\". However, in the following lines, both versions are translated as \"is not only\". Can you explain this inconsistency? Also, you do not translate 吃. How about, e.g., \"Bamboo is not only a food for pandas, but it can also be made into a lot of furniture.\"?\n\n4) Could you please show/include the prompts you used for zero-shot and few-shot inference?\n\n5) Are there similar datasets in other languages than EN, CH, JA and VI, or did you focus on purpose on this selection of languages?\n\n6) On line 198, you say that CS2W offers two annotation paradigms. Could you clarify what you mean here?\n\n7) Figure 3 b): Unfortunately, I don't understand this figure. Is it the number of spans per samples? (If so, I think you can remove the colors and the labels, as this seems to be redundant.)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments and suggestions.\n\n**Reason to reject:**\n\n**Q1:** The paper does not describe the creation of the corpus in full detail. For example, it does not explain how the samples were selected from MagicData RAMC, and it does not describe the annotation guidelines in detail, and it does not explain how the disagreements were resolved.\n\n**A1:** We will describe data extraction, detailed labeling guidelines, and how to resolve disagreements below.\n1. Data Extraction\n\n    MagicData-RAMC is an ASR transcription dataset consisting of 351 sets of spontaneous conversations in Chinese Mandarin. Each set features natural conversations between two speakers on a single topic, and it includes audio files and transcribed texts that retain disfluencies, grammatical errors, and ASR transcription errors. We manually select sentences from the ASR transcriptions, which are self-contained in meaning but with conversion problems for annotation.\n\n    In transcribed text, we do not extract the following three types of sentences:\n\n    (1) Incomplete sentences. In spontaneous conversations, a speaker often break off abruptly or is interrupted by another speaker, resulting in many incomplete sentences in transcriptions. \n\n     e.g. 我和他刚刚准备出去玩，就(He and I were just getting ready to hang out but)\n\n    This indicates that the speaker stops speaking at the word \"but\" or is interrupted by someone else. Incomplete sentences cannot be understood by annotators and hence cannot be accurately annotated.\n\n    (2) Sentences that require context. Spontaneous conversations have continuity and many sentences need to be understood in context. \n\n    e.g. \n\n    Speaker1：你这次数学考试考了多少分？(Speaker1：What grade did you get on this math test? )\n\n    Speaker2：一百多一点，比不上你。(Speaker2：A little over a hundred, no more than you. )\n\n    Speaker2's speech is a response to Speaker1's question, which is not understandable without Speaker1’s speech. The sentences that require context also can hardly be annotated. \n\n    (3) Sentences that are too short. In spoken language, phrases such as \"no problem\", \"yes\" and \"okay\" are often used, which are too general. Therefore, we do not select sentences with the number of tokens being less than 5. \n\n    In conclusion, we select complete sentences with conversion problems whose length are appropriate.\n\n2. Detailed Annotation Guidelines\n\n    (1) Task Statement\n\n    Automatic speech recognition (ASR) plays a vital role in a wide range of NLP application scenarios. Spoken language is transcribed into spoken style texts which serve as fundamental inputs to plenty of downstream tasks. However, they are often inherently containing disfluencies, grammatical errors, and colloquial words. This dataset consists of transcribed texts with conversion problems for annotation. For each sentence, the annotator needs to annotate the type and the range of conversion problems, and write the corresponding written language. \n\n    (2) Conversion type definitions and examples\n\n    We further categorize the conversion problem with a two-level classification system as: Disfluency (R-type, Filter Word), ASR Transcription Errors, Grammatical Errors (Missing Words, Redundant Words, Incorrect Word Order), Colloquial Words.\n\n    Each conversion problem is described in detail next.\n\n    ① Disfluency\n\n    The elements that make a sentence not fluent are referred to as \"disfluency\", which can be categorized into R-type and Filter Word based on their structures [4].\n\n    **Filter Word:**\n\n    Filter word have no meaning and are often used to indicate pauses and hesitations of the speaker, which includes \"uh\", \"ah\" and so on. Many times, words like \"yeah\", \"okay\" are also marked as filler words. \n\n    e.g. Well, this is, you know, a good plan.\n\n    In this sentence, the phrases \"well\" and \"you know\" have no meaning, and they should be annotated as \"Filter Word\". \n\n    **R-type:**\n\n    The standard structure of R-type disfluency indicates the reparandum (words that the speaker intends to discard), an optional interregnum (words that have no meaning), and the associated repair. Interregnums are often fixed phrases, e.g. \"uh\", \"you know\". \n\n    Reparandum contains those words which are originally not intended to be in the utterance. Thus this section consists of one or more words that will be repeated or corrected. Interregnum is a non-lexicalized part like filter word, including \"uh\", \"I mean\" and so on. The last part is repair. Words from the reparandum are finally corrected or repeated. \n\n    e.g. 让我们，我的意思是，让我来解决这个问题。(Let us, I mean, let me work on the problem.)\n\n    In this sentence, “Let us” is the reparandum, and “let me” is the repair, which corrects the reparandum. \"I mean\" have no meaning so it is the interregnum \n\n    ② ASR Transcription Errors\n\n    ASR Transcription Errors are the occasional homophone mistakes in ASR transcriptions. CS2W is built upon the existing MagicData-RAMC dataset, which consists of 351 sets of spontaneous conversations speech in Chinese Mandarin and their ASR transcriptions. Thus there are some occasional homephone mistakes in some sentences.\n\n    e.g. 这个艺术家很有菜花。(The artist is very cauliflower.) \n\n     (footprint: In Chinese, \"cauliflower\" and \"talent\" have the same pronunciation.)\n\n    According to the meaning of the sentence, the correct sentence should be \"The artist is very talented\". So \"cauliflower\" needs to be annotated as “ASR Transcription Error”.\n\n    ③ Grammatical Errors\n\n    The transcription text in spoken language often includes grammatical errors because speakers in conversations often lack careful thinking. And common grammatical errors in spoken language include Missing Words, Redundant Words, and Incorrect Word Order.\n\n    **Missing Words:**\n\n    Missing Words include missing subject, missing predicate, missing object, missing function word, and missing modifier.\n\n    e.g. 那时我们有机会扳平比分，但是我们没有机会。(We had a chance to equalize, but we didn't it.)\n\n    This sentence is missing a verb. The correct sentence is \"We had a chance to equalize, but we didn't take it\".\n\n    **Redundant Words:**\n\n    Redundant Words include redundant subject, redundant predicate, redundant object, redundant function word, and redundant modifier. \n\n    e.g. 它们的皮毛很有光泽,可以用肉眼很难看出来。(Their fur is shiny and can be hardly seen with the naked eyes.)\n\n    The modifier in this sentence is redundant. The fur is shiny so it should be visible to the naked eyes. The word \"hardly\" should be deleted. \n\n    **Incorrect Word Order:**\n\n    Incorrect Word Order is also common in spoken transcribed texts because of the frequent inversions in spoken language.\n\n    e.g. 昨天我看了新买的一部电影在电视上。(Yesterday I watched a newly purchased movie on TV.)\n\n    In Chinese, the correct sentence is \"Yesterday on TV I watched a newly purchased movie\".\n \n    ④ Colloquial Words\n\n    Spoken language often contains informal expressions, such as some popular Internet phrases, which is called “Colloquial Words”.\n\n    e.g. 这明明是你的功劳，却被同事抢走了，你真是一个大怨种。(This is obviously your credit, but your coworkers took it away. You're such an unlucky guy.)\n\n    (In Chinese, \"大怨种\" is a popular Internet phrase which is used to describe people who are aggrieved but have no way to complain.)\n\n    We need to replace all informal expressions with formal ones. The written language is \"This is obviously your credit, but your coworkers took it away. You're such an unlucky guy\".\n\n    ⑤ Mixed Type\n\n    In real spontaneous conversations, a single sentence often contains multiple conversions. \n\n    e.g. 在国内成立野牛，这个，水牛研究中心，有利于帮助适应人工环境。(The establishment of the bison, I mean, buffalo research center in the country will help to adapt in an artificial environment.)\n\n    This sentence contains both disfluency and grammatical errors. The written language should be “The establishment of the buffalo research center in the country will help them to adapt in an artificial environment.”.\n\n    (3) Annotation rules\n\n    We built an annotation platform to accelerate our annotation progress. When using it, the annotator needs to select the conversion type and annotation range of the current sentence. Then, the annotator provides the written language corresponding to this spoken language. Next we will present the annotation rules for different conversions. Please note that on the annotation platform, each sentence is word-segmented into individual words, each of which can be selected to make it easier for the annotator to annotate the conversion range.\n\n    ① Disfluency\n\n    The annotator selects the disfluency button on the annotation platform. For Filter Word, the annotator should annotate its range. For R-type, the reparandum and the interregnum should be annotated but the repair does not need to be annotated. Because the sentence can be corrected with the reparandum and the interregnum being deleted.\n\n    ② ASR Transcription Errors\n\n    The annotator selects the ASR transcription errors button on the annotation platform and annotates the range of the ASR Transcription Errors.  \n\n    ③ Grammatical Errors\n\n    The annotator selects the grammatical errors button on the annotation platform. For Missing Words, the annotator needs to annotate the two words before and after the missing part. For Redundant Words, the annotator needs to annotate the redundant part. For Incorrect Word Order, the whole sentence has to be annotated. \n\n    ④ Colloquial Words\n\n    The annotator selects the colloquial words button on the annotation platform and annotates the range of the Colloquial Words.\n\n    ⑤ Mixed Type\n\n    First, the annotator selects the button for the first conversion type, and then annotates the range of this conversion. Then the annotator selects the buttons of the other conversion types in turn and annotate the range of the conversion.\n\n    (4) Disagreement Resolution\n\n    To guarantee annotation consistency and quality, we introduce a two-round annotation process.\n\n    In the first round of annotation, we recruit eight Chinese native speakers as part-time annotators to start annotation after pre-annotation training.\n\n    In the second round, we perform manual evaluation and re-annotation with other two annotators, and one of the authors of this paper serves as a senior annotator.\n\n    When an inter-annotator disagreement arises, the annotator in the second round reannotates the sentence and submits the results of two rounds to the senior annotator.\n\n    There are two cases of inter-annotator disagreements. \n\n    The first case is that the annotator from the first or second round makes an incorrect annotation.\n\n    e.g. 当他回到车车间时，已经有了明显的变化。(When he returned to the ga-, garage, had changed markedly.)\n\n    The sentence has two conversion problems including Disfluency and Missing Words. The annotator in the first round only annotates disfluency correctly but he/she does not catch the grammatical errors. The annotator in the second round annotates two conversion problems. Then the senior annotator will choose the correct one.\n\n    The second case is that the sentence can be corrected in multiple ways, which is common in grammatical errors.\n\n    e.g.  \n\n    Source sentence: 如果人们连续看上四五个小时的电视节目，就会感到十分疲劳。(If people watch TV programs for four or five hours in a row, will feel very tired.)\n\n    Target sentence 1: 人们如果连续看上四五个小时的电视节目，就会感到十分疲劳。(People in case of watching TV programs for four or five hours in a row will feel very tired.)\n\n    Target sentence 2: 如果人们连续看上四五个小时的电视节目，他们就会感到十分疲劳。(If people watch TV programs for four or five hours in a row, they will feel very tired.)\n\n    This sentence lacks a subject and has two solutions. First, consider the conversion as Incorrect Word Order, the written language is target sentence 1. Second, consider the conversion as Missing Word, the written language is target sentence 2. In this case, the senior annotator chooses the solution with a smaller edit distance. If the edit distances are the same, the senior annotator chooses the first-round solution.\n\n**R2:** Thank you for this comment. We indeed demonstrate the value of the dataset created. We use CS2W to fine-tune two models, BART, CPT, and the results show that the fine-tuned CPT-large achieves the best results on all metrics and substantially outperforms the advanced LLM under zero-shot and few-shot (5-shot) settings, which indicates that even in the era of LLMs, CS2W still plays an important role in the Chinese Spoken-to-Written Style Conversion task. \n\n**A2:** Thank you for this comment. We indeed demonstrate the value of the dataset created. We use CS2W to fine-tune two models, BART, CPT, and the results show that the fine-tuned CPT-large achieves the best results on all metrics and substantially outperforms the advanced LLM under zero-shot and few-shot (5-shot) settings, which indicates that even in the era of LLMs, CS2W still plays an important role in the Chinese Spoken-to-Written Style Conversion task. \n\nRegarding the prompts used for zero- and few-shot setting, we tried two different prompts on BLOOM-7B under the zero-shot and 5-shot setting.\n\nPrompt1: \"下面有一个口语到书面语风格转换任务, 请把口语修改为书面语: \\\\n\\\\n口语：\\\\n {源句} \\\\n书面语: \\\\n\"\n(Here's a spoken- to written-language style conversion task, please rewrite the spoken language into the written language: \\\\n\\\\n Spoken: {source sentence} \\\\n Written: \\\\n).\n\nPrompt2: \"下面有一个语法纠错任务, 请把错误的文本修改为正确的文本: \\\\n\\\\n错误文本: \\\\n{源句} \\\\n 正确文本: \\\\n\"\n(Here's a grammatical error correction task, please correct the wrong text into the right text: \\\\n\\\\n wrong text: {source sentence} \\\\n correct text: \\\\n)\n\nPrompt2 outperforms Prompt1 on all metrics. We speculate that the model is not capable of understanding the definition of the spoken-to-written language conversion task. However, grammatical error correction is a widely-used task and the four conversion problems, except for colloquial words, can be regarded as either simple or complex grammatical errors, thus making it work well. Hence we adopt prompt2 for the rest of our experiments.\n\nUnder the zero-shot setting, the prompt is “下面有一个语法纠错任务，请把错误的文本修改为正确的文本：\\\\n\\\\n错误文本：\\\\n {源句} \\\\n正确文本：\\\\n” (Here's a grammatical error correction task, please correct the wrong text into the right text: \\\\n\\\\n wrong text: {source sentence} \\\\n\\\\n correct text:).\n\nWe expect LLMs to be able to correct all types of conversion problems. Therefore, we need to ensure the diversity of demonstrations given to LLMs. Considering that there are four types of conversion problems in CS2W, we add demonstrations of different types in turn as we have more demonstrations. The demonstrations of each conversion problem are randomly selected from the validation set, which are different from the input sentence in the test set and contain only one conversion problem. In addition, CS2W is dominated by disfluency (87.4%) as the primary conversion problem, so we select two demonstrations which are R-type and Filter word respectively under the 5-shot setting. In section 5.4, the prompt used in 5-shot experiments is “下面有一个语法纠错任务，请把错误的文本修改为正确的文本：\\\\n\\\\n错误文本：我不不太喜欢听那种歌曲；\\\\n正确文本：我不太喜欢听那种歌曲。\\\\n\\\\n错误文本：你呃喝酒喝的比较多了；\\\\n正确文本：你喝酒喝的比较多了。\\\\n\\\\n错误文本：我爸会糖醋丸子；\\\\n正确文本：我爸会炸糖醋丸子。\\\\n\\\\n错误文本：我想给游戏氪金；\\\\n正确文本：我想给游戏充钱。\\\\n\\\\n错误文本：这件事情我无可奉报；\\\\n正确文本：这件事情我无可奉告。\\\\n\\\\n错误文本：{源句}\\\\n正确文本：\\\\n” (Here's a grammatical error correction task, please correct the wrong text into the right text: \\\\n\\\\n wrong text: I don’t don’t like this type of music;\\\\n correct text: I don’t like this type of music. \\\\n\\\\n wrong text: You, uh, have been drinking a lot;\\\\n correct text: You have been drinking a lot.\\\\n\\\\n wrong text: My dad can sweet and sour dumplings;\\\\n correct text: My dad can make sweet and sour dumplings.\\\\n\\\\n wrong text:  I'd like to load up the game;\\\\n correct text: I'd like to top up the game.\\\\n\\\\n wrong text: I have nothing to report in this matter;\\\\n correct text: I have nothing to say in this matter. \\\\n\\\\n wrong text: {source sentence}\\\\n\\\\n correct text: ).\n\nIn section 5.7, the prompt used for zero-shot to 5-shot is the same as the prompt above.    \n\n**Question for authors:**\n\n**Q1:** A new sub-type of disfluency is described in section 4.4. Please quantify how common this type of disfluency is.\n\n**A1:** In CS2W, there are 17 sentences containing this new sub-type of disfluency. Although the number of sentences with this new type is small, it has been demonstrated that LLMs can learn to follow specific response formats from only a handful of examples in the training data without any reinforcement learning or human preference modeling [5]. Therefore, LLMs may be able to learn the feature of the new sub-type of disfluency when CS2W is used in SFT. Moreover, under the few-shot setting, the number of instances with this sub-type of disfluency is also enough to prompt LLMs. When CS2W is used for evaluation, the number is sufficient too.\n\n**Q2:** Do I understand correctly that you do not use the originally created training, development and test sets (described on lines 323f) in your experiments (because you use a different split described on lines 400f )? If so. what is the benefit of the original splits?\n\n**A2:** The experimental section (400f) shows the initial data division strategy. However, given that the dataset is relatively small, such a division according to conversion types does not make sense. Thus, the conventional randomized division is used (323f). We apologize for the confusion caused and we will correct this in the next version.\n\n**Q3:** In Figure 4, the English translation of the example is confusing: In the first line, you translate 不单 as \"is not only and 不单独 as \"is not\". However, in the following lines, both versions are translated as \"is not only\". Can you explain this inconsistency? Also, you do not translate 吃. How about, e.g, \"Bamboo is not only a food for pandas, but it can also be made into a lot of furniture.\"?\n\n**A3:** Many thanks for this. In Figure 4, \"不单\" should be translated as \"is not only\" while \"不单独\" should be translated as \"is not individually\". And your translation is better than ours. We’ll correct these in the new version. \n\n**Q4:** Could you please show/include the prompts you used for zero-shot and few-shot inference?\n\n**A4:** Please refer to our answer to the “Reason to reject”.\n\n**Q5:** Are there similar datasets in other languages than EN, CH, JA and VI, or did you focus on purpose on this selection of languages?\n\n**A5:** Thank you for your question. \n\nPublicly available spoken-to-written conversion datasets are scarce. And as of the time we completed this work, the only related dataset is Japanese S2W [1], which is mentioned in the section of related work (Line 219). Similar to ours, \nS2W is extracted from transcribed texts of spontaneous conversations. Chinese and Japanese have different linguistic conventions, thus making the type of conversions different. Japanese S2W contains conversion types: notation correction, postpositional particle expressions restoration, disfluency, and simplification. After the submission of our paper, a parallel corpus of formal-informal sentences in Persian [2] has been posted to arXiv on Aug. 10, 2023. In this corpus, informal sentences are spoken language or in a written style but used in informal conversations and social media. The data for this corpus comes from social networking sites, books, and movie scripts, while our data source is transcribed text from spontaneous conversations. It is demonstrated that disfluencies are more prevalent in spontaneous speech [3]. However, in the Informal-Formal Persian Corpus, the authors define only two types of transformations, lexical and syntactic, without focusing on disfluencies. Besides the above two datasets, there is no other spoken-to-written conversion dataset.\n\nGiven that disfluency and grammatical errors dominate CS2W as the primary conversion problem, we introduce datasets for disfluency detection and grammatical error correction, respectively, in the section of related work. In the field of disfluency detection, Switchboard and PhoDisfluency are the only publicly available datasets. In addition, there are two in-house Chinese datasets [6][7], which are not publicly available. In the field of Grammatical Error Correction (GEC), there is a significant increase in open-source datasets. In addition to the Chinese and English datasets mentioned in Table 2, there are also Japanese and Spanish GEC datasets [8][9]. The GEC datasets we list in Table 2 are the more popular ones.\n\nIn conclusion, for spoken-to-written style conversion and disfluency, we list all open-source datasets and the language was not chosen intentionally. For GEC, we list several most popular datasets. \n\n**Q6:** On line 198, you say that CS2W offers two annotation paradigms. Could you clarify what you mean here?\n\n**A6:** Thank you for your question. Because the dataset for the spoken-to-written style conversion task is relatively small, we refer to the annotation paradigm of many grammar error correction datasets and disfluency detection datasets for our annotation [10]. \n\nThere are mainly two types of annotation paradigms for constructing GEC data, i.e., error-coding and direct rewriting. \n\nThe error-coding paradigm requires annotators to explicitly mark the erroneous span in the original sentence, then choose its error type, and finally make corrections. However, it is demonstrated that annotators would pay less attention to the fluency of the resulting reference when there are too many categories to consider.\n\nInstead, the direct rewriting paradigm requires annotators to directly rewrite the whole sentence, as long as the resulting sentence does not change the original meaning and is grammatically correct and fluent.\n\nIn CS2W, we not only annotate the type, range of the conversion (the error-coding paradigm), but also manually rewrite the corresponding written language (the direct rewriting paradigm). That is, compared to other datasets that adopt one annotation paradigm in Table 2, CS2W adopts two annotation paradigms. We’ll make this clearer in the new version.\n\n**Q7:** Figure 3 b): Unfortunately, I don't understand this figure. Is it the number of spans per samples? (If so, I think you can remove the colors and the labels, as this seems to be redundant.)\n\n**A7:** Thank you for bringing this issue to our attention. During the annotation process, we have noticed that a sentence often has multiple conversion problems. \n\ne.g. \n\nSource sentence: 他的儿子呀是上一届奥运会的冠军嘛，并且当年世界锦标赛夺得金牌。(His son, you know, was the last Olympic champion, uh, and won the gold medal the World Championships that year.)\n\nTarget sentence: 他的儿子是上一届奥运会的冠军，并且在当年世界锦标赛夺得金牌。(His son was the last Olympic champion and won the gold medal at the World Championships that year.)\n\nThere are two types of conversion problems in the source sentence: Disfluency and Missing word. And there are two disfluencies including “you know” and “uh”. So, the number of conversion problem types for this sentence is 2 and the number of conversion problems is 3. And Figure 3 b)counts the number of conversion problems on a single sentence. Additionally, the colors and labels are indeed redundant, which we will correct in the next version.\n\n**Typos Grammar Style And Presentation Improvements:**\n\n**I1:** In Table 3, \"The milk is a little bit very hot.\" Isn't this an example of an R-type disfluency instead of a redundant word? And The phrase \"Incorrect Word Order\" is outside the cell.\n\n**A1:** Many thanks for this. \"The milk is a little bit very hot.\" is an example of an R-type Disfluency and we will change it with an example of redundant words in the next version.\n\n**I2:** Figure 1: In the caption, you have the abbreviation FP=Filled Pause. However, this does not appear in the example. Or should FW be FP?\n\n**A2:** Many thanks for this. In the caption of Figure 3, the abbreviation “FP=Filled” should be “FW=Filter Word”. We will change it in the next version.\n\n**I3:** Can you give an example of collocation errors (155)?\n\n**A3:** Collocation errors include subject-predicate collocation errors, predicate-object collocation errors, subject-object collocation errors, and modifier collocation errors.\n\ne.g. \n\nSource Sentence: 女足的队员就是一个球，把球踢好就是她们的任务。(The women's soccer player is a ball, and it's their job to play it well.)\n\nTarget Sentence: 女足的队员就是一个整体，把球踢好就是她们的任务。(The women's soccer players are a team, and it's their job to play football well.)\n\nThis is an example of a subject-object collocation error. The player cannot be a ball. So it would be more appropriate to say “the players are a team”.\n\n**I4:** What do you mean by \"doesn't require annotators\" (178)?\n\n**A4:** Thank you for this feedback. JFLEG adopts the rewriting paradigm while FCE and AESW adopt the error coding paradigm. The error coding paradigm requires annotation under the minimal edit distance principle [11], which always selects a reference with fewer edits when correcting errors. In contrast, the rewriting paradigm only requires that the reference be consistent with the meaning of the source sentence. Therefore, annotators are not required to perform minimal edit distance principle. We are sorry for this confusion. We’ll make it more clear in the next version.\n\n**I5:** This sentence is incorrectly constructed and confusing, please check it (277f).\n\n**A5:** The correct expression should be \"CS2W is a spoken-written style conversion dataset, and the conversion problems are categorized according to Chinese linguistic characteristics. \" We will correct it in the next version.\n\n**I6:** This sentence is unclear to me (311f).\n\n**A6:** In the second round of annotations, The first question (296) is to check the accuracy of the first round of annotations under the error coding paradigm (we refer to it as the annotation accuracy). The second question (298) is to check the accuracy of the first round of annotations under the rewriting paradigm (we call it as the meaning-preserving accuracy). The third question (301) is to check the consistency of the first round of annotations for both paradigms. The direct rewriting annotation requires annotators to rewrite the whole sentence, which is more flexible than the error coding paradigm. When two rounds of annotations diverge under the error coding paradigm, there is also a high probability that the results of the rewritten sentences will differ even more. So the meaning-preserving accuracy is further affected when the annotation accuracy is low. We are sorry that the expression making you feel confused, we will make this clearer in the next version. \n\n**I7:** \"with disfluency/grammatical errors …\" (366f): This sentence is unclear to me, isn't it a repetition of the main sentence (\"Among the mixed types …\")?\n\n**A7:** Thank you for this question. The meaning of the sentence (366f) is that \"disfluencies are so common that many sentences with grammatical errors or colloquial words also contain disfluencies.\". We are sorry that the expression making you feel confused, we will make it clearer in the next version.\n\n**I8:** Please explain what the P/R/F scores are (452).\n\n**A8:** Thank you for this question. P is precision, R is recall and F is F0.5 score, which are calculated with the MaxMatch method widely used in GEC [12]. Firstly, the gold sentence and the model’s output are word-segmented using PKUNLP word segmentation (WS) tool [13], and then we calculate the number of maximal matches according to the WS results. The P, R, F0.5 measure the different rates between the set of the model’s output edits {e1,...,e2} and the set of gold sentence edits {g1,...,g2} for all sentences:\n\n$$\nP = \\frac{\\sum_{i=1}^n|e_i\\cap g_i|}{\\sum_{i=1}^n|e_i|}\n$$\n\n$$\nR = \\frac{\\sum_{i=1}^n|e_i\\cap g_i|}{\\sum_{i=1}^n|g_i|}\n$$\n\n$$\nF_{0.5} = \\frac{1.25\\*P\\*R}{0.25\\*P+R}\n$$\n\nWhere we define the intersection between $e_i$ and $g_i$ as:\n\n$$\ne_i \\cap p_i = \\lbrace e \\in e_i |\\exists g \\in g_i(match(e,g)) \\rbrace\n$$\n\nOther typos, grammatical and presentation issues will be fixed in the next version following your suggestion. Thank you very much.\n\n[1] Mana Ihori, Akihiko Takashima, and Ryo Masumura. 2020. Parallel corpus for japanese spoken-to-written style conversion. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 6346–6353. European Language Resources Association.\n\n[2] Vahide Tajalli, Fateme Kalantari, and Mehrnoush Shamsfard. 2023. Developing an informal-formal persian corpus. CoRR, abs/2308.05336\n\n[3] Elizabeth Ellen Shriberg. 1994. Preliminaries to a theory of speech disfluencies. Ph.D. thesis, Citeseer.\n\n[4] Rohit Kundu, Preethi Jyothi, and Pushpak Bhattacharyya. 2022. Survey: Exploring disfluencies for speech-to-speech machine translation. \n\n[5] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: less is more for alignment. CoRR, abs/2305.11206.\n\n[6] Shaolei Wang, Wanxiang Che, and Ting Liu. 2016. A neural attention model for disfluency detection. In COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan, pages 278–287. ACL\n\n[7] Qianqian Dong, Feng Wang, Zhen Yang, Wei Chen, Shuang Xu, and Bo Xu. 2019. Adapting translation models for transcript disfluency detection. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6351–6358. AAAI Press. \n\n[8] Daisuke Suzuki, Yujin Takahashi, Ikumi Yamashita, Taichi Aida, Tosho Hirasawa, Michitaka Nakatsuji, Masato Mita, and Mamoru Komachi. 2022. Construction of a quality estimation dataset for automatic evaluation of japanese grammatical error correction. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022, pages 5565–5572. European Language Resources Association.\n\n[9] Antonios Anastasopoulos, Alison Lui, Toan Q. Nguyen, and David Chiang. 2019. Neural machine translation of text from non-native speakers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 3070–3080. Association for Computational Linguistics.\n\n[10] Yue Zhang, Zhenghua Li, Zuyi Bao, Jiacheng Li, Bo Zhang, Chen Li, Fei Huang, and Min Zhang. 2022. Mucgec: a multi-reference multi-source evaluation dataset for chinese grammatical error correction. In Proceedings of the 2022 Conference of the North249\nAmerican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 3118–3130. Association for Computational Linguistics.\n\n[11] Ryo Nagata and Keisuke Sakaguchi. 2016. Phrase structure annotation and parsing for learner english. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.\n\n[12] Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 3-8, 2012, Montréal, Canada, pages 568–572. The Association for Computational Linguistics. \n\n[13] Ruixuan Luo, Jingjing Xu, Yi Zhang, Xuancheng Ren, and Xu Sun. 2019. PKUSEG: A toolkit for multi-domain chinese word segmentation. CoRR, abs/1906.11455.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors create and publish a dataset for a task for which few public datasets exist to date. Unlike other similar datasets, they annotate multiple style conversions in parallel (disfluencies, ASR transcription errors, grammatical errors, and spoken-style vocabulary), which makes it suitable for real-world scenarios. Ideally, their corpus may serve as a blueprint for similar datasets for languages other than Chinese.\n\nTheir experiments show that LMs finetuned on this dataset can outperform zero-shot and few-shot models on several metrics, demonstrating the usefulness of such a manually curated dataset even in the era of pretrained LMs.", "reasons_to_reject": "The paper does not describe the creation of the corpus in full detail. For example, it does not explain how the samples were selected from the larger MagicData RAMC of Yang et al. (2022), and it does not describe the annotation guidelines in detail. While it is mentioned that there were inter-annotator disagreements, no examples of these are given, nor is it explained how the disagreements were resolved.\n\nIn addition, most experiments use zero-shot or few-shot settings, disregarding the dataset created. Although the performance of models finetuned on the dataset is indeed better, this point is not sufficiently emphasized or discussed. Again, some details are missing, e.g., which prompts were used for zero-shot and few-shot experiments.", "questions_to_authors": "1) In section 4.4, you describe a new sub-type of disfluency. Can you quantify how common this type of disfluency is?\n\n2) Do I understand correctly that you do not use the originally created training, development and test sets (described on lines 323f.) in your experiments (because you use a different split described on lines 400f.)? If so, what is the benefit of the original splits?\n\n3) In Figure 4, the English translation of the example is confusing: In the first line, you translate 不单 as \"is not only\" and 不单独 as \"is not\". However, in the following lines, both versions are translated as \"is not only\". Can you explain this inconsistency? Also, you do not translate 吃. How about, e.g., \"Bamboo is not only a food for pandas, but it can also be made into a lot of furniture.\"?\n\n4) Could you please show/include the prompts you used for zero-shot and few-shot inference?\n\n5) Are there similar datasets in other languages than EN, CH, JA and VI, or did you focus on purpose on this selection of languages?\n\n6) On line 198, you say that CS2W offers two annotation paradigms. Could you clarify what you mean here?\n\n7) Figure 3 b): Unfortunately, I don't understand this figure. Is it the number of spans per samples? (If so, I think you can remove the colors and the labels, as this seems to be redundant.)", "paper_topic_and_main_contributions": "The paper addresses the conversion of spoken-style language to written-style language as a postprocessing step for automatic speech recognition (ASR). The main contributions are the creation and publication of a manually annotated dataset for Chinese and benchmark evaluations with large language models (LMs) on this dataset.\n\nThe dataset is a manually collected subset of the Real Spontaneous Dialogue Speech dataset MagicData-RAMC by Yang et al. (2022) and consists of 7237 samples containing relevant spoken-to-written-style issues. Each sample is annoted with the types and spans of issues, together with a manually normalized written-style version. Four different types of issues are annotated: disfluencies (repetitions, restarts, and repairs), ASR transcription errors, grammatical errors, and spoken-style vocabulary. According to the authors, it is the first dataset with annotation of multiple error types for Chinese.\n\nIn addition, the paper analyzes six different LMs to benchmark their performance on the Chinese spoken-to-written conversion task: BART, CPT, BLOOM, BELLE, ChatGLM and GPT3.5. Only the first two are actually trained on the curated dataset. For the others, both zero-shot and few-shot (5-shot) settings are tested. The result shows that CPT-large, trained on the curated dataset, has the best performance across various metrics.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "For Baoli (2011), Shriberg (1996) and Zayats et al. (2016), may you add where the papers were published (e.g. arxiv)?"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "For all Chinese texts, I'd recommend to also add Pinyin (including diacritics) for accessibility.\n\nTable 2:\n- NLPCC18-task: \"Rewriting\"> \"Direct Rewriting\" (for consistenc with other lines, unless there is a difference in which case please clarify)\n- PhoDisfluency: \"JA\" > \"VI\" (Vietnamese)\n- \"HSK\" > \"HSK Exam\" (like \"TOEFL Exam\")\n\nTable 3:\n- \"The milk is a little bit very hot.\" Isn't this an example of an R-type disfluency instead of a redundant word?\n- \"Incorrect Word / Order\": The word \"Order\" is in the next cell, which is confusing\n\nTable 4:\n- \"w/ 2 error\" > \"w/ 2 errors\" (and accordingly for 3 and 4 errors)\n\nTable 5:\n- \"Bloom-7b\" > \"BLOOM-7B\"\n- The table would be clearer if you would arrange it into three parts: 1) fine-tuning, 2) zero-shot, 3) 5-shot\n\nTable 6: \"GPT-large\" > \"CPT-large\"\n\nTable 7: \"BLEU.2\" > \"BLEU-2\"\n\nFigure 1: In the caption, you have the abbreviation FP=Filled Pause. However, this does not appear in the example. Or should FW be FP?\n\nFigure 3:\n- \"conversion\" > \"Conversion\"\n- maybe change order of categories, e.g. from small to large (in this case, switch the first two, and switch GE and CW further down)\n\n155: Can you give an example of collocation errors?\n\n156: remove \"which\"\n\n178: What do you mean by \"doesn't require annotators\"?\n\n189: remove \"Pinyin of\"\n\n247: \"&\" > \"and\"\n\n267: \"by Sakaguchi et al. (2016)\" (i.e., change citation type)\n\n277f: This sentence is incorrectly constructed and confusing, please check it\n\n311f: This sentence is unclear to me.\n\n347: remove \"much\"\n\n355: \"exhibit\" > \"our dataset exhibits\"\n\n361: \"prolems\" > \"problems\"\n\n366f: \"with disfluency/grammatical errors …\": This sentence is unclear to me, isn't it a repetition of the main sentence (\"Among the mixed types …\")?\n\n383: \"Figure\" > \"Figure 4\"\n\n424: \"We used cpt-base\" > \"We used cpt-base and cpt-large\"\n\n426f: add reference for the ROOTS corpus\n\n445: \"We test fine-tuning with […] and zero-shot and 5-shot with BLOOM, BELLE, ChatGLM, and GPT3.5-turbo.\n\n452: Please explain what the P/R/F scores are."}}, {"review_id": "sqrgIehjUS", "reviewer": "Reviewer_i9a8", "summary": "", "questions": "Question A: Regarding ASR transcription errors, you have chosen a single source of ASR transcriptions. What confidence should we have that this ASR transcriber produces errors in a similar manner to other ASR systems? \n\nQuestion B: For the zero to five demonstrations given to the LLMs, were they selected at random? Did they contain the same type of error as you expected the model to find? When you have more demonstrations, did you ensure diversity in the examples given to the LLM?\n\nQuestion C: Did the annotators give informed consent for the use of their contributions in training AI/NLP systems?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments and suggestions. We are very encouraged by your positive comments on our work.\n\n**Q1**: Regarding ASR transcription errors, you have chosen a single source of ASR transcriptions, What confidence should we have that this ASR transcriber produces errors in a similar manner to other ASR systems?\n\n**A1**: Thank you for bringing this issue to our attention. We have double-checked the MagicData-RAMC dataset and the description of data construction, and found no information about the ASR system used for transcription of this dataset.\n\n**Q2**: For the zero to five demonstrations given to the LLMs, were they selected at random? Did they contain the same type of error as you expeted the model to find? When you have more demonstrations, did you ensure diversity in the examples given to the LLM? \n\n**A2**: Thank you for your questions. \n\n(1) Yes, the zero to five demonstrations given to the LLMs were chosen at random. There are four types of conversion problems in CS2W. As we expected the LLMs to be able to correct all types of conversion problems and the conversion problems presented in the demonstration may enhance the understanding of LLMs towards the corresponding conversion problems, we enforced diversity in the demonstrations given to LLMs during the demonstration selection process. Considering that there are four types of conversion problems in CS2W, we continued to add demonstrations of different types until there was no room to add more demonstrations. During the demonstration selection process, the demonstrations of each conversion problem were randomly selected from the validation set, which were different from the input sentence in the test set and contained only one conversion problem. In addition, CS2W is dominated by the disfluency (87.4%) conversion problem, so we selected two demonstrations which were R-type and Filter word respectively under the 5-shot setting. In sections 5.4 and 5.7, the prompt used in 5-shot experiments is “下面有一个语法纠错任务，请把错误的文本修改为正确的文本：\\\\n\\\\n错误文本：我不不太喜欢听那种歌曲；\\\\n正确文本：我不太喜欢听那种歌曲。\\\\n\\\\n错误文本：你呃喝酒喝的比较多了；\\\\n正确文本：你喝酒喝的比较多了。\\\\n\\\\n错误文本：我爸会糖醋丸子；\\\\n正确文本：我爸会炸糖醋丸子。\\\\n\\\\n错误文本：我想给游戏氪金；\\\\n正确文本：我想给游戏充钱。\\\\n\\\\n错误文本：这件事情我无可奉报；\\\\n正确文本：这件事情我无可奉告。\\\\n\\\\n错误文本：{源句}\\\\n正确文本：\\\\n” (Here's a grammatical error correction task, please correct the wrong text into the right text: \\\\n\\\\n wrong text: I don’t don’t like this type of music;\\\\n correct text: I don’t like this type of music. \\\\n\\\\n wrong text: You, uh, have been drinking a lot;\\\\n correct text: You have been drinking a lot.\\\\n\\\\n wrong text: My dad can sweet and sour dumplings;\\\\n correct text: My dad can make sweet and sour dumplings.\\\\n\\\\n wrong text:  I'd like to load up the game;\\\\n correct text: I'd like to top up the game.\\\\n\\\\n wrong text: I have nothing to report in this matter;\\\\n correct text: I have nothing to say in this matter. \\\\n\\\\n wrong text: {source sentence}\\\\n\\\\n correct text: ).\n\n(2) Yes, when there were five demonstrations, they contained the same type of error as we expected the model to find. As we added demonstrations with disfluency conversion problems first and there were two demonstrations with disfluency conversion problems, if the number of demonstrations was less than five, we also attempted to enforce diversity in the demonstrations by continually adding demonstrations with different conversion problems until there was no room to add more demonstrations.\n\n(3) Yes, we indeed ensured diversity in the examples given to LLMs during the demonstration selection process. Please refer to the detailed demonstration selection process described above for more information. However, although we had an intuition that having more types of conversions in demonstrations would enable LLMs to resolve more types of conversions and improve results, we found that this might not hold true under all conditions. We will provide a detailed illustration of the experiment below:\n\nFor convenience, the results in section 5.4 is shown in the table below.\n\n| Model\\BLEU | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 |\n| :------: | :---: | :---: | :---: | :---: |\n|  BELLE-7B-2M   |   0.4885    |   0.3519    |    0.279   |    0.2167   |\n| ChatGLM-6B  |    0.5924   |    0.4872   |    0.4261   |   0.3785    |\n|  GPT3.5-turbo  |   0.6882    |    0.5862   |    0.5087   |    0.4393   |\n\nWe randomly selected three groups of five demonstrations from the validation set:\n\nThe first group:\n\ndemonstration1:(colloquial word)\n\nwrong text: 我为什么要套路女孩子; correct text: 我为什么要哄骗女孩子。\n\nwrong text: Why I'm trying to trap a girl. correct text: Why I'm trying to cajole a girl.\n\ndemonstration 2: (disfluency)\n\nwrong text: 他他让我假装睡觉; correct text: 他让我假装睡觉。\n\nwrong text: He he asked me to pretend to sleep; correct text: He asked me to pretend to sleep.\n\ndemonstration 3: (disfluency)\n\nwrong text: 这个电视剧的男主男一号叫靳燃;correct text: 这个电视剧的男一号叫靳燃。\n\nwrong text: The male protagonist main male character of this drama is named Jin Yan; correct text: The main male character of this drama is named Jin Yan.\n\ndemonstration 4: (colloquial word)\n\nwrong text: 腿可以够到很多你胳膊够不到的地方;correct text: 腿可以碰到很多你胳膊碰不到的地方。\n\nwrong text: the leg can reach a lot of places your arm can't; correct text: the leg can touch a lot of places your arm can't.\n\ndemonstration 5: (disfluency, grammatical error)\n\nwrong text: 他消失了二十八年他就终于研究出了那个就是氢弹原子弹这些; correct text: 他消失了二十八年他就终于研究出了氢弹原子弹这些武器。\n\nwrong text: He disappeared for 28 years and he finally developed the, the, hydrogen bomb and the atomic bomb and so on; \ncorrect text: He disappeared for 28 years and he finally developed the hydrogen bomb and the atomic bomb.\n\nThe results of the demonstrations for the first group are presented in the table below:\n\n| Model\\BLEU | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 |\n| :------: | :---: | :---: | :---: | :---: |\n|  BELLE-7B-2M   |   0.4593  $\\downarrow$  |   0.3465  $\\downarrow$ |    0.2688 $\\downarrow$  |    0.2042  $\\downarrow$ |\n| ChatGLM-6B  |    0.66 $\\uparrow$   |    0.5445  $\\uparrow$  |    0.4596 $\\uparrow$  |   0.3847 $\\uparrow$   |\n|  GPT3.5-turbo  |   0.626  $\\downarrow$  |    0.5215  $\\downarrow$ |    0.4442 $\\downarrow$  |    0.3748 $\\downarrow$  |\n\nThe results indicate that the BLEU score of ChatGLM-6B increased slightly, while the BLEU scores of the other two LLMs decreased.\n\nThe second group with five demonstrations with disfluency and only one conversion type:\n\ndemonstration1:(disfluency)\n\nwrong text: 人家会做什么一大桌桌子的菜; correct text: 人家会做一大桌子的菜。\n\nwrong text: They cook a whole table and a table of food; correct text: They cook a whole table of food.\n\ndemonstration 2: (disfluency)\n\nwrong text: 这就是职业带来的好一些好处; correct text: 这就是职业带来的一些好处。\n\nwrong text: These are some of the be- benefits that come with a career. correct text: These are some of the benefits that come with a career.\n\ndemonstration 3: (disfluency)\n\nwrong text: 你我们理性分析的话会发现他很聪明; correct text: 我们理性分析的话会发现他很聪明。\n\nwrong text: If you, we analyze him rationally, we will find that he is very clever. correct text: If we analyze rationally, we will find that he is very clever.\n\ndemonstration 4: (disfluency)\n\nwrong text: 你你就算吃饭了还是会饿; correct text: 你就算吃饭了还是会饿。\n\nwrong text: You you will still be hungry even if you eat; correct text: You will still be hungry even after eating.\n\ndemonstration 5: (disfluency)\n\nwrong text: 她她师傅是凭着真实力取得了冠军; correct text: 她师傅是凭着真实力取得了冠军。\n\nwrong text: She, her master won the championship by virtue of true strength; correct text: Her master won the championship with real strength.\n\nThe results of the demonstrations for the second group are presented in the table below:\n\n| Model\\BLEU | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 |\n| :------: | :---: | :---: | :---: | :---: |\n|  BELLE-7B-2M   |   0.4558  $\\downarrow$  |   0.3532 $\\uparrow$  |    0.279 $\\uparrow$  |    0.2166 $\\downarrow$  |\n| ChatGLM-6B  |    0.6932 $\\uparrow$ |   0.5977 $\\uparrow$  |    0.5225 $\\uparrow$  |    0.453 $\\uparrow$   |\n|  GPT3.5-turbo  |   0.6918   $\\uparrow$ |    0.5971  $\\uparrow$ |    0.5249  $\\uparrow$ |    0.4586  $\\uparrow$ |\n\nThe results indicate that the BLEU score of BELLE-7B-2M is consistent with the results presented in section 5.4, while the BLEU scores of the other two LLMs increased significantly.\n\nThe third group with four demonstrations with disfluency and two conversion type:\n\ndemonstration1: (disfluency)\n\nwrong text: 除工除了选择工作以外，还有很多人继续学习深造；correct text: 除了选择工作以外，还有很多人继续学习深造。\n\nwrong text: In addition, in addition to choosing a job, many people continue their studies; correct text: In addition to choosing jobs, many people continue their studies.\n\n\ndemonstration 2: (disfluency)\n\nwrong text: 你喜不喜欢就是旅游呀啥的；correct text: 你喜不喜欢旅游呀。\n\nwrong text: 你喜不喜欢就是旅游呀啥的; correct text: 你喜不喜欢旅游呀。\nwrong text: Do you like , uh, traveling or not? correct text: Do you like traveling?\n\ndemonstration 3: (grammatical error)\n\nwrong text: 张三丰这个人是有史料可卡找的; correct text: 张三丰这个人是有史料可找的。\n\nwrong text: Zhang Sanfeng has historical materials to trap and find; correct text: Zhang Sanfeng has historical materials to find;\n\ndemonstration 4: (disfluency)\n\nwrong text: 让我印象最深的一次一次兼职就是快递快递分拣；correct text: 让我印象最深的一次兼职就是快递分拣。\n\nwrong text: One one of the most impressive part-time job I've ever had was courier, courier sorting; correct text: One of the most impressive part-time job I've ever had was courier sorting.\n\ndemonstration 5: (disfluency)\n\nwrong text: 我习惯了习惯了早睡早起；correct text: 我习惯了早睡早起。\n\nwrong text: I'm used to getting used to going to bed early and getting up early; correct text: I'm used to going to bed early and getting up early.\n\n\nThe results of the demonstrations for the third group are presented in the table below:\n\n| Model\\BLEU | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 |\n| :------: | :---: | :---: | :---: | :---: |\n|  BELLE-7B-2M   |   0.4559 $\\downarrow$   |   0.3598  $\\uparrow$ |    0.291 $\\uparrow$  |    0.2312 $\\uparrow$  |\n| ChatGLM-6B  |    0.6842 $\\uparrow$ |   0.5822 $\\uparrow$  |    0.5043 $\\uparrow$  |    0.4324  $\\uparrow$  |\n|  GPT3.5-turbo  |   0.7331  $\\uparrow$  |    0.6131 $\\uparrow$  |    0.5333  $\\uparrow$ |    0.4644 $\\uparrow$  |\n\nThe results indicate that the BLEU scores of all LLMs increased.\n\nIn summary, the results of the experiment indicate that although we surmised that greater diversity in examples would enable LLMs to resolve more types of conversions and improve results, this was not always the case. In fact, the first group, which had the greatest diversity in examples, performed the worst. The second and third groups, which had less diversity but more demonstrations with disfluencies, performed better. We surmise that this is because disfluency dominates CS2W, and when the number of demonstrations with disfluencies exceeds a certain threshold, the results improve. If the four conversion types in the dataset were balanced, demonstrations with greater diversity may have performed better.\n\nWe will provide more details on the prompts and outputs of LLMs in the next version.\n\n**Q3**: Did the annotators give informed consent for the use of their contributions in training AI/NLP systems?\n\n**A3**: Yes, the annotators gave informed consent for the use of their contributions in training AI/NLP systems.\n\n**Typos Grammar Style And Presentation Improvements**:\n\n**I1**: Section beginning line 412 - please indicate the parameter count of the various models used, so their performance can be measured in comparison with their parameter efficiency.\n\n**A1**: Thank you for your feedback. The parameter count of the various models used is shown in the table below.\n\n|      Model      | BART-base | BART-large | CPT-base | CPT-large | BLOOM | BELLE | ChatGLM |\n| :-------------: | :-------: | :--------: | :------: | :-------: | :---: | :---: | :-----: |\n| **Parameter Count** |   139M    |    406M    |   121M   |   393M    |  7B   |  7B   |   6B    |\n\n**I2**: Line 465 - \"5 - shot\" in this context could either mean five examples were provided in the prompt or that five generations were used (assuming a temperature that is not fully deterministic). It may help to clarify that five examples were provided, and to specify the model temperature for generation.\n\n**A2**: Thank you for this comment. The “5-shot” in this context means five examples were provided in the prompt. \n\nRegarding the prompts used for zero- and few-shot setting, we tried two different prompts on BLOOM-7B under the zero-shot and 5-shot setting.\n\nPrompt1: “下面有一个口语到书面语风格转换任务，请把口语修改为书面语：\\\\n\\\\n口语：\\\\n {源句} \\\\n书面语：\\\\n”\n\n(Here's a spoken- to written-language style conversion task, please rewrite the spoken language into the written language: \\\\n\\\\n Spoken: {source sentence} \\\\n Written: \\\\n).\n\nPrompt2: \"下面有一个语法纠错任务，请把错误的文本修改为正确的文本：\\\\n\\\\n错误文本：\\\\n{源句} \\\\n 正确文本：\\\\n\"\n\n(Here's a grammatical error correction task, please correct the wrong text into the right text: \\\\n\\\\n wrong text: {source sentence} \\\\n correct text: \\\\n)\n\nPrompt2 outperforms Prompt1 on all metrics. We speculate that the model is not capable of understanding the definition of the spoken-to-written language conversion task. However, grammatical error correction is a widely-used task and the four conversion problems, except for colloquial words, can be regarded as either simple or complex grammatical errors, thus making it work well. Hence we adopt prompt2 for the rest of our experiments.\n\nUnder the zero-shot setting, the prompt is “下面有一个语法纠错任务，请把错误的文本修改为正确的文本：\\\\n\\\\n错误文本：\\\\n {源句} \\\\n正确文本：\\\\n” (Here's a grammatical error correction task, please correct the wrong text into the right text: \\\\n\\\\n wrong text: {source sentence} \\\\n\\\\n correct text:).\n\nThe five examples please refer to the response for Q2.\n\nThe model temperature for generation is shown in the table below.\n\n|    Model    | BLOOM-7B | BELLE-7B-0.2M | BELLE-7B-2M | ChatGLM-6B | GPT3.5-turbo |\n| :---------: | :------: | :-----------: | :---------: | :--------: | :----------: |\n| **Temperature** |   1.0    |     0.35      |     1.0     |    0.95    |     0.7      |\n\n**I3**: Table 7 - the description says BLEU 1, 2, and 3 the labels and text say 1, 2, and 4. Also, the header labels are inconsistent. This table is also presented out of order for the numbering.\n\n**A3**: Many thanks for this. The correct caption is “BLEU-1, BLEU-2, BLEU-4”. We will change it in the next version.\n\nOther typos, grammatical errors and presentation issues will be fixed all in the next version. Many thanks again.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper sets a standard for addressing multiple types of errors in spoken transcription through one model, with an adequate-sized corpus for future academic work.The paper appears to be free of major methodological errors, demonstrates the utility of multiple large language models against the task, and follows up by demonstrating a substantial improvement on a downstream task.", "reasons_to_reject": "There are some errors in visualization, some typos, and some missing citations, but none of these rise to the level of being reasons to reject this paper.\n\n", "questions_to_authors": "Question A: Regarding ASR transcription errors, you have chosen a single source of ASR transcriptions. What confidence should we have that this ASR transcriber produces errors in a similar manner to other ASR systems? \n\nQuestion B: For the zero to five demonstrations given to the LLMs, were they selected at random? Did they contain the same type of error as you expected the model to find? When you have more demonstrations, did you ensure diversity in the examples given to the LLM?\n\nQuestion C: Did the annotators give informed consent for the use of their contributions in training AI/NLP systems?", "paper_topic_and_main_contributions": "This paper describes the problem of taking transcribed Chinese text and performs a style transformation to remove disfluencies and grammatical errors, presenting an output that is more consistent with written style. To solve this problem they have prepared an open-source (non-commercial) corpus of annotated data based on an existing dataset, they have fine-tuned several existing models to establish baselines for this dataset, and have demonstrated the utility of the dataset in addressing a downstream machine translation task. ", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "line 329 - PKU seg (available at https://arxiv.org/abs/1906.11455)\n\nline 425 - BLOOM (cite: https://arxiv.org/abs/2211.05100)\n\nline 435 - ChatGLM (cite either: https://arxiv.org/abs/2103.10360 for the 6B model or https://arxiv.org/abs/2210.02414 for the 130B model)", "Typos_Grammar_Style_and_Presentation_Improvements": "line 030 - \"canonicalized\" can just be \"canonical\"\n\nline 039 - Please indicate \"Mandarin Chinese\" not just \"Chinese\" at the first mention. It is okay to use \"Chinese\" subsequently, where it will be understood to mean Mandarin, but it may be clearer to use Mandarin throughout.\n\nline 383 - \"Figure\" should say \"Figure 3\"\n\nline 383-385 - would read more clearly with \"said incorrectly\" in place of \"incorrectly\"\n\nsection beginning line 412 - please indicate the parameter count of the various models used, so their performance can be measured in comparison with their parameter efficiency\n\nFigure 2 - the graph on the left should start at zero, not one. The graph on the right has three bars labeled \"2\" that are of different lengths\n\nFigure 4 - the sentence at the bottom should say \"is not\" not \"is not only\" based on the transcript above. Also ... in this example, the typical correct English would be either \"... is not only for pandas, ...\" or \"... is not for pandas alone, ...\" The example might confuse English readers (even if it is correct for Chinese)\n\nline 465 - \"5 - shot\" in this context could either mean five examples were provided in the prompt or that five generations were used (assuming a temperature that is not fully deterministic). It may help to clarify that five examples were provided, and to specify the model temperature for generation.\n\nline 475 - you have been using the label \"colloquial words\" for this category, and you change to \"spoken words\" here. It should be consistent, and \"colloquial words\" is the better choice.\n\nTable 7 - the description says BLEU 1, 2, and 3 the labels and text say 1, 2, and 4. Also, the header labels are inconsistent. This table is also presented out of order for the numbering\n\nTable 6 - GPT should be CPT in the top left. Also, use \"Colloquial Words\" not \"Spoken words\" here\n"}}]

Author rebuttal:

Meta-review: This paper describes the problem of taking transcribed Chinese text and performs a style transformation to remove disfluencies and grammatical errors, presenting an output that is more consistent with written style. To solve this problem they have prepared an open-source (non-commercial) corpus of annotated data based on an existing dataset, they have fine-tuned several existing models to establish baselines for this dataset, and have demonstrated the utility of the dataset in addressing a downstream machine translation task.

Main contributions can be summarized as:

A new curated dataset CS2W, a Chinese Spoken-to-Written style conversion dataset comprising 7,237 spoken sentences extracted from transcribed conversational texts with four types of conversion problems.
Thorough analysis of the dataset, and benchmark evaluation of the on spoken-to-written language conversion with five SoTA baseline models. The analysis led to the discovery of a new type of disfluency.
Annotation guidelines and methodology behind the dataset preparation are shared with satisfactory detail.

Reasons To Accept:
- This paper sets a standard for addressing multiple types of errors in spoken transcription through one model, with an adequate-sized corpus for future academic work.The paper appears to be free of major methodological errors, demonstrates the utility of multiple large language models against the task, and follows up by demonstrating a substantial improvement on a downstream task.

Reasons To Reject:
- There are some errors in visualization, some typos, and some missing citations, but none of these rise to the level of being reasons to reject this paper.
Decision: Accept-Main