Dialect Transfer for Swiss German Speech Translation
Authors: Claudio Paonessa
Yanick Schraner
Jan Milan Deriu
Manuela Hürlimann
Manfred Vogel
Mark Cieliebak
Venue: EMNLP 2023 Findings (2023)
Keywords: speech to text
speech translation
swiss german
low resource

Abstract: This paper investigates the challenges in building Swiss German speech translation systems, specifically focusing on the impact of dialect diversity and differences between Swiss German and Standard German. Swiss German is a spoken language with no formal writing system, it comprises many diverse dialects and is a low-resource language with only around 5 million speakers. The study is guided by two key research questions: how does the inclusion and exclusion of dialects during the training of speech translation models for Swiss German impact the performance on specific dialects, and how do the differences between Swiss German and Standard German impact the performance of the systems? We show that dialect diversity and linguistic differences pose significant challenges to Swiss German speech translation, which is in line with linguistic hypotheses derived from empirical investigations.

does in- Dialect Transfer for Swiss German Speech Translation
Claudio Paonessa', Yanick Schraner', Jan Deriu', Manuela Hürlimann'.
Manfred Vogel', Mark Cieliebak2
University of Applied Sciences and Arts Northwestern Switzerland, Windisch Zuurich University of Applied Sciences, Winterthur
claudio.paonesse @ fhnw.ch, deri @ zhaw.ch, schick.schranerd fhnw.ch, hueu@zhaw.ch, manfred. vogel @ fhnw.ch, ciel @ zhaw.ch
Abstract
dialects differ significantly yields even fewer speak- ers for each dialect (e.g., Valais only has about 80K speakers).
This paper investigates the challenges in build- ing Swiss German speech translation systems, specifically focusing on the impact of dialect diversity and differences between Swiss Ger- man and Standard German. Swiss German is a spoken language with no formal writing sys- tem, comprises many diverse dialects and is a low- low-resource language with only around 5 million speakers. The study is guided by two key research questions:
These three challenges motivate the investigation of the following two questions:
How does the inclusion and exclusion of di- alects during the training of speech translation
(ST) models for Swiss German impact the per- formance on specific dialects?
Do the differences between Swiss German and Standard German negatively impact the performance of the ST systems?
clusion and exclusion of dialects during the training of speech translation models for Swiss German impact the performance on specific dialects, and how do the differences hetween Swiss German and Standard German impact the performance of the systems? We show that dialect diversity and linguistic differences pose sigmificant challenges to Swiss German speech translation, which is in line with linguistic hy- potheses derived from empirical investigations.
The first question investigates the issue of the diversity of dialects in low-resource setting, while the second question investigates the differences between Swiss German and Standard German.
Contributions. To answer these questions, we first review the Swiss German dialect landscape and the differences Standard German. In particular, we devise set of hypotheses from the literature stat- ing which dialects are expected to differ from each other and which Swiss German phenomena are ex- pected to impact the performance. We then investi- gate empirically whether the hypotheses match the results of training ST models in various settings.
1 Introduction
There are three main challenges when building Swiss German speech-to-ted systems. First, Swiss German is spoken language with no formal writ- ing system. Thus, the task is formulated as the translation of Swiss German audio into Standard German text (Plüss et al., 2021) without access to intermediate textual representation of the source language. This leads difficulties where Swiss German and Standard German differ, e.g. in the usage of tenses (where Swiss German does not use preterite) or lexical items that are distinct berween the two languages. The second challenge is that Swiss German consists of many dialects that tend to differ. This issuc makes training a speech trans- lation system from Swiss German Standard Ger- man a task of translating many dialects into single language. The third challenge that Swiss Ger- man a low-resource language with only around Million speakers. Furthermore, the fact that some
Our findings show that the empirical results fol- low the linguistic investigations. That is, there are dialects that expect to differ significantly from others, which is confirmed by our empirical results. Furthermore, the differences between Swiss Ger- man and Standard German impact the performance. where the past tense has the highest impact.
2 Swiss German Dialects
2.1 Linguistic background
"Swiss German" commonly refers to the dialects spoken in German-speaking Switzerland. All

---

Swiss German dialects are Alemannic dialects' be- longing the High German languages within the West Germanic language family. The Alemannic dialects in Switzerland can be further split into High Alemannic and Highest Alemannic variants
(ieect mejiess STT4S6-115
0090
carevici
4000
The sociolinguistic situation in German- speaking Switzerland is particular: unlike in other linguistic areas, high prestige is associated with dialects (Ender and Kaiser, 2009), which are considered important markers of regional identity. This means that Swiss German dialects are spoken in most everyday situations, including formal ones. They are, however, only written in informal contexts, such as social media/text messages, and sometimes also for advertisement purposes. On the other hand, formal correspondence, laws, and newspapers are always written in Standard German. The Standard German of Switzerland differs from the varieties in Germany and Austria and is therefore often referred to as "Swiss Standard German."
66/6
Figure I: Map showing the seven dialect regions in STT4SG-350.
German / Alemannic vocabulary items which are not intelligible to speakers of Standard German In Swiss German, some of these originate from French (e-g.. "trottoir" for pavement). Further, a highly typical Swiss German word formation pro- cess is "-li" suffixation (e.g., "Hündli" little dog)-
In this work, we will investigate the impact of two differences between Swiss German and Stan- dard German: first, preterite tense, which does not exist in Swiss German (Section 6.1), and second, Swiss German vocabulary items that are expected to notably differ from Standard German (Section 6.2).
2.2 Differences between Swiss German and Standard German
Alemannic dialects (AL), including Swiss German, differ from Standard German (DE) in several ways such as:
Phonology: Middle German monophthongs and diphthongs are preserved in Swiss German (AL "huus" vs DE "Haus" (house) and AL "[liob]" vs DE "[li:b]" (dear)) and most dialects have completed the High German consonant shift (AL "['xafta]" vs DE "["kastu]" (box)). Stress is placed on the first syllable of word in Swiss German more often than in DE.
2.3 Dialect regions and dialect differences
The STT4SG-350 speech translation corpus (Plüss et al., 2023a), which we use for the experiments reported in this paper, defines seven dialect regions, They correspond to Swiss cantons as follows:
Basel (BS): Basel-City, Basel-Country and parts of Aargau
Grammar: AL nouns have no genitive case but rather use possessive constructions ("s huus vom buur" "the house the farmer" and "im buur sis huus"- "the farmer his house") and the accusative and nominative cases are conflated (except for per- sonal pronouns). The verbal system has preterite tense, perfect constructs used instead. Relative clauses always use the particle "wo". and some verbs reduplicate in present infinite form when forming a complex predicate with another verb (e.g. "du lohsch mi lo ässe" "you let me [lct] eat"). Lexis: There are a rather large number of Swiss
Berne (BE): Berne, Fribourg, Solothurn and parts of Aargau
Central Switzerland (CS): Lucerne, most Schwyz, Ob- and Nidwalden, Uri, Glarus and parts of Aargau
Eastern Switzerland (ES): St.Gallen, Thur- gau, Schaffhausen, Appenzell Innerrhoden and Appenzell Ausserrhoden
Grisons (GR): Grisons
Valais (VS): Wallis
Zurich (ZH)) Zurich, Höfe district of Schwyz
and parts of Aargau
Figure visualizes these seven dialect regions.
'Except for the dialect Samnaun Grisons, which variant the Bavarian language.
We will investigate similarities between the di-
"The traditional dialect of Basel City has many features of Low Alemannic German, but due to internal migration, these have mostly leveled and are now close to High Alemannic.
SSee, e.g. //dps://de.... http//ewwikkin wwwipedia..ooo.oo. org/wiki/ SchweizerrdaurschhWorttsss

---

alect regions impact ST performance. To quantify the differences between the dialects, we use dialec- tometric data from the Dialect Atlas of German- Speaking Switzerland (DAGSS; (Hotzenköcherle et al., 1962-1997) presented by (Scherrer and Stoeckle, 2016). The digitised DAGSS data set, version 3 (DDAGSS), consists of 289 linguistic features (107 phonological, 118 morphosyntactic, and 64 lexical features) collected from local respon- dents in a field survey across 565 Swiss locations.
Linguistic Distance: Global (239 features)
cs
on
15
2H
0.48
3.58
0.90
0.44
037
0.3
0.21
0
3.51
0.48
0.58
0.52
0.48
0.51
0.60
0.37
0.53
8
g
0.56
2.63
0.57
0.67
0.13
0.64
04
F
We use the DDAGSS features to calculate lin- guistic distance indices based on the Relative Iden- tity Value (RIV) metric (Goebl, 2010). To ap- ply RIV to the STT4SG-350 data, we match the DDAGGS survey sites our seven dialect regions, then calculate RIV for all site pairs* and average this for all region pairs. The result when using all DDAGSS features" is shown in Figure 2. From the pairwise distances, we derive the following hy- potheses:
Figure 2: Matrix of linguistic distances between dialect regions.
(ASR) model different English accents leads faster training and higher accuracy than a model trained from scratch. Transfer learning from En- glish ASR to different languages such as German, Spanish, and Russian also leads to a higher accu- racy than training from scratch (Luo et al., 2021). In (Woldemariam, 2020), transfer learning for low- resource speech recognition in the case of Amharic from English and Mandarin has been studied. For both pre-training languages fine-tuning ASR for Amharic led to an absolute WER reduction 11.22% and 10.25%, respectively. In our work, we extend the research on transfer learning for differ- ent dialects from the ASR set to speech translation task.
VS-hyp. VS is the most distant dialect from other dialect regions, which corresponds well with local perceptions. Thus, we expect that systems that are trained only on VS per- form badly on other dialects, and systems not trained on VS will not perform well on VS. CS/ZH- CS and ZH have less pronounced differences from most other di- alects. Thus, expect systems trained only on ZH or CS to work better than systems trained only on other dialects.
ZH/CS--E-BB-hypp ZH has a very small distance from BS, CS, and ES, respectively. Thus, we expect that systems trained on only BS, CS, or F.S, respectively, will perform well ZH.
Swiss German Speech Translation. Recent high quality datasets such as SwissDial (Dogan- Schönberger et al., 2021), SPC (Plüss et al., 2021), and SDS- (Plüss et al., 2022) enabled notable advancements in Swiss German Speech Transla- tion (ST). These datasets successfully cover a wide range of Swiss German dialects. However, we are not aware of any comprehensive investigation into the interactions between these dialects in ASR ST.
BE/ES-GR-hyp BE has larger distance to ES and GR than the other dialects. Thus, we expect systems trained on BE to perform poorly on ES and GR and vice versa.
3 Related Work
4 General Setup
General Dialeet Transfer. Luo et al. (2021) and Hassan et al. (2022) showed that fine-tuning a pre-trained English Automatic Speech Recognition
4.1 Data
For this investigation, we rely on the STT4SG-350 corpus (Plüss et al., 2023a), which consists of 343 hours of speech translation data. That is, it con- tains audio in Swiss German and the corresponding Standard German sentence. The sentences were presented to speakers who were asked to translate them to their dialect (without writing down the
'Specifically, we use the following process calculate RIV: for two survey sites, we first identify all features for which they both have response (sotal). We then count all features where the responses are not identical (different). We finally calculate the distance metric RIV different total IIn Appendix C, we display the distance matrices per fea- ture group.

---

train all (bal)
valid
test
full
small 244M parameters. We avoid larger models they make the experiments prohibitively time and cost intensive
Hours
276 (2.39)
34
34
343
Rec.
200K (173K)
23K
25K
248K
Unique sent.
192K (167K)
4K
218K
Speakers
219 (192)
21
76
316
Avg. Ree./speaker
912 (902)
1106
324
783
5 Transfer Experiments
Table 1: Corpus statistics per split. For the train set, the balanced (bal) version is in parentheses. Taken from (Plüss et al., 2023a).
We are interested in the interplay of the dialects: which dialects benefit from other dialects being in the training data, and are there dialects where specialized fine-tuning is more adequate? For this, we run two experiments:
translation) and record them. One effect of this setting is that speakers may use synonyms and dif- ferent tenses, word order, etc. when saying the sen- tence aloud in their dialect, leading deviation from the reference sentence. The corpus consists of a train and test set. The particular setting of the test set that contains hours of audio for each the seven dialect regions (cp. Section 2.3) based on the same set of sentences. This allows for conclusive comparisons of dialect transfer effects. STT4SG- 350 also provides balanced training split with 34 hours for each dialect region (see Table 1). In the train set, the sentences are diverse across the various regions. Furthermore, STT4SG-350 is also balanced with respect to gender and age, achieving an almost uniform distribution across these dimen- sions.
1) Leave-One One-Ont (LOO), where we train the models on all dialects except one and measure the impact on the performance of the left-out dialect well as on the other dialects,
2) Single-Dialect (SD), where we fine-tune the models on a single dialect and measure the impact on the performance of the various dialects.
For both experiments, we compare the results to the All-dialects setting, which consists in train- ing the model on the full STT4SG-350 corpus, i.c.. all dialects. For all experiments, we report the BLEU score (Papineni al., 2002), which is cal- culated using the script provided by (Plüss et al., 2023b). To analyse the difference between All- dialects and the different settings, we compute the ratio between the BLEU score achieved by the All- dialects setting and the BLEU score achieved by the specific experiment (retainment ratio). We visu- alise this performance retainment using heatmaps, which show the percentage of performance retained on the jth (column) when leaving out the ith (row) dialect. Thus, value of 0.9 is to be read as the model achieving 90% of the BLEU score that All- dialects achieved. The average at the end each row indicates the average retainment of the other dialects (i.e., excluding the value of the diagonal), which summarizes the influence of one dialect on other dialects. The absolute values are provided in Table 2.
4.2 Models
For the subsequent experiments, we used three models to ensure that the results generalize to vari- architectures.
XLS-R. The first model is pre-trained XLS- R-300M model (Babu et al., 2021). We ini- tially conducted two different XLS-R-300M experi- ments, with CTC decoding and with a sequence-to- sequence architecture. The experiments reported here used the CTC architecture, as they perform better, and the sequence-to-sen costenneett-sequuee se-queecce results do not provide any additional insights. A comparison of CTC and in Appendix B.
results is reported
5.1 Leave-One-Out
Trafo. The second model a randomly initialized transformer model implemented the FAIRSEQ S2T library (Ott et al., 2019) and replicating the model of (Plüss et al., 2022), which we trained from scratch on our data.
The leave-one-out (LOO) experiment fine-tunes the models with the data of all dialects while leaving one dialect out. This tests the impact one dialect has on the others through ablation. Table 2 presents the BLEU scores of the LOO experiment. For each dialect column, the LOO row shows the perfor- mance of the LOO model on this dialect's test set
Whisper. The third model is Whisper small (Rad- ford et al., 2022), a pre-trained transformer-based sequencc-to--sqquence model.
For the investigation, we relied on medium-sized models: the XLSSR model has 317M parameters, the Trafo model 72M parameters, and Whisper
"See Appendix A for a comparison of the 300M and the IB model of XL.S-R which show robust correlation between the two models on dialect-specific metrics, indicating that the relative outcomes are independent of model size.

---

Model
Experiment
Full
BS
BB
GR
CS
ES
VS
ZH
X
All dialects
70.9
69.8
68.4
72.7
72.9
69.5
70.6
72.5
Leave One Out
70.4
67.7
64.9
71.0
72.3
67.4
61.2
72.2
Single Dialect
$7.6
68.7
65.5
69.3
71.2
69.0
71.8
69.5
Tralo
All dialects
62.0
59.1
57.5
64.8
65.9
62.1
61.1
63.8
Leave Out
50.5
49.9
59.3
$6.2
47.7
59.0
Single Dialect
0.1
0.1
0.1
0.1
0.2
0.1
0
Whisper
All dialects
62.9
60.0
59.7
64.6
65.9
62.8
62.2
65.2
Leave One Out
60.2
55.4
54.6
64.6
66.3
61.2
$1.9
63.5
Single Dialect
45.4
49.5
50.5
53.2
57.4
50.1
54.9
52.9
Table 2: BLEU scores of all experiments. Each experiment is tuple of model and experiment type. There three types: All-dialects, Leave-One-Our and Single Dialect. For each setting, we report the score achieved on each dialect separately, and the average score achieved on the full test set.
B5
BE
GR
CS
ES
VS
ZH
Avg
leaving out a dialect leads to lower performance on the left-out dialect compared to using the full dataset. For deeper analysis of the dependence of the dialects, Figure 3 shows the retainment ratios. We make the following observations:
0.97
0.99
0.99
1.00
1.00
1.01
0.99
1.00
1.0
1.00
1.00
1.00
1.00
GR
0.99
0.99
0.98
0.99 0.99
1.00
0.99
0.99
CS
1.00
1.00
1.00
0.9
VS needs in-dialect data. The strongest reduc- tion is measured when leaving out the VS dialect. The drop is consistent across all three models. For XLS-R, the BLEU score only retains 87% of the performance. Trafo only retains 78% of the origi- nal BLEU score, and Whisper retains 83% of the All-dialects score. This drop was expected, the VS dialect is the one that differs the most from all the other dialects (cp. Figure 2).
0.99
0.99
1.00 0.97
1.00
0.8
VS
1.00
1.00
1.00
1.01 1.01
0.87
1:01
1.00
0.7
1.00
1.00 1.00 1.00
0.5
(a) XI.S-
VS
Avg
B5
0.85
0.90
0.93 0.95 0.92 0.92 0.91 0.92
BE
0.93
0.87
0.95 0.95 0.94 0.94
0.94
GR
0.93
0.94 0.92 0.95 0.94 0.94 0.94
0.94
ZH and CS do not require much in-dialect data. Other dialects, such as ZH or CS, experience a smaller reduction. In fact, with XLS-R, both di- alects almost completely retain their performance, and with Trafo, they retain 92% and 94% of the performance average, respectively. This finding matches the CS/ZH--cntral--ypp hypothesis since ZH and CS benefit the most from other dialects. Mutual dependence. We note that when one di- alect is left out with XLS-R, the other dialects do not suffer performance loss. In some cases, the performance even exceeds the All-dialects setting, with ratios above 1. This is due XLSSR's exten- sive pre-training, which allows languages with relatively small amounts of data. Thus, the results with Trafo are more revealing. When VS is left out, the other dialects suffer al- most no deterioration, indicating the special status of this dialect, which is in line with the VS-hyp hypothesis. For the other dialects, omitting them leads to greater performance loss on the other dialects. The largest overall drop in performance occurs when BS or ZH left out, where the aver- age retainment ratio of the other dialects For BS, this drop is not expected, as BS does not have as high pairwise similarities as ZH or CS. For
CS
0.94
0.92 0.94 0.93
0.94
ES
0.92
0.93
0.94 0.95 0.90 0.94 0.93
0.93
0:95 0.96
0.78
0.7
ZH
0.92
0.91
0.93 0.94 0.91 0.92 0.92 0.92
0.6
(b) Trafo
85
BE
GR
C5
E5 V5
ZH Avg
0.95 0.96
1.0
BE
0.97
0.91
0.95
0.96 0.98 0.96 0.97
0.96
3.04
1.00
1.03 1.04
1.03
CS
1.03
1.03
1.02
1.01 1.02
1.02
1.02
1.02
0.9
1.02
V5
1.02
1.02
1.02 1.02 1.03 0.83 1.02
1.02
adapt to new
0.6
0.5
(c) Whisper
Figure 3: Results of LOO Experiment: heatmap with retainment ratio. Each row shows the scores achieved for each dialect when leaving out the dialect the row. The last column shows the average of the ratios (the average excludes the diagonal values.)
92%.
(so when BS is left out, the BLEU score on the BS test set is 67.7). is apparent (and expected) that

---

Avg
ZH is a central dialect. The SD experiments also confirm that the similarity of ZH to ES and CS is reflected in the empirical results. When training on ZH only, the CS and ES scores retain at least 90% (for Whisper the retainment is also at least 76%). Vice versa, we observe the same pattern. The simi- larity between ZH and BS is less pronounced with both KLS-R and Whisper, which is in line with Fig- 2. As with the LOO experiments, BS-only also reaches a high average score, which not expected from the hypotheses.
B5
0.98
1.0
0.71
0.96
0.76
0.85
0.60 0.64 0.78 0.72
GR
0.79
0.95
0.87 0.87
0.81
0.88
0.87
0.98
0.79
0.79
0.94
0.84
ES
0.81
0.00
0.92
0.84
0.99
0.50
0.88
0.78
0.7
0.62
0.67
0.49
0.6
ZH
0.85
0.76
0.88 0.94 0.90 0.71
0.96...84
0.96
0.84
(a)
XLS-R
85
BE
GR
C5
E5
VS
ZH
Avg
0.75
0.70
1.0
BE dissimilar to both GR and ES. The SD experiments also confirm the BE/ES-GR-hyp hy- pothesis. For systems trained only on BE, the GR and ES scores suffer the most (next to VS). Also, when the system is trained either GR ES, the BE scores suffer the most (next to VS).
BE
0.09
0.85
0.72 0.77 0.64 0.50 0.74 0.09
0.9
CS
0.75
0.77
0.81
0.87
0.75
0.66
0.85
0.76
0.8
0.65
0.65
VS
0.64
0.62
0.64
0.69 0.55 0.88 0.64
Overall, the experiments confirm the hypotheses that we derived from the dialect distances based on the DDAGGS data. Most notably, the difference of VS to all the other dialects is as pronounced as the hypotheses predicted. Also, the status of ZH as central dialect is confirmed by our experiments. On the model side, we note that both XLS-R R and Whisper have strong pre-training, where leaving out one dialect does not hurt performance. On the other hand, training only on single dialect leads to subpar performance on samples of a different dialect.
0.73 0.66
0.5
(b) Whisper
Figure 4: Results of SD Experiment, when the system trained on the dialect of the row. The last column shows the average score ratios (the average excludes the diagonal values.)
ZH, the decrease is consistent with the centrality hypothesis of Z... However, there pairs of di- alects where omitting them leads loss of more than 10% of the original performance.
Swiss German differences to Standard German
5.2 Single Dialect Fine-Tuning
We investigate two main differences between Swiss German and Standard German (cp. Section 2.2). First, the usage of the past tense, i.e., samples where the Standard German text includes the preterite (simple past), which is not used in Swiss German. Second, the usage of words which are different in Swiss German compared to Standard German.
The single dialect fine-tuning (SD) experiment fine- tunes the models on single dialect and measures the effect all dialects. Table 2 gives an overview of the results. We can see that the final BLEU scores for Trafo were O, as 50 hours of data are too little to train such a model from scratch. Thus, we only show and discuss the XLS-R and Whisper retainment ratios in Fig 4.
Training single dialect is sufficient for that dialect. The results show that with XLS-R, single dialect training leads to at least 95% of the All- dialects performance.
6.1 Preterite
We expect to see mismatch between the transcript and the reference in those samples where the Stan- dard German reference contains the simple past tense. We applied spaCy? to find samples in the test set where the preterite tense used the Standard German text. There are 5908 samples containing preterite tense (23.4%). We used the All-dialects model to compute the BLEU scores separately for
VS Status confirmed. The SD experiments again confirm VS's status, the most distinct dialect training on VS alone leads to an improvement of 2% over the All-dialects setting. We also note that VS and ES have the highest distance in both the XLS-R and the Whisper experiments, which corresponds to the linguistic distance in Figure 2.
hhttps://spacy.io

---

Text Type
Correct
Response
TARGET
der gemeinderat hatte hier bercits auf 10000000 gekürzt der gemeinderat hat hier bereits auf 10000000 gekürzt
REWRITTEN
HYPOTHESIS
Yes
der gemeinderat hat hier bereits auf 100 000 franken gekürzt
TARGET
wir wurden sehr gut informiert wir sind sehr gut informiert und dokumentiert worden wir sind sehr gut informiert und dokumentiert
dokumentiert
REWRITTEN
HYPOTHESIS
No
TARGET
der 42 jährige hatte dort heimlich trainiert der 42 jährige hatte dort heimlich trainiert
REWRITTEN
HYPOTHESIS
der 42 jährige trainierte dort heimlich
Table 3: Examples of Target, Rewritten Target, and generated Hypothesis the Past Tense experiments.
impact is shown (Past Tense + Reference). Extend- ing the references yields improvement of almost 5 points in BLEU score, i.e., there 1197 out of 5908 samples where the BLEU score improves when adding the rewritten reference.
Neon 72.27
Sentence BLEU
Hoar: $2.50
This result shows that measuring the correct- ness of samples with past tense is not trivial since there two cases to consider: 1) the transcript is correct but uses the past perfect form, which does not match the reference. the transcript is wrong due difficulties handling the past tense. In Table 3, we show some examples. In the first example, the hypothesis uses the past perfect form correctly, while the target uses the past perfect. In the second example, which is in the passive voice, the hypothesis uses the present perfect but omits the participle. In the last example, the hypothesis uses the simple past while the target uses the past perfect. These examples illustrate the difficulty of measuring the effects of past tense.
20
No Past
Tense Refermnew
Figure 5: Comparison of distributions of sentence level BLEU scores for sentences with (Past Tense) and with- out past tense (No Past Tense). The Past Tense + Ref- erence shows the BLEU distribution for samples with the rephrased reference. The differences between "No Past Tense" and "Past Tense" are significant according to Welch's t-test (p=5.95e-104..
the samples with past tense and those without past tense. Figure 5 shows the results. Samples with preterite tense in the Standard German text perform significantly worse than the ones without (mean BLEU of 0.625 vs. 0.722).
6.2 Vocabulary
We collect lists of words where Swiss German and Standard German are likely differ. We then mea- sure the differences BLEU scores between sam- ples that contain such words and those that do not. To our knowledge, there is no inventory of Stan- dard German words which are realised differently in Swiss German dialects. Therefore, use three data sources proxy and apply (semi-)manual filtering to each.
Qualitative Analysis. qualitative analysis re- vealed two types of errors among the samples con- taining preterite tense in the target. The first cate- gory of mistakes is those that could not be traced back to the use of past tense, i.e., those where other mistakes yielded lower BLEU score.
In the second category, the mistakes are due to the past tense. There, we noticed that the generated text often did not match the tense of the hypothesis. Thus, following the idea of paraphrasing to extend the set of references (Paonessa et al., 2023), we extended the set of references for all the past tense samples by translating them into the past perfect form. For this, we used ShatGPTS In Figure 5, the
1) Volumes 5-8 of the DDAGGS contain 64 lexical features that are expected to differ between the different dialects. We keep those entries where none of the realisations dard German equivalent e.g., we keep "heraus" ("out" in the directional sense) with realisations "us", "use", "usi", "uss". "usse", "uus", "uuse", "uusi" and "uuss", but we discard "flicken" ("to mend") with realisations "flicke", "nääje", "büeze",
cognates of the Stan-
We used the g-t-3.5-turbo modellhtttps://plafffoo openai. con.i.ccomdoeee cow/docs// dww.s//gide//gpttttt

---

Type
Correct
Response
TARGET
ctwas weniger als eine stunde war er dort gewesen etwas weniger als eine stunde war er da und schaut es aus beim direkten vergleich und wie sieht es aus beim direkten vergleich
HYPOTHESIS
Yes
TARGET
HYPOTHESIS
Yes
TARGET
ich arbeite sehr oft mit diesen werten ich arbeite sehr viel mit den werten
HYPOTHESIS
Yes
Table 4: Examples of Targets and generated Hypotheses for the vocabulary experiments.
etc. After this filtering step, 18 items remain.
100
2) The Wikipedia entry on Helvetisms con- tains 503 entries representing lexical items specific to Swiss (Standard) German. remove proverbs and obsolete expressions and then manually filter the remaining entries: we try to intuit whether a given Standard German word is usually realised in Swiss German with different lexeme (because the Standard German lexeme does not exist or is aare/dispreferred.. This is subjective process that is limited by the evaluator's knowledge and percep- tion of the different dialects. After filtering, 262 words remain.
Mean: 70.35
Neia: 06.
Sentence BLEU
60
20
No Special
All
Special
Figure 6: Comparison of distributions of sentence level BLEU scores for sentences containing words of spe- cial vocabulary. The differences between "No Special" and "Special" are significant according to Welch's 's t-test (p=5.3e-13).
3) The GSWNorm 2022 Shared Task (von Däniken al., 2022) presented data set of 7284 Swiss German sentences (106K tokens / 20K types) with word-level normalisation into Standard Ger- man. order to reduce the set of types tered manually, we apply heuristic: we keep only those pairs where the first letters of the two words are not identical, leading to 2569 candidate types. We then filter these in the same way Wikipedia Helvetisms, resulting in final list of 267 items. We combine these three word lists and after elimi- nating duplicates between the lists, 522 vocabulary items remain. There are 2975 samples (12.1%% that contain at least one special vocabulary item. Results. Figure 6 shows the difference in BLEU score between those samples without special cabulary (No Special), the samples with special vocabulary (Special), and All-dialects (All). The difference in BLEU score is large (66. 13 vs. 70.39). However, as with the Past Tense, the source of the BLEU difference is unclear, i.e., we do not know whether the difference stems from the transcript being wrong or the hypothesis using synonym not covered by the reference (such synonyms may be introduced during translation and recording, Section 4.1). To illustrate this issue, Table shows examples mismatches between the transcript and
fil-
the target. In most examples, the Swiss version of the word was used. For instance, Swiss German "wie schaut es aus", vs. Standard German "wie sieht es aus". Thus, a fair amount of the mismatch in terms of BLEU score can likely be attributed to using the Standard German version in the target.
Conclusion
Our findings show that the empirical results consistent with the linguistic distances between the dialects and the differences between Swiss Ger- man and Standard German. For example. dialects similar to other dialects (such as ZH) positively affect others when included in the training data. In contrast, dialects that differ from others (such as VS) need in-dialect data to perform well. The usage of preterite and vocabulary specific to Swiss German impact the BLEU score negatively. Thus, future work is concerned with investigating meth- ods dealing with more efficient dialect transfer and handling the differences between Swiss German and Standard German.
Phttps://de. http:://dewwwwiiiie http://wwiiii....giiii/iiiissss. _oo_evoo_tee von... HelvettismentWortsscaaazz accessed 12 May 2023

---

Limitations
Hofmann. 2021.
SwissDial: Parallel Multidi- alectal Corpus of Spoken Swiss German. CoRR. 103/2103. 1401.
No Learning Curve. In our experiments, we ei- ther used the full set of dialect data (SD) or none of the dialect data (LOO). A more fine-grained anal- ysis would include computing learning curves to understand how much "new" dialect data is needed. This would also give more hints the interplay be- tween dialects. However, these experiments would be very costly to run.
Andrea Ender and Irmtraud Kaiser. 2009. Zum Stellen- wert von Dialekt und Standard im österreichischen und Schweizer Ergebnisse einer Umfrage.
Hans Goebl, 2010. Dialectometry and Quantitative Mapping. In Language and Space. An International Handbook of Linguistic Variation, vol. 2, pages 433- 457. De Gruyter.
No Tests on very large SOTA models. We lim- ited our experiments to models with around 300M parameters and not very large dallion-parameter models. There are two reasons: first, the training time and cost would have become prohibitively large, and second, shown in Appendix A, the insights are expected to be largely the same. Error Attribution Past Tense. Measuring the mis- takes caused by the past tense is not trivial since the mistakes could have two main sources: 1) a non-tense-related error or 2) a tense-related error. From the latter, there are again two subcases: 1) the model could not handle the tense and made mis- takes due to that (e.g., second example in Table 3) 2) the model behaves well, but the reference does not match the tense generated by the model (which could be caused by the translation and recording process, see Section 4.1). Thus, we cannot mea- sure how many errors are due to which of the above error types, only the impact on the BLEU score. Error Attribution Vocabulary. Similarly, for the special vocabulary, we can only measure the impact on the BLEU score. The errors could also be due to a noo-vocabulary-reaatee source. If they are caused by vocabulary, it is still not clear whether the error stems from using wrong word or a synonym that is not covered by the reference.
Muhammad Ahmed Hassan, Asim Rehmat, Muham- mad Usman Ghani Khan, and Muhammad Haroon Yousaf. 2022. Improvement in Automatic Speech Recognition of South Asian Accent Using Transfer Learning of DeepSpeech2. Mathematical Problems in Engineering, 2022:6825555.
Rudolf Hotzenköcherle, Robert Schläpfer, and Paul Zinsli. 1962-1997. Sprachatlas der deutschen Schweiz.
Jian Luo, Jianzong Wang, Ning Cheng, Edward Xiao, Jing Xiao, Georg Kucsko, Patrick O'Neill, Jagadeesh Balam, Slyne Deng, Adriana Flores, Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev, Vitaly Lavrukhin, and Jason Li. 2021. Cross-Language Transfer Learn- ing and Domain Adaptation for End-to-End Auto- matic Speech Recognition. In 2021 IEEE Interna- tional Conference on Multimedia and Expo (ICME), pages 1-6.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demon- strations), pages 48-53, Minneupolis, Minnesota. As- sociation for Computational Linguistics.
Claudio Paonessa, Dominik Frefel, and Manfred Vogel. 2023. Improving Metrics for Speech Translation. arXiv preprint arXiv:2305. 12918.
Word List Subjectivity. The creation of the word list was done mostly ad-hoe and based on the sub- jective interpretations of the word-list creator. Fur- thermore, we did not differentiate between dialects and some dialects may use vocabulary similar to Standard German.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a Method for Automatic Eval- uation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Com- putational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Michel Plüss, Jan Deriu, Yanick Schraner, Claudio Paonessa, Julia Hartmann, Larissa Schmidt, Chris- tian Scheller, Manuela Hürlimann, Tanja Samardžić, Manfred Vogel, al. 2023a. STT4SG-350: A Speech Corpus for All Swiss German Dialect Re- gions. arXiv preprint 2305 18855.
References
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021. XLS-R: supervupee cross- pross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296.
Michel Plüss, Manuela Hürlimann, Marc Cuny, Alla Stöckli, Nikolaos Kapotis, Julia Hartmann, Mal- gorzata Anna Ulasik, Christian Scheller, Yanick Schraner, Amit Jain, Jan Deriu, Mark Cieliehak, and
Pelin Dogan-Schönnerge Julian Mäder, and Thomas

---

Manfred Vogel. 2022. SDS-200: Swiss German speech to Standard German text corpus. In Pro- ceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3250-3256, Marseille, France. European Language Resources Association.
EucL (1. AAP0) 421
0L0L 26,900 9X53)
0730
0.79
0772
0771
0696
Michel Plüss, Lukas Neukom, Christian Scheller, and Manfred Vogel. 2021. Swiss Parliaments Corpus, an Automatically Aligned Swiss German Speech to Standard German Text Corpus. In Swiss Text Analyt- ics Conference 2021, Proceedings of the Swiss Text Analytics Conference 2021.
SiasY
Michel Plúss, Yanick Schraner, Christian Scheller, and Manfred Vogel. 2023b. 2nd Swiss German Speech Standard German Text Shared Task at SwissText 2022. arXiv preprint arXiv:2301.0677
Figure 7: Comparison of two XLS-R models: XLS-R IB and XLS-R 300M.
the STT4SG-350 balanced dataset for 80K steps. The 1B model took 48 hours to complete the train- ing, while the 300M parameter model took 28 hours. The IB model obtains a higher BLEU score on all dialects in the test set. Both models have the same strengths and weaknesses the different di- alects. Therefore, our findings on the 300M model can transferred to the IB parameter model. This study uses the 300M model as we aim for shorter training times.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2022. Robust Speech Recognition via Large-Scale Weak Supervision.
Yves Scherrer and Philipp Stoeckle. 2016. A quan- titative approach to Swiss German Dialectometric analyses and comparisons of linguistic levels. Di- alectologia et Geolinguistica, 24(1):92--25.
Pius von Däniken, Manuela Hürlimann, and Mark Cieliebak. 2022, GSWNORM2022- Shared Task on Text Normalization for Swiss German. In Pro- ceedings of the 7th Swiss Text Analytics Conference, SwissText Conference Proceedings, Lugano. CEUR- WS.
B Comparison of XLS-R 300M with CTC vs. sequence-to-sequeee decoder
We executed all XLS-R-300M experiments with two different decoders: The standard CTC de- coder and sequence-to-sequennee (Seq2Seq) de- coder. In Table 5 we compare the two decoders the STT4SG-350 balanced dataset. The CTC obtains overall BLEU score of 0.709, whereas the seq2seq decoder achieves 0.659. In the low data regime with the dialect from scratch exper- iments, CTC model results an average 0.576 + 0.044 BLEU score and seq2seq in an av- erage 0.107 + 0.026 BLEU score. The average BLEU score is calculated over seven training runs using different dialect for training each time. We observe, the decoder in the Seq2Seq architecture cannot learn decoding with only 34 hours of train- ing data.
Changhan Wang, Yun Tang, Xutai Ma. Anne Wu, Dmytro Okhonko, and Juan Pino. 2020. Fairseq S2T: Fast speech-to-text modeling with fairseq. In Proceedings of the 1st Conference of the Asia-Pucifei Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference on Natural Language Processing: System Demon- strations. pages 33-39, Suzhou, China. Association for Computational Linguistics.
Yonas Woldemariam. 2020. Transfer Learning for Less- Resourced Semitic Languages Speech Recognition: the Case of Amharic. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resoureed languages (SLTU) and Collab- oration and Computing for Minder-Resurreel Lan- guages (CCURL). pages 61-69, Marseille, France. European Language Resources association.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: art thetthhrrttttt natural language processing. CoRR, abs/1910.03771.
C Linguistic Distance Matrices by Feature Type
State-of-thhe
In Figures 8a, 8b and 8c we show the linguistic feature matrices by feature type. While Figure 2 shows the Relative Identity Values (RIV) across all 289 features of the DDAGGS, they are presented individually here: Figure 8a shows the values based on the 107 phonological features, Figure 8b based
A Comparison XLS-R 1B model vs XLS-R 300M model
Figure 7, we compare the XLS-R-IB and the XL.S- R- 300M models. We trained both models on

---

Model
Full
B8
BF
GR
CS
ES
VS
ZH
CTC
0.709
0.698
0.684
0.727
0.729
0.695
0.706
0.725
Seq2Seq
0.659
0.645
0.635
0.675
0.676
0.651
0.663
0.671
Table 5: Comparison of two XLS-R models: XLS-R with CTC decoding and XLS-R with Seq2Seq head.
Uiguissic: Distance: Phonoloiccal (3C7 "estures)
D Experiment Details
dansage
The vocabulary used to preprocess the sentences is limited to lower-case characters and the German umlauts ä, ö, and ü. All characters with other ac- cents are transformed into their corresponding char- acter without accents, and hyphens are replaced with a space.
0.43
0.20
0.45
140
3:2
0.22
0.42
23
0:11
0:30
0.41
0.35
05
0:55
0.92
2
8-
6:4
2:40
645
0.47
346
0:46
0,47
08
042
344
0.32
XLS-R. We use the fairseq implementation 10 and replicate the training procedure and model settings from (Plüss et al., 2023a). The runs on the full dataset and the Leave-One-Out (LOO) experiment are trained for 80K steps. The Single Dialect Fine- Tuning (SD) models are only trained for another 20K steps without any freezing during the warmup phase. All the final models correspond to the check- point with the best Word Error Rate on the vali- dation dataset during training. There is no task- specific fine-tuning of the hyperparameters. The training for the 300M version of the model is con- ducted on 2 NVIDIA A100 40 GB GPUs. The IB All-dialects model is trained on 4 NVIDIA A A100 40 GB GPUs
0:65
6.55
30
0.9
(a) Matrix of linguistic distance between dialect regions:
phonological features
Unguisiic Distance: Morphossytactic 1115 features)
Samsage
C4
0.42
0.21
0.54
0.50
0.40
2
0:45
641
0.47
0:48
055
0:5
0/96
8-
0.54
0:54
0.48
0.54
144
A4:
0.52
0.33
035
9:40
028
855
6.39
3:5
6:96
(b) Matrix of linguistic distance between dialect regions:
Trafo We replicate the model architecture and train- ing procedure from Plüss et al. (2022). This model is based on the FAIRSEQ S2T library (Ott et al., 2019; Wang et al., 2020). The runs the full dataset and the Leave-One-Out (LOO) experiment are trained for 80K steps. The Single Dialect Fine- Tuning (SD) models are only trained for another 10K steps. There is no task-specific fine-tuning of the hyperparameters. The models are trained on a single NVIDIA A 100 40 GB GPUs.
morphosyntactic features
Linqulstic Distance:
Lexical 164 featurasi
C5
OR
2H
0.24
0.72
9:70
E.70
0.69
0.75
6.72
0.7
22
0.64
3.00
0.3
0:T
6:7
05
0.62
Blice
0uth
0.3
127
0.74
072
0.70
0.2
096
6:75
Whisper We use the Huggingface (Wolf et al., 2019) implementation" of the Whisper small model including the provided fine-tuning proce- dure. The runs on the full dataset and the Leave- One-Out (LOO) experiments are trained for 80K steps and a warmup phase of 10K steps. The Single Dialect Fine-Tuning (SD) models are only trained for another 10K steps with warmup phase of 1K steps. The learning rate is set to le 5. There is no task task-specific specific fine-tuning of the hyper-
(c) Matrix of linguistic distance between dialect regions:
lexical features.
Figure
on the 118 morphosyntactic features and Figure 8c based on 64 lexical features. We can see that VS the most distinet dialect across all feature types, followed by GR, BE and ES in changing order changing depending on the feature.
1n "nttps://githud cow.e/acbboorrsssra//aaaaarsrsss tree/ hwwe//////eeaampll///eeeeeeeee/e thttpps://huginng
wo//ooo//tttteoooooooooooooooooo httss/ttaasfformreeo model_ model_doc/whisper

---

Modd
Moric
Full
BS
GR
CS
ES
VS
211
XISK
WHR
16.1
16.6
18.3
15.0
14.1
16.9
16.2
14.9
chrP
88.1
87.0
89.5
89.8
58.0
88.5
CER
TER
Trado
WER
24.1
26.1
29-1
21.8
20.3
23.5
25.1
22.3
shef
62.7
x1.2
79.0
84.6
85.5
83.3
81.7
84.0
CER
14.0
14.7
17.8
12.2
11.6
13.7
14.5
13.0
TER
23.9
76.0
20.3
21.5
20.3
23.2
25.0
22.3
Whisper
25.4
cheP
23.3
23.6
TER
24.2
25.0
21.5
21.1
22.9
23.0
21.4
LOO
WER SURR-XA XLSA
Table BLEU scores of all experiments. Each experi- ment is tuple of model and experiment type. There three types: All-dialects, Seeve-One-ee Out, and Single Di- alect. each setting, we report score achieved on each dialect separately, and the average score achieved on the full test set.
B5
BE
GR
C5
ES
VS
ZH Avg
BS
0.92
1.0
BE
0.00
0.99
GR
0.97
0.90
0.93
0.97 0.96 0.90
0.90
0.97
C$
1.00
0.99 0.99
0.97 0.99 0.99
0.58
0.99
0.9
ES
0.998
1.00 0.99
1.03 0.92 1.00
0.96
1.00
0.8
VS
1:00
1.01 1.01
1.01 1.01
0.71
1.01 1.01
parameters. The models are trained on 4 NVIDIA A100 40 GB GPUs.
ZH
1.00
1.00 0.99
0.59 1.00 1.09 0.98 1.00
E Experiments Alternative Metrics
XLS-R
In the main text, we use the BLEU score for our analysis, here, we present the same type of results using different metrics: WER, TER and chff. Fig- ure shows the LOO experiment results using WER, figure 10 shows the LOO experiment re- sults using TER, and figure shows the LOO experiment using chrF. All the metrics yield the same underlying results and conclusions as using the BLEU score. In table 6, the All dialects scores for the different metrics are presented. Figure 12 shows the SD experiment results using WER, fig- ure 13 shows the SD experiment results using TER, and figure 14 shows the SD experiment using chrlF. All the metrics yield the same underlying results and conclusions as using the BLEU score.
0.75 0.84 a.8
0.88
C5
ES 0.55
VS 0.91 0.92 0.93 0.92 0.94 0.67
0.93
ZH 0.85 0.86 0.85 0.85 0.83 0.84 0.84 0.85
0.5
(b) Trafo
LOO WEB
B5
BE
GR
B5 0.05 0.98 0.84 0.53 0.91 0.88 0.92 0.00
BE 0.92 0.86 0.82 0.87 0.93 0.86 0.88 0.80
-1.2 1:15 1.11
0.31 1.14 1.14 1.15
1.14
3.14
0.7
0.6
(c) Whisper
Figure 9: Results of LOO Experiment: heatmap with retainment ratio. Each row shows the scores achieved each dialect when leaving out the dialect of row. The last column shows the average score of the ratios (the average excludes the diagonal values.)

---

LOO
TER XLSR
LOO
chrf XLSI
85
BE
GR
CS ES
VS
ZH Avg
B5
BE
GR
CS
ES
VS
ZH Avg
0.95 0.98 1.01
BS
1.0
1.0
0.09 1.00 1.01
BE
1.00
0.98
GR
0.96
9.90
0.93
0.96 0.96 0.90 0.97
0.97
GR
0.99
0.99
0.99
1.00 0.99 1.00 1.00 1.00
CS
1.00
0.99
0.99
0.97 0.99 0.99
0.99
0.9
C$
1.00
1.00 1.00
1.00 1.00 1.00 1.00
1.00
0.9
ES
0.98
1.00 0.99
1.01 0.91 1.01
0.95 0.99
0.8
ES
1.00 1.00 1.00
1.00 0.99 1.00 1.00
1.00
0.8
VS
1.00
1.01 1.11
1.00
101
0.71
3.01
1.01
1.01
VS
1.00 1.00 1.00
1.00 1.00 0.94
1.00
1.00
ZH
1.00
1.00
0.99
0.99 1.00
1.02
0.97 1.00
ZH
1.00
1.00 1.00
1.00 1.00 1.00 1.00 1.00
(a) XLS-R
(a) XLS-R
Avg
8.8 0.88 0.84 0.82 0.85 0.87 0.04
0.91 0.98 0.66
6.95 0.95
1.0
0.88 0.88
0.96
C5
0.37
ES
0.00
0.96 0.97
VS 0.91 0.92 0.93 0.92 0.94 0.67 0.92
VS
0.97 0.97 0.90 0.98 0.99 0.89 0.90 0.90
ZH
0.85 0.86 0.85 0.85 0.82 0.84 0.84
ZH
0.95 0.95 0.96 0.97 0.95 0:95 0.96 0.96
0.5
0.5
(b) Trafo
(b) Trafo
Whisper ES VS ZH Avg
chif
85
BE
GR
85
0.85 0.93 0.86 0.93 0.91 0.88 0.03 6.01
B5
0.00
1.00
1.00 100 1.00 1.00 1.00 1.00
BE 0.92
0.85
0.84 0.87 0.94 0.88 0.89
BE
1.01 0.90 1.01 1.00 1.01 1.01 1.00 1.01
0.15
1.0 1.14 113 1.38
-.002
1.01 1.01
1.10 1009 1.08 2.31
0.00
1.01 1.01
1.00
1.01
0.7
0.7
0.6
0.6
(c) Whisper
(c) Whisper
Figure 10: Results of LOO Experiment: heatmap with retainment ratio. Each row shows the scores achieved for each dialect when leaving out the dialect the row. The last column shows the average of the ratios (the average excludes the diagonal values.)
Figure 11: Results of LOO Experiment: heatmap with retainment ratio. Each row shows the scores achieved each dialect when leaving out the dialect of row. The last column shows the average score of the ratios (the average excludes the diagonal values.)

---

SD
WER
XLSR
5D
TER XLSR
BS
BE
GR
CS
ES VS
ZH Avg
BS
BE
GR
CS
ES VS
ZH Avg
85
0.75 0.77 0.52
0.70
85
0.94
0.00
8E
8.66 0.68 6.06
BE
0.50
0.9
GR
0.62
8.57
0.87 0.70 0.72
0.50
0.65
0.65
GR
0.66
0.56
0.87
0.69
0.71
0.40
0.67
0.65
C5
0.68
0.76 0.71
0.91 0.62 0.51
0.82
0.69
0.8
C5
0.66
0.76
0.71
0.53
0.82 0.60
0.61
0.69
0.8
0.64
0.51
0.80
0.65 0.96
0.41
0.62
ES
0.64
0.50
0.79
0.64
0.96
0.40
0.69
0.61
0.53
0:30
1.04
0.50
0.47
0.45
0.46
0.09
0.07
0.09
0.00
0.72
0.06
(a) XLS-R
(a) XLS-R
SD WER Whisper GR
5D TER
Whisper
ES VS ZH Avg
85
BE
GR
CS
Avg
1.0
1.0
GR
GR
0.51
0.8
C5
0.66
0.8
ES
0.64
0.7
ES
0.53
0.51
0.7
VS 0.60
8.59 0.57 0.60 0.54
0.80
0.58
VS 0.57
0.56 0.54
0.57
0.50
0.77
0.58
0.56
0.6
0.6
ZH 0.62 8.59 0.62 0.67 0.68 0.53
0.65 0.62
ZH
0.52
0.58 0.61
0.85 0.84
0.85
0.60
0.5
0.5
(b) Whisper
(b) Whisper
Figure 12: Results of SD Experiment, when the system is trained on the dialect of the row. The last column shows the average score of the ratios (the average ex- cludes the diagonal values.)
Figure 13: Results of SD Experiment, when the system trained on the dialect of the row. The last column shows the average score of the ratios (the average ex- cludes the diagonal values.)

---

SD
ChiF
XLSR
BS
BE
GR
CS
ES
VS
ZH Avg
85
0.96
0.96 0.33
8E
0.83
0.9
GR
0.91
0.88
0.59
0.05
0.05
0.56
0.91
0.56
0.01
C5
0.92
0.96 0.95
3.00 0.91 0.92
0.96
0.96
0.8
0.92
0.64
0.97
0.93
1.00 0.80
0.95
0.90
0.75
1:01
0.87
0.06
0.90 0.96 0.89 0.99
(a) XLS-R
SD chif Whisper ES ZH Avg
1.0
0.8
ES
0.86
0.7
VS 0.89
8.88 0.89 0.51 0.85 0.99
0.92
0.6
ZH 0.95 8.50 0.94 0.55 0.54 0.56
0.96
0.5
(b) Whisper
Figure 14: Results of SD Experiment, when the system is trained on the dialect of the row. The last column shows the average score of the ratios (the average ex- cludes the diagonal values.)


Reviews:
[{"review_id": "8ae95gBYlq", "reviewer": "Reviewer_qYnP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the positive feedback, we appreciate this a lot.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I'd like this paper to be accepted because it goes beyond the current \"macro-language\" perspective in NLP and delves into dialects and non-standard varieties. Also, the low-resourced setting and how to deal with it is a major contribution to the field.", "reasons_to_reject": "No reason.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper sheds light on the Swiss German varieties and investigates the interplay of dialects vs. Standard German in speech translation. The methodology is compelling and the findings are very interesting. I particularly like the detailed description of various linguistic features of the targetted dialects. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "v5Lr4qTN7G", "reviewer": "Reviewer_xjs8", "summary": "", "questions": "Question A: Could you clarify what data the Trafo model was trained on? (pretraining and finetuning)\n\nQuestion B: Given the other experiments showing CTC performs significantly better than a seq2seq decoder trained from scratch, why not use CTC here as well for a more appropriate comparison to a model trained directly on/for Swiss German? What is the size of the seq2seq decoder? Given that the target language is a high resource language (Standard German) why not use a pretrained decoder or additional language modelling data? \n\nQuestion C: For context for the scores presented here, if a faithful dialectal *transcript* was output rather than the Standard German translation, what would the BLEU scores with the Standard German reference translation be? Given that the vocabulary analysis says some models leave special vocabulary in Swiss German, this would be particularly good to report. \n\nQuestion D: Would it be possible to include Standard German in the linguistic distances for context? \n\nQuestion E: What were the model vocabularies used? It is said that the preprocessing limits characters to lowercase characters (ASCII characters?) and the German umlauts ä, ö, and ü. Does this mean all (non-Whisper) models use character-level vocabularies, or is this just a normalization step?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions, we will answer them here and address some of your concerns.\n\nQuestion A:\n* The Trafo model is a randomly initialized Encoder-Decoder model, which is trained from scratch. Meaning that there is no pre-training using SDS-200 or any other pre-training, only trained on the various settings in the paper (All-dialects, LOO, SingleDialect from the SG4STT corpus). \n\nQuestion B: \n* The experiments in Appendix B only focus on the XLS-R 300M model. In that case, using CTC performed better than adding a neural decoder. \n* Why not use a pre-trained decoder? In this work, we were more interested in the interplay between the dialects, and not in maximizing the scores. The Trafo case covers the setting where the model is trained completely from scratch without any pretraining. We only employed FairseqLM during the decoding as an additional LM. The Whisper experiments cover the setting of having a pre-trained encoder and decoder architecture.\n* Trafo details: 2-layer convolutional subsampler, 12 transformer encoder layers, 6 transformer decoder layers. Each transformer layer has 8 attention heads, and an embedding size of 512. This model corresponds to the T-Md (Transformer Medium) model from fairseq S2T (Wang et al., 2020) with 72 million parameters.\n\nQuestion C:\n* To check how well a Swiss German transcript would perform when compared to a Standard German Reference, we ran a new experiment where we used the SwissDial corpus (Dogan et al 2021) that contains parallel Data of different dialects and Standard German texts. We computed the BLEU score between the Dialect text and the Standard German texts for each dialect separately. The BLEU scores are very low and range between 1.3 (CS) and 8.6 (Grisons). The reason is that written Swiss German differs significantly from Standard German as 1) it is written roughly phonetically, 2) there are lexical differences (e.g. “stoßen…aus” vs “macheds” in the example), and 3) differences in word ordering. For example:\n* * Standard German: “Wenn sie sich versammeln oder berühren stoßen sie ein nasales Krächzen aus.”\n* * Zurich Dialect: “Wenn si sich versammled oder berüered macheds es nasals Chrächze.”\n\nDogan-Schönberger, P., Mäder, J., & Hofmann, T. (2021). Swissdial: Parallel multidialectal corpus of spoken swiss german. arXiv preprint arXiv:2103.11401.\t\n\nQuestion D:\n* The (D)DAGS dataset that we use contains only Swiss German dialects but not Standard German, since the data was elicited in Switzerland.\n* In the literature, we found that Scherrer et al. (2019) calculate a dialectality measure for Swiss German on data from the ArchiMob dataset, which contains Swiss German recordings with dialectal and normalised transcriptions. In a simplified procedure, they calculate the Levenshtein distance between the dialectal transcripts and the normalisation layer of ArchiMob to assess the dialectality of the different respondents  (see Section 5.2.2 / Figure 9). The emerging picture is not very clear and does not correlate with geography.  \n\nScherrer, Yves, Tanja Samardžić, and Elvira Glaser. \"Digitising Swiss German: how to process and study a polycentric spoken language.\" Language Resources and Evaluation 53.4 (2019): 735-769.\n\n\nQuestion E:\n* For XLS-R CTC, we used the lower-cased alphabet, the umlaute (ä,ö,ü), and the digits from 0-9 (i.e., string.ascii_lowercase + string.digits + 'äöü ')\n* For Whisper, we used the vocabulary provided by them.\n* For Trafo: 5000 Sentence Piece Model fitted on sentences normalized with the same vocab (only lowercase). To fit the sentence piece model, we used over 67 million sentences from Europarl v7 de, News crawl (WMT) 2019 de, Parlspeech v2 (Bundestag, Nationalrat)\n\n\nOther comments:\n* We will update the descriptions of the experiments in our paper to make the experimental setting clearer, especially the Trafo case.\n* We will update Baseline -> All Dialects, Report BLEU*100, and also put WER info into the Appendix (also requested by Reviewer LM7U).\n* Regarding BLEU by Pluss.  We used the Pluss version, since most papers on Swiss-German to Standard German use this script. We can clarify the use of the NLTK implementation. Furthermore, we were more interested in the relative values than the absolute ones. We also computed SacreBLEU, which yielded the same relative outcome, just with slightly higher scores. \n* Use of space: we believe that a discussion of the heatmaps is helpful to some readers since they compress a lot of information and scores into one number.  However, we will check whether this can be shortened.\n* Adding chrF: we computed the chrF scores for all experiments, and will add them into the appendix, alongside WER, and TER as requested by R-LM7U. In all cases, the effects that we measured using BLEU scores remain the same just with different magnitudes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Experimental comparison of speech translation of Swiss German dialects into Standard German, with three different models, and three different transfer/finetuning settings (all, some, one)\n- Detailed analysis of differences between dialects with examples", "reasons_to_reject": "- Narrow scope using existing models and techniques; differences between dialects are the focus, rather than the techniques, making the conclusions potentially helpful to a specific target audience but not necessarily generalizable to other cases, particularly given that there is no task-specific parameter tuning and not a direct comparison between model settings \n- Some experimental details are missing or confusingly stated. A number of basic model/data details are referred to other works and should be included here (at least in the appendix) for clarity instead of requiring lookups in multiple other papers (for example, the size of the Trafo data, the model architecture, the size of the seq2seq decoder). \n- The space could be used more effectively; analysis is at times redundant with the text as it lists what is in figures and tables (for example, significant space is used for heatmaps whose numbers which are then summarized in the text, rather than further discussion of why there may be differences in performance or trends such as between Whisper and XLS-R).", "questions_to_authors": "Question A: Could you clarify what data the Trafo model was trained on? (pretraining and finetuning)\n\nQuestion B: Given the other experiments showing CTC performs significantly better than a seq2seq decoder trained from scratch, why not use CTC here as well for a more appropriate comparison to a model trained directly on/for Swiss German? What is the size of the seq2seq decoder? Given that the target language is a high resource language (Standard German) why not use a pretrained decoder or additional language modelling data? \n\nQuestion C: For context for the scores presented here, if a faithful dialectal *transcript* was output rather than the Standard German translation, what would the BLEU scores with the Standard German reference translation be? Given that the vocabulary analysis says some models leave special vocabulary in Swiss German, this would be particularly good to report. \n\nQuestion D: Would it be possible to include Standard German in the linguistic distances for context? \n\nQuestion E: What were the model vocabularies used? It is said that the preprocessing limits characters to lowercase characters (ASCII characters?) and the German umlauts ä, ö, and ü. Does this mean all (non-Whisper) models use character-level vocabularies, or is this just a normalization step?", "paper_topic_and_main_contributions": "This paper addresses transfer between Swiss German dialects in speech translation from Swiss German to Standard German.  \nSpecifically, it looks at translation of 7 different Swiss German dialects to German using the STT4SG-350 corpus with the balanced train splits of 34 hours per dialect. Compared are directly finetuning pretrained models (XLS-R and Whisper) on all dialects on a single dialect, or leave one out; and finetuning a transformer model possibly pretrained on SDS-200 (a Swiss German speech translation corpus) on each of these conditions (called \"Trafo\"). Two different model sizes are used for XLS-R to check whether model size has an effect (conclusions are shown to be robust across model scales).  \nResults suggest that transfer ability roughly corresponds to linguistic distances between dialects derived from DDAGGS features.  \nAnalysis on the preterite and vocabulary likely to be specific to Swiss German using crawled word lists with semi-manual filtering is conducted (each leading to the conclusion that there is a difference in these two cases, but stated that it is hard to pinpoint why).  \nResults suggest that leave one out is typically similar to including all dialects, but with a weaker pretrained model (Trafo) more distantly related dialects suffer as more data is left out, and finetuning on a single dialect is insufficient in all cases. There is a greater gap between all/leave one out/single dialect finetuning for Whisper than XLS-R (presumably because it is trained on less non-English data, and so more sensitive to having less finetuning data?).  Reasons for differences between XLS-R and Whisper are not discussed, and the comparison with Trafo is a bit murky.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "It is not completely clear what data the Trafo model is trained on: \n- Section 4.2 says 'the baseline of Plüss et al 2022 is replicated and fine-tuned on our data', and the balanced training is said to be 34 hours per dialect, and looking up Plüss et al 2022 it is a 200 hour Swiss German to Standard German corpus and the baseline is a 12-6 layer encoder-decoder trained on all of the data. On L373 it says re Trafo \"50 hours of data are little to fine-tune such a model from scratch\" when mentioning that the single dialect fine-tuning resulting in a score of 0.  \n- Is it the case that the Trafo model is pretrained on the 200 hours from SDS-200, and then finetuned on each condition? (all dialects, LOO, single dialect). Or, has it been trained from scratch on each condition, using the baseline architecture only from Plüss et al 2022, in which case these models have been trained on between 34 and 276 hours? Should L373 say \"34 hours is too little to fine-tune such a model\" or \"34 hours is too little to train such a model from scratch\" ? If they are in fact fine-tuned, it is surprising that with 200+ hours of Swiss German to Standard German data there are scores of 0?\n\nTable 2: Consider renaming \"Baseline\" to \"All dialects\" for clarity and to help explain why in most cases it performs the best\n\nBLEU scores are typically reported multiplied by 100 to be 0-100, and here presented 0-1; the presented scores are presumably quite high (45-73) and not very low (<1), consider multiplying by 100 to standardize. \n\nIt is stated that model checkpoints are chosen based on WER, but WER is not reported. Consider adding WER for context in the results table, or at least in the appendix. \n\nBLEU is known to be less appropriate than other metrics such as chrF for dialects which are often marked by differences of 1 to 2 characters, causing potentially large differences in BLEU for small differences in system outputs; consider reporting at least chrF as an additional metric.\n\nWhy use the BLEU implementation from Plüss 2023b (which uses NLTK and is not common practice because it is not comparable to other implementations), and not SacreBLEU [(Post 2018)](https://aclanthology.org/W18-6319/)? It should be noted not just that BLEU is calculated using the script provided by (Plüss et al, 2023b) but include \"which uses the NLTK implementation\" so this is clear and scores are not compared with a different implementation.\n\nTypo: L772 \"task-specific specific\""}}, {"review_id": "64XpjttRQo", "reviewer": "Reviewer_LM7U", "summary": "", "questions": "The BLEU scores that you report are in general rather high, more than 50% on average. This means that there is quite a large overlap between the translation and the reference, which means that the task could be considered more an ASR task vs. MT task. Could you provide also WER (word error rate) and TER (translation edit rate) scores at least for some experiments? These would give insights: how different is translation from speech in Swiss German into High German text in terms of WER from the case where the speech and text are in the same language? Also, how much re-ordering is going on (difference between WER and TER numbers will tell us). \n\nAlso, if you compute WER and TER, you can also obtain edit distance alignment between words from the hypothesis and the reference, and this would help with the attribution of vocabulary errors (for example, you can directly look to which words the preteritum words are aligned, etc.)\n\nAnother question: assuming that you would have had recordings of the same utterances (from the test set) but spoken in High German directly, how high do you think would WER be? Would it be close to 0 given the recording conditions? An experiment like that could be done by using a TTS system to pronounce the utterances. This would give a kind of a lower bound on the quality of \"translation\" from Swiss German (I put translation in quotes here to highlight that at least for some of the spoken words it is speech recognition, not translation if they have a one-to-one correspondence with their written high German counterparts). \n\nRelated to this, if you fine-tune a pre-trained model like XLS-R or Whisper on Swiss German, would its performance degrade on High German? This is an important practical question, since, as far as I know, there is a lot of code switching going on in Switzerland between the two, and adding High German to the mix of Swiss German dialect is a good idea for practical applications of your MT/ASR system.\n\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments and interesting questions. We will address them here.\n\nOn using WER and TER\n* It is true that the BLEU scores are rather high, and might not be the ideal measure. However, using the WER is too strict since we are dealing with two language varieties and leads to having many false negatives. Generally, it is a hard setting to evaluate. Furthermore, we are interested in the relative differences in scores for the different scenarios. Having said that, we will gladly add WER and TER scores to the appendix. We will create the same types of Figures and will add the analog to Table 2 into the Appendix. \n* We computed the WER and TER for all our experiments. Regarding the question of the difference between WER and TER: it is small, with TER being only slightly lower. \n* * XLS-R Baseline: WER 16.07 vs. TER 15.82\n* * Trafo Baseline: WER 24.14 vs. TER 23.92\n* In order to keep the response relatively short, we show the WER scores for the pair VS and ZH (the rest will be added to the Appendix):\n*  * WER for the XLS-R Baseline: 16.07 overall, for VS 16.18 for ZH: 14.92\n* * WER for XLS-R leaving out VS on VS: 22.67 for ZH: 14.72 -> thus, confirming that VS needs in-dialect data and ZH even benefits from lack of VS data.\n* * WER for XLS-R with VS from scratch for VS: 15.62, and ZH: 29.75 again confirming the VS statement\n* We will add the plots for WER, TER, and chrF in the Appendix. However, the main conclusions remain the same under these other metrics as well. \n\nOn the performance of Swiss German models on Standard German. \n* We ran both types of experiments that you asked for.\n*  * Performance of Standard German on XLS-R trained on Swiss German.\n* * Change in performance of Whisper after Swiss German fine-tuning.\n* To evaluate the performance of the Swiss German model on Standard German, we leverage the CommonVoice 11 DE data (the TTS route is a bit out of scope in this short timespan). Note that we used the XLS-R 1B model and additional data for Swiss German (SG4STT + SDS-200 + SPC), we report those results (we ran them already some weeks ago). This is why the BLEU scores for Swiss German are much higher than in the paper. We show 4 scenarios 2 using XLS-R and 2 using Whisper:\n* * XLS-R  \n* * * (1) Fine-tune XLS-R on Swiss German data. Results: Common Voice 11 DE testset BLEU of 46.31 (WER: 36.30) & STT4SG-350 test: 76.69 (WER: 12.81)\n* * * (2) Fine-tune XLS-R on Swiss German data+CommonVoice Train.  Results: Common Voice 11 DE testset BLEU of  85.09 (WER: 8.04) & STT4SG-350 test: 75.28 (WER: 13.53)\n* * Whisper\n* * * (1) Whisper Performance on CommonVoice Test before fine-tuning on STT4SG. Results: Common Voice 11 DE testset BLEU of 64.55 (WER: 22.48) & STT4SG-350 test: 26.03 (WER: 51.27)\n* * * (2) Whisper Performance on CommonVoice Test after fine-tuning on STT4SG. Results:  Common Voice 11 DE testset BLEU of 18.93 (WER: 69.42) & STT4SG-350 test: 62.93 (WER: 25.01)\n* Thus, we see the following insights:\n* * Training only on Swiss German does not suffice to learn a good Standard German model.\n* *  Training on a mix of Standard German and Swiss German yields good results for both languages with a slight decrease in Swiss German performance.\n* *  Fine-tuning Whisper on Swiss German lets it forget all about Standard German (catastrophic forgetting). On the other hand, the pre-trained Whisper performs reasonably well on Standard German, but very poorly on Swiss German.\n* *  The results indicate that there is quite a difference between Standard German and Swiss German. \n* *  We’ll add this to the Appendix\n\nOn the difference between ASR vs. ST/MT.\n* The issues that you raise are very interesting. That is, is our task to be interpreted better as a Speech Translation or ASR task? Recently, it was mostly framed as Speech Translation since Swiss German words require a translation and some reordering as well. However, as you rightly point out, it is not as different as French is to English. We think that this would be a very interesting topic for a separate followup paper. \n* We think that for the purpose of our work, the setting of Speech Translation and using BLEU scores suffices, since also using WER or TER yield the same overall conclusions and outcomes as the BLEU analysis - however, we will report them in the Appendix. \n\nVarious things that we will update.\n* On the p-value in Figure 5. Yes, this is the statistical significance, which in our case is significantly below the p < 0.05 value. \n* You are right, we will reference Figure 3 in Section 6.2\n* When talking about “from scratch” we will normalize it to “train from scratch”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiments are very thorough, well-described, and well-justified. The paper will be of value for researchers who not only work on Swiss German but also on other languages and dialects without a standard written form.\n\nThe authors show an example of good scientific work, with a deep analysis of the experiments, focusing on specific phenomena like the past tense in Swiss German vs. high German, showing examples and trying to find explanations for the automatic scoring results.\n\nThree models of different complexity are fine-tuned or trained from scratch, one of them being Whisper, which is the latest pre-trained speech model in fashion at the moment. However, it is interesting to see that another, smaller model XLS-R lead to better results. Also, the authors showed that it is sufficient to use a version of this model with fewer parameters to arrive at similar conclusions, thus performing experiments with less computational resources.\n\nThe paper is overall very clearly written and is easy to follow.\n\n", "reasons_to_reject": "I don't see any major reasons for a rejection apart from maybe more ground-breaking work by other researchers that is more novel. In this paper, the novelty arises from the availability of a new dataset plus the excellent experimental methodology itself.\n\nI would like the authors to address my questions below, though, especially regarding the computation of WER/TER which puts their experiments in the context of speech recognition as opposed to speech translation (and in my opinion, it is necessary to do so given the significant lexical and word order similarity of Swiss German and High German, despite their differences).\n", "questions_to_authors": "The BLEU scores that you report are in general rather high, more than 50% on average. This means that there is quite a large overlap between the translation and the reference, which means that the task could be considered more an ASR task vs. MT task. Could you provide also WER (word error rate) and TER (translation edit rate) scores at least for some experiments? These would give insights: how different is translation from speech in Swiss German into High German text in terms of WER from the case where the speech and text are in the same language? Also, how much re-ordering is going on (difference between WER and TER numbers will tell us). \n\nAlso, if you compute WER and TER, you can also obtain edit distance alignment between words from the hypothesis and the reference, and this would help with the attribution of vocabulary errors (for example, you can directly look to which words the preteritum words are aligned, etc.)\n\nAnother question: assuming that you would have had recordings of the same utterances (from the test set) but spoken in High German directly, how high do you think would WER be? Would it be close to 0 given the recording conditions? An experiment like that could be done by using a TTS system to pronounce the utterances. This would give a kind of a lower bound on the quality of \"translation\" from Swiss German (I put translation in quotes here to highlight that at least for some of the spoken words it is speech recognition, not translation if they have a one-to-one correspondence with their written high German counterparts). \n\nRelated to this, if you fine-tune a pre-trained model like XLS-R or Whisper on Swiss German, would its performance degrade on High German? This is an important practical question, since, as far as I know, there is a lot of code switching going on in Switzerland between the two, and adding High German to the mix of Swiss German dialect is a good idea for practical applications of your MT/ASR system.\n\n\n", "paper_topic_and_main_contributions": "The paper focuses on translating from Swiss German Speech to text in high German. There are many thorough experiments regarding the influence of different dialects of Swiss German on each other, and how some dialects are more distinct and thus are more helpful when data from a dialect like that is used in training.  Three model architectures are tried out, and a very thorough analysis of the experimental results is provided.\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 373: \"to fine-tune such a model from scratch\" - this phrase is contradictory - is it fine-tuning or training from scratch?\nTable 3: this Table is not referenced in the main text of the paper, at least I could not find any reference to it.\nCaption of Figure 5: are you sure about the p= ... of the statistical significance here, is it really this low, or is it a typo?\n"}}]

Author rebuttal:

Meta-review: This work addresses the problem of speech translation from spoken Swiss German dialects into written Standard German and investigates the performance in many different scenarios to demonstrate the effects of dialect diversity empirically.
The reviewers agree on its substance but do not fully acknowledge its “technical” contributions.

Here is a list of the pros and cons of this paper.
* Pros
- Thorough experimental comparison in different scenarios (xjs8, LM7U)
- Valuable for future studies in speech translation for low-resource languages/dialects without a standard written form (qYnP, LM7U)
- Detailed analyses and discussions on the differences among dialects (xjs8, LM7U)
* Cons
- Limited focus on Swiss German, which is not necessarily generalizable to other languages/dialects (xjs8); It is not so serious unless there are any studies working on very similar problems.
Decision: Accept-Findings