Automatic Dialect Adaptation in Finnish and its Effect on Perceived Creativity
Mika Hämäläinen1

mika.hamalainen@helsinki.fi

Niko Partanen2

niko.partanen@helsinki.fi

Jack Rueter1

thierry.poibeau@ens.fr

Digital Humanities, 2 Finnish, Finno-Ugrian and Scandinavian Studies, 3 Computer Science, University of Helsinki, FI
4Lab. LATTICE, ENS/PSL & CNRS & Univ. Sorbonne nouvelle, FR
Abstract

the local varieties have certainly became endangered. Similar processes of dialect convergence have been reported fromdifferent regions in Europe, although with substantial variation (Auer 2018). In the case of Finnish this has not, however, resulted in merging of the written and spoken standards, but the spoken Finnish has remained, to our day, very
distinct from the written standard. In a late 1950s, a program was set up to document extant spoken dialects, withthe goal of recording 30 hours of speech from each municipality. This work resulted in very large collections of dialectal recordings (Lyytikäinen 1984, 448-449). Many of thesehave been published, and some portion has also been manually normalized. Dataset used is described in more detail in
Section Data and Preprocessing.

We present a novel approach for adapting text written
in standard Finnish to different dialects. We experimentwith character level NMT models both by using a multidialectal and transfer learning approaches. The models
are tested with over 20 different dialects. The results
seem to favor transfer learning, although not strongly
over the multi-dialectal approach. We study the influence dialectal adaptation has on perceived creativity ofcomputer generated poetry. Our results suggest that themore the dialect deviates from the standard Finnish, thelower scores people tend to give on an existing evaluation metric. However, on a word association test, people associate creativity and originality more with dialect
and fluency more with standard Finnish.

Introduction
We present a novel method for adapting text written in standard Finnish to different Finnish dialects. The models developed in this paper have been released in an open-source
Python library1 to boost the limited Finnish NLP resources,
and to encourage both replication of the current study andfurther research in this topic. In addition to the new methodological contribution, we use our models to test the effectthey have on perceived creativity of poems authored by acomputationally creative system.
Finnish language exhibits numerous differences between
colloquial spoken regional varieties and the written standard.
This situation is a result of a long historical development.
Literary Finnish variety known as Modern Finnish developed into its current form in late 19th century, after which
the changes have been mainly in the details (Häkkinen 1994,
16). Many of the changes have been lexical due to technicalinnovations and modernization of the society: orthographic
spelling conventions have largely remained the same. Spoken Finnish, on the other hand, traditionally represents anareally divided dialect continuum, with several sharp boundaries, and many regions of gradual differentiation from one
municipality to another municipality.
Especially in the later parts of 21th century the spoken
varieties have been leveling away from very specific local
dialects, and although regional varieties still exist, most of1khalid.alnajjar@helsinki.fi

Thierry Poibeau4

jack.rueter@helsinki.fi
1

Khalid Alnajjar3

https://github.com/mikahama/murre

Proceedings of the 11th International
Conference on Computational Creativity (ICCC’20)
ISBN: 978-989-54160-2-8

204

Finnish orthography is largely phonemic within the language variety used in that representation, although, as discussed above, the relationship to actual spoken Finnish iscomplicated. Phonemicity of the orthography is still a very
important factor here, as the differences between different
varieties are mainly displaying historically developed differences, and not orthographic particularities that would be essentially random from contemporary point of view. Thereby
the differences between Finnish dialects, spoken Finnish andStandard Finnish are highly systematic and based to historical sound correspondences and sound changes, instead of
more random adaptation of historical spelling conventions
that would be typical for many languages.
Due to the phonemicity of the Finnish writing system,
dialectal differences are also reflected in informal writing.
People speaking a dialect oftentimes also write it as theywould speak it when communicating with friends and family members. This is different from English in that, for example, although Australians and Americans pronounce theword today differently, they would still write the word in
the same way. In Finnish, such a dialectal difference would
result in a different written form as well.
We hyphotesize that dialect increases the perceived valueof computationally created artefacts. Dialectal text is something that people are not expecting from a machine as muchas they would expect standard Finnish. The effect dialect has
on results can be revealing of the shortcomings of evaluation
methods used in the field.

Related Work

Data and Preprocessing

Text adaptation has received some research attention in thepast. The task consists of adapting or transferring a text toa new form that follows a certain style or domain. As theparticular task of dialect adaptation has not received a wide
research interest, we dedicate this section in describing different text adaptation systems in a mode broad sense.
Adaptation of written language to a more spoken language style has previously been tackled as a lexical adaptation problem (Kaji and Kurohashi 2005). They use styleand topic classification to gather data representing written
and spoken language styles, thereafter, they learn the probabilities of lexemes occurring in both categories. This way
they can learn the differences between the spoken and thewritten on a lexical level and use this information for style
adaptation. The difference to our approach is that we approach the problem on a character level rather than lexicallevel. This makes it possible for our approach to deal with
out-of-vocabulary words and to learn inflectional differences
as well without additional modeling.
Poem translation has been tackled from the point of viewof adaptation as well (Ghazvininejad, Choi, and Knight2018). The authors train a neural model to translate French
poetry into English while making the output adapt to specified rhythm and rhyme patterns. They use an FSA (finitestate acceptor) to enforce a desired rhythm and rhyme.
Back-translation is also a viable starting point for style
adaptation (Prabhumoye et al. 2018). They propose amethod consisting of two neural machine translation systems and style generators. They first translate the English input into French and then back again to English in the hopesof reducing the characteristics of the initial style. A stylespecific bi-LSTM model is then used to adapt the back translated sentence to a given style based on gender, political orientation and sentiment.
A recent line of work within the paradigm of computational creativity presents a creative contextual style adaptation in video game dialogs (Hämäläinen and Alnajjar 2019).
They adapt video game dialog to better suit the state of thevideo game character. Their approach works in two steps:
first, they use a machine translation model to paraphrase thesyntax of the sentences in the dialog to increase the varietyof the output. After this, they refill the new syntax with thewords from the dialog and adapt some of the content words
with a word embedding model to fit better the domain dictated by the player’s condition.
A recent style adaptation (Li et al. 2019) learns to separate stylistic information from content information, so that it
can maximize the preservation of the content while adapting
the text to a new style. They propose an encoder-decoder architecture for solving this task and evaluate it on two tasks;
sentiment transfer and formality transfer.
Earlier work on Finnish dialect normalization to standard Finnish has shown that the relationship between spoken
Finnish varieties and literary standard language can be modeled as a character level machine translation task (Partanen,Hämäläinen, and Alnajjar 2019).

Proceedings of the 11th International
Conference on Computational Creativity (ICCC’20)
ISBN: 978-989-54160-2-8

205

We use a corpus called Samples of Spoken Finnish (Institute for the Languages of Finland 2014) for dialect adaptation. This corpus consists of over 51,000 hand annotatedsentences of dialectal Finnish. These sentences have beennormalized on a word level to standard Finnish. This provides us with an ideal parallel data set consisting of dialectal
text and their standard Finnish counterparts.
The corpus was designed so that all main dialects andthe transition varieties would be represented. The last dialect booklet in the series of 50 items was published in
2000, and the creation process was summarised there by
Rekunen (2000). For each location there is one hour of transcribed text from two different speakers. Almost all speakers are born in the 19th century. Transcriptions are done insemi-narrow transcription that captures well the dialect specific particularities, without being phonetically unnecessarily narrow.
The digitally available version of the corpus has a manual normalization for 684,977 tokens. The entire normalized
corpus was used in our experiments.
Dialect
Etelä-Häme
Etelä-Karjala
Etelä-Pohjanmaa
Etelä-Satakunta
Etelä-Savo
Eteläinen Keski-Suomi
Inkerinsuomalaismurteet
Kaakkois-Häme
KainuuKeski-Karjala
Keski-PohjanmaaLänsi-SatakuntaLänsi-UusimaaLänsipohjaLäntinen Keski-Suomi
Peräpohjola
Pohjoinen Keski-Suomi
Pohjoinen Varsinais-Suomi
Pohjois-Häme
Pohjois-Karjala
Pohjois-Pohjanmaa
Pohjois-Satakunta
Pohjois-Savo

Short
EH
EK
EP
ES
ESa
EKS
IS
KH
K
KK
KP
LS
LU
LP
LKS
P
PKS
PVS
PH
PK
PP
PS
PSa

Sentences
1860
813
2684
848
1744
2168
4035
8026
3995
1640
900
1288
1171
1026
857
1913
733
3885
859
4292
1801
2371
2344

Table 1: Dialects and the number of sentences in each dialect
in the corpus
Despite the attempts of the authors of the corpus to include all dialects, the dialects are not equally represented in
the corpus. One reason for this is certainly the different sizesof the dialect areas, and the variation introduced by different
speech rates of individual speakers. The difference in thenumber of sentences per dialect can be seen in Table 1. We
do not consider this uneven distribution to be a problem, as
it is mainly a feature of this dataset, but we have paid at-

tention to these differences in data splitting. In order to get
proportionally even numbers of each dialect in the different
data sets, we split the sentences of each dialect into training(70%), validation (15%) and testing (15%) the split is done
after shuffling the data. The same split is used throughout
this paper.
The dialectal data contains non-standard annotations thatare meant to capture phonetic and prosodic features that areusually not represented in the writing. These include the useof the acute accent to represent stress, superscripted characters, IPA characters and others. We go through all charactersin the dialectal sentences that do not occur in the normalizations, i.e all characters that are not part of the Finnish
alphabets and ordinary punctuation characters. We remove
all annotations that mark prosodic features as these are notusually expressed in writing. This is done entirely manually
as sometimes the annotations are additional characters thatcan be entirely removed and sometimes the annotations areadded to vowels and consonants, in which case they form
new Unicode characters and need to be replaced with their
non-annotated counterparts.

Automatic Dialect Adaptation

In order to adapt text written in standard Finnish to dialects,
we train several different models on the data set. As a character level sequence-to-sequence neural machine translation(NMT) approach has been proven successful in the past forthe opposite problem of normalization of dialectal or historical language variant to the standard language (see (Bollmann
2019; Hämäläinen et al. 2019; Veliz, De Clercq, and Hoste
2019; Hämäläinen and Hengchen 2019)), we approach the
problem form a similar character based methodology. The
advantage of character level models to word level models istheir adaptability to out of vocabulary words; a requirement
which needs to be satisfied for our experiments to be successful. In practice, this means splitting the words into characters separated by white-spaces and marking word boundaries with a special character, which is underscore ( ) in our
approach.
In NMT, language flags have been used in the past to train
multi-lingual models (Johnson et al. 2017). The idea is thatthe model can benefit from the information in multiple languages when predicting the translation for a particular language a expressed by a language specific flag given to thesystem. We train one model with all the dialect data, appending a dialect flag to the source side. The model will thenlearn to use the flag when adapting the standard Finnish text
the the desired dialect.
Additionally, we train one model without any flags or dialectal cues. This model is trained to predict from standard
Finnish to dialectal text (without any specification in termsof the dialect). This model serves two purposes, firstly if it
performs poorly on individual dialects, it means that there isa considerable distance between each dialect so that a single
model that adapts text to a generic dialect cannot sufficiently
capture all of the dialects. Secondly, this model is used as astarting point for dialect specific transfer learning.
We use the generic model without flags for training dialectspecific models. We do this by freezing the first layer of theProceedings of the 11th International
Conference on Computational Creativity (ICCC’20)
ISBN: 978-989-54160-2-8

206

encoder, as the encoder only sees standard Finnish, it does
not require any further training. Then we train the dialectspecific models from the generic model by continuing thetraining with only the training and validation data specific toa given dialect. We train each dialect specific model in thedescribed transfer learning fashion for an additional 20,000
steps.
Our models are recurrent neural networks. The architecture consists of two encoding layers and two decoding layers
and the general global attention model (Luong, Pham, andManning 2015). We train the models by using the OpenNMT Python package (Klein et al. 2017) with otherwise thedefault settings. The model with flags and the generic model
are trained for 100,000 steps. We train the models by providing chunks of three words at a time as opposed to training one word or whole sentence at a time, as a chunk of threewords has been suggested to be more effective in a characterlevel text normalization task (Partanen, Hämäläinen, and Alnajjar 2019).
Table 2 shows an example of the sequences used for training. The model receiving the dialect flag has the name of thedialect appended to the beginning of the source data, whereas the generic model has no additional information apartfrom the character sequences. The dialect specific transfer
learning models are also trained without an additional flag,but rather the exposure solely to the dialect specific data is
considered sufficient for the model to better learn the desired
dialect.

Results and Evaluation
In this section, we present the results of the dialect adaptation models on different dialects. We use a commonly usedmetric called word error rate (WER) and compare the dialect adaptations of the test sets of each dialect to the gold
standard. WER is calculated for each sentence by using thefollowing formula:
S+D+I
(1)
S+D+C
WER is derived from Levenshtein edit distance (Levenshtein 1966) as a better measurement for calculating wordlevel errors. It takes into account the number of deletions
D, substitutions S, insertions I and the number of correct
words C.
The results are shown in Tables 3 and 4. On the verticalaxis are the models. Flags represents the results of the modelthat was trained with initial tokens indicating the desired dialect the text should be adapted in. No flags is the modeltrained without any dialectal information, and the rest of themodels are dialect specific transfer learning models trainedon the no flags model.
The results are to be interpreted as the lower the better,i.e. the lower the WER, the closer the output is to the gold
dialect data in a given dialect. These results indicate thatthe no flag model does not get the best results for any of thedialects, which is to be expected, as if it reached to good results, that would indicate that the dialects do not differ fromeach other. Interestingly, we can observe that the transfer
W ER =

FlagsNo flags

SourceInkerinsuomalaismurteet m i n ä k u n n ä i n
m i n ä k u n n ä i nTargetm i e k o n ä i n
m i e k o n ä i n

Table 2: Example of the training data. The sentence reads ”when I saw” in English
model
Flags
No flags
EH
EK
EP
ES
ESa
EKS
IS
KH
K
KK
KP
LS
LU
LP
LKS
P
PKS
PVS
PH
PK
PP
PS
PSa

EH
24.37
38.87
24.21
48.65
38.8
34.36
46.06
44.3
52.09
43.98
42.59
54.1
35.58
36.05
38.45
40.58
33.25
39.05
45.73
50.34
31.26
44.14
34.73
28.42
43.12

EK
19.8
36.21
43.6
19.28
50.37
44.81
32.28
37.3
28.4
38.34
45.05
30
43.94
39.56
45.07
44.55
40.03
44.38
43.03
41.51
44.72
44.33
44.38
46.29
40.86

EP
25.13
41.98
37.64
52.63
24.9
41.49
49.5
47.06
55.13
47.75
45.11
55.59
38.58
42.77
44.24
42.07
37.48
40.83
48.96
52.91
38.38
47.18
37.87
35.51
47.81

ES
28.09
42.16
35.77
47.57
42.3
29.03
50.38
51.05
49.53
47.66
50.11
51.52
40.2
35.73
39.17
41.94
39.88
42.72
51.9
44.13
37.56
50.83
41.85
36.62
49.71

ESa
27.22
37.71
46.83
35.69
49.2
49.35
26.81
34.15
41.52
45.46
39.79
40.52
44.54
46.21
51.68
46.1
39.42
45.09
36.41
50.96
44.61
36.98
43.24
46.96
34.74

EKS
25.19
37.35
42.98
39.94
46.3
47.8
32.43
25.07
44.57
43.23
35.97
43.12
41.53
45.34
51.03
44.94
35.24
42.25
33.39
53.29
39.4
37.08
39.46
42.46
33.12

IS
21.09
39.38
51.51
31.86
54.47
50.05
42.01
45.56
19.69
41.16
50.56
29.21
51.03
47.7
47.35
49.32
49.09
50.06
48.55
44.48
52.07
46.76
52
53.15
46.47

KH
28.73
39.03
41.05
42.97
46.39
45.56
44.26
42.97
41.13
28.43
48.17
43.65
45.84
43.4
41.04
46.35
42.59
46.11
47.2
46.03
42.19
46.09
45.12
41.84
44.95

K
25.56
37.05
42.38
47.14
44.71
47.74
38.4
36.5
50.24
47.88
25.56
49.74
39.26
46.73
51.14
44.42
33.99
41.1
33.37
55.99
38.51
33.51
36.91
42.31
32.01

KK
24.59
42.43
53.26
33.13
55.68
51.16
40.32
42.84
29.14
44.36
49.34
24.87
52.04
48.19
49.54
50.71
49.82
51.14
46.73
46.38
52.73
46.5
52.84
53.84
45.44

KP
22.51
39.08
38.95
49.76
39.21
38.02
45.88
42.65
52.26
47.9
40.89
53.52
22.51
40.76
46.74
35
32.79
35.12
43.46
53.35
35.43
42.58
27.12
36.63
45.28

LS
30.49
42.3
37.53
45.51
44.24
37.12
47.9
47.86
46.65
45.76
49.63
50.21
44.32
29.71
38.97
44.57
42.64
44.34
52.63
43.09
40.82
51.05
43.25
38.6
51.15

Table 3: WER for the different models for dialects from Etelä-Häme to Länsi-Satakunta
learning method gives the best scores for almost all the dialect, except for Etelä-Satakunta (ES), Keski-Karjala (KK),
Keski-Pohjanmaa (KP), Pohjois-Karjala (PK) and PohjoisSatakunta (PS), for which the model with flags gives the best
results. Both methods are equally good for Pohjois-Häme(PH). All in all, the difference between the two methods israther small in the WER. An example of the dialectal adaptation can be seen in Table 5.
Based on these results it is difficult to suggest one method
over the other as both of them are capable of reaching to
the best results on different dialects. On the practical side,
the model with dialectal flags trains faster and requires lesscomputational resources, as the model is trained once only
and it works for all the dialects immediately, where as transfer learning has to be done for each dialect individually after
training a generic model.
Evaluation of the models with and without dialectal flags
shows that especially in word forms that are highly divergent in the dialect, it is almost impossible for the model topredict the correct result that is in the test set. This doesn’tmean that the model’s output would necessarily be entirely
incorrect, as the result may still be perfectly valid dialectal
representation, it just is in a different variety.
There are also numerous examples of features that are invariation also within one dialect. In these cases the model
may produce a form different from that in the specific row
of a test set. These kind of problems are particularly prominent in examples where the dialectal transcription contains
prosodic phenomena at the word boundary level. Since themodel starts the prediction from standard Finnish input, itcannot have any knowledge about specific prosodic featuresof the individual examples in test data. Some phonologi-

Proceedings of the 11th International
Conference on Computational Creativity (ICCC’20)
ISBN: 978-989-54160-2-8

207

cal features such as assimilation of nasals seem to be overgeneralized by the model, and also in this case it would be
impossible for the model to predict the instances where suchphenomena does not take place due to particularly careful
pronunciation.
Another interesting feature of the model is that it seems
to be able to generalize its predictions into unseen words,
as long as they exhibit morphology common for the training
data. There are, however, instances of clearly contemporary
word types, such as recent international loans, that have general shape and phonotactics that are entirely absent from thetraining data. The problems caused by this are somewhat
mitigated by fact that in many cases the standard Finnish
word can be left intact, and it will pass within the dialectal
text relatively well.
This has a consequence that the scores reported here
are possibly slightly worse than the model’s true abilities.
The resulting dialectal text can still be very accurate andclosely approximate the actual dialect, although the prediction would slightly differ from the test instances. At thesame time if the model slips into predicted text some literary Finnish forms, the result is still perfectly understandable, and also in real use the dialects would rarely be used inentire isolation from the standard language.
It must also be taken into account that only either a native dialect speaker or an advanced specialist in Finnish dialectology can reliably detect minute disfluencies in dialectal predictions, especially when the error is introduced bya form of other dialect. Similarly it would be very uncommon to have such knowledge about all the Finnish dialects
the model operates on. After this careful examination of themodels, we proceed to the generation of dialectal poems andmodelFlags
No flags
EH
EK
EP
ES
ESa
EKS
IS
KH
K
KK
KP
LS
LU
LP
LKS
P
PKS
PVS
PH
PK
PP
PS
PSa

LU
27.87
43.49
39.9
50.59
47.04
43.01
53.26
52.05
48.72
44.26
52.71
51.83
48.5
42.57
25.9
49.41
45.39
47.29
55.67
46.42
44.15
53.77
48.43
42.17
52.19

LP
20.02
37.1
35.63
45.08
37.78
36.6
40.5
40.5
47.29
44.17
39.03
48.19
27.67
36.9
43.04
19.57
33.2
28
41.86
49.26
33.2
41.01
30.77
35.18
42.13

LKS
21.89
35.06
33.65
46.23
39.13
40.35
38.89
35.72
49.67
43.45
34.47
50.94
34.21
39.88
44.97
38.2
21.41
35.54
39.87
54.99
31.47
38.61
32.43
32.03
36.28

P
27.53
38.35
39.92
46.75
41.56
42.12
43.85
41.94
48.39
46.91
39.75
49.37
35.92
42.58
45.66
32.23
34.97
26.81
42.62
52.31
36.83
42.59
34.85
38.9
42.45

PKS
28.73
40.54
48.42
46.03
52.06
52.27
37.36
36.11
49.59
49.09
35.07
49.41
43.07
51.71
54.87
47.75
40.88
46.23
28.65
57.36
44.44
38.34
43.11
47.14
35.29

PVS
32.4
49.19
51.05
47.19
56.16
47.34
54.04
55.63
45.74
49.42
58.24
48.7
56.07
47.31
43.76
55.87
55.13
56.06
57.68
31.67
55.22
57.18
57.03
54.2
56.52

PH
20.03
34.9
27.61
45.62
33.28
34.46
39.56
38.27
49.89
42.14
35.52
50.84
30.26
34.74
40.63
35.04
26.9
33.45
39.28
52.13
20.03
37.99
28.9
26.49
38.29

PK
27.15
36.44
41.9
43.82
44.64
46.08
35.68
37.34
46.45
45.54
33.57
45.66
40.42
46.31
49.41
43.57
36.47
40.98
35.8
52.69
38.14
27.28
38.09
41.34
32.8

PP
21.51
35.12
32.54
45.84
34.32
37.09
40.02
38.33
48.51
44.09
35.43
50.27
25.17
38.71
43.76
33.96
28.22
32.7
38.08
51.34
30.76
37.87
21.04
31.54
37.67

PS
21.56
38.54
27.46
51.85
35.23
35.13
46.65
42.99
54.29
43.86
41.77
55.14
35.28
36.28
40.15
37.92
31.57
37.82
46.01
53.44
28.94
46.02
34.2
22
43.96

PSa
27.6
37.86
43.91
43.21
46.35
48.23
35.55
35.35
46.18
44.09
33.67
45.86
42.05
47.7
50.93
45.85
36.37
43.27
33.67
53.39
40.5
34.98
39.94
42.98
27.24

Table 4: WER for the different models for dialects from Länsi-Uusimaa to Pohjois-Savo
their further evaluation by native Finnish speakers.

to pien ? adaptation. For this reason, we only consider as
many words as in the original chunks when doing the dialectal adaptation.

Effect on Perceived Creativity
In this section, we apply the dialect adaptation trained in
the earlier sections to text written in standard Finnish. We
are interested in seeing what the effect of the automatically
adapted dialect is on computer generated text. We use an
existing Finnish poem generator (Hämäläinen 2018) that
produces standard Finnish (SF) text as it relies heavily on
hand defined syntactic structures that are filled with lemmatized words that are inflected with a normative Finnish morphological generator by using a tool called Syntax Maker(Hämäläinen and Rueter 2018). We use this generator to
generate 10 different poems.
The poems generated by the system are then adapted todialects with the models we elaborated in this paper. As
the number of different dialects is extensive and conducting
human questionnaire with such a myriad of dialects is not
feasible, we limit our study to three dialects. We pick EteläKarjala (EK) and Inkerinsuomalaismurteet (IS) dialects because they are the best performing ones in terms of WER andPohjoinen Varsinais-Suomi (PVS) dialect as it is the worst
performing in terms of WER. For this study, we use the dialect specific models tuned with transfer learning.
A qualitative look at the predictions revealed that the dialectal models have a tendency of over generating when aword chunk has less than three words. The models tend to
predict one or two additional words in such cases, however,if the chunk contains three words, the models do not over nor
under generate words. Fortunately this is easy to overcome
by ensuring that only as many dialectal words are considered from the prediction as there were in the chunk written
in standard Finnish. For instance olen vanha (I am old) gets
predicted in IS as olev vanha a. The first two words are correctly adapted to the dialect, while the third word a is aninvention by the model. However, the models do not systematically predict too many words as in pieni ? (small?)Proceedings of the 11th International
Conference on Computational Creativity (ICCC’20)
ISBN: 978-989-54160-2-8

208

Replicating the Poem Generator Evaluation
In our first experiment, we replicate the poem generator
evaluation that was used to evaluate the Finnish poem generator used in this experiment. We are interested in seeing
whether dialectal adaptation has an effect on the evaluation
results of the creative system. They evaluated their system
based on the evaluation questions initially elaborated in astudy on an earlier Finnish poem generator (Toivanen et al.
2012). The first evaluation question is a binary one Is the
text a poem?. The rest of the evaluation questions are asked
on a 5-point Likert scale:
1. How typical is the text as a poem?
2. How understandable is it?
3. How good is the language?
4. Does the text evoke mental images?
5. Does the text evoke emotions?
6. How much do you like the text?
The subjects are not told that they are to read poetry northat they are reading fully computer generated and dialectally adapted text. We conduct dialectal adaptation to the 10
generated poems to the three different dialects, this meansthat there are altogether four variants of each poem, one instandard Finnish, and three in dialects. We produce the questionnaires automatically in such a fashion that each questionnaire has the 10 different poems shuffled in random order each time. The variants of each poem are picked randomly so that each questionnaire has randomly picked variant for each of the poems. Every questionnaire contains poems from all of the different variant types, but none of them
contains the same poem more than once. Each questionnaire
is unique in the order and combination of the variants. We

SFhimo on palo,
se syttyy herkästitaas intona se kokoaa
milloin into on eloisa?
näemmekö me,ennen kuin into jää pois?
mikäli innot pysyisivät,
sinä huomaisit innon
minä alan maksamaan innon
olenko liiallinen?

EKhimo om palos,
se syttyy herkästtaas intonna se kokovaa
millo into on elosa?
näämmekö met,enne ku into jää pois?
mikäli innot pysysiit,
sie huomasit inno
mie alan maksamaa inno
olenko siialli?

PVS
himo om palo,
se sytty herkästitaas inton se kokko
millon innoo on elosa?
näämekö me,ennen ku into jää pois?
mikäl innop pysysivät,siä huamasit innomää ala maksaman inno
olenko liialline?

IShimo om palloo,
se syttyy herkästtoas inton se kokohoa
millon into on eloisa?
neämmäks meä,ennen ku into jää pois?
mikält innot pysysiit,
sie huomaisit inno
mie ala maksamaa inno
olenko liialine?

Translation
desire is a fire,it gets easily ignited
again, as an ardor it shall risewhen is ardor vivacious?
will we see
before ardor disappears?
if ardors stayed,
you would notice the ardor
I will start paying for the ardor
Am I extravagant?

Table 5: An example poem generated in standard Finnish and its dialectal adaptations to three different dialects
introduce all this randomness to reduce constant bias thatmight otherwise be present if the poem variants were always
presented in the same order.
We print out the questionnaires and recruit people nativein Finnish in the university campus. We recruit 20 people toevaluate the questionnaires each of which consisting of 10
poems. This means that each variant of a poem is evaluatedby five different people.
Table 6 shows the results from this experiment, however
some evaluators did not complete the task for all poems intheir pile2 . Interestingly, the results drop on all the parameters when the poems are adapted into the different dialects
in question. The best performing dialect in the experiment
was the Etelä-Karjala dialect, and the worst performing one
was the Pohjoinen Varsinais-Suomi dialect all though it got
the exact same average scores with Inkerinsuomalaismurteet
on the last three questions. Now these results are not to beinterpreted as that dialectal poems would always get worse
results, as we only used a handful of dialects form the possibilities. However, the results indicate an interesting finding that something as superficial as a dialect can affect theresults. It is to be noted that the dialectal adaptation only
alters the words to be more dialectal, it does not substitutethe words with new ones, nor does it alter their order.
In order to better understand why the dialects were ranked
in this order, we compare the dialectal poems to the standard
Finnish poems automatically by calculating WER. These
WERs should not be understood as ”error rates” since weare not comparing the dialects to a gold standard, but rather
to the standard Finnish poems. The idea is that the higher
the WER, the more they differ from the standard. Table 7
shows the results of this experiment. The results seem to bein line with the human evaluation results; the further away
the dialect is from the standard Finnish, the lower it scores in
the human evaluation. This is a potential indication of familiarity bias; people tend to prefer the more familiar language
variety.

Word Association Test
In the second experiment, we are interested in seeing how
people associate words when they are presented with a standard Finnish version and a dialectally adapted variant of the2

The data is based on 47 observations for SF, 46 for EK, 43 for
PVS and 49 for IS out of the maximum of 50.

Proceedings of the 11th International
Conference on Computational Creativity (ICCC’20)
ISBN: 978-989-54160-2-8

209

same poem. The two poems are presented on the same page,labeled as A and B. The order is randomized again, whichmeans that both the order of poems in the questionnarie andwhether the dialectal one is A or B is randomized. This isdone again to reduce bias in the results that might be causedby always maintaining the same order. The concepts westudy are the following:• emotive• original• creative• poem-like• artificial• fluent

The subjects are asked to associate each concept with Aor B, one of which is the dialectal and the other the standard
Finnish version of the same poem. We use the same dialects
as before, but which dialect gets used is not controlled inthis experiment. We divide each questionnaire of 10 poems
into piles of two to reduce the work load on each annotator
as each poem is presented in two different variant forms.
This way, we recruit altogether 10 different people for thistask, again native speakers from the university campus. Eachpoem with a dialectal variant gets annotated by five different
people.
Table 8 shows the results of this experiment. Some ofthe people did not answer to all questions for some poems.
This is reflected in the no answer column. The results indicate that the standard Finnish variant poems were considered considerably more fluent than the dialectal poems, andslightly more emotive and artificial. The dialectal poems
were considered considerably more original and creative,
and slightly more poem-like.
It is interesting that while dialectal poems can get clearly
better results on some parameters on this experiment, they
still scored lower on all the parameters in the first experiment. This potentially highlights a more general problem
on evaluation in the field of computational creativity, as results are heavily dependent on the metric that happened to bechosen. The problems arising from this ”ad hoc” evaluation
practice are also discussed by (Lamb, Brown, and Clarke2018).

SF
EK
IS
PVS

Poem
%
87.2%
82.6%
77.6%
77.0%

M
2.85
2.5
2.69
2.51

Typical
Mo Me
4
3
2
2
2
3
2
2

Understandable
M Mo Me
3.62
4
4
3
4
3
2.90 3, 4 3
2.80
2
3

Language
M Mo Me
3.51 44
2.87 33
2.78 23
2.58 2
3

Mental images
M Mo Me
3.57 44
3.26 43
3.27 43
3.27 4
3

Emotions
M Mo Me
2.94 23
2.67 22
2.86 23
2.86 2
3

M
3.02
2.70
2.61
2.61

Liking
Mo Me
4
3
2
3
3
3
3
3

Table 6: Results form the first human evaluation. Mean, mode and median are reported for the questions on Likert-scale

WER

EK
34.38

IS
43.41

PVS
54.69

Table 7: The distance of the dialectal poems form the original poem written in standard Finnish
emotiveoriginal
creative
poem-likeartificial
fluent

SF
48%
40%
32%
46%
50%
74%

Dialect
46%
60%
64%
50%
46%
24%

No answer
6%
0%
4%
4%
4%
2%

Table 8: Results of the second experiment with human annotators

Conclusions
We have presented our work on automatic dialect adaptation by using a character-level NMT approach. Based on
our automatic evaluation, both the transfer learning method
and a multi-dialectal model with flags can achieve the best
results in different dialects. The transfer learning method,
however, receives the highest scores on most of the dialects.
Nevertheless, the difference in WERs of the two methods
is generally small, therefore it is not possible to clearly recommend one over another to be used for different characterlevel data sets. If the decision is based on the computational
power used, then the multi-dialectal model with flags should
be used as it only needs to be trained once and it can handle
all the dialects.
The dialect adaptation models elaborated in this paper
have been made publicly available as an open-source Python
library3 . This not only makes the replication of the results easier but also makes it possible to apply these uniqueFinnish NLP tools on other related research or tasks outsideof the academia as well.
Our study shows that automatic dialect adaptation has aclear impact to how different attributes of the text are perceived. In the first experiment that was based on existingevaluation questions, a negative impact was found as thescores dropped on all the metrics in comparison to the original standard Finnish poem. However, when inspecting the
distance the dialects have from the standard Finnish, we noticed that the further away the dialect is form the standard,
the lower it scores.
We believe that the low scores might be an indication of3

https://github.com/mikahama/murre

Proceedings of the 11th International
Conference on Computational Creativity (ICCC’20)
ISBN: 978-989-54160-2-8

210

familiarity bias, which means that people have a tendency
of preferring things they are more familiar with. Especiallysince the evaluation was conducted in a region in Finlandwith a high number of migrants from different parts of thecountry. This leads to a situation where the most familiar
language variety for everyone regardless of their dialectal
background is the standard Finnish variety. Also, as the dialectal data used in our model originates from the Finnish
speakers born in the 19th century, it remains possible thatthe poems were transformed into a variety not entirely familiar to the individuals who participated into our survey. In
the upcoming research it is necessary to investigate the perceptions of wider demographics, taking into account larger
areal representation.
Based on our results, it is too early to generalize that familiarity bias is a problem in evaluation of computationallycreative systems. However, it is an important aspect to take
into consideration in the future research. We are interested
in testing this particular bias out in the future in a more controlled fashion. Nevertheless, the fact that a variable, such as
dialect that is never controlled in the computational creativity evaluations, has a clear effect on the evaluation results,
raises a real question about the validity of such evaluation
methods. As abstract questions on 5-point Likert scale are
a commonly used evaluation methodology, the question of
narrowing down the unexpected variables, such as dialect,that affect the evaluation results positively or negatively isvital for the progress in the field in terms of comparabilityof results from different systems.
Even though the initial hypothesis we had on dialects increasing the perceived value of computationally created artefacts was proven wrong by the first experiment, the second
experiment showed that dialects can indeed have a positiveeffect on the results as well, in terms of perceived creativity and originality. This finding is also troublesome form
the point of view of computational creativity evaluation in alarger context. Our dialect adaptation system is by no means
designed to exhibit any creative behavior of its own, yet people are more prone to associating the concept creativity withdialectally adapted poetry.
The results of the first and second experiment give a very
different picture of the impact dialect adaptation has on perceived creativity. This calls for a more thorough researchon the effect different evaluation practices have on the results of a creative system. Is the difference in results fully
attributable to subjectivity in the task, what was asked on
how it was asked. Does making people pick between two(dialectal and standard Finnish in our case) introduce a bias
not present when people rate the poems individually? It isimportant these questions be systematically addressed in the
future research.

Acknowledgments
Thierry Poibeau is partly supported by a PRAIRIE 3IA Institute fellowship (”Investissements d’avenir” program, reference ANR-19-P3IA-0001).

References
Auer, P. 2018. Dialect change in europe–leveling and convergence. The Handbook of Dialectology 159–76.
Bollmann, M. 2019. A large-scale comparison of historicaltext normalization systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3885–3898. Minneapolis, Minnesota: Association for Computational Linguistics.
Ghazvininejad, M.; Choi, Y.; and Knight, K. 2018. Neural
poetry translation. In Proceedings of the 2018 Conferenceof the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 67–71.
Häkkinen, K. 1994. Agricolasta nykykieleen: suomen kirjakielen historia. Söderström.
Hämäläinen, M., and Alnajjar, K. 2019. Creative contextual dialog adaptation in an open world rpg. In Proceedingsof the 14th International Conference on the Foundations ofDigital Games, 1–7.
Hämäläinen, M., and Hengchen, S. 2019. From the paft
to the fiiture: a fully automatic nmt and word embeddings
method for ocr post-correction. In Recent Advances in Natural Language Processing, 432–437. INCOMA.
Hämäläinen, M., and Rueter, J. 2018. Development of anopen source natural language generation tool for finnish. In
Proceedings of the Fourth International Workshop on Computational Linguistics of Uralic Languages, 51–58.
Hämäläinen, M.; Säily, T.; Rueter, J.; Tiedemann, J.; andMäkelä, E. 2019. Revisiting NMT for normalization of early
English letters. In Proceedings of the 3rd Joint SIGHUM
Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, 71–75.
Minneapolis, USA: Association for Computational Linguistics.
Hämäläinen, M. 2018. Harnessing nlg to create finnish
poetry automatically. In International Conference on Computational Creativity, 9–15. Association for Computational
Creativity (ACC).
Institute for the Languages of Finland. 2014. Suomen kielen näytteitä - Samples of Spoken Finnish [online-corpus],
version 1.0. http://urn.fi/urn:nbn:fi:lb-201407141.
Johnson, M.; Schuster, M.; Le, Q. V.; Krikun, M.; Wu, Y.;
Chen, Z.; Thorat, N.; Viégas, F.; Wattenberg, M.; Corrado,G.; Hughes, M.; and Dean, J. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot

Proceedings of the 11th International
Conference on Computational Creativity (ICCC’20)
ISBN: 978-989-54160-2-8211translation. Transactions of the Association for Computational Linguistics 5:339–351.
Kaji, N., and Kurohashi, S. 2005. Lexical choice via topic
adaptation for paraphrasing written language to spoken language. In International Conference on Natural Language
Processing, 981–992. Springer.
Klein, G.; Kim, Y.; Deng, Y.; Senellart, J.; and Rush, A. M.
2017. OpenNMT: Open-Source Toolkit for Neural MachineTranslation. In Proc. ACL.
Lamb, C.; Brown, D. G.; and Clarke, C. L. 2018. Evaluating
computational creativity: An interdisciplinary tutorial. ACM
Computing Surveys (CSUR) 51(2):1–34.
Levenshtein, V. I. 1966. Binary codes capable of correcting
deletions, insertions, and reversals. Soviets Physics Doklady
10(8):707–710.
Li, D.; Zhang, Y.; Gan, Z.; Cheng, Y.; Brockett, C.; Sun, M.T.; and Dolan, B. 2019. Domain adaptive text style transfer.
arXiv preprint arXiv:1908.09395.
Luong, M.-T.; Pham, H.; and Manning, C. D. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025.
Lyytikäinen, E. 1984. Suomen kielen nauhoitearkiston
neljännesvuosisata. Virittäjä 88(4):448–448.
Partanen, N.; Hämäläinen, M.; and Alnajjar, K. 2019. Dialect text normalization to normative standard finnish. In
Proceedings of the 5th Workshop on Noisy User-generated
Text (W-NUT 2019), 141–146.
Prabhumoye, S.; Tsvetkov, Y.; Salakhutdinov, R.; andBlack, A. W. 2018. Style transfer through back-translation.
In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 866–876. Melbourne, Australia: Association for
Computational Linguistics.
Rekunen, J. 2000. Suomen kielen näytteitä 50. Kotimaistenkielten tutkimuskeskus.
Toivanen, J.; Toivonen, H.; Valitutti, A.; and Gross, O. 2012.
Corpus-Based Generation of Content and Form in Poetry. In
Proceedings of the Third International Conference on Computational Creativity.
Veliz, C. M.; De Clercq, O.; and Hoste, V. 2019. Benefits ofdata augmentation for nmt-based text normalization of usergenerated content. In Proceedings of the 5th Workshop on
Noisy User-generated Text (W-NUT 2019), 275–285.