# Proceedings of the 5th Conference on CMC and Social Media Corpora for the Humanities (cmccorpora17)

Egon W. Stemle, Ciara R. Wigham

► To cite this version:

Egon W. Stemle, Ciara R. Wigham. Proceedings of the 5th Conference on CMC and Social Media Corpora for the Humanities (cmccorpora17). 5th Conference on CMC and Social Media Corpora for the Humanities (cmccorpora17), Oct 2017, Bolzano, Italy. cmc-corpora conference series, 2017, cmc-corpora pre-conference proceedings, 10.5281/zenodo.1040714. hal-01614310v2

HAL Id: hal-01614310

https://hal.science/hal-01614310v2

Submitted on 7 Nov 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Conference Proceedings

Proceedings of the 5th Conference on CMC and Social
Media Corpora for the Humanities (cmccorpora17)

3-4 October 2017, Eurac Research, Italy

Editors

Egon W. Stemle

Ciara R. Wigham
Proceedings of the 5th Conference on CMC and Social Media Corpora for the
Humanities (cmccorpora17)

Editors: Egon W. Stemle, Ciara R. Wigham

Bolzano, 2017
Second edition

Conference web site: https://cmc-corpora2017.eurac.edu

This publication is available from:

https://cmc-corpora2017.eurac.edu/proceedings/

https://hal.archives-ouvertes.fr/hal-01614310

DOI 10.5281/zenodo.1040713

This publication was supported by

eurac
research

<figure><img src="image_2.png" /></figure>

cmc-corpora Conference Series

<figure><img src="image_3.png" /></figure>

CLARIN

Common Language Resources and
Technology Infrastructure

This publication was compiled using LuaLATEX. The LATEX template is based on KOMA script. The individual contributions are based on style files in LATEX, MSWord, or OpenDocument format. These style files are modifications of the style files for cmccorpora16 which are modifications of the Language Resources and Evaluation Conference (LREC) 2016 style files. The template and the style files are available online: https://github.com/cmc-corpora/.

This work is licensed under a Creative Commons “Attribution 4.0 International”
license.

<figure><img src="image_4.png" /></figure>
# Preface

This volume presents the proceedings of the 5th edition of the annual conference series on CMC and Social Media Corpora for the Humanities (cmc-corpora2017). This conference series is dedicated to the collection, annotation, processing, and exploitation of corpora of computer-mediated communication (CMC) and social media for research in the humanities. The annual event brings together language-centered research on CMC and social media in linguistics, philologies, communication sciences, media and social sciences with research questions from the fields of corpus and computational linguistics, language technology, text technology, and machine learning.

The 5th Conference on CMC and Social Media Corpora for the Humanities was held at Eurac Research on October, 4th and 5th, in Bolzano, Italy. This volume contains extended abstracts of the invited talks, papers, and extended abstracts of posters presented at the event. The conference attracted 26 valid submissions. Each submission was reviewed by at least two members of the scientific committee. This committee decided to accept 16 papers and 8 posters of which 14 papers and 3 posters were presented at the conference. The programme also includes three invited talks: two keynote talks by *Aivars Glaznieks* (Eurac Research, Italy) and *A. Seza Doğruöz* (Independent researcher) and an invited talk on the Common Language Resources and Technology Infrastructure (CLARIN) given by *Darja Fišer*, the CLARIN ERIC Director of User Involvement.

We wish to thank all colleagues who have contributed to the conference and to this volume with their papers, posters, and invited talks. Thanks also to all members of the scientific committee and to the local coordinating committee without whom the conference would not have taken place. Whilst previous events in the conference cycle were held in Dortmund, Germany (2013 and 2014), Rennes, France (2015) and Ljubljana, Slovenia (2016), we hope that the Bolzano 2017 conference will mark another step towards a lively exchange of approaches, expertise, resources, tools, and best practices between researchers and existing networks in the field and pave the ground for future standards in building and using CMC and social media corpora for research in the humanities.

We look forward to welcoming colleagues at the 2018 conference to be held in Antwerp, Belgium to continue the scientific exchange.

September 30, 2017
Bolzano/Bozen

Ciara R. Wigham, Université Clermont Auvergne (France)
Egon W. Stemle, Eurac Research (Italy)

Chair of the Scientific Committee and chair of the Organizing Committee.
Table of Contents

<table><tr><td>Preface</td><td>iii</td></tr><tr><td>Committees</td><td>vi</td></tr><tr><td>Invited Talks</td><td>1</td></tr><tr><td>Think Global, Write Local – Patterns of Writing Dialect on SNS</td><td>2</td></tr><tr><td>Aivars Glaznieks</td><td></td></tr><tr><td>Small vs. Big Data in Language Research: Challenges and Opportunities</td><td>3</td></tr><tr><td>A. Seza Doğruöz</td><td></td></tr><tr><td>CLARIN Survey of CMC Resources and Tools</td><td>4</td></tr><tr><td>Darja Fišer</td><td></td></tr><tr><td>Papers</td><td>5</td></tr><tr><td>The Impact of WhatsApp on Dutch Youths' School Writing</td><td>6</td></tr><tr><td>Lieke Verheijen and Wilbert Spooren</td><td></td></tr><tr><td>Modeling Non-Standard Language Use in Adolescents' CMC: The Impact and Interaction of Age, Gender and Education</td><td>11</td></tr><tr><td>Lisa Hilte, Reinhild Vandekerckhove and Walter Daelemans</td><td></td></tr><tr><td>Investigating Interaction Signs across Genres, Modes and Languages: The Example of OKAY</td><td>16</td></tr><tr><td>Laura Herzberg and Angelika Storrer</td><td></td></tr><tr><td>Anonymisation of the Dortmund Chat Corpus 2.1</td><td>21</td></tr><tr><td>Harald Lüngen, Michael Beißwenger, Laura Herzberg and Cathrin Pichler</td><td></td></tr><tr><td>Emoticons as multifunctional and pragmatic Resources: a corpus-based Study on Twitter</td><td>25</td></tr><tr><td>Stefania Spina</td><td></td></tr><tr><td>Corpus-Based Analysis of Demonyms in Slovene Twitter</td><td>30</td></tr><tr><td>Taja Kuzman and Darja Fišer</td><td></td></tr><tr><td>European Language Ecology and Bilingualism with English on Twitter</td><td>35</td></tr><tr><td>Steven Coats</td><td></td></tr><tr><td>Reliable Part-of-Speech Tagging of Low-frequency Phenomena in the Social Media Domain</td><td>39</td></tr><tr><td>Tobias Horsmann, Michael Beißwenger and Torsten Zesch</td><td></td></tr><tr><td>Developing a protocol for collecting data in Higher Education: assessing natural language metadata for a Databank of Oral Teletandem Interactions.</td><td>44</td></tr><tr><td>Paola Leone</td><td></td></tr><tr><td>The #Idéo2017 Platform</td><td>46</td></tr><tr><td>Julien Longhi, Claudia Marinica, Nader Hassine, Abdulhafiz Alkhouli and Boris Borzic</td><td></td></tr><tr><td>Connecting Resources: Which Issues Have to be Solved to Integrate CMC Corpora from Heterogeneous Sources and for Different Languages?</td><td>52</td></tr><tr><td>Michael Beißwenger, Ciara Wigham, Carole Etienne, Holger Grumt Suárez, Laura Herzberg, Darja Fišer, Erhard Hinrichs, Tobias Horsmann, Natali Karlova-Bourbonus, Lothar Lemnitzer, Julien Longhi, Harald Lüngen, Lydia-Mai Ho-Dac, Christophe Parisse, Céline Poudat, Thomas Schmidt, Egon Stemle, Angelika Storrer and Torsten Zesch</td><td></td></tr><tr><td>"You're trolling because..." – A Corpus-based Study of Perceived Trolling and Motive Attribution in the Comment Threads of Three British Political Blogs</td><td>56</td></tr><tr><td>Márton Petykó</td><td></td></tr><tr><td>Fear and Loathing on Twitter: Attitudes towards Language</td><td>61</td></tr><tr><td>Damjan Popič and Darja Fišer</td><td></td></tr><tr><td>A Comparative Study of Computer-mediated and Spoken Conversations from Pakistani and U.S. English using Multidimensional Analysis</td><td>65</td></tr><tr><td>Muhammad Shakir and Dagmar Deuber</td><td></td></tr></table>
<table><tr><td>Posters</td><td>70</td></tr><tr><td>MoCoDa 2: Creating a Database and Web Frontend for the Repeated Collection of Mobile Communication (WhatsApp, SMS &amp; Co)</td><td>71</td></tr><tr><td>Michael Beißwenger, Marcel Fladrich, Wolfgang Imo and Evelyn Ziegler</td><td></td></tr><tr><td>Public Service News on Facebook: Exploring Journalistic Usage Patterns and Reaction Data</td><td>72</td></tr><tr><td>Daniel Pfurtscheller</td><td></td></tr><tr><td>The graphic realization of /l/-vocalization in Swiss German WhatsApp messages</td><td>73</td></tr><tr><td>Simone Ueberwasser</td><td></td></tr><tr><td>Appendix</td><td>74</td></tr><tr><td>Author Index</td><td>75</td></tr><tr><td>Keyword Index</td><td>76</td></tr></table>
# Committees

## Scientific Committee

**Chair**

Ciara Wigham Université Clermont Auvergne

**Co-Chairs**

Michael Beißwenger Universität Duisburg-Essen
Darja Fišer Faculty of Arts, University of Ljubljana

**Members**

Andrea Abel Eurac Research
Steven Coats University of Oulu
Daria Dayter University of Basel
Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute
Jennifer Frey Eurac Research
Aivars Glaznieks Eurac Research
Axel Herold Berlin-Brandenburgische Akademie der Wissenschaften
Julien Longhi Université de Cergy-Pontoise
Harald Lüngen Institut für Deutsche Sprache
Maja Miličević University of Belgrade
María-Teresa Ortego-Antón Departamento de Lengua Española (Área de Traducción e Interpretación)
- Universidad de Valladolid
Céline Poudat BCL, Université Nice-Sophia Antipolis, CNRS
H. Müge Satar Newcastle University
Stefania Spina Università per Stranieri di Perugia
Egon W. Stemle Eurac Research
Angelika Storrer Universität Dortmund

## Coordinating Committee

Michael Beißwenger Universität Duisburg-Essen
Darja Fišer Faculty of Arts, University of Ljubljana
Ciara Wigham Université Clermont Auvergne

## Organizing Committee

**Chair**

Egon W. Stemle Eurac Research

**Members**

Irina Iokhno Eurac Research
Daniela Gasser Eurac Research
# Invited Talks

## Contents

<table><tr><td>Think Global, Write Local – Patterns of Writing Dialect on SNS<br><i>Aivars Glaznieks</i></td><td>2</td></tr><tr><td>Small vs. Big Data in Language Research: Challenges and Opportunities<br><i>A. Seza Doğruöz</i></td><td>3</td></tr><tr><td>CLARIN Survey of CMC Resources and Tools<br><i>Darja Fišer</i></td><td>4</td></tr></table>
# Think Global, Write Local – Patterns of Writing Dialect on SNS

Aivars Glaznieks

Institute for Applied Linguistics, Eurac Research
Viale Druso 1, 39100 Bolzano, Italy
aivars.glaznieks@eurac.edu

## Abstract

Social Network Sites (SNS) claim that they are on a mission to connect the world. They facilitate communication among people wherever they are located. Consequently, many users of SNS communicate with a broad and heterogenic group of friends on different occasions and thereby express various aspects of their identities (such as gender, age, ethnic background etc.). One aspect may also be a local identity.

Users of SNS can show their local identity linguistically by using a regional variety. Sometimes, the use of single regionally marked words or sporadic regiolectal spellings are sufficient to identify the regional background of the writer (Androutsopoulos and Ziegler, 2003); in other cases entire text messages and conversations appear in dialectal spellings meaning that the dialect appears as the main variety of the conversation (Siebenhaar, 2008). The extent of dialect use in computer-mediated communication (CMC) may depend on various factors such as the individual dialect skills, the vividness and prestige of the respective dialect in the community, emotional involvement in the given topic, age, gender, the intended recipient, and other factors probably interacting with each other (Peersman et al., 2016).

The use of regional dialects in written CMC is one reason (amongst others) why language in CMC often differs from the respective standard languages. Since no orthographic rules are usually available for writing in dialect, it is up to the users to represent their dialect in a proper but readable and comprehensible way. Users have to construct their regiolectal language variety on the basis of the orthography of the respective standard language, which usually allows also for variation. One reason for this may be various adequate possibilities to represent a dialect word within a given writing system (e.g. German, cf. Dürscheid and Stark (2013)). Another reason may be the (sometimes very slight) phonetic differences between regionally close dialects that writers want (or do not want) to turn up in the dialect respelling (Sebba, 2007). Therefore, dialect respellings are not always coherent (neither with respect to a group of dialect speakers nor with respect to individual writers) but usually appear in various forms (Müller, 2011). However, unifications of respellings in CMC are described for pidgin languages (Heyd, 2016) and also occur in dialectal CMC (Tophinke, 2008).

Over the last decade, researchers started to compile corpora containing different genres of CMC. Such CMC corpora enable a systematic analysis of the way dialect features are reflected in written communication. In my talk, I will focus on patterns of the regional dialect(s) in the DiDi Corpus, a collection of Facebook messages from around 100 South Tyrolean writers (http://www.eurac.edu/didi). I will provide examples of regional features, analyse the distribution of such features, and discuss challenges of identifying local writings on SNS.

**Keywords:** orthography, dialect writing, facebook messages

## References

Androutsopoulos, J. and Ziegler, E. (2003). *Sprachvariation und Internet: Regionalismen in einer Chat-Gemeinschaft*. In Jannis Androutsopoulos et al., editors, *Standardfragen: soziolinguistische Perspektiven auf Sprachgeschichte, Sprachkontakt und Sprachvariation*, page 251279. Peter Lang, Frankfurt a. M.

Dürscheid, C. and Stark, E. (2013). Anything goes? SMS, phonographisches Schreiben und Morphemkonstanz. In Martin Neef et al., editors, *Die Schnittstelle von Morphologie und geschriebener Sprache*, page 189209. de Gruyter, Berlin.

Heyd, T. (2016). Global varieties of English gone digital: Orthographic and semantic variation in digital Nigerian Pidgin. In Lauren Squires, editor, *English in Computer-Mediated Communication*, pages 101–122. de Gruyter, Berlin.

Müller, C. M. (2011). *Dialektverschriftung im Spannungsfeld zwischen standardnah und lautnah*. Eine korpuslinguistische Untersuchung der Rubrik 'Dein SMS' in der Aargauer Zeitung. In Helen Christen, et al., editors, *Struktur, Gebrauch und Wahrnehmung von Dialekt. Beiträge zum 3. Kongress der Internationalen Gesellschaft für Dialektologie des Deutschen (IGDD), Zürich, 7.-9. September 2009*, pages 155–178, Wien. Praesens Verlag.

Peersman, C., Daelemans, W., Vandekerckhove, R., Vandekerckhove, B., and Van Vaerenbergh, L. (2016). The Effects of Age, Gender and Region on Non-standard Linguistic Variation in Online Social Networks. *eprint arXiv:1601.02431*, pages 1–24, 1.

Sebba, M. (2007). *Spelling in Society*. Cambridge University Press, Cambridge.

Siebenhaar, B. (2008). Quantitative Approaches to Linguistic Variation in IRC: Implications for Qualitative Research. *Language@Internet*, 5(4):1–14.

Tophinke, D. (2008). Regional schreiben: Weblogs zwischen Orthographie und Phonographie. In Helen Christen et al., editors, *Sprechen, Schreiben, Hören. Zur Produktion und Perzeption von Dialekt und Standardsprache zu Beginn des 21. Jahrhunderts. Beiträge zum 2. Kongress der Internationalen Gesellschaft für Dialektologie des Deutschen, Wien, 20.-23. September 2006*, pages 153–179, Wien. Praesens Verlag.
# Small vs. Big Data in Language Research: Challenges and Opportunities

A. Seza Doruz

Independent Researcher
a.s.dogruoz@gmail.com

## Abstract

Mobile communication tools and platforms provide various opportunities for users to interact over social media. With the recent developments in computational research and machine learning, it has become possible to analyze large chunks of language related data automatically and fast. However, these tools are not readily available to handle data in all languages and there are also challenges handling social media data. Even when these issues are resolved, asking the right research question to the right set and amount of data becomes crucially important.

Both qualitative and quantitative methods have attracted respectable researchers in language related areas of research. When tackling similar research problems, there is need for both top-down and bottom-up data-based approaches to reach a solution. Sometimes, this solution is hidden under an in-depth analysis of a small data set and sometimes it is revealed only through analyzing and experimenting with large amounts of data. However, in most cases, there is need for linking the findings of small data sets to understand the bigger picture revealed through patterns in large sets.

Having worked with both small and large language related data in various forms, I will compare pros and cons of working with both types of data across media and contexts and share my own experiences with highlights and lowlights.

**Keywords:** social media data, machine learning, small vs. large data sets, multilingualism

## References

Nguyen, D. and Doğruöz, A. (2014). Word level language identification in online multilingual communication. In *EMNLP*.

Nguyen, D., Doğruöz, A. S., Rosé, C. P., and de Jong, F. (2016). Computational sociolinguistics: A survey. *Computational linguistics*.

Papalexakis, E. and Doğruöz, A. S. (2015). Understanding multilingual social networks in online immigrant communities. In *Proceedings of the 24th International Conference on World Wide Web*, pages 865–870. ACM.
# CLARIN Survey of CMC Resources and Tools

Darja Fišer

University of Ljubljana

Ljubljana, Slovenia

darja.fiser@ff.uni-lj.si

## Abstract

With the growing volume and importance of computer-mediated communication, the need to understand its linguistic and social dimensions, along with CMC-robust language technologies is on the rise as well. This is reflected in the increasing number of conferences, projects and positions involving analysis of CMC in a wide range of disciplines in Digital Humanities, Social Sciences and Computer Science. As a result, a number of valuable CMC corpora, datasets and tools are being developed (Beißwenger et al., 2017) but unfortunately, due to non-negligible technical, legal and ethical obstacles, not many are being shared and reused.

Since it is the mission of CLARIN to create and maintain an infrastructure to support the sharing, use and sustainability of language data and tools for researchers in Digital Humanities and Social Sciences (Krauwer and Hinrichs, 2014), it is our goal to have a good overview of the available resources and tools, to offer support to their developers to overcome the technical, legal and ethical obstacles and deposit them to the CLARIN infrastructure, as well as to the researchers with diverse backgrounds, such as linguistics, media studies, psychology etc., but also to interested parties from the educational, commercial, political, medical and legal sectors of the society who are interested in using them.

The first step in this direction was an interdisciplinary workshop<fnref n="1" /> on the creation and use of social media which was organized within the Horizon 2020 CLARIN-PLUS project on 18 and 19 May 2017 in Kaunas, Lithuania. The aims of the workshop were to demonstrate the possibilities of social media resources and natural language processing tools for researchers with a diverse research background and an interest in empirical research of language and social practices in computer-mediated communication, to promote interdisciplinary cooperation possibilities, and to initiate a discussion on the various approaches to social media data collection and processing.

The workshop also served as a platform to conduct a survey<fnref n="2" /> of corpora, datasets and tools of computer-mediated communication in the languages spoken in countries that are members and observers of CLARIN ERIC. Apart from identifying the existing resources and tools, our motivation was to establish to which extent they are accessible through the CLARIN infrastructure and how the information and accessibility of them could be further optimized from a user perspective.

In this talk, I will give an overview of the identified corpora, the smaller, more focused datasets and tools that are tailored to processing computer-mediated communication. The focus of the talk will be on the comprehensiveness of the provided metadata, level of availability and accessibility of the identified resources and tools and the degree of their actual or potential inclusion in the CLARIN infrastructure. I will also discuss the simple and long-term possibilities of enriching the current state of the infrastructure and provide guidelines for creating and depositing CMC resources with a CLARIN center.

**Keywords:** CLARIN ERIC, research infrastructure, language resources, NLP tools, computer-mediated communication

## References

Beißwenger, M., Chanier, T., Erjavec, T., Fišer, D., Herold, A., Lubešić, N., Lüngen, H., Poudat, C., Stemle, E., Storrer, A., and Wigham, C. (2017). Closing a Gap in the Language Resources Landscape: Groundwork and Best Practices from Projects on Computer-mediated Communication in four European Countries. In *Selected Papers from the CLARIN Annual Conference 2016, Aix-en-Provence, 2628 October 2016, CLARIN Common Language Resources and Technology Infrastructure*, pages 1–18. Linköping University Electronic Press, Linköpings universitet.

Krauwer, S. and Hinrichs, E. (2014). The clarin research infrastructure: resources and tools for e-humanities scholars. In *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014)*, pages 1525–1531. European Language Resources Association (ELRA).

<footnote n="1">¹https://www.clarin.eu/event/2017/clarin-plus-workshop-creation-and-use-social-media-resources</footnote>

<footnote n="2">²https://office.clarin.eu/v/CE-2017-1064-Resources-for-computer-mediated-communication.docx</footnote>
# Papers

## Contents

<table><tr><td>The Impact of WhatsApp on Dutch Youths' School Writing<br>Lieke Verheijen and Wilbert Spooren</td><td>6</td></tr><tr><td>Modeling Non-Standard Language Use in Adolescents' CMC: The Impact and Interaction of Age, Gender and Education<br>Lisa Hilte, Reinhild Vandekerckhove and Walter Daelemans</td><td>11</td></tr><tr><td>Investigating Interaction Signs across Genres, Modes and Languages: The Example of OKAY<br>Laura Herzberg and Angelika Storrer</td><td>16</td></tr><tr><td>Anonymisation of the Dortmund Chat Corpus 2.1<br>Harald Lüngen, Michael Beißwenger, Laura Herzberg and Cathrin Pichler</td><td>21</td></tr><tr><td>Emoticons as multifunctional and pragmatic Resources: a corpus-based Study on Twitter<br>Stefania Spina</td><td>25</td></tr><tr><td>Corpus-Based Analysis of Demonyms in Slovene Twitter<br>Taja Kuzman and Darja Fišer</td><td>30</td></tr><tr><td>European Language Ecology and Bilingualism with English on Twitter<br>Steven Coats</td><td>35</td></tr><tr><td>Reliable Part-of-Speech Tagging of Low-frequency Phenomena in the Social Media Domain<br>Tobias Horsmann, Michael Beißwenger and Torsten Zesch</td><td>39</td></tr><tr><td>Developing a protocol for collecting data in Higher Education: assessing natural language metadata for a Databank of Oral Teletandem Interactions<br>Paola Leone</td><td>44</td></tr><tr><td>The #Idéo2017 Platform<br>Julien Longhi, Claudia Marinica, Nader Hassine, Abdulhafiz Alkhouli and Boris Borzic</td><td>46</td></tr><tr><td>Connecting Resources: Which Issues Have to be Solved to Integrate CMC Corpora from Heterogeneous Sources and for Different Languages?<br>Michael Beißwenger, Ciara Wigham, Carole Etienne, Holger Grumt Suárez, Laura Herzberg, Darja Fišer, Erhard Hinrichs, Tobias Horsmann, Natali Karlova-Bourbonus, Lothar Lemnitzer, Julien Longhi, Harald Lüngen, Lydia-Mai Ho-Dac, Christophe Parisse, Céline Poudat, Thomas Schmidt, Egon Stemle, Angelika Storrer and Torsten Zesch</td><td>52</td></tr><tr><td>"You're trolling because..." – A Corpus-based Study of Perceived Trolling and Motive Attribution in the Comment Threads of Three British Political Blogs<br>Márton Petykó</td><td>56</td></tr><tr><td>Fear and Loathing on Twitter: Attitudes towards Language<br>Damjan Popič and Darja Fišer</td><td>61</td></tr><tr><td>A Comparative Study of Computer-mediated and Spoken Conversations from Pakistani and U.S. English using Multidimensional Analysis<br>Muhammad Shakir and Dagmar Deuber</td><td>65</td></tr></table>
# The Impact of WhatsApp on Dutch Youths' School Writing

Lieke Verheijen, Wilbert Spooren

Radboud University (Nijmegen, the Netherlands)

Email: lieke.verheijen@let.ru.nl; w.spooren@let.ru.nl

## Abstract

Today's youths are continuously engaged with social media. The informal language they use in computer-mediated communication (CMC) often deviates from spelling and grammar rules of the standard language. Therefore, parents and teachers fear that social media have a negative impact on youths' literacy skills. This paper examines whether such worries are justifiable. An experimental study was conducted with 500 Dutch youths of different educational levels and age groups, to find out if social media affect their productive or perceptive writing skills. We measured whether chatting via WhatsApp directly impacts the writing quality of Dutch youths' narratives or their ability to detect 'spelling errors' (deviations from Standard Dutch) in grammaticality judgement tasks. The use of WhatsApp turned out to have no short-term effects on participants' performances on either of the writing tasks. Thus, the present study gives no cause for great concern about any impact of WhatsApp on youths' school writing.

**Keywords:** computer-mediated communication, social media, WhatsApp, writing, literacy

## 1. Introduction

Youths are nowadays constantly using computer-mediated communication such as WhatsApp, Facebook chat, Snapchat, and Twitter. Examples (1)–(3) present chat messages by Dutch youths:

(1) **OMG!** Had je mijn mijn verhaal gezien
**Hahahahhaahhaaha** kwam ik pas vanochtend achter
**k** kan me **nie** eens herinneren **da** ik die gemaakt heb
**Miss** in **mn** slaap **ofzo** hagahagagaa
('OMG! Did you see my my story
Hahahahhaahhaaha only found out this morning
i cant even remember making dat
Mayb in me sleep or somethin hagahagagaa')

(2) Beetje te vroeg ik val echt in slaap maar alvast **happy birthdayyyyyy** **tooooooo** **youuuuuuuuuuu!**
(‘Bit too early I’m really falling asleep but anyway…’)

(3) **Liefie** ♥ gaat ie weer met jou? Wat het je 😊 bel me
**weneer** je online bent 👅ly♥♥♥♥ zie je
**morgen** 😊 **BEL ME** 😊 **chatt** 😊
('Luv♥ you doin okay again? What hare you 😊 call
me whn you are online 👅ly♥♥♥♥ see you
tomorrow 😊 **CALL ME** 😊 **honey** 😊')

All the words in bold deviate from the Dutch standard language norms. They contain non-standard abbreviations, letter repetitions, phonetic respellings, overuse of capitalisation, and emoji. Such deviations from Standard Dutch lead to fears that informal written CMC may negatively interfere with writing in more formal settings (Spooren, 2009). These fears have existed for decades now;<fnref n="1" /> it is just the medium under critique that changes every few years. Yet there are also scholars who point out that youths' literacy skills may benefit from social media use, via creativity with language, greater exposure to written texts, more engagement in writing, and greater metalinguistic awareness (Wood, Kemp, & Plester, 2013).

## 2. Research Goals and Hypotheses

Since prior research does not provide a conclusive answer about the impact of CMC on literacy (Verheijen, 2013), our research aims to contribute to this debate. The goal of this study is to determine whether Dutch youngsters' constant use of social media affects the way they write at school. We conducted an experiment to find out whether engaging in synchronous written CMC directly impacts their productive or perceptive school writing skills. Our focus was on the chat program WhatsApp, because this is currently a very popular medium among Dutch youths. Furthermore, we aim to establish whether the demographic variables of age, education, and gender have an impact on youths' writing skills – specifically, if they have a mediating effect on the possible impact of WhatsApp on those writing skills. Therefore, the analysis will include four independent variables: not only condition (exposure vs. non-exposure to WhatsApp), but also educational level (lower, intermediate, higher), age group (adolescents vs. young adults), and gender (boys vs. girls). We hypothesize that a greater impact of CMC use on school writing skills may be displayed by youths of a younger age group or lower educational level, as it may be more difficult for them to keep these registers separate.

## 3. Methodology

### 3.1 Participants

Participants were 500 youths from secondary and tertiary educational institutions, from different educational levels and age groups, in Nijmegen and surroundings. The data collection period lasted from October to December 2016. Testing took place in an educational setting. Most participants were tested in class; only a small number in higher tertiary education voluntarily took part outside of class. The latter were reimbursed for their participation with gift certificates of € 5. Students from higher tertiary education belonged to different faculties and studies, including communication and information sciences,

<footnote n="1">¹ Or, for that matter, for centuries. See Deutscher (2005) for a historical note on the concerns about language deterioration.</footnote>
biology, and literary and cultural studies. The adolescents (N = 300) were around 14 years old (x̄ age = 14.2 yrs, range 13-16; 151 male, 149 female), all in the third grade. The young adults (N = 200) were around 20 years old (x̄ age = 20.4 yrs, range 18-27; 72 male, 128 female). Table 1 shows an overview of the participants. Afterwards, underage participants were given a document with more information about the study and the researchers' contact details, to take home to their parents or caretakers.

<table><caption>Table 1: Overview of participants.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">Educational level</th></tr><tr><th>lower</th><th>intermediate</th><th>higher</th></tr></thead><tbody><tr><td rowspan="2">Age group</td><td>adolescents: secondary education</td><td>101</td><td>92</td><td>107</td></tr><tr><td>young adults: tertiary education</td><td>102</td><td>-</td><td>98</td></tr></tbody></table>

## 3.2 Data Collection

### 3.2.1. Priming: WhatsApp vs. Colouring

All classes that were tested were divided into two groups. The experimental groups were primed with CMC via social media: they were instructed to chat via WhatsApp on their own smartphones for fifteen minutes, in small groups of three or four students. They could chat about whatever they preferred; no specific conversation topics were provided, in order to generate as natural chat conversations as possible. During that time, the control groups performed a non-CMC-related control task, namely colouring mandalas. These tasks were chosen because, in a pilot study, they proved to be effective in revealing differences with respect to orthography and language correctness (Riemens, 2016).

### 3.2.2. Measuring Productive Writing Skills: Stories

To test their productive writing skills, all participants wrote a story in class, starting with the following sentence: "I was alone in a dark room. My hand groped for the light switch, but suddenly..." [translated from Dutch]. The formal writing genre that was tested was that of narrative storytelling. Since not all classes had easy access to computers and laptops, all stories were handwritten for consistency's sake.

### 3.2.3. Measuring Perceptive Writing Skills: GJTs

Participants also completed grammaticality judgement tasks (GJTs), to test their receptive grammar and spelling skills. These consisted of twenty sentences in which they had to spot and correct 'language errors'. These were orthographic deviations typical of CMC: various types of textisms (phonetic respelling, reduplication of letter, shortening, single letter homophone, initialism); missing capitalisation, diacritics, and punctuation; spelling 'errors' that are heavily frowned upon by Dutch language prescriptivists (is/eens, d/t, jou/jouw); emoticons; omissions; English borrowings; and extra spacing. Five sentences contained no orthographic deviations, so participants could spot and correct fifteen 'errors'.

## 3.3 Data Analysis

### 3.3.1. GJT Scores

For the grammaticality judgement tasks, two scores were computed for each participant. First, the choice score: whether they correctly identified the sentence as containing an 'error' or not (max. 20 points). Second, the correction score: whether they correctly managed to correct that 'error' (max. 15).

### 3.3.2. T-Scan Analysis

The stories were automatically analysed with T-Scan, software for conducting complexity analyses of Dutch texts (Pander Maat et al., 2014). T-Scan provided us with a staggering 411 variables for each text, out of which a theory-based selection of 27 relevant variables was made:

1) Zin_per_doc: number of sentences per essay

2) Word_per_doc: number of words per essay

3) Let_per wrd: number of letters per word

4) Wrd_per_zin: number of words per sentence

5) Bijzin_per_zin: number of subordinate clauses per sentence

6) Pv_Frog_d: density of finite verbs

7) D_level: D-level

8) Nom_d: density of nominalisations

9) Lijdv_d: density of passive forms

10) AL_gem: average of all dependency lengths per sentence

11) AL_max: maximal dependency length per sentence

12) Bijw_bep_d: density of adverbials

13) TTR wrd: type-token ratio (for words)

14) MTLD wrd: measure of textual lexical diversity (for words)

15) Inhwrdd: density of content words

16) Pers_vnw_d: density of personal and possessive pronouns

17) Ww_mod_d: density of modal verbs

18) Huww_tijd_d: density of auxiliary verbs of time

19) Koppelww_d: density of copula verbs

20) Imp_ellips_d: density of imperatives and elliptical constructions

21) Vg_d: density of conjunctions

22) Lidw_d: density of articles

23) Nw_d: density of nouns

24) Tuss_d: density of interjections

25) Spec_d: density of names and special words

26) Interp_d: density of punctuation

27) Afk_d: density of abbreviations

### 3.3.3. Exploratory Factor Analysis

Because the twenty-seven variables selected from T-Scan were still too many to put into a regression analysis, we used an exploratory factor analysis (with the extraction method of principal component analysis, PCA), to further reduce these to a set of writing components indicative of the writing quality of stories.

An orthogonal rotation method was chosen, namely varimax with Kaiser normalization: this method, which does not allow correlations between factors, facilitated the interpretation of results, since it maximizes the spread of loadings for a variable across all factors. There was no multicollinearity, because none of the correlation coefficients were $r \ge .84$. Missing values were replaced with the mean, because listwise deletion would result in a
loss of participants in the analysis, and pairwise deletion would lead to a non-positive definite matrix. The Keyser-Meyer-Olkin measure was well above .5 (KMO = .644), which verified the sampling adequacy for the analysis. Bartlett's test of sphericity showed that correlations between items were sufficiently large for PCA: $\chi^2(351) = 6267.569, p < .001$. The proportion of residuals with an absolute value greater than 0.05 was 50%. An initial analysis yielded eigenvalues for each component in the data. The large sample size of this study (500 participants) allowed us to use a scree plot with eigenvalues over 1 for deciding how many components to extract. The inflexion of the scree plot justified retaining three components.

Table 2 shows the results of the PCA after rotation. The items that cluster on the same components suggest that component 1 represents syntactic complexity, 2 lexical richness, and 3 writing productivity. The total variance explained by the three factors is 38.08%. The resulting factor scores were saved as Anderson-Rubin variables, so they did not correlate.

<table><caption>Rotated Component Matrix</caption><thead><tr><th rowspan="2">Writing variable</th><th colspan="3">Rotated factor loadings</th></tr><tr><th>1</th><th>2</th><th>3</th></tr></thead><tbody><tr><td>AL_max</td><td>.868</td><td>.141</td><td>.106</td></tr><tr><td>D_level</td><td>.818</td><td>-.193</td><td>.016</td></tr><tr><td>Bijzin_per_zin</td><td>.792</td><td>-.089</td><td>.077</td></tr><tr><td>AL_gem</td><td>.764</td><td>.221</td><td>.232</td></tr><tr><td>Wrd_per_zin</td><td>.720</td><td>.004</td><td>-.087</td></tr><tr><td>Interp_d</td><td>-.718</td><td>-.077</td><td>.065</td></tr><tr><td>Vg_d</td><td>.556</td><td>-.287</td><td>.072</td></tr><tr><td>Tuss_d</td><td>-.240</td><td>-.117</td><td>.186</td></tr><tr><td>Bijw_bep_d</td><td>.221</td><td>-.004</td><td>.107</td></tr><tr><td>Spec_d</td><td>-.147</td><td>.016</td><td>-.057</td></tr><tr><td>Pv_Frog_d</td><td>-.167</td><td>-.762</td><td>-.037</td></tr><tr><td>Nw_d</td><td>-.059</td><td>.698</td><td>-.166</td></tr><tr><td>Pers_vnw_d</td><td>-.102</td><td>-.680</td><td>.066</td></tr><tr><td>Let_per_wrd</td><td>-.002</td><td>.624</td><td>-.153</td></tr><tr><td>Inhwrd_d</td><td>-.045</td><td>.554</td><td>.054</td></tr><tr><td>Lidw_d</td><td>-.047</td><td>.520</td><td>-.232</td></tr><tr><td>MTLD_wrd</td><td>-.060</td><td>.450</td><td>.004</td></tr><tr><td>Nom_d</td><td>-.034</td><td>.423</td><td>.028</td></tr><tr><td>Koppelww_d</td><td>-.127</td><td>-.189</td><td>.020</td></tr><tr><td>Ww_mod_d</td><td>.057</td><td>-.145</td><td>.136</td></tr><tr><td>Imp_ellips_d</td><td>.039</td><td>-.099</td><td>.066</td></tr><tr><td>Word_per_doc</td><td>.023</td><td>.004</td><td>.917</td></tr><tr><td>TTR_wrd</td><td>-.141</td><td>.281</td><td>-.800</td></tr><tr><td>Zin_per_doc</td><td>-.529</td><td>-.038</td><td>.782</td></tr><tr><td>Huww_tijd_d</td><td>-.078</td><td>-.043</td><td>-.378</td></tr><tr><td>Lijdv_d</td><td>-.038</td><td>.028</td><td>-.139</td></tr><tr><td>Afk_d</td><td>-.095</td><td>.053</td><td>-.095</td></tr><tr><td>Eigenvalues</td><td>4.496</td><td>3.246</td><td>2.539</td></tr><tr><td>% of variance</td><td>16.652</td><td>12.021</td><td>9.405</td></tr></tbody></table>

Note: loadings > .40 appear in bold and colour.

<caption>Table 2: PCA rotated factor loadings for the story analysis.</caption>

### 3.3.4. Linear Multiple Regression

The next step of the statistical analysis was linear multiple regression. The outcome variables were the three A-R factor scores resulting from the exploratory factor analysis of the stories and the two GJT scores. The predictor variables were condition (colouring versus WhatsApp),

the three demographic variables educational level, age group, and gender, plus all interactions between condition and the demographic variables. As we had no preconceived ideas about which variables would be significant predictors, they were all entered with the forced entry method. The first block of the regression only contained the main effects. The interactions were entered in subsequent blocks.<fnref n="2" />

## 4. Results and Discussion

Table 3 shows the means and standard deviations of participants' performances on the writing tasks:

<table><caption>Table 3: Descriptive statistics.</caption><thead><tr><th rowspan="3">Independent variables</th><th colspan="4">Dependent variables</th></tr><tr><th colspan="2">Stories</th><th colspan="2">GJTs</th></tr><tr><th>syntactic complexity: x̄(SD)</th><th>lexical richness: x̄(SD)</th><th>writing productivity: x̄(SD)</th><th>choice correction score: x̄(SD)</th></tr></thead><tbody><tr><td><strong>Condition:</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Colouring, N = 207</td><td>-0.02<br>(1.00)</td><td>0.06<br>(1.04)</td><td>0.00<br>(1.09)</td><td>14.44<br>(3.16)</td><td>13.82<br>(1.03)</td></tr><tr><td>WhatsApp, N = 201</td><td>0.07<br>(1.05)</td><td>0.09<br>(0.96)</td><td>0.03<br>(0.99)</td><td>14.71<br>(3.04)</td><td>13.74<br>(0.99)</td></tr><tr><td><strong>Educational level:</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Lower, N = 203</td><td>0.19<br>(1.06)</td><td>-0.03<br>(0.93)</td><td>-0.17<br>(1.04)</td><td>12.68<br>(2.87)</td><td>13.53<br>(0.99)</td></tr><tr><td>Higher, N = 205</td><td>-0.13<br>(0.96)</td><td>0.18<br>(1.06)</td><td>0.20<br>(1.00)</td><td>16.44<br>(2.00)</td><td>14.03<br>(0.96)</td></tr><tr><td><strong>Age group:</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Adolescents, N = 208</td><td>0.01<br>(1.14)</td><td>-0.24<br>(0.92)</td><td>0.00<br>(1.08)</td><td>14.10<br>(3.09)</td><td>13.75<br>(1.01)</td></tr><tr><td>Young adults, N = 200</td><td>0.05<br>(0.88)</td><td>0.40<br>(0.98)</td><td>0.03<br>(1.00)</td><td>15.06<br>(3.04)</td><td>13.81<br>(1.01)</td></tr><tr><td><strong>Gender:</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Male, N = 179</td><td>0.16<br>(1.16)</td><td>0.09<br>(1.00)</td><td>-0.14<br>(1.11)</td><td>14.25<br>(3.14)</td><td>13.60<br>(1.02)</td></tr><tr><td>Female, N = 229</td><td>-0.08<br>(0.89)</td><td>0.06<br>(1.0)</td><td>0.14<br>(0.97)</td><td>14.82<br>(3.06)</td><td>13.92<br>(0.98)</td></tr><tr><td>TOTAL, N = 408</td><td>0.03<br>(1.02)</td><td>0.07<br>(1.00)</td><td>0.01<br>(1.04)</td><td>14.57<br>(3.10)</td><td>13.78<br>(1.01)</td></tr></tbody></table>

### 4.1 Syntactic Complexity

One writing component was syntactic complexity, presented in Table 2 in column '1'. Educational level was a significant negative predictor: higher educated youths wrote syntactically less complex stories. At a first glance, this may seem surprising. However, this rather fits the genre of narrative storytelling, which does not require complex, long sentences – as opposed to, for example, expository discussion as in essays. So the higher educated youths showed more mastery of the genre of stories. Gender was a significant negative predictor: male participants wrote syntactically more complex stories.

<footnote n="2">² Participants of the intermediate secondary educational level were eventually omitted, as they were not part of the original research plan and would decrease the reliability of the analyses because of an empty cell in the research design: no youths of intermediate tertiary education were tested (see Table 1).</footnote>
<table><caption>Table 4: Regression results for syntactic complexity.<fnref n="3" /></caption><thead><tr><th colspan="4">Dependent variable: syntactic complexity</th></tr><tr><th>Independent variables</th><th><i>B</i></th><th><i>SE B</i></th><th><i>β</i></th></tr></thead><tbody><tr><td>Condition</td><td>0.09</td><td>0.10</td><td>0.04</td></tr><tr><td><strong>Educational level</strong></td><td><strong>-0.32</strong></td><td><strong>0.10</strong></td><td><strong>-0.16**</strong></td></tr><tr><td>Age group</td><td>0.06</td><td>0.10</td><td>0.03</td></tr><tr><td><strong>Gender</strong></td><td><strong>-0.24</strong></td><td><strong>0.10</strong></td><td><strong>-0.12*</strong></td></tr><tr><td><i>R<sup>2</sup></i> / Adjusted <i>R<sup>2</sup></i></td><td>.04 / .03</td><td></td><td></td></tr><tr><td><i>ANOVA</i></td><td colspan="3"><i>F</i> (4, 403) = 4.24 (<i>p</i> &lt; .01)</td></tr></tbody></table>

## 4.2 Lexical Richness

Another writing component was lexical richness. Table 2 shows the variables that loaded onto this component, in the column labelled ‘2’. Lexical richness was positively predicted by educational level and age group: the stories of higher educated and of older participants were lexically richer. In addition, there was a significant interaction between gender and condition. For boys, WhatsApp had a small significant positive effect on their stories’ lexis; for girls, the effect was negative but non-significant.

<table><caption>Table 5: Regression results for lexical richness.</caption><thead><tr><th colspan="4">Dependent variable: lexical richness</th></tr><tr><th>Independent variables</th><th><i>B</i></th><th><i>SE B</i></th><th><i>β</i></th></tr></thead><tbody><tr><td>Condition</td><td>0.32</td><td>0.19</td><td>0.16</td></tr><tr><td><strong>Educational level</strong></td><td><strong>0.31</strong></td><td><strong>0.13</strong></td><td><strong>0.16*</strong></td></tr><tr><td><strong>Age group</strong></td><td><strong>0.64</strong></td><td><strong>0.13</strong></td><td><strong>0.32***</strong></td></tr><tr><td>Gender</td><td>0.11</td><td>0.13</td><td>0.06</td></tr><tr><td>Educational level × condition</td><td>-0.14</td><td>0.19</td><td>-0.06</td></tr><tr><td>Age group × condition</td><td>0.08</td><td>0.19</td><td>0.03</td></tr><tr><td><strong>Gender × condition</strong></td><td><strong>-0.50</strong></td><td><strong>0.19</strong></td><td><strong>-0.22**</strong></td></tr><tr><td><i>R<sup>2</sup></i> / Adjusted <i>R<sup>2</sup></i></td><td>.13 / .12</td><td></td><td></td></tr><tr><td><i>ANOVA</i></td><td colspan="3"><i>F</i> (7, 400) = 8.90 (<i>p</i> &lt; .001)</td></tr></tbody></table>

## 4.3 Writing Productivity

The third component of the stories, writing productivity, is presented in column ‘3’ in Table 2. It was positively predicted by educational level: youths with a higher educational level produced significantly longer stories. Gender was a significant positive predictor of writing productivity too: female participants wrote longer stories.

<table><caption>Table 6: Regression results for writing productivity.</caption><thead><tr><th colspan="4">Dependent variable: writing productivity</th></tr><tr><th>Independent variables</th><th><i>B</i></th><th><i>SE B</i></th><th><i>β</i></th></tr></thead><tbody><tr><td>Condition</td><td>0.03</td><td>0.10</td><td>0.01</td></tr><tr><td><strong>Educational level</strong></td><td><strong>0.37</strong></td><td><strong>0.10</strong></td><td><strong>0.18***</strong></td></tr><tr><td>Age group</td><td>-0.01</td><td>0.10</td><td>0.00</td></tr><tr><td><strong>Gender</strong></td><td><strong>0.27</strong></td><td><strong>0.10</strong></td><td><strong>0.13**</strong></td></tr><tr><td><i>R<sup>2</sup></i> / Adjusted <i>R<sup>2</sup></i></td><td>.05 / .04</td><td></td><td></td></tr><tr><td><i>ANOVA</i></td><td colspan="3"><i>F</i> (4, 403) = 5.19 (<i>p</i> &lt; .001)</td></tr></tbody></table>

## 4.4 GJT Choice Score

For the grammaticality judgement tasks, educational level and age group were significant positive predictors of the

choice score, so higher educated youths and older youths were more successful in spotting ‘language errors’.

<table><caption>Table 7: Regression results GJT choice score.</caption><thead><tr><th colspan="4">Dependent variable: GJT choice score</th></tr><tr><th>Independent variables</th><th><i>B</i></th><th><i>SE B</i></th><th><i>β</i></th></tr></thead><tbody><tr><td>Condition</td><td>0.21</td><td>0.24</td><td>0.03</td></tr><tr><td><strong>Educational level</strong></td><td><strong>3.77</strong></td><td><strong>0.24</strong></td><td><strong>0.61***</strong></td></tr><tr><td><strong>Age group</strong></td><td><strong>0.99</strong></td><td><strong>0.24</strong></td><td><strong>0.16***</strong></td></tr><tr><td>Gender</td><td>0.35</td><td>0.24</td><td>0.06</td></tr><tr><td><i>R<sup>2</sup></i> / Adjusted <i>R<sup>2</sup></i></td><td>.40 / .39</td><td></td><td></td></tr><tr><td><i>ANOVA</i></td><td colspan="3"><i>F</i> (4, 403) = 67.21 (<i>p</i> &lt; .001)</td></tr></tbody></table>

## 4.5 GJT Correction Score

The correction score was significantly positively predicted by educational level and gender: both higher educated and female participants were more successful in correcting ‘language errors’. The interaction between gender and condition was also significant. For girls, WhatsApp had a small significant negative effect on their correction score; for boys, the effect was positive but non-significant.

<table><caption>Table 8: Regression results for GJT correction score.</caption><thead><tr><th colspan="4">Dependent variable: GJT correction score</th></tr><tr><th>Independent variables</th><th><i>B</i></th><th><i>SE B</i></th><th><i>β</i></th></tr></thead><tbody><tr><td>Condition</td><td>-0.09</td><td>0.19</td><td>-0.05</td></tr><tr><td><strong>Educational level</strong></td><td><strong>0.39</strong></td><td><strong>0.13</strong></td><td><strong>0.20**</strong></td></tr><tr><td>Age group</td><td>-0.11</td><td>0.14</td><td>-0.06</td></tr><tr><td><strong>Gender</strong></td><td><strong>0.51</strong></td><td><strong>0.14</strong></td><td><strong>0.25***</strong></td></tr><tr><td>Educational level × condition</td><td>0.22</td><td>0.19</td><td>0.10</td></tr><tr><td>Age group × condition</td><td>0.31</td><td>0.19</td><td>0.13</td></tr><tr><td><strong>Gender × condition</strong></td><td><strong>-0.44</strong></td><td><strong>0.20</strong></td><td><strong>-0.20*</strong></td></tr><tr><td><i>R<sup>2</sup></i> / Adjusted <i>R<sup>2</sup></i></td><td>.11 / .09</td><td></td><td></td></tr><tr><td><i>ANOVA</i></td><td colspan="3"><i>F</i> (7, 400) = 6.71 (<i>p</i> &lt; .001)</td></tr></tbody></table>

An overview of the results of all the linear multiple regressions is presented in Table 9 below:

<table><thead><tr><th colspan="6">Dependent variables</th></tr><tr><th colspan="3">Stories</th><th colspan="3">GJTs</th></tr><tr><th>Independent variables</th><th>syntactic complexity</th><th>lexical richness</th><th>writing productivity</th><th>choice score</th><th>correction score</th></tr></thead><tbody><tr><td colspan="6"><strong>Main variables:</strong></td></tr><tr><td>Condition</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Educational level</strong></td><td>-</td><td>+</td><td>+</td><td>+</td><td>+</td></tr><tr><td><strong>Age group</strong></td><td></td><td>+</td><td></td><td>+</td><td></td></tr><tr><td><strong>Gender</strong></td><td>-</td><td></td><td>+</td><td></td><td>+</td></tr><tr><td colspan="6"><strong>Interactions:</strong></td></tr><tr><td>EL × C</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>AG × C</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>G × C</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>EL × AG × C</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>EL × G × C</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>AG × G × C</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>EL × AG × G × C</td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

Note: + = positive predictor, - = negative predictor, C = condition, EL = educational level, AG = age group, G = gender.

<caption>Table 9: Overview of regression results.</caption>

<footnote n="3">³ For all tables with results, *p* < .05, **p* < .01, ***p* < .001.</footnote>
## 5. Conclusion

This paper reports on an experimental study measuring whether the use of WhatsApp has a direct impact on the writing quality of Dutch youths' stories or on their ability to detect 'spelling errors' in grammaticality judgement tasks. Educational level was a significant positive predictor for four writing variables, and age group for two. Gender predicted three writing variables. Condition did not affect the writing variables. We can thus conclude that WhatsApp does not appear to impact Dutch youths' productive or perceptive writing skills. Only two minor interactions between condition and gender were found, which suggests that perhaps there might be a slight impact of WhatsApp, moderated by gender, in which boys' lexical richness might benefit from CMC and girls' ability to correct language errors might be affected by it.

Two objections to this conclusion may be raised. One might doubt whether our measuring instrument was sensitive enough to detect differences in writing quality. However, the effectiveness of our testing method is confirmed by finding main effects for three demographic variables: these show that analysing the stories with T-Scan, as well as the GJT scores, are successful ways to detect differences in youths' writing skills. One might also argue that our experimental manipulation, the use of WhatsApp for fifteen minutes, was not strong enough to generate any effects. That cannot be the case either, because we found some interactions with gender; moreover, the prime already proved to yield significant results in a pilot study conducted in advance. All in all, the present study gives no cause for concern about the impact of WhatsApp on school writing.

## 6. Future Work

We hypothesized that particularly writers of a younger age group and lower educational level could experience possible interference of social media on their school writings. Prior research also suggests that youths of a lower educational track have more trouble distinguishing informal online writing (CMC) from more formal offline writing repertoires (Vandekerckhove & Sandra, 2016). Further research could explore other ways to test for such interference. Perhaps effects of social media crop up in minor orthographic details of their school writings, such as non-standard punctuation, capitalisation, spacing, or diacritics, because in the pilot study, these were the items on which WhatsApp use had the greatest impact. The frequent omission of punctuation and capitalization (sentence-initial or with proper names) in school writings was also noted by Vandekerckhove and Sandra (2016). The stories written for the present experiment could thus be analysed for the occurrence of such non-standard orthographic details.

In addition, the WhatsApp chats produced by roughly half of the participants during the priming phase were nearly all collected afterwards, of course with their consent (sent to the first author via email), but were not analysed. If properly formatted and annotated, the CMC data thus compiled could be a valuable corpus for further analysis.

We could study the nature of these WhatsApp interactions, e.g. for the use of textisms, to find out to which extent these chats actually differ from Standard Dutch in terms of orthography and grammar and whether the amount of deviations affected the direct impact of CMC use on the writing tasks.

## 7. Acknowledgements

This study is part of a research project funded by the Dutch Organisation for Scientific Research (NWO), project number 322-70-006.

## 8. References

- Deutscher, G. (2005). *The Unfolding of Language*. London: Random House.

- Pander Maat, H., et al. (2014). T-Scan: A new tool for analyzing Dutch text. *Computational Linguistics in the Netherlands Journal*, 4, pp. 53--74.

- Riemens, N. (2016). De directe invloed van WhatsApp op schrijfvaardigheid. BA thesis, Radboud University.

- Spooren, W. (2009). Bezorgde ouders? De relatie tussen chat en schrijfkwaliteit. In W. Spooren, M. Onrust, & J. Sanders (Eds.), *Studies in Taalbeheersing* 3. Assen: Van Gorcum, pp. 331--342.

- Vandekerckhove, R., and Sandra, D. (2016). De potentiële impact van informele online communicatie op de spellingpraktijk van Vlaamse tieners in schoolcontext. *Tijdschrift voor Taalbeheersing*, 38(3), pp. 201--234.

- Verheijen, L. (2013). The effects of text messaging and instant messaging on literacy. *English Studies*, 94(5), pp. 582--602.

- Wood, C., Kemp, N. and Plester, B. (2013). *Text Messaging and Literacy: The Evidence*. New York, NY: Routledge.
# Modeling Non-Standard Language Use in Adolescents' CMC: The Impact and Interaction of Age, Gender and Education

Lisa Hilte, Reinhild Vandekerckhove, Walter Daelemans

CLIPS Research Center, University of Antwerp

Postal address: Prinsstraat 13, B-2000 Antwerp, Belgium

E-mail: {lisa.hilte, reinhild.vandekerckhove, walter.daelemans}@uantwerpen.be

## Abstract

The present paper deals with Flemish adolescents' informal computer-mediated communication (CMC) in a large corpus (2.9 million tokens) of chat conversations. We analyze deviations from written standard Dutch and possible correlations with the teenagers' gender, age and educational track. The concept of non-standardness is operationalized by means of a wide range of features that serve different purposes, related to the chatspeak maxims of orality, brevity and expressiveness. It will be demonstrated how the different social variables impact on non-standard writing, and, more importantly, how they interact with each other. While the findings for age and education correspond to our expectations (more non-standard markers are used by younger adolescents and students in practice-oriented educational tracks), the results for gender (no significant difference between girls and boys) do not: they call for a more fine-grained analysis of non-standard writing, in which features relating to different chat principles are examined separately.

**Keywords:** computer-mediated communication, non-standardness, teenage talk, language modeling

## 1. Introduction

Adolescents' informal CMC tends to deviate from formal standard writing in many ways: alternative spelling, non-standard capitalization, emoticons, ... These deviations can be related to the three main principles behind chatspeak, i.e. the principles of orality, economy and expressive compensation (Androutsopoulos, 2011, 149; see section 3.2 for definitions and examples). While many CMC-studies report on just one type of features or present a small selection, the present study examines a wide array of 11 non-standard features and relates their frequency to three independent variables.

In the following sections, we will describe the goal of this study (section 2), as well as the dependent and independent variables (section 3). Next, we present the corpus and methodology (section 4), and finally, we will discuss and evaluate the results (section 5).

## 2. Goal of the Paper

We try to capture the impact of three aspects of the adolescent authors' profile on their CMC writing practices: their gender, age, and educational track. The latter variable has been largely neglected in CMC research. The same accounts for potential interactions between these variables: as boys and girls age, do their online writing practices evolve in a similar way? And do the same age and gender patterns emerge in different education types? In the end we want to demonstrate that the inclusion of a wide range of both independent and dependent variables is a prerequisite for a correct assessment of variation patterns in adolescents' CMC.

## 3. Dependent and Independent Variables

### 3.1. Independent Variables

All participants are high school students living in Flanders, the Dutch-speaking part of Belgium. We examine three social variables: the adolescents' gender, their age and their type of education (i.e. educational track).

Both gender and age are treated as binary variables: boys are compared to girls, and younger teenagers (13-16) to older ones (17-20). For educational track, we distinguish the three main types of secondary education in Belgium: ASO, TSO and BSO. ASO or General Secondary Education is theory-oriented and prepares students for higher education, whereas BSO or Vocational Secondary Education is practice-oriented, preparing students for a manual profession. TSO or Technical Secondary Education constitutes a more hybrid in-between level.

### 3.2. Dependent Variables

We selected 11 different linguistic features which are all deviations from the formal writing standard.

The largest set of features consists of 7 expressive markers which convey emotional or social involvement (see Hilte, Vandekerckhove & Daelemans, 2016 for a detailed analysis of these expressive markers):

1. non-standard capitalization
e.g. *IK ZWEER HET* 'I swear'

2. emoticons and emoji

e.g. *dammn we look so hot* 😃*
* 😂
* 😂
* 😂

3. combinations question and exclamation marks
e.g. *Echt?!* 'Really?!'

4. deliberate repetition ('flooding') of letters
e.g. *yeeesss* 'yes'

5. deliberate repetition ('flooding') of punctuation marks
e.g. *Wat???* 'What???'
6. onomatopoeic rendering of laughter
e.g. *hahaha*

7. rendering of kisses and hugs
e.g. *Dankje xxx* 'Thank you xxx'

For orality (i.e. the underlying chatspeak principle to write 'as you speak'), we take one feature into account:

8. non-standard Dutch lexemes (informal Dutch, ranging from colloquial speech, regiolect/dialect words and slang to youth language ...)
e.g. *vertel het sebiet* (std. Dutch: *vertel het straks*, 'tell it later')

The economy principle ('make your message as concise as possible') was operationalized with:

9. chatspeak abbreviations and acronyms (i.e. non-standard shortened words or phrases)
e.g. *Omg yes* (full version: **Oh my god** yes)

The final category contains two features that do not really fit into one of the three chat principle categories but are characteristic of (Dutch) CMC and atypical of formal standard writing:

10. English words<fnref n="1" /> (used in a Dutch conversation)
e.g. *echt nice* 'really nice'

11. Discourse markers # (*hashtag*, to indicate a topic or express a feeling about it) and @ (*at*, to directly address one person in a group conversation)
e.g. #bestfriends
e.g. @sarah

# 4. Corpus and Methodology

## 4.1 Corpus

The corpus consists of Flemish teenagers' informal chat conversations and contains 2 885 084 tokens or 488 014 posts. The number of chatters in the corpus is 1384. The distributions for the social variables age, gender and education can be found in Table 1. We note that (dialect) region is a quasi-constant: almost all tokens (over 96%) are collected from participants living in the central Antwerp-Brabant region. The same holds for medium and year: almost all tokens (over 99%) are extracted from instant (i.e. synchronous) messages on Facebook/Messenger, WhatsApp or iMessage, and the vast majority of the tokens (87%) were produced in 2015-2016. Students consented to donate their conversations, and for minors, parents' consent was asked too. All chat material was anonymized before analysis – the participants' names were replaced by serial numbers, which are linked to the features of their social profiles (e.g. gender).

<table><caption>Table 1: Distributions for gender, age and education.</caption><thead><tr><th>Variable</th><th>Subgroups</th><th>Tokens</th></tr></thead><tbody><tr><td rowspan="2">Gender</td><td>Boys</td><td>985 928 (34%)</td></tr><tr><td>Girls</td><td>1 899 156 (66%)</td></tr><tr><td rowspan="2">Age</td><td>Younger (13-16)</td><td>1 584 373 (55%)</td></tr><tr><td>Older (17-20)</td><td>1 300 711 (45%)</td></tr><tr><td rowspan="3">Education</td><td>General (ASO)</td><td>920 114 (34%)</td></tr><tr><td>Technical (TSO)</td><td>1 213 483 (42%)</td></tr><tr><td>Vocational (BSO)</td><td>751 487 (26%)</td></tr><tr><td>Total</td><td></td><td>2 885 084</td></tr></tbody></table>

## 4.2. Methodology

### 4.2.1. Feature Extraction

All feature occurrences were extracted automatically using Python scripts. For a random test set, the software's output was compared to human annotation, which rendered a satisfying f-score of 0.90 (average for all 11 features).

### 4.2.2. Statistical Language Modeling

We statistically analyze the use of non-standard features by constructing a generalized linear mixed model (GLMM) on a token-level, using the R package 'lme4' (Bates et al., 2017). The GLMM tries to model and predict the response variable, which is the probability of a token containing at least one non-standard feature. As a random effect, we add the chatters' ID to account for individual variation between the participants as well as for their unbalanced contributions.

# 5. Results

## 5.1 Modeling Non-Standardness

Our model of non-standard language use (conceptualized in a binary way, i.e. the probability that a token contains at least one non-standard feature) lets the three social factors age, gender and education interact with each other, while adding a random effect for individual variation among the chatters. Table 2 shows the raw output of the model, i.e. the estimates and significance scores for the different levels of the factors, always in comparison to the reference category (older teenage boys in the General Education System/ASO). To evaluate a factor's significance as a whole (and not just in comparison to the reference group, but to all other levels as well), we performed extra Anova analyses. These results are shown in Table 3. Furthermore, we added extra effect tests using the 'Effects' package in R (Fox et al., 2016; Fox, 2003).

<footnote n="1">¹ Although the use of English words in Dutch conversations could also be related to the orality principle, we argue that it should be</footnote>

<footnote>dealt with separately, since it is indicative of the extent to which youngsters connect with international chat culture.</footnote>
<table><caption>Table 2: Output of the GLMM in comparison to the reference level (older boys ASO).</caption><thead><tr><th></th><th>estimate</th><th>std. error</th><th>z</th><th>p</th><th>sig.</th></tr></thead><tbody><tr><td>(Intercept)</td><td>-1.18023</td><td>0.03466</td><td>-34.06</td><td>&lt; 2e-16</td><td>***</td></tr><tr><td>ageYoung</td><td>0.07991</td><td>0.02159</td><td>3.70</td><td>0.000215</td><td>***</td></tr><tr><td>genderGirls</td><td>-0.15825</td><td>0.04613</td><td>-3.43</td><td>0.000603</td><td>***</td></tr><tr><td>eduBSO</td><td>0.10890</td><td>0.05611</td><td>1.94</td><td>0.052263</td><td>.</td></tr><tr><td>eduTSO</td><td>0.03414</td><td>0.05153</td><td>0.66</td><td>0.507682</td><td></td></tr><tr><td>ageYoung: genderGirls</td><td>0.24210</td><td>0.02486</td><td>9.74</td><td>&lt; 2e-16</td><td>***</td></tr><tr><td>ageYoung: eduBSO</td><td>0.03532</td><td>0.04537</td><td>0.78</td><td>0.436242</td><td></td></tr><tr><td>ageYoung: eduTSO</td><td>0.06508</td><td>0.02485</td><td>2.62</td><td>0.008826</td><td>**</td></tr><tr><td>genderGirls: eduBSO</td><td>0.17528</td><td>0.07487</td><td>2.34</td><td>0.019227</td><td>*</td></tr><tr><td>genderGirls: eduTSO</td><td>0.06882</td><td>0.07001</td><td>0.98</td><td>0.325614</td><td></td></tr><tr><td>ageYoung: genderGirls: eduBSO</td><td>-0.09562</td><td>0.04986</td><td>-1.92</td><td>0.055141</td><td>.</td></tr><tr><td>ageYoung: genderGirls: eduTSO</td><td>-0.06868</td><td>0.02905</td><td>-2.36</td><td>0.018077</td><td>*</td></tr></tbody><tfoot><tr><td colspan="6">Signif. codes: 0'***' 0.001 '**' 0.01 '*' 0.05 '' 0.1 '' 1</td></tr></tfoot></table>

<table><caption>Table 3: Output of the GLMM's Anova.</caption><thead><tr><th></th><th>Chisq</th><th>Df</th><th>Pr(&gt;Chisq)</th><th>Sig.</th></tr></thead><tbody><tr><td>age</td><td>2207.4925</td><td>1</td><td>&lt; 2.2e-16</td><td>***</td></tr><tr><td>gender</td><td>0.4481</td><td>1</td><td>0.503223</td><td></td></tr><tr><td>education</td><td>30.1873</td><td>2</td><td>2.786e-07</td><td>***</td></tr><tr><td>age:gender</td><td>232.6871</td><td>1</td><td>&lt; 2.2e-16</td><td>***</td></tr><tr><td>age:education</td><td>10.6975</td><td>2</td><td>0.004754</td><td>**</td></tr><tr><td>gender: education</td><td>3.0855</td><td>2</td><td>0.213793</td><td></td></tr><tr><td>age:gender:education</td><td>6.5998</td><td>2</td><td>0.036887</td><td>*</td></tr></tbody><tfoot><tr><td colspan="5">Signif. codes: 0'***' 0.001 '**' 0.01 '*' 0.05 '' 0.1 '' 1</td></tr></tfoot></table>

## 5.2. Effects and Interpretation

The different effects captured by the model are visualized in Figure 1 (i.e. the plot of the three-way interaction), on which the predicted probabilities for non-standardness are plotted for the different social variables.

The red dotted lines, representing the younger teenagers, are consistently higher than the black solid ones, representing the older teenagers, across all gender and education groups. This is a very consistent main effect for age, which is also significant (see Table 3, also confirmed by additional effect tests in which the other variables are kept constant): the younger teenagers (13-16 years old) use significantly more non-standard features than the older ones (17-20 years old). These results correspond to our expectations: non-standard language use is said to peak during adolescence, around the age of 16 ('the adolescent peak' – which is also the boundary between our two age categories) and thus decreasing as the teenagers age (Holmes, 1992, 184).

<figure><img src="image_5.png" /><figcaption>Figure 1: Interaction age*gender*education for non-standardness.</figcaption></figure>

The two panels in Figure 1 represent the two genders, with the males on the left and females on the right. Clearly, at a younger age (compare the red dotted lines), girls outperform boys in non-standardness in each education type. However, this is no longer true at an older age, where girls only very slightly outperform boys in the Vocational System but use fewer non-standard features in the General and Technical Education Systems. The Anova (Table 3) and additional effect tests reveal that there is no significant main effect for gender, i.e. the model does not predict significantly different probabilities for non-standard features for girls compared to boys. However, the interactions between gender and age and between gender, age and education are significant. Consequently, gender is still an important factor in the model, as it is part of higher-order (interaction) terms which significantly impact on the response variable: in other words, in order to truly capture the gender effect, age and education have to be included in the analyses. As for the interaction between age and gender, Figure 1 shows that the decrease in non-standardness as the adolescents age is much stronger for the girls than for the boys. Again, these results correspond to our expectations, as in previous research, girls were found to converge more towards the adult standard as they grew older than boys (see Eisikovits, 2006, 43-44). Eisikovits ascribes this different age pattern to a difference between (working class) boys' and girls' attitude towards society when they graduate from high school; while accepting the responsibilities of adulthood, girls converge towards mainstream societal norms, whereas boys more strongly insist on their autonomy (2006, 48-49). We note that these preference patterns are confirmed for middle class participants by Vandekerckhove (2000, 302).
Finally, the Anova (Table 3) and additional effect tests reveal a significant main effect for type of education. The separate data points in Figure 1 reveal a consistent pattern across gender and age groups: the lowest probability of non-standardness is predicted for the teenagers in the General (theoretical) System (ASO), followed by the ones in the Technical (hybrid) System (TSO), and then by the ones in the Vocational (practical) System (BSO). Furthermore, additional effect tests showed that all three types are significantly different from each other. A possible explanation for these results concerns the level of proficiency in and familiarity with written standard Dutch in the different education types, which might increase as the school type becomes more theoretical. Apart from linguistic *skills*, *attitudinal* differences might be a factor too, as the prototypical chatspeak features may simply be more popular and considered to be *cooler* among students in the Vocational System. (For a more thorough analysis, see Hilte, Vandekerckhove & Daelemans, *fc*) Finally, the differences between education types are larger for the girls than for the boys. This could indicate a higher sensitivity for girls for this social factor.

Below, we present an alternative way to visualize the effects captured by the model. Figure 2 facilitates grasping the different ‘age*gender’ interactions in the three school systems. Clearly, in the more theoretical education types (General and Technical Education / ASO resp. TSO), the gender effect is opposite in the two age groups. At a younger age, the girls outperform the boys in non-standardness, but at a later age, they use fewer non-standard markers. In the Vocational System (BSO), however, the girls outperform the boys in non-standardness at a younger age and use more or less the same number of non-standard markers at an older age. Although there is still an interaction (girls’ use of non-standard features decreasing more strongly than boys’), it is much less outspoken than the ‘classical’ pattern in the other two education types, and results in a convergence of the two genders rather than in an (opposite) divergence.

<figure><img src="image_6.png" /><figcaption>Figure 2: Interaction age*gender*education for non-standardness, alternative visualization.</figcaption></figure>

## 6. Conclusion

We modeled Flemish adolescents' non-standard language use in their informal computer-mediated communication. We found that age, gender and education interact and influence the occurrence of non-standard features. Whereas the impact of age (lower frequencies in older teenagers' CMC) and education (lower frequencies for students in more theoretical educational tracks) might confirm expectations based on related research, the gender findings are quite surprising. The observation for the main effect of gender (i.e. no significant difference) does not correspond to previous research, as female language use is generally found to be more 'standard-oriented'.

However, this might be related to the operationalization of the notion of non-standardness in our research design: clearly expressive markers, which appear to be highly favored by women (see Hilte, Vandekerckhove & Daelemans, 2016, 31-32), might behave completely different in terms of indexing non-standardness from markers of regional non-standard speech. Consequently, a priority for future research will be the declustering of the set of ‘non-standard’ features and the consequent construction of different models for each subset, so that potential different preference patterns for these subsets can emerge. Still then, as we have shown in this preliminary study, gender cannot be studied in isolation, since the interactions with age and education are a prerequisite for a correct and nuanced evaluation of its impact.

## 7. Acknowledgements

We thank Giovanni Cassani, Dominiek Sandra and Koen Plevoets for their help and advise with the statistical modeling.

## 8. References

Androutsopoulos, J. (2011). Language Change and Digital Media: A Review of Conceptions and Evidence. In T. Kristiansen & N. Coupland (Eds.), *Standard Languages and Language Standards in a Changing Europe*. Oslo: Novus Press, pp. 145--161.

Bates, D., Maechler, M., Bolker, B., & Walker, S. (2017). Package 'lme4'. URL: https://cran.r-project.org/web/packages/lme4/lme4.pdf

Eisikovits, E. (2006). Girl-Talk/Boy-Talk: Sex Differences in Adolescent Speech. In J. Coates (Ed.), *Language and Gender: A Reader*. Oxford: Blackwell, pp. 42--54.

Fox, J. (2003). Effect Displays in R for Generalised Linear Models. *Journal of Statistical Software*, 8(15).

Fox, J., Weisberg, S., Friendly, M., Hong, J., Andersen, R., Firth, D., & Taylor, S. (2016). Package ‘effects’. URL: http://www.r-project.org, http://socserv.socsci.mcmaster.ca/jfox/

Hilte, L., Vandekerckhove, R., & Daelemans, W. (2016). Expressiveness in Flemish Online Teenage Talk: A
Corpus-Based Analysis of Social and Medium-Related Linguistic Variation. In D. Fišer, & M. Beisswenger (Eds.), *Proceedings of the 4th Conference on CMC and Social Media Corpora for the Humanities, Ljubljana, Slovenia, 27-28 September 2016*, pp. 30--33.

Hilte, L., Vandekerckhove, R., & Daelemans, W. (forthcoming). Adolescents' Social Background and Non-Standard Writing in Online Communication.

Holmes, J. (1992). *An Introduction to Sociolinguistics*. London / New York: Longman.

Parkins, R. (2012). Gender and Emotional Expressiveness: An Analysis of Prosodic Features in Emotional Expression. *Griffith Working Papers in Pragmatics and Intercultural Communication*, 5(1), pp. 46--54.

Vandekerckhove, R. (2000). Structurele en sociale aspecten van dialectverandering. De dynamiek van het Deerlijkse dialect. Gent: Koninklijke Academie voor Nederlandse Taal- en Letterkunde.

Wolf, A. (2000). Emotional Expression Online: Gender Differences in Emoticon Use. *Cyberpsychology & Behavior*, 3(5), pp. 827--833.
# Investigating Interaction Signs across Genres, Modes and Languages: The Example of OKAY

Laura Herzberg, Angelika Storrer

Department of German Linguistics, University of Mannheim

Schloss, Ehrenhof West, D-68131 Mannheim

E-mail: lherzber@mail.uni-mannheim.de, astorrer@mail.uni-mannheim.de

## Abstract

This paper presents results of a case study that compared the usage of OKAY across genre types (Wikipedia articles vs. talk pages), across modes (spoken vs. written language), and across languages (German vs. French CMC data from Wikipedia talk pages). The cross-genre study builds on the results of Herzberg (2016), who compared the usage of OKAY in German Wikipedia articles with its usage in Wikipedia talk pages. These results also form the basis for comparing the CMC genre of Wikipedia talk pages with occurrences of OKAY in the German spoken language corpus FOLK. Finally, we compared the results on the usage of OKAY in German Wikipedia talk pages with the usage of OKAY in French Wikipedia talk pages. With our case study, we want to demonstrate that it is worthwhile to investigate interaction signs across genres and languages, and to compare the usage in written CMC with the usage in spoken interaction.

**Keywords:** interaction signs, cross-lingual CMC study, Wikipedia talk pages

## 1. Background and Motivation

Interaction signs are elements that are not integrated in the syntactic structure of utterances, but serve as devices for discourse management: they can be used to express reactions to a partner's utterances or to display emotions. The category "interaction sign" was defined in Beißwenger et al. (2012), building on the grammar framework of the "Grammatik der deutschen Sprache" (henceforth GDS), which already included interjections ("hm", "well", "oh my god", "oops") and responsives ("yes", "no", "okay"). This framework was expanded with categories which have similar functions as interjections and responsives but typically occur in computer-mediated communication (CMC), e.g. emoticons, addressing terms (@USERNAME), action words ("lol", "grin") etc. (cf. Beißwenger et al., 2012).

The focus of this paper is on OKAY, which is an interesting object of study because it is used in many languages with a wide range of functions (cf. Figure 2). OKAY is not a CMC-specific interaction sign (like emoticons or "lol"), but is used in both written and spoken language. In our studies, the meta-lemma OKAY represents the different variants of spelling and pronunciation. Using OKAY as an example, we want to demonstrate that comparing the usage of interaction signs in speech corpora with its usage in written CMC corpora can yield interesting results. In our cross-genre and cross-lingual studies, we also explore which spelling variants are preferred by the users and whether these variants are compliant with spelling rules.

Most of the previous work on OKAY deals with spoken language: Schegloff/Sacks (1973) investigate OKAY in pre-closing sequences of spoken conversation. The studies of Beach (1993) and Bangerter et al. (2003) examine the usage of OKAY in phone calls. Levin/Gray (1983) describe the usage of OKAY in lecturer's presentations. Condon/Čech (2007) investigate the role of OKAY in decision making processes, comparing

face-to-face interaction with CMC data. All these studies deal with the usage of OKAY in English. Studies on other languages are rare, although OKAY is used in many languages: Delahaie (2009) studies the usage of OKAY as an agreement marker in the learning of French as a foreign language. Kaiser (2011) investigates the usage of OKAY in German spoken doctor-patient communication. Cirko (2016) describes the usage of OKAY in German examination talks.

In our paper, we investigate the usage of OKAY across genre types (comparing CMC with text genres), across modes (comparing the usage in spoken interaction and written CMC), and across languages (comparing the same CMC genre in German and French). The cross-genre study builds on the results of Herzberg (2016), who compared the usage of OKAY in German Wikipedia article talk pages with its usage in Wikipedia articles. These results also form the basis for contrasting the usage of OKAY in written CMC and in spoken interaction (using data from the German speech corpus FOLK). Finally, we compare the usage of OKAY in the German Wikipedia talk pages with its usage in French Wikipedia talk pages.

## 2. Cross-genre study

### 2.1 Corpus Data

For the cross-genre study we compared data from two linguistically annotated Wikipedia corpora (cf. Margaretha/Lüngen, 2014): a corpus with German Wikipedia articles (Wiki-A-de; appr. 797 million tokens) and a corpus with German Wikipedia article talk pages (Wiki-D-de; 310 million tokens). Wikipedia articles represent a text genre (monologous structure, standard language etc.), while talk pages have features of CMC genres (dialoguous structure, informal writing style with non-standard language etc., cf. Storrer, 2017). The two corpora were downloaded from the Institute for the
German Language (IDS) and queried in RAPIDMINER-KOBRA<fnref n="1" />.

## 2.2 Classification: Categories and Procedure

(1) First, we analysed the frequency of different spelling variants of OKAY in both corpora. The assumption was that OKAY is quite more frequent in the CMC corpus Wiki-D-de due to the dialogical structure and conversation-like nature of Wikipedia discussions. Different spelling variants had been queried and combined to draw the samples for the article and the talk pages (cf. Herzberg, 2016 for details). Since not all spelling variants occurred equally in both corpora, the two samples differ in their totals. The procedure resulted in a Wiki-A-de sample of 6,336 OKAY occurrences in total, and in a Wiki-D-de sample of 10,554 occurrences in total. All occurrences in both samples were manually checked and the false positives were sorted out. The distribution of true and false positives is illustrated in Table 1. It shows absolute frequencies as well as normalised frequencies as pmw values (occurrence per million words). Three types of false positives were distinguished: a) OKAY was mentioned as a word, e.g. in an article about interjections, b) OKAY was cited, e.g. in a song title or c) spelling variants of OKAY were homographic with abbreviations of proper names, such as a volcano ("Ok [...] is a shield volcano in Iceland")<fnref n="2" />.

(2) Second, each spelling variant had been investigated individually. The two categories “conformant vs. non-conformant” and “speedy vs. non-speedy” served as objects of study. Because CMC writing is less norm-conformant, we expected to find spelling variants that do not comply with the German spelling norm. In German okay, Okay, o.k. and O.K. are the norm-conformant spelling variants<fnref n="3" />. It has to be noted, that the variants “o. k.” and “O. K.” have to display a blank space between O and K to be norm-conformant. Therefore, the spelling variants ok, OK, Ok, o.k., and O.K. are non-conformant spellings.

Another hypothesis was that CMC users prefer “speedy” spelling variants (ok, Ok, OK) because speed writing is a general feature of CMC. We classified ok, OK and Ok as “speedy” and all other variants as “non-speedy”.

## 2.3 Results and Discussion

(1) The results of the cross-genre frequency study on OKAY are presented in Table 1.

<table><caption>Table 1: Distribution of true and false positives of OKAY in the German Wikipedia.</caption><thead><tr><th rowspan="2"></th><th colspan="2">true positives</th><th colspan="2">false positives</th></tr><tr><th>abs.</th><th>pmw</th><th>abs.</th><th>pmw</th></tr></thead><tbody><tr><td>Wiki-A-de</td><td>25</td><td>0.03</td><td>6,311</td><td>7.92</td></tr><tr><td>Wiki-D-de</td><td>8,248</td><td>26.62</td><td>2,306</td><td>7.44</td></tr></tbody></table>

As expected, OKAY is quite more frequent in the CMC corpus (talk pages) than in the text corpus (Wikipedia articles). Interestingly, the two corpora considerably differ in their number of false positives: in the CMC sample, 2,306 (21.8 %) were classified as being false positives. In the text sample 6,311 (99.6 %) occurrences of OKAY turned out to be false positives: only 25 (0.4 %) of all occurrences were true positives. As 25 items is a very small data set, we restricted our studies on the frequency of spelling variants on the CMC sample.

(2) Table 2 shows the results of the studies on norm-conformance and frequency of OKAY spelling variants in the German CMC corpus Wiki-D-de and in the French CMC corpus Wiki-D-fr. In this section, we discuss the results of the German data; the cross-lingual aspects are treated in section 4.3⁴.

<table><caption>Table 2: Frequency and norm-conformance of OKAY spelling variants in German and French.</caption><thead><tr><th rowspan="2">Spelling Variant</th><th colspan="2">Norm-conformance</th><th colspan="2">Frequency Wiki-D-de</th><th colspan="2">Frequency Wiki-D-fr</th></tr><tr><th>DE</th><th>FR</th><th>abs.</th><th>pmw</th><th>abs.</th><th>pmw</th></tr></thead><tbody><tr><td>OK</td><td></td><td></td><td>17,796</td><td>57.43</td><td>9,281</td><td>67.69</td></tr><tr><td>ok</td><td></td><td></td><td>16,048</td><td>51.78</td><td>5,476</td><td>39.94</td></tr><tr><td>Ok</td><td></td><td></td><td>15,431</td><td>49.79</td><td>7,495</td><td>54.67</td></tr><tr><td>okay</td><td>✓</td><td></td><td>8,421</td><td>27.17</td><td>86</td><td>0.63</td></tr><tr><td>Okay</td><td>✓</td><td></td><td>8,287</td><td>26.74</td><td>163</td><td>1.19</td></tr><tr><td>o. k.</td><td>✓</td><td></td><td>96</td><td>0.31</td><td>0</td><td>0</td></tr><tr><td>O. K.</td><td>✓</td><td></td><td>86</td><td>0.28</td><td>6</td><td>0.04</td></tr><tr><td>o.k.</td><td></td><td></td><td>80</td><td>0.26</td><td>0</td><td>0</td></tr><tr><td>O.K.</td><td></td><td>✓</td><td>21</td><td>0.07</td><td>3</td><td>0.02</td></tr></tbody></table>

The results in Table 2 clearly support the assumption that non-conformant variants are more frequently used than the conformant ones in the German CMC corpus. Moreover, the results support the hypothesis that the three speedy variants ok, OK, and Ok are preferred, although they do not conform to German spelling rules.

# 3. Cross-modal study

## 3.1 Corpus Data

There are significant differences between the usage of interaction signs in spoken and written language. In spoken interaction intonation plays a crucial role in interpreting a positive, negative, or doubting evaluation expressed by an interaction sign. Interaction signs are relevant for organizing turn-taking in spoken interaction: hearers use interaction signs to encourage the floor holder to continue (so-called “continuers”, cf. Schegloff, 1982 p. 81). While these functions have been widely investigated in spoken language (see cited works in section 1), studies on CMC or cross-modal studies are still rare.

<footnote n="1">¹ Details on the queries are provided in Herzberg (2016).</footnote>

<footnote n="2">² Cf. https://en.wikipedia.org/wiki/Ok_(volcano) [15.06.17].</footnote>

<footnote n="3">³ Cf. Duden-Rechtschreibung, 2013 p. 781.</footnote>

<footnote n="4">⁴ We integrated the French data in Table 2 in order to save space. The table presents absolute frequencies as well as normalised frequencies as pmw values.</footnote>
In our cross-modal case study, we compared data from the CMC corpus Wiki-D-de (310 million tokens), with spoken interaction data taken from the German FOLK corpus (1.9 million tokens). The speech data was queried automatically via the DGD.

## 3.2 Classification: Categories and Procedure

(1) We distinguished between two main functional categories: OKAY as a syntactic unit (used in predictive, adverbial, attributive function or as a noun) and OKAY as an interaction sign (used in responsive, reactive, interrogative, and structural function). In Herzberg (2016), all true positives in the Wiki-D-de sample described in section 2 (8,248 occurrences in total, cf. Table 1) have been classified as follows: 5,045 (61.2 %) occurrences are used as interaction signs and 3,203 (39.8 %) as syntactic units. An interesting finding concerns the functional category “responsive”, i.e. a (positive) answer to a polar question. This function is described as being the main function of OKAY in the German grammar GDS (1997) p. 63. However, the study revealed that only a very small amount (20 occurrences, i.e. 0.4 % of all interactive OKAY occurrences) in the examined Wiki-D-de data were used as responsives. We assumed that this mismatch between the Grammar description and our data was due to the fact that the classifications in this Grammar refers to the usage of OKAY in spoken interaction. We thus used data from the FOLK corpus to investigate whether the responsive function of OKAY is a main function in spoken interaction. We manually checked how often the responsive function of OKAY was used in a FOLK corpus sample with 1,500 occurrences of OKAY.

(2) In a second study we analysed the positions of OKAY in samples taken from FOLK, Wiki-D-de, and the French Wikipedia talk pages (Wiki-D-fr; 137 million tokens) with 500 occurrences in each sample. These samples only contain true positives; false positives have been manually sorted out. The data has then been classified according to four positional categories: initial (directly at the beginning of a ⁵post/utterance); middle (within a post/utterance); final (end of a post/utterance) and standalone (OKAY forms post/utterance).

## 3.3 Results and Discussion

(1) The analysed FOLK sample had a similar outcome as the study on the CMC data: Only 15 (1 %) of the 1,500 examined occurrences are used as responsives. In both corpora, the responsive function which is claimed to be the main function in the GDS grammar description, only rarely occurs in both written CMC and spoken language. These results demonstrate that it is worthwhile to further evaluate assumptions about the functions of interaction signs on the basis of corpus data.

<figure><img src="image_7.png" /><figcaption>Figure 1: Positional distribution of OKAY in spoken and CMC interaction<fnref n="6" />.</figcaption></figure>

(2) The results of our comparison of positions in Figure 1 reveal significant differences between the two modes. Whereas OKAY is used variably and nearly equally often in German utterances across the three categories “standalone” (32.8 %), “middle” (27.0 %) and “initial” (25.4 %), the German CMC data presents a different picture: OKAY is preferably used at the beginning (47.5 %) and within (41.6 %) a post. These two positions make up nearly 90 % of all investigated occurrences. Interestingly, the positional distribution patterns in the German and French CMC data are quite similar<fnref n="7" />. There are two possible explanations for these results that have to be verified in further work: (a) The standalone position is typical for “continuers” (see above) and the final position is typical for the usage of OKAY as a tag question. Both functions are particularly relevant for organising turn-taking in spoken interaction. This may explain the lower rate of standalone and final positions in the CMC data, where turn-taking mechanism are substituted by other mechanisms of interaction management (cf. Beißwenger, 2008). (b) As it is shown in Figure 3, OKAY is mostly used as an interaction sign in the speech data from FOLK. In the two CMC corpora however, OKAY is also used as a syntactic unit. These syntactically integrated units (nouns, adverbials, predicatives) often occur in a middle position. This may be one factor to explain the higher rate of middle positions in CMC corpora in the results presented in Figure 1.

To get a clearer image of the differences in the usage of OKAY in spoken and written interaction, we want to annotate the functional and the positional categories presented in Figure 2 on two different layers and explore correlations between the positional and functional categories in more detail.

<table><caption>Figure 2: Formal and functional annotation categories.</caption><thead><tr><th colspan="5">OKAY</th></tr></thead><tbody><tr><td>1. Layer<br>Position</td><td>initial</td><td>middle</td><td>final</td><td>standalone</td></tr><tr><td>2. Layer<br>Function</td><td>Interactive<br>Responsive<br>grammatical<br>Predicative</td><td>Reactive</td><td>Interrogative</td><td>Structural</td></tr><tr><td></td><td></td><td>Adverbial</td><td>Nominal</td><td>Attributive</td></tr></tbody></table>

<footnote n="5">⁵ Following the proposals of the TEI CMC group, we use the term “posts” for units in CMC interaction (cf. Beißwenger et al., 2012). The segments in spoken interaction are termed as “utterances”.</footnote>

<footnote n="6">⁶ The figures contain absolute frequencies of true positives.</footnote>

<footnote n="7">⁷ Cross-lingual aspects are treated in section 4.3.</footnote>
# 4. Cross-lingual study

## 4.1 Corpus Data

In our cross-lingual study we compared the corpus of German Wikipedia talk pages (Wiki-D-de; 310 million tokens) with a corpus of French Wikipedia talk pages (Wiki-D-fr; 137 million tokens). Both corpora are available within the German Reference Corpus DeReKo at the IDS. The data has been queried automatically via COSMAS II.

## 4.2 Classification: Categories and Procedure

(1) In a first study, we manually classified two samples of German and French, each containing 500 OKAY occurrences, in three categories: “syntactic units”, “interaction signs” (cf. 3.2) and “others”. We assumed that the usage of OKAY as a syntactic unit, signalling a deeper integration of the loan word in the host language system, is less frequent in the French corpus.

(2) The focus on the second investigation was again on spelling variants. We expected that the speedy and non-conformant variants are also preferred in the French CMC corpus. Similar to the study of German, *ok*, *OK* and *Ok* were classified as speedy variants whereas *okay*, *Okay*, *o.k.*, *O.K.*, *o. k.* and *O. K.* are non-speedy variants. In French, only the variant *O.K.* is conformant<fnref n="8" />. Therefore, the variants *okay*, *Okay*, *ok*, *OK*, *Ok*, *o.k.*, *O. K.* and *o. k.* were classified as being non-conformant.

(3) We integrated the French CMC sample in our cross-modal study on positional differences between spoken and written CMC, described in section 3.2., to investigate the distribution patterns in the CMC data of both languages.

## 4.3 Results and Discussion

(1) The results in Figure 3 support our assumption that OKAY is less frequently used as a syntactic unit in French than in German<fnref n="9" />.

<figure><img src="image_8.png" /><figcaption>Figure 3: Functional distribution of OKAY.</figcaption></figure>

In the German CMC data, 135 (28.4 %) occurrences of OKAY were tagged as syntactic units and 330 (69.3 %) occurrences as interaction signs. In French, 285 (58.9 %) occurrences were classified as interaction signs whereas 50 (10.3 %) were tagged as syntactic units. The French

data included a considerably high amount of OKAY occurrences that could not clearly be classified as either interactive or syntactic (149 occurrences; 30.8 %)<fnref n="10" />. The aforementioned results had been achieved by manually checking and tagging the samples. Using a tagger that automatically assigns part-of-speech (POS) tags to distinguish between interactive or grammatical usages of OKAY did not achieve satisfactory results. The applied taggers either tagged all occurrences as being interactive, e.g. in FOLK, or as being grammatical, e.g. in Wiki-D-de. Studying OKAY exemplifies that there is still a need for improvement in the field of POS-tagging (cf. Lüngen et al., 2016 for details).

(2) The results of our cross-lingual study on the frequency of spelling variants are presented in Table 2 of section 2.2. The most frequent variants in the corpora are non-conformant, but support speed-writing. In both languages the non-speedy variants including a space (*o. k.* and *O. K.*) are rarely used. The variants *okay* and *Okay* are less frequent in French than in German, where these forms are norm-conformant.

(3) In terms of the positional distribution, shown in Figure 1, there is a clear distinction between speech and written CMC corpora. The distributional patterns in the French and the German CMC data do not differ to a vast extent and therefore seem to be language independent.

# 5. Conclusion

We investigated the usage of OKAY across genre types (German Wikipedia articles vs. talk pages), modes (German spoken vs. written interaction), and across languages (German vs. French CMC). The cross-genre study illustrated that OKAY is quite more frequently used in the CMC genre and that speedy writing variants are preferred over rule-conformant non-speedy ones. The cross-lingual study revealed that the grammatically integrated functions of OKAY occur more frequently in the German than in the French data. This may be an effect of the French language policy that recommends to avoid English loan elements. By comparing the frequency of spelling variants we found that the “speedy” variants are highly preferred in French and in German, although these variants are not rule-conformant. The cross-modal study showed that the function of a responsive, described as being the main function in the GDS grammar, is rarely used in both written and spoken corpora. It is thus worthwhile to investigate the functions of OKAY on the basis of corpus data. The results of the comparison of positional categories in Figure 2 revealed that the distribution patterns in the French and the German CMC corpora are quite similar, whereas the patterns in the CMC corpora differ considerably from the distribution in the spoken language corpus FOLK. Further work will study the usage of interaction signs in spoken and written CMC interaction on the basis of a more fine-grained annotation of functional categories.

<footnote n="8">⁸ Cf. Le Petit Robert, 2017 p. 1736.</footnote>

<footnote n="9">⁹ The samples contain absolute frequencies of true positives.</footnote>

<footnote n="10">¹⁰ Examples are posts containing elliptical constructions like “Donc, OK pour moi” or “OK pour la date de la mort”.</footnote>
## 6. References

Bangerter, A.; Clark, H.H.; Katz, A.R. (2003). Navigating Joint Projects in Telephone Conversations. In *Discourse Processes* 37, pp. 1-23.

Beach, W. (1993). Transitional regularities for 'casual' "Okay" usages. In *Journal of Pragmatics* 19, pp. 325-352.

Beißwenger, M. (2008). Situated Chat Analysis as a Window to the User's Perspective: Aspects of Temporal and Sequential Organization. In J. Androutsopoulos, M. Beißwenger (Eds.), *Data and Methods in Computer-Mediated Discourse Analysis* (= Language@Internet 5).

Beißwenger, M.; Ermakova, M.; Geyken, A.; Lemnitzer, L.; Storrer, A. (2012). A TEI Schema for the Representation of Computer-mediated Communication. In *Journal of the Text Encoding Initiative (jTEI)*. Issue 3/2012 (DOI: 10.4000/jtei.476).

Delahaie, J. (2009). Oui, voilà ou d'accord? Enseigner les marqueurs d'accord en classe de FL. In *Synergies Pays Scandinaves* 4, pp. 17-34.

Duden-Rechtschreibung (2013). *Duden – Die Grammatik*. 26., völlig neu erarbeitete und erweiterte Auflage. Berlin: Bibliographisches Institut GmbH. (= Band 1 – Der Duden in 12 Bänden).

Condon, S.L.; Čech, C.G. (2007). OK, next one: Discourse markers of common ground. In A. Fetzer, K. Fischer (Eds.), *Lexical Markers of Common Grounds*. London: Elsevier, pp. 18-45.

GDS (1997). *Grammatik der deutschen Sprache*. Zifonun, G.; Hoffmann, L.; Strecker, B.; et al. (Eds.). 3 Bände. Berlin/New York: de Gruyter.

Herzberg, L. (2016). *Korpuslinguistische Analyse interaktiver Einheiten: das Beispiel okay*. Master thesis. University of Mannheim.

Kaiser, J. (2011). *okay in ärztlichen Gesprächen – eine linguistische Gesprächsanalyse*. State examination thesis. Ruprecht-Karls-University Heidelberg.

Le Petit Robert (2017). *Dictionnaire Alphabétique Et Analogique De La Langue Française*. Paris: Dictionnaires Le Robert.

Levin, H.; Gray, D. (1983). The Lecture's OK. In *American Speech* 58, pp. 195-200.

Lüngen, H.; Beißwenger, M.; Herold, A.; Storrer, A. (2016). Integrating corpora of computer-mediated communication in CLARIN-D: Results from the curation project ChatCorpus2CLARIN. In S. Dipper, F. Neubarth, H. Zinsmeister (Eds.), *Proceedings of the 13th Conference on Natural Language Processing (KONVENS 2016)*, pp. 156-164.

Margaretha, E.; Lüngen, H. (2014). Building linguistic corpora from Wikipedia articles and discussions. In *Journal of Language Technologie and Computational Linguistics JLCL* 29(2), pp. 59-83.

Schegloff, E.A.; Sacks, H. (1973). Opening up Closings. In *Semiotica* 8, pp. 289-327.

Schegloff, E.A. (1982). Discourse as an interactional achievement: Some uses of 'uh huh' and other things that come between sentences. In D. Tannen (Ed.),

*Analyzing discourse: Text and talk*, Washington, DC: Georgetown University Press, pp. 71-93.

Storrer, A. (2017). Grammatische Variation in Gespräch, Text und internetbasierter Kommunikation. In M. Konopka, A. Wöllstein (Eds.), *Grammatische Variation. Empirische Zugänge und theoretische Modellierung*, Berlin/New York: de Gruyter, pp. 105-125.

Corpus tools and resources:

COSMAS I/II: Corpus Search, Management and Analysis System. Institute for the German language Mannheim, http://www.ids-mannheim.de/cosmas2/.

DeReKo: Das Deutsche Referenzkorpus. Institute for the German language Mannheim, http://www.ids-mannheim.de/kl/projekte/korpora/.

DGD: Datenbank gesprochenes Deutsch. Institute for the German language Mannheim, http://agd.ids-mannheim.de/folk.shtml.

FOLK: Forschungs- und Lehrkorpus für gesprochenes Deutsch. Institute for the German language Mannheim, http://dgd.ids-mannheim.de/dgd/pragdb.dgd_extern.welcome.

RAPIDMINER-KOBRA: RapidMiner Software, www.rapidminer.com and KobRA Plugin, http://www.kobra.tu-dortmund.de/mediawiki/index.php?title=Software.

Wiki-A-de: Corpus with all articles of the German Wikipedia (Version 17.11.2015). Institute for the German language Mannheim, http://corpora.ids-mannheim.de/pub/wikipedia-deutsch/2015/.

Wiki-D-de: Corpus with all article talk pages of the German Wikipedia (Version 17.11.2015). Institute for the German language Mannheim, http://corpora.ids-mannheim.de/pub/wikipedia-deutsch/2015/.

Wiki-D-fr: Corpus with all article talk pages of the French Wikipedia (Version 17.11.2015). Institute for the German language Mannheim, http://corpora.ids-mannheim.de/pub/wikipedia-fremdspr/2015/.
# Anonymisation of the Dortmund Chat Corpus 2.1

Harald Lüngen¹, Michael Beißwenger², Laura Herzberg³, Cathrin Pichler²

¹Institut für Deutsche Sprache, R5 6-13, D-68161 Mannheim

²Universität Duisburg-Essen, Institut für Germanistik, Berliner Platz 6-8, D-45127 Essen

³Universität Mannheim, Germanistische Linguistik, Schloss, D-68131 Mannheim

E-mail: luengen@ids-mannheim.de, michael.beisswenger@uni-due.de, lherzber@mail.uni-mannheim.de, cathrin.pichler@tu-dortmund.de

## Abstract

As a consequence of a recent curation project, the Dortmund Chat Corpus is available in CLARIN-D research infrastructures for download and querying. In a legal expertise it had been recommended that standard measures of anonymisation be applied to the corpus before its republication. This paper reports about the anonymisation campaign that was conducted for the corpus. Anonymisation has been realised as categorisation, and the taxonomy of anonymisation categories applied is introduced and the method of applying it to the TEI files is demonstrated. The results of the anonymisation campaign as well as issues of quality assessment are discussed. Finally, pseudonymisation as an alternative to categorisation as a method of the anonymisation of CMC data is discussed, as well as possibilities of an automatisation of the process.

**Keywords:** Corpora, Computer-mediated communication, Anonymisation

## 1. Introduction

In the CLARIN-D curation project "Integration of the Dortmund Chat Corpus into CLARIN-D" (Lüngen et al., 2016), a legal expertise was sought to clarify issues concerning the possibility to republish the material, which had been collected between 2004-2008 partly without written consent of the participants, in the CLARIN-D infrastructures (Lüngen et al., 2016; Beißwenger et al., 2017). The legal expertise was composed by the company iRights.law (Berlin), which specialises on legal issues concerning digital media. Below is a summary of the recommendations that were given to the hosting institutions. They follow from considerations of personality and data protection rights. Other legal statuses like copyright and intellectual property rights were also considered in the expertise, but are not discussed here.

1. Remove the chats which originally came from psycho-social counselling platforms completely (10 out of 480 logfiles)

2. Grant access to chats collected from closed platforms only for authorised scientific use

3. Apply "standard measures" of anonymisation to all chat files

a. Randomise/replace host names, nicknames, place names, and platform names

b. Remove or permute the time stamps

Medlock (2006) distinguishes between categorisation and pseudonymisation. The latter is a procedure of permutation or replacement of the sensitive references with instances of the same ontological category (e.g. replacing occurrences of the male name *Holger* with the male name *Werner*. To get an idea of possible types and categories of sensitive references in chat and CMC, we also looked at the anonymisation in previous CMC corpus projects. Among them was no project dealing with chat data, however an email corpus (Medlock, 2006), a

Facebook corpus (DiDi, 2015), two SMS corpora (Panckhurst, 2013; Ueberwasser, 2015), as well as one spoken conversation corpus (FOLK, cf. Winterscheid, 2015). In all the CMC corpora, anonymisation was realised as categorisation, only in the spoken corpus was it realised as pseudonymisation.

## 2. Anonymisation by categorisation

Categorisation preserves some of the information so that a corpus can still reasonably be used for linguistic analyses. It implies the replacement of a sensitive reference with a placeholder string that indicates its ontological category, such as *Person_name* or *Place_name*. Since most of the references that had to be anonymised in the chat corpus are names, we firstly included the five named entity categories PER, ORG, LOC, GPE, OTH from the TüBa-D/Z treebank (Telljohann et al., 2004), which had already been used in NER experiments with DeReKo (Bingel & Haider, 2015) in our category inventory. Because these five NER categories are relatively coarse-grained, and because the annotations in the original chat corpus resource contained already more specific information, we extended the set by the categories NICK (for nickname, a subcategory of PER) and ROOM (for chat room). Moreover, we added the category GEO_DE for a noun or adjective derived from a LOC or a GPE (a union of the categories _GeoNE_ and _GeoADJA_ in DiDi, 2015). Besides these, three categories for more formal references were added: URL (for a web address), email (for an email address), and NUMBER (for any kind of referencing number, see Table 1 for examples). Following Winterscheid (2015) and DiDi (2015), we also introduced the two rarer categories IMPLICIT (for an implicit reference), and CITATION (for a quote by which an individual might be identified). The 13 anonymisation categories used are shown in Table 1 with their definitions, example(s), and the source of or inspiration for the category.
<table><thead><tr><th>#</th><th>Category short form</th><th>Category long form and definition</th><th>Examples</th><th>Source</th></tr></thead><tbody><tr><td>1</td><td>PER</td><td>PERSONNAME:<br>A first name or second name or a sequence out of first name and second name</td><td>“Erwin”,<br>“Meike”,<br>“Anna Hein”</td><td>TüBa-DZ<br>(Telljohann<br>et al., 2004)</td></tr><tr><td>2</td><td>NICK</td><td>NICKNAME:<br>User name chosen by a chat participant, or a variant therof</td><td>“superman”,<br>“lela2”, “Tiger”, “Lan5”,<br>“KainPech”</td><td>DO chat<br>corpus</td></tr><tr><td>3</td><td>ORG</td><td>ORGANISATIONNAME:<br>Company (e.g. the employer of a participant), sports club, institute, university etc.</td><td>“RUB”,<br>“John Deere”,<br>“ASV Schifferstadt”</td><td></td></tr><tr><td>4</td><td>LOC</td><td>LOCATIONNAME:<br>A place or area which is not a GPE, e.g. mountains, valleys, rivers, roads, motorways, etc.</td><td>“Augustaanlage”,<br>“Neckar”, “Königstuhl”<br>“A6”</td><td></td></tr><tr><td>5</td><td>GPE</td><td>GEOPOLITICALENTITTYNAME:<br>A geo-political entity, i.e. a place or area of which the borders are officially defined, i.e. cities, municipalities, countries, states, suburbs etc., including their spelling variants and abbreviations</td><td>“Mannheim”, “NRW”,<br>“Italien” “doaaadmund”<br>“DO”</td><td>TüBa-DZ</td></tr><tr><td>6</td><td>GEO_DE</td><td>GEODERIVATIONNAME:<br>Noun or adjective that is morphologically derived from a (mostly GPE or LOC) name and which expresses an association or a quality (adjectives) or a group or inhabitants (noun)</td><td>“Mannheimer”,<br>“Mannheimerinnen”,<br>“Gelbfüßler”</td><td></td></tr><tr><td>7</td><td>OTH</td><td>OTHERNAME:<br>Residual category for all sensitive names and references that cannot be categorised otherwise</td><td>“unicum”</td><td></td></tr><tr><td>8</td><td>ROOM</td><td>CHATROOMNAME:<br>Name of a chatroom</td><td>“Welcome”,<br>“blue”</td><td>DO chat<br>corpus</td></tr><tr><td>9</td><td>URL</td><td>WWWURL:<br>Web address</td><td>“http://wwwids-mannheim.de/”</td><td></td></tr><tr><td>10</td><td>EMAIL</td><td>EMAIL:<br>Email address</td><td>“fix@ids-mannheim.de”</td><td></td></tr><tr><td>11</td><td>NUMBER</td><td>NUMBER:<br>Any number or code that can be associated with a person: e.g. house number, serial number, postal code, telephone number, passport number, account number, IP address, password</td><td>“0621/1581418”,<br>“10.0.1.45”, “68161”</td><td></td></tr><tr><td>12</td><td>IMPLICIT</td><td>IMPLICIT:<br>Implicit reference: Revealing descriptions and pieces of information from which the identity of a chat participant or a third party can be inferred (e.g. someone's job)</td><td>„IT-Operator“</td><td>FOLK<br>(Wintersche<br>id, 2015)</td></tr><tr><td>13</td><td>CITATION</td><td>CITATION:<br>A quote, e.g. from a song, which can be used to identify a chat participant or a third party</td><td></td><td>FOLK</td></tr></tbody></table>
<caption>Table 1: Anonymisation categories in the Dortmund Chat Corpus 2.1.</caption>

```xml
<post auto="false" rends="color:#808080" synch="#f427" type="standard" who="#A26" xml:id="m576">
  <w lemma="wollen" type="VMFIN" xml:id="m576.t1">willst</w>
  <w lemma="du" type="PPER" xml:id="m576.t2">du</w>
  <w lemma="cine" type="ART" xml:id="m576.t3">ne</w>
  <w lemma="Therapie" type="NN" xml:id="m576.t4">therapie</w>
  <ref corresp="#A31" type="addressingTerm">
    <w lemma="@" type="ADRIND" xml:id="m576.t5">@</w>
    <w type="NE" xml:id="m576.t6">[MALE-PARTICIPANT-A31_]</w>
  </ref>
  <w lemma="ich" type="PPER" xml:id="m576.t7">ich</w>
  <w lemma="studieren" type="VVFIN" xml:id="m576.t8">studier</w>
  <ref type="IMPLICIT">
    <w type="NE" xml:id="m576.t9">[IMPLICIT-1_]</w>
  </ref>
</post>
```

<caption>Listing 4: Anonymisation of an implicit reference.</caption>

Below are the stats of the annotation of categories in the hole corpus - remember that the chat corpus contains roughly 1 million tokens

<table><caption>Table 2: Occurrences of categories for sensitive references in Dortmund Chat Corpus 2.1.</caption><thead><tr><th>Category</th><th># Occurrences</th></tr></thead><tbody><tr><td>NICK:</td><td>30,022</td></tr><tr><td>ROOM:</td><td>2,409</td></tr><tr><td>OTH:</td><td>1,819</td></tr><tr><td>URL:</td><td>1,742</td></tr><tr><td>GPE:</td><td>1,309</td></tr><tr><td>PER:</td><td>838</td></tr><tr><td>ORG:</td><td>741</td></tr><tr><td>GEO_DE:</td><td>178</td></tr><tr><td>NUMBER:</td><td>169</td></tr><tr><td>IMPLICIT:</td><td>130</td></tr><tr><td>LOC:</td><td>107</td></tr><tr><td>EMAIL:</td><td>50</td></tr><tr><td>CITATION:</td><td>5</td></tr><tr><td>Σ =</td><td>39,519</td></tr></tbody></table>

### 3. Anonymisation campaign and results

The major bulk of the anonymisation by categorisation process was carried out by four student assistants of Mannheim University, Duisburg-Essen University and the Institute for the German Language (IDS), Mannheim. The sensible references that had not been pre-annotated were identified and annotated with the category inventory in Table 1 using the “author mode” of the XML editor Oxygen. The campaign lasted from August till December 2016 (five months) and took approximately 625 hours of manual annotation work. Subsequently an XSLT post-processing step was implemented to insert the replacement strings and to provide TEI annotation in terms of the elements `<name>` and `<ref>`.

Listings 1-4 contain XML code snippets that show what the result of the anonymisation looks like in CLARIN-D TEI (cf. Lüngen et al., 2016).

```xml
<partDesc>
  <!-- 1301005 -->
  <listPerson>
    <!-- ... -->
    <person role="celebrity" xml:id="A03">
      <persName type="nickname">Günther Beckstein</persName>
      <sex evidence="estimated">male</sex>
    </person>
    <!-- ... -->
    <person role="moderator" xml:id="A04">
      <persName type="nickname">[MALE-MODERATOR-A04_]</persName>
      <sex evidence="estimated">male</sex>
    </person>
    <!-- ... -->
    <person role="participant" xml:id="A07">
      <persName type="nickname">[FEMALE-PARTICIPANT-A07_]</persName>
      <sex evidence="estimated">female</sex>
    </person>
    <!-- ... -->
    <person role="participant" xml:id="A08">
      <persName type="nickname">[PARTICIPANT-A08_]</persName>
      <sex evidence="estimated">unknown</sex>
    </person>
    <!-- ... -->
  </listPerson>
</partDesc>
```

Listing 1: Anonymisation of metadata (participant list). Mentions of celebrities and politicians are from the public sphere and are not anonymised.

```xml
<post auto="false" rends="color:lime" type="event" who="#A14" xml:id="m487">
  <name corresp="#A14" type="nickname">
    <w lemma="[_PARTICIPANT-A14_]" type="NE" xml:id="m487.t1"> [PARTICIPANT-A14_]</w>
  </name>
  <w lemma="werden" type="VVFIN" xml:id="m487.t2">wird</w>
  <w lemma="schlecht" type="ADJD" xml:id="m487.t3">schlecht</w>
</post>
```

Listing 2: Anonymisation of a nickname without role entry in participant list.

```xml
<w lemma="auch" type="ADV" xml:id="m40.t1">auch</w>
<w lemma="bei" type="APPR" xml:id="m40.t2">bei</w>
<w lemma="die" type="ART" xml:id="m40.t3">den</w>
<name type="GEO_DE">
  <w lemma="[_GEODERIVATIONNAME_]" type="NN" xml:id="m40.t4">[_GEODERIVATIONNAME-4_]</w>
</name>
```

Listing 3: Anonymisation of a derivation of a name (like *Düsseldorfern*).

### 4. Quality assessment

To get some impression of the agreement between our coders, we asked all four of them to annotate the chat logfile with the ID 1102001 immediately after the training session. The file contains 675 chat posts, and the union of the sensitive references identified by the four coders contained 126 references. This figure was subsequently used as N (number of items to be coded) in the Kappa calculation described in the following. We calculated Fleiss' Kappa using the IRR package for the programming language R (function *kappam.fleiss*)<fnref n="1" />. The agreement between the four coders was κ=0.582. According to the interpretation scale by Landis & Koch (1997), this corresponds to “moderate” agreement.

A closer inspection of the disagreements revealed that

<footnote n="1">¹ Cf. https://cran.r-project.org/package=irr [22.06.2017].</footnote>
Coder 1 was the source of an unusual great deal of the mismatches. For instance, in cases where the name of a person was given fully as first name + last name, Coder 1 had, contrary to the training instructions, always annotated them as separate instances. There were at least 15 such first name + last name combinations. (Coder 1 was subsequently made aware of this error i.e. before anonymising her share of the corpus.)

We additionally calculated Fleiss' Kappa among the remaining three coders Coder 2, Coder 3, and Coder 4 only. For them, Kappa was found to be κ=0.827 after all, which according to Landis & Koch (1977) can be interpreted as "almost perfect agreement".

Because of these results, we believe that our method is appropriate for achieving an anonymisation of the chat corpus that conforms to legal standards as put forth in the legal expertise. Ideally, one would have had more material annotated by all four coders, and calculated the inter-rater reliability not only at the beginning of the annotation campaign but also in the middle and at the end of it. Moreover, it would have been interesting if we had even checked for *intra*-rater reliability of each or at least some of the coders. Unfortunately, in the present campaign there was no more time for coding, coordination, and evaluation work. But for future projects, this should be kept in mind.

## 5. Discussion

During the campaign we noticed that for several chats, a more fine-grained category scheme would have been desirable from a discourse linguist's point of view. In some chats, for instance, many locations were mentioned, and in the anonymised version one would have wished to have more information on the kind of location discussed (e.g. restaurant, shop, school) available. On the other hand, a more complex encoding scheme usually affects inter-rater agreement to the negative. A simple solution to this could be to allow coders to add free information strings.

Another way to address all kinds of problems with the category scheme could be to aim for corpus *pseudonymisation* such as in the spoken conversation corpus FOLK. However, to achieve a full pseudonymisation is even more costly than our anonymisation by categorisation method, and besides has its own drawbacks, such as the possibility of introducing inconsistencies in the dialogue.

Finally, it seems obvious that we need an automatisation of the anonymisation process. A campaign like the one described above is simply not feasible for larger corpora, and the need for anonymisation is potentially given with many kinds of CMC, even web corpora. However, the task is non-trivial and comprises more than standard Named Entity Recognition. The anonymised Dortmund Chat Corpus 2.1 can also serve as training data for future developments of corpus anonymisation tools.

## 6. References

Beißwenger, M., Lüngen, H., Schallaböck, J., Weitzmann, J.H., Herold, A., Kamocki, P., Storrer, A., Wildgans, J. (2017, to appear). Rechtliche Bedingungen für die Bereitstellung eines Chat-Korpus in CLARIN-D: Ergebnisse eines Rechtsgutachtens. In Michael Beißwenger (Ed.), *Empirische Erforschung internetbasierter Kommunikation*. Berlin/Bew York: de Gryuter (Empirische Linguistik).

Bingel, J., Haider, T. (2014). Named Entity Tagging a Very Large Unbalanced Corpus. Training and Evaluating NE Classifiers. In *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC '14)*. Reykjavik: European Language Resources Association (ELRA).

DiDi (2015). *Beschreibung der Anonymisierung im DiDi-Korpus*. Available online http://www.eurac.edu/en/research/autonomies/commul/Documents/DiDi/DiDi_anonymisation_DE.pdf.

Fleiss, J.L. (1971). Measuring nominal scale agreement among many raters. In *Psychological Bulletin* 76 (5), pp. 378–382.

Landis, J.R., Koch, G.G. (1977). The measurement of observer agreement for categorical data. In *Biometrics* 33, pp. 159–174.

Lüngen, H., Beißwenger, M., Herold, A., Storrer, A. (2016). Integrating corpora of computer-mediated communication in CLARIN-D: Results from the curation project ChatCorpus2CLARIN. In S. Dipper, F. Neubarth & Heike Zinsmeister (Eds.), *Proceedings of the 13th Conference on Natural Language Processing (KONVENS 2016)*, pp. 156-164.

Medlock, B. (2006). An Introduction to NLP-based Textual Anonymisation. In *Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC '06)*. Genoa: European Language Resources Association (ELRA). Available online http://www.lrec-conf.org/proceedings/lrec2006/.

Panckhurst, R. (2013). A large SMS Corpus in French: from Design and Collation to Anonymisation, Transcoding and Analysis. In *Procedia - Social and Behavioral Sciences* 95, pp. 96-104.

Telljohann, H., Hinrichs, E., Kübler, S. (2004). The TüBa-D/Z Treebank: Annotating German with a Contet-Free Backbone. In *Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004)*.

Ueberwasser, S. (2015). *Anonymisation (SMS4Science.ch)*. Available online https://sms.linguistik.uzh.ch/bin/view/SMS4Science/A nonymisation.

Winterscheid, J. (2015). GAIS Web: Maskierung. Document in the IDS Project. Available online http://prowiki.ids-mannheim.de/bin/view/GAIS/MasK ierung.
# Emoticons as multifunctional and pragmatic Resources:
a corpus-based Study on Twitter

Stefania Spina

University for Foreigners of Perugia
Piazza Fortebraccio, 4 – Perugia, Italy
E-mail: stefania.spina@unistrapg.it

## Abstract

Emoticons play an important role in digital written communication: they can serve as markers either of emotions or social relationship and familiarity, and they can intensify or downgrade the pragmatic force of a text.

The aim of this study is to investigate the use of emoticons in Twitter by Italian users, and to verify, by relying on corpus data and on statistical methodologies, some of the prevailing opinions on the use of emoticons: that they are technically-driven resources, that they are mostly used by young people, and more often by females, and that they are superficial and easy ways of expressing emotions using images instead of words.

A mixed-effects model analysis has shown that the use of emoticons on Twitter is affected by a complex interaction of cultural, technological, situational and sociolinguistic variables.

**Keywords:** emoticons, Twitter, mixed-effects models.

## 1. Introduction

Emoticons (the graphic signs, such as the smiley face, that often accompany digital written communication) are an integral part of digital culture since its beginnings: they have followed its development over the last decades, evolving alongside with the rapid spread of new written communication environments, such as social media or text messaging systems.

Many studies have outlined the key role of emoticons in digital written communication (e.g. Amaghlobeli, 2012; Baron, 2009; Danesi, 2016; Derks, Bos & von Grumbkow, 2007; Dresner & Herring, 2010; Spina, 2016; Vandergriff, 2014; Walther & D'Addario, 2001; Yus & Yus, 2014): as people use writing more and more instead of face to face interactions or phone calls, the need for overcoming limitations in communicating emotional tone arises. The widespread use of emoticons allows to convey nonlinguistic information that in face-to-face communication is expressed through facial expression and other bodily indicators. Emoticons, therefore, are primarily “emotion icons”: additional opportunities to convey emotions through the use of graphic symbols, directly mapped onto facial expressions.

The role of emoticon in digital written communication, however, is much more nuanced and not limited to the expression of emotions. Following Dresner & Herring (2010), Vandergriff (2014), and Spina (2016), they are developing at least two other important pragmatic functions, that are not necessarily mapped onto facial expressions, or aimed at the expression of emotions:

* the function of social markers of familiarity and empathy. In this sense, they are relational icons, that promote rapport and play a social and affiliative role;

* the function of markers of the pragmatic force of a text, aimed at intensifying or downgrading its meaning. In this function, they are contextualization cues (Gumperz, 1982; Auer, 1992), that provide information on how to interpret the verbal message.

As a consequence, emoticons are multifunctional and highly context-sensitive resources, whose different functions most often tend to overlap and to occur simultaneously within the use of a single emoticon. This claim is illustrated by the examples (1) and (2):

(1)
@user2 Ci vieni in piscina domani?
[@user2 Are you coming to the pool tomorrow?]

@user1 No :-(

(2)
@user2 Hai visto la foto del mio profilo?
[@user2 Have you seen my profile picture?]

@user1 Bellissima!! :-)))
[@user1 So beautiful!! :-)))

In example (1), the sad emoticon serves both as a mitigation resource, aimed at softening the refusal of an invitation, and as a means of expressing regret for this refusal. In example (2), the smiley is both a marker of intensification of the positive emotion expressed verbally by “so beautiful” and graphically by the exclamation marks, and a marker of familiarity, aimed at expressing empathy and friendliness. Emoticons, therefore, are not just a ludic and extralinguistic supplement to language, with the exclusive role of expressing emotions, but rather linguistic resources that
play other important pragmatic functions in digital written communication, such as conveying the intentions of the writer (Tagg, 2012), supporting social relationships among participants, and providing new opportunities for creative expressions.

## 2. Motivation

The main aim of the present investigation was to examine the use of emoticons in Twitter by Italian users. More specifically, it tried to verify, by relying on corpus data and on statistical methodologies, some of the prevailing findings on the use of emoticons in digital written communication: that they are technically-driven resources, whose spread is mainly due to the diffusion of mobile devices (Baron, 2008); that they are mostly used by young people (Merchant, 2001; Tagliamonte & Denis, 2008), and more often by females (Baron, 2008; Huffaker & Calvert, 2005; Spina & Cancila, 2013; Tossel et al., 2012); finally, that they are easy ways of expressing emotions using images instead of words (Provine, Spencer & Mandell, 2007).

Conversely, the nuanced social and pragmatic functions played by emoticons in digital communication suggest that their use and distribution should be affected by a more complex interaction of technological, cultural, situational and sociolinguistic variables.

The questions that this study tried to answer were: what are the variables that, at a discourse level, affect the use of emoticons in Twitter interactions? How is the use of emoticons influenced by these variables?

## 3. Method

To answer these questions, a large corpus of tweets extracted from the Italian timeline was used. The Ita_twitter corpus (Spina, 2016) contains more than 550,000 tweets sent in a time span of seven months (November 2012-May 2013). The 8,842,450 tokens were pos-tagged through an ad hoc version of TreeTagger (Schmid, 1994), purposely trained to automatically detect emoticons.

From the Ita_twitter corpus, a subset written by 290 users was randomly selected. This subset consists of 4,441 tweets and contains information on the authors (sex, geographical provenance), on their level of mastery within Twitter environment (date of registration on Twitter, number of tweets sent), on the technical context (the software device from which each tweet was sent), and on the type of tweet (a simple status update, or the reply to a previous tweet written by someone else). Information on the authors' age was obtained by manually checking each of the 290 profiles. The subset of 4,441 tweets contained 15 different types of ASCII emoticons, that are listed in table 1. Each of the types is represented in the corpus by a number of different graphic forms, depending on the combination of ASCII symbols. The classic smiley :-), for example, is represented by a number of different forms, such as :), :)), :-)), etc.

<table><caption>Table 1: the 15 types of emoticons used in the corpus</caption><thead><tr><td>Emoticon</td><td>Meaning</td></tr></thead><tbody><tr><td>:-)</td><td>smile</td></tr><tr><td>;-)</td><td>wink</td></tr><tr><td>:-()</td><td>sad or frown</td></tr><tr><td>:*</td><td>kiss</td></tr><tr><td>&lt;3</td><td>heart</td></tr><tr><td>*_*</td><td>dazed</td></tr><tr><td>:</td><td>tears of happiness</td></tr><tr><td>^_^</td><td>happy</td></tr><tr><td>:P</td><td>tongue sticking out</td></tr><tr><td>x.x</td><td>dead</td></tr><tr><td>:</td><td>crying</td></tr><tr><td>-.-</td><td>annoyed</td></tr><tr><td>:D</td><td>laughing</td></tr><tr><td>O.O</td><td>surprised</td></tr><tr><td>u.u</td><td>sarcastic</td></tr></tbody></table>

In addition, given that the corpus was pos-tagged and lemmatized, a range of other linguistic information could be added to the selected tweets, including the type of sentence (question, exclamation, etc.), the co-occurrence of other discourse elements relevant to Twitter interactions (hashtags, mentions), and the length of each tweet (in number of tokens).

In order to explore how Italian participants use emoticons in their Twitter interactions, a mixed-effects model analysis was performed on the selected data. Mixed-effects modeling (e.g., Baayen, Davidson, & Bates, 2008) is particularly suited to corpus data (Gries, 2015), because it can integrate multiple categorical and numeric variables (fixed effects), and, at the same time, it can address the idiosyncrasies deriving from the analysis of data produced by the same subjects (random effects).

The mixed-effect model was built using R version 3.3.3 and the R packages lme4 (version 1.1–13; Bates, Maechler, & Bolker, 2012), lmerTest (version 2.0–33), and sjPlot (version 2.3.1). The number of emoticons used in each tweet (range: 0-27, mean: 0.26, sd 0.68) was used as dependent variable, and the following predictors were initially included in the model: the age (range: 16-67, mean: 31.72, sd 10.57) and sex of participants (f: 1443; m: 2998); the device from which the tweets were sent (mobile or desktop); the level of mastery within the Twitter environment (measured as the number of tweets sent from the date of registration to the date of each tweet in the corpus), distributed in five bands, from the lowest (a) to the highest (e); the type of tweet (status update: 1842, or reply: 2599); the number of co-occurring hashtags (range: 0-8, mean: 0.23; sd 0.63); the tweet length in number of tokens
(range: 1-41, mean: 13.85, sd 7.16), and the type of sentence (declarative: 3053, or non-declarative, that is exclamative or interrogative: 1388).

In order to model the individual differences in the use of emoticons, the authors of the tweets were used as random effect, by assuming different random intercepts for each author. In addition, as random slope models allow the predictors to have a different effect for each subject, random slopes were included in the model, with the aim of accounting for the different effects that the “type of tweet” and “sentence” variables have on each single author (Winter, 2013).

Finally, in the model building process, a backward selection approach was adopted, starting with a full model, including all the fixed effects mentioned above, and then dropping one variable at a time, and excluding a variable from the model if non-significant (Gries, 2015).

## 4. Results

Preliminary results show that four predictors affect the use of emoticons in Twitter as fixed effects (see the plot in figure 1). The predictor with the stronger effect is the type of tweet: emoticons occur far more in replies to the tweets of other participants, which are the most interactive form of tweet (Honeycutt & Herring, 2009; Schnoebelen, 2012), than in status updates. This finding clearly confirms the hypothesis that emoticons are one of the linguistic resources that participants rely on when they need support for establishing or maintaining a social relationship with their interlocutors. Replies, that automatically include a mention to the addressee, are in fact one-to-one or one-to-few interactions, even if they also presuppose the presence of the more numerous audience of followers. In this context, emoticons seem particularly suited to convey reactions to opinions or feelings expressed by others.

<figure><img src="image_9.png" /><figcaption>Figure 1: A plot of the estimates of the four significant fixed effects.</figcaption></figure>

Another highly affecting predictor is sex: females use emoticons significantly more than males. This finding confirms previous research in computer-mediated communication (e.g. Vandergriff, 2014; Wolf, 2000; but

Danesi, 2016), according to which “females are the prime users of emoticons” (Baron, 2008:65). The same is not true, however, for the age of participants: this variable is never significant in the model as a fixed effect, and cannot therefore be considered a major predictor of the use of emoticons in Twitter.

The different types of sentence used in the tweets are another significant predictor; the model shows that the presence of exclamative or interrogative sentences, signalled by the ? and ! punctuation marks, produces a decrease in the use of emoticons. This finding could seem surprising, since emoticons are a graphical means of expressing emotions, and exclamatives often convey feelings and emotions, but it can be interpreted as an evidence of the multifunctionality of emoticons: one of their roles in digital written communication is that of syntactic markers, often serving as punctuation in place of traditional punctuation marks (Amaghlobeli, 2012). We can conclude, then, that exclamative and interrogative sentences decrease where emoticons replace question and exclamation marks, as in examples (3) and (4):

(3)
@user ciao... Buon lunedi... Che bell'inizio di settimana :-D

[@user Hello... Have a good Monday! What a great start of the week :-D]

(4)
@user Ma che ci fai per DUE MESI a new york :D

[@user What are you going to do for TWO MONTHS in New York :D]

The last significant fixed effect of the model is the hashtag. Emoticons and hashtags tend to have a complementary distribution: when more emoticons are used, less hashtags are found in the tweets. This finding seems coherent with the respective functions of the two discourse elements: while emoticons express either emotions or familiarity, or mark the pragmatic force of a text, the hashtag has an informative function (marking the topic of a text), or addresses the social need of aggregating communities of participants around a common theme or interest (Zappavigna, 2015). In this sense, emoticons seem to serve the pragmatic function of supporting social relationships among few participants, whereas the hashtag plays an important role in affiliating large masses of people in flows of conversations on shared topics.

Going further with the analysis, the picture described so far gets clearer if interactions between different predictors are considered. The type of tweet and the level of mastery, for example, have an interaction effect on the use of emoticons: while replies always contain a greater number of emoticons, this effect seems to slightly increase if the users are more familiar with Twitter. In the case of status updates, the opposite is true: the less proficient the users, the less emoticons they use. In addition, the mastery of Twitter rules and conventions also interacts with the length of tweets in the effect on the use of emoticons: as shown in Figure 2, low mastery levels (a) produce shorter tweets with less emoticons; medium and high mastery levels (b, c, d and e), conversely, tend to increase the number of
emoticons, together with the text length.

<figure><img src="image_10.png" /><figcaption>Figure 2: A plot of the interaction between mastery and length of tweets in their effect on the use of emoticons.</figcaption></figure>

From the above, we can conclude that the use of emoticons in Twitter is not technically-driven (the predictor “software device” is never significant in the model, neither as a fixed effect nor in interaction with other factors), and it is not exclusive of young people; rather, it is influenced by a number of pragmatic, cultural and sociolinguistic factors interacting with each other. As a result, far from being only an add-on feature or a frivolous way of expressing emotions, emoticons are constitutive of CMC (Vandergriff, 2014), since they are assuming more and more sophisticated social and pragmatic roles.

After this quantitative investigation, a more in-depth and qualitative analysis needs to be conducted on emoticons, in order to investigate in more detail the linguistic context that favors their use, and their distribution among the previously mentioned functions. A specific attention should be paid to the linguistic features that are traditionally associated to the purpose of establishing and maintaining relationships with other participants (discourse markers and personal pronouns, for example) and of modulating the pragmatic force of texts (intensifiers, affective vocabulary, etc.).

## 5. References

Amaghlobeli, N. (2012). Linguistic features of typographic emoticons in SMS discourse. *Theory and Practice in Language Studies, 2*(2), pp. 348--354.

Auer, P. (1992). *The contextualization of language*. Amsterdam: Benjamins.

Baayen, R. H., Davidson, D. J. and Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. *Journal of Memory and Language, 59*, pp. 390--412.

Bates, D. M., Maechler, M. and Bolker, B. (2012). *lme4: Linear mixed-effects models using Eigen and S4*.

Baron, N. (2008). *Always On: Language in an Online and Mobile World*. Oxford: OUP.

Baron, N. (2009). The Myth of Impoverished Signal: Dispelling the Spoken Language Fallacy for Emoticons in Online Communication. In J. Vincent & L. Fortunati

(Eds.), *Electronic Emotion: The Mediation of Emotion via Information and Communication Technologies*. Bern: Peter Lang, pp. 107--135.

Baron, N. (2010). *Alphabet to Email: How Written English Evolved and Where It's Heading*. London: Routledge.

Danesi, M. (2016). *The Semiotics of Emoji. The Rise of Visual Language in the Age of the Internet*. London: Bloomsbury

Derks, D.; Bos, A.E.R and von Grumbkow, J. (2007). Emoticons and social interaction on the Internet: the importance of social context. *Computers in Human Behavior, 23*, pp. 842--849.

Dresner, E., Herring, S.C. (2010). Functions of the Nonverbal in CMC: Emoticons and Illocutionary Force. *Communication Theory, 20*(3), pp. 249--268.

Gries, S. Th. (2015). The most underused statistical method in corpus linguistics: Multi-level (and mixed-effects) models. *Corpora, 10*(1), 95--125.

Gumperz, J. (1982). *Discourse strategies*. Cambridge: CUP.

Honeycutt, C., Herring S.C. (2009). Beyond microblogging: Conversation and collaboration via Twitter. In *Proceedings of the Forty-Second Hawai'i International Conference on System Sciences (HICSS-42)*, Los Alamitos: IEEE Press.

Huffaker, D. A., and Calvert, S. L. (2005). Gender, Identity, and Language Use in Teenage Blogs. *Journal of Computer - Mediated Communication, 10*(2).

Lo, S.K. (2008). The Nonverbal Communication Functions of Emoticons in Computer-Mediated Communication. *CyberPsychology & Behavior, 11*(5), pp. 595--597.

Merchant, G. (2001). Teenagers in cyberspace: An investigation of language use and language change in Internet chatrooms. *Journal of Research in Reading, 24*(3), 293--306.

Provine, R.R.; Spencer, R. and Mandell, D. (2007). Emotional expression online: Emoticons punctuate website text messages. *Journal of Language and Social Psychology, 26*(3), pp. 299--307.

Schmid, H, (1994). Probabilistic part-of-speech tagging using decision trees. In *Proceedings of the International Conference on New Methods in Language Processing*.

Schnoebelen, T.J. (2012). Emotions Are Relational: Positioning and the Use of Affective Linguistic Resources. Ph.D. thesis, Department of Linguistics, Stanford University.

Spina, S. (2016). *Fiumi di parole. Discorso e grammatica delle interazioni scritte in Twitter*. Loreto: StreetLib.

Spina, S., Cancila, J. (2013). Gender issues in the interactions of Italian politicians on Twitter: Identity, representation and flows of conversation. *International Journal of Cross-cultural Studies and Environmental Communication, 2*(2), pp. 147--157.

Tagg, C. (2012). *Discourse of Text Messaging: Analysis of SMS Communication*. London: Continuum.

Tagliamonte, S., Denis, D. (2008). Linguistic ruin? LOL! Instant Messaging and teen language. *American Speech, 83*(1), pp. 3--34.

Tossell, C. C., Kortum, P., Shepard, C., Barg-Walkow, L. H., Rahmati, A., and Zhong, L. (2012). A longitudinal
study of emoticon use in text messaging from
smartphones. *Computers in Human Behavior*, 28(2), pp.
659--663.

Vandergriff, I. (2014). A pragmatic investigation of emoticon use in nonnative/native speaker text chat. *Language@Internet*, 11.

Walther, J., D'Addario, K. (2001). The impacts of emoticons on message interpretation in computer mediated communication. *Social Science Computer Review*, 19, 324--347.

Winter, B. (2013). Linear models and linear mixed effects models in R with linguistic applications. [http://arxiv.org/pdf/1308.5499.pdf](http://arxiv.org/pdf/1308.5499.pdf).

Wolf, A. (2000). Emotional Expression Online: Gender Differences in Emoticon Use. *CyberPsychology & Behavior, 3*(5), pp. 827--833.

Yus, F., and Yus, F. (2014). Not all emoticons are created equal. *Linguagem Em (Dis) Curso*, 14(3), pp. 511--529

Zappavigna, M. (2015). Searchable talk: The linguistic functions of hashtags. *Social Semiotics*, 25(3), pp. 274--291.
# Corpus-Based Analysis of Demonyms in Slovene Twitter

Taja Kuzman, Darja Fišer

Faculty of Arts, University of Ljubljana

Aškerčeva 2, 1000 Ljubljana, Slovenia

E-mail: kuzman.taja@gmail.com, darja.fiser@ff.uni-lj.si

## Abstract

This paper reports on a corpus-based analysis of demonym mentions in the corpus of Slovene tweets. First, we analyze the frequency of demonym mentions for the inhabitants of the European and G8 countries. Then, we focus on the representation of demonyms for residents of Slovenia's neighboring countries: Austria, Italy, Hungary and Croatia. The main topic of the tweets mentioning Croatians, Austrians and Italians is sport, whereas Hungarians occur most often in relation to the Eurovision. Some economic and political issues are also represented, such as the selling of Slovene companies to foreign firms, the refugee crisis and the arbitration procedure between Slovenia and Croatia. A collocation analysis revealed a highly stereotypical treatment of the neighboring nations and hostility of some Slovene Twitter users to inhabitants of Slovenia's neighboring countries.

**Keywords:** demonyms, nationalities, Twitter, discourse analysis, Slovene

## 1. Introduction

A corpus of user-generated content, especially of tweets, offers an insight into people's beliefs, opinions and attitudes, including attitudes towards residents of other countries. This paper presents an analysis of demonyms (i.e. nouns, used to denote inhabitants of a particular city, country etc.) for the nations which are members of the European Union and of non-European G8 nations that are mentioned in the corpus of Slovene tweets Janes Tweet v4.0 in order to analyze how often Slovene Twitter users talk about other nationalities and in which contexts. Next, a detailed analysis of the representation of the neighboring nationalities was performed in order to establish the general attitude of Slovene Twitter users towards their neighbors.

## 2. Related Work

Phrases that appear together multiple times provide cultural information and analyzing them can “provide empirical evidence of how the culture is expressed in lexical patterns” (Stubbs, 1996: 169). It is therefore not surprising that many corpus-based discourse analyses have been conducted to observe how people present other nations in written text.

For instance, Bang (2008) examined the representation of foreign countries in the corpus of US news reports. The premodifiers of the keywords ‘country’, ‘countries’, ‘nation’ and ‘nations’ were analyzed, and collocates indicating verbal and mental actions of Arab and European leaders were examined. Furthermore, the lexical collocates of ‘said’ and the grammatical collocates of keywords ‘China’, ‘North Korea’, ‘South Korea’ and ‘Japan’ were analyzed. The study revealed that the representation of foreign counties in US news reports is characterized by stereotyping and asymmetry (ibid.).

Similarly, Tarasheva (2009) used critical discourse analysis to study the representation of Bulgaria in a corpus of articles, published on the BBC website.

Articles about Bulgaria were compared to the ones about Belgium, Portugal, Finland and Denmark in a comparable corpora. The research examined the topics of the articles, most frequent keywords and collocations. The results showed that events from Bulgaria are presented differently than those from the other examined countries: articles about crime appear much more often and the most frequent keywords indicate that Bulgarians are mainly portrayed as crime victims. Tarasheva (2009) concludes that “negative coverage for Bulgaria is deliberately sought and achieved”.

Our study differs from other corpus-based work mentioned above in that it does not examine texts ordered, authored and edited by professionals but rather unsolicited user-generated content posted by the general public.

In contrast to the abundance of corpus-based studies of representations of countries, representations of inhabitants have not yet received much attention. However, the complex topic regarding the Slovenes' attitudes towards their neighbors has been the subject of many academic works. Throughout history, Slovenes lived in multicultural countries – until the early 20th century in the Austro-Hungarian Empire and then in Yugoslavia until the 1990s (Zupančič & Arbeiter, 2016). Furthermore, during the world wars, they were occupied by Italians, Austrians and Hungarians. Hence, Slovenes began to perceive themselves as inferior to their neighbors. Moreover, they perceived them as their enemies and felt threatened by them (Romih, 2013). Thus, Slovenes have become introverted and developed negative attitudes towards their neighbors in order to feel superior to them as well as to strengthen their nationalistic feelings (Šabec, 2007; Zupančič & Arbeiter, 2016). The growth of negative attitudes has also been influenced by the media in former Yugoslavia which tended to portray other nations as crude and violent (Zupančič & Arbeiter, 2016). Today, Slovenes still distrust their neighbors, especially Croatians, who are perceived to be the least trustworthy peoples from former
Yugoslavia, according to surveys in 2009 and 2010 (Salihović, 2012).

### 3. The Janes v4.0 Tweet Corpus

The Janes v4.0 Tweet corpus is a subcorpus of Slovene user-generated corpus Janes (Fišer et al., 2016), which contains tweets, written by Slovene Twitter users in the period June 2013–July 2016. The corpus contains 107 million tokens and has been richly linguistically annotated (rediacritization, word-form normalization, part-of-speech tagging and lemmatization) and enriched with metadata, obtained directly from the Twitter API (author, title, time of post, number of retweets and favorites), but also through specialized processes, e.g. sentiment (“neutral”, “positive” or “negative”), the gender of the author, the type of the user (“private” for individuals or “corporate” for companies, news agencies etc.) and the linguistic and technical level of (non)standardness of the text.

### 4. Demonyms in the Slovene Twittosphere

#### 4.1 Subcorpus

The study was performed in the Sketch Engine concordancer. For the purposes of our study, we constructed a subcorpus of tweets, written by individuals (annotated as “private”) in the Slovene language. The subcorpus contains 77,250,014 tokens.

Since we were interested in opinions of the general public, we only examined private users' tweets in order to exclude tweets from companies or news outlets that often have a persuasive function, trying to influence the readers' opinion or attract customers.

#### 4.2 Methodology

In the first part of the study, we examined the frequency of demonym mentions for inhabitants of all European

nations that were part of European Union in April, 2017 (including Great Britain) and of non-European members of the G8 (Canada, Japan, Russia and the USA). Due to length restrictions of this paper, only official demonyms as they occur in the Slovene orthography manual *Slovenski pravopis* (Toporišič et al., 2014) were analyzed. We examined the occurrence of both masculine and feminine form of demonyms.

#### 4.3 Results

As can be seen from Figure 1, Slovene Twitter users most frequently mention their southern neighbors, Croatians, much more often than inhabitants of other neighboring countries. After Croatians, Slovene tweets most frequently feature residents from the most influential nations of the world—Germany, Russia and the United States of America—which is not surprising as the actions of these countries have a profound influence on the rest of the world. Interestingly, Greeks also occur frequently: regarding a random sample of tweets, we could presume that Slovene Twitter users mostly mention Greeks in connection with the economic crisis in Greece and when commenting their decisions regarding the European Union, as they have an important impact on the economic and political situation in the whole European Union. The least frequent demonyms represent residents of smaller European nations, such as Luxembourg, Cyprus, Malta etc.

Feminine forms of all nationality names are rather rare, which is not surprising as in Slovene the masculine form of the demonym is used as the generic noun that includes both men and women. The only feminine form that stands out is the form for ‘Slovene woman’ *Slovenka*. It must also be taken into account that when users generalize actions of members of their own nation, they likely substitute ‘Slovenes’ by ‘us’. That could be the reason why the frequency of the demonym ‘Slovenes’ (*Slovenci*) is lower than frequency of ‘Croatians’, ‘Germans’ etc.

<figure><img src="image_11.png" /><figcaption>Figure 1: Frequency of Selected Demonyms in Slovene Twitter Corpus</figcaption></figure>
# 5. Representations of the Neighboring Nations

## 5.1 Methodology

In the second part of the study, the representations of demonyms for Slovenia's neighboring nations were compared. The keywords *Avstrijec*, *Avstrijka* (masculine and feminine form for 'Austrian'), *Italijan*, *Italijanka* (masculine and feminine form for 'Italian'), *Madžar*, *Madžarka* (masculine and feminine form for 'Hungarian') and *Hrvat*, *Hrvatica* (masculine and feminine form for 'Croatian') were examined in terms of the users that mentioned them (frequency of different users, their gender), the annotated sentiment of the tweets, the number of retweets and favorites and the topics of the tweets. Furthermore, the collocations of these keywords with nouns, adjectives and verbs were analyzed. The aim of this part of the study was to analyze how often Slovenes mention their neighbors and in connection with which topic, whether these tweets receive a lot of attention and what the general attitude of Slovenes towards their neighbors is.

The same subcorpus and concordancer were used as in Section 4. The analysis was conducted using the metadata in the corpus. Collocation analysis was performed with the Word Sketch feature in the Sketch Engine. However, it was limited to masculine forms of the nationality names as the frequency of the feminine forms was too low. All collocations that appeared five times or more were examined. Topics of the tweets in which the relevant nationalities were mentioned were deducted from the accompanying hashtags. Tweets without hashtags were not considered in this final step.

## 5.2 Results

### 5.2.1 Metadata

As was already shown in Section 4, Slovene Twitter users most often talk about Croatians: 6,285 tweets were found that contain either masculine or feminine form of this demonym. The second most mentioned neighbors of Slovenes are Italians with 2,231 hits, closely followed by Austrians with 2,068 hits, while Hungarians are mentioned in only 952 tweets. Tweets mentioning Hungarians had the highest frequency of different users, which means that a user rarely wrote about Hungarians more than once; whereas there is the largest number of users who recurrently wrote about Croatians.

A very small amount of all tweets containing demonyms was retweeted (7) or favorited (20) more than 20 times. Tweets mentioning Croatians were retweeted or favorited the most, which is not unusual, given that 'Croatian' is the most frequent demonym mentioned by Slovene Twitter users. Interestingly, the most retweeted (47 times) and favorited (119 times) tweet does not refer to Croatians, but to Austrians. As feminine forms of demonyms rarely appear, it is not surprising that the most

retweeted post, containing feminine form of a demonym, was retweeted only 17 times, mentioning Italian women. Approximately three quarters of tweets, featuring masculine forms, were written by men. Surprisingly, feminine forms are much more frequently used by feminine users (*Hrvatica* 'Croatian woman'—39% of authors of tweets were females, *Italijanka* 'Italian woman'—52%, *Avstrijka* 'Austrian woman'—45% and *Madžarka* 'Hungarian woman'—24%).

While the sentiment of tweets containing the masculine form of demonyms are mostly negative, tweets with the feminine form are mostly neutral, except tweets containing *Avstrijka* 'Austrian woman', which are mostly negative. However, in comparison with others, the demonym for Austrian women also has the largest percentage of positive tweets (25%). Positive tweets, which represent 17%–25% of all hits, are the least frequent for all examined nationality names. Interestingly, more than half of the tweets about Hungarians are negative, which makes this nationality the most negatively presented, according to the sentiment annotation.

### 5.2.2 Collocations

The analysis of demonym + noun collocations revealed not only that analyzed demonyms almost exclusively appear in coordination with other demonyms, mostly with demonyms for residents of Slovenia's neighboring countries. Interestingly, Croatians co-occur with Slovenes much more often than the other three demonyms. Croatians also frequently occurs in coordination with demonyms for inhabitants from the Balkans (Serbians and Bosnians). Italians and Austrians frequently co-occur with Germans. Hungarians appear more often in connection with Italians, Croatians, Austrians, Czechs and Slovaks than with Slovenes.

Due to a low frequency count, no adjective + demonym collocations that pass the frequency threshold (5) were found for *Madžar* 'Hungarian'. On the other hand, *Hrvat* 'Croatian', *Avstrijec* 'Austrian' and *Italijan* 'Italian' collocate with various different adjectives, which indicate how differently they are represented in Slovene Twitter.

‘Croatian’ collocates with adjectives that are otherwise associated with Slovenes: *podalpski* ‘sub-alpine’, *alpski* ‘alpine’ and *brdski* ‘from the hills’. These adjectives are used in order to shock readers and to declare that Slovenes are becoming Croatians, or acting as them, as in the tweet “Unfortunately, too many Slovenes are actually Alpine Croatians.” (*Žal je preveč Slovencev v resnici Alpskih Hrvatov.*). Furthermore, most adjectives that collocate with ‘Croatian’ are used ironically. Such adjective is ‘poor’ (*ubog*) as in “Poor Croatians are left with only 1,000 km of coast…” (*Ubogim hrvatom ostane samo še 1000 km obale …*) Positive adjectives ‘grand’ (*veliki*) and ‘dear’ (*dragi*) are also used ironically. Another adjective that also occurs frequently with this keyword is ‘guilty’ (*kriv*). It mostly appears in ironic
tweets in which users mock Slovene tendency to blame
Croatians for everything, for example "Listening to the
news reports, one would say that Croatians are guilty for
the unpreparedness of our government." (Po poročanju
medijev bi človek rekel, da so za nepripravljenost naše
vlade krivi Hrvati.) Furthermore, 'Croatian' also
collocates with 'true' (pravi) in tweets from which
stereotypes about Croatians can be easily presumed.
Such example is "Refugees are not true Croatians. True
Croatians never flee" (Begunci niso pravi Hrvati. Pravi
Hrvat nikoli ne beži).

In contrast, ‘Italian’ and ‘Austrian’ do not appear in
collocations with adjectives that are used ironically.
‘Italian’ mostly collocates with ‘loud’ (glasen), which is
generally perceived as a negative trait. That can be seen
from the following example: “So a coffee in peace
changed into ‘a coffee in a coffee shop, filled with loud
Italians.’ Yay” (In kava v miru se je spremenila v ‘kava v
kaficu polnem glasnih italijanov’. Yay). Interestingly, a
collocation ‘old Italian’ (star Italijan) also occurs quite
often, mostly with a negative connotation, as in “What,
can these old Italians smell in which sauna is a woman.
Suddenly, a whole bunch of them is next to her” (Kva ti
stari italijani zavohajo v keri savni je ženska. Naenkrat
jih je cel kombi ob njej).

The keyword ‘Austrian’ frequently collocates only with
one adjective, which is ‘rich’ (bogat). It appears mostly
in its superlative form, for instance as in “This year, the
richest Austrian is 80 times richer than the richest
Slovene” (Najbogatejši Avstrijec je letos 80-krat
bogatejši od najbogatejšega Slovenca).

As the direct object, the nouns ‘Croatian’ and ‘Italian’
frequently appear with the verb ‘to defeat’ (premagati).
This collocation appears in connection with sport and
Slovene Twitter users mostly hope that their team or
foreign teams would beat Croatians or Italians and tweet
about it with excitement when it happens. A collocation
‘to have a Croatian for neighbor’ (imeti Hrvata za
soseda) also occurs quite often. Generally, there is not
enough context to determine whether this is meant in a
positive or negative way. However, there are some very
telling examples which are clearly negative, such as
“Who needs an enemy when you have a Croatian for a
neighbor!” (Kdo rabi sovražnika če imaš Hrvata za
soseda!). Furthermore, the verb ‘to hate’ also co-occurs
relatively frequently with Croatians, but it was
discovered that it actually occurs in only one sentence
that had been then retweeted by different users: “Who
doesn’t hate Croatians, ain’t Slovene” (Kdor ne sovraži
Hrvatov, ni Sloven’c—an allusion to a Slovene popular
soccer fan slogan “Who doesn’t jump, ain’t Slovene”).
The keywords ‘Croatian’ and ‘Austrian’ often collocate
with the verb ‘to sell’ (prodati), as Slovene Twitter users
mention or disapprove the fact that many Slovene firms
have been sold to Croatian and Austrian companies.

The keyword ‘Italian’ frequently collocates with the verb
‘support’ (navijati), connected with sport. In most
analyzed tweets, Twitter users declare (sometimes
surprised) that they support the Italian team. Such

example is “I see that you support the Italian football
team. And I support the Croatians. Who would thought
so?!” (navijaš za italijane u fuzbalu, vidim. Jst za hrvate.
Kdu bi si mislil..?!) In contrast to that, ‘Italian’ also quite
often appears with ne marati, meaning ‘dislike’.
However, these tweets seem much less negative than
tweets about Croatians and some have a positive turn, as
in “I don’t like Italians, but today I supported them” (ne
maram italijanov ampak danes sem za njih navijal).

5.2.3 Topics

As can be already presumed from the collocations of the
keywords with verbs, frequency analysis of hashtags
shows that the topic of a majority of the tweets
mentioning ‘Croatians’, ‘Austrians’ and ‘Italians’ is sport
(e.g. #eurobasket, #sochi...) The only exception are
Hungarians, for which the most frequent topic is
Eurovision, also a popular topic in tweets with the other
three demonyms. In terms of politics, a number of
tags relate to the arbitration procedure to define
border between Slovenia and Croatia, as well as to
refugees.

6. Conclusion

In this paper we examined demonym mentions in the
corpus of Slovene tweets. The results showed that
Slovene Twitter users mostly talk about their southern
neighbors, Croatians. According to sentiment annotation,
tweets comprising masculine forms of demonyms for
Slovenia's neighboring countries are mostly negative,
while feminine forms mostly occur in neutral tweets. The
collocation analysis revealed that Croatians are generally
disliked by the Slovene Twitter users, occurring in ironic
or negative context that presents them as unwanted
neighbors and reveals deeply rooted stereotypes. Italians
are presented as being sometimes unpleasant, but still
more likeable than Croatians. When referring to
Austrians, Slovene Twitter users mostly connect them
with being rich (or richer than Slovenes). Hashtag
analysis revealed that Slovenes predominantly mention
these nationalities in connection with sport. Some events
and political issues are also represented, such as the
selling of Slovene companies to Croatian and Austrian
firms, the refugee crisis and the arbitration procedure
between Slovenia and Croatia.

The analysis was sometimes difficult as some errors in
annotation occurred due to polysemy and multilinguality
issues (e.g. Danka ‘Danish woman’ or danka ‘rectum’,
Japonka ‘Japanese woman’ or japonka ‘slipper’,
Maltežan ‘Maltese man’ or maltežan ‘Maltese dog’). The
results are also limited because there were included only
official demonyms.

The corpus offers numerous opportunities for extending
the research, e.g. the usage of derogatory or
discriminatory terms for nationalities, representation of
the peoples from the Balkans, as well as comparison
between the representation of various nationalities by
private and corporate Twitter accounts.
7. Acknowledgements

The work described in this paper was funded by the Slovenian Research Agency within the national basic research project "Resources, Tools and Methods for the Research of Nonstandard Internet Slovene" (J6-6842, 2014-2017).

8. References

Bang, M. (2008). *Representation of Foreign Countries in the US Press: A Corpus Study*. A thesis for the degree of doctor of philosophy. The University of Birmingham.

Fišer, D., Erjavec, T., Ljubešić, N. (2017). The compilation, processing and analysis of the Janes corpus of Slovene user-generated content. *Corpus de communication médiée par les réseaux: construction, structuration, analyse*. Collection Humanités Numériques. Paris: L'Harmattan.

Romih, K. (2013). Humor in slovenska nacionalna identiteta. Bachelor thesis. University of Ljubljana.

Šabec, K. (2007). Conflicting memories and stereotypical images of Italians in the Slovenian collective perception: the case of Slovenian literature from Trieste. *Družboslovne razprave*, 23(55), pp. 95–113.

Salihović, S. (2012). Odnos Slovencev do pripadnikov nekdanje Jugoslavije: diplomska naloga visokošolskega univerzitetnega študijskega programa prve stopnje. Bachelor thesis. School of Advanced Social Studies.

Stubbs, M. (1996). *Text and corpus analysis: computer-assisted studies of language and culture*. Oxford, Cambridge: Blackwell.

Tarasheva, E. (2009). The Image of Bulgaria in International Media: A Critical Discourse Analysis and Corpus Study. In *Year Book of the Department of Foreign Languages and Cultures NBU*. http://ebox.nbu.bg/cel/cult07_en.html

Thomas, A. L. (2016). The Agent Across the Border: "Russia" and "Ukraine" as Actors in the News, 2013-2015. In *Thesis and Dissertations - Linguistics*. Paper 15. http://uknowledge.uky.edu/ltt_etds/15

Toporišič, J. and Fran Ramovš Institute of the Slovenian Language (2014): *Slovenski pravopis*. Ljubljana: Založba ZRC SAZU. http://bos.zrc-sazu.si/sp2001.html.

Zupančič, R., Arbeiter, J. (2016). Primitive, cruel and blood-thirsty savages: stereotypes in and about the Western Balkans. *Teorija in praksa*, 53(5), pp. 1051–1063.
# European Language Ecology and Bilingualism with English on Twitter

Steven Coats

University of Oulu, Finland

English Philology, Faculty of Humanities, 90014 University of Oulu, Finland

Email: steven.coats@oulu.fi

## Abstract

Societal and demographic changes have contributed to increasing bi- and multilingualism in European countries in recent years, and communication on social media platforms such as Twitter reflects this linguistic diversity. While high rates of English use online have been attested for many European countries by survey research, relatively little work has quantified the extent to which English is used on social media in European contexts. In this study, English use and bilingualism with English in Europe are investigated on Twitter. A large corpus of Twitter messages with geographical metadata was created by accessing the Twitter APIs. After language detection and filtering, linguistic profiles for European countries were created and the behavior of bi- and multilingual users examined. The analysis supports some previous findings that suggest that a large-scale language shift towards English may be ongoing in Europe in some communicative domains. Geographical differences shed light on the dynamics of this process.

**Keywords:** Bilingualism, social media, Twitter, corpus linguistics, quantitative methods

## 1. Introduction and Background

Recent years have seen an increase in the relative prominence of computer-mediated communication (CMC) modalities such as texting, instant messaging, or posting on social media, and platforms such as Twitter have become multilingual sites with global representation (Mocanu et al. 2013; Leetaru et al. 2013). At the same time, population movements and changes in education and media consumption have contributed to an increasing bi- and multilingualization of local environments, particularly with English – trends that are particularly evident in online communicative domains in some European societies. Although national languages continue to receive reinforcement in education and media, bilingualism with English has become the norm for many within Europe, particularly for young people.

In this study, bi- and multilingualism with English are investigated by means of a quantitative analysis of Twitter messages with location metadata in order to establish a language ecology (Haugen 1972). The research poses the following questions: Which languages are favored by multilinguals on Twitter in Europe? How linguistically diverse are European societies on the platform, and what role does English play? And to what extent do national languages play a role in the discourse of European Twitter users? Addressing these questions may allow us to characterize European Twitter discourse in terms of a language ecology that can “tell us something about where [a] language stands and where it is going in comparison with other languages of the world” (Haugen 1972, p. 337).

In a first step, the linguistic behavior on Twitter of users who can be reliably located within European countries is examined according to country in order to provide an overview of the language ecology of Europe. In a second step, the aggregate network behavior of bi- and multilingual users is examined more closely: Which languages do multilinguals favor in which places? The structure of the network of multilinguals between languages can shed light on the relative status of English and national languages and, due to

the prevailing demographics of Twitter users, perhaps provide an indication of middle- to long-term language shift for European societies.

## 2. Previous Work: Twitter Language and Multilingualism

A number of studies of CMC and Twitter language have investigated aspects of English, including phenomena such as the discourse functions of hashtags (Wikström 2014; Squires 2015), lexical innovation in American English (Eisenstein et al. 2014), African-American Vernacular English dialect on Twitter (Jørgensen, Hovy, and Søgaard 2015), grammatical variation in English-language Twitter from Finland and the Nordic countries (Coats 2016a; Coats 2016b), or the interaction between demographic parameters such as gender with lexical and grammatical features in American English (Bamann, Eisenstein, and Schnoebel 2014).

Ronen et al. (2014) found that English plays an important central role in multilingual networks of Wikipedia editors, book translations, and Twitter users. Hale (2014) investigated global multilingual networks on Twitter, including the network associations of retweets and user mentions, and found that while most interaction networks are language-based and English is the most important single mediating language, other languages collectively represent a larger bridging force. Eleta and Golbeck (2014) examined the tweets of 92 multilingual Twitter users and showed that their language choice on the Twitter reflects the predominant language of their social networks. Kim et al. (2014) used Shannon Entropy to quantify linguistic diversity on Twitter in Switzerland, Quebec and Qatar. They created networks of mono-, bi- and multilinguals, and demonstrated that while English mediates between language communities, users of local languages have more influence. Topic selection may also influence language choice. Such findings have confirmed the status of English as the global lingua franca, but the dynamics of multilingualism in a large social media data
set from all of Europe has to our knowledge not yet been subject to research attention.

Other studies have used surveys to investigate online exposure to and use of languages, their relative status in various media or communicative contexts, and attitudes towards them in Europe (e.g. the *Eurobarometer* surveys conducted by the European Commission or Leppänen et al. 2011 for Finland). Increasing knowledge of English has cemented the language’s “hypercentral” position within the language ecology of Europe (Swaan 2001; Soler-Carbonell 2016), and there may be evidence that English has now displaced some local languages in certain functional domains in some European societies (Görlach 2002, p. 16; for a discussion see the contributions in Linn 2016).

Few large-scale studies of aggregate online language use in Europe, however, have been based on documented usage, and empirical research into aggregate use on Twitter has typically offered only an overview of language frequencies. Additionally, while language-use profiles at country level for Twitter data exist (e.g., Mocanu et al. 2013; Leetaru et al. 2013; Magdy et al. 2014; Graham, Hale, and Gaffney 2014), relatively few studies focus specifically on bi- or multilingualism.

### 3. Methods

Corpus-based and NLP methods were employed in the study. They comprised the collection of data online, filtering of data, quantification of multilingualism, and the construction and visualization of language networks.

#### 3.1. Data Collection

Over 140 million tweets with *place* attributes from European countries or territories were collected from the Twitter Streaming API from November 2016 until June 2017 using the *Tweepy* library in Python (Roesslein 2015). From this “seed” dataset of tweets by 2.9 million users, the tweets of those with at least 20 tweets and at least 50% of tweets from a single country (654,676 users) were retained for analysis.<fnref n="1" /> In total, the data used for analysis comprised over 69.8 million tweets from 55 European countries or territories.

#### 3.2. Data Filtering and Language Detection

Not all tweet user messages are composed by humans: A substantial proportion of tweets is generated automatically by apps or bots that interact with the Twitter API (Haustein et al. 2016). Because many apps post content that is not user-composed but rather consists of automatically-generated text, filtering tweets by the *source* value can reduce the amount of noise in the data set. A manual analysis of a selection of tweets showed that widely-used Twitter apps such as “Twitter for iPhone” or the Twitter Web

Client (i.e. www.twitter.com) were less likely to broadcast automatically-generated text than were some infrequently-used apps. For this reason, the data was filtered to retain only those tweets broadcast by the following apps: Twitter Web Client, Twitter for iOS, Twitter for iPhone, Twitter for Android, Twitter for Windows Phone, Twitter for Instagram, Tweetbot for iOS, and Tweetbot for iPhone. Tweets with these sources collectively comprised over 87% of all those by European users.

A consideration of bi- and multilingualism on the Twitter platform critically depends on accurate characterization of the language of individual tweets, but automatic language detection of tweets can pose difficulties. Character sequences present in URL addresses, usernames, hashtags, emojis, and non-standard orthography can create problems for automatic language detection algorithms, as they rarely correspond to items in the lexicons of natural languages. Even after removing such sequences, very short texts are not handled well by language detection algorithms (Figure 1). To increase detected language accuracy, the data was therefore filtered to include tweets that exhibited three-way agreement between the native Twitter language detection algorithm and the algorithms *langid* (Lui and Baldwin 2014) and *compact language detector 2* (Sites 2014) after removal of URLs, usernames, hashtags, and emojis. For some less-widely-used languages not identified by all three algorithms, such as Faroese, Nynorsk, Albanian, or Somali (among others), two-way language identification or identification by a single algorithm with a high probabilistic accuracy value was used to assign languages to tweets.<fnref n="2" />

<table><caption>Figure 1: Language misidentification on short texts by *langid*</caption><thead><tr><th></th><th>text</th><th>langlangid</th></tr></thead><tbody><tr><td>12260</td><td>normalee, ili dzeperice :)</td><td>it</td></tr><tr><td>12272</td><td>luicrede divincere mann saràcosì! referendum trascu...</td><td>it</td></tr><tr><td>12397</td><td>gol dla 2:1!</td><td>it</td></tr><tr><td>12459</td><td>koje crno malo :))</td><td>it</td></tr><tr><td>12736</td><td>no hej</td><td>it</td></tr><tr><td>154</td><td>ssshes so besyitful</td><td>de</td></tr><tr><td>251</td><td>popieram:-)</td><td>de</td></tr><tr><td>344</td><td>gesehen? fantastische serie.</td><td>de</td></tr><tr><td>518</td><td>i am bored</td><td>de</td></tr><tr><td>773</td><td>a legend</td><td>de</td></tr></tbody></table>

#### 3.3. Quantification of Bilingualism Strength

A user in the dataset was determined to be bilingual for languages *i*, *j* if he or she had authored at least 10% of the total number of tweets in each of the two languages. The connection strength between languages *i*, *j* was quantified on the basis of all users with the phi coefficient, calculated from a contingency table (Table 1 and Equation 1).

<footnote n="1">¹For this data, correlation between the center of the *place* bounding box and the precise GPS coordinates from the *coordinates* object, if both were present, was found to be quite high (= 0.992). For this reason, the *place* field was considered an accurate indication of true user location when posting a tweet.</footnote>

<footnote n="2">²The presence of unique vocabulary markers (Ljubešić, Fišer, and Erjavec 2014) can be used to collect tweets in less-used languages, but the method is not applicable to the detection of already-collected tweets.</footnote>
<table><caption>Table 1: Contingency Table for Number of Bilinguals</caption><thead><tr><th></th><th>language<sub>i</sub></th><th>~ language<sub>i</sub></th><th></th></tr></thead><tbody><tr><th>language<sub>j</sub></th><td>O<sub>11</sub></td><td>O<sub>12</sub></td><td>= R<sub>1</sub></td></tr><tr><th>~ language<sub>j</sub></th><td>O<sub>21</sub></td><td>O<sub>22</sub></td><td>= R<sub>2</sub></td></tr><tr><th></th><td>= C<sub>1</sub></td><td>= C<sub>2</sub></td><td>= N</td></tr></tbody></table>

$$ \phi_{ij} = \frac{(O_{11}O_{22} - O_{12}O_{21})}{\sqrt{R_1 R_2 C_1 C_2}} \quad (1) $$

$\phi$ is equivalent to Pearson's product-moment correlation coefficient for occurrence frequency of two binary variables, and ranges in value from -1 to 1. Positive values indicate the language pairs are more strongly connected than would be expected based on the prevalence of the languages in the multilingual dataset.

A t-statistic was calculated to test the significance of the correlation between languages according to the formula

$$ t_{ij} = \frac{\phi_{ij} \sqrt{D - 2}}{\sqrt{1 - \phi_{ij}^2}} \quad (2) $$

where $D = max(R_1, C_1)$.

A multilingualism network for Europe was created in which node size corresponds to the number of multilingual users for a language and edge width corresponds to the strength of the connection (number of bilinguals) for a language pair. The network for those nodes and edges with at least 10 bilinguals and t-test p-values < 0.05 was retained (Figure 2). Network relationships were visualized using the R packages *igraph* and *visNetwork* (Csardi and Nepusz 2006; Thieurmel 2016). Additionally, nodes and vertices were annotated with information about total number of users, total number of bilinguals, and average message length.

## 4. Results

In terms of overall language representation in the European Twitter bi- and multilingualism network, English is the most prevalent language, with approximately 30% of all tweets in English. For Europe as a whole, a network of 42 languages and 68 edges describes the statistically significant bilingual links (Figure 2). English clearly plays the most important role: it is connected to almost all of the languages in the network. Other languages with large numbers of users, such as French, Spanish, Turkish, and Russian, have multiple connections to other languages.

It should be remarked that the bi- and multilingual network only accounts for productive language use (i.e. authorship of discourse in a particular language), not passive understanding of languages. In an additional step, follower and friend statistics will be used to estimate these values as well.

Multilingual networks were also created for individual European countries. In them, English serves as a bridge between linguistic communities, but the principal national

Bilingualism by language (green: φ significant at p < 0.05)

<figure><img src="image_12.png" /><figcaption>Figure 2: Bilingualism Network</figcaption></figure>

language(s) figure prominently. Nonetheless, high levels of bilingualism with English, when considered in light of previously reported demographic statistics (European Commission 2012), may indicate a shift towards English and away from traditional languages in Europe, particularly for some minority languages with official status.

## References

Bamann, David, Jacob Eisenstein, and Tyler Schnoebelen (2014). “Gender Identity and Lexical Variation in Social Media”. In: *Journal of Sociolinguistics* 18.2, pp. 135–160.

Coats, Steven (2016a). “Grammatical feature frequencies of English on Twitter in Finland”. In: *English in Computer-mediated Communication: Variation, Representation, and Change*. Ed. by Lauren Squires. Berlin: De Gruyter, pp. 179–210.

- (2016b). “Grammatical frequencies and gender in Nordic Twitter Englishes”. In: *Proceedings of the 4th conference on CMC and social media corpora for the humanities*. Ljubljana, pp. 12–16. URL: http://nl.ijs.si/janes/wp-content/uploads/2016/09/CMC-conference-proceedings-2016.pdf.

Csardi, Gabor and Tamas Nepusz (2006). “The igraph software package for complex network research”. In: *International Journal Complex Systems*, p. 1695. URL: http://igraph.org.

Eisenstein, Jacob et al. (2014). “Diffusion of Lexical Change in Social Media”. In: *PLoS ONE* 9.1.

Eleta, Irene and Jennifer Golbeck (2014). “Multilingual use of Twitter: Social networks at the language frontier”. In: *Computers in Human Behavior* 41, pp. 424–432.
European Commission (2006). "Europeans and their languages: Special Eurobarometer 243". In: URL: http://ec.europa.eu/public_opinion/archives/ebs/ebs_243_en.pdf.

- (2011). "User language preference online: Flash Eurobarometer 313". In: URL: http://ec.europa.eu/public_opinion/flash/fl_313_en.pdf.

- (2012). "Europeans and their languages: Special Eurobarometer 386". In: URL: http://ec.europa.eu/public_opinion/archives/ebs/ebs_386_sum_en.pdf.

Graham, Mark, Scott A. Hale, and Devin Gaffney (2014). "Where in the World Are You? Geolocation and Language Identification in Twitter". In: *The Professional Geographer* 66.4, pp. 568–578. URL: http://dx.doi.org/10.1080/00330124.2014.907699.

Görlach, Manfred (2002). *Still More Englishes*. Amsterdam: John Benjamins.

Grosjean, François (2008). "Studying bilinguals: Methodological and conceptual issues". In: *Handbook of bilingualism*. Ed. by Tej K. Bhatia and William C. Ritchie. Malden, MA: Wiley-Blackwell, pp. 32–63.

Hale, Scott (2014). "Global connectivity and multilinguals in the Twitter network". In: *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems*. ACM. New York, pp. 833–842.

Haugen, Einar (1972). "The ecology of language". In: *The ecology of language*. Ed. by Einar Haugen and Anwar Dil. Palo Alto: Stanford University Press, pp. 325–339.

Haustein, Stefanie et al. (2016). "Tweets as impact indicators: Examining the implications of automated "bot" accounts on Twitter". In: *Journal of the Association for Information Science and Technology* 67.1, pp. 232–238. ISSN: 2330-1643. DOI: 10.1002/asi.23456. URL: http://dx.doi.org/10.1002/asi.23456.

Jørgensen, Anna Katrine, Dirk Hovy, and Anders Søgaard (2015). "Challenges of studying and processing dialects in social media". In: *Proceedings of the ACL 2015 Workshop on Noisy User-generated Text*. Association for Computational Linguistics. Stroudsburg, PA, pp. 9–18. URL: http://aclweb.org/anthology/W15-4302.

Kim, Suin et al. (2014). "Sociolinguistic Analysis of Twitter in Multilingual Societies". In: *Proceedings of the 25th ACM Conference on Hypertext and Social Media*. HT ’14. Santiago, Chile: ACM, pp. 243–248. URL: http://doi.acm.org/10.1145/2631775.2631824.

Leetaru, Kalev H. et al. (2013). "Mapping the global Twitter heartbeat: The geography of Twitter". In: *First Monday* 18.5/6.

Leppänen, Sirpa et al. (2011). *National Survey on the English Language in Finland: Uses, meanings and attitudes (= Studies in Variation, Contacts and Change in English, Volume 5)*. Helsinki: Varieng.

Linn, Andrew, ed. (2016). *Investigating English in Europe: Contexts and agendas*. Berlin and Boston: De Gruyter Mouton.

Ljubešić, Nikola, Darja Fišer, and Tomaž Erjavec (2014). "TweetCaT: a tool for building Twitter corpora of smaller languages". In: *LREC*.

Lui, Marco and Timothy Baldwin (2012). "Langid.py: An off-the-shelf language identification tool". In: *50th Proceedings of the Association for Computational Linguistics*. Association for Computational Linguistics. Stroudsburg, PA, pp. 25–30.

- (2014). "Accurate language identification of Twitter messages". In: *Proceedings of the 5th Workshop on Language Analysis for Social Media (LASM) EACL 2014*. Association for Computational Linguistics. Stroudsburg, PA, pp. 17–25.

Magdy, Amr et al. (2014). "Exploiting geo-tagged tweets to understand localized language diversity". In: *Proceedings of Workshop on Managing and Mining Enriched Geo-Spatial Data*. GeoRich’14. Snowbird, UT, USA: ACM, 2:1–6. URL: http://doi.acm.org/10.1145/2619112.2619114.

Mocanu, Delia et al. (2013). "The Twitter of babel: Mapping world languages through microblogging platforms". In: *PLoS ONE* 8.4.

Roesslein, Joshua (2015). *Tweepy*. Python programming language module. URL: https://github.com/tweepy/tweepy.

Ronen, Shahar et al. (2014). "Links that speak: The global language network and its association with global fame". In: *PNAS* 111.52, E5616–E5622.

Sites, Dick (2014). *Compact language detector 2*. R package version 1.0.2. URL: https://github.com/CLD2Owners/cld2.

Soler-Carbonell, Josep (2016). "English in the language ecology of Europe". In: *Investigating English in Europe: Contexts and agendas*. Ed. by Andrew Linn. Berlin and Boston: De Gruyter Mouton, pp. 53–58.

Squires, Lauren (2015). "Twitter: Design, discourse, and implications of public text". In: *The Routledge Handbook of Language and Digital Communication*. Ed. by Alexandra Georgakopoulou and Tereza Spilioti. London and New York: Routledge, pp. 239–256.

Swaan, Abram De (2001). *Words of the world: The global language system*. Cambridge: Polity.

Thieurmel, Benoit (2016). *visNetwork: Network Visualization using 'vis.js' Library*. R package version 1.0.2. URL: https://CRAN.R-project.org/package=visNetwork.

Wikström, Peter (2014). "#srynotfunny: Communicative functions of hashtags on Twitter". In: *SKY Journal of Linguistics* 27, pp. 127–152.
# Reliable Part-of-Speech Tagging
of Low-frequency Phenomena in the Social Media Domain

Tobias Horsmann, Michael Beißwenger and Torsten Zesch

Language Technology Lab

Department of Computer Science and Applied Cognitive Science

University of Duisburg-Essen, Germany

`{tobias.horsmann,michael.beisswenger,torsten.zesch}@uni-due.de`

## Abstract

We present a series of experiments to fit a part-of-speech (PoS) tagger towards tagging extremely infrequent PoS tags of which we only have a limited amount of training data. The objective is to implement a tagger that tags this phenomenon with a high degree of correctness in order to be able to use it as a corpus query tool on plain text corpora, so that new instances of this phenomenon can be easily found. We focused on avoiding manual annotation as much as possible and experimented with altering the frequency weight of the PoS tag of interest in the small training data set we have. This approach was compared to adding machine tagged training data in which only the phenomenon of interest is manually corrected. We find that adding more training data is unavoidable but machine tagging data and hand correcting the tag of interest suffices. Furthermore, the choice of the tagger plays an important role as some taggers are equipped to deal with rare phenomena more adequately than others. The best trade off between precision and recall of the phenomenon of interest was achieved by a separation of the tagging into two steps. An evaluation of this phenomenon-fitted tagger on social media plain-text confirmed that the tagger serves as a useful corpus query tool that retrieves instances of the phenomenon including many unseen ones.

**Keywords:** Part-of-speech, Social Media, CMC, Rare Phenomena

## 1. Introduction

This paper reports on experiments on adapting part-of-speech (PoS) taggers for tagging rare phenomena found in genres of computer-mediated communication (CMC). Our work is motivated by a use case in which a linguist wants to study a rarely occurring CMC phenomenon using Twitter data from the social media domain. The central problem here is how to find such rare instances of the phenomenon under observation without spending hours of screening through plain text. A filtering tool would be desirable that facilitates the retrieval process for the linguist. The tool should find many instances of the phenomenon and at the same time achieve reasonably correct results in order to decrease the workload considerably. The project we present here investigates how to adapt a PoS tagger for tagging a certain rarely occurring phenomenon in order to use the PoS tagger as a filtering tool that linguists can use to query a corpus.

The main challenge of adapting a PoS tagger to the language use in the social media domain lies in dealing with the notorious lack of training data and many out-of-vocabulary words. This problem becomes even more severe when the tagger shall be adapted for dealing with a phenomenon that is under-represented in the already small training data sets. We will, thus, investigate methods to improve tagging of under-represented phenomena while laying emphasis on avoiding manual annotation as much as possible. We aim on detecting a German verb-pronoun contraction phenomenon that the linguist wishes to study in detail on the basis of a broad set of instances automatically retrieved from social media data. To deal with the lack of training data, we experiment with (i) adjusting the frequency weight of the under-represented phenomena by under- and oversampling and (ii) adding automatically tagged but new data in which only the tag of the phenomenon of interest is manually corrected. In a concluding case study, we optimize a tagger towards finding this contraction phenomenon and evaluate how well the filtering works in a real world setup on plain text Twitter messages.

<table><caption>Table 1: Full verb + pers. pronoun (VVPPER) contraction</caption><tr><td>wiederholen (to repeat) + es (it)</td><td>1st person<br/>ich <strong>wiederhols</strong> nochmal, ihr redet hier öffentlich!<br/>I repeat it [repeat-it] again, you're talking in public!</td></tr><tr><td>kommen (to come) + du (you)</td><td>2nd person<br/>wieso? wo <strong>kommste</strong> denn her?<br/>why? where do you come [come-you] from?</td></tr></table>

## 2. German Verb-Pronoun Contraction

We are interested in a particular phenomenon in which a verb and a following personal pronoun are contracted into a single form. Table 1 shows examples of this type of contractions taken from the Dortmund Chat Corpus (Beißwenger, 2013). Verb-pronoun contractions belong to the class of phenomena that are not unique for CMC discourse but typical for spontaneous - spoken or 'conceptually oral' - language in colloquial registers. Phenomena of this type are of special interest for linguists who want to use corpora to compare written discourse from the social media domain to the language of edited text and the language found in informal, spoken interactions. If we use a tagger as a filtering tool, we need a high precision to avoid screening through countless false positive instances. At the same time, we want to find new lexical forms unknown from the training set, which requires a high recall (i.e., high generalization).

We have a data set of 23k tokens of German social media discourse that was annotated for a shared task on PoS tagging for German CMC and social media.
dia data (Beißwenger et al., 2016). The data are annotated with an extended version of the Stuttgart-Tübingen tagset (STTS) (Schiller et al., 1999) that has been expanded by tags needed for tagging social media phenomena (Beißwenger et al., 2015). In this tagset, verb-pronoun contractions are labelled by an own tag, VVPPER, which occurs 13 times in total. Results of the shared task showed that this infrequency prevents taggers from learning the phenomenon in a reliable manner (Horsmann and Zesch, 2016b). Since the VVPPER tag is not included in the canonical STTS, as those contractions do not occur in the domain of edited text, existing STTS annotated corpora (e.g., newspaper corpora) cannot be used to obtain additional instances of the phenomenon for training. Although there is a small amount of annotated data that we can build upon, there are still not enough VVPPER instances to make the phenomenon recognizable when training PoS taggers.

### 3. Dealing with Infrequency

In this experiment, we test different strategies to improve the tagging of VVPPER instances. With a total of 13 instances in our data set, we have to annotate at least some additional data in order to train the tagger but also to arrive at meaningful results during evaluation. At the same time we keep the manual annotation effort at a minimum.

#### 3.1. Data Set

We base our experiment on the data set from the aforementioned shared task. We enrich the data by selecting 230 user posts that contain this phenomenon from the Dortmund Chat Corpus. We automatically tagged these additional data by using the Stanford tagger that assigns PoS tags of the canonical STTS and manually corrected the tag for the verb-pronoun contraction that only exists in the extended STTS. This is the most minimalistic amount of manual annotation one can possibly perform which - as we will see soon - suffices. Of the additional 230 instances, we add one half to the testing set and one-sixth to the training set. The remaining two-sixths are our development set in the following experiments and are held back for the moment. The enhanced training set now contains 45 (38+7) sequences with the phenomenon and the testing set 121 (115+6) sequences. This should be enough instances for learning and evaluating the phenomenon.

#### 3.2. Frequency Weight vs. Lexical Knowledge

An option to circumvent annotation of a larger amount of data is boosting the signal for a certain PoS tag in the already existing data. This can either be done by oversampling (Daumé III, 2007) the few instances one has by adding them N times to the training set, or by downsampling, i.e. removing sequences *without* the PoS tag of interest i.e. VVPPER. Both approaches lead to an increased frequency weight of the phenomenon relative to the other PoS tags in the corpus. We experiment with both strategies in the following setup: *Downsampling*: We remove 25, 50 and 75 percent of the training data instances that do not contain any verb contractions. *Oversampling/new Instances*: We choose oversampling rates that add a number

of instances which we can also provide from the held back annotated sequences. This allows a direct comparison between oversampling instances and adding fresh ones. We will, thus, oversample two and three times, and compare this to adding the same amount of instances from the set of new sequences in the held back development set.
We conduct these experiments with the following taggers to learn about the empirical differences between tagger implementations for our objective:

**Stanford** (Toutanova et al., 2003) a PoS tagger that is frequently used in the community due to its good reputation and high accuracy.

**HunPos** (Halácsy et al., 2007), a tagger with a good reputation based on Hidden-Markov models and a re-implementation of the TNT tagger (Brants, 2000).

**LSTM** A deep learning PoS tagger by Plank et al. (2016), that is based on Long-Short-Term-Memory (Hochreiter and Schmidhuber, 1997) neural networks. We use the same parametrization as Plank et al. (2016) and self-trained German word embeddings trained on German Twitter messages with $195 \cdot 10^6$ tokens.

**Two-Step** Horsmann and Zesch (2016a) proposed a tagger architecture for social media data that first uses a highly generalized *coarse-grained* tagger, and as a second step applies a specialized non-sequential tagger for *fine-grained* tagging. The second tagger is tailored towards recognizing the tag of interest while the first tagging step constraints the application of the second tagger.

We implement this approach by using a CRF tagger (Lafferty et al., 2001) in the first step and an SVM in the second step. For training the coarse-grained sequence model, we map the extended-STTS tags of the training data to the coarse-grained tagset used by the Universal Dependency project and map VVPPER to *verb*. We include a PoS dictionary and Brown (Brown et al., 1992) clusters created over German Twitter messages to compensate for the lack of training data. This coarse-grained tagger reaches a $F_1$ of 0.93 on the tag *Verb* in the test data, which means that some VVPPER instances will be missed because the coarse model did not predict *verb*.

**Results** In Figure 1, we show the results of the three strategies on the VVPPER tag. We focus on *out-of-vocabulary* instances which perform considerably poorer than *in-vocabulary* instances ($F_1$ between 0.96 to 0.99), and thus offer more opportunities for improvements. We see that neither downsampling nor oversampling helps to reach a substantial improvement on the tag. Furthermore, downsampling shows that the anyway low amount of training data becomes a large problem for the LSTM if further reduced. The Stanford tagger stays behind the other taggers with both sampling methods. The only effective method is, without much surprise, providing new data. The LSTM needs considerably more data to improve while the other taggers improve linearly with each new data set.

**Discussion** Table 2 shows details of the two best taggers HunPoS and Two-Step. Once again, we focus on the out-of-vocabulary instances, while also showing precision (P) and recall (R). The $F_1$ score shows that both taggers reach