dataesr/scientific-paragraphs-categorization huggingface dataset:
Funding Évora team funded by national funds through FCT -Fundação para a Ciência e Tecnologia, I. P. , in the framework of the ICT project with the references UIDB/04683/2020 and UIDP/04683/2020 and by TOMA-QAPA (PTDC/CTAMET/29678/2017). The authors acknowledge the GRASP-ACE (Grant agreement ID: 778349), ACTRIS-IMP (Grant agreement ID:871115), ATMO-ACCESS (Grant Agreement ID: 101008004), PROBE (COST Action number: CA18235), HARMONIA (COST Action number: CA21119), EUMETNET through the E-PROFILE program and REALISTIC (Grant agreement ID:101086690) projects. The authors acknowledge the support through ACTRIS-2 under grant agreement no. 654109. This work was partially supported by the Spanish national projects PID2019-103886RB-I00/AEI/10. 13039/501100011033,.
INTRODUCTION Driven by advances in AI and cloud computing, large technology companies are increasingly integrating Automatic Speech Recognition (ASR) systems-those that turn speech to text-into products and services. These advances have expanded the range of languages able to access such services: Google's Cloud Speech-to-Text API, for example, currently supports 73 languages in 139 dialects and variants. 1  There are, however, many less-prominent languages that are unsupported, leaving communities unheard. Research across HCI and ASR has also expressed concern at the current direction of ASR development towards unsupervised approaches with extremely high data requirements  [35, 45]. Such approaches to ASR development, even if aimed at broadening access to language technologies for smaller languages, bypass local language speakers  [6]  and have been criticised for getting "caught up in an information-theoretic view of the problem"  [35].
The term 'local languages' in its plural form also better represents the realities of many speech communities that "have a repertoire of languages, each one playing a diferent role in the local linguistic ecosystems"  [6, p. 7819]. Local, in the context of this paper, is Langa -an isiXhosa-speaking community we partnered with on the outskirts of Cape Town, South Africa. With the help of our toolkit we gathered 318 communitygenerated stories collected in public settings across Langa, totalling just under two hours of audio from voices that are marginalised and not currently refected in existing datasets  [3, 54]. The toolkit also supported fve community members to comprehensively transcribe the collection of stories, which exhibit vibrancy, innovation, and mixing of language(s). The hardware and software components of the toolkit thus embody  [26]  the main HCI contribution of this work; namely, to address the technological barriers and work through more social/structural barriers to participation in order to engage communities of minoritised language speakers in the development of speech and language technologies suited to their ways of speaking.
Leveraging the data and insights the community engagement processes surfaced, we improved an isiXhosa ASR system from a character-error rate of 51. 2 % to 27. 7 %, but were also challenged to consider how we might re-imagine and further situate ASR evaluation processes and metrics in community contexts. Beyond the toolkit and spoken/transcript data, our research makes the following key contributions: Reitmaier, et al. • to show how community engagement, facilitated through the toolkit, surfaces salient social and technical issues and opportunities for language technologies; • to highlight how voice-based interactions can broaden digital participation in communities that are linguistically and economically marginalised; • and to demonstrate the impact of speech and language technologies to Language Technology for All (LT4All)  [28]  and Information and Communication Technology for Development (ICT4D)  [61]  research communities.
Data-gathering To support ASR development, researchers have developed smartphone applications and other services designed to collect speech samples directly from users  [1, 17, 33]. However, these record users as they read out a list of prompts, and are therefore not representative of how people speak in everyday life. In Langa specifcally, this also does not account for how people mix languages. To more accurately capture speech in everyday life, Reitmaier et al. [45]  have previously demonstrated how WhatsApp itself can be utilised as an 'unplatformed'  [36]  tool for collecting voice message samples from community members, leveraging users' familiarity with the messaging app and the authentic form of expression it represents -especially because it is typically used to communicate with friends, family, and community.
That being said, we are inspired by how mobile interface innovations can be leveraged to situate the transcription tasks in marginalised community contexts and create new economic opportunities in the process. Relatedly, Reitmaier et al. have recently demonstrated some success in creating a modest ASR dataset by working with community members, but also acknowledge that they did not adequately support community-generated transcriptions  [45]. Finally, web and desktop-based transcription software used by professional transcription service providers are comparatively complex and require bespoke hardware (e. g. , foot pedals to control playback speed), signifcant transcriber training, and are targeted towards users with more advanced digital and media literacy, such as the ability to touch-type.
ASR development Of course, context, language, data collection and transcription also fgure heavily in the ASR development process. Here we draw on the ASR, HCI and sociolinguistic disciplinary perspectives of our research team to summarise and critique the current state of the art of ASR development. Typical data-collection approaches utilised in ASR development see data as something that can be harvested or scraped online, turning data collections into a neutral, inert, and interchangeable substrate  [14]  which bypasses communities  [12]. In our case, dataharvesting  [46]  (e. g. , from YouTube) and crowd-sourcing through Amazon's Mechanical Turk platform  [10] , are not as efective, not least because digital inequalities lead to poor online representation and in efect render the voices of people living in places like Langa 'digitally invisible'  [60].
While state-of-the-art ASR systems  [2]  have been served well by building complex acoustic and language models using large data collections, languages that have less data available-so-called 'low-resource' languages in the ASR argotare not well served by these approaches and consequently are left behind  [14]. While there are existing isiXhosa ASR datasets containing read speech  [3] , and code-switched isiXhosa-English datasets of TV soap operas  [54] , voices of people from places like Langa are poorly represented therein. However, rather than dismissing these scarce language resources altogether, we are inspired by the late Gary Marsden's pragmatic design approach that emphasises leveraging the most out of existing resources  [37] , in our case by training and tuning an isiXhosa ASR system using both existing resources as well as data, feedback and provocations generated through engagement with a particular community.
We also align ourselves with Joshi et al. [35] 's research in this regard and scholarship more broadly that advocates ground-up approaches to ASR development  [6]. However, these are the exception rather than the rule. Community and context are critical diferentiating factors here, for conventional approaches to ASR and language model development invest more heavily into deep learning models than data collecting or curation  [46]. The danger in focusing too narrowly on the 'model-work' of ASR is that it confgures the task of ASR-turning speech into text-into a mathematical problem to solve and optimise in the lab using objective metrics of error rates, and operating on existing datasets of speech data paired with 'gold-standard' human transcripts. Another nuance that is unaccounted for, especially in the context of Langa, is that transcription and writing practices in minoritised, unstandardised and local language varieties can difer from the requirements of technologies designed for 'high-resource' standardised languages  [6].
A segmented timeline shows how the current audio recording is segmented into smaller parts of a maximum of fve seconds. Audio recordings longer than fve seconds are not themselves split; instead the playback of the recording pauses to minimise cognitive load and give users the opportunity to transcribe the current segment, re-listen if necessary, before moving to the next. The TranscriptTool also overlaps these segments by 750 ms to ensure that if a word is inadvertently clipped at the end of one segment it can be picked up in its entirety at the start of the next segment instead; this avoids the need to detect natural pauses that may not be possible to accurately identify if recordings are noisy. Transcripts are stored in a database on the TranscriptTool, which also logs how a transcript is built up over time-which audio segment of a longer recording was active when text was entered-in order to help capture time alignment data.
Generated data The data gathered and transcribed by community members using the above tools was furthermore an invaluable conceptual tool that surfaced and provided concrete data on more anticipated challenges, such as noisy recordings and mixed language use, as well as less anticipated challenges, such as considerable variability between transcribers of the same audio content. This suggests that isiXhosa has a lower degree of language standardisation in its written form (see  [18] ). Engaging with communities in public and working with less standardised languages calls into question terminologies and standards that ASR development ascribes to data, for instance of refned audio made in quiet environments and of gold-standard human transcriptions without variability (see  [5] ). We qualify these challenges as more or less anticipated, because even when we deliberately try to escape them we are nevertheless infuenced by the models and mindsets of monolingualism and the gold-standard status of human transcriptions that are ingrained in the orthodoxy of ASR practice.
Formative design and public engagement workshop The workshop facilitator recruited 11 participants (7F, 4M) local to Langa and determined a locally appropriate participation incentive payment. Opening the workshop, we introduced ourselves and explained that we wanted to work with participants to think of ways to improve the efectiveness of speech systems in local languages, like isiXhosa. We explained that we had a initial isiXhosa ASR system (introduced later), but that we need to improve it by  (1)  gathering data via and improvements to the SpeechBox and (2) transcribing that data using the TranscriptTool. Finally, we wanted to (3) showcase the current status of the ASR system using the mobile demonstrator app. 4. 2. 1 SpeechBox trial. We then introduced the SpeechBox system and demonstrated how it works.
TranscriptTool trial. Next, we turned our attention to the TranscriptTool and outlined how ASR systems are trained not just on speech data, but that they also need text data so the system can learn how sounds relate to text. We explained that transcription is typically done by experts on desktop machines, and invited participants to experiment with us to see if it could be done on their phones. We then asked participants to install the TranscriptTool app on their own phones. This was a time-consuming process, as participants struggled to complete the installation process. Some mobiles had severely cracked screens (see Fig. 2 ), making navigation in unfamiliar apps such as the Google Play store difcult, while other mobiles were out of battery or had run out of space. By clearing space, charging phones and demonstrating the precise steps required to install the app, nine participants (82 %) were ultimately able to install the app.
The remaining two were ultimately unable to install the TranscriptTool because their mobiles were running much older and unsupported versions of Android. We preloaded the app with nine transcription tasks. Eight were short recordings (2. 2 to 5. 2 s) from van der Westhuizen and Niesler  [54] 's soap opera corpus, split between mixed isiXhosa and English 6 The frst language that a person has been exposed to from birth. Reitmaier, et al. ( = 4) and solely isiXhosa ( = 4). The ninth task was 120 s long and contained all of the audio prompts from the SpeechBox, which were recorded in isiXhosa but also contain some commonly-used English terms (e. g. , 'button' or 'airtime') and even some mixed words, such as 'iCovid' (= isiXhosa noun prefx + Covid  [24] ). Participants completed all of the tasks, but also uncovered some basic usability issues that we subsequently addressed.
ASR demonstration trial. In the concluding part of the workshop we focused on our mobile ASR speech-to-text demonstrator app so that users could experiment with ASR capabilities and limitations on their own devices. Here only fve participants (~45 %) were able to install the app without compatibility issues, as it required a minimum version of Android 7. 0 (late 2016). We had chosen this confguration based on usage fgures reported in the Android Studio IDE, which claims 92. 7 % of devices would be supported. However, these fgures are likely weighted very diferently across the globe, and do not account for mobiles that remain largely ofine and typically 'sideload' apps, using software such as SHAREit  7  , rather than obtaining them through Google Play. The ASR demonstrator used a baseline isiXhosa ASR 'hybrid' system.
We trained the model on 50 hours of read speech from the NCHLT South African speech corpus  [3] : a dataset that has previously been used in the isiXhosa ASR literature (cf. [7, 34, 55] ) and used the smaller isiXhosa-English TV soap opera dataset  [54]  for testing and tuning. When evaluated on the soap opera data, this baseline model achieved a word error rate (WER) of 93. 6 %, and a character error rate (CER) of 51. 2 %  8 In the workshop, we contextualised the metrics of the ASR system and explained that it would likely make lots of errors. In groups of two (and one group of three) we asked participants to record and transcribe messages to their group partner(s) on three diferent topics: organising a party together; informing each other about a community activity happening in the area; and, chatting about work.
The partner(s) would then try to gauge the content of the recording from the transcript before listening to the recording. They would then respond with their own recording and repeat the process. One group reported that they could not make sense of any of the transcripts. The remaining groups said that the ASR recognised some things correctly and that they could get the gist of most of the messages, because they also knew the context from our instruction. Common words, like 'enkosi' ('thanks') would normally be correct, but the ASR also transcribed some English words (e. g. , a shopping item, "Hennessy Whiskey" for the party scenario) as incomprehensible isiXhosa. That being said, the ASR system performed slightly better than we had expected and the metrics might have suggested. We thanked participants for their feedback on all aspects of the workshop and reiterated that their SpeechBox and TranscriptTool use and feedback would help reduce ASR errors in future.
4. 2. 4 Discussion: SpeechBox and TranscriptTool refinements. In response to participants' feedback during the workshop we improved how button presses are interpreted on the SpeechBox appliance, especially during the recording phase in order to accommodate both the 'press to begin and press again to stop' as well as 'press while recording and release to stop' behaviours observed. We also ensured that the system was listening for button presses even if a prompt was playing, for instance so users could start entering their phone number instead of having to listen to the entire prompt frst. As recommended, we designed an English version of the SpeechBox signage, and also refned the precise language of each prompt. We addressed issues and implemented changes to the Transcript-Tool that participants uncovered and suggested: to support task navigation (e.
With the existing app they occasionally improvised methods to achieve thisfor instance, when an exasperated person recorded a story about being tired of COVID-19, one of the participants added a transcription comment in brackets "(mama in background telling him to mention umsebenzi [jobs])" whereas the other participant focused on the main speaker. The two disagreed about the place of inappropriate language: one saw it as a part of how people speak and, given the goal of building ASR systems that refected real isiXhosa, wanted this refected in the dataset and so transcribed it; the other fagged any instances of profanity (as per instructions). Finally, participants thought the video tutorial with the instruction was helpful, but found it difcult to reference as they only had one device, so they could either watch the video or open the TranscriptTool.
The ASR system Informed by the new datasets and fndings reported in the literature  [18, 45]  showing that speakers code-switch between many South-African languages, not just between isiXhosa and English, we decided to make the ASR system multilingual. The model was designed to recognise fve languages: isiXhosa (xho), isiZulu (zul), Sesotho (sot), Setswana (tsn)-all belonging to the same Southern Bantu family of languages-as well as English (eng). We built a hybrid ASR system using the Kaldi toolkit  [43] , with a model consisting of an acoustic model using a grapheme lexicon and a language model. Similar hybrid systems have been successfully used to recognise a diverse range of Indic languages, which also exhibit code-switching  [20] , providing evidence that training on multiple languages (especially if these languages are acoustically/linguistically similar) is more robust than training on one language alone.
Although integrating the Covid Stories data into our training pipeline would have been ideal, for reasons we unpack in Section 7, we did not use these recordings for training/testing directly and instead used the 'test-set' component of the South African soap opera (SO) dataset  [54] , which covers all fve of the languages listed above. We reasoned that the SO dataset would be a closer match in domain and style to the type of speech the ASR system would be used to recognise during our deployment. We initially built the acoustic model component of this system by pretraining on the NCHLT corpora  [3] , followed by fne-tuning on the (train-set proponent of the) South African Soap Opera (SO) dataset  [54]. Although NCHLT is a large dataset that covers all fve languages of interest, there is a big domain mismatch between NCHLT and SO, because NCHLT contains read speech and SO contains conversational speech from TV programmes.
To alleviate this mismatch and to better suit the model for the COVID-19 stories use-case, we instead pre-trained the acoustic model on 500 hours of speech from the the British English Multi-Genre Broadcast (MGB) corpus  [4] , which was also fne-tuned on the training part of SO. Note that, unlike the NCHLT corpus, the MGB dataset contains only English speech. Nevertheless, given its matching domain, we found pre-training on this corpus to yield better results. This model achieved a WER of 58. 1 % and CER of 27. 7 % when evaluated on the SO test set. For the language model component, we concatenated text data from all fve languages, using the data-sources listed in Table  2. Given the success and popularity of large pre-trained self supervised models for low-resource ASR speech recognition (e.
The ASR demonstrator appliance The ASR demonstrator appliance, which utilises the above ASR system, is shown in Fig. 4. Compared to the SpeechBox, the enclosure is much smaller and designed to look like a smart speaker. The keypad is replaced with two rating buttons (+/-). An LED ring light mounted around the main button and behind the semi-transparent front panel creates a difusing efect, and replaces the original version's display. On the software side, we integrated the demonstrator appliance with the ASR system using bidirectional streaming protocols  10 to transmit audio requests and concurrently receive transcript responses, so users can expect a result shortly after they fnish speaking. We also used the improved ASR system to re-transcribe all audio recordings that were not fagged as blank, inappropriate, or underage.
Following von Holy et al. 's research on accessing digital libraries through search in resource-scarce South African languages including isiXhosa  [59] , we indexed and queried COVID-19 story transcripts as is (without stemming or n-grams) using the Apache Lucene 11  open-source search software. When users walk up to the device, pressing the main button initiates an interaction fow. The user is greeted by a prompt, explaining that the device contains stories collected from Langa about COVID-19 and that they can press and hold the button again and tell the device what they are interested in. While the device is recording, the ASR system transcribes the audio data, which is not otherwise retained on the device or on the server. The LED ring light visualises the microphone signal, to indicate to the user that the device is actively listening.
Community-engaged COVID-19 stories & transcripts Because the design of the Speechbox and its deployment encouraged wide participation in a community that is linguistically and structurally marginalised, the collection of spoken story contributions users made are a rare and useful resource that difers from existing corpora of isiXhosa  [3, 54]. The dataset features (unscripted) conversational speech, often draws on diferent languages and sometimes multiple speakers, and overall more accurately refects the way Langa residents speak in day-to-day life. In fact a salient feature of the dataset, and of language-use in Langa more generally, is fuid and non-domain-specifc code-switching. In other contexts, code-switching tends to be more domain-bound -see for instance  [27, 31] , or contain only a single switch point, such as the example from the Soap Opera dataset shown in Table  1.
Improving the codeswitching capabilities of our ASR system-e. g. , by modelling likely switch points between languages or how particular topics (jobs, money, medicine) infuence code-switching-is thus an integral next step. A further feature of the dataset is the variability of transcriptions, discussed in Section 5. 3, which are not particularly amenable to integration into traditional ASR development pipelines without at least signifcant normalisation. But, more troubling for ASR orthodoxy, this variability evidences the notion that we cannot assume that 'gold-standard' transcripts exist, particularly for vernacular languages-such as isiXhosa-that do not necessarily have a well-defned (or widely used) written standard  [6]  and where such standards are entwined with colonial encounters  [18].
By comparing multiple transcripts from several annotators, we can better understand and respond to variations in spelling, lexis, and interpretation. These variations and lack of written standards, have important implications for how we develop and evaluate ASR systems, for conventional ASR approaches rely on both 'gold standard' transcripts and 'standard' metrics (e. g. WER/CER), which have considerable faws even in 'high-resource' settings  [23, 40]. Consider that, on average, there was a 63. 9 % disagreement rate between annotators at a word level, and a 30. 4 % disagreement rate at a character level. Note then that we would infer very diferent things about our ASR system's quality and what required improving depending on how we normalise transcripts. The dataset therefore reveals, how this standard approach to ASR development and evaluation breaks down in the context of Langa.