prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How is non-standard pronunciation identified?
Context 1: Introduction
Recent years have seen unprecedented progress for Natural Language Processing (NLP) on almost every NLP subtask  Even though low-resource settings have also been explored, this progress has overwhelmingly been observed in languages with significant data resources that can be leveraged to train deep neural networks  Low-resource languages still lag behind 
Endangered languages pose an additional challenge  The process of documenting an endangered language typically includes the creation of word lists, audio and video recordings, notes, or grammar fragments, with the created resources then stored in large online linguistics archives  This process is often hindered by the Transcription Bottleneck: the linguistic fieldworker and the language community may not have time to transcribe all of the recordings and may only transcribe segments that are linguistically salient for publication or culturally significant for the creation of community resources 
With this work we make publicly available a large corpus in Mapudungun, a language of the indigenous Mapuche people of southern Chile and western Argentina  We hope to ameliorate the resource gap and the transcription bottleneck in two ways  First, we are providing a larger data set than has previously been available, and second, we are providing baselines for NLP tasks (speech recognition, speech synthesis, and machine translation)  In providing baselines and datasets splits, we hope to further facilitate research on low-resource NLP for this language through our data set  Research on low-resource speech recognition is particularly important in relieving the transcription bottleneck, while tackling the research challenges that speech synthesis and machine translation pose for such languages could lead to such systems being deployed to serve more under-represented communities 
The Mapudungun Language
Mapudungun (iso 639-3: arn) is an indigenous language of the Americas spoken natively in Chile and Argentina, with an estimated 100 to 200 thousand speakers in Chile and 27 to 60 thousand speakers in Argentina BIBREF0  It is an isolate language and is classified as threatened by Ethnologue, hence the critical importance of all documentary efforts  Although the morphology of nouns is relatively simple, Mapudungun verb morphology is highly agglutinative and complex  Some analyses provide as many as 36 verb suffix slots BIBREF1  A typical complex verb form occurring in our corpus of spoken Mapudungun consists of five or six morphemes 
Mapudungun has several interesting grammatical properties  It is a polysynthetic language in the sense of BIBREF2; see BIBREF3 for explicit argumentation  As with other polysynthetic languages, Mapudungun has Noun Incorporation; however, it is unique insofar as the Noun appears to the right of the Verb, instead of to the left, as in most polysynthetic languages BIBREF4  One further distinction of Mapudungun is that, whereas other polysynthetic languages are characterized by a lack of infinitives, Mapudungun has infinitival verb forms; that is, while subordinate clauses in Mapudungun closely resemble possessed nominals and may occur with an analytic marker resembling possessor agreement, there is no agreement inflection on the verb itself  One further remarkable property of Mapudungun is its inverse voice system of agreement, whereby the highest agreement is with the argument highest in an animacy hierarchy regardless of thematic role BIBREF5 
The Resource
The resource is comprised of 142 hours of spoken Mapudungun that was recorded during the AVENUE project BIBREF6 in 2001 to 2005  The data was recorded under a partnership between the AVENUE project, funded by the US National Science Foundation at Carnegie Mellon University, the Chilean Ministry of Education (Mineduc), and the Instituto de Estudios Indígenas at Universidad de La Frontera, originally spanning 170 hours of audio  We have recently cleaned the data and are releasing it publicly for the first time (although it has been shared with individual researchers in the past) along with NLP baselines 
The recordings were transcribed and translated into Spanish at the Instituto de Estudios Indígenas at Universidad de La Frontera  The corpus covers three dialects of Mapudungun: about 110 hours of Nguluche, 20 hours of Lafkenche and 10 hours of Pewenche  The three dialects are quite similar, with some minor semantic and phonetic differences  The fourth traditionally distinguished dialect, Huilliche, has several grammatical differences from the other three and is classified by Ethnologue as a separate language, iso 639-3: huh, and as nearly extinct 
The recordings are restricted to a single domain: primary, preventive, and treatment health care, including both Western and Mapuche traditional medicine  The recording sessions were conducted as interactive conversations so as to be natural in Mapuche culture, and they were open-ended, following an ethnographic approach  The interviewer was trained in these methods along with the use of the digital recording systems that were available at the time  We also followed human subject protocol  Each person signed a consent form to release the recordings for research purposes and the data have been accordingly anonymized  Because Machi (traditional Mapuche healers) were interviewed, we asked the transcribers to delete any culturally proprietary knowledge that a Machi may have revealed during the conversation  Similarly, we deleted any names or any information that may identify the participants 
The corpus is culturally relevant because it was created by Mapuche people, using traditional ways of relating to each other in conversations  They discussed personal experiences with primary health care in the traditional Mapuche system and the Chilean health care system, talking about illnesses and the way they were cured  The participants ranged from 16 years old to 100 years old, almost in equal numbers of men and women, and they were all native speakers of Mapudungun 
The Resource ::: Orthography
At the time of the collection and transcription of the corpus, the orthography of Mapudungun was not standardized  The Mapuche team at the Instituto de Estudios Indígenas (IEI – Institute for Indigenous Studies) developed a supra-dialectal alphabet that comprises 28 letters that cover 32 phones used in the three Mapudungun variants  The main criterion for choosing alphabetic characters was to use the current Spanish keyboard that was available on all computers in Chilean offices and schools  The alphabet used the same letters used in Spanish for those phonemes that sound like Spanish phonemes  Diacritics such as apostrophes were used for sounds that are not found in Spanish 
As a result, certain orthographic conventions that were made at the time deviate from the now-standard orthography of Mapudungun, Azumchefe  We plan to normalize the orthography of the corpus, and in fact a small sample has already been converted to the modern orthography  However, we believe that the original transcriptions will also be invaluable for academic, historical, and cultural purposes, hence we release the corpus using these conventions 
The Resource ::: Additional Annotations
In addition, the transcription includes annotations for noises and disfluencies including aborted words, mispronunciations, poor intelligibility, repeated and corrected words, false starts, hesitations, undefined sound or pronunciations, non-verbal articulations, and pauses  Foreign words, in this case Spanish words, are also labelled as such 
The Resource ::: Cleaning
The dialogues were originally recorded using a Sony DAT recorder (48kHz), model TCD-D8, and Sony digital stereo microphone, model ECM-DS70P  Transcription was performed with the TransEdit transcription tool v 1 1 beta 10, which synchronizes the transcribed text and the wave files 
However, we found that a non-trivial number of the utterance boundaries and speaker annotations were flawed  Also some recording sessions did not have a complete set of matching audio, transcription, and translation files  Hence, in an effort to provide a relatively “clean" corpus for modern computational experiments, we converted the encoding of the textual transcription from Latin-1 to Unicode, DOS to UNIX line endings, a now more standard text encoding format than what was used when the data was first collected  Additionally, we renamed a small portion of files which had been misnamed and removed several duplicate files 
Although all of the data was recorded with similar equipment in relatively quiet environments, the acoustics are not as uniform as we would like for building speech synthesizers  Thus we applied standardized power normalization  We also moved the boundaries of the turns to standardize the amount of leading and trailing silence in each turn  This is a standard procedure for speech recognition and synthesis datasets  Finally we used the techniques in BIBREF7 for found data to re-align the text to the audio and find out which turns are best (or worst) aligned so that we can select segments that give the most accurate alignments  Some of the misalignments may in part be due to varied orthography, and we intend, but have not yet, to investigate normalization of orthography (i e  spelling correction) to mitigate this 
The Resource ::: Training, Dev, and Test Splits
We create two training sets, one appropriate for single-speaker speech synthesis experiments, and one appropriate for multiple-speaker speech recognition and machine translation experiments  In both cases, our training, development, and test splits are performed at the dialogue level, so that all examples from each dialogue belong to exactly one of these sets 
For single-speaker speech synthesis, we only use the dialog turns of the speaker with the largest volume of data (nmlch – one of the interviewers)  The training set includes $221 8$ thousand sentences from 285 dialogues, with 12 and 46 conversations reserved for the development and test set 
For speech recognition experiments, we ensure that our test set includes unique speakers as well as speakers that overlap with the training set, in order to allow for comparisons of the ability of the speech recognition system to generalize over seen and new speakers  For consistency, we use the same dataset splits for the machine translation experiments  The statistics in Table reflect this split 
Applications
Our resource has the potential to be the basis of computational research in Mapudungun across several areas  Since the collected audio has been transcribed, our resource is appropriate for the study of automatic speech recognition and speech synthesis  The Spanish translations enable the creation of machine translation systems between Mapudungun and Spanish, as well as end-to-end (or direct) speech translation  We in fact built such speech synthesis, speech recognition, and machine translation systems as a showcase of the usefulness of our corpus in that research direction 
Furthermore, our annotations of the Spanish words interspersed in Mapudungun speech could allow for a study of code-switching patterns within the Mapuche community  In addition, our annotations of non-standardized orthographic transcriptions could be extremely useful in the study of historical language and orthography change as a language moves from predominantly oral to being written in a standardized orthography, as well as in building spelling normalization and correction systems  The relatively large amount of data that we collected will also allow for the training of large language models, which in turn could be used as the basis for predictive keyboards tailored to Mapudungun  Last, since all data are dialogues annotated for the different speaker turns, they could be useful for building Mapudungun dialogue systems and chatbot-like applications 
The potential applications of our resource, however, are not exhausted in language technologies  The resource as a whole could be invaluable for ethnographic and sociological research, as the conversations contrast traditional and Western medicine practices, and they could reveal interesting aspects of the Mapuche culture 
In addition, the corpus is a goldmine of data for studying the morphostyntax of Mapudungun BIBREF8  As an isolate polysynthetic language, the study of Mapudungun can provide insights into the range of possibilities within human languages can work 
Baseline Results
Using the aforementioned higher quality portions of the corpus, we trained baseline systems for Mapudungun speech recognition and speech synthesis, as well as Machine Translation systems between Mapudungun and Spanish 
Baseline Results ::: Speech Synthesis
In our previous work on building speech systems on found data in 700 languages, BIBREF7, we addressed alignment issues (when audio is not segmented into turn/sentence sized chunks) and correctness issues (when the audio does not match the transcription)  We used the same techniques here, as described above 
For the best quality speech synthesis we need a few hours of phonetically-balanced, single-speaker, read speech  Our first step was to use the start and end points for each turn in the dialogues, and select those of the most frequent speaker, nmlch  This gave us around 18250 segments  We further automatically removed excessive silence from the start, middle and end of these turns (based on occurrence of F0)  This gave us 13 hours and 48 minutes of speech 
We phonetically aligned this data and built a speech clustergen statistical speech synthesizer BIBREF9 from all of this data  We resynthesized all of the data and measured the difference between the synthesized data and the original data using Mel Cepstral Distortion, a standard method for automatically measuring quality of speech generation BIBREF10  We then ordered the segments by their generation score and took the top 2000 turns to build a new synthesizer, assuming the better scores corresponded to better alignments, following the techniques of BIBREF7 
The initial build gave an MCD on held out data of 6 483  While the 2000 best segment dataset gives an MCD of 5 551, which is a large improvement  The quality of the generated speech goes from understandable, only if you can see the text, to understandable, and transcribable even for non-Mapudungun speakers  We do not believe we are building the best synthesizer with our current (non-neural) techniques, but we do believe we are selecting the best training data for other statistical and neural training techniques in both speech synthesis and speech recognition 
Baseline Results ::: Speech Recognition
For speech recognition (ASR) we used Kaldi BIBREF11  As we do not have access to pronunciation lexica for Mapudungun, we had to approximate them with two settings  In the first setting, we make the simple assumption that each character corresponds to a pronunced phoneme  In the second setting, we instead used the generated phonetic lexicon also used in the above-mentioned speech synthesis techniques  The train/dev/test splits are across conversations, as described above 
Under the first setting, we obtained a 60% character error rate, while the generated lexicon significantly boosts performance, as our systems achieve a notably reduced 30% phone error rate  Naturally, these results are relatively far from the quality of ASR systems trained on large amounts of clean data such as those available in English  Given the quality of the recordings, and the lack of additional resources, we consider our results fairly reasonable and they would still be usable for simple dialog-like tasks  We anticipate, though, that one could significantly improve ASR quality over our dataset, by using in-domain language models, or by training end-to-end neural recognizers leveraging languages with similar phonetic inventories BIBREF12 or by using the available Spanish translations in a multi-source scenario BIBREF13 
Baseline Results ::: Mapudungun–Spanish Machine Translation
We built neural end-to-end machine translation systems between Mapudungun and Spanish in both directions, using state-of-the-art Transformer architecture BIBREF14 with the toolkit of BIBREF15  We train our systems at the subword level using Byte-Pair Encoding BIBREF16 with a vocabulary of 5000 subwords, shared between the source and target languages  We use five layers for each of the encoder and the decoder, an embedding size of 512, feed forward transformation size of 2048, and eight attention heads  We use dropout BIBREF17 with $0 4$ probability as well as label smoothing set to $0 1$  We train with the Adam optimizer BIBREF18 for up to 200 epochs using learning decay with a patience of six epochs 
The baseline results using different portions of the training set (10k, 50k, 100k, and all (220k) parallel sentences) on both translation directions are presented in Table , using detokenized BLEU BIBREF19 (a standard MT metric) and chrF BIBREF20 (a metric that we consider to be more appropriate for polysynthetic languages, as it does not rely on word n-grams) computed with the sacreBLEU toolkit BIBREF21  It it worth noting the difference in quality between the two directions, with translation into Spanish reaching 20 4 (almost 21) BLEU points in the development set, while the opposite direction (translating into Mapudungun) shows about a 7 BLEU points worse performance  This is most likely due to Mapudungun being a polysynthetic language, with its complicated morphology posing a challenge for proper generation 
Related Work
Mapudungun grammar has been studied since the arrival of European missionaries and colonizers hundreds of years ago  More recent descriptions of Mapudungun grammar BIBREF1 and BIBREF0 informed the collection of the resource that we are presenting in this paper 
Portions of our resource have been used in early efforts to build language systems for Mapudungun  In particular, BIBREF22 focused on Mapudungun morphology in order to create spelling correction systems, while BIBREF23, BIBREF6, BIBREF24, and BIBREF25 developed hybrid rule- and phrase-based Statistical Machine Translation systems 
Naturally, similar works in collecting corpora in Indigenous languages of Latin America are abundant, but very few, if any, have the scale and potential of our resource to be useful in many downstream language-specific and inter-disciplinary applications  A general overview of the state of NLP for the under-represented languages of the Americas can be found at BIBREF26  To name a few of the many notable works, BIBREF27 created a parallel Mixtec-Spanish corpus for Machine Translation and BIBREF28 created lexical resources for Arapaho, while BIBREF29 and BIBREF30 focused on building speech corpora for Southern Quechua and Chatino respectively 
Conclusion
With this work we present a resource that will be extremely useful for building language systems in an endangered, under-represented language, Mapudungun  We benchmark NLP systems for speech synthesis, speech recognition, and machine translation, providing strong baseline results  The size of our resource (142 hours, more than 260k total sentences) has the potential to alleviate many of the issues faced when building language technologies for Mapudungun, in contrast to other indigenous languages of the Americas that unfortunately remain low-resource 
Our resource could also be used for ethnographic and anthropological research into the Mapuche culture, and has the potential to contribute to intercultural bilingual education, preservation activities and further general advancement of the Mapudungun-speaking community 
Acknowledgements
The data collection described in this paper was supported by NSF grants IIS-0121631 (AVENUE) and IIS-0534217 (LETRAS), with supplemental funding from NSF's Office of International Science and Education  Preliminary funding for work on Mapudungun was also provided by DARPA The experimental material is based upon work generously supported by the National Science Foundation under grant 1761548 
Context 2: INTRODUCTION
The idea of language identification is to classify a given audio signal into a particular class using a classification algorithm  Commonly language identification task was done using i-vector systems [1]  A very well known approach for language identification proposed by N  Dahek et al  [1] uses the GMM-UBM model to obtain utterance level features called i-vectors  Recent advances in deep learning [15,16] have helped to improve the language identification task using many different neural network architectures which can be trained efficiently using GPUs for large scale datasets  These neural networks can be configured in various ways to obtain better accuracy for language identification task  Early work on using Deep learning for language Identification was published by Pavel Matejka et al  [2], where they used stacked bottleneck features extracted from deep neural networks for language identification task and showed that the bottleneck features learned by Deep neural networks are better than simple MFCC or PLP features  Later the work by I  Lopez-Moreno et al  [3] from Google showed how to use Deep neural networks to directly map the sequence of MFCC frames into its language class so that we can apply language identification at the frame level  Speech signals will have both spatial and temporal information, but simple DNNs are not able to capture temporal information  Work done by J  Gonzalez-Dominguez et al  [4] by Google developed an LSTM based language identification model which improves the accuracy over the DNN based models  Work done by Alicia et al  [5] used CNNs to improve upon i-vector [1] and other previously developed systems  The work done by Daniel Garcia-Romero et al  [6] has used a combination of Acoustic model trained for speech recognition with Time-delay neural networks where they train the TDNN model by feeding the stacked bottleneck features from acoustic model to predict the language labels at the frame level  Recently X-vectors [7] is proposed for speaker identification task and are shown to outperform all the previous state of the art speaker identification algorithms and are also used for language identification by David Snyder et al  [8] 
In this paper, we explore multiple pooling strategies for language identification task  Mainly we propose Ghost-VLAD based pooling method for language identification  Inspired by the recent work by W  Xie et al  [9] and Y  Zhong et al  [10], we use Ghost-VLAD to improve the accuracy of language identification task for Indian languages  We explore multiple pooling strategies including NetVLAD pooling [11], Average pooling and Statistics pooling( as proposed in X-vectors [7]) and show that Ghost-VLAD pooling is the best pooling strategy for language identification  Our model obtains the best accuracy of 98 24%, and it outperforms all the other previously proposed pooling methods  We conduct all our experiments on 635hrs of audio data for 7 Indian languages collected from $\textbf {All India Radio}$ news channel  The paper is organized as follows  In section 2, we explain the proposed pooling method for language identification  In section 3, we explain our dataset  In section 4, we describe the experiments, and in section 5, we describe the results 
POOLING STRATEGIES
In any language identification model, we want to obtain utterance level representation which has very good language discriminative features  These representations should be compact and should be easily separable by a linear classifier  The idea of any pooling strategy is to pool the frame-level representations into a single utterance level representation  Previous works by [7] have used simple mean and standard deviation aggregation to pool the frame-level features from the top layer of the neural network to obtain the utterance level features  Recently [9] used VLAD based pooling strategy for speaker identification which is inspired from [10] proposed for face recognition  The NetVLAD [11] and Ghost-VLAD [10] methods are proposed for Place recognition and face recognition, respectively, and in both cases, they try to aggregate the local descriptors into global features  In our case, the local descriptors are features extracted from ResNet [15], and the global utterance level feature is obtained by using GhostVLAD pooling  In this section, we explain different pooling methods, including NetVLAD, Ghost-VLAD, Statistic pooling, and Average pooling 
POOLING STRATEGIES ::: NetVLAD pooling
The NetVLAD pooling strategy was initially developed for place recognition by R  Arandjelovic et al  [11]  The NetVLAD is an extension to VLAD [18] approach where they were able to replace the hard assignment based clustering with soft assignment based clustering so that it can be trained with neural network in an end to end fashion  In our case, we use the NetVLAD layer to map N local features of dimension D into a fixed dimensional vector, as shown in Figure 1 (Left side) 
The model takes spectrogram as an input and feeds into CNN based ResNet architecture  The ResNet is used to map the spectrogram into 3D feature map of dimension HxWxD  We convert this 3D feature map into 2D by unfolding H and W dimensions, creating a NxD dimensional feature map, where N=HxW  The NetVLAD layer is kept on top of the feature extraction layer of ResNet, as shown in Figure 1  The NetVLAD now takes N features vectors of dimension D and computes a matrix V of dimension KxD, where K is the number clusters in the NetVLAD layer, and D is the dimension of the feature vector  The matrix V is computed as follows 
Where $w_k$,$b_k$ and $c_k$ are trainable parameters for the cluster $k$ and V(j,k) represents a point in the V matrix for (j,k)th location  The matrix is constructed using the equation (1) where the first term corresponds to the soft assignment of the input $x_i$ to the cluster $c_k$, whereas the second term corresponds to the residual term which tells how far the input descriptor $x_i$ is from the cluster center $c_k$ 
POOLING STRATEGIES ::: GhostVLAD pooling
GhostVLAD is an extension of the NetVLAD approach, which we discussed in the previous section  The GhostVLAD model was proposed for face recognition by Y  Zhong [10]  GhostVLAD works exactly similar to NetVLAD except it adds Ghost clusters along with the NetVLAD clusters  So, now we will have a K+G number of clusters instead of K clusters  Where G is the number of ghost clusters, we want to add (typically 2-4)  The Ghost clusters are added to map any noisy or irrelevant content into ghost clusters and are not included during the feature aggregation stage, as shown in Figure 1 (Right side)  Which means that we compute the matrix V for both normal cluster K and ghost clusters G, but we will not include the vectors belongs to ghost cluster from V during concatenation of the features  Due to which, during feature aggregation stage the contribution of the noisy and unwanted features to normal VLAD clusters are assigned less weights while Ghost clusters absorb most of the weight  We illustrate this in Figure 1(Right Side), where the ghost clusters are shown in red color  We use Ghost clusters when we are computing the V matrix, but they are excluded during the concatenation stage  These concatenated features are fed into the projection layer, followed by softmax to predict the language label 
POOLING STRATEGIES ::: Statistic and average pooling
In statistic pooling, we compute the first and second order statistics of the local features from the top layer of the ResNet model  The 3-D feature map is unfolded to create N features of D dimensions, and then we compute the mean and standard deviation of all these N vectors and get two D dimensional vectors, one for mean and the other for standard deviation  We then concatenate these 2 features and feed it to the projection layer for predicting the language label 
In the Average pooling layer, we compute only the first-order statistics (mean) of the local features from the top layer of the CNN model  The feature map from the top layer of CNN is unfolded to create N features of D dimensions, and then we compute the mean of all these N vectors and get D dimensional representation  We then feed this feature to the projection layer followed by softmax for predicting the language label 
DATASET
In this section, we describe our dataset collection process  We collected and curated around 635Hrs of audio data for 7 Indian languages, namely Kannada, Hindi, Telugu, Malayalam, Bengali, and English  We collected the data from the All India Radio news channel where an actor will be reading news for about 5-10 mins  To cover many speakers for the dataset, we crawled data from 2010 to 2019  Since the audio is very long to train any deep neural network directly, we segment the audio clips into smaller chunks using Voice activity detector  Since the audio clips will have music embedded during the news, we use Inhouse music detection model to remove the music segments from the dataset to make the dataset clean and our dataset contains 635Hrs of clean audio which is divided into 520Hrs of training data containing 165K utterances and 115Hrs of testing data containing 35K utterances  The amount of audio data for training and testing for each of the language is shown in the table bellow 
EXPERIMENTS
In this section, we describe the feature extraction process and network architecture in detail  We use spectral features of 256 dimensions computed using 512 point FFT for every frame, and we add an energy feature for every frame giving us total 257 features for every frame  We use a window size of 25ms and frame shift of 10ms during feature computation  We crop random 5sec audio data from each utterance during training which results in a spectrogram of size 257x500 (features x number of features)  We use these spectrograms as input to our CNN model during training  During testing, we compute the prediction score irrespective of the audio length 
For the network architecture, we use ResNet-34 architecture, as described in [9]  The model uses convolution layers with Relu activations to map the spectrogram of size 257x500 input into 3D feature map of size 1x32x512  This feature cube is converted into 2D feature map of dimension 32x512 and fed into Ghost-VLAD/NetVLAD layer to generate a representation that has more language discrimination capacity  We use Adam optimizer with an initial learning rate of 0 01 and a final learning rate of 0 00001 for training  Each model is trained for 15 epochs with early stopping criteria 
For the baseline, we train an i-vector model using GMM-UBM  We fit a small classifier on top of the generated i-vectors to measure the accuracy  This model is referred as i-vector+svm   To compare our model with the previous state of the art system, we set up the x-vector language identification system [8]  The x-vector model used time-delay neural networks (TDNN) along with statistic-pooling  We use 7 layer TDNN architecture similar to [8] for training  We refer to this model as tdnn+stat-pool   Finally, we set up a Deep LSTM based language identification system similar to [4] but with little modification where we add statistics pooling for the last layers hidden activities before classification  We use 3 layer Bi-LSTM with 256 hidden units at each layer  We refer to this model as LSTM+stat-pool  We train our i-vector+svm and TDNN+stat-pool using Kaldi toolkit  We train our NetVLAD and GhostVLAD experiments using Keras by modifying the code given by [9] for language identification  We train the LSTM+stat-pool and the remaining experiments using Pytorch [14] toolkit, and we will opensource all the codes and data soon 
RESULTS
In this section, we compare the performance of our system with the recent state of the art language identification approaches  We also compare different pooling strategies and finally, compare the robustness of our system to the length of the input spectrogram during training  We visualize the embeddings generated by the GhostVLAD method and conclude that the GhostVLAD embeddings shows very good feature discrimination capabilities 
RESULTS ::: Comparison with different approaches
We compare our system performance with the previous state of the art language identification approaches, as shown in Table 2  The i-vector+svm system is trained using GMM-UBM models to generate i-vectors as proposed in [1]  Once the i-vectors are extracted, we fit SVM classifier to classify the audio  The TDNN+stat-pool system is trained with a statistics pooling layer and is called the x-vector system as proposed by David Snyder et al  [11] and is currently the state of the art language identification approach as far as our knowledge  Our methods outperform the state of the art x-vector system by absolute 1 88% improvement in F1-score, as shown in Table 2 
RESULTS ::: Comparison with different pooling techniques
We compare our approach with different pooling strategies in Table 3  We use ResNet as our base feature extraction network  We keep the base network the same and change only the pooling layers to see which pooling approach performs better for language identification task  Our experiments show that GhostVLAD pooling outperforms all the other pooling methods by achieving 98 43% F1-Score 
RESULTS ::: Duration analysis
To observe the performance of our method with different input durations, we conducted an experiment where we train our model on different input durations  Since our model uses ResNet as the base feature extractor, we need to feed fixed-length spectrogram  We conducted 4 different experiments where we trained the model using 2sec, 3sec, 4sec and 5sec spectrograms containing 200,300,400 and 500 frames respectively  We observed that the model trained with a 5sec spectrogram is the best model, as shown in Table 4 
RESULTS ::: Visualization of embeddings
We visualize the embeddings generated by our approach to see the effectiveness  We extracted 512-dimensional embeddings for our testing data and reduced the dimensionality using t-sne projection  The t-sne plot of the embeddings space is shown in Figure 3  The plot shows that the embeddings learned by our approach has very good discriminative properties
Conclusion
In this work, we use Ghost-VLAD pooling approach that was originally proposed for face recognition to improve language identification performance for Indian languages  We collected and curated 630 hrs audio data from news All India Radio news channel for 7 Indian languages  Our experimental results shows that our approach outperforms the previous state of the art methods by an absolute 1 88% F1-score  We have also conducted experiments with different pooling strategies proposed in the past, and the GhostVLAD pooling approach turns out to be the best approach for aggregating frame-level features into a single utterance level feature  Our experiments also prove that our approach works much better even if the input during training contains smaller durations  Finally, we see that the embeddings generated by our method has very good language discriminative features and helps to improve the performance of language identification 
Context 3: Introduction
Decoding intended speech or motor activity from brain signals is one of the major research areas in Brain Computer Interface (BCI) systems BIBREF0 , BIBREF1   In particular, speech-related BCI technologies attempt to provide effective vocal communication strategies for controlling external devices through speech commands interpreted from brain signals BIBREF2   Not only do they provide neuro-prosthetic help for people with speaking disabilities and neuro-muscular disorders like locked-in-syndrome, nasopharyngeal cancer, and amytotropic lateral sclerosis (ALS), but also equip people with a better medium to communicate and express thoughts, thereby improving the quality of rehabilitation and clinical neurology BIBREF3 , BIBREF4   Such devices also have applications in entertainment, preventive treatments, personal communication, games, etc  Furthermore, BCI technologies can be utilized in silent communication, as in noisy environments, or situations where any sort of audio-visual communication is infeasible 
Among the various brain activity-monitoring modalities in BCI, electroencephalography (EEG) BIBREF5 , BIBREF6 has demonstrated promising potential to differentiate between various brain activities through measurement of related electric fields  EEG is non-invasive, portable, low cost, and provides satisfactory temporal resolution  This makes EEG suitable to realize BCI systems  EEG data, however, is challenging: these data are high dimensional, have poor SNR, and suffer from low spatial resolution and a multitude of artifacts  For these reasons, it is not particularly obvious how to decode the desired information from raw EEG signals  Although the area of BCI based speech intent recognition has received increasing attention among the research community in the past few years, most research has focused on classification of individual speech categories in terms of discrete vowels, phonemes and words BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15   This includes categorization of imagined EEG signal into binary vowel categories like /a/, /u/ and rest BIBREF7 , BIBREF8 , BIBREF9 ; binary syllable classes like /ba/ and /ku/ BIBREF1 , BIBREF10 , BIBREF11 , BIBREF12 ; a handful of control words like 'up', 'down', 'left', 'right' and 'select' BIBREF15 or others like 'water', 'help', 'thanks', 'food', 'stop' BIBREF13 , Chinese characters BIBREF14 , etc  Such works mostly involve traditional signal processing or manual feature handcrafting along with linear classifiers (e g , SVMs)  In our recent work BIBREF16 , we introduced deep learning models for classification of vowels and words that achieved 23 45% improvement of accuracy over the baseline 
Production of articulatory speech is an extremely complicated process, thereby rendering understanding of the discriminative EEG manifold corresponding to imagined speech highly challenging  As a result, most of the existing approaches failed to achieve satisfactory accuracy on decoding speech tokens from the speech imagery EEG data  Perhaps, for these reasons, very little work has been devoted to relating the brain signals to the underlying articulation  The few exceptions include BIBREF17 , BIBREF18   In BIBREF17 , Zhao et al  used manually handcrafted features from EEG data, combined with speech audio and facial features to achieve classification of the phonological categories varying based on the articulatory steps  However, the imagined speech classification accuracy based on EEG data alone, as reported in BIBREF17 , BIBREF18 , are not satisfactory in terms of accuracy and reliability  We now turn to describing our proposed models 
Proposed Framework
Cognitive learning process underlying articulatory speech production involves incorporation of intermediate feedback loops and utilization of past information stored in the form of memory as well as hierarchical combination of several feature extractors  To this end, we develop our mixed neural network architecture composed of three supervised and a single unsupervised learning step, discussed in the next subsections and shown in Fig  FIGREF1   We formulate the problem of categorizing EEG data based on speech imagery as a non-linear mapping INLINEFORM0 of a multivariate time-series input sequence INLINEFORM1 to fixed output INLINEFORM2 , i e, mathematically INLINEFORM3 : INLINEFORM4 , where c and t denote the EEG channels and time instants respectively 
Preprocessing step
We follow similar pre-processing steps on raw EEG data as reported in BIBREF17 (ocular artifact removal using blind source separation, bandpass filtering and subtracting mean value from each channel) except that we do not perform Laplacian filtering step since such high-pass filtering may decrease information content from the signals in the selected bandwidth 
Joint variability of electrodes
Multichannel EEG data is high dimensional multivariate time series data whose dimensionality depends on the number of electrodes  It is a major hurdle to optimally encode information from these EEG data into lower dimensional space  In fact, our investigation based on a development set (as we explain later) showed that well-known deep neural networks (e g , fully connected networks such as convolutional neural networks, recurrent neural networks and autoencoders) fail to individually learn such complex feature representations from single-trial EEG data  Besides, we found that instead of using the raw multi-channel high-dimensional EEG requiring large training times and resource requirements, it is advantageous to first reduce its dimensionality by capturing the information transfer among the electrodes  Instead of the conventional approach of selecting a handful of channels as BIBREF17 , BIBREF18 , we address this by computing the channel cross-covariance, resulting in positive, semi-definite matrices encoding the connectivity of the electrodes  We define channel cross-covariance (CCV) between any two electrodes INLINEFORM0 and INLINEFORM1 as: INLINEFORM2   Next, we reject the channels which have significantly lower cross-covariance than auto-covariance values (where auto-covariance implies CCV on same electrode)  We found this measure to be essential as the higher cognitive processes underlying speech planning and synthesis involve frequent information exchange between different parts of the brain  Hence, such matrices often contain more discriminative features and hidden information than mere raw signals  This is essentially different than our previous work BIBREF16 where we extract per-channel 1-D covariance information and feed it to the networks  We present our sample 2-D EEG cross-covariance matrices (of two individuals) in Fig  FIGREF2  
CNN & LSTM
In order to decode spatial connections between the electrodes from the channel covariance matrix, we use a CNN BIBREF19 , in particular a four-layered 2D CNN stacking two convolutional and two fully connected hidden layers  The INLINEFORM0 feature map at a given CNN layer with input INLINEFORM1 , weight matrix INLINEFORM2 and bias INLINEFORM3 is obtained as: INLINEFORM4   At this first level of hierarchy, the network is trained with the corresponding labels as target outputs, optimizing a cross-entropy cost function  In parallel, we apply a four-layered recurrent neural network on the channel covariance matrices to explore the hidden temporal features of the electrodes  Namely, we exploit an LSTM BIBREF20 consisting of two fully connected hidden layers, stacked with two LSTM layers and trained in a similar manner as CNN 
Deep autoencoder for spatio-temporal information
As we found the individually-trained parallel networks (CNN and LSTM) to be useful (see Table TABREF12 ), we suspected the combination of these two networks could provide a more powerful discriminative spatial and temporal representation of the data than each independent network  As such, we concatenate the last fully-connected layer from the CNN with its counterpart in the LSTM to compose a single feature vector based on these two penultimate layers  Ultimately, this forms a joint spatio-temporal encoding of the cross-covariance matrix 
In order to further reduce the dimensionality of the spatio-temporal encodings and cancel background noise effects BIBREF21 , we train an unsupervised deep autoenoder (DAE) on the fused heterogeneous features produced by the combined CNN and LSTM information  The DAE forms our second level of hierarchy, with 3 encoding and 3 decoding layers, and mean squared error (MSE) as the cost function 
Classification with Extreme Gradient Boost
At the third level of hierarchy, the discrete latent vector representation of the deep autoencoder is fed into an Extreme Gradient Boost based classification layer BIBREF22 , BIBREF23 motivated by BIBREF21   It is a regularized gradient boosted decision tree that performs well on structured problems  Since our EEG-phonological pairwise classification has an internal structure involving individual phonemes and words, it seems to be a reasonable choice of classifier  The classifier receives its input from the latent vectors of the deep autoencoder and is trained in a supervised manner to output the final predicted classes corresponding to the speech imagery 
Dataset
We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)  The dataset consists of 14 participants, with each prompt presented 11 times to each individual  Since our intention is to classify the phonological categories from human thoughts, we discard the facial and audio information and only consider the EEG data corresponding to imagined speech  It is noteworthy that given the mixed nature of EEG signals, it is reportedly challenging to attain a pairwise EEG-phoneme mapping BIBREF18   In order to explore the problem space, we thus specifically target five binary classification problems addressed in BIBREF17 , BIBREF18 , i e presence/absence of consonants, phonemic nasal, bilabial, high-front vowels and high-back vowels 
Training and hyperparameter selection
We performed two sets of experiments with the single-trial EEG data  In PHASE-ONE, our goals was to identify the best architectures and hyperparameters for our networks with a reasonable number of runs  For PHASE-ONE, we randomly shuffled and divided the data (1913 signals from 14 individuals) into train (80%), development (10%) and test sets (10%)  In PHASE-TWO, in order to perform a fair comparison with the previous methods reported on the same dataset, we perform a leave-one-subject out cross-validation experiment using the best settings we learn from PHASE-ONE 
The architectural parameters and hyperparameters listed in Table TABREF6 were selected through an exhaustive grid-search based on the validation set of PHASE-ONE  We conducted a series of empirical studies starting from single hidden-layered networks for each of the blocks and, based on the validation accuracy, we increased the depth of each given network and selected the optimal parametric set from all possible combinations of parameters  For the gradient boosting classification, we fixed the maximum depth at 10, number of estimators at 5000, learning rate at 0 1, regularization coefficient at 0 3, subsample ratio at 0 8, and column-sample/iteration at 0 4  We did not find any notable change of accuracy while varying other hyperparameters while training gradient boost classifier 
Performance analysis and discussion
To demonstrate the significance of the hierarchical CNN-LSTM-DAE method, we conducted separate experiments with the individual networks in PHASE-ONE of experiments and summarized the results in Table TABREF12 From the average accuracy scores, we observe that the mixed network performs much better than individual blocks which is in agreement with the findings in BIBREF21   A detailed analysis on repeated runs further shows that in most of the cases, LSTM alone does not perform better than chance  CNN, on the other hand, is heavily biased towards the class label which sees more training data corresponding to it  Though the situation improves with combined CNN-LSTM, our analysis clearly shows the necessity of a better encoding scheme to utilize the combined features rather than mere concatenation of the penultimate features of both networks 
The very fact that our combined network improves the classification accuracy by a mean margin of 14 45% than the CNN-LSTM network indeed reveals that the autoencoder contributes towards filtering out the unrelated and noisy features from the concatenated penultimate feature set  It also proves that the combined supervised and unsupervised neural networks, trained hierarchically, can learn the discriminative manifold better than the individual networks and it is crucial for improving the classification accuracy  In addition to accuracy, we also provide the kappa coefficients BIBREF24 of our method in Fig  FIGREF14   Here, a higher mean kappa value corresponding to a task implies that the network is able to find better discriminative information from the EEG data beyond random decisions  The maximum above-chance accuracy (75 92%) is recorded for presence/absence of the vowel task and the minimum (49 14%) is recorded for the INLINEFORM0  
To further investigate the feature representation achieved by our model, we plot T-distributed Stochastic Neighbor Embedding (tSNE) corresponding to INLINEFORM0 and V/C classification tasks in Fig  FIGREF8   We particularly select these two tasks as our model exhibits respectively minimum and maximum performance for these two  The tSNE visualization reveals that the second set of features are more easily separable than the first one, thereby giving a rationale for our performance 
Next, we provide performance comparison of the proposed approach with the baseline methods for PHASE-TWO of our study (cross-validation experiment) in Table TABREF15   Since the model encounters the unseen data of a new subject for testing, and given the high inter-subject variability of the EEG data, a reduction in the accuracy was expected  However, our network still managed to achieve an improvement of 18 91, 9 95, 67 15, 2 83 and 13 70 % over BIBREF17   Besides, our best model shows more reliability compared to previous works: The standard deviation of our model's classification accuracy across all the tasks is reduced from 22 59% BIBREF17 and 17 52% BIBREF18 to a mere 5 41% 
Conclusion and future direction
In an attempt to move a step towards understanding the speech information encoded in brain signals, we developed a novel mixed deep neural network scheme for a number of binary classification tasks from speech imagery EEG data  Unlike previous approaches which mostly deal with subject-dependent classification of EEG into discrete vowel or word labels, this work investigates a subject-invariant mapping of EEG data with different phonological categories, varying widely in terms of underlying articulator motions (eg: involvement or non-involvement of lips and velum, variation of tongue movements etc)  Our model takes an advantage of feature extraction capability of CNN, LSTM as well as the deep learning benefit of deep autoencoders  We took BIBREF17 , BIBREF18 as the baseline works investigating the same problem and compared our performance with theirs  Our proposed method highly outperforms the existing methods across all the five binary classification tasks by a large average margin of 22 51% 
Acknowledgments
This work was funded by the Natural Sciences and Engineering Research Council (NSERC) of Canada and Canadian Institutes for Health Research (CIHR) 

ground_truth: Unanswerable

Original transcription was labeled with additional labels in [] brackets with nonstandard pronunciation.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the GhostVLAD approach?
Context 4: Introduction
Abusive language refers to any type of insult, vulgarity, or profanity that debases the target; it also can be anything that causes aggravation BIBREF0 , BIBREF1   Abusive language is often reframed as, but not limited to, offensive language BIBREF2 , cyberbullying BIBREF3 , othering language BIBREF4 , and hate speech BIBREF5  
Recently, an increasing number of users have been subjected to harassment, or have witnessed offensive behaviors online BIBREF6   Major social media companies (i e  Facebook, Twitter) have utilized multiple resources—artificial intelligence, human reviewers, user reporting processes, etc —in effort to censor offensive language, yet it seems nearly impossible to successfully resolve the issue BIBREF7 , BIBREF8  
The major reason of the failure in abusive language detection comes from its subjectivity and context-dependent characteristics BIBREF9   For instance, a message can be regarded as harmless on its own, but when taking previous threads into account it may be seen as abusive, and vice versa  This aspect makes detecting abusive language extremely laborious even for human annotators; therefore it is difficult to build a large and reliable dataset BIBREF10  
Previously, datasets openly available in abusive language detection research on Twitter ranged from 10K to 35K in size BIBREF9 , BIBREF11   This quantity is not sufficient to train the significant number of parameters in deep learning models  Due to this reason, these datasets have been mainly studied by traditional machine learning methods  Most recently, Founta et al  founta2018large introduced Hate and Abusive Speech on Twitter, a dataset containing 100K tweets with cross-validated labels  Although this corpus has great potential in training deep models with its significant size, there are no baseline reports to date 
This paper investigates the efficacy of different learning models in detecting abusive language  We compare accuracy using the most frequently studied machine learning classifiers as well as recent neural network models  Reliable baseline results are presented with the first comparative study on this dataset  Additionally, we demonstrate the effect of different features and variants, and describe the possibility for further improvements with the use of ensemble models 
Related Work
The research community introduced various approaches on abusive language detection  Razavi et al  razavi2010offensive applied Naïve Bayes, and Warner and Hirschberg warner2012detecting used Support Vector Machine (SVM), both with word-level features to classify offensive language  Xiang et al  xiang2012detecting generated topic distributions with Latent Dirichlet Allocation BIBREF12 , also using word-level features in order to classify offensive tweets 
More recently, distributed word representations and neural network models have been widely applied for abusive language detection  Djuric et al  djuric2015hate used the Continuous Bag Of Words model with paragraph2vec algorithm BIBREF13 to more accurately detect hate speech than that of the plain Bag Of Words models  Badjatiya et al  badjatiya2017deep implemented Gradient Boosted Decision Trees classifiers using word representations trained by deep learning models  Other researchers have investigated character-level representations and their effectiveness compared to word-level representations BIBREF14 , BIBREF15  
As traditional machine learning methods have relied on feature engineering, (i e  n-grams, POS tags, user information) BIBREF1 , researchers have proposed neural-based models with the advent of larger datasets  Convolutional Neural Networks and Recurrent Neural Networks have been applied to detect abusive language, and they have outperformed traditional machine learning classifiers such as Logistic Regression and SVM BIBREF15 , BIBREF16   However, there are no studies investigating the efficiency of neural models with large-scale datasets over 100K 
Methodology
This section illustrates our implementations on traditional machine learning classifiers and neural network based models in detail  Furthermore, we describe additional features and variant models investigated 
Traditional Machine Learning Models
We implement five feature engineering based machine learning classifiers that are most often used for abusive language detection  In data preprocessing, text sequences are converted into Bag Of Words (BOW) representations, and normalized with Term Frequency-Inverse Document Frequency (TF-IDF) values  We experiment with word-level features using n-grams ranging from 1 to 3, and character-level features from 3 to 8-grams  Each classifier is implemented with the following specifications:
Naïve Bayes (NB): Multinomial NB with additive smoothing constant 1
Logistic Regression (LR): Linear LR with L2 regularization constant 1 and limited-memory BFGS optimization
Support Vector Machine (SVM): Linear SVM with L2 regularization constant 1 and logistic loss function
Random Forests (RF): Averaging probabilistic predictions of 10 randomized decision trees
Gradient Boosted Trees (GBT): Tree boosting with learning rate 1 and logistic loss function
Neural Network based Models
Along with traditional machine learning approaches, we investigate neural network based models to evaluate their efficacy within a larger dataset  In particular, we explore Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and their variant models  A pre-trained GloVe BIBREF17 representation is used for word-level features 
CNN: We adopt Kim's kim2014convolutional implementation as the baseline  The word-level CNN models have 3 convolutional filters of different sizes [1,2,3] with ReLU activation, and a max-pooling layer  For the character-level CNN, we use 6 convolutional filters of various sizes [3,4,5,6,7,8], then add max-pooling layers followed by 1 fully-connected layer with a dimension of 1024 
Park and Fung park2017one proposed a HybridCNN model which outperformed both word-level and character-level CNNs in abusive language detection  In order to evaluate the HybridCNN for this dataset, we concatenate the output of max-pooled layers from word-level and character-level CNN, and feed this vector to a fully-connected layer in order to predict the output 
All three CNN models (word-level, character-level, and hybrid) use cross entropy with softmax as their loss function and Adam BIBREF18 as the optimizer 
RNN: We use bidirectional RNN BIBREF19 as the baseline, implementing a GRU BIBREF20 cell for each recurrent unit  From extensive parameter-search experiments, we chose 1 encoding layer with 50 dimensional hidden states and an input dropout probability of 0 3  The RNN models use cross entropy with sigmoid as their loss function and Adam as the optimizer 
For a possible improvement, we apply a self-matching attention mechanism on RNN baseline models BIBREF21 so that they may better understand the data by retrieving text sequences twice  We also investigate a recently introduced method, Latent Topic Clustering (LTC) BIBREF22   The LTC method extracts latent topic information from the hidden states of RNN, and uses it for additional information in classifying the text data 
Feature Extension
While manually analyzing the raw dataset, we noticed that looking at the tweet one has replied to or has quoted, provides significant contextual information  We call these, “context tweets"

ground_truth: extension of the NetVLAD, adds Ghost clusters along with the NetVLAD clusters

An extension of NetVLAD which replaces hard assignment-based clustering with soft assignment-based clustering with the additon o fusing Ghost clusters to deal with noisy content.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What data was presented to the subjects to elicit event-related responses?
Context 4:  As humans can better understand a tweet with the reference of its context, our assumption is that computers also benefit from taking context tweets into account in detecting abusive language 
As shown in the examples below, (2) is labeled abusive due to the use of vulgar language  However, the intention of the user can be better understood with its context tweet (1) 
(1) I hate when I'm sitting in front of the bus and somebody with a wheelchair get on 
INLINEFORM0 (2) I hate it when I'm trying to board a bus and there's already an as**ole on it 
Similarly, context tweet (3) is important in understanding the abusive tweet (4), especially in identifying the target of the malice 
(3) Survivors of #Syria Gas Attack Recount `a Cruel Scene' 
INLINEFORM0 (4) Who the HELL is “LIKE" ING this post? Sick people    
Huang et al  huang2016modeling used several attributes of context tweets for sentiment analysis in order to improve the baseline LSTM model  However, their approach was limited because the meta-information they focused on—author information, conversation type, use of the same hashtags or emojis—are all highly dependent on data 
In order to avoid data dependency, text sequences of context tweets are directly used as an additional feature of neural network models  We use the same baseline model to convert context tweets to vectors, then concatenate these vectors with outputs of their corresponding labeled tweets  More specifically, we concatenate max-pooled layers of context and labeled tweets for the CNN baseline model  As for RNN, the last hidden states of context and labeled tweets are concatenated 
Dataset
Hate and Abusive Speech on Twitter BIBREF10 classifies tweets into 4 labels, “normal", “spam", “hateful" and “abusive"  We were only able to crawl 70,904 tweets out of 99,996 tweet IDs, mainly because the tweet was deleted or the user account had been suspended  Table shows the distribution of labels of the crawled data 
Data Preprocessing
In the data preprocessing steps, user IDs, URLs, and frequently used emojis are replaced as special tokens  Since hashtags tend to have a high correlation with the content of the tweet BIBREF23 , we use a segmentation library BIBREF24 for hashtags to extract more information 
For character-level representations, we apply the method Zhang et al  zhang2015character proposed  Tweets are transformed into one-hot encoded vectors using 70 character dimensions—26 lower-cased alphabets, 10 digits, and 34 special characters including whitespace 
Training and Evaluation
In training the feature engineering based machine learning classifiers, we truncate vector representations according to the TF-IDF values (the top 14,000 and 53,000 for word-level and character-level representations, respectively) to avoid overfitting  For neural network models, words that appear only once are replaced as unknown tokens 
Since the dataset used is not split into train, development, and test sets, we perform 10-fold cross validation, obtaining the average of 5 tries; we divide the dataset randomly by a ratio of 85:5:10, respectively  In order to evaluate the overall performance, we calculate the weighted average of precision, recall, and F1 scores of all four labels, “normal”, “spam”, “hateful”, and “abusive” 
Empirical Results
As shown in Table , neural network models are more accurate than feature engineering based models (i e  NB, SVM, etc ) except for the LR model—the best LR model has the same F1 score as the best CNN model 
Among traditional machine learning models, the most accurate in classifying abusive language is the LR model followed by ensemble models such as GBT and RF  Character-level representations improve F1 scores of SVM and RF classifiers, but they have no positive effect on other models 
For neural network models, RNN with LTC modules have the highest accuracy score, but there are no significant improvements from its baseline model and its attention-added model  Similarly, HybridCNN does not improve the baseline CNN model  For both CNN and RNN models, character-level features significantly decrease the accuracy of classification 
The use of context tweets generally have little effect on baseline models, however they noticeably improve the scores of several metrics  For instance, CNN with context tweets score the highest recall and F1 for “hateful" labels, and RNN models with context tweets have the highest recall for “abusive" tweets 
Discussion and Conclusion
While character-level features are known to improve the accuracy of neural network models BIBREF16 , they reduce classification accuracy for Hate and Abusive Speech on Twitter  We conclude this is because of the lack of labeled data as well as the significant imbalance among the different labels  Unlike neural network models, character-level features in traditional machine learning classifiers have positive results because we have trained the models only with the most significant character elements using TF-IDF values 
Variants of neural network models also suffer from data insufficiency  However, these models show positive performances on “spam" (14%) and “hateful" (4%) tweets—the lower distributed labels  The highest F1 score for “spam" is from the RNN-LTC model (0 551), and the highest for “hateful" is CNN with context tweets (0 309)  Since each variant model excels in different metrics, we expect to see additional improvements with the use of ensemble models of these variants in future works 
In this paper, we report the baseline accuracy of different learning models as well as their variants on the recently introduced dataset, Hate and Abusive Speech on Twitter  Experimental results show that bidirectional GRU networks with LTC provide the most accurate results in detecting abusive language  Additionally, we present the possibility of using ensemble models of variant models and features for further improvements 
Acknowledgments
K  Jung is with the Department of Electrical and Computer Engineering, ASRI, Seoul National University, Seoul, Korea  This work was supported by the National Research Foundation of Korea (NRF) funded by the Korea government (MSIT) (No  2016M3C4A7952632), the Technology Innovation Program (10073144) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea) 
We would also like to thank Yongkeun Hwang and Ji Ho Park for helpful discussions and their valuable insights 

ground_truth: 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)

KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)