prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What cyberbulling topics did they address?
Context 1: Introduction
Cyberbullying has been defined by the National Crime Prevention Council as the use of the Internet, cell phones or other devices to send or post text or images intended to hurt or embarrass another person  Various studies have estimated that between to 10% to 40% of internet users are victims of cyberbullying BIBREF0   Effects of cyberbullying can range from temporary anxiety to suicide BIBREF1   Many high profile incidents have emphasized the prevalence of cyberbullying on social media  Most recently in October 2017, a Swedish model Arvida Byström was cyberbullied to the extent of receiving rape threats after she appeared in an advertisement with hairy legs 
Detection of cyberbullying in social media is a challenging task  Definition of what constitutes cyberbullying is quite subjective  For example, frequent use of swear words might be considered as bullying by the general population  However, for teen oriented social media platforms such as Formspring, this does not necessarily mean bullying (Table TABREF9 )  Across multiple SMPs, cyberbullies attack victims on different topics such as race, religion, and gender  Depending on the topic of cyberbullying, vocabulary and perceived meaning of words vary significantly across SMPs  For example, in our experiments we found that for word `fat', the most similar words as per Twitter dataset are `female' and `woman' (Table TABREF23 )  However, other two datasets do not show such particular bias against women  This platform specific semantic similarity between words is a key aspect of cyberbullying detection across SMPs  Style of communication varies significantly across SMPs  For example, Twitter posts are short and lack anonymity  Whereas posts on Q&A oriented SMPs are long and have option of anonymity (Table TABREF7 )  Fast evolving words and hashtags in social media make it difficult to detect cyberbullying using swear word list based simple filtering approaches  The option of anonymity in certain social networks also makes it harder to identify cyberbullying as profile and history of the bully might not be available 
Past works on cyberbullying detection have at least one of the following three bottlenecks  First (Bottleneck B1), they target only one particular social media platform  How these methods perform across other SMPs is unknown  Second (Bottleneck B2), they address only one topic of cyberbullying such as racism, and sexism  Depending on the topic, vocabulary and nature of cyberbullying changes  These models are not flexible in accommodating changes in the definition of cyberbullying  Third (Bottleneck B3), they rely on carefully handcrafted features such as swear word list and POS tagging  However, these handcrafted features are not robust against variations in writing style  In contrast to existing bottlenecks, this work targets three different types of social networks (Formspring: a Q&A forum, Twitter: microblogging, and Wikipedia: collaborative knowledge repository) for three topics of cyberbullying (personal attack, racism, and sexism) without doing any explicit feature engineering by developing deep learning based models along with transfer learning 
We experimented with diverse traditional machine learning models (logistic regression, support vector machine, random forest, naive Bayes) and deep neural network models (CNN, LSTM, BLSTM, BLSTM with Attention) using variety of representation methods for words (bag of character n-gram, bag of word unigram, GloVe embeddings, SSWE embeddings)  Summary of our findings and research contributions is as follows 
Datasets
Please refer to Table TABREF7 for summary of datasets used  We performed experiments using large, diverse, manually annotated, and publicly available datasets for cyberbullying detection in social media  We cover three different types of social networks: teen oriented Q&A forum (Formspring), large microblogging platform (Twitter), and collaborative knowledge repository (Wikipedia talk pages)  Each dataset addresses a different topic of cyberbullying  Twitter dataset contains examples of racism and sexism  Wikipedia dataset contains examples of personal attack  However, Formspring dataset is not specifically about any single topic  All three datasets have the problem of class imbalance where posts labeled as cyberbullying are in the minority as compared to neutral posts  Variation in the number of posts across datasets also affects vocabulary size that represents the number of distinct words encountered in the dataset  We measure the size of a post in terms of the number of words in the post  For each dataset, there are only a few posts with large size  We truncate such large posts to the size of post ranked at 95 percentile in that dataset  For example, in Wikipedia dataset, the largest post has 2846 words  However, size of post ranked at 95 percentile in that dataset is only 231  Any post larger than size 231 in Wikipedia dataset will be truncated by considering only first 231 words  This truncation affects only a small minority of posts in each dataset  However, it is required for efficiently training various models in our experiments  Details of each dataset are as follows 
Formspring BIBREF2 : It was a question and answer based website where users could openly invite others to ask and answer questions  The dataset includes 12K annotated question and answer pairs  Each post is manually labeled by three workers  Among these pairs, 825 were labeled as containing cyberbullying content by at least two Amazon Mechanical turk workers 
Twitter BIBREF3 : This dataset includes 16K annotated tweets  The authors bootstrapped the corpus collection, by performing an initial manual search of common slurs and terms used pertaining to religious, sexual, gender, and ethnic minorities  Of the 16K tweets, 3117 are labeled as sexist, 1937 as racist, and the remaining are marked as neither sexist nor racist 
Wikipedia BIBREF4 : For each page in Wikipedia, a corresponding talk page maintains the history of discussion among users who participated in its editing  This data set includes over 100k labeled discussion comments from English Wikipedia's talk pages  Each comment was labeled by 10 annotators via Crowdflower on whether it contains a personal attack  There are total 13590 comments labeled as personal attack 
Use of Swear Words and Anonymity
Please refer to Table TABREF9   We use the following short forms in this section: B=Bullying, S=Swearing, A=Anonymous  Some of the values for Twitter dataset are undefined as Twitter does not allow anonymous postings  Use of swear words has been repeatedly linked to cyberbullying  However, preliminary analysis of datasets reveals that depending on swear word usage can neither lead to high precision nor high recall for cyberbullying detection  Swear word list based methods will have low precision as P(B INLINEFORM0 S) is not close to 1  In fact, for teen oriented social network Formspring, 78% of the swearing posts are non-bullying  Swear words based filtering will be irritating to the users in such SMPs where swear words are used casually  Swear word list based methods will also have a low recall as P(S INLINEFORM1 B) is not close to 1  For Twitter dataset, 82% of bullying posts do not use any swear words  Such passive-aggressive cyberbullying will go undetected with swear word list based methods  Anonymity is another clue that is used for detecting cyberbullying as bully might prefer to hide its identity  Anonymity definitely leads to increased use of swear words (P(S INLINEFORM2 A) INLINEFORM3 P(S)) and cyberbullying (P(B INLINEFORM4 A) INLINEFORM5 P(B), and P(B INLINEFORM6 A&S)) INLINEFORM7 P(B))  However, significant fraction of anonymous posts are non-bullying (P(B INLINEFORM8 A) not close to 1) and many of bullying posts are not anonymous (P(A INLINEFORM9 B) not close to 1)  Further, anonymity might not be allowed by many SMPs such as Twitter 
Related Work
Cyberbullying is recognized as a phenomenon at least since 2003 BIBREF5   Use of social media exploded with launching of multiple platforms such as Wikipedia (2001), MySpace (2003), Orkut (2004), Facebook (2004), and Twitter (2005)  By 2006, researchers had pointed that cyberbullying was as serious phenomenon as offline bullying BIBREF6   However, automatic detection of cyberbullying was addressed only since 2009 BIBREF7   As a research topic, cyberbullying detection is a text classification problem  Most of the existing works fit in the following template: get training dataset from single SMP, engineer variety of features with certain style of cyberbullying as the target, apply a few traditional machine learning methods, and evaluate success in terms of measures such as F1 score and accuracy  These works heavily rely on handcrafted features such as use of swear words  These methods tend to have low precision for cyberbullying detection as handcrafted features are not robust against variations in bullying style across SMPs and bullying topics  Only recently, deep learning has been applied for cyberbullying detection BIBREF8   Table TABREF27 summarizes important related work 
Deep Neural Network (DNN) Based Models
We experimented with four DNN based models for cyberbullying detection: CNN, LSTM, BLSTM, and BLSTM with attention  These models are listed in the increasing complexity of their neural architecture and amount of information used by these models  Please refer to Figure 1 for general architecture that we have used across four models  Various models differ only in the Neural Architecture layer while having identical rest of the layers  CNNs are providing state-of-the-results on extracting contextual feature for classification tasks in images, videos, audios, and text  Recently, CNNs were used for sentiment classification BIBREF9   Long Short Term Memory networks are a special kind of RNN, capable of learning long-term dependencies  Their ability to use their internal memory to process arbitrary sequences of inputs has been found to be effective for text classification BIBREF10   Bidirectional LSTMs BIBREF11 further increase the amount of input information available to the network by encoding information in both forward and backward direction  By using two directions, input information from both the past and future of the current time frame can be used  Attention mechanisms allow for a more direct dependence between the state of the model at different points in time  Importantly, attention mechanism lets the model learn what to attend to based on the input sentence and what it has produced so far 
The embedding layer processes a fixed size sequence of words  Each word is represented as a real-valued vector, also known as word embeddings  We have experimented with three methods for initializing word embeddings: random, GloVe BIBREF12 , and SSWE BIBREF13   During the training, model improves upon the initial word embeddings to learn task specific word embeddings  We have observed that these task specific word embeddings capture the SMP specific and topic specific style of cyberbullying  Using GloVe vectors over random vector initialization has been reported to improve performance for some NLP tasks  Most of the word embedding methods such as GloVe, consider only syntactic context of the word while ignoring the sentiment conveyed by the text  SSWE method overcomes this problem by incorporating the text sentiment as one of the parameters for word embedding generation  We experimented with various dimension size for word embeddings  Experimental results reported here are with dimension size as 50  There was no significant variation in results with dimension size ranging from 30 to 200 
To avoid overfitting, we used two dropout layers, one before the neural architecture layer and one after, with dropout rates of 0 25 and 0 5 respectively  Fully connected layer is a dense output layer with the number of neurons equal to the number of classes, followed by softmax layer that provides softmax activation  All our models are trained using backpropagation  The optimizer used for training is Adam and the loss function is categorical cross-entropy  Besides learning the network weights, these methods also learn task-specific word embeddings tuned towards the bullying labels (See Section SECREF21 )  Our code is available at: https://github com/sweta20/Detecting-Cyberbullying-Across-SMPs 
Experiments
Existing works have heavily relied on traditional machine learning models for cyberbullying detection  However, they do not study the performance of these models across multiple SMPs  We experimented with four models: logistic regression (LR), support vector machine (SVM), random forest (RF), and naive Bayes (NB), as these are used in previous works (Table TABREF27 )  We used two data representation methods: character n-gram and word unigram  Past work in the domain of detecting abusive language have showed that simple n-gram features are more powerful than linguistic and syntactic features, hand-engineered lexicons, and word and paragraph embeddings BIBREF14   As compared to DNN models, performance of all four traditional machine learning models was significantly lower  Please refer to Table TABREF11  
All DNN models reported here were implemented using Keras  We pre-process the data, subjecting it to standard operations of removal of stop words, punctuation marks and lowercasing, before annotating it to assigning respective labels to each comment  For each trained model, we report its performance after doing five-fold cross-validation  We use following short forms 
Effect of Oversampling Bullying Instances
The training datasets had a major problem of class imbalance with posts marked as bullying in the minority  As a result, all models were biased towards labeling the posts as non-bullying  To remove this bias, we oversampled the data from bullying class thrice  That is, we replicated bullying posts thrice in the training data  This significantly improved the performance of all DNN models with major leap in all three evaluation measures  Table TABREF17 shows the effect of oversampling for a variety of word embedding methods with BLSTM Attention as the detection model  Results for other models are similar BIBREF15   We can notice that oversampled datasets (F+, T+, W+) have far better performance than their counterparts (F, T, W respectively)  Oversampling particularly helps the smallest dataset Formspring where number of training instances for bullying class is quite small (825) as compared to other two datasets (about 5K and 13K)  We also experimented with varying the replication rate for bullying posts BIBREF15   However, we observed that for bullying posts, replication rate of three is good enough 
Choice of Initial Word Embeddings and Model
Initial word embeddings decide data representation for DNN models  However during the training, DNN models modify these initial word embeddings to learn task specific word embeddings  We have experimented with three methods to initialize word embeddings  Please refer to Table TABREF19   This table shows the effect of varying initial word embeddings for multiple DNN models across datasets  We can notice that initial word embeddings do not have a significant effect on cyberbullying detection when oversampling of bullying posts is done (rows corresponding to F+, T+, W+)  In the absence of oversampling (rows corresponding to F, T W), there is a gap in performance of simplest (CNN) and most complex (BLSTM with attention) models  However, this gap goes on reducing with the increase in the size of datasets 
Table TABREF20 compares the performance of four DNN models for three evaluation measures while using SSWE as the initial word embeddings  We have noticed that most of the time LSTM performs weaker than other three models  However, performance gap in the other three models is not significant 
Task Specific Word Embeddings
DNN models learn word embeddings over the training data  These learned embeddings across multiple datasets show the difference in nature and style of bullying across cyberbullying topics and SMPs  Here we report results for BLSTM with attention model  Results for other models are similar  We first verify that important words for each topic of cyberbullying form clusters in the learned embeddings  To enable the visualization of grouping, we reduced dimensionality with t-SNE BIBREF16 , a well-known technique for dimensionality reduction particularly well suited for visualization of high dimensional datasets  Please refer to Table TABREF22   This table shows important clusters observed in t-SNE projection of learned word embeddings  Each cluster shows that words most relevant to a particular topic of bullying form cluster 
We also observed changes in the meanings of the words across topics of cyberbullying  Table TABREF23 shows most similar words for a given query word for two datasets  Twitter dataset which is heavy on sexism and racism, considers word slave as similar to targets of racism and sexism  However, Wikipedia dataset that is about personal attacks does not show such bias 
Transfer Learning
We used transfer learning to check if the knowledge gained by DNN models on one dataset can be used to improve cyberbullying detection performance on other datasets  We report results where BLSTM with attention is used as the DNN model  Results for other models are similar BIBREF15   We experimented with following three flavors of transfer learning 
Complete Transfer Learning (TL1): In this flavor, a model trained on one dataset was directly used to detect cyberbullying in other datasets without any extra training  TL1 resulted in significantly low recall indicating that three datasets have different nature of cyberbullying with low overlap (Table TABREF25 )  However precision was relatively higher for TL1, indicating that DNN models are cautious in labeling a post as bully (Table TABREF25 )  TL1 also helps to measure similarity in nature of cyberbullying across three datasets  We can observe that bullying nature in Formspring and Wikipedia datasets is more similar to each other than the Twitter dataset  This can be inferred from the fact that with TL1, cyberbullying detection performance for Formspring dataset is higher when base model is Wikipedia (precision =0 51 and recall=0 66)as compared to Twitter as the base model (precision=0 38 and recall=0 04)  Similarly, for Wikipedia dataset, Formspring acts as a better base model than Twitter while using TL1 flavor of transfer learning  Nature of SMP might be a factor behind this similarity in nature of cyberbullying  Both Formspring and Wikipedia are task oriented social networks (Q&A and collaborative knowledge repository respectively) that allow anonymity and larger posts  Whereas communication on Twitter is short, free of anonymity and not oriented towards a particular task 
Feature Level Transfer Learning (TL2): In this flavor, a model was trained on one dataset and only learned word embeddings were transferred to another dataset for training a new model  As compared to TL1, recall score improved dramatically with TL2 (Table TABREF25 )  Improvement in precision was also significant (Table TABREF25 )  These improvements indicate that learned word embeddings are an essential part of knowledge transfer across datasets for cyberbullying detection 
Model Level Transfer Learning (TL3): In this flavor, a model was trained on one dataset and learned word embeddings, as well as network weights, were transferred to another dataset for training a new model  TL3 does not result in any significant improvement over TL2  This lack of improvement indicates that transfer of network weights is not essential for cyberbullying detection and learned word embeddings is the key knowledge gained by the DNN models 
DNN based models coupled with transfer learning beat the best-known results for all three datasets  Previous best F1 scores for Wikipedia BIBREF4 and Twitter BIBREF8 datasets were 0 68 and 0 93 respectively  We achieve F1 scores of 0 94 for both these datasets using BLSTM with attention and feature level transfer learning (Table TABREF25 )  For Formspring dataset, authors have not reported F1 score  Their method has accuracy score of 78 5% BIBREF2   We achieve F1 score of 0 95 with accuracy score of 98% for the same dataset 
Conclusion and Future Work
We have shown that DNN models can be used for cyberbullying detection on various topics across multiple SMPs using three datasets and four DNN models  These models coupled with transfer learning beat state of the art results for all three datasets  These models can be further improved with extra data such as information about the profile and social graph of users  Most of the current datasets do not provide any information about the severity of bullying  If such fine-grained information is made available, then cyberbullying detection models can be further improved to take a variety of actions depending on the perceived seriousness of the posts 

ground_truth: personal attack, racism, and sexism

racism, sexism, personal attack, not specifically about any single topic

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How large is their data set?
Context 2: Introduction
The task of document quality assessment is to automatically assess a document according to some predefined inventory of quality labels  This can take many forms, including essay scoring (quality = language quality, coherence, and relevance to a topic), job application filtering (quality = suitability for role + visual/presentational quality of the application), or answer selection in community question answering (quality = actionability + relevance of the answer to the question)  In the case of this paper, we focus on document quality assessment in two contexts: Wikipedia document quality classification, and whether a paper submitted to a conference was accepted or not 
Automatic quality assessment has obvious benefits in terms of time savings and tractability in contexts where the volume of documents is large  In the case of dynamic documents (possibly with multiple authors), such as in the case of Wikipedia, it is particularly pertinent, as any edit potentially has implications for the quality label of that document (and around 10 English Wikipedia documents are edited per second)  Furthermore, when the quality assessment task is decentralized (as in the case of Wikipedia and academic paper assessment), quality criteria are often applied inconsistently by different people, where an automatic document quality assessment system could potentially reduce inconsistencies and enable immediate author feedback 
Current studies on document quality assessment mainly focus on textual features  For example, BIBREF0 examine features such as the article length and the number of headings to predict the quality class of a Wikipedia article  In contrast to these studies, in this paper, we propose to combine text features with visual features, based on a visual rendering of the document  Figure 1 illustrates our intuition, relative to Wikipedia articles  Without being able to read the text, we can tell that the article in Figure 1 has higher quality than Figure 1 , as it has a detailed infobox, extensive references, and a variety of images  Based on this intuition, we aim to answer the following question: can we achieve better accuracy on document quality assessment by complementing textual features with visual features?
Our visual model is based on fine-tuning an Inception V3 model BIBREF1 over visual renderings of documents, while our textual model is based on a hierarchical biLSTM  We further combine the two into a joint model  We perform experiments on two datasets: a Wikipedia dataset novel to this paper, and an arXiv dataset provided by BIBREF2 split into three sub-parts based on subject category  Experimental results on the visual renderings of documents show that implicit quality indicators, such as images and visual layout, can be captured by an image classifier, at a level comparable to a text classifier  When we combine the two models, we achieve state-of-the-art results over 3/4 of our datasets 
This paper makes the following contributions:
All code and data associated with this research will be released on publication 
Related Work
A variety of approaches have been proposed for document quality assessment across different domains: Wikipedia article quality assessment, academic paper rating, content quality assessment in community question answering (cQA), and essay scoring  Among these approaches, some use hand-crafted features while others use neural networks to learn features from documents  For each domain, we first briefly describe feature-based approaches and then review neural network-based approaches  Wikipedia article quality assessment: Quality assessment of Wikipedia articles is a task that assigns a quality class label to a given Wikipedia article, mirroring the quality assessment process that the Wikipedia community carries out manually  Many approaches have been proposed that use features from the article itself, meta-data features (e g , the editors, and Wikipedia article revision history), or a combination of the two  Article-internal features capture information such as whether an article is properly organized, with supporting evidence, and with appropriate terminology  For example, BIBREF3 use writing styles represented by binarized character trigram features to identify featured articles  BIBREF4 and BIBREF0 explore the number of headings, images, and references in the article  BIBREF5 use nine readability scores, such as the percentage of difficult words in the document, to measure the quality of the article  Meta-data features, which are indirect indicators of article quality, are usually extracted from revision history, and the interaction between editors and articles  For example, one heuristic that has been proposed is that higher-quality articles have more edits BIBREF6 , BIBREF7   BIBREF8 use the percentage of registered editors and the total number of editors of an article  Article–editor dependencies have also been explored  For example, BIBREF9 use the authority of editors to measure the quality of Wikipedia articles, where the authority of editors is determined by the articles they edit  Deep learning approaches to predicting Wikipedia article quality have also been proposed  For example, BIBREF10 use a version of doc2vec BIBREF11 to represent articles, and feed the document embeddings into a four hidden layer neural network  BIBREF12 first obtain sentence representations by averaging words within a sentence, and then apply a biLSTM BIBREF13 to learn a document-level representation, which is combined with hand-crafted features as side information  BIBREF14 exploit two stacked biLSTMs to learn document representations 
Academic paper rating: Academic paper rating is a relatively new task in NLP/AI, with the basic formulation being to automatically predict whether to accept or reject a paper  BIBREF2 explore hand-crafted features, such as the length of the title, whether specific words (such as outperform, state-of-the-art, and novel) appear in the abstract, and an embedded representation of the abstract as input to different downstream learners, such as logistic regression, decision tree, and random forest  BIBREF15 exploit a modularized hierarchical convolutional neural network (CNN), where each paper section is treated as a module  For each paper section, they train an attention-based CNN, and an attentive pooling layer is applied to the concatenated representation of each section, which is then fed into a softmax layer 
Content quality assessment in cQA: Automatic quality assessment in cQA is the task of determining whether an answer is of high quality, selected as the best answer, or ranked higher than other answers  To measure answer content quality in cQA, researchers have exploited various features from different sources, such as the answer content itself, the answerer's profile, interactions among users, and usage of the content  The most common feature used is the answer length BIBREF16 , BIBREF17 , with other features including: syntactic and semantic features, such as readability scores  BIBREF18 ; similarity between the question and the answer at lexical, syntactic, and semantic levels BIBREF18 , BIBREF19 , BIBREF20 ; or user data (e g , a user's status points or the number of answers written by the user)  There have also been approaches using neural networks  For example, BIBREF21 combine CNN-learned representations with hand-crafted features to predict answer quality  BIBREF22 use a 2-dimensional CNN to learn the semantic relevance of an answer to the question, and apply an LSTM to the answer sequence to model thread context  BIBREF23 and BIBREF24 model the problem similarly to machine translation quality estimation, treating answers as competing translation hypotheses and the question as the reference translation, and apply neural machine translation to the problem  Essay scoring: Automated essay scoring is the task of assigning a score to an essay, usually in the context of assessing the language ability of a language learner  The quality of an essay is affected by the following four primary dimensions: topic relevance, organization and coherence, word usage and sentence complexity, and grammar and mechanics  To measure whether an essay is relevant to its “prompt” (the description of the essay topic), lexical and semantic overlap is commonly used BIBREF25 , BIBREF26   BIBREF27 explore word features, such as the number of verb formation errors, average word frequency, and average word length, to measure word usage and lexical complexity  BIBREF28 use sentence structure features to measure sentence variety  The effects of grammatical and mechanic errors on the quality of an essay are measured via word and part-of-speech $n$ -gram features and “mechanics” features BIBREF29 (e g , spelling, capitalization, and punctuation), respectively  BIBREF30 , BIBREF31 , and BIBREF32 use an LSTM to obtain an essay representation, which is used as the basis for classification  Similarly, BIBREF33 utilize a CNN to obtain sentence representation and an LSTM to obtain essay representation, with an attention layer at both the sentence and essay levels 
The Proposed Joint Model
We treat document quality assessment as a classification problem, i e , given a document, we predict its quality class (e g , whether an academic paper should be accepted or rejected)  The proposed model is a joint model that integrates visual features learned through Inception V3 with textual features learned through a biLSTM  In this section, we present the details of the visual and textual embeddings, and finally describe how we combine the two  We return to discuss hyper-parameter settings and the experimental configuration in the Experiments section 
Visual Embedding Learning
A wide range of models have been proposed to tackle the image classification task, such as VGG BIBREF34 , ResNet BIBREF35 , Inception V3 BIBREF1 , and Xception BIBREF36   However, to the best of our knowledge, there is no existing work that has proposed to use visual renderings of documents to assess document quality  In this paper, we use Inception V3 pretrained on ImageNet (“Inception” hereafter) to obtain visual embeddings of documents, noting that any image classifier could be applied to our task  The input to Inception is a visual rendering (screenshot) of a document, and the output is a visual embedding, which we will later integrate with our textual embedding 
Based on the observation that it is difficult to decide what types of convolution to apply to each layer (such as 3 $\times $ 3 or 5 $\times $ 5), the basic Inception model applies multiple convolution filters in parallel and concatenates the resulting features, which are fed into the next layer  This has the benefit of capturing both local features through smaller convolutions and abstracted features through larger convolutions  Inception is a hybrid of multiple Inception models of different architectures  To reduce computational cost, Inception also modifies the basic model by applying a 1 $\times $ 1 convolution to the input and factorizing larger convolutions into smaller ones 
Textual Embedding Learning
We adopt a bi-directional LSTM model to generate textual embeddings for document quality assessment, following the method of BIBREF12 (“biLSTM” hereafter)  The input to biLSTM is a textual document, and the output is a textual embedding, which will later integrate with the visual embedding 
For biLSTM, each word is represented as a word embedding BIBREF37 , and an average-pooling layer is applied to the word embeddings to obtain the sentence embedding, which is fed into a bi-directional LSTM to generate the document embedding from the sentence embeddings  Then a max-pooling layer is applied to select the most salient features from the component sentences 
The Joint Model
The proposed joint model (“Joint” hereafter) combines the visual and textual embeddings (output of Inception and biLSTM) via a simple feed-forward layer and softmax over the document label set, as shown in Figure 2   We optimize our model based on cross-entropy loss 
Experiments
In this section, we first describe the two datasets used in our experiments: (1) Wikipedia, and (2) arXiv  Then, we report the experimental details and results 
Datasets
The Wikipedia dataset consists of articles from English Wikipedia, with quality class labels assigned by the Wikipedia community  Wikipedia articles are labelled with one of six quality classes, in descending order of quality: Featured Article (“FA”), Good Article (“GA”), B-class Article (“B”), C-class Article (“C”), Start Article (“Start”), and Stub Article (“Stub”)  A description of the criteria associated with the different classes can be found in the Wikipedia grading scheme page  The quality class of a Wikipedia article is assigned by Wikipedia reviewers or any registered user, who can discuss through the article's talk page to reach consensus  We constructed the dataset by first crawling all articles from each quality class repository, e g , we get FA articles by crawling pages from the FA repository: https://en wikipedia org/wiki/Category:Featured_articles  This resulted in around 5K FA, 28K GA, 212K B, 533K C, 2 6M Start, and 3 2M Stub articles 
We randomly sampled 5,000 articles from each quality class and removed all redirect pages, resulting in a dataset of 29,794 articles  As the wikitext contained in each document contains markup relating to the document category such as {Featured Article} or {geo-stub}, which reveals the label, we remove such information  We additionally randomly partitioned this dataset into training, development, and test splits based on a ratio of 8:1:1  Details of the dataset are summarized in Table 1  
We generate a visual representation of each document via a 1,000 $\times $ 2,000-pixel screenshot of the article via a PhantomJS script over the rendered version of the article, ensuring that the screenshot and wikitext versions of the article are the same version  Any direct indicators of document quality (such as the FA indicator, which is a bronze star icon in the top right corner of the webpage) are removed from the screenshot 
The arXiv dataset BIBREF2 consists of three subsets of academic articles under the arXiv repository of Computer Science (cs), from the three subject areas of: Artificial Intelligence (cs ai), Computation and Language (cs cl), and Machine Learning (cs lg)  In line with the original dataset formulation BIBREF2 , a paper is considered to have been accepted (i e  is positively labeled) if it matches a paper in the DBLP database or is otherwise accepted by any of the following conferences: ACL, EMNLP, NAACL, EACL, TACL, NIPS, ICML, ICLR, or AAAI  Failing this, it is considered to be rejected (noting that some of the papers may not have been submitted to one of these conferences)  The median numbers of pages for papers in cs ai, cs cl, and cs lg are 11, 10, and 12, respectively  To make sure each page in the PDF file has the same size in the screenshot, we crop the PDF file of a paper to the first 12; we pad the PDF file with blank pages if a PDF file has less than 12 pages, using the PyPDF2 Python package  We then use ImageMagick to convert the 12-page PDF file to a single 1,000 $\times $ 2,000 pixel screenshot  Table 2 details this dataset, where the “Accepted” column denotes the percentage of positive instances (accepted papers) in each subset 
Experimental Setting
As discussed above, our model has two main components — biLSTM and Inception— which generate textual and visual representations, respectively  For the biLSTM component, the documents are preprocessed as described in BIBREF12 , where an article is divided into sentences and tokenized using NLTK BIBREF38   Words appearing more than 20 times are retained when building the vocabulary  All other words are replaced by the special UNK token  We use the pre-trained GloVe BIBREF39 50-dimensional word embeddings to represent words  For words not in GloVe, word embeddings are randomly initialized based on sampling from a uniform distribution $U(-1, 1)$   All word embeddings are updated in the training process  We set the LSTM hidden layer size to 256  The concatenation of the forward and backward LSTMs thus gives us 512 dimensions for the document embedding  A dropout layer is applied at the sentence and document level, respectively, with a probability of 0 5 
For Inception, we adopt data augmentation techniques in the training with a “nearest” filling mode, a zoom range of 0 1, a width shift range of 0 1, and a height shift range of 0 1  As the original screenshots have the size of 1,000 $\times 2$ ,000 pixels, they are resized to 500 $\times $ 500 to feed into Inception, where the input shape is (500, 500, 3)  A dropout layer is applied with a probability of 0 5  Then, a GlobalAveragePooling2D layer is applied, which produces a 2,048 dimensional representation 
For the Joint model, we get a representation of 2,560 dimensions by concatenating the 512 dimensional representation from the biLSTM with the 2,048 dimensional representation from Inception  The dropout layer is applied to the two components with a probability of 0 5  For biLSTM, we use a mini-batch size of 128 and a learning rate of 0 001  For both Inception and joint model, we use a mini-batch size of 16 and a learning rate of 0 0001  All hyper-parameters were set empirically over the development data, and the models were optimized using the Adam optimizer BIBREF40  
In the training phase, the weights in Inception are initialized by parameters pretrained on ImageNet, and the weights in biLSTM are randomly initialized (except for the word embeddings)  We train each model for 50 epochs  However, to prevent overfitting, we adopt early stopping, where we stop training the model if the performance on the development set does not improve for 20 epochs  For evaluation, we use (micro-)accuracy, following previous studies BIBREF5 , BIBREF2  
Baseline Approaches
We compare our models against the following five baselines:
Majority: the model labels all test samples with the majority class of the training data 
Benchmark: a benchmark method from the literature  In the case of Wikipedia, this is BIBREF5 , who use structural features and readability scores as features to build a random forest classifier; for arXiv, this is BIBREF2 , who use hand-crafted features, such as the number of references and TF-IDF weighted bag-of-words in abstract, to build a classifier based on the best of logistic regression, multi-layer perception, and AdaBoost 
Doc2Vec: doc2vec BIBREF11 to learn document embeddings with a dimension of 500, and a 4-layer feed-forward classification model on top of this, with 2000, 1000, 500, and 200 dimensions, respectively 
biLSTM: first derive a sentence representation by averaging across words in a sentence, then feed the sentence representation into a biLSTM and a maxpooling layer over output sequence to learn a document level representation with a dimension of 512, which is used to predict document quality 
Inception $_{\text{fixed}}$ : the frozen Inception model, where only parameters in the last layer are fine-tuned during training 
The hyper-parameters of Benchmark, Doc2Vec, and biLSTM are based on the corresponding papers except that: (1) we fine-tune the feed forward layer of Doc2Vec on the development set and train the model 300 epochs on Wikipedia and 50 epochs on arXiv; (2) we do not use hand-crafted features for biLSTM as we want the baselines to be comparable to our models, and the main focus of this paper is not to explore the effects of hand-crafted features (e g , see BIBREF12 ) 
Experimental Results
Table 3 shows the performance of the different models over our two datasets, in the form of the average accuracy on the test set (along with the standard deviation) over 10 runs, with different random initializations 
On Wikipedia, we observe that the performance of biLSTM, Inception, and Joint is much better than that of all four baselines  Inception achieves 2 9% higher accuracy than biLSTM  The performance of Joint achieves an accuracy of 59 4%, which is 5 3% higher than using textual features alone (biLSTM) and 2 4% higher than using visual features alone (Inception)  Based on a one-tailed Wilcoxon signed-rank test, the performance of Joint is statistically significant ( $p<0 05$ )  This shows that the textual and visual features complement each other, achieving state-of-the-art results in combination 
For arXiv, baseline methods Majority, Benchmark, and Inception $_{\text{fixed}}$ outperform biLSTM over cs ai, in large part because of the class imbalance in this dataset (90% of papers are rejected)  Surprisingly, Inception $_{\text{fixed}}$ is better than Majority and Benchmark over the arXiv cs lg subset, which verifies the usefulness of visual features, even when only the last layer is fine-tuned  Table 3 also shows that Inception and biLSTM achieve similar performance on arXiv, showing that textual and visual representations are equally discriminative: Inception and biLSTM are indistinguishable over cs cl; biLSTM achieves 1 8% higher accuracy over cs lg, while Inception achieves 1 3% higher accuracy over cs ai  Once again, the Joint model achieves the highest accuracy on cs ai and cs cl by combining textual and visual representations (at a level of statistical significance for cs ai)  This, again, confirms that textual and visual features complement each other, and together they achieve state-of-the-art results  On arXiv cs lg, Joint achieves a 0 6% higher accuracy than Inception by combining visual features and textual features, but biLSTM achieves the highest accuracy  One characteristic of cs lg documents is that they tend to contain more equations than the other two arXiv datasets, and preliminary analysis suggests that the biLSTM is picking up on a correlation between the volume/style of mathematical presentation and the quality of the document 
Analysis
In this section, we first analyze the performance of Inception and Joint  We also analyze the performance of different models on different quality classes  The high-level representations learned by different models are also visualized and discussed  As the Wikipedia test set is larger and more balanced than that of arXiv, our analysis will focus on Wikipedia 
Inception
To better understand the performance of Inception, we generated the gradient-based class activation map BIBREF41 , by maximizing the outputs of each class in the penultimate layer, as shown in Figure 3   From Figure 3 and Figure 3 , we can see that Inception identifies the two most important regions (one at the top corresponding to the table of contents, and the other at the bottom, capturing both document length and references) that contribute to the FA class prediction, and a region in the upper half of the image that contributes to the GA class prediction (capturing the length of the article body)  From Figure 3 and Figure 3 , we can see that the most important regions in terms of B and C class prediction capture images (down the left and right of the page, in the case of B and C), and document length/references  From Figure 3 and Figure 3 , we can see that Inception finds that images in the top right corner are the strongest predictor of Start class prediction, and (the lack of) images/the link bar down the left side of the document are the most important for Stub class prediction 
Joint
Table 4 shows the confusion matrix of Joint on Wikipedia  We can see that more than 50% of documents for each quality class are correctly classified, except for the C class where more documents are misclassified into B  Analysis shows that when misclassified, documents are usually misclassified into adjacent quality classes, which can be explained by the Wikipedia grading scheme, where the criteria for adjacent quality classes are more similar 
We also provide a breakdown of precision (“ $\mathcal {P}$ ”), recall (“ $\mathcal {R}$ ”), and F1 score (“ $\mathcal {F}_{\beta =1}$ ”) for biLSTM, Inception, and Joint across the quality classes in Table 5   We can see that Joint achieves the highest accuracy in 11 out of 18 cases  It is also worth noting that all models achieve higher scores for FA, GA, and Stub articles than B, C and Start articles  This can be explained in part by the fact that FA and GA articles must pass an official review based on structured criteria, and in part by the fact that Stub articles are usually very short, which is discriminative for Inception, and Joint  All models perform worst on the B and C quality classes  It is difficult to differentiate B articles from C articles even for Wikipedia contributors  As evidence of this, when we crawled a new dataset including talk pages with quality class votes from Wikipedia contributors, we found that among articles with three or more quality labels, over 20% percent of B and C articles have inconsistent votes from Wikipedia contributors, whereas for FA and GA articles the number is only 0 7% 
We further visualize the learned document representations of biLSTM, Inception, and Joint in the form of a t-SNE plot BIBREF42 in Figure 4   The degree of separation between Start and Stub achieved by Inception is much greater than for biLSTM, with the separation between Start and Stub achieved by Joint being the clearest among the three models  Inception and Joint are better than biLSTM at separating Start and C  Joint achieves slightly better performance than Inception in separating GA and FA  We can also see that it is difficult for all models to separate B and C, which is consistent with the findings of Tables 4 and 5  
Conclusions
We proposed to use visual renderings of documents to capture implicit document quality indicators, such as font choices, images, and visual layout, which are not captured in textual content  We applied neural network models to capture visual features given visual renderings of documents  Experimental results show that we achieve a 2 9% higher accuracy than state-of-the-art approaches based on textual features over Wikipedia, and performance competitive with or surpassing state-of-the-art approaches over arXiv  We further proposed a joint model, combining textual and visual representations, to predict the quality of a document  Experimental results show that our joint model outperforms the visual-only model in all cases, and the text-only model on Wikipedia and two subsets of arXiv  These results underline the feasibility of assessing document quality via visual features, and the complementarity of visual and textual document representations for quality assessment 
Context 3: INTRODUCTION
The idea of language identification is to classify a given audio signal into a particular class using a classification algorithm  Commonly language identification task was done using i-vector systems [1]  A very well known approach for language identification proposed by N  Dahek et al  [1] uses the GMM-UBM model to obtain utterance level features called i-vectors  Recent advances in deep learning [15,16] have helped to improve the language identification task using many different neural network architectures which can be trained efficiently using GPUs for large scale datasets  These neural networks can be configured in various ways to obtain better accuracy for language identification task  Early work on using Deep learning for language Identification was published by Pavel Matejka et al  [2], where they used stacked bottleneck features extracted from deep neural networks for language identification task and showed that the bottleneck features learned by Deep neural networks are better than simple MFCC or PLP features  Later the work by I  Lopez-Moreno et al  [3] from Google showed how to use Deep neural networks to directly map the sequence of MFCC frames into its language class so that we can apply language identification at the frame level  Speech signals will have both spatial and temporal information, but simple DNNs are not able to capture temporal information  Work done by J  Gonzalez-Dominguez et al  [4] by Google developed an LSTM based language identification model which improves the accuracy over the DNN based models  Work done by Alicia et al  [5] used CNNs to improve upon i-vector [1] and other previously developed systems  The work done by Daniel Garcia-Romero et al  [6] has used a combination of Acoustic model trained for speech recognition with Time-delay neural networks where they train the TDNN model by feeding the stacked bottleneck features from acoustic model to predict the language labels at the frame level  Recently X-vectors [7] is proposed for speaker identification task and are shown to outperform all the previous state of the art speaker identification algorithms and are also used for language identification by David Snyder et al  [8] 
In this paper, we explore multiple pooling strategies for language identification task  Mainly we propose Ghost-VLAD based pooling method for language identification  Inspired by the recent work by W  Xie et al  [9] and Y  Zhong et al  [10], we use Ghost-VLAD to improve the accuracy of language identification task for Indian languages  We explore multiple pooling strategies including NetVLAD pooling [11], Average pooling and Statistics pooling( as proposed in X-vectors [7]) and show that Ghost-VLAD pooling is the best pooling strategy for language identification  Our model obtains the best accuracy of 98 24%, and it outperforms all the other previously proposed pooling methods  We conduct all our experiments on 635hrs of audio data for 7 Indian languages collected from $\textbf {All India Radio}$ news channel  The paper is organized as follows  In section 2, we explain the proposed pooling method for language identification  In section 3, we explain our dataset  In section 4, we describe the experiments, and in section 5, we describe the results 
POOLING STRATEGIES
In any language identification model, we want to obtain utterance level representation which has very good language discriminative features  These representations should be compact and should be easily separable by a linear classifier  The idea of any pooling strategy is to pool the frame-level representations into a single utterance level representation  Previous works by [7] have used simple mean and standard deviation aggregation to pool the frame-level features from the top layer of the neural network to obtain the utterance level features  Recently [9] used VLAD based pooling strategy for speaker identification which is inspired from [10] proposed for face recognition  The NetVLAD [11] and Ghost-VLAD [10] methods are proposed for Place recognition and face recognition, respectively, and in both cases, they try to aggregate the local descriptors into global features  In our case, the local descriptors are features extracted from ResNet [15], and the global utterance level feature is obtained by using GhostVLAD pooling  In this section, we explain different pooling methods, including NetVLAD, Ghost-VLAD, Statistic pooling, and Average pooling 
POOLING STRATEGIES ::: NetVLAD pooling
The NetVLAD pooling strategy was initially developed for place recognition by R  Arandjelovic et al  [11]  The NetVLAD is an extension to VLAD [18] approach where they were able to replace the hard assignment based clustering with soft assignment based clustering so that it can be trained with neural network in an end to end fashion  In our case, we use the NetVLAD layer to map N local features of dimension D into a fixed dimensional vector, as shown in Figure 1 (Left side) 
The model takes spectrogram as an input and feeds into CNN based ResNet architecture  The ResNet is used to map the spectrogram into 3D feature map of dimension HxWxD  We convert this 3D feature map into 2D by unfolding H and W dimensions, creating a NxD dimensional feature map, where N=HxW  The NetVLAD layer is kept on top of the feature extraction layer of ResNet, as shown in Figure 1  The NetVLAD now takes N features vectors of dimension D and computes a matrix V of dimension KxD, where K is the number clusters in the NetVLAD layer, and D is the dimension of the feature vector  The matrix V is computed as follows 
Where $w_k$,$b_k$ and $c_k$ are trainable parameters for the cluster $k$ and V(j,k) represents a point in the V matrix for (j,k)th location  The matrix is constructed using the equation (1) where the first term corresponds to the soft assignment of the input $x_i$ to the cluster $c_k$, whereas the second term corresponds to the residual term which tells how far the input descriptor $x_i$ is from the cluster center $c_k$ 
POOLING STRATEGIES ::: GhostVLAD pooling
GhostVLAD is an extension of the NetVLAD approach, which we discussed in the previous section  The GhostVLAD model was proposed for face recognition by Y  Zhong [10]  GhostVLAD works exactly similar to NetVLAD except it adds Ghost clusters along with the NetVLAD clusters  So, now we will have a K+G number of clusters instead of K clusters  Where G is the number of ghost clusters, we want to add (typically 2-4)  The Ghost clusters are added to map any noisy or irrelevant content into ghost clusters and are not included during the feature aggregation stage, as shown in Figure 1 (Right side)  Which means that we compute the matrix V for both normal cluster K and ghost clusters G, but we will not include the vectors belongs to ghost cluster from V during concatenation of the features  Due to which, during feature aggregation stage the contribution of the noisy and unwanted features to normal VLAD clusters are assigned less weights while Ghost clusters absorb most of the weight  We illustrate this in Figure 1(Right Side), where the ghost clusters are shown in red color  We use Ghost clusters when we are computing the V matrix, but they are excluded during the concatenation stage

ground_truth: a sample of  29,794 wikipedia articles and 2,794 arXiv papers 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the GhostVLAD approach?
Context 3:  These concatenated features are fed into the projection layer, followed by softmax to predict the language label 
POOLING STRATEGIES ::: Statistic and average pooling
In statistic pooling, we compute the first and second order statistics of the local features from the top layer of the ResNet model  The 3-D feature map is unfolded to create N features of D dimensions, and then we compute the mean and standard deviation of all these N vectors and get two D dimensional vectors, one for mean and the other for standard deviation  We then concatenate these 2 features and feed it to the projection layer for predicting the language label 
In the Average pooling layer, we compute only the first-order statistics (mean) of the local features from the top layer of the CNN model  The feature map from the top layer of CNN is unfolded to create N features of D dimensions, and then we compute the mean of all these N vectors and get D dimensional representation  We then feed this feature to the projection layer followed by softmax for predicting the language label 
DATASET
In this section, we describe our dataset collection process  We collected and curated around 635Hrs of audio data for 7 Indian languages, namely Kannada, Hindi, Telugu, Malayalam, Bengali, and English  We collected the data from the All India Radio news channel where an actor will be reading news for about 5-10 mins  To cover many speakers for the dataset, we crawled data from 2010 to 2019  Since the audio is very long to train any deep neural network directly, we segment the audio clips into smaller chunks using Voice activity detector  Since the audio clips will have music embedded during the news, we use Inhouse music detection model to remove the music segments from the dataset to make the dataset clean and our dataset contains 635Hrs of clean audio which is divided into 520Hrs of training data containing 165K utterances and 115Hrs of testing data containing 35K utterances  The amount of audio data for training and testing for each of the language is shown in the table bellow 
EXPERIMENTS
In this section, we describe the feature extraction process and network architecture in detail  We use spectral features of 256 dimensions computed using 512 point FFT for every frame, and we add an energy feature for every frame giving us total 257 features for every frame  We use a window size of 25ms and frame shift of 10ms during feature computation  We crop random 5sec audio data from each utterance during training which results in a spectrogram of size 257x500 (features x number of features)  We use these spectrograms as input to our CNN model during training  During testing, we compute the prediction score irrespective of the audio length 
For the network architecture, we use ResNet-34 architecture, as described in [9]  The model uses convolution layers with Relu activations to map the spectrogram of size 257x500 input into 3D feature map of size 1x32x512  This feature cube is converted into 2D feature map of dimension 32x512 and fed into Ghost-VLAD/NetVLAD layer to generate a representation that has more language discrimination capacity  We use Adam optimizer with an initial learning rate of 0 01 and a final learning rate of 0 00001 for training  Each model is trained for 15 epochs with early stopping criteria 
For the baseline, we train an i-vector model using GMM-UBM  We fit a small classifier on top of the generated i-vectors to measure the accuracy  This model is referred as i-vector+svm   To compare our model with the previous state of the art system, we set up the x-vector language identification system [8]  The x-vector model used time-delay neural networks (TDNN) along with statistic-pooling  We use 7 layer TDNN architecture similar to [8] for training  We refer to this model as tdnn+stat-pool   Finally, we set up a Deep LSTM based language identification system similar to [4] but with little modification where we add statistics pooling for the last layers hidden activities before classification  We use 3 layer Bi-LSTM with 256 hidden units at each layer  We refer to this model as LSTM+stat-pool  We train our i-vector+svm and TDNN+stat-pool using Kaldi toolkit  We train our NetVLAD and GhostVLAD experiments using Keras by modifying the code given by [9] for language identification  We train the LSTM+stat-pool and the remaining experiments using Pytorch [14] toolkit, and we will opensource all the codes and data soon 
RESULTS
In this section, we compare the performance of our system with the recent state of the art language identification approaches  We also compare different pooling strategies and finally, compare the robustness of our system to the length of the input spectrogram during training  We visualize the embeddings generated by the GhostVLAD method and conclude that the GhostVLAD embeddings shows very good feature discrimination capabilities 
RESULTS ::: Comparison with different approaches
We compare our system performance with the previous state of the art language identification approaches, as shown in Table 2  The i-vector+svm system is trained using GMM-UBM models to generate i-vectors as proposed in [1]  Once the i-vectors are extracted, we fit SVM classifier to classify the audio  The TDNN+stat-pool system is trained with a statistics pooling layer and is called the x-vector system as proposed by David Snyder et al  [11] and is currently the state of the art language identification approach as far as our knowledge  Our methods outperform the state of the art x-vector system by absolute 1 88% improvement in F1-score, as shown in Table 2 
RESULTS ::: Comparison with different pooling techniques
We compare our approach with different pooling strategies in Table 3  We use ResNet as our base feature extraction network  We keep the base network the same and change only the pooling layers to see which pooling approach performs better for language identification task  Our experiments show that GhostVLAD pooling outperforms all the other pooling methods by achieving 98 43% F1-Score 
RESULTS ::: Duration analysis
To observe the performance of our method with different input durations, we conducted an experiment where we train our model on different input durations  Since our model uses ResNet as the base feature extractor, we need to feed fixed-length spectrogram  We conducted 4 different experiments where we trained the model using 2sec, 3sec, 4sec and 5sec spectrograms containing 200,300,400 and 500 frames respectively  We observed that the model trained with a 5sec spectrogram is the best model, as shown in Table 4 
RESULTS ::: Visualization of embeddings
We visualize the embeddings generated by our approach to see the effectiveness  We extracted 512-dimensional embeddings for our testing data and reduced the dimensionality using t-sne projection  The t-sne plot of the embeddings space is shown in Figure 3  The plot shows that the embeddings learned by our approach has very good discriminative properties
Conclusion
In this work, we use Ghost-VLAD pooling approach that was originally proposed for face recognition to improve language identification performance for Indian languages  We collected and curated 630 hrs audio data from news All India Radio news channel for 7 Indian languages  Our experimental results shows that our approach outperforms the previous state of the art methods by an absolute 1 88% F1-score  We have also conducted experiments with different pooling strategies proposed in the past, and the GhostVLAD pooling approach turns out to be the best approach for aggregating frame-level features into a single utterance level feature  Our experiments also prove that our approach works much better even if the input during training contains smaller durations  Finally, we see that the embeddings generated by our method has very good language discriminative features and helps to improve the performance of language identification 

ground_truth: extension of the NetVLAD, adds Ghost clusters along with the NetVLAD clusters

An extension of NetVLAD which replaces hard assignment-based clustering with soft assignment-based clustering with the additon o fusing Ghost clusters to deal with noisy content.