prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what dataset is used in this paper?
Context 1: Introduction
Users of photo-sharing websites such as Flickr often provide short textual descriptions in the form of tags to help others find the images  With the availability of GPS systems in current electronic devices such as smartphones, latitude and longitude coordinates are nowadays commonly made available as well  The tags associated with such georeferenced photos often describe the location where these photos were taken, and Flickr can thus be regarded as a source of environmental information  The use of Flickr for modelling urban environments has already received considerable attention  For instance, various approaches have been proposed for modelling urban regions BIBREF0 , and for identifying points-of-interest BIBREF1 and itineraries BIBREF2 , BIBREF3   However, the usefulness of Flickr for characterizing the natural environment, which is the focus of this paper, is less well-understood 
Many recent studies have highlighted that Flickr tags capture valuable ecological information, which can be used as a complementary source to more traditional sources  To date, however, ecologists have mostly used social media to conduct manual evaluations of image content with little automated exploitation of the associated tags BIBREF4 , BIBREF5 , BIBREF6   One recent exception is BIBREF7 , where bag-of-words representations derived from Flickr tags were found to give promising result for predicting a range of different environemental phenomena 
Our main hypothesis in this paper is that by using vector space embeddings instead of bag-of-words representations, the ecological information which is implicitly captured by Flickr tags can be utilized in a more effective way  Vector space embeddings are representations in which the objects from a given domain are encoded using relatively low-dimensional vectors  They have proven useful in natural language processing, especially for encoding word meaning BIBREF8 , BIBREF9 , and in machine learning more generally  In this paper, we are interested in the use of such representations for modelling geographic locations  Our main motivation for using vector space embeddings is that they allow us to integrate the textual information we get from Flickr with available structured information in a very natural way  To this end, we rely on an adaptation of the GloVe word embedding model BIBREF9 , but rather than learning word vectors, we learn vectors representing locations  Similar to how the representation of a word in GloVe is determined by the context words surrounding it, the representation of a location in our model is determined by the tags of the photos that have been taken near that location  To incorporate numerical features from structured environmental datasets (e g  average temperature), we associate with each such feature a linear mapping that can be used to predict that feature from a given location vector  This is inspired by the fact that salient properties of a given domain can often be modelled as directions in vector space embeddings BIBREF10 , BIBREF11 , BIBREF12   Finally, evidence from categorical datasets (e g  land cover types) is taken into account by requiring that locations belonging to the same category are represented using similar vectors, similar to how semantic types are sometimes modelled in the context of knowledge graph embedding BIBREF13  
While our point-of-departure is a standard word embedding model, we found that the off-the-shelf GloVe model performed surprisingly poorly, meaning that a number of modifications are needed to achieve good results  Our main findings are as follows  First, given that the number of tags associated with a given location can be quite small, it is important to apply some kind of spatial smoothing, i e  the importance of a given tag for a given location should not only depend on the occurrences of the tag at that location, but also on its occurrences at nearby locations  To this end, we use a formulation which is based on spatially smoothed version of pointwise mutual information  Second, given the wide diversity in the kind of information that is covered by Flickr tags, we find that term selection is in some cases critical to obtain vector spaces that capture the relevant aspects of geographic locations  For instance, many tags on Flickr refer to photography related terms, which we would normally not want to affect the vector representation of a given location  Finally, even with these modifications, vector space embeddings learned from Flickr tags alone are sometimes outperformed by bag-of-words representations  However, our vector space embeddings lead to substantially better predictions in cases where structured (scientific) information is also taken into account  In this sense, the main value of using vector space embeddings in this context is not so much about abstracting away from specific tag usages, but rather about the fact that such representations allow us to integrate numerical and categorical features in a much more natural way than is possible with bag-of-words representations 
The remainder of this paper is organized as follows  In the next section, we provide a discussion of existing work  Section SECREF3 then presents our model for embedding geographic locations from Flickr tags and structured data  Next, in Section SECREF4 we provide a detailed discussion about the experimental results  Finally, Section SECREF5 summarizes our conclusions 
Vector space embeddings
The use of low-dimensional vector space embeddings for representing objects has already proven effective in a large number of applications, including natural language processing (NLP), image processing, and pattern recognition  In the context of NLP, the most prominent example is that of word embeddings, which represent word meaning using vectors of typically around 300 dimensions  A large number of different methods for learning such word embeddings have already been proposed, including Skip-gram and the Continuous Bag-of-Words (CBOW) model BIBREF8 , GloVe BIBREF9 , and fastText BIBREF14   They have been applied effectively in many downstream NLP tasks such as sentiment analysis BIBREF15 , part of speech tagging BIBREF16 , BIBREF17 , and text classification BIBREF18 , BIBREF19   The model we consider in this paper builds on GloVe, which was designed to capture linear regularities of word-word co-occurrence  In GloVe, there are two word vectors INLINEFORM0 and INLINEFORM1 for each word in the vocabulary, which are learned by minimizing the following objective: DISPLAYFORM0
where INLINEFORM0 is the number of times that word INLINEFORM1 appears in the context of word INLINEFORM2 , INLINEFORM3 is the vocabulary size, INLINEFORM4 is the target word bias, INLINEFORM5 is the context word bias  The weighting function INLINEFORM6 is used to limit the impact of rare terms  It is defined as 1 if INLINEFORM7 and as INLINEFORM8 otherwise, where INLINEFORM9 is usually fixed to 100 and INLINEFORM10 to 0 75  Intuitively, the target word vectors INLINEFORM11 correspond to the actual word representations which we would like to find, while the context word vectors INLINEFORM12 model how occurrences of INLINEFORM13 in the context of a given word INLINEFORM14 affect the representation of this latter word  In this paper we will use a similar model, which will however be aimed at learning location vectors instead of the target word vectors 
Beyond word embeddings, various methods have been proposed for learning vector space representations from structured data such as knowledge graphs BIBREF20 , BIBREF21 , BIBREF22 , social networks BIBREF23 , BIBREF24 and taxonomies BIBREF25 , BIBREF26   The idea of combining a word embedding model with structured information has also been explored by several authors, for example to improve the word embeddings based on information coming from knowledge graphs BIBREF27 , BIBREF28   Along similar lines, various lexicons have been used to obtain word embeddings that are better suited at modelling sentiment BIBREF15 and antonymy BIBREF29 , among others  The method proposed by BIBREF30 imposes the condition that words that belong to the same semantic category are closer together than words from different categories, which is somewhat similar in spirit to how we will model categorical datasets in our model 
Embeddings for geographic information
The problem of representing geographic locations using embeddings has also attracted some attention  An early example is BIBREF31 , which used principal component analysis and stacked autoencoders to learn low-dimensional vector representations of city neighbourhoods based on census data  They use these representations to predict attributes such as crime, which is not included in the given census data, and find that in most of the considered evaluation tasks, the low-dimensional vector representations lead to more faithful predictions than the original high-dimensional census data 
Some existing works combine word embedding models with geographic coordinates  For example, in BIBREF32 an approach is proposed to learn word embeddings based on the assumption that words which tend to be used in the same geographic locations are likely to be similar  Note that their aim is dual to our aim in this paper: while they use geographic location to learn word vectors, we use textual descriptions to learn vectors representing geographic locations 
Several methods also use word embedding models to learn representations of Points-of-Interest (POIs) that can be used for predicting user visits BIBREF33 , BIBREF34 , BIBREF35   These works use the machinery of existing word embedding models to learn POI representations, intuitively by letting sequences of POI visits by a user play the role of sequences of words in a sentence  In other words, despite the use of word embedding models, many of these approaches do not actually consider any textual information  For example, in BIBREF34 the Skip-gram model is utilized to create a global pattern of users' POIs  Each location was treated as a word and the other locations visited before or after were treated as context words  They then use a pair-wise ranking loss BIBREF36 which takes into account the user's location visit frequency to personalize the location recommendations  The methods of BIBREF34 were extended in BIBREF35 to use a temporal embedding and to take more account of geographic context, in particular the distances between preferred and non-preferred neighboring POIs, to create a “geographically hierarchical pairwise preference ranking model”  Similarly, in BIBREF37 the CBOW model was trained with POI data  They ordered POIs spatially within the traffic-based zones of urban areas  The ordering was used to generate characteristic vectors of POI types  Zone vectors represented by averaging the vectors of the POIs contained in them, were then used as features to predict land use types  In the CrossMap method BIBREF38 they learned embeddings for spatio-temporal hotspots obtained from social media data of locations, times and text  In one form of embedding, intended to enable reconstruction of records, neighbourhood relations in space and time were encoded by averaging hotspots in a target location's spatial and temporal neighborhoods  They also proposed a graph-based embedding method with nodes of location, time and text  The concatenation of the location, time and text vectors were then used as features to predict peoples' activities in urban environments  Finally, in BIBREF39 , a method is proposed that uses the Skip-gram model to represent POI types, based on the intuition that the vector representing a given POI type should be predictive of the POI types that found near places of that type 
Our work is different from these studies, as our focus is on representing locations based on a given text description of that location (in the form of Flickr tags), along with numerical and categorical features from scientific datasets 
Analyzing Flickr tags
Many studies have focused on analyzing Flickr tags to extract useful information in domains such as linguistics BIBREF40 , geography BIBREF0 , BIBREF41 , and ecology BIBREF42 , BIBREF7 , BIBREF43   Most closely related to our work, BIBREF7 found that the tags of georeferenced Flickr photos can effectively supplement traditional scientific environmental data in tasks such as predicting climate features, land cover, species occurrence, and human assessments of scenicness  To encode locations, they simply combine a bag-of-words representation of geographically nearby tags with a feature vector that encodes associated structured scientific data  They found that the predictive value of Flickr tags is roughly on a par with that of the scientific datasets, and that combining both types of information leads to significantly better results than using either of them alone  As we show in this paper, however, their straightforward way of combining both information sources, by concatenating the two types of feature vectors, is far from optimal 
Despite the proven importance of Flickr tags, the problem of embedding Flickr tags has so far received very limited attention  To the best of our knowledge, BIBREF44 is the only work that generated embeddings for Flickr tags  However, their focus was on learning embeddings that capture word meaning (being evaluated on word similarity tasks), whereas we use such embeddings as part of our method for representing locations 
Model Description
In this section, we introduce our embedding model, which combines Flickr tags and structured scientific information to represent a set of locations INLINEFORM0   The proposed model has the following form: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are parameters to control the importance of each component in the model  Component INLINEFORM2 will be used to constrain the representation of the locations based on their textual description (i e  Flickr tags), INLINEFORM3 will be used to constrain the representation of the locations based on their numerical features, and INLINEFORM4 will impose the constraint that locations belonging to the same category should be close together in the space  We will discuss each of these components in more detail in the following sections 
Tag Based Location Embedding
Many of the tags associated with Flickr photos describe characteristics of the places where these photos were taken BIBREF45 , BIBREF46 , BIBREF47   For example, tags may correspond to place names (e g  Brussels, England, Scandinavia), landmarks (e g  Eiffel Tower, Empire State Building) or land cover types (e g  mountain, forest, beach)  To allow us to build location models using such tags, we collected the tags and meta-data of 70 million Flickr photos with coordinates in Europe (which is the region our experiments will focus on), all of which were uploaded to Flickr before the end of September 2015  In this section we first explain how tags can be weighted to obtain bag-of-words representations of locations from Flickr  Subsequently we describe a tag selection method, which will allow us to specialize the embedding depending on which aspects of the considered locations are of interest, after which we discuss the actual embedding model 
Tag weighting  Let INLINEFORM0 be a set of geographic locations, each characterized by latitude and longitude coordinates  To generate a bag-of-words representation of a given location, we have to weight the relevance of each tag to that location  To this end, we have followed the weighting scheme from BIBREF7 , which combines a Gaussian kernel (to model spatial proximity) with Positive Pointwise Mutual Information (PPMI) BIBREF48 , BIBREF49  
Let us write INLINEFORM0 for the set of users who have assigned tag INLINEFORM1 to a photo with coordinates near INLINEFORM2   To assess how relevant INLINEFORM3 is to the location INLINEFORM4 , the number of times INLINEFORM5 occurs in photos near INLINEFORM6 is clearly an important criterion  However, rather than simply counting the number of occurrences within some fixed radius, we use a Gaussian kernel to weight the tag occurrences according to their distance from that location: INLINEFORM7
where the threshold INLINEFORM0 is assumed to be fixed, INLINEFORM1 is the location of a Flickr photo, INLINEFORM2 is the Haversine distance, and we will assume that the bandwidth parameter INLINEFORM3 is set to INLINEFORM4   A tag occurrence is counted only once for all photos by the same user at the same location, which is important to reduce the impact of bulk uploading  The value INLINEFORM5 reflects how frequent tag INLINEFORM6 is near location INLINEFORM7 , but it does not yet take into account the total number of tag occurrences near INLINEFORM8 , nor how popular the tag INLINEFORM9 is overall  To measure how strongly tag INLINEFORM10 is associated with location INLINEFORM11 , we use PPMI, which is a commonly used measure of association in natural language processing  However, rather than estimating PPMI scores from term frequencies, we will use the INLINEFORM12 values instead: INLINEFORM13
where: INLINEFORM0
with INLINEFORM0 the set of all tags, and INLINEFORM1 the set of locations 
Tag selection  Inspired by BIBREF50 , we use a term selection method in order to focus on the tags that are most important for the tasks that we want to consider and reduce the impact of tags that might relate only to a given individual or a group of users  In particular, we obtained good results with a method based on Kullback-Leibler (KL) divergence, which is based on BIBREF51   Let INLINEFORM0 be a set of (mutually exclusive) properties of locations in which we are interested (e g  land cover categories)  For the ease of presentation, we will identify INLINEFORM1 with the set of locations that have the corresponding property  Then, we select tags from INLINEFORM2 that maximize the following score: INLINEFORM3
where INLINEFORM0 is the probability that a photo with tag INLINEFORM1 has a location near INLINEFORM2 and INLINEFORM3 is the probability that an arbitrary tag occurrence is assigned to a photo near a location in INLINEFORM4   Since INLINEFORM5 often has to be estimated from a small number of tag occurrences, it is estimated using Bayesian smoothing: INLINEFORM6
where INLINEFORM0 is a parameter controlling the amount of smoothing, which will be tuned in the experiments  On the other hand, for INLINEFORM1 we can simply use a maximum likelihood estimation: INLINEFORM2
Location embedding  We now want to find a vector INLINEFORM0 for each location INLINEFORM1 such that similar locations are represented using similar vectors  To achieve this, we use a close variant of the GloVe model, where tag occurrences are treated as context words of geographic locations  In particular, with each location INLINEFORM2 we associate a vector INLINEFORM3 and with each tag INLINEFORM4 we associate a vector INLINEFORM5 and a bias term INLINEFORM6 , and consider the following objective (which in our full model ( EQREF7 ) will be combined with components that are derived from the structured information): INLINEFORM7
Note how tags play the role of the context words in the GloVe model, while instead of learning target word vectors we now learn location vectors  In contrast to GloVe, our objective does not directly refer to co-occurrence statistics, but instead uses the INLINEFORM0 scores  One important consequence of this is that we can also consider pairs INLINEFORM1 for which INLINEFORM2 does not occur in INLINEFORM3 at all; such pairs are usually called negative examples  While they cannot be used in the standard GloVe model, some authors have already reported that introducing negative examples in variants of GloVe can lead to an improvement BIBREF52   In practice, evaluating the full objective above would not be computationally feasible, as we may need to consider millions of locations and millions of tags  Therefore, rather than considering all tags in INLINEFORM4 for the inner summation, we only consider those tags that appear at least once near location INLINEFORM5 together with a sample of negative examples 
Structured Environmental Data
There is a wide variety of structured data that can be used to describe locations  In this work, we have restricted ourselves to the same datasets as BIBREF7   These include nine (real-valued) numerical features, which are latitude, longitude, elevation, population, and five climate related features (avg  temperature, avg  precipitation, avg  solar radiation, avg  wind speed, and avg  water vapor pressure)  In addition, 180 categorical features were used, which are CORINE land cover classes at level 1 (5 classes), level 2 (15 classes) and level 3 (44 classes) and 116 soil types (SoilGrids)  Note that each location should belong to exactly 4 categories: one CORINE class at each of the three levels and a soil type 
Numerical features  Numerical features can be treated similarly to the tag occurrences, i e  we will assume that the value of a given numerical feature can be predicted from the location vectors using a linear mapping  In particular, for each numerical feature INLINEFORM0 we consider a vector INLINEFORM1 and a bias term INLINEFORM2 , and the following objective: INLINEFORM3
where we write INLINEFORM0 for set of all numerical features and INLINEFORM1 is the value of feature INLINEFORM2 for location INLINEFORM3 , after z-score normalization 
Categorical features  To take into account the categorical features, we impose the constraint that locations belonging to the same category should be close together in the space  To formalize this, we represent each category type INLINEFORM0 as a vector INLINEFORM1 , and consider the following objective: INLINEFORM2
Evaluation Tasks
We will use the method from BIBREF7 as our main baseline  This will allow us to directly evaluate the effectiveness of embeddings for the considered problem, since we have used the same structured datasets and same tag weighting scheme  For this reason, we will also follow their evaluation methodology  In particular, we will consider three evaluation tasks:
Predicting the distribution of 100 species across Europe, using the European network of nature protected sites Natura 2000 dataset as ground truth  For each of these species, a binary classification problem is considered  The set of locations INLINEFORM0 is defined as the 26,425 distinct sites occurring in the dataset 
Predicting soil type, again each time treating the task as a binary classification problem, using the same set of locations INLINEFORM0 as in the species distribution experiments  For these experiments, none of the soil type features are used for generating the embeddings 
Predicting CORINE land cover classes at levels 1, 2 and level 3, each time treating the task as a binary classification problem, using the same set of locations INLINEFORM0 as in the species distribution experiments  For these experiments, none of the CORINE features are used for generating the embeddings 
In addition, we will also consider the following regression tasks:
Predicting 5 climate related features: the average precipitation, temperature, solar radiation, water vapor pressure, and wind speed  We again use the same set of locations INLINEFORM0 as for species distribution in this experiment  None of the climate features is used for constructing the embeddings for this experiment 
Predicting people's subjective opinions of landscape beauty in Britain, using the crowdsourced dataset from the ScenicOrNot website as ground truth  The set INLINEFORM0 is chosen as the set of locations of 191 605 rated locations from the ScenicOrNot dataset for which at least one georeferenced Flickr photo exists within a 1 km radius 
Experimental Setup
In all experiments, we use Support Vector Machines (SVMs) for classification problems and Support Vector Regression (SVR) for regression problems to make predictions from our representations of geographic locations  In both cases, we used the SVM INLINEFORM0 implementation BIBREF53   For each experiment, the set of locations INLINEFORM1 was split into two-thirds for training, one-sixth for testing, and one-sixth for tuning the parameters  All embedding models are learned with Adagrad using 30 iterations  The number of dimensions is chosen for each experiment from INLINEFORM2 based on the tuning data  For the parameters of our model in Equation EQREF7 , we considered values of INLINEFORM3 from {0 1, 0 01, 0 001, 0 0001} and values of INLINEFORM4 from {1, 10, 100, 1000, 10 000, 100 000}  To compute KL divergence, we need to determine a set of classes INLINEFORM5 for each experiment  For classification problems, we can simply consider the given categories, but for the regression problems we need to define such classes by discretizing the numerical values  For the scenicness experiments, we considered scores 3 and 7 as cut-off points, leading to three classes (i e  less than 3, between 3 and 7, and above 7)  Similarly, for each climate related features, we consider two cut-off values for discretization: 5 and 15 for average temperature, 50 and 100 for average precipitation, 10 000 and 17 000 for average solar radiation, 0 7 and 1 for average water vapor pressure, and 3 and 5 for wind speed  The smoothing parameter INLINEFORM6 was selected among INLINEFORM7 based on the tuning data  In all experiments where term selection is used, we select the top 100 000 tags  We fixed the radius INLINEFORM8 at 1km when counting the number of tag occurrences  Finally, we set the number of negative examples as 10 times the number of positive examples for each location, but with a cap at 1000 negative examples in each region for computational reasons  We tune all parameters with respect to the F1 score for the classification tasks, and Spearman INLINEFORM9 for the regression tasks 
Variants and Baseline Methods
We will refer to our model as EGEL (Embedding GEographic Locations), and will consider the following variants  EGEL-Tags only uses the information from the Flickr tags (i e  component INLINEFORM0 ), without using any negative examples and without feature selection  EGEL-Tags+NS is similar to EGEL-Tags but with the addition of negative examples  EGEL-KL(Tags+NS) additionally considers term selection  EGEL-All is our full method, i e  it additionally uses the structured information  We also consider the following baselines  BOW-Tags represents locations using a bag-of-words representation, using the same tag weighting as the embedding model  BOW-KL(Tags) uses the same representation but after term selection, using the same KL-based method as the embedding model  BOW-All combines the bag-of-words representation with the structured information, encoded as proposed in BIBREF7   GloVe uses the objective from the original GloVe model for learning location vectors, i e  this variant differs from EGEL-Tags in that instead of INLINEFORM1 we use the number of co-occurrences of tag INLINEFORM2 near location INLINEFORM3 , measured as INLINEFORM4  
Results and Discussion
We present our results for the binary classification tasks in Tables TABREF23 – TABREF24 in terms of average precision, average recall and macro average F1 score  The results of the regression tasks are reported in Tables TABREF25 and TABREF29 in terms of the mean absolute error between the predicted and actual scores, as well as the Spearman INLINEFORM0 correlation between the rankings induced by both sets of scores  It can be clearly seen from the results that our proposed method (EGEL-All) can effectively integrate Flickr tags with the available structured information  It outperforms the baselines for all the considered tasks  Furthermore, note that the PPMI-based weighting in EGEL-Tags consistently outperforms GloVe and that both the addition of negative examples and term selection lead to further improvements  The use of term selection leads to particularly substantial improvements for the regression problems 
While our experimental results confirm the usefulness of embeddings for predicting environmental features, this is only consistently the case for the variants that use both the tags and the structured datasets  In particular, comparing BOW-Tags with EGEL-Tags, we sometimes see that the former achieves the best results  While this might seem surprising, it is in accordance with the findings in BIBREF54 , BIBREF38 , among others, where it was also found that bag-of-words representations can sometimes lead to surprisingly effective baselines  Interestingly, we note that in all cases where EGEL-KL(Tags+NS) performs worse than BOW-Tags, we also find that BOW-KL(Tags) performs worse than BOW-Tags  This suggests that for these tasks there is a very large variation in the kind of tags that can inform the prediction model, possibly including e g  user-specific tags  Some of the information captured by such highly specific but rare tags is likely to be lost in the embedding 
To further analyze the difference in performance between BoW representations and embeddings, Figure TABREF29 compares the performance of the GloVe model with the bag-of-words model for predicting place scenicness, as a function of the number of tag occurrences at the considered locations  What is clearly noticeable in Figure TABREF29 is that GloVe performs better than the bag-of-words model for large corpora and worse for smaller corpora  This issue has been alleviated in our embedding method by the addition of negative examples 
 Conclusions
In this paper, we have proposed a model to learn geographic location embeddings using Flickr tags, numerical environmental features, and categorical information  The experimental results show that our model can integrate Flickr tags with structured information in a more effective way than existing methods, leading to substantial improvements over baseline methods on various prediction tasks about the natural environment 
Acknowledgments
Shelan Jeawak has been sponsored by HCED Iraq  Steven Schockaert has been supported by ERC Starting Grant 637277 

ground_truth:  the same datasets as BIBREF7

same datasets as BIBREF7

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What data was presented to the subjects to elicit event-related responses?
Context 2: Introduction
Decoding intended speech or motor activity from brain signals is one of the major research areas in Brain Computer Interface (BCI) systems BIBREF0 , BIBREF1   In particular, speech-related BCI technologies attempt to provide effective vocal communication strategies for controlling external devices through speech commands interpreted from brain signals BIBREF2   Not only do they provide neuro-prosthetic help for people with speaking disabilities and neuro-muscular disorders like locked-in-syndrome, nasopharyngeal cancer, and amytotropic lateral sclerosis (ALS), but also equip people with a better medium to communicate and express thoughts, thereby improving the quality of rehabilitation and clinical neurology BIBREF3 , BIBREF4   Such devices also have applications in entertainment, preventive treatments, personal communication, games, etc  Furthermore, BCI technologies can be utilized in silent communication, as in noisy environments, or situations where any sort of audio-visual communication is infeasible 
Among the various brain activity-monitoring modalities in BCI, electroencephalography (EEG) BIBREF5 , BIBREF6 has demonstrated promising potential to differentiate between various brain activities through measurement of related electric fields  EEG is non-invasive, portable, low cost, and provides satisfactory temporal resolution  This makes EEG suitable to realize BCI systems  EEG data, however, is challenging: these data are high dimensional, have poor SNR, and suffer from low spatial resolution and a multitude of artifacts  For these reasons, it is not particularly obvious how to decode the desired information from raw EEG signals  Although the area of BCI based speech intent recognition has received increasing attention among the research community in the past few years, most research has focused on classification of individual speech categories in terms of discrete vowels, phonemes and words BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15   This includes categorization of imagined EEG signal into binary vowel categories like /a/, /u/ and rest BIBREF7 , BIBREF8 , BIBREF9 ; binary syllable classes like /ba/ and /ku/ BIBREF1 , BIBREF10 , BIBREF11 , BIBREF12 ; a handful of control words like 'up', 'down', 'left', 'right' and 'select' BIBREF15 or others like 'water', 'help', 'thanks', 'food', 'stop' BIBREF13 , Chinese characters BIBREF14 , etc  Such works mostly involve traditional signal processing or manual feature handcrafting along with linear classifiers (e g , SVMs)  In our recent work BIBREF16 , we introduced deep learning models for classification of vowels and words that achieved 23 45% improvement of accuracy over the baseline 
Production of articulatory speech is an extremely complicated process, thereby rendering understanding of the discriminative EEG manifold corresponding to imagined speech highly challenging  As a result, most of the existing approaches failed to achieve satisfactory accuracy on decoding speech tokens from the speech imagery EEG data  Perhaps, for these reasons, very little work has been devoted to relating the brain signals to the underlying articulation  The few exceptions include BIBREF17 , BIBREF18   In BIBREF17 , Zhao et al  used manually handcrafted features from EEG data, combined with speech audio and facial features to achieve classification of the phonological categories varying based on the articulatory steps  However, the imagined speech classification accuracy based on EEG data alone, as reported in BIBREF17 , BIBREF18 , are not satisfactory in terms of accuracy and reliability  We now turn to describing our proposed models 
Proposed Framework
Cognitive learning process underlying articulatory speech production involves incorporation of intermediate feedback loops and utilization of past information stored in the form of memory as well as hierarchical combination of several feature extractors  To this end, we develop our mixed neural network architecture composed of three supervised and a single unsupervised learning step, discussed in the next subsections and shown in Fig  FIGREF1   We formulate the problem of categorizing EEG data based on speech imagery as a non-linear mapping INLINEFORM0 of a multivariate time-series input sequence INLINEFORM1 to fixed output INLINEFORM2 , i e, mathematically INLINEFORM3 : INLINEFORM4 , where c and t denote the EEG channels and time instants respectively 
Preprocessing step
We follow similar pre-processing steps on raw EEG data as reported in BIBREF17 (ocular artifact removal using blind source separation, bandpass filtering and subtracting mean value from each channel) except that we do not perform Laplacian filtering step since such high-pass filtering may decrease information content from the signals in the selected bandwidth 
Joint variability of electrodes
Multichannel EEG data is high dimensional multivariate time series data whose dimensionality depends on the number of electrodes  It is a major hurdle to optimally encode information from these EEG data into lower dimensional space  In fact, our investigation based on a development set (as we explain later) showed that well-known deep neural networks (e g , fully connected networks such as convolutional neural networks, recurrent neural networks and autoencoders) fail to individually learn such complex feature representations from single-trial EEG data  Besides, we found that instead of using the raw multi-channel high-dimensional EEG requiring large training times and resource requirements, it is advantageous to first reduce its dimensionality by capturing the information transfer among the electrodes  Instead of the conventional approach of selecting a handful of channels as BIBREF17 , BIBREF18 , we address this by computing the channel cross-covariance, resulting in positive, semi-definite matrices encoding the connectivity of the electrodes  We define channel cross-covariance (CCV) between any two electrodes INLINEFORM0 and INLINEFORM1 as: INLINEFORM2   Next, we reject the channels which have significantly lower cross-covariance than auto-covariance values (where auto-covariance implies CCV on same electrode)  We found this measure to be essential as the higher cognitive processes underlying speech planning and synthesis involve frequent information exchange between different parts of the brain  Hence, such matrices often contain more discriminative features and hidden information than mere raw signals  This is essentially different than our previous work BIBREF16 where we extract per-channel 1-D covariance information and feed it to the networks  We present our sample 2-D EEG cross-covariance matrices (of two individuals) in Fig  FIGREF2  
CNN & LSTM
In order to decode spatial connections between the electrodes from the channel covariance matrix, we use a CNN BIBREF19 , in particular a four-layered 2D CNN stacking two convolutional and two fully connected hidden layers  The INLINEFORM0 feature map at a given CNN layer with input INLINEFORM1 , weight matrix INLINEFORM2 and bias INLINEFORM3 is obtained as: INLINEFORM4   At this first level of hierarchy, the network is trained with the corresponding labels as target outputs, optimizing a cross-entropy cost function  In parallel, we apply a four-layered recurrent neural network on the channel covariance matrices to explore the hidden temporal features of the electrodes  Namely, we exploit an LSTM BIBREF20 consisting of two fully connected hidden layers, stacked with two LSTM layers and trained in a similar manner as CNN 
Deep autoencoder for spatio-temporal information
As we found the individually-trained parallel networks (CNN and LSTM) to be useful (see Table TABREF12 ), we suspected the combination of these two networks could provide a more powerful discriminative spatial and temporal representation of the data than each independent network  As such, we concatenate the last fully-connected layer from the CNN with its counterpart in the LSTM to compose a single feature vector based on these two penultimate layers  Ultimately, this forms a joint spatio-temporal encoding of the cross-covariance matrix 
In order to further reduce the dimensionality of the spatio-temporal encodings and cancel background noise effects BIBREF21 , we train an unsupervised deep autoenoder (DAE) on the fused heterogeneous features produced by the combined CNN and LSTM information  The DAE forms our second level of hierarchy, with 3 encoding and 3 decoding layers, and mean squared error (MSE) as the cost function 
Classification with Extreme Gradient Boost
At the third level of hierarchy, the discrete latent vector representation of the deep autoencoder is fed into an Extreme Gradient Boost based classification layer BIBREF22 , BIBREF23 motivated by BIBREF21   It is a regularized gradient boosted decision tree that performs well on structured problems  Since our EEG-phonological pairwise classification has an internal structure involving individual phonemes and words, it seems to be a reasonable choice of classifier  The classifier receives its input from the latent vectors of the deep autoencoder and is trained in a supervised manner to output the final predicted classes corresponding to the speech imagery 
Dataset
We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)  The dataset consists of 14 participants, with each prompt presented 11 times to each individual  Since our intention is to classify the phonological categories from human thoughts, we discard the facial and audio information and only consider the EEG data corresponding to imagined speech  It is noteworthy that given the mixed nature of EEG signals, it is reportedly challenging to attain a pairwise EEG-phoneme mapping BIBREF18   In order to explore the problem space, we thus specifically target five binary classification problems addressed in BIBREF17 , BIBREF18 , i e presence/absence of consonants, phonemic nasal, bilabial, high-front vowels and high-back vowels 
Training and hyperparameter selection
We performed two sets of experiments with the single-trial EEG data  In PHASE-ONE, our goals was to identify the best architectures and hyperparameters for our networks with a reasonable number of runs  For PHASE-ONE, we randomly shuffled and divided the data (1913 signals from 14 individuals) into train (80%), development (10%) and test sets (10%)  In PHASE-TWO, in order to perform a fair comparison with the previous methods reported on the same dataset, we perform a leave-one-subject out cross-validation experiment using the best settings we learn from PHASE-ONE 
The architectural parameters and hyperparameters listed in Table TABREF6 were selected through an exhaustive grid-search based on the validation set of PHASE-ONE  We conducted a series of empirical studies starting from single hidden-layered networks for each of the blocks and, based on the validation accuracy, we increased the depth of each given network and selected the optimal parametric set from all possible combinations of parameters  For the gradient boosting classification, we fixed the maximum depth at 10, number of estimators at 5000, learning rate at 0 1, regularization coefficient at 0 3, subsample ratio at 0 8, and column-sample/iteration at 0 4  We did not find any notable change of accuracy while varying other hyperparameters while training gradient boost classifier 
Performance analysis and discussion
To demonstrate the significance of the hierarchical CNN-LSTM-DAE method, we conducted separate experiments with the individual networks in PHASE-ONE of experiments and summarized the results in Table TABREF12 From the average accuracy scores, we observe that the mixed network performs much better than individual blocks which is in agreement with the findings in BIBREF21   A detailed analysis on repeated runs further shows that in most of the cases, LSTM alone does not perform better than chance  CNN, on the other hand, is heavily biased towards the class label which sees more training data corresponding to it  Though the situation improves with combined CNN-LSTM, our analysis clearly shows the necessity of a better encoding scheme to utilize the combined features rather than mere concatenation of the penultimate features of both networks 
The very fact that our combined network improves the classification accuracy by a mean margin of 14 45% than the CNN-LSTM network indeed reveals that the autoencoder contributes towards filtering out the unrelated and noisy features from the concatenated penultimate feature set  It also proves that the combined supervised and unsupervised neural networks, trained hierarchically, can learn the discriminative manifold better than the individual networks and it is crucial for improving the classification accuracy  In addition to accuracy, we also provide the kappa coefficients BIBREF24 of our method in Fig  FIGREF14   Here, a higher mean kappa value corresponding to a task implies that the network is able to find better discriminative information from the EEG data beyond random decisions  The maximum above-chance accuracy (75 92%) is recorded for presence/absence of the vowel task and the minimum (49 14%) is recorded for the INLINEFORM0  
To further investigate the feature representation achieved by our model, we plot T-distributed Stochastic Neighbor Embedding (tSNE) corresponding to INLINEFORM0 and V/C classification tasks in Fig  FIGREF8   We particularly select these two tasks as our model exhibits respectively minimum and maximum performance for these two  The tSNE visualization reveals that the second set of features are more easily separable than the first one, thereby giving a rationale for our performance 
Next, we provide performance comparison of the proposed approach with the baseline methods for PHASE-TWO of our study (cross-validation experiment) in Table TABREF15   Since the model encounters the unseen data of a new subject for testing, and given the high inter-subject variability of the EEG data, a reduction in the accuracy was expected  However, our network still managed to achieve an improvement of 18 91, 9 95, 67 15, 2 83 and 13 70 % over BIBREF17   Besides, our best model shows more reliability compared to previous works: The standard deviation of our model's classification accuracy across all the tasks is reduced from 22 59% BIBREF17 and 17 52% BIBREF18 to a mere 5 41% 
Conclusion and future direction
In an attempt to move a step towards understanding the speech information encoded in brain signals, we developed a novel mixed deep neural network scheme for a number of binary classification tasks from speech imagery EEG data  Unlike previous approaches which mostly deal with subject-dependent classification of EEG into discrete vowel or word labels, this work investigates a subject-invariant mapping of EEG data with different phonological categories, varying widely in terms of underlying articulator motions (eg: involvement or non-involvement of lips and velum, variation of tongue movements etc)  Our model takes an advantage of feature extraction capability of CNN, LSTM as well as the deep learning benefit of deep autoencoders  We took BIBREF17 , BIBREF18 as the baseline works investigating the same problem and compared our performance with theirs  Our proposed method highly outperforms the existing methods across all the five binary classification tasks by a large average margin of 22 51% 
Acknowledgments
This work was funded by the Natural Sciences and Engineering Research Council (NSERC) of Canada and Canadian Institutes for Health Research (CIHR) 
Context 3: Introduction
Reading Comprehension (RC) has become a central task in natural language processing, with great practical value in various industries  In recent years, many large-scale RC datasets in English BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 have nourished the development of numerous powerful and diverse RC models BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11  The state-of-the-art model BIBREF12 on SQuAD, one of the most widely used RC benchmarks, even surpasses human-level performance  Nonetheless, RC on languages other than English has been limited due to the absence of sufficient training data  Although some efforts have been made to create RC datasets for Chinese BIBREF13, BIBREF14 and Korean BIBREF15, it is not feasible to collect RC datasets for every language since annotation efforts to collect a new RC dataset are often far from trivial  Therefore, the setup of transfer learning, especially zero-shot learning, is of extraordinary importance 
Existing methods BIBREF16 of cross-lingual transfer learning on RC datasets often count on machine translation (MT) to translate data from source language into target language, or vice versa  These methods may not require a well-annotated RC dataset for the target language, whereas a high-quality MT model is needed as a trade-off, which might not be available when it comes to low-resource languages 
In this paper, we leverage pre-trained multilingual language representation, for example, BERT learned from multilingual un-annotated sentences (multi-BERT), in cross-lingual zero-shot RC  We fine-tune multi-BERT on the training set in source language, then test the model in target language, with a number of combinations of source-target language pair to explore the cross-lingual ability of multi-BERT  Surprisingly, we find that the models have the ability to transfer between low lexical similarity language pair, such as English and Chinese  Recent studies BIBREF17, BIBREF12, BIBREF18 show that cross-lingual language models have the ability to enable preliminary zero-shot transfer on simple natural language understanding tasks, but zero-shot transfer of RC has not been studied  To our knowledge, this is the first work systematically exploring the cross-lingual transferring ability of multi-BERT on RC tasks 
Zero-shot Transfer with Multi-BERT
Multi-BERT has showcased its ability to enable cross-lingual zero-shot learning on the natural language understanding tasks including XNLI BIBREF19, NER, POS, Dependency Parsing, and so on  We now seek to know if a pre-trained multi-BERT has ability to solve RC tasks in the zero-shot setting 
Zero-shot Transfer with Multi-BERT ::: Experimental Setup and Data
We have training and testing sets in three different languages: English, Chinese and Korean  The English dataset is SQuAD BIBREF2  The Chinese dataset is DRCD BIBREF14, a Chinese RC dataset with 30,000+ examples in the training set and 10,000+ examples in the development set  The Korean dataset is KorQuAD BIBREF15, a Korean RC dataset with 60,000+ examples in the training set and 10,000+ examples in the development set, created in exactly the same procedure as SQuAD  We always use the development sets of SQuAD, DRCD and KorQuAD for testing since the testing sets of the corpora have not been released yet 
Next, to construct a diverse cross-lingual RC dataset with compromised quality, we translated the English and Chinese datasets into more languages, with Google Translate  An obvious issue with this method is that some examples might no longer have a recoverable span  To solve the problem, we use fuzzy matching to find the most possible answer, which calculates minimal edit distance between translated answer and all possible spans  If the minimal edit distance is larger than min(10, lengths of translated answer - 1), we drop the examples during training, and treat them as noise when testing  In this way, we can recover more than 95% of examples  The following generated datasets are recovered with same setting 
The pre-trained multi-BERT is the official released one  This multi-lingual version of BERT were pre-trained on corpus in 104 languages  Data in different languages were simply mixed in batches while pre-training, without additional effort to align between languages  When fine-tuning, we simply adopted the official training script of BERT, with default hyperparameters, to fine-tune each model until training loss converged 
Zero-shot Transfer with Multi-BERT ::: Experimental Results
Table TABREF6 shows the result of different models trained on either Chinese or English and tested on Chinese  In row (f), multi-BERT is fine-tuned on English but tested on Chinese, which achieves competitive performance compared with QANet trained on Chinese  We also find that multi-BERT trained on English has relatively lower EM compared with the model with comparable F1 scores  This shows that the model learned with zero-shot can roughly identify the answer spans in context but less accurate  In row (c), we fine-tuned a BERT model pre-trained on English monolingual corpus (English BERT) on Chinese RC training data directly by appending fastText-initialized Chinese word embeddings to the original word embeddings of English-BERT  Its F1 score is even lower than that of zero-shot transferring multi-BERT (rows (c) v s  (e))  The result implies multi-BERT does acquire better cross-lingual capability through pre-training on multilingual corpus  Table TABREF8 shows the results of multi-BERT fine-tuned on different languages and then tested on English , Chinese and Korean  The top half of the table shows the results of training data without translation  It is not surprising that when the training and testing sets are in the same language, the best results are achieved, and multi-BERT shows transfer capability when training and testing sets are in different languages, especially between Chinese and Korean 
In the lower half of Table TABREF8, the results are obtained by the translated training data  First, we found that when testing on English and Chinese, translation always degrades the performance (En v s  En-XX, Zh v s  Zh-XX)  Even though we translate the training data into the same language as testing data, using the untranslated data still yield better results  For example, when testing on English, the F1 score of the model training on Chinese (Zh) is 53 8, while the F1 score is only 44 1 for the model training on Zh-En  This shows that translation degrades the quality of data  There are some exceptions when testing on Korean  Translating the English training data into Chinese, Japanese and Korean still improve the performance on Korean  We also found that when translated into the same language, the English training data is always better than the Chinese data (En-XX v s  Zh-XX), with only one exception (En-Fr v s  Zh-Fr when testing on KorQuAD)  This may be because we have less Chinese training data than English  These results show that the quality and the size of dataset are much more important than whether the training and testing are in the same language or not 
Zero-shot Transfer with Multi-BERT ::: Discussion ::: The Effect of Machine Translation
Table TABREF8 shows that fine-tuning on un-translated target language data achieves much better performance than data translated into the target language  Because the above statement is true across all the languages, it is a strong evidence that translation degrades the performance We notice that the translated corpus and untranslated corpus are not the same  This may be another factor that influences the results  Conducting an experiment between un-translated and back-translated data may deal with this problem 
Zero-shot Transfer with Multi-BERT ::: Discussion ::: The Effect of Other Factors
Here we discuss the case that the training data are translated  We consider each result is affected by at least three factors: (1) training corpus, (2) data size, (3) whether the source corpus is translated into the target language  To study the effect of data-size, we conducted an extra experiment where we down-sampled the size of English data to be the same as Chinese corpus, and used the down-sampled corpus to train  Then We carried out one-way ANOVA test and found out the significance of the three factors are ranked as below: (1) > (2) >> (3)  The analysis supports that the characteristics of training data is more important than translated into target language or not  Therefore, although translation degrades the performance, whether translating the corpus into the target language is not critical 
What Does Zero-shot Transfer Model Learn? ::: Unseen Language Dataset
It has been shown that extractive QA tasks like SQuAD may be tackled by some language independent strategies, for example, matching words in questions and context BIBREF20  Is zero-shot learning feasible because the model simply learns this kind of language independent strategies on one language and apply to the other?
To verify whether multi-BERT largely counts on a language independent strategy, we test the model on the languages unseen during pre-training  To make sure the languages have never been seen before, we artificially make unseen languages by permuting the whole vocabulary of existing languages  That is, all the words in the sentences of a specific language are replaced by other words in the same language to form the sentences in the created unseen language  It is assumed that if multi-BERT used to find answers by language independent strategy, then multi-BERT should also do well on unseen languages  Table TABREF14 shows that the performance of multi-BERT drops drastically on the dataset  It implies that multi-BERT might not totally rely on pattern matching when finding answers 
What Does Zero-shot Transfer Model Learn? ::: Embedding in Multi-BERT
PCA projection of hidden representations of the last layer of multi-BERT before and after fine-tuning are shown in Fig  FIGREF15  The red points represent Chinese tokens, and the blue points are for English  The results show that tokens from different languages might be embedded into the same space with close spatial distribution  Even though during the fine-tuning only the English data is used, the embedding of the Chinese token changed accordingly  We also quantitatively evaluate the similarities between the embedding of the languages  The results can be found in the Appendix 
What Does Zero-shot Transfer Model Learn? ::: Code-switching Dataset
We observe linguistic-agnostic representations in the last subsection  If tokens are represented in a language-agnostic way, the model may be able to handle code-switching data  Because there is no code-switching data for RC, we create artificial code-switching datasets by replacing some of the words in contexts or questions with their synonyms in another language  The synonyms are found by word-by-word translation with given dictionaries  We use the bilingual dictionaries collected and released in facebookresearch/MUSE GitHub repository  We substitute the words if and only if the words are in the bilingual dictionaries 
Table TABREF14 shows that on all the code-switching datasets, the EM/F1 score drops, indicating that the semantics of representations are not totally disentangled from language  However, the examples of the answers of the model (Table TABREF21) show that multi-BERT could find the correct answer spans although some keywords in the spans have been translated into another language 
What Does Zero-shot Transfer Model Learn? ::: Typology-manipulated Dataset
There are various types of typology in languages  For example, in English the typology order is subject-verb-object (SVO) order, but in Japanese and Korean the order is subject-object-verb (SOV)  We construct a typology-manipulated dataset to examine if the typology order of the training data influences the transfer learning results  If the model only learns the semantic mapping between different languages, changing English typology order from SVO to SOV should improve the transfer ability from English to Japanese  The method used to generate datasets is the same as BIBREF21 
The source code is from a GitHub repository named Shaul1321/rnn_typology, which labels given sentences to CoNLL format with StanfordCoreNLP and then re-arranges them greedily 
Table TABREF23 shows that when we change the English typology order to SOV or OSV order, the performance on Korean is improved and worsen on English and Chinese, but very slightly  The results show that the typology manipulation on the training set has little influence  It is possible that multi-BERT normalizes the typology order of different languages to some extent 
Conclusion
In this paper, we systematically explore zero-shot cross-lingual transfer learning on RC with multi-BERT  The experimental results on English, Chinese and Korean corpora show that even when the languages for training and testing are different, reasonable performance can be obtained  Furthermore, we created several artificial data to study the cross-lingual ability of multi-BERT in the presence of typology variation and code-switching  We showed that only token-level pattern matching is not sufficient for multi-BERT to answer questions and typology variation and code-switching only caused minor effects on testing performance 
Supplemental Material ::: Internal Representation of multi-BERT
The architecture of multi-BERT is a Transformer encoder BIBREF25  While fine-tuning on SQuAD-like dataset, the bottom layers of multi-BERT are initialized from Google-pretrained parameters, with an added output layer initialized from random parameters  Tokens representations from the last layer of bottom-part of multi-BERT are inputs to the output layer and then the output layer outputs a distribution over all tokens that indicates the probability of a token being the START/END of an answer span 
Supplemental Material ::: Internal Representation of multi-BERT ::: Cosine Similarity
As all translated versions of SQuAD/DRCD are parallel to each other  Given a source-target language pair, we calculate cosine similarity of the mean pooling of tokens representation within corresponding answer-span as a measure of how much they look like in terms of the internal representation of multi-BERT  The results are shown in Fig  FIGREF26 
Supplemental Material ::: Internal Representation of multi-BERT ::: SVCCA
Singular Vector Canonical Correlation Analysis (SVCCA) is a general method to compare the correlation of two sets of vector representations  SVCCA has been proposed to compare learned representations across language models BIBREF24  Here we adopt SVCCA to measure the linear similarity of two sets of representations in the same multi-BERT from different translated datasets, which are parallel to each other  The results are shown in Fig FIGREF28 
Supplemental Material ::: Improve Transfering
In the paper, we show that internal representations of multi-BERT are linear-mappable to some extent between different languages  This implies that multi-BERT model might encode semantic and syntactic information in language-agnostic ways and explains how zero-shot transfer learning could be done 
To take a step further, while transfering model from source dataset to target dataset, we align representations in two proposed way, to improve performance on target dataset 
Supplemental Material ::: Improve Transfering ::: Linear Mapping Method
Algorithms proposed in BIBREF23, BIBREF22, BIBREF26 to unsupervisedly learn linear mapping between two sets of embeddings are used here to align representations of source (training data) to those of target  We obtain the mapping generated by embeddings from one specific layer of pre-trained multi-BERT then we apply this mapping to transform the internal representations of multi-BERT while fine-tuning on training data 
Supplemental Material ::: Improve Transfering ::: Adversarial Method
In Adversarial Method, we add an additional transform layer to transform representations and a discrimination layer to discriminate between transformed representations from source language (training set) and target language (development set)  And the GAN loss is applied in the total loss of fine-tuning 
Supplemental Material ::: Improve Transfering ::: Discussion
As table TABREF33 shows, there are no improvements among above methods  Some linear mapping methods even causes devastating effect on EM/F1 scores 

ground_truth: 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)

KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the model performance on target language reading comprehension?
Context 4:  Next, the embeddings are given as input to a bidirectional LSTM BIBREF14 , in order to create context-dependent representations for every token  The hidden states from forward- and backward-LSTMs are concatenated for each word position, resulting in representations that are conditioned on the whole sequence  This concatenated vector is then passed through an additional feedforward layer, and a softmax over the two possible labels (correct and incorrect) is used to output a probability distribution for each token  The model is optimised by minimising categorical cross-entropy with respect to the correct labels  We use AdaDelta BIBREF15 for calculating an adaptive learning rate during training, which accounts for a higher baseline performance compared to previous results 
Evaluation
We trained our error generation models on the public FCE training set BIBREF16 and used them to generate additional artificial training data  Grammatically correct text is needed as the starting point for inserting artificial errors, and we used two different sources: 1) the corrected version of the same FCE training set on which the system is trained (450K tokens), and 2) example sentences extracted from the English Vocabulary Profile (270K tokens)   While there are other text corpora that could be used (e g , Wikipedia and news articles), our development experiments showed that keeping the writing style and vocabulary close to the target domain gives better results compared to simply including more data 
We evaluated our detection models on three benchmarks: the FCE test data (41K tokens) and the two alternative annotations of the CoNLL 2014 Shared Task dataset (30K tokens) BIBREF3   Each artificial error generation system was used to generate 3 different versions of the artificial data, which were then combined with the original annotated dataset and used for training an error detection system  Table TABREF1 contains example sentences from the error generation systems, highlighting each of the edits that are marked as errors 
The error detection results can be seen in Table TABREF4   We use INLINEFORM0 as the main evaluation measure, which was established as the preferred measure for error correction and detection by the CoNLL-14 shared task BIBREF3   INLINEFORM1 calculates a weighted harmonic mean of precision and recall, which assigns twice as much importance to precision – this is motivated by practical applications, where accurate predictions from an error detection system are more important compared to coverage  For comparison, we also report the performance of the error detection system by Rei2016, trained using the same FCE dataset 
The results show that error detection performance is substantially improved by making use of artificially generated data, created by any of the described methods  When comparing the error generation system by Felice2014a (FY14) with our pattern-based (PAT) and machine translation (MT) approaches, we see that the latter methods covering all error types consistently improve performance  While the added error types tend to be less frequent and more complicated to capture, the added coverage is indeed beneficial for error detection  Combining the pattern-based approach with the machine translation system (Ann+PAT+MT) gave the best overall performance on all datasets  The two frameworks learn to generate different types of errors, and taking advantage of both leads to substantial improvements in error detection 
We used the Approximate Randomisation Test BIBREF17 , BIBREF18 to calculate statistical significance and found that the improvement for each of the systems using artificial data was significant over using only manual annotation  In addition, the final combination system is also significantly better compared to the Felice2014a system, on all three datasets  While Rei2016 also report separate experiments that achieve even higher performance, these models were trained on a considerably larger proprietary corpus  In this paper we compare error detection frameworks trained on the same publicly available FCE dataset, thereby removing the confounding factor of dataset size and only focusing on the model architectures 
The error generation methods can generate alternative versions of the same input text – the pattern-based method randomly samples the error locations, and the SMT system can provide an n-best list of alternative translations  Therefore, we also investigated the combination of multiple error-generated versions of the input files when training error detection models  Figure FIGREF6 shows the INLINEFORM0 score on the development set, as the training data is increased by using more translations from the n-best list of the SMT system  These results reveal that allowing the model to see multiple alternative versions of the same file gives a distinct improvement – showing the model both correct and incorrect variations of the same sentences likely assists in learning a discriminative model 
Related Work
Our work builds on prior research into AEG  Brockett2006 constructed regular expressions for transforming correct sentences to contain noun number errors  Rozovskaya2010a learned confusion sets from an annotated corpus in order to generate preposition errors  Foster2009 devised a tool for generating errors for different types using patterns provided by the user or collected automatically from an annotated corpus  However, their method uses a limited number of edit operations and is thus unable to generate complex errors  Cahill2013 compared different training methodologies and showed that artificial errors helped correct prepositions  Felice2014a learned error type distributions for generating five types of errors, and the system in Section SECREF3 is an extension of this model  While previous work focused on generating a specific subset of error types, we explored two holistic approaches to AEG and showed that they are able to significantly improve error detection performance 
Conclusion
This paper investigated two AEG methods, in order to create additional training data for error detection  First, we explored a method using textual patterns learned from an annotated corpus, which are used for inserting errors into correct input text  In addition, we proposed formulating error generation as an MT framework, learning to translate from grammatically correct to incorrect sentences 
The addition of artificial data to the training process was evaluated on three error detection annotations, using the FCE and CoNLL 2014 datasets  Making use of artificial data provided improvements for all data generation methods  By relaxing the type restrictions and generating all types of errors, our pattern-based method consistently outperformed the system by Felice2014a  The combination of the pattern-based method with the machine translation approach gave further substantial improvements and the best performance on all datasets 

ground_truth: Table TABREF6, Table TABREF8

when testing on English, the F1 score of the model training on Chinese (Zh) is 53.8,  F1 score is only 44.1 for the model training on Zh-En