prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they test their framework performance on commonly used language pairs, such as English-to-German?
Context 1: Introduction
Neural Machine Translation (NMT) has shown its effectiveness in translation tasks when NMT systems perform best in recent machine translation campaigns BIBREF0 , BIBREF1   Compared to phrase-based Statistical Machine Translation (SMT) which is basically an ensemble of different features trained and tuned separately, NMT directly modeling the translation relationship between source and target sentences  Unlike SMT, NMT does not require much linguistic information and large monolingual data to achieve good performances 
An NMT consists of an encoder which recursively reads and represents the whole source sentence into a context vector and a recurrent decoder which takes the context vector and its previous state to predict the next target word  It is then trained in an end-to-end fashion to learn parameters which maximizes the likelihood between the outputs and the references  Recently, attention-based NMT has been featured in most state-of-the-art systems  First introduced by BIBREF2 , attention mechanism is integrated in decoder side as feedforward layers  It allows the NMT to decide which source words should take part in the predicting process of the next target words  It helps to improve NMTs significantly  Nevertheless, since the attention mechanism is specific to a particular source sentence and the considering target word, it is also specific to particular language pairs 
Some recent work has focused on extending the NMT framework to multilingual scenarios  By training such network using parallel corpora in number of different languages, NMT could benefit from additional information embedded in a common semantic space across languages  Basically, the proposed NMT are required to employ multiple encoders or multiple decoders to deal with multilinguality  Furthermore, in order to avoid the tight dependency of the attention mechanism to specific language pairs, they also need to modify their architecture to combine either the encoders or the attention layers  These modifications are specific to the purpose of the tasks as well  Thus, those multilingual NMTs are more complicated, much more free parameters to learn and more difficult to perform standard trainings compared to the original NMT 
In this paper, we introduce a unified approach to seamlessly extend the original NMT to multilingual settings  Our approach allows us to integrate any language in any side of the encoder-decoder architecture with only one encoder and one decoder for all the languages involved  Moreover, it is not necessary to do any network modification to enable attention mechanism in our NMT systems  We then apply our proprosed framework in two demanding scenarios: under-resourced translation and zero-resourced translation  The results show that bringing multilinguality to NMT helps to improve individual translations  With some insightful analyses of the results, we set our goal toward a fully multilingual NMT framework 
The paper starts with a detailed introduction to attention-based NMT  In Section SECREF3 , related work about multi-task NMT is reviewed  Section SECREF5 describes our proposed approach and thorough comparisons to the related work  It is followed by a section of evaluating our systems in two aforementioned scenarios, in which different strategies have been employed under a unified approach (Section SECREF4 )  Finally, the paper ends with conclusion and future work 
This work is licenced under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/
Neural Machine Translation: Background
An NMT system consists of an encoder which automatically learns the characteristics of a source sentence into fix-length context vectors and a decoder that recursively combines the produced context vectors with the previous target word to generate the most probable word from a target vocabulary 
More specifically, a bidirectional recurrent encoder reads every words INLINEFORM0 of a source sentence INLINEFORM1 and encodes a representation INLINEFORM2 of the sentence into a fixed-length vector INLINEFORM3 concatinated from those of the forward and backward directions: INLINEFORM4
Here INLINEFORM0 is the one-hot vector of the word INLINEFORM1 and INLINEFORM2 is the word embedding matrix which is shared across the source words  INLINEFORM3 is the recurrent unit computing the current hidden state of the encoder based on the previous hidden state  INLINEFORM4 is then called an annotation vector, which encodes the source sentence up to the time INLINEFORM5 from both forward and backward directions  Recurrent units in NMT can be a simple recurrent neural network unit (RNN), a Long Short-Term Memory unit (LSTM) BIBREF3 or a Gated Recurrent Unit (GRU) BIBREF4
Similar to the encoder, the recurrent decoder generates one target word INLINEFORM0 to form a translated target sentence INLINEFORM1 in the end  At the time INLINEFORM2 , it takes the previous hidden state of the decoder INLINEFORM3 , the previous embedded word representation INLINEFORM4 and a time-specific context vector INLINEFORM5 as inputs to calculate the current hidden state INLINEFORM6 : INLINEFORM7
Again, INLINEFORM0 is the recurrent activation function of the decoder and INLINEFORM1 is the shared word embedding matrix of the target sentences  The context vector INLINEFORM2 is calculated based on the annotation vectors from the encoder  Before feeding the annotation vectors into the decoder, an attention mechanism is set up in between, in order to choose which annotation vectors should contribute to the predicting decision of the next target word  Intuitively, a relevance between the previous target word and the annotation vectors can be used to form some attention scenario  There exists several ways to calculate the relevance as shown in BIBREF5 , but what we describe here follows the proposed method of BIBREF2 DISPLAYFORM0
In BIBREF2 , this attention mechanism, originally called alignment model, has been employed as a simple feedforward network with the first layer is a learnable layer via INLINEFORM0 , INLINEFORM1 and INLINEFORM2   The relevance scores INLINEFORM3 are then normalized into attention weights INLINEFORM4 and the context vector INLINEFORM5 is calculated as the weighted sum of all annotation vectors INLINEFORM6   Depending on how much attention the target word at time INLINEFORM7 put on the source states INLINEFORM8 , a soft alignment is learned  By being employed this way, word alignment is not a latent variable but a parametrized function, making the alignment model differentiable  Thus, it could be trained together with the whole architecture using backpropagation 
One of the most severe problems of NMT is handling of the rare words, which are not in the short lists of the vocabularies, i e  out-of-vocabulary (OOV) words, or do not appear in the training set at all  In BIBREF6 , the rare target words are copied from their aligned source words after the translation  This heuristic works well with OOV words and named entities but unable to translate unseen words  In BIBREF7 , their proposed NMT models have been shown to not only be effective on reducing vocabulary sizes but also have the ability to generate unseen words  This is achieved by segmenting the rare words into subword units and translating them  The state-of-the-art translation systems essentially employ subword NMT BIBREF7  
Universal Encoder and Decoder for Multilingual Neural Machine Translation
While the majority of previous research has focused on improving the performance of NMT on individual language pairs with individual NMT systems, recent work has started investigating potential ways to conduct the translation involved in multiple languages using a single NMT system  The possible reason explaining these efforts lies on the unique architecture of NMT  Unlike SMT, NMT consists of separated neural networks for the source and target sides, or the encoder and decoder, respectively  This allows these components to map a sentence in any language to a representation in an embedding space which is believed to share common semantics among the source languages involved  From that shared space, the decoder, with some implicit or explicit relevant constraints, could transform the representation into a concrete sentence in any desired language  In this section, we review some related work on this matter  We then describe a unified approach toward an universal attention-based NMT scheme  Our approach does not require any architecture modification and it can be trained to learn a minimal number of parameters compared to the other work 
Related Work
By extending the solution of sequence-to-sequence modeling using encoder-decoder architectures to multi-task learning, Luong2016 managed to achieve better performance on some INLINEFORM0 tasks such as translation, parsing and image captioning compared to individual tasks  Specifically in translation, the work utilizes multiple encoders to translate from multiple languages, and multiple decoders to translate to multiple languages  In this view of multilingual translation, each language in source or target side is modeled by one encoder or decoder, depending on the side of the translation  Due to the natural diversity between two tasks in that multi-task learning scenario, e g  translation and parsing, it could not feature the attention mechanism although it has proven its effectiveness in NMT  There exists two directions which proposed for multilingual translation scenarios where they leverage the attention mechanism  The first one is indicated in the work from BIBREF8 , where it introduce an one-to-many multilingual NMT system to translates from one source language into multiple target languages  Having one source language, the attention mechanism is then handed over to the corresponding decoder  The objective function is changed to adapt to multilingual settings  In testing time, the parameters specific to a desired language pair are used to perform the translation 
Firat2016 proposed another approach which genuinely delivers attention-based NMT to multilingual translation  As in BIBREF9 , their approach utilizes one encoder per source language and one decoder per target language for many-to-many translation tasks  Instead of a quadratic number of independent attention layers, however, one single attention mechanism is integrated into their NMT, performing an affine transformation between the hidden layer of INLINEFORM0 source languages and that one of INLINEFORM1 target languages  It is required to change their architecture to accomodate such a complicated shared attention mechanism 
In a separate effort to achieve multilingual NMT, the work of Zoph2016 leverages available parallel data from other language pairs to help reducing possible ambiguities in the translation process into a single target language  They employed the multi-source attention-based NMT in a way that only one attention mechanism is required despite having multiple encoders  To achieve this, the outputs of the encoders were combined before feeding to the attention layer  They implemented two types of encoder combination; One is adding a non-linear layer on the concatenation of the encoders' hidden states  The other is using a variant of LSTM taking the respective gate values from the individual LSTM units of the encoders  As a result, the combined hidden states contain information from both encoders , thus encode the common semantic of the two source languages 
Universal Encoder and Decoder
Inspired by the multi-source NMT as additional parallel data in several languages are expected to benefit single translations, we aim to develop a NMT-based approach toward an universal framework to perform multilingual translation  Our solution features two treatments: 1) Coding the words in different languages as different words in the language-mixed vocabularies and 2) Forcing the NMT to translating a representation of source sentences into the sentences in a desired target language 
Language-specific Coding  When the encoder of a NMT system considers words across languages as different words, with a well-chosen architecture, it is expected to be able to learn a good representation of the source words in an embedding space in which words carrying similar meaning would have a closer distance to each others than those are semantically different  This should hold true when the words have the same or similar surface form, such as (@de@Obama; @en@Obama) or (@de@Projektion; @en@projection)  This should also hold true when the words have the same or similar meaning across languages, such as (@en@car; @en@automobile) or (@de@Flussufer; @en@bank)  Our encoder then acts similarly to the one of multi-source approach BIBREF10 , collecting additional information from other sources for better translations, but with a much simpler embedding function  Unlike them, we need only one encoder, so we could reduce the number of parameters to learn  Furthermore, we neither need to change the network architecture nor depend on which recurrent unit (GRU, LSTM or simple RNN) is currently using in the encoder 
We could apply the same trick to the target sentences and thus enable many-to-many translation capability of our NMT system  Similar to the multi-target translation BIBREF8 , we exploit further the correlation in semantics of those target sentences across different languages  The main difference between our approach and the work of BIBREF8 is that we need only one decoder for all target languages  Given one encoder for multiple source languages and one decoder for multiple target languages, it is trivial to incorporate the attention mechanism as in the case of a regular NMT for single language translation  In training, the attention layers were directed to learn relevant alignments between words in specific language pair and forward the produced context vector to the decoder  Now we rely totally on the network to learn good alignments between source and target sides  In fact, giving more information, our system are able to form nice alignments 
In comparison to other research that could perform complete multi-task learning, e g  the work from BIBREF9 or the approach proposed by BIBREF11 , our method is able to accommodate the attention layers seemlessly and easily  It also draws a clear distinction from those works in term of the complexity of the whole network: considerably less parameters to learn, thus reduces overfitting, with a conventional attention mechanism and a standard training procedure 
Target Forcing  While language-specific coding allows us to implement a multilingual attention-based NMT, there are two issues we have to consider before training the network  The first is that the number of rare words would increase in proportion with the number of languages involved  This might be solved by applying a rare word treatment method with appropriate awareness of the vocabularies' size  The second one is more problematic: Ambiguity level in the translation process definitely increases due to the additional introduction of words having the same or similar meaning across languages at both source and target sides  We deal with the problem by explicitly forcing the attention and translation to the direction that we prefer, expecting the information would limit the ambiguity to the scope of one language instead of all target languages  We realize this idea by adding at the beginning and at the end of every source sentences a special symbol indicating the language they would be translated into  For example, in a multilingual NMT, when a source sentence is German and the target language is English, the original sentence (already language-specific coded) is:
@de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag
Now when we force it to be translated into English, the target-forced sentence becomes:
<E> @de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag <E>
Due to the nature of recurrent units used in the encoder and decoder, in training, those starting symbols encourage the network learning the translation of following target words in a particular language pair  In testing time, information of the target language we provided help to limit the translated candidates, hence forming the translation in the desired language 
Figure FIGREF8 illustrates the essence of our approach  With two steps in the preprocessing phase, namely language-specific coding and target forcing, we are able to employ multilingual attention-based NMT without any special treatment in training such a standard architecture  Our encoder and attention-enable decoder can be seen as a shared encoder and decoder across languages, or an universal encoder and decoder  The flexibitily of our approach allow us to integrate any language into source or target side  As we will see in Section SECREF4 , it has proven to be extremely helpful not only in low-resourced scenarios but also in translation of well-resourced language pairs as it provides a novel way to make use of large monolingual corpora in NMT 
Evaluation
In this section, we describe the evaluation of our proposed approach in comparisons with the strong baselines using NMT in two scenarios: the translation of an under-resource language pair and the translation of a language pair that does not exist any paralled data at all 
Experimental Settings
Training Data  We choose WIT3's TED corpus BIBREF12 as the basis of our experiments since it might be the only high-quality parallel data of many low-resourced language pairs  TED is also multilingual in a sense that it includes numbers of talks which are commonly translated into many languages  In addition, we use a much larger corpus provided freely by WMT organizers when we evaluate the impact of our approach in a real machine translation campaign  It includes the paralled corpus extracted from the digital corpus of European Parliament (EPPS), the News Commentary (NC) and the web-crawled parallel data (CommonCrawl)  While the number of sentences in popular TED corpora varies from 13 thousands to 17 thousands, the total number of sentences in those larger corpus is approximately 3 million sentences 
Neural Machine Translation Setup  All experiments have been conducted using NMT framework Nematus, Following the work of Sennrich2016a, subword segmentation is handled in the prepocessing phase using Byte-Pair Encoding (BPE)  Excepts stated clearly in some experiments, we set the number of BPE merging operations at 39500 on the joint of source and target data  When training all NMT systems, we take out the sentence pairs exceeding 50-word length and shuffle them inside every minibatch  Our short-list vocabularies contain 40,000 most frequent words while the others are considered as rare words and applied the subword translation  We use an 1024-cell GRU layer and 1000-dimensional embeddings with dropout at every layer with the probability of 0 2 in the embedding and hidden layers and 0 1 in the input and ourput layers  We trained our systems using gradient descent optimization with Adadelta BIBREF13 on minibatches of size 80 and the gradient is rescaled whenever its norm exceed 1 0  All the trainings last approximately seven days if the early-stopping condition could not be reached  At a certain time, an external evaluation script on BLEU BIBREF14 is conducted on a development set to decide the early-stopping condition  This evaluation script has also being used to choose the model archiving the best BLEU on the development set instead of the maximal loglikelihood between the translations and target sentences while training  In translation, the framework produces INLINEFORM0 -best candidates and we then use a beam search with the beam size of 12 to get the best translation 
Under-resourced Translation
First, we consider the translation for an under-resourced pair of languages  Here a small portion of the large parallel corpus for English-German is used as a simulation for the scenario where we do not have much parallel data: Translating texts in English to German  We perform language-specific coding in both source and target sides  By accommodating the German monolingual data as an additional input (German INLINEFORM0 German), which we called the mix-source approach, we could enrich the training data in a simple, natural way  Given this under-resourced situation, it could help our NMT obtain a better representation of the source side, hence, able to learn the translation relationship better  Including monolingual data in this way might also improve the translation of some rare word types such as named entities  Furthermore, as the ultimate goal of our work, we would like to investigate the advantages of multilinguality in NMT  We incorporate a similar portion of French-German parallel corpus into the English-German one  As discussed in Section SECREF5 , it is expected to help reducing the ambiguity in translation between one language pair since it utilizes the semantic context provided by the other source language  We name this mix-multi-source 
Table TABREF16 summarizes the performance of our systems measured in BLEU on two test sets, tst2013 and tst2014  Compared to the baseline NMT system which is solely trained on TED English-German data, our mix-source system achieves a considerable improvement of 2 6 BLEU points on tst2013 and 2 1 BLEU points on and tst2014   Adding French data to the source side and their corresponding German data to the target side in our mix-multi-source system also help to gain 2 2 and 1 6 BLEU points more on tst2013 tst2014, respectively  We observe a better improvement from our mix-source system compared to our mix-multi-source system  We speculate the reason that the mix-source encoder utilize the same information shared in two languages while the mix-multi-source receives and processes similar information in the other language but not necessarily the same  We might validate this hypothesis by comparing two systems trained on a common English-German-French corpus of TED  We put it in our future work's plan 
As we expected Figure FIGREF19 shows how different words in different languages can be close in the shared space after being learned to translate into a common language  We extract the word embeddings from the encoder of the mix-multi-source (En,Fr INLINEFORM0 De,De) after training, remove the language-specific codes (@en@ and @fr@)and project the word vectors to the 2D space using t-SNE BIBREF15  
Using large monolingual data in NMT 
A standard NMT system employs parallel data only  While good parallel corpora are limited in number, getting monolingual data of an arbitrary language is trivial  To make use of German monolingual corpus in an English INLINEFORM0 German NMT system, sennrich2016b built a separate German INLINEFORM1 English NMT using the same parallel corpus, then they used that system to translate the German monolingual corpus back to English, forming a synthesis parallel data  gulcehre2015 trained another RNN-based language model to score the monolingual corpus and integrate it to the NMT system through shallow or deep fusion  Both methods requires to train separate systems with possibly different hyperparameters for each  Conversely, by applying mix-source method to the big monolingual data, we need to train only one network  We mix the TED parallel corpus and the substantial monolingual corpus (EPPS+NC+CommonCrawl) and train a mix-source NMT system from those data 
The first result is not encouraging when its performance is even worse than the baseline NMT which is trained on the small parallel data only  Not using the same information in the source side, as we discussed in case of mix-multi-source strategy, could explain the degrading in performance of such a system  But we believe that the magnitude and unbalancing of the corpus are the main reasons  The data contains nearly four millions sentences but only around twenty thousands of them (0 5%) are the genuine parallel data  As a quick attempt, after we get the model with that big data, we continue training on the real parallel corpus for some more epochs  When this adaptation is applied, our system brings an improvement of +1 52 BLEU on tst2013 and +1 06 BLEU on tst2014 (Table TABREF21 ) 
Zero-resourced Translation
Among low-resourced scenarios, zero-resourced translation task stands in an extreme level  A zero-resourced translation task is one of the most difficult situation when there is no parallel data between the translating language pair  To the best of our knowledge, there have been yet existed a published work about using NMT for zero-resourced translation tasks up to now  In this section, we extend our strategies using the proposed multilingual NMT approach as first attempts to this extreme situation 
We employ language-specific coding and target forcing in a strategy called bridge  Unlike the strategies used in under-resourced translation task, bridge is an entire many-to-many multilingual NMT  Simulating a zero-resourced German INLINEFORM0 French translation task given the available German-English and English-French parallel corpora, after applying language-specific coding and target forcing for each corpus, we mix those data with an English-English data as a “bridge” creating some connection between German and French  We also propose a variant of this strategy that we incorporate French-French data  And we call it universal 
We evaluate bridge and universal systems on two German INLINEFORM0 French test sets  They are compared to a direct system, which is an NMT trained on German INLINEFORM1 French data, and to a pivot system, which essentially consists of two separate NMTs trained to translate from German to English and English to French  The direct system should not exist in a real zero-resourced situation  We refer it as the perfect system for comparison purpose only  In case of the pivot system, to generate a translated text in French from a German sentence, we first translate it to English, then the output sentence is fed to the English INLINEFORM2 German NMT system to obtain the French translation  Since there are more than two languages involved in those systems, we increase the number of BPE merging operations proportionally in order to reduce the number of rare words in such systems  We do not expect our proposed systems to perform well with this primitive way of building direct translating connections since this is essentially a difficult task  We report the performance of those systems in Table TABREF23  
Unsupprisingly, both bridge and universal systems perform worse than the pivot one  We consider two possible reasons:
Our target forcing mechanism is moderately primitive  Since the process is applied after language-specific coding, the target forcing symbol is the same for all source sentences in every languages  Thus, the forcing strength might not be enough to guide the decision of the next words  Once the very first word is translated into a word in wrong language, the following words tend to be translated into that wrong language again  Table TABREF24 shows some statistics of the translated words and sentences in wrong language 
Balancing of the training corpus  Although it is not severe as in the case of mix-source system for large monolingual data, the limited number of sentences in target language can affect the training  The difference of 1 07 BLEU points between bridge and universal might explain this assumption as we added more target data (French) in universal strategy, thus reducing the unbalance in training 
Those issues would be addressed in our following future work toward the multilingual attention-based NMT 
Conclusion and Future Work
In this paper, we present our first attempts in building a multilingual Neural Machine Translation framework  By treating words in different languages as different words and force the attention and translation to the direction of desired target language, we are able to employ attention-enable NMT toward a multilingual translation system  Our proposed approach alleviates the need of complicated architecture re-designing when accommodating attention mechanism  In addition, the number of free parameters to learn in our network does not go beyond that magnitute of a single NMT system  With its universality, our approach has shown its effectiveness in an under-resourced translation task with considerable improvements  In addition, the approach has achieved interesting and promising results when applied in the translation task that there is no direct parallel corpus between source and target languages 
Nevertheless, there are issues that we can continue working on to address in future work  A more balancing data would be helpful for this framework  The mechanism of forcing the NMT system to the right target language could be improved  We could conduct more detailed analyses of the various strategies under the framework to show its universarity 
Context 3: Introduction
Natural languages evolve and words have always been subject to semantic change over time BIBREF1  With the rise of large digitized text resources recent NLP technologies have made it possible to capture such change with vector space models BIBREF2, BIBREF3, BIBREF4, BIBREF5, topic models BIBREF6, BIBREF7, BIBREF8, and sense clustering models BIBREF9  However, many approaches for detecting LSC differ profoundly from each other and therefore drawing comparisons between them can be challenging BIBREF10  Not only do architectures for detecting LSC vary, their performance is also often evaluated without access to evaluation data or too sparse data sets  In cases where evaluation data is available, oftentimes LSCD systems are not evaluated on the same data set which hinders the research community to draw comparisons 
For this reason we report the results of the first shared task on unsupervised lexical semantic change detection in German that is based on an annotated data set to guarantee objective reasoning throughout different approaches  The task was organized as part of the seminar 'Lexical Semantic Change Detection' at the IMS Stuttgart in the summer term of 2019 
Task
The goal of the shared task was to create an architecture to detect semantic change and to rank words according to their degree of change between two different time periods  Given two corpora Ca and Cb, the target words had to be ranked according to their degree of lexical semantic change between Ca and Cb as annotated by human judges  A competition was set up on Codalab and teams mostly consisting of 2 people were formed to take part in the task  There was one group consisting of 3 team members and two individuals who entered the task on their own  In total there were 12 LSCD systems participating in the shared task 
The shared task was divided into three phases, i e , development, testing and analysis phase  In the development phase each team implemented a first version of their model based on a trial data set and submitted it subsequently  In the testing phase the testing data was made public and participants applied their models to the test data with a restriction of possible result uploads to 30  The leaderboard was public at all times  Eventually, the analysis phase was entered and the models of the testing phase were evaluated in terms of the predictions they made and parameters could be tuned further  The models and results will be discussed in detail in sections 7 and 8 
Corpora
The task, as framed above, requires to detect the semantic change between two corpora  The two corpora used in the shared task correspond to the diachronic corpus pair from BIBREF0: DTA18 and DTA19  They consist of subparts of DTA corpus BIBREF11 which is a freely available lemmatized, POS-tagged and spelling-normalized diachronic corpus of German containing texts from the 16th to the 20th century  DTA18 contains 26 million sentences published between 1750-1799 and DTA19 40 million between 1850-1899  The corpus version used in the task has the following format: "year [tab] lemma1 lemma2 lemma3    " 
Evaluation
The Diachronic Usage Relatedness (DURel) gold standard data set includes 22 target words and their varying degrees of semantic change BIBREF12  For each of these target words a random sample of use pairs from the DTA corpus was retrieved and annotated  The annotators were required to rate the pairs according to their semantic relatedness on a scale from 1 to 4 (unrelated - identical meanings) for two time periods  The average Spearman's $\rho $ between the five annotators was 0 66 for 1,320 use paris

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does Gaussian-masked directional multi-head attention works?
Context 2: Introduction
Chinese word segmentation (CWS) is a task for Chinese natural language process to delimit word boundary  CWS is a basic and essential task for Chinese which is written without explicit word delimiters and different from alphabetical languages like English  BIBREF0 treats Chinese word segmentation (CWS) as a sequence labeling task with character position tags, which is followed by BIBREF1, BIBREF2, BIBREF3  Traditional CWS models depend on the design of features heavily which effects the performance of model  To minimize the effort in feature engineering, some CWS models BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11 are developed following neural network architecture for sequence labeling tasks BIBREF12  Neural CWS models perform strong ability of feature representation, employing unigram and bigram character embedding as input and approach good performance 
The CWS task is often modeled as one graph model based on a scoring model that means it is composed of two parts, one part is an encoder which is used to generate the representation of characters from the input sequence, the other part is a decoder which performs segmentation according to the encoder scoring  Table TABREF1 summarizes typical CWS models according to their decoding ways for both traditional and neural models  Markov models such as BIBREF13 and BIBREF4 depend on the maximum entropy model or maximum entropy Markov model both with a Viterbi decoder  Besides, conditional random field (CRF) or Semi-CRF for sequence labeling has been used for both traditional and neural models though with different representations BIBREF2, BIBREF15, BIBREF10, BIBREF17, BIBREF18  Generally speaking, the major difference between traditional and neural network models is about the way to represent input sentences 
Recent works about neural CWS which focus on benchmark dataset, namely SIGHAN Bakeoff BIBREF21, may be put into the following three categories roughly 
Encoder  Practice in various natural language processing tasks has been shown that effective representation is essential to the performance improvement  Thus for better CWS, it is crucial to encode the input character, word or sentence into effective representation  Table TABREF2 summarizes regular feature sets for typical CWS models including ours as well  The building blocks that encoders use include recurrent neural network (RNN) and convolutional neural network (CNN), and long-term memory network (LSTM) 
Graph model  As CWS is a kind of structure learning task, the graph model determines which type of decoder should be adopted for segmentation, also it may limit the capability of defining feature, as shown in Table 2, not all graph models can support the word features  Thus recent work focused on finding more general or flexible graph model to make model learn the representation of segmentation more effective as BIBREF9, BIBREF11 
External data and pre-trained embedding  Whereas both encoder and graph model are about exploring a way to get better performance only by improving the model strength itself  Using external resource such as pre-trained embeddings or language representation is an alternative for the same purpose BIBREF22, BIBREF23  SIGHAN Bakeoff defines two types of evaluation settings, closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation BIBREF21  In this work, we will focus on the closed test setting by finding a better model design for further CWS performance improvement 
Shown in Table TABREF1, different decoders have particular decoding algorithms to match the respective CWS models  Markov models and CRF-based models often use Viterbi decoders with polynomial time complexity  In general graph model, search space may be too large for model to search  Thus it forces graph models to use an approximate beam search strategy  Beam search algorithm has a kind low-order polynomial time complexity  Especially, when beam width $b$=1, the beam search algorithm will reduce to greedy algorithm with a better time complexity $O(Mn)$ against the general beam search time complexity $O(Mnb^2)$, where $n$ is the number of units in one sentences, $M$ is a constant representing the model complexity  Greedy decoding algorithm can bring the fastest speed of decoding while it is not easy to guarantee the precision of decoding when the encoder is not strong enough 
In this paper, we focus on more effective encoder design which is capable of offering fast and accurate Chinese word segmentation with only unigram feature and greedy decoding  Our proposed encoder will only consist of attention mechanisms as building blocks but nothing else  Motivated by the Transformer BIBREF24 and its strength of capturing long-range dependencies of input sentences, we use a self-attention network to generate the representation of input which makes the model encode sentences at once without feeding input iteratively  Considering the weakness of the Transformer to model relative and absolute position information directly BIBREF25 and the importance of localness information, position information and directional information for CWS, we further improve the architecture of standard multi-head self-attention of the Transformer with a directional Gaussian mask and get a variant called Gaussian-masked directional multi-head attention  Based on the newly improved attention mechanism, we expand the encoder of the Transformer to capture different directional information  With our powerful encoder, our model uses only simple unigram features to generate representation of sentences 
For decoder which directly performs the segmentation, we use the bi-affinal attention scorer, which has been used in dependency parsing BIBREF26 and semantic role labeling BIBREF27, to implement greedy decoding on finding the boundaries of words  In our proposed model, greedy decoding ensures a fast segmentation while powerful encoder design ensures a good enough segmentation performance even working with greedy decoder together  Our model will be strictly evaluated on benchmark datasets from SIGHAN Bakeoff shared task on CWS in terms of closed test setting, and the experimental results show that our proposed model achieves new state-of-the-art 
The technical contributions of this paper can be summarized as follows 
We propose a CWS model with only attention structure  The encoder and decoder are both based on attention structure 
With a powerful enough encoder, we for the first time show that unigram (character) featues can help yield strong performance instead of diverse $n$-gram (character and word) features in most of previous work 
To capture the representation of localness information and directional information, we propose a variant of directional multi-head self-attention to further enhance the state-of-the-art Transformer encoder 
Models
The CWS task is often modelled as one graph model based on an encoder-based scoring model  The model for CWS task is composed of an encoder to represent the input and a decoder based on the encoder to perform actual segmentation  Figure FIGREF6 is the architecture of our model  The model feeds sentence into encoder  Embedding captures the vector $e=(e_1,   ,e_n)$ of the input character sequences of $c=(c_1,   ,c_n)$  The encoder maps vector sequences of $ {e}=(e_1,  ,e_n)$ to two sequences of vector which are $ {v^b}=(v_1^b,   ,v_n^b)$ and ${v^f}=(v_1^f,   v_n^f)$ as the representation of sentences  With $v^b$ and $v^f$, the bi-affinal scorer calculates the probability of each segmentation gaps and predicts the word boundaries of input  Similar as the Transformer, the encoder is an attention network with stacked self-attention and point-wise, fully connected layers while our encoder includes three independent directional encoders 
Models ::: Encoder Stacks
In the Transformer, the encoder is composed of a stack of N identical layers and each layer has one multi-head self-attention layer and one position-wise fully connected feed-forward layer  One residual connection is around two sub-layers and followed by layer normalization BIBREF24  This architecture provides the Transformer a good ability to generate representation of sentence 
With the variant of multi-head self-attention, we design a Gaussian-masked directional encoder to capture representation of different directions to improve the ability of capturing the localness information and position information for the importance of adjacent characters  One unidirectional encoder can capture information of one particular direction 
For CWS tasks, one gap of characters, which is from a word boundary, can divide one sequence into two parts, one part in front of the gap and one part in the rear of it  The forward encoder and backward encoder are used to capture information of two directions which correspond to two parts divided by the gap 
One central encoder is paralleled with forward and backward encoders to capture the information of entire sentences  The central encoder is a special directional encoder for forward and backward information of sentences  The central encoder can fuse the information and enable the encoder to capture the global information 
The encoder outputs one forward information and one backward information of each positions  The representation of sentence generated by center encoder will be added to these information directly:
where $v^{b}=(v^b_1,   ,v^b_n)$ is the backward information, $v^{f}=(v^f_1,   ,v^f_n)$ is the forward information, $r^{b}=(r^b_1,   ,r^b_n)$ is the output of backward encoder, $r^{c}=(r^c_1,   ,r^c_n)$ is the output of center encoder and $r^{f}=(r^f_1,   ,r^f_n)$ is the output of forward encoder 
Models ::: Gaussian-Masked Directional Multi-Head Attention
Similar as scaled dot-product attention BIBREF24, Gaussian-masked directional attention can be described as a function to map queries and key-value pairs to the representation of input  Here queries, keys and values are all vectors  Standard scaled dot-product attention is calculated by dotting query $Q$ with all keys $K$, dividing each values by $\sqrt{d_k}$, where $\sqrt{d_k}$ is the dimension of keys, and apply a softmax function to generate the weights in the attention:
Different from scaled dot-product attention, Gaussian-masked directional attention expects to pay attention to the adjacent characters of each positions and cast the localness relationship between characters as a fix Gaussian weight for attention  We assume that the Gaussian weight only relys on the distance between characters 
Firstly we introduce the Gaussian weight matrix $G$ which presents the localness relationship between each two characters:
where $g_{ij}$ is the Gaussian weight between character $i$ and $j$, $dis_{ij}$ is the distance between character $i$ and $j$, $\Phi (x)$ is the cumulative distribution function of Gaussian, $\sigma $ is the standard deviation of Gaussian function and it is a hyperparameter in our method  Equation (DISPLAY_FORM13) can ensure the Gaussian weight equals 1 when $dis_{ij}$ is 0  The larger distance between charactersis, the smaller the weight is, which makes one character can affect its adjacent characters more compared with other characters 
To combine the Gaussian weight to the self-attention, we produce the Hadamard product of Gaussian weight matrix $G$ and the score matrix produced by $Q{K^{T}}$
where $AG$ is the Gaussian-masked attention  It ensures that the relationship between two characters with long distances is weaker than adjacent characters 
The scaled dot-product attention models the relationship between two characters without regard to their distances in one sequence  For CWS task, the weight between adjacent characters should be more important while it is hard for self-attention to achieve the effect explicitly because the self-attention cannot get the order of sentences directly  The Gaussian-masked attention adjusts the weight between characters and their adjacent character to a larger value which stands for the effect of adjacent characters 
For forward and backward encoder, the self-attention sublayer needs to use a triangular matrix mask to let the self-attention focus on different weights:
where $pos_i$ is the position of character $c_i$  The triangular matrix for forward and backward encode are:
$\left[ \begin{matrix} 1 & 0 & 0 & \cdots &0\\ 1 & 1 & 0 & \cdots &0\\ 1 & 1 & 1 & \cdots &0\\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 1 & 1 & 1 & \cdots & 1\\ \end{matrix} \right]$ $\left[ \begin{matrix} 1 & 1 & 1 & \cdots &1 \\ 0 & 1 & 1 & \cdots &1 \\ 0 & 0& 1 & \cdots &1 \\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 0 & 0 & 0 & \cdots & 1\\ \end{matrix}\right]$
Similar as BIBREF24, we use multi-head attention to capture information from different dimension positions as Figure FIGREF16 and get Gaussian-masked directional multi-head attention  With multi-head attention architecture, the representation of input can be captured by
where $MH$ is the Gaussian-masked multi-head attention, ${W_i^q, W_i^k,W_i^v} \in \mathbb {R}^{d_k \times d_h}$ is the parameter matrices to generate heads, $d_k$ is the dimension of model and $d_h$ is the dimension of one head 
Models ::: Bi-affinal Attention Scorer
Regarding word boundaries as gaps between any adjacent words converts the character labeling task to the gap labeling task  Different from character labeling task, gap labeling task requires information of two adjacent characters  The relationship between adjacent characters can be represented as the type of gap  The characteristic of word boundaries makes bi-affine attention an appropriate scorer for CWS task 
Bi-affinal attention scorer is the component that we use to label the gap  Bi-affinal attention is developed from bilinear attention which has been used in dependency parsing BIBREF26 and SRL BIBREF27  The distribution of labels in a labeling task is often uneven which makes the output layer often include a fixed bias term for the prior probability of different labels BIBREF27  Bi-affine attention uses bias terms to alleviate the burden of the fixed bias term and get the prior probability which makes it different from bilinear attention  The distribution of the gap is uneven that is similar as other labeling task which fits bi-affine 
Bi-affinal attention scorer labels the target depending on information of independent unit and the joint information of two units  In bi-affinal attention, the score $s_{ij}$ of characters $c_i$ and $c_j$ $(i < j)$ is calculated by:
where $v_i^f$ is the forward information of $c_i$ and $v_i^b$ is the backward information of $c_j$  In Equation (DISPLAY_FORM21), $W$, $U$ and $b$ are all parameters that can be updated in training  $W$ is a matrix with shape $(d_i \times N\times d_j)$ and $U$ is a $(N\times (d_i + d_j))$ matrix where $d_i$ is the dimension of vector $v_i^f$ and $N$ is the number of labels 
In our model, the biaffine scorer uses the forward information of character in front of the gap and the backward information of the character behind the gap to distinguish the position of characters  Figure FIGREF22 is an example of labeling gap  The method of using biaffine scorer ensures that the boundaries of words can be determined by adjacent characters with different directional information  The score vector of the gap is formed by the probability of being a boundary of word  Further, the model generates all boundaries using activation function in a greedy decoding way 
Experiments ::: Experimental Settings ::: Data
We train and evaluate our model on datasets from SIGHAN Bakeoff 2005 BIBREF21 which has four datasets, PKU, MSR, AS and CITYU  Table TABREF23 shows the statistics of train data  We use F-score to evaluate CWS models  To train model with pre-trained embeddings in AS and CITYU, we use OpenCC to transfer data from traditional Chinese to simplified Chinese 
Experiments ::: Experimental Settings ::: Pre-trained Embedding
We only use unigram feature so we only trained character embeddings  Our pre-trained embedding are pre-trained on Chinese Wikipedia corpus by word2vec BIBREF29 toolkit  The corpus used for pre-trained embedding is all transferred to simplified Chinese and not segmented  On closed test, we use embeddings initialized randomly 
Experiments ::: Experimental Settings ::: Hyperparameters
For different datasets, we use two kinds of hyperparameters which are presented in Table TABREF24  We use hyperparameters in Table TABREF24 for small corpora (PKU and CITYU) and normal corpora (MSR and AS)  We set the standard deviation of Gaussian function in Equation (DISPLAY_FORM13) to 2  Each training batch contains sentences with at most 4096 tokens 
Experiments ::: Experimental Settings ::: Optimizer
To train our model, we use the Adam BIBREF30 optimizer with $\beta _1=0 9$, $\beta _2=0 98$ and $\epsilon =10^{-9}$  The learning rate schedule is the same as BIBREF24:
where $d$ is the dimension of embeddings, $step$ is the step number of training and $warmup_step$ is the step number of warmup  When the number of steps is smaller than the step of warmup, the learning rate increases linearly and then decreases 
Experiments ::: Hardware and Implements
We trained our models on a single CPU (Intel i7-5960X) with an nVidia 1080 Ti GPU  We implement our model in Python with Pytorch 1 0 
Experiments ::: Results
Tables TABREF25 and TABREF26 reports the performance of recent models and ours in terms of closed test setting  Without the assistance of unsupervised segmentation features userd in BIBREF20, our model outperforms all the other models in MSR and AS except BIBREF18 and get comparable performance in PKU and CITYU  Note that all the other models for this comparison adopt various $n$-gram features while only our model takes unigram ones 
With unsupervised segmentation features introduced by BIBREF20, our model gets a higher result  Specially, the results in MSR and AS achieve new state-of-the-art and approaching previous state-of-the-art in CITYU and PKU  The unsupervised segmentation features are derived from the given training dataset, thus using them does not violate the rule of closed test of SIGHAN Bakeoff 
Table TABREF36 compares our model and recent neural models in terms of open test setting in which any external resources, especially pre-trained embeddings or language models can be used  In MSR and AS, our model gets a comparable result while our results in CITYU and PKU are not remarkable 
However, it is well known that it is always hard to compare models when using open test setting, especially with pre-trained embedding  Not all models may use the same method and data to pre-train  Though pre-trained embedding or language model can improve the performance, the performance improvement itself may be from multiple sources  It often that there is a success of pre-trained embedding to improve the performance, while it cannot prove that the model is better 
Compared with other LSTM models, our model performs better in AS and MSR than in CITYU and PKU  Considering the scale of different corpora, we believe that the size of corpus affects our model and the larger size is, the better model performs  For small corpus, the model tends to be overfitting 
Tables TABREF25 and TABREF26 also show the decoding time in different datasets  Our model finishes the segmentation with the least decoding time in all four datasets, thanks to the architecture of model which only takes attention mechanism as basic block 
Related Work ::: Chinese Word Segmentation
CWS is a task for Chinese natural language process to delimit word boundary  BIBREF0 for the first time formulize CWS as a sequence labeling task  BIBREF3 show that different character tag sets can make essential impact for CWS  BIBREF2 use CRFs as a model for CWS, achieving new state-of-the-art  Works of statistical CWS has built the basis for neural CWS 
Neural word segmentation has been widely used to minimize the efforts in feature engineering which was important in statistical CWS  BIBREF4 introduce the neural model with sliding-window based sequence labeling  BIBREF6 propose a gated recursive neural network (GRNN) for CWS to incorporate complicated combination of contextual character and n-gram features  BIBREF7 use LSTM to learn long distance information  BIBREF9 propose a neural framework that eliminates context windows and utilize complete segmentation history  BIBREF33 explore a joint model that performs segmentation, POS-Tagging and chunking simultaneously  BIBREF34 propose a feature-enriched neural model for joint CWS and part-of-speech tagging  BIBREF35 present a joint model to enhance the segmentation of Chinese microtext by performing CWS and informal word detection simultaneously  BIBREF17 propose a character-based convolutional neural model to capture $n$-gram features automatically and an effective approach to incorporate word embeddings  BIBREF11 improve the model in BIBREF9 and propose a greedy neural word segmenter with balanced word and character embedding inputs  BIBREF23 propose a novel neural network model to incorporate unlabeled and partially-labeled data  BIBREF36 propose two methods that extend the Bi-LSTM to perform incorporating dictionaries into neural networks for CWS  BIBREF37 propose Switch-LSTMs to segment words and provided a more flexible solution for multi-criteria CWS which is easy to transfer the learned knowledge to new criteria 
Related Work ::: Transformer
Transformer BIBREF24 is an attention-based neural machine translation model  The Transformer is one kind of self-attention networks (SANs) which is proposed in BIBREF38  Encoder of the Transformer consists of one self-attention layer and a position-wise feed-forward layer  Decoder of the Transformer contains one self-attention layer, one encoder-decoder attention layer and one position-wise feed-forward layer  The Transformer uses residual connections around the sublayers and then followed by a layer normalization layer 
Scaled dot-product attention is the key component in the Transformer  The input of attention contains queries, keys, and values of input sequences  The attention is generated using queries and keys like Equation (DISPLAY_FORM11)  Structure of scaled dot-product attention allows the self-attention layer generate the representation of sentences at once and contain the information of the sentence which is different from RNN that process characters of sentences one by one  Standard self-attention is similar as Gaussian-masked direction attention while it does not have directional mask and gaussian mask  BIBREF24 also propose multi-head attention which is better to generate representation of sentence by dividing queries, keys and values to different heads and get information from different subspaces 
Conclusion
In this paper, we propose an attention mechanism only based Chinese word segmentation model  Our model uses self-attention from the Transformer encoder to take sequence input and bi-affine attention scorer to predict the label of gaps  To improve the ability of capturing the localness and directional information of self-attention based encoder, we propose a variant of self-attention called Gaussian-masked directional multi-head attention to replace the standard self-attention  We also extend the Transformer encoder to capture directional features  Our model uses only unigram features instead of multiple $n$-gram features in previous work  Our model is evaluated on standard benchmark dataset, SIGHAN Bakeoff 2005, which shows not only our model performs segmentation faster than any previous models but also gives new higher or comparable segmentation performance against previous state-of-the-art models 
Context 3:  The resulting word ranking of the DURel data set is determined by the mean usage relatedness across two time periods and is used as the benchmark to compare the models’ performances in the shared task 
Evaluation ::: Metric
The output of a system with the target words in the predicted order is compared to the gold ranking of the DURel data set  As the metric to assess how well the model's output fits the gold ranking Spearman's $\rho $ was used  The higher Spearman's rank-order correlation the better the system's performance 
Evaluation ::: Baselines
Models were compared to two baselines for the shared task:
log-transformed normalized frequency difference (FD)
count vectors with column intersection and cosine distance (CNT + CI + CD)
The window size for CNT + CI + CD was 10  Find more information on these models in BIBREF0 
Participating Systems
Participants mostly rely on the models compared in BIBREF0 and apply modifications to improve them  In particular, most teams make use of skip-gram with negative sampling (SGNS) based on BIBREF13 to learn the semantic spaces of the two time periods and orthogonal procrustes (OP) to align these vector spaces, similar to the approach by BIBREF14  Different meaning representations such as sense clusters are used as well  As measure to detect the degree of LSC all teams except one choose cosine distance (CD)  This team uses Jensen-Shannon distance (JSD) instead, which computes the distance between probability distributions BIBREF15 
The models of each team will be briefly introduced in this section 
Participating Systems ::: sorensbn
Team sorensbn makes use of SGNS + OP + CD to detect LSC  They use similar hyperparameters as in BIBREF0 to tune the SGNS model  They use an open-sourced noise-aware implementation to improve the OP alignment BIBREF16 
Participating Systems ::: tidoe
Team tidoe builds on SGNS + OP + CD, but they add a transformation step to receive binarized representations of matrices BIBREF17  This step is taken to counter the bias that can occur in vector-space models based on frequencies BIBREF18 
Participating Systems ::: in vain
The team applies a model based on SGNS with vector initialization alignment and cosine distance (SGNS + VI + CD)  Vector initialization is an alignment strategy where the vector space learning model for $t_2$ is initialized with the vectors from $t_1$ BIBREF19  Since SGNS + VI + OP does not perform as well as other models in BIBREF0, they alter the vector initialization process by initializing on the complete model instead of only the word matrix of $t_1$ to receive improved results 
Participating Systems ::: Evilly
In line with previous approaches, team Evilly builds upon SGNS + OP + CD  They alter the OP step by using only high-frequency words for alignment 
Participating Systems ::: DAF
Team DAF uses an architecture based on learning vectors with fastText, alignment with unsupervised and supervised variations of OP, and CD, using the MUSE package BIBREF20, BIBREF21  For the supervised alignment stop words are used  The underlying assumption is that stop words serve as functional units of language and their usage should be consistent over time 
Participating Systems ::: SnakesOnAPlane
The team learns vector spaces with count vectors, positive pointwise mutual information (PPMI), SGNS and uses column intersection (CI) and OP as alignment techniques where applicable  Then they compare two distance measures (CD and JSD) for the different models CNT + CI, PPMI + CI and SGNS + OP to identify which measure performs better for these models  They also experiment with different ways to remove negative values from SGNS vectors, which is needed for JSD 
Participating Systems ::: TeamKulkarni15
TeamKulkarni15 uses SGNS + OP + CD with the modification of local alignment with k nearest neighbors, since other models often use global alignment that can be prone to noise BIBREF22 
Participating Systems ::: Bashmaistori
They use word injection (WI) alignment on PPMI vectors with CD  This approach avoids the complex alignment procedure for embeddings and is applicable to embeddings and count-based methods  They compare two implementations of word injection BIBREF23, BIBREF0 as these showed different results on different data sets 
Participating Systems ::: giki
Team giki uses PPMI + CI + CD to detect LSC

ground_truth: pays attentions to adjacent characters and casts a localness relationship between the characters as a fixed Gaussian weight assuming the weight relies on the distance between characters

Gaussian-masked directional attention can be described as a function to map queries and key-value pairs to the representation of input, Gaussian-masked directional attention expects to pay attention to the adjacent characters of each positions and cast the localness relationship between characters as a fix Gaussian weight for attention, Gaussian weight only relys on the distance between characters

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the corpus used for the task?
Context 3:  They state that a word sense is determined by its context, but relevant context words can also be found outside a predefined window  Therefore, they use tf-idf to select relevant context BIBREF24 
Participating Systems ::: Edu-Phil
Similar to team DAF they also use fastText + OP + CD  Their hypothesis is that fastText may increase the performance for less frequent words in the corpus since generating word embeddings in fasttext is based on character n-grams 
Participating Systems ::: orangefoxes
They use the model by BIBREF5 which is based on SGNS, but avoids alignment by treating time as a vector that may be combined with word vectors to get time-specific word vectors 
Participating Systems ::: Loud Whisper
Loud Whisper base their approach on BIBREF9 which is a graph-based sense clustering model  They process the data set to receive bigrams, create a co-occurence graph representation and after clustering assess the type of change per word by comparing the results against an intersection table  Their motivation is not only to use a graph-based approach, but to extend the approach by enabling change detection for all parts of speech as opposed to the original model 
Results and Discussion
Table TABREF8 shows the results of the shared task  All teams receive better results than baseline 1 (FD), of which a total of 8 teams outperform baseline 2 (CNT + CI + CD)  The 4 top scores with $\rho $ $>$ 0 7 are either modified versions of SGNS + OP + CD or use SGNS + VI + CD  The following 4 scores in the range of 0 5 $<$ $\rho $ $<$ 0 6 are generated by the models fastText + OP + CD, SGNS + OP + CD/JSD, and PPMI + WI + CD 
Contrary to the results by BIBREF0 the modified version of vector initialization shows high performance similar to OP alignment, as previously reported by BIBREF14  Some modifications to the SGNS + OP + CD approach are able to yield better results than others, e g  noise-aware alignment and binarized matrices as compared to frequency-driven OP alignment or local alignment with KNN  Team SnakesOnAPlane compare two distance measures and their results show that JSD ($\rho $ $=$  561) performs minimally worse than CD ($\rho $ $=$  565) as the semantic change measure for their model 
The overall best-performing model is Skip-Gram with orthogonal alignment and cosine distance (SGNS + OP + CD) with similar hyperparameters as in the model architecture described previously BIBREF0  Said architecture was used as the basis for the two best performing models  Team tidoe reports that binarizing matrices leads to a generally worse performance ($\rho $ $=$  811) compared to the unmodified version of SGNS + OP + CD ($\rho $ $=$ 0 9)  The noise aware alignment approach applied by team sorensbn obtains a higher score ($\rho $ $=$  854) compared to the result reported by tidoe, but is unable to exceed the performance of the unmodified SNGS + OP + CD for the same set of hyperparameters (window size = 10, negative sampling = 1; subsampling = None)  Of the 8 scores above the second baseline, 5 use an architecture that builds upon SGNS + OP + CD  Whereas in the lower score segment $\rho $ $<$ 0 5 none of the models use SGNS + OP + CD  These findings are in line with the results reported by BIBREF0, however the overall best results are lower in this shared task, which is expected from the smaller number of parameter combinations explored  Additionally, in the shared task the objective was to report the best score and not to calculate the mean which makes it more difficult to compare the robustness of the models presented here 

ground_truth: DTA18, DTA19

Diachronic Usage Relatedness (DURel) gold standard data set