prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what previous RNN models do they compare with?
Context 1: Introduction
Long short term memory (LSTM) units BIBREF1 are popular for many sequence modeling tasks and are used extensively in language modeling  A key to their success is their articulated gating structure, which allows for more control over the information passed along the recurrence  However, despite the sophistication of the gating mechanisms employed in LSTMs and similar recurrent units, the input and context vectors are treated with simple linear transformations prior to gating  Non-linear transformations such as convolutions BIBREF2 have been used, but these have not achieved the performance of well regularized LSTMs for language modeling BIBREF3  
A natural way to improve the expressiveness of linear transformations is to increase the number of dimensions of the input and context vectors, but this comes with a significant increase in the number of parameters which may limit generalizability  An example is shown in Figure FIGREF1 , where LSTMs performance decreases with the increase in dimensions of the input and context vectors  Moreover, the semantics of the input and context vectors are different, suggesting that each may benefit from specialized treatment 
Guided by these insights, we introduce a new recurrent unit, the Pyramidal Recurrent Unit (PRU), which is based on the LSTM gating structure  Figure FIGREF2 provides an overview of the PRU  At the heart of the PRU is the pyramidal transformation (PT), which uses subsampling to effect multiple views of the input vector  The subsampled representations are combined in a pyramidal fusion structure, resulting in richer interactions between the individual dimensions of the input vector than is possible with a linear transformation  Context vectors, which have already undergone this transformation in the previous cell, are modified with a grouped linear transformation (GLT) which allows the network to learn latent representations in high dimensional space with fewer parameters and better generalizability (see Figure FIGREF1 ) 
We show that PRUs can better model contextual information and demonstrate performance gains on the task of language modeling  The PRU improves the perplexity of the current state-of-the-art language model BIBREF0 by up to 1 3 points, reaching perplexities of 56 56 and 64 53 on the Penn Treebank and WikiText2 datasets while learning 15-20% fewer parameters  Replacing an LSTM with a PRU results in improvements in perplexity across a variety of experimental settings  We provide detailed ablations which motivate the design of the PRU architecture, as well as detailed analysis of the effect of the PRU on other components of the language model 
Related work
Multiple methods, including a variety of gating structures and transformations, have been proposed to improve the performance of recurrent neural networks (RNNs)  We first describe these approaches and then provide an overview of recent work in language modeling 
Pyramidal Recurrent Units
We introduce Pyramidal Recurrent Units (PRUs), a new RNN architecture which improves modeling of context by allowing for higher dimensional vector representations while learning fewer parameters  Figure FIGREF2 provides an overview of PRU  We first elaborate on the details of the pyramidal transformation and the grouped linear transformation  We then describe our recurrent unit, PRU 
Pyramidal transformation for input
The basic transformation in many recurrent units is a linear transformation INLINEFORM0 defined as: DISPLAYFORM0
where INLINEFORM0 are learned weights that linearly map INLINEFORM1 to INLINEFORM2   To simplify notation, we omit the biases 
Motivated by successful applications of sub-sampling in computer vision (e g , BIBREF22 , BIBREF23 , BIBREF9 , BIBREF24 ), we subsample input vector INLINEFORM0 into INLINEFORM1 pyramidal levels to achieve representation of the input vector at multiple scales  This sub-sampling operation produces INLINEFORM2 vectors, represented as INLINEFORM3 , where INLINEFORM4 is the sampling rate and INLINEFORM5   We learn scale-specific transformations INLINEFORM6 for each INLINEFORM7   The transformed subsamples are concatenated to produce the pyramidal analog to INLINEFORM8 , here denoted as INLINEFORM9 : DISPLAYFORM0
where INLINEFORM0 indicates concatenation  We note that pyramidal transformation with INLINEFORM1 is the same as the linear transformation 
To improve gradient flow inside the recurrent unit, we combine the input and output using an element-wise sum (when dimension matches) to produce residual analog of pyramidal transformation, as shown in Figure FIGREF2 BIBREF25  
We sub-sample the input vector INLINEFORM0 into INLINEFORM1 pyramidal levels using the kernel-based approach BIBREF8 , BIBREF9   Let us assume that we have a kernel INLINEFORM2 with INLINEFORM3 elements  Then, the input vector INLINEFORM4 can be sub-sampled as: DISPLAYFORM0
where INLINEFORM0 represents the stride and INLINEFORM1  
The number of parameters learned by the linear transformation and the pyramidal transformation with INLINEFORM0 pyramidal levels to map INLINEFORM1 to INLINEFORM2 are INLINEFORM3 and INLINEFORM4 respectively  Thus, pyramidal transformation reduces the parameters of a linear transformation by a factor of INLINEFORM5   For example, the pyramidal transformation (with INLINEFORM6 and INLINEFORM7 ) learns INLINEFORM8 fewer parameters than the linear transformation 
Grouped linear transformation for context
Many RNN architectures apply linear transformations to both the input and context vector  However, this may not be ideal due to the differing semantics of each vector  In many NLP applications including language modeling, the input vector is a dense word embedding which is shared across all contexts for a given word in a dataset  In contrast, the context vector is highly contextualized by the current sequence  The differences between the input and context vector motivate their separate treatment in the PRU architecture 
The weights learned using the linear transformation (Eq  EQREF9 ) are reused over multiple time steps, which makes them prone to over-fitting BIBREF26   To combat over-fitting, various methods, such as variational dropout BIBREF26 and weight dropout BIBREF0 , have been proposed to regularize these recurrent connections  To further improve generalization abilities while simultaneously enabling the recurrent unit to learn representations at very high dimensional space, we propose to use grouped linear transformation (GLT) instead of standard linear transformation for recurrent connections BIBREF27   While pyramidal and linear transformations can be applied to transform context vectors, our experimental results in Section SECREF39 suggests that GLTs are more effective 
The linear transformation INLINEFORM0 maps INLINEFORM1 linearly to INLINEFORM2   Grouped linear transformations break the linear interactions by factoring the linear transformation into two steps  First, a GLT splits the input vector INLINEFORM3 into INLINEFORM4 smaller groups such that INLINEFORM5   Second, a linear transformation INLINEFORM6 is applied to map INLINEFORM7 linearly to INLINEFORM8 , for each INLINEFORM9   The INLINEFORM10 resultant output vectors INLINEFORM11 are concatenated to produce the final output vector INLINEFORM12   DISPLAYFORM0
GLTs learn representations at low dimensionality  Therefore, a GLT requires INLINEFORM0 fewer parameters than the linear transformation  We note that GLTs are subset of linear transformations  In a linear transformation, each neuron receives an input from each element in the input vector while in a GLT, each neuron receives an input from a subset of the input vector  Therefore, GLT is the same as a linear transformation when INLINEFORM1  
Pyramidal Recurrent Unit
We extend the basic gating architecture of LSTM with the pyramidal and grouped linear transformations outlined above to produce the Pyramidal Recurrent Unit (PRU), whose improved sequence modeling capacity is evidenced in Section SECREF4  
At time INLINEFORM0 , the PRU combines the input vector INLINEFORM1 and the previous context vector (or previous hidden state vector) INLINEFORM2 using the following transformation function as: DISPLAYFORM0
where INLINEFORM0 indexes the various gates in the LSTM model, and INLINEFORM1 and INLINEFORM2 represents the pyramidal and grouped linear transformations defined in Eqns  EQREF10 and EQREF15 , respectively 
We will now incorporate INLINEFORM0 into LSTM gating architecture to produce PRU  At time INLINEFORM1 , a PRU cell takes INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 as inputs to produce forget INLINEFORM5 , input INLINEFORM6 , output INLINEFORM7 , and content INLINEFORM8 gate signals  The inputs are combined with these gate signals to produce context vector INLINEFORM9 and cell state INLINEFORM10   Mathematically, the PRU with the LSTM gating architecture can be defined as: DISPLAYFORM0
where INLINEFORM0 represents the element-wise multiplication operation, and INLINEFORM1 and INLINEFORM2 are the sigmoid and hyperbolic tangent activation functions  We note that LSTM is a special case of PRU when INLINEFORM3 = INLINEFORM4 =1 
Experiments
To showcase the effectiveness of the PRU, we evaluate the performance on two standard datasets for word-level language modeling and compare with state-of-the-art methods  Additionally, we provide a detailed examination of the PRU and its behavior on the language modeling tasks 
Set-up
Following recent works, we compare on two widely used datasets, the Penn Treebank (PTB) BIBREF28 as prepared by BIBREF29 and WikiText2 (WT-2) BIBREF20   For both datasets, we follow the same training, validation, and test splits as in BIBREF0  
We extend the language model, AWD-LSTM BIBREF0 , by replacing LSTM layers with PRU  Our model uses 3-layers of PRU with an embedding size of 400  The number of parameters learned by state-of-the-art methods vary from 18M to 66M with majority of the methods learning about 22M to 24M parameters on the PTB dataset  For a fair comparison with state-of-the-art methods, we fix the model size to 19M and vary the value of INLINEFORM0 and hidden layer sizes so that total number of learned parameters is similar across different configurations  We use 1000, 1200, and 1400 as hidden layer sizes for values of INLINEFORM1 =1,2, and 4, respectively  We use the same settings for the WT-2 dataset  We set the number of pyramidal levels INLINEFORM2 to two in our experiments and use average pooling for sub-sampling  These values are selected based on our ablation experiments on the validation set (Section SECREF39 )  We measure the performance of our models in terms of word-level perplexity  We follow the same training strategy as in BIBREF0  
To understand the effect of regularization methods on the performance of PRUs, we perform experiments under two different settings: (1) Standard dropout: We use a standard dropout BIBREF12 with probability of 0 5 after embedding layer, the output between LSTM layers, and the output of final LSTM layer  (2) Advanced dropout: We use the same dropout techniques with the same dropout values as in BIBREF0   We call this model as AWD-PRU 
Results
Table TABREF23 compares the performance of the PRU with state-of-the-art methods  We can see that the PRU achieves the best performance with fewer parameters 
PRUs achieve either the same or better performance than LSTMs  In particular, the performance of PRUs improves with the increasing value of INLINEFORM0   At INLINEFORM1 , PRUs outperform LSTMs by about 4 points on the PTB dataset and by about 3 points on the WT-2 dataset  This is explained in part by the regularization effect of the grouped linear transformation (Figure FIGREF1 )  With grouped linear and pyramidal transformations, PRUs learn rich representations at very high dimensional space while learning fewer parameters  On the other hand, LSTMs overfit to the training data at such high dimensions and learn INLINEFORM2 to INLINEFORM3 more parameters than PRUs 
With the advanced dropouts, the performance of PRUs improves by about 4 points on the PTB dataset and 7 points on the WT-2 dataset  This further improves with finetuning on the PTB (about 2 points) and WT-2 (about 1 point) datasets 
For similar number of parameters, the PRU with standard dropout outperforms most of the state-of-the-art methods by large margin on the PTB dataset (e g  RAN BIBREF7 by 16 points with 4M less parameters, QRNN BIBREF33 by 16 points with 1M more parameters, and NAS BIBREF31 by 1 58 points with 6M less parameters)  With advanced dropouts, the PRU delivers the best performance  On both datasets, the PRU improves the perplexity by about 1 point while learning 15-20% fewer parameters 
PRU is a drop-in replacement for LSTM, therefore, it can improve language models with modern inference techniques such as dynamic evaluation BIBREF21   When we evaluate PRU-based language models (only with standard dropout) with dynamic evaluation on the PTB test set, the perplexity of PRU ( INLINEFORM0 ) improves from 62 42 to 55 23 while the perplexity of an LSTM ( INLINEFORM1 ) with similar settings improves from 66 29 to 58 79; suggesting that modern inference techniques are equally applicable to PRU-based language models 
Analysis
It is shown above that the PRU can learn representations at higher dimensionality with more generalization power, resulting in performance gains for language modeling  A closer analysis of the impact of the PRU in a language modeling system reveals several factors that help explain how the PRU achieves these gains 
As exemplified in Table TABREF34 , the PRU tends toward more confident decisions, placing more of the probability mass on the top next-word prediction than the LSTM  To quantify this effect, we calculate the entropy of the next-token distribution for both the PRU and the LSTM using 3687 contexts from the PTB validation set  Figure FIGREF32 shows a histogram of the entropies of the distribution, where bins of size 0 23 are used to effect categories  We see that the PRU more often produces lower entropy distributions corresponding to higher confidences for next-token choices  This is evidenced by the mass of the red PRU curve lying in the lower entropy ranges compared to the blue LSTM's curve  The PRU can produce confident decisions in part because more information is encoded in the higher dimensional context vectors 
The PRU has the ability to model individual words at different resolutions through the pyramidal transform; which provides multiple paths for the gradient to the embedding layer (similar to multi-task learning) and improves the flow of information  When considering the embeddings by part of speech, we find that the pyramid level 1 embeddings exhibit higher variance than the LSTM across all POS categories (Figure FIGREF33 ), and that pyramid level 2 embeddings show extremely low variance  We hypothesize that the LSTM must encode both coarse group similarities and individual word differences into the same vector space, reducing the space between individual words of the same category  The PRU can rely on the subsampled embeddings to account for coarse-grained group similarities, allowing for finer individual word distinctions in the embedding layer  This hypothesis is strengthened by the entropy results described above: a model which can make finer distinctions between individual words can more confidently assign probability mass  A model that cannot make these distinctions, such as the LSTM, must spread its probability mass across a larger class of similar words 
Saliency analysis using gradients help identify relevant words in a test sequence that contribute to the prediction BIBREF34 , BIBREF35 , BIBREF36   These approaches compute the relevance as the squared norm of the gradients obtained through back-propagation  Table TABREF34 visualizes the heatmaps for different sequences  PRUs, in general, give more relevance to contextual words than LSTMs, such as southeast (sample 1), cost (sample 2), face (sample 4), and introduced (sample 5), which help in making more confident decisions  Furthermore, when gradients during back-propagation are visualized BIBREF37 (Table TABREF34 ), we find that PRUs have better gradient coverage than LSTMs, suggesting PRUs use more features than LSTMs that contributes to the decision  This also suggests that PRUs update more parameters at each iteration which results in faster training  Language model in BIBREF0 takes 500 and 750 epochs to converge with PRU and LSTM as a recurrent unit, respectively 
Ablation studies
In this section, we provide a systematic analysis of our design choices  Our training methodology is the same as described in Section SECREF19 with the standard dropouts  For a thorough understanding of our design choices, we use a language model with a single layer of PRU and fix the size of embedding and hidden layers to 600  The word-level perplexities are reported on the validation sets of the PTB and the WT-2 datasets 
The two hyper-parameters that control the trade-off between performance and number of parameters in PRUs are the number of pyramidal levels INLINEFORM0 and groups INLINEFORM1   Figure FIGREF35 provides a trade-off between perplexity and recurrent unit (RU) parameters 
Variable INLINEFORM0 and fixed INLINEFORM1 : When we increase the number of pyramidal levels INLINEFORM2 at a fixed value of INLINEFORM3 , the performance of the PRU drops by about 1 to 4 points while reducing the total number of recurrent unit parameters by up to 15%  We note that the PRU with INLINEFORM4 at INLINEFORM5 delivers similar performance as the LSTM while learning about 15% fewer recurrent unit parameters 
Fixed INLINEFORM0 and variable INLINEFORM1 : When we vary the value of INLINEFORM2 at fixed number of pyramidal levels INLINEFORM3 , the total number of recurrent unit parameters decreases significantly with a minimal impact on the perplexity  For example, PRUs with INLINEFORM4 and INLINEFORM5 learns 77% fewer recurrent unit parameters while its perplexity (lower is better) increases by about 12% in comparison to LSTMs  Moreover, the decrease in number of parameters at higher value of INLINEFORM6 enables PRUs to learn the representations in high dimensional space with better generalizability (Table TABREF23 ) 
Table TABREF43 shows the impact of different transformations of the input vector INLINEFORM0 and the context vector INLINEFORM1   We make following observations: (1) Using the pyramidal transformation for the input vectors improves the perplexity by about 1 point on both the PTB and WT-2 datasets while reducing the number of recurrent unit parameters by about 14% (see R1 and R4)  We note that the performance of the PRU drops by up to 1 point when residual connections are not used (R4 and R6)  (2) Using the grouped linear transformation for context vectors reduces the total number of recurrent unit parameters by about 75% while the performance drops by about 11% (see R3 and R4)  When we use the pyramidal transformation instead of the linear transformation, the performance drops by up to 2% while there is no significant drop in the number of parameters (R4 and R5) 
We set sub-sampling kernel INLINEFORM0 (Eq  EQREF12 ) with stride INLINEFORM1 and size of 3 ( INLINEFORM2 ) in four different ways: (1) Skip: We skip every other element in the input vector  (2) Convolution: We initialize the elements of INLINEFORM3 randomly from normal distribution and learn them during training the model  We limit the output values between -1 and 1 using INLINEFORM4 activation function to make training stable  (3) Avg  pool: We initialize the elements of INLINEFORM5 to INLINEFORM6   (4) Max pool: We select the maximum value in the kernel window INLINEFORM7  
Table TABREF45 compares the performance of the PRU with different sampling methods  Average pooling performs the best while skipping give comparable performance  Both of these methods enable the network to learn richer word representations while representing the input vector in different forms, thus delivering higher performance  Surprisingly, a convolution-based sub-sampling method does not perform as well as the averaging method  The INLINEFORM0 function used after convolution limits the range of output values which are further limited by the LSTM gating structure, thereby impeding in the flow of information inside the cell  Max pooling forces the network to learn representations from high magnitude elements, thus distinguishing features between elements vanishes, resulting in poor performance 
Conclusion
We introduce the Pyramidal Recurrent Unit, which better model contextual information by admitting higher dimensional representations with good generalizability  When applied to the task of language modeling, PRUs improve perplexity across several settings, including recent state-of-the-art systems  Our analysis shows that the PRU improves the flow of gradient and expand the word embedding subspace, resulting in more confident decisions  Here we have shown improvements for language modeling  In future, we plan to study the performance of PRUs on different tasks, including machine translation and question answering  In addition, we will study the performance of the PRU on language modeling with more recent inference techniques, such as dynamic evaluation and mixture of softmax 
Acknowledgments
This research was supported by NSF (IIS 1616112, III 1703166), Allen Distinguished Investigator Award, and gifts from Allen Institute for AI, Google, Amazon, and Bloomberg  We are grateful to Aaron Jaech, Hannah Rashkin, Mandar Joshi, Aniruddha Kembhavi, and anonymous reviewers for their helpful comments 
Context 2: Introduction
Stance detection (also called stance identification or stance classification) is one of the considerably recent research topics in natural language processing (NLP)  It is usually defined as a classification problem where for a text and target pair, the stance of the author of the text for that target is expected as a classification output from the set: {Favor, Against, Neither} BIBREF0  
Stance detection is usually considered as a subtask of sentiment analysis (opinion mining) BIBREF1 topic in NLP  Both are mostly performed on social media texts, particularly on tweets, hence both are important components of social media analysis  Nevertheless, in sentiment analysis, the sentiment of the author of a piece of text usually as Positive, Negative, and Neutral is explored while in stance detection, the stance of the author of the text for a particular target (an entity, event, etc ) either explicitly or implicitly referred to in the text is considered  Like sentiment analysis, stance detection systems can be valuable components of information retrieval and other text analysis systems BIBREF0  
Previous work on stance detection include BIBREF2 where a stance classifier based on sentiment and arguing features is proposed in addition to an arguing lexicon automatically compiled  The ultimate approach performs better than distribution-based and uni-gram-based baseline systems BIBREF2   In BIBREF3 , the authors show that the use of dialogue structure improves stance detection in on-line debates  In BIBREF4 , Hasan and Ng carry out stance detection experiments using different machine learning algorithms, training data sets, features, and inter-post constraints in on-line debates, and draw insightful conclusions based on these experiments  For instance, they find that sequence models like HMMs perform better at stance detection when compared with non-sequence models like Naive Bayes (NB) BIBREF4   In another related study BIBREF5 , the authors conclude that topic-independent features can be exploited for disagreement detection in on-line dialogues  The employed features include agreement, cue words, denial, hedges, duration, polarity, and punctuation BIBREF5   Stance detection on a corpus of student essays is considered in BIBREF6   After using linguistically-motivated feature sets together with multivalued NB and SVM as the learning models, the authors conclude that they outperform two baseline approaches BIBREF6   In BIBREF7 , the author claims that Wikipedia can be used to determine stances about controversial topics based on their previous work regarding controversy extraction on the Web 
Among more recent related work, in BIBREF8 stance detection for unseen targets is studied and bidirectional conditional encoding is employed  The authors state that their approach achieves state-of-the art performance rates BIBREF8 on SemEval 2016 Twitter Stance Detection corpus BIBREF0   In BIBREF9 , a stance-community detection approach called SCIFNET is proposed  SCIFNET creates networks of people who are stance targets, automatically from the related document collections BIBREF9 using stance expansion and refinement techniques to arrive at stance-coherent networks  A tweet data set annotated with stance information regarding six predefined targets is proposed in BIBREF10 where this data set is annotated through crowdsourcing  The authors indicate that the data set is also annotated with sentiment information in addition to stance, so it can help reveal associations between stance and sentiment BIBREF10   Lastly, in BIBREF0 , SemEval 2016's aforementioned shared task on Twitter Stance Detection is described  Also provided are the results of the evaluations of 19 systems participating in two subtasks (one with training data set provided and the other without an annotated data set) of the shared task BIBREF0  
In this paper, we present a tweet data set in Turkish annotated with stance information, where the corresponding annotations are made publicly available  The domain of the tweets comprises two popular football clubs which constitute the targets of the tweets included  We also provide the evaluation results of SVM classifiers (for each target) on this data set using unigram, bigram, and hashtag features 
To the best of our knowledge, the current study is the first one to target at stance detection in Turkish tweets  Together with the provided annotated data set and the corresponding evaluations with the aforementioned SVM classifiers which can be used as baseline systems, our study will hopefully help increase social media analysis studies on Turkish content 
The rest of the paper is organized as follows: In Section SECREF2 , we describe our tweet data set annotated with the target and stance information  Section SECREF3 includes the details of our SVM-based stance classifiers and their evaluation results with discussions  Section SECREF4 includes future research topics based on the current study, and finally Section SECREF5 concludes the paper with a summary 
A Stance Detection Data Set
We have decided to consider tweets about popular sports clubs as our domain for stance detection  Considerable amounts of tweets are being published for sports-related events at every instant  Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey  As is the case for the sentiment analysis tools, the outputs of the stance detection systems on a stream of tweets about these clubs can facilitate the use of the opinions of the football followers by these clubs 
In a previous study on the identification of public health-related tweets, two tweet data sets in Turkish (each set containing 1 million random tweets) have been compiled where these sets belong to two different periods of 20 consecutive days BIBREF11   We have decided to use one of these sets (corresponding to the period between August 18 and September 6, 2015) and firstly filtered the tweets using the possible names used to refer to the target clubs  Then, we have annotated the stance information in the tweets for these targets as Favor or Against  Within the course of this study, we have not considered those tweets in which the target is not explicitly mentioned, as our initial filtering process reveals 
For the purposes of the current study, we have not annotated any tweets with the Neither class  This stance class and even finer-grained classes can be considered in further annotation studies  We should also note that in a few tweets, the target of the stance was the management of the club while in some others a particular footballer of the club is praised or criticised  Still, we have considered the club as the target of the stance in all of the cases and carried out our annotations accordingly 
At the end of the annotation process, we have annotated 700 tweets, where 175 tweets are in favor of and 175 tweets are against Target-1, and similarly 175 tweets are in favor of and 175 are against Target-2  Hence, our data set is a balanced one although it is currently limited in size  The corresponding stance annotations are made publicly available at http://ceng

ground_truth: Variational LSTM, CharCNN, Pointer Sentinel-LSTM, RHN, NAS Cell, SRU, QRNN, RAN, 4-layer skip-connection LSTM, AWD-LSTM, Quantized LSTM

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which sports clubs are the targets?
Context 2: metu edu tr/ INLINEFORM0 e120329/ Turkish_Stance_Detection_Tweet_Dataset csv in Comma Separated Values (CSV) format  The file contains three columns with the corresponding headers  The first column is the tweet id of the corresponding tweet, the second column contains the name of the stance target, and the last column includes the stance of the tweet for the target as Favor or Against 
To the best of our knowledge, this is the first publicly-available stance-annotated data set for Turkish  Hence, it is a significant resource as there is a scarcity of annotated data sets, linguistic resources, and NLP tools available for Turkish  Additionally, to the best of our knowledge, it is also significant for being the first stance-annotated data set including sports-related tweets, as previous stance detection data sets mostly include on-line texts on political/ethical issues 
Stance Detection Experiments Using SVM Classifiers
It is emphasized in the related literature that unigram-based methods are reliable for the stance detection task BIBREF2 and similarly unigram-based models have been used as baseline models in studies such as BIBREF0   In order to be used as a baseline and reference system for further studies on stance detection in Turkish tweets, we have trained two SVM classifiers (one for each target) using unigrams as features  Before the extraction of unigrams, we have employed automated preprocessing to filter out the stopwords in our annotated data set of 700 tweets  The stopword list used is the list presented in BIBREF12 which, in turn, is the slightly extended version of the stopword list provided in BIBREF13  
We have used the SVM implementation available in the Weka data mining application BIBREF14 where this particular implementation employs the SMO algorithm BIBREF15 to train a classifier with a linear kernel  The 10-fold cross-validation results of the two classifiers are provided in Table TABREF1 using the metrics of precision, recall, and F-Measure 
The evaluation results are quite favorable for both targets and particularly higher for Target-1, considering the fact that they are the initial experiments on the data set  The performance of the classifiers is better for the Favor class for both targets when compared with the performance results for the Against class  This outcome may be due to the common use of some terms when expressing positive stance towards sports clubs in Turkish tweets  The same percentage of common terms may not have been observed in tweets during the expression of negative stances towards the targets  Yet, completely the opposite pattern is observed in stance detection results of baseline systems given in BIBREF0 , i e , better F-Measure rates have been obtained for the Against class when compared with the Favor class BIBREF0   Some of the baseline systems reported in BIBREF0 are SVM-based systems using unigrams and ngrams as features similar to our study, but their data sets include all three stance classes of Favor, Against, and Neither, while our data set comprises only tweets classified as belonging to Favor or Against classes  Another difference is that the data sets in BIBREF0 have been divided into training and test sets, while in our study we provide 10-fold cross-validation results on the whole data set  On the other hand, we should also note that SVM-based sentiment analysis systems (such as those given in BIBREF16 ) have been reported to achieve better F-Measure rates for the Positive sentiment class when compared with the results obtained for the Negative class  Therefore, our evaluation results for each stance class seem to be in line with such sentiment analysis systems  Yet, further experiments on the extended versions of our data set should be conducted and the results should again be compared with the stance detection results given in the literature 
We have also evaluated SVM classifiers which use only bigrams as features, as ngram-based classifiers have been reported to perform better for the stance detection problem BIBREF0   However, we have observed that using bigrams as the sole features of the SVM classifiers leads to quite poor results  This observation may be due to the relatively limited size of the tweet data set employed  Still, we can conclude that unigram-based features lead to superior results compared to the results obtained using bigrams as features, based on our experiments on our data set  Yet, ngram-based features may be employed on the extended versions of the data set to verify this conclusion within the course of future work 
With an intention to exploit the contribution of hashtag use to stance detection, we have also used the existence of hashtags in tweets as an additional feature to unigrams  The corresponding evaluation results of the SVM classifiers using unigrams together the existence of hashtags as features are provided in Table TABREF2  
When the results given in Table TABREF2 are compared with the results in Table TABREF1 , a slight decrease in F-Measure (0 5%) for Target-1 is observed, while the overall F-Measure value for Target-2 has increased by 1 8%  Although we could not derive sound conclusions mainly due to the relatively small size of our data set, the increase in the performance of the SVM classifier Target-2 is an encouraging evidence for the exploitation of hashtags in a stance detection system  We leave other ways of exploiting hashtags for stance detection as a future work 
To sum up, our evaluation results are significant as reference results to be used for comparison purposes and provides evidence for the utility of unigram-based and hashtag-related features in SVM classifiers for the stance detection problem in Turkish tweets 
Future Prospects
Future work based on the current study includes the following:
Conclusion
Stance detection is a considerably new research area in natural language processing and is considered within the scope of the well-studied topic of sentiment analysis  It is the detection of stance within text towards a target which may be explicitly specified in the text or not  In this study, we present a stance-annotated tweet data set in Turkish where the targets of the annotated stances are two popular sports clubs in Turkey  The corresponding annotations are made publicly-available for research purposes  To the best of our knowledge, this is the first stance detection data set for the Turkish language and also the first sports-related stance-annotated data set  Also presented in this study are SVM classifiers (one for each target) utilizing unigram and bigram features in addition to using the existence of hashtags as another feature  10-fold cross validation results of these classifiers are presented which can be used as reference results by prospective systems  Both the annotated data set and the classifiers with evaluations are significant since they are the initial contributions to stance detection problem in Turkish tweets 
Context 3: Introduction and Background
Many reinforcement learning algorithms are designed for relatively small discrete or continuous action spaces and so have trouble scaling  Text-adventure games—or interaction fictions—are simulations in which both an agents' state and action spaces are in textual natural language  An example of a one turn agent interaction in the popular text-game Zork1 can be seen in Fig  FIGREF1  Text-adventure games provide us with multiple challenges in the form of partial observability, commonsense reasoning, and a combinatorially-sized state-action space  Text-adventure games are structured as long puzzles or quests, interspersed with bottlenecks  The quests can usually be completed through multiple branching paths  However, games can also feature one or more bottlenecks  Bottlenecks are areas that an agent must pass through in order to progress to the next section of the game regardless of what path the agent has taken to complete that section of the quest BIBREF0  In this work, we focus on more effectively exploring this space and surpassing these bottlenecks—building on prior work that focuses on tackling the other problems 
Formally, we use the definition of text-adventure games as seen in BIBREF1 and BIBREF2  These games are partially observable Markov decision processes (POMDPs), represented as a 7-tuple of $\langle S,T,A,\Omega , O,R, \gamma \rangle $ representing the set of environment states, mostly deterministic conditional transition probabilities between states, the vocabulary or words used to compose text commands, observations returned by the game, observation conditional probabilities, reward function, and the discount factor respectively  For our purposes, understanding the exact state and action spaces we use in this work is critical and so we define each of these in relative depth 
Action-Space  To solve Zork1, the cannonical text-adventure games, requires the generation of actions consisting of up to five-words from a relatively modest vocabulary of 697 words recognized by the game’s parser  This results in $\mathcal {O}(697^5)={1 64e14}$ possible actions at every step  To facilitate text-adventure game playing, BIBREF2 introduce Jericho, a framework for interacting with text-games  They propose a template-based action space in which the agent first selects a template, consisting of an action verb and preposition, and then filling that in with relevant entities $($e g  $[get]$ $ [from] $ $)$  Zork1 has 237 templates, each with up to two blanks, yielding a template-action space of size $\mathcal {O}(237 \times 697^2)={1 15e8}$  This space is still far larger than most used by previous approaches applying reinforcement learning to text-based games 
State-Representation  Prior work has shown that knowledge graphs are effective in terms of dealing with the challenges of partial observability $($BIBREF3 BIBREF3; BIBREF4$)$  A knowledge graph is a set of 3-tuples of the form $\langle subject, relation, object \rangle $  These triples are extracted from the observations using Stanford's Open Information Extraction (OpenIE) BIBREF5  Human-made text-adventure games often contain relatively complex semi-structured information that OpenIE is not designed to parse and so they add additional rules to ensure that the correct information is parsed  The graph itself is more or less a map of the world, with information about objects' affordances and attributes linked to the rooms that they are place in a map  The graph also makes a distinction with respect to items that are in the agent's possession or in their immediate surrounding environment  An example of what the knowledge graph looks like and specific implementation details can be found in Appendix SECREF14 
BIBREF6 introduce the KG-A2C, which uses a knowledge graph based state-representation to aid in the section of actions in a combinatorially-sized action-space—specifically they use the knowledge graph to constrain the kinds of entities that can be filled in the blanks in the template action-space  They test their approach on Zork1, showing the combination of the knowledge graph and template action selection resulted in improvements over existing methods  They note that their approach reaches a score of 40 which corresponds to a bottleneck in Zork1 where the player is eaten by a “grue” (resulting in negative reward) if the player has not first lit a lamp  The lamp must be lit many steps after first being encountered, in a different section of the game; this action is necessary to continue exploring but doesn’t immediately produce any positive reward  That is, there is a long term dependency between actions that is not immediately rewarded, as seen in Figure FIGREF1  Others using artificially constrained action spaces also report an inability to pass through this bottleneck BIBREF7, BIBREF8  They pose a significant challenge for these methods because the agent does not see the correct action sequence to pass the bottleneck enough times  This is in part due to the fact that for that sequence to be reinforced, the agent needs to reach the next possible reward beyond the bottleneck 
More efficient exploration strategies are required to pass bottlenecks  Our contributions are two-fold  We first introduce a method that detects bottlenecks in text-games using the overall reward gained and the knowledge graph state  This method freezes the policy used to reach the bottleneck and restarts the training from there on out, additionally conducting a backtracking search to ensure that a sub-optimal policy has not been frozen  The second contribution explore how to leverage knowledge graphs to improve existing exploration algorithms for dealing with combinatorial action-spaces such as Go-Explore BIBREF9  We additionally present a comparative ablation study analyzing the performance of these methods on the popular text-game Zork1 
Exploration Methods
In this section, we describe methods to explore combinatorially sized action spaces such as text-games—focusing especially on methods that can deal with their inherent bottleneck structure  We first describe our method that explicitly attempts to detect bottlenecks and then describe how an exploration algorithm such as Go Explore BIBREF9 can leverage knowledge graphs 
KG-A2C-chained An example of a bottleneck can be seen in Figure FIGREF1  We extend the KG-A2C algorithm as follows  First, we detect bottlenecks as states where the agent is unable to progress any further  We set a patience parameter and if the agent has not seen a higher score in patience steps, the agent assumes it has been limited by a bottleneck  Second, when a bottleneck is found, we freeze the policy that gets the agent to the state with the highest score  The agent then begins training a new policy from that particular state 
Simply freezing the policy that led to the bottleneck, however, can potentially result in a policy one that is globally sub-optimal  We therefore employ a backtracking strategy that restarts exploration from each of the $n$ previous steps—searching for a more optimal policy that reaches that bottleneck  At each step, we keep track of a buffer of $n$ states and admissible actions that led up to that locally optimal state  We force the agent to explore from this state to attempt to drive it out of the local optima  If it is further unable to find itself out of this local optima, we refresh the training process again, but starting at the state immediately before the agent reaches the local optima  If this continues to fail, we continue to iterate through this buffer of seen states states up to that local optima until we either find a more optimal state or we run out of states to refresh from, in which we terminate the training algorithm 
KG-A2C-Explore Go-Explore BIBREF9 is an algorithm that is designed to keep track of sub-optimal and under-explored states in order to allow the agent to explore upon more optimal states that may be a result of sparse rewards  The Go-Explore algorithm consists of two phases, the first to continuously explore until a set of promising states and corresponding trajectories are found on the basis of total score, and the second to robustify this found policy against potential stochasticity in the game  Promising states are defined as those states when explored from will likely result in higher reward trajectories  Since the text games we are dealing with are mostly deterministic, with the exception of Zork in later stages, we only focus on using Phase 1 of the Go-Explore algorithm to find an optimal policy  BIBREF10 look at applying Go-Explore to text-games on a set of simpler games generated using the game generation framework TextWorld BIBREF1  Instead of training a policy network in parallel to generate actions used for exploration, they use a small set of “admissible actions”—actions guaranteed to change the world state at any given step during Phase 1—to explore and find high reward trajectories  This space of actions is relatively small (of the order of $10^2$ per step) and so finding high reward trajectories in larger action-spaces such as in Zork would be infeasible
Go-Explore maintains an archive of cells—defined as a set of states that map to a single representation—to keep track of promising states  BIBREF9 simply encodes each cell by keeping track of the agent's position and BIBREF10 use the textual observations encoded by recurrent neural network as a cell representation  We improve on this implementation by training the KG-A2C network in parallel, using the snapshot of the knowledge graph in conjunction with the game state to further encode the current state and use this as a cell representation  At each step, Go-Explore chooses a cell to explore at random (weighted by score to prefer more advanced cells)  The KG-A2C will run for a number of steps, starting with the knowledge graph state and the last seen state of the game from the cell  This will generate a trajectory for the agent while further training the KG-A2C at each iteration, creating a new representation for the knowledge graph as well as a new game state for the cell  After expanding a cell, Go-Explore will continue to sample cells by weight to continue expanding its known states  At the same time, KG-A2C will benefit from the heuristics of selecting preferred cells and be trained on promising states more often 
Evaluation
We compare our two exploration strategies to the following baselines and ablations:
KG-A2C This is the exact same method presented in BIBREF6 with no modifications 
A2C Represents the same approach as KG-A2C but with all the knowledge graph components removed  The state representation is text only encoded using recurrent networks 
A2C-chained Is a variation on KG-A2C-chained where we use our policy chaining approach with the A2C method to train the agent instead of KG-A2C 
A2C-Explore Uses A2C in addition to the exploration strategy seen in KG-A2C-Explore  The cell representations here are defined in terms of the recurrent network based encoding of the textual observation 
Figure FIGREF10 shows that agents utilizing knowledge-graphs in addition to either enhanced exploration method far outperform the baseline A2C and KG-A2C  KG-A2C-chained and KG-A2C-Explore both pass the bottleneck of a score of 40, whereas A2C-Explore gets to the bottleneck but cannot surpass it 
There are a couple of key insights that can be drawn from these results The first is that the knowledge graph appears to be critical; it is theorized to help with partial observability  However the knowledge graph representation isn't sufficient in that the knowledge graph representation without enhanced exploration methods cannot surpass the bottleneck  A2C-chained—which explores without a knowledge graph—fails to even outperform the baseline A2C  We hypothesize that this is due to the knowledge graph aiding implicitly in the sample efficiency of bottleneck detection and subsequent exploration  That is, exploring after backtracking from a potentially detected bottleneck is much more efficient in the knowledge graph based agent 
The Go-Explore based exploration algorithm sees less of a difference between agents  A2C-Explore converges more quickly, but to a lower reward trajectory that fails to pass the bottleneck, whereas KG-A2C-Explore takes longer to reach a similar reward but consistently makes it through the bottleneck  The knowledge graph cell representation appears to thus be a better indication of what a promising state is as opposed to just the textual observation 
Comparing the advanced exploration methods when using the knowledge graph, we see that both agents successfully pass the bottleneck corresponding to entering the cellar and lighting the lamp and reach comparable scores within a margin of error  KG-A2C-chained is significantly more sample efficient and converges faster  We can infer that chaining policies by explicitly detecting bottlenecks lets us pass it more quickly than attempting to find promising cell representations with Go-Explore  This form of chained exploration with backtracking is particularly suited to sequential decision making problems that can be represented as acyclic directed graphs as in Figure FIGREF1 
Appendix ::: Zork1
Zork1 is one of the first text-adventure games and heavily influences games released later in terms of narrative style and game structure  It is a dungeon crawler where the player must explore a vast world and collect a series of treasures  It was identified by BIBREF2 as a moonshot game and has been the subject of much work in leaning agents BIBREF12, BIBREF7, BIBREF11, BIBREF8  Rewards are given to the player when they collect treasures as well as when important intermediate milestones needed to further explore the world are passed  Figure FIGREF15 and Figure FIGREF1 show us a map of the world of Zork1 and the corresponding quest structure 
The bottleneck seen at a score of around 40 is when the player first enters the cellar on the right side of the map  The cellar is dark and you need to immediately light the lamp to see anything  Attempting to explore the cellar in the dark results in you being instantly killed by a monster known as a “grue” 
Appendix ::: Knowledge Graph Rules
We make no changes from the graph update rules used by BIBREF6  Candidate interactive objects are identified by performing part-of-speech tagging on the current observation, identifying singular and proper nouns as well as adjectives, and are then filtered by checking if they can be examined using the command $examine$ $OBJ$  Only the interactive objects not found in the inventory are linked to the node corresponding to the current room and the inventory items are linked to the “you” node  The only other rule applied uses the navigational actions performed by the agent to infer the relative positions of rooms, e g  $\langle kitchen,down,cellar \rangle $ when the agent performs $go$ $down$ when in the kitchen to move to the cellar 
Appendix ::: Hyperparameters
Hyperparameters used for our agents are given below  Patience and buffer size are used for the policy chaining method as described in Section SECREF2  Cell step size is a parameter used for Go-Explore and describes how many steps are taken when exploring in a given cell state  Base hyperparameters for KG-A2C are taken from BIBREF6 and the same parameters are used for A2C 

ground_truth: Galatasaray, Fenerbahçe

Galatasaray , Fenerbahçe 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the results from these proposed strategies?
Context 4: 
Sequence Labeling  Predict each token in a sequence into predefined types  Common tasks include NER, POS tagging, Slot tagging, etc 
Knowledge Distillation BIBREF7   Teacher-Student based knowledge distillation is one common approach for model compression  NeuronBlocks provides knowledge distillation template to improve the inference speed of heavy DNN models like BERT/GPT 
Extractive Machine Reading Comprehension  Given a pair of question and passage, predict the start and end positions of the answer spans in the passage 
User Interface
NeuronBlocks provides convenient user interface for users to build, train, and test DNN models  The details are described in the following 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
I/O interface  This part defines model input/output, such as training data, pre-trained models/embeddings, model saving path, etc 
Model Architecture interface  This is the key part of the configuration file, which defines the whole model architecture  Figure FIGREF19 shows an example of how to specify a model architecture using the blocks in NeuronBlocks  To be more specific, it consists of a list of layers/blocks to construct the architecture, where the blocks are supplied in the gallery of Block Zoo 
Training Parameters interface  In this part, the model optimizer as well as all other training hyper parameters are indicated 
Workflow
Figure FIGREF34 shows the workflow of building DNN models in NeuronBlocks  Users only need to write a JSON configuration file  They can either instantiate an existing template from Model Zoo, or construct a new architecture based on the blocks from Block Zoo  This configuration file is shared across training, test, and prediction  For model hyper-parameter tuning or architecture modification, users just need to change the JSON configuration file  Advanced users can also contribute novel customized blocks into Block Zoo, as long as they follow the same interface guidelines with the existing blocks  These new blocks can be further shared across all users for model architecture design  Moreover, NeuronBlocks has flexible platform support, such as GPU/CPU, GPU management platforms like PAI 
Experiments
To verify the performance of NeuronBlocks, we conducted extensive experiments for common NLP tasks on public data sets including CoNLL-2003 BIBREF14 , GLUE benchmark BIBREF13 , and WikiQA corpus BIBREF15   The experimental results showed that the models built with NeuronBlocks can achieve reliable and competitive results on various tasks, with productivity greatly improved 
Sequence Labeling
For sequence labeling task, we evaluated NeuronBlocks on CoNLL-2003 BIBREF14 English NER dataset, following most works on the same task  This dataset includes four types of named entities, namely, PERSON, LOCATION, ORGANIZATION, and MISC  We adopted the BIOES tagging scheme instead of IOB, as many previous works indicated meaningful improvement with BIOES scheme BIBREF16 , BIBREF17   Table TABREF28 shows the results on CoNLL-2003 Englist testb dataset, with 12 different combinations of network layers/blocks, such as word/character embedding, CNN/LSTM and CRF  The results suggest that the flexible combination of layers/blocks in NeuronBlocks can easily reproduce the performance of original models, with comparative or slightly better performance 
GLUE Benchmark
The General Language Understanding Evaluation (GLUE) benchmark BIBREF13 is a collection of natural language understanding tasks  We experimented on the GLUE benchmark tasks using BiLSTM and Attention based models  As shown in Table TABREF29 , the models built by NeuronBlocks can achieve competitive or even better results on GLUE tasks with minimal coding efforts 
Knowledge Distillation
We evaluated Knowledge Distillation task in NeuronBlocks on a dataset collected from one commercial search engine  We refer to this dataset as Domain Classification Dataset  Each sample in this dataset consists of two parts, i e , a question and a binary label indicating whether the question belongs to a specific domain  Table TABREF36 shows the results, where Area Under Curve (AUC) metric is used as the performance evaluation criteria and Queries per Second (QPS) is used to measure inference speed  By knowledge distillation training approach, the student model by NeuronBlocks managed to get 23-27 times inference speedup with only small performance regression compared with BERTbase fine-tuned classifier 
WikiQA
The WikiQA corpus BIBREF15 is a publicly available dataset for open-domain question answering  This dataset contains 3,047 questions from Bing query logs, each associated with some candidate answer sentences from Wikipedia  We conducted experiments on WikiQA dataset using CNN, BiLSTM, and Attention based models  The results are shown in Table TABREF41   The models built in NeuronBlocks achieved competitive or even better results with simple model configurations 
Conclusion and Future Work
In this paper, we introduce NeuronBlocks, a DNN toolkit for NLP tasks built on PyTorch  NeuronBlocks targets three types of engineers, and provides a two-layer solution to satisfy the requirements from all three types of users  To be more specific, the Model Zoo consists of various templates for the most common NLP tasks, while the Block Zoo supplies a gallery of alternative layers/modules for the networks  Such design achieves a balance between generality and flexibility  Extensive experiments have verified the effectiveness of this approach  NeuronBlocks has been widely used in a product team of a commercial search engine, and significantly improved the productivity for developing NLP DNN approaches 
As an open-source toolkit, we will further extend it in various directions  The following names a few examples 

ground_truth: Reward of 11.8 for the A2C-chained model, 41.8 for the KG-A2C-chained model, 40 for A2C-Explore and 44 for KG-A2C-Explore.

KG-A2C-chained and KG-A2C-Explore both pass the bottleneck of a score of 40