prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What features are used?
Context 1: Introduction
Cancer is one of the leading causes of death in the world, with over 80,000 deaths registered in Canada in 2017 (Canadian Cancer Statistics 2017)  A computer-aided system for cancer diagnosis usually involves a pathologist rendering a descriptive report after examining the tissue glass slides obtained from the biopsy of a patient  A pathology report contains specific analysis of cells and tissues, and other histopathological indicators that are crucial for diagnosing malignancies  An average sized laboratory may produces a large quantity of pathology reports annually (e g , in excess of 50,000), but these reports are written in mostly unstructured text and with no direct link to the tissue sample  Furthermore, the report for each patient is a personalized document and offers very high variability in terminology due to lack of standards and may even include misspellings and missing punctuation, clinical diagnoses interspersed with complex explanations, different terminology to label the same malignancy, and information about multiple carcinoma appearances included in a single report BIBREF0  
In Canada, each Provincial and Territorial Cancer Registry (PTCR) is responsible for collecting the data about cancer diseases and reporting them to Statistics Canada (StatCan)  Every year, Canadian Cancer Registry (CCR) uses the information sources of StatCan to compile an annual report on cancer and tumor diseases  Many countries have their own cancer registry programs  These programs rely on the acquisition of diagnostic, treatment, and outcome information through manual processing and interpretation from various unstructured sources (e g , pathology reports, autopsy/laboratory reports, medical billing summaries)  The manual classification of cancer pathology reports is a challenging, time-consuming task and requires extensive training BIBREF0  
With the continued growth in the number of cancer patients, and the increase in treatment complexity, cancer registries face a significant challenge in manually reviewing the large quantity of reports BIBREF1 , BIBREF0   In this situation, Natural Language Processing (NLP) systems can offer a unique opportunity to automatically encode the unstructured reports into structured data  Since, the registries already have access to the large quantity of historically labeled and encoded reports, a supervised machine learning approach of feature extraction and classification is a compelling direction for making their workflow more effective and streamlined  If successful, such a solution would enable processing reports in much lesser time allowing trained personnel to focus on their research and analysis  However, developing an automated solution with high accuracy and consistency across wide variety of reports is a challenging problem 
For cancer registries, an important piece of information in a pathology report is the associated ICD-O code which describes the patient's histological diagnosis, as described by the World Health Organization's (WHO) International Classification of Diseases for Oncology BIBREF2   Prediction of the primary diagnosis from a pathology report provides a valuable starting point for exploration of machine learning techniques for automated cancer surveillance  A major application for this purpose would be “auto-reporting” based on analysis of whole slide images, the digitization of the biopsy glass slides  Structured, summarized and categorized reports can be associated with the image content when searching in large archives  Such as system would be able to drastically increase the efficiency of diagnostic processes for the majority of cases where in spite of obvious primary diagnosis, still time and effort is required from the pathologists to write a descriptive report 
The primary objective of our study is to analyze the efficacy of existing machine learning approaches for the automated classification of pathology reports into different diagnosis categories  We demonstrate that TF-IDF feature vectors combined with linear SVM or XGBoost classifier can be an effective method for classification of the reports, achieving up to 83% accuracy  We also show that TF-IDF features are capable of identifying important keywords within a pathology report  Furthermore, we have created a new dataset consisting of 1,949 pathology reports across 37 primary diagnoses  Taken together, our exploratory experiments with a newly introduced dataset on pathology reports opens many new opportunities for researchers to develop a scalable and automatic information extraction from unstructured pathology reports 
Background
NLP approaches for information extraction within the biomedical research areas range from rule-based systems BIBREF3 , to domain-specific systems using feature-based classification BIBREF1 , to the recent deep networks for end-to-end feature extraction and classification BIBREF0   NLP has had varied degree of success with free-text pathology reports BIBREF4   Various studies have acknowledge the success of NLP in interpreting pathology reports, especially for classification tasks or extracting a single attribute from a report BIBREF4 , BIBREF5  
The Cancer Text Information Extraction System (caTIES) BIBREF6 is a framework developed in a caBIG project focuses on information extraction from pathology reports  Specifically, caTIES extracts information from surgical pathology reports (SPR) with good precision as well as recall 
Another system known as Open Registry BIBREF7 is capable of filtering the reports with disease codes containing cancer  In BIBREF8 , an approach called Automated Retrieval Console (ARC) is proposed which uses machine learning models to predict the degree of association of a given pathology or radiology with the cancer  The performance ranges from an F-measure of 0 75 for lung cancer to 0 94 for colorectal cancer  However, ARC uses domain-specific rules which hiders with the generalization of the approach to variety of pathology reports 
This research work is inspired by themes emerging in many of the above studies  Specifically, we are evaluating the task of predicting the primary diagnosis from the pathology report  Unlike previous approaches, the system does not rely on custom rule-based knowledge, domain specific features, balanced dataset with fewer number of classes 
Materials and Methods
We assembled a dataset of 1,949 cleaned pathology reports  Each report is associated with one of the 37 different primary diagnoses based on IDC-O codes  The reports are collected from four different body parts or primary sites from multiple patients  The distribution of reports across different primary diagnoses and primary sites is reported in tab:report-distribution  The dataset was developed in three steps as follows 
Collecting pathology reports: The total of 11,112 pathology reports were downloaded from NCI's Genomic Data Commons (GDC) dataset in PDF format BIBREF9   Out of all PDF files, 1,949 reports were selected across multiple patients from four specific primary sites—thymus, testis, lung, and kidney  The selection was primarily made based on the quality of PDF files 
Cleaning reports: The next step was to extract the text content from these reports  Due to the significant time expense of manually re-typing all the pathology reports, we developed a new strategy to prepare our dataset  We applied an Optical Character Recognition (OCR) software to convert the PDF reports to text files  Then, we manually inspected all generated text files to fix any grammar/spelling issues and irrelevant characters as an artefact produced by the OCR system 
Splitting into training-testing data: We split the cleaned reports into 70% and 30% for training and testing, respectively  This split resulted in 1,364 training, and 585 testing reports 
Pre-Processing of Reports
We pre-processed the reports by setting their text content to lowercase and filtering out any non-alphanumeric characters  We used NLTK library to remove stopping words, e g , `the', `an', `was', `if' and so on BIBREF10   We then analyzed the reports to find common bigrams, such as “lung parenchyma”, “microscopic examination”, “lymph node” etc  We joined the biagrams with a hyphen, converting them into a single word  We further removed the words that occur less than 2% in each of the diagnostic category  As well, we removed the words that occur more than 90% across all the categories  We stored each pre-processed report in a separate text file 
TF-IDF features
TF-IDF stands for Term Frequency-Inverse Document Frequency, and it is a useful weighting scheme in information retrieval and text mining  TF-IDF signifies the importance of a term in a document within a corpus  It is important to note that a document here refers to a pathology report, a corpus refers to the collection of reports, and a term refers to a single word in a report  The TF-IDF weight for a term INLINEFORM0 in a document INLINEFORM1 is given by DISPLAYFORM0
We performed the following steps to transform a pathology report into a feature vector:
Create a set of vocabulary containing all unique words from all the pre-processed training reports 
Create a zero vector INLINEFORM0 of the same length as the vocabulary 
For each word INLINEFORM0 in a report INLINEFORM1 , set the corresponding index in INLINEFORM2 to INLINEFORM3  
The resultant INLINEFORM0 is a feature vector for the report INLINEFORM1 and it is a highly sparse vector 
Keyword extraction and topic modelling
The keyword extraction involves identifying important words within reports that summarizes its content  In contrast, the topic modelling allows grouping these keywords using an intelligent scheme, enabling users to further focus on certain aspects of a document  All the words in a pathology report are sorted according to their TF-IDF weights  The top INLINEFORM0 sorted words constitute the top INLINEFORM1 keywords for the report  The INLINEFORM2 is empirically set to 50 within this research  The extracted keywords are further grouped into different topics by using latent Dirichlet allocation (LDA) BIBREF11   The keywords in a report are highlighted using the color theme based on their topics 
Evaluation metrics
Each model is evaluated using two standard NLP metrics—micro and macro averaged F-scores, the harmonic mean of related metrics precision and recall  For each diagnostic category INLINEFORM0 from a set of 37 different classes INLINEFORM1 , the number of true positives INLINEFORM2 , false positives INLINEFORM3 , and false negatives INLINEFORM4 , the micro F-score is defined as DISPLAYFORM0
whereas macro F-score is given by DISPLAYFORM0
In summary, micro-averaged metrics have class representation roughly proportional to their test set representation (same as accuracy for classification problem with a single label per data point), whereas macro-averaged metrics are averaged by class without weighting by class prevalence BIBREF12  
Experimental setting
In this study, we performed two different series of experiments: i) evaluating the performance of TF-IDF features and various machine learning classifiers on the task of predicting primary diagnosis from the text content of a given report, and ii) using TF-IDF and LDA techniques to highlight the important keywords within a report  For the first experiment series, training reports are pre-processed, then their TF-IDF features are extracted  The TF-IDF features and the training labels are used to train different classification models  These different classification models and their hyper-parameters are reported in tab:classifier  The performance of classifiers is measured quantitatively on the test dataset using the evaluation metrics discussed in the previous section  For the second experiment series, a random report is selected and its top 50 keywords are extracted using TF-IDF weights  These 50 keywords are highlighted using different colors based on their associated topic, which are extracted through LDA  A non-expert based qualitative inspection is performed on the extracted keywords and their corresponding topics 
Experiment Series 1
A classification model is trained to predict the primary diagnosis given the content of the cancer pathology report  The performance results on this task are reported in tab:results  We can observe that the XGBoost classifier outperformed all other models for both the micro F-score metric, with a score of 0 92, and the macro F-score metric, with a score of 0 31  This was an improvement of 7% for the micro F-score over the next best model, SVM-L, and a marginal improvement of 5% for macro F-score  It is interesting to note that SVM with linear kernels performs much better than SVM with RBF kernel, scoring 9% on the macro F-score and 12% more on the micro F-score  It is suspected that since words used in primary diagnosis itself occur in some reports, thus enabling the linear models to outperform complex models 
Experiment Series 2
fig:keywords shows the top 50 keywords highlighted using TF-IDF and LDA  The proposed approach has performed well in highlighting the important regions, for example the topic highlighted with a red color containing “presence range tumor necrosis” provides useful biomarker information to readers 
Conclusions
We proposed a simple yet efficient TF-IDF method to extract and corroborate useful keywords from pathology cancer reports  Encoding a pathology report for cancer and tumor surveillance is a laborious task, and sometimes it is subjected to human errors and variability in the interpretation  One of the most important aspects of encoding a pathology report involves extracting the primary diagnosis  This may be very useful for content-based image retrieval to combine with visual information  We used existing classification model and TF-IDF features to predict the primary diagnosis  We achieved up to 92% accuracy using XGBoost classifier  The prediction accuracy empowers the adoption of machine learning methods for automated information extraction from pathology reports 
Context 2: Introduction
This work is licenced under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/ Deep neural networks have been widely used in text classification and have achieved promising results BIBREF0 , BIBREF1 , BIBREF2   Most focus on content information and use models such as convolutional neural networks (CNN) BIBREF3 or recursive neural networks BIBREF4   However, for user-generated posts on social media like Facebook or Twitter, there is more information that should not be ignored  On social media platforms, a user can act either as the author of a post or as a reader who expresses his or her comments about the post 
In this paper, we classify posts taking into account post authorship, likes, topics, and comments  In particular, users and their “likes” hold strong potential for text mining  For example, given a set of posts that are related to a specific topic, a user's likes and dislikes provide clues for stance labeling  From a user point of view, users with positive attitudes toward the issue leave positive comments on the posts with praise or even just the post's content; from a post point of view, positive posts attract users who hold positive stances  We also investigate the influence of topics: different topics are associated with different stance labeling tendencies and word usage  For example we discuss women's rights and unwanted babies on the topic of abortion, but we criticize medicine usage or crime when on the topic of marijuana BIBREF5   Even for posts on a specific topic like nuclear power, a variety of arguments are raised: green energy, radiation, air pollution, and so on  As for comments, we treat them as additional text information  The arguments in the comments and the commenters (the users who leave the comments) provide hints on the post's content and further facilitate stance classification 
In this paper, we propose the user-topic-comment neural network (UTCNN), a deep learning model that utilizes user, topic, and comment information  We attempt to learn user and topic representations which encode user interactions and topic influences to further enhance text classification, and we also incorporate comment information  We evaluate this model on a post stance classification task on forum-style social media platforms  The contributions of this paper are as follows: 1  We propose UTCNN, a neural network for text in modern social media channels as well as legacy social media, forums, and message boards — anywhere that reveals users, their tastes, as well as their replies to posts  2  When classifying social media post stances, we leverage users, including authors and likers  User embeddings can be generated even for users who have never posted anything  3  We incorporate a topic model to automatically assign topics to each post in a single topic dataset  4  We show that overall, the proposed method achieves the highest performance in all instances, and that all of the information extracted, whether users, topics, or comments, still has its contributions 
Extra-Linguistic Features for Stance Classification
In this paper we aim to use text as well as other features to see how they complement each other in a deep learning model  In the stance classification domain, previous work has showed that text features are limited, suggesting that adding extra-linguistic constraints could improve performance BIBREF6 , BIBREF7 , BIBREF8   For example, Hasan and Ng as well as Thomas et al  require that posts written by the same author have the same stance BIBREF9 , BIBREF10   The addition of this constraint yields accuracy improvements of 1–7% for some models and datasets  Hasan and Ng later added user-interaction constraints and ideology constraints BIBREF7 : the former models the relationship among posts in a sequence of replies and the latter models inter-topic relationships, e g , users who oppose abortion could be conservative and thus are likely to oppose gay rights 
For work focusing on online forum text, since posts are linked through user replies, sequential labeling methods have been used to model relationships between posts  For example, Hasan and Ng use hidden Markov models (HMMs) to model dependent relationships to the preceding post BIBREF9 ; Burfoot et al  use iterative classification to repeatedly generate new estimates based on the current state of knowledge BIBREF11 ; Sridhar et al  use probabilistic soft logic (PSL) to model reply links via collaborative filtering BIBREF12   In the Facebook dataset we study, we use comments instead of reply links  However, as the ultimate goal in this paper is predicting not comment stance but post stance, we treat comments as extra information for use in predicting post stance 
Deep Learning on Extra-Linguistic Features
In recent years neural network models have been applied to document sentiment classification BIBREF13 , BIBREF4 , BIBREF14 , BIBREF15 , BIBREF2   Text features can be used in deep networks to capture text semantics or sentiment  For example, Dong et al  use an adaptive layer in a recursive neural network for target-dependent Twitter sentiment analysis, where targets are topics such as windows 7 or taylor swift BIBREF16 , BIBREF17 ; recursive neural tensor networks (RNTNs) utilize sentence parse trees to capture sentence-level sentiment for movie reviews BIBREF4 ; Le and Mikolov predict sentiment by using paragraph vectors to model each paragraph as a continuous representation BIBREF18   They show that performance can thus be improved by more delicate text models 
Others have suggested using extra-linguistic features to improve the deep learning model  The user-word composition vector model (UWCVM) BIBREF19 is inspired by the possibility that the strength of sentiment words is user-specific; to capture this they add user embeddings in their model  In UPNN, a later extension, they further add a product-word composition as product embeddings, arguing that products can also show different tendencies of being rated or reviewed BIBREF20   Their addition of user information yielded 2–10% improvements in accuracy as compared to the above-mentioned RNTN and paragraph vector methods  We also seek to inject user information into the neural network model  In comparison to the research of Tang et al  on sentiment classification for product reviews, the difference is two-fold  First, we take into account multiple users (one author and potentially many likers) for one post, whereas only one user (the reviewer) is involved in a review  Second, we add comment information to provide more features for post stance classification  None of these two factors have been considered previously in a deep learning model for text stance classification  Therefore, we propose UTCNN, which generates and utilizes user embeddings for all users — even for those who have not authored any posts — and incorporates comments to further improve performance 
Method
In this section, we first describe CNN-based document composition, which captures user- and topic-dependent document-level semantic representation from word representations  Then we show how to add comment information to construct the user-topic-comment neural network (UTCNN) 
User- and Topic-dependent Document Composition
As shown in Figure FIGREF4 , we use a general CNN BIBREF3 and two semantic transformations for document composition   We are given a document with an engaged user INLINEFORM0 , a topic INLINEFORM1 , and its composite INLINEFORM2 words, each word INLINEFORM3 of which is associated with a word embedding INLINEFORM4 where INLINEFORM5 is the vector dimension  For each word embedding INLINEFORM6 , we apply two dot operations as shown in Equation EQREF6 : DISPLAYFORM0
where INLINEFORM0 models the user reading preference for certain semantics, and INLINEFORM1 models the topic semantics; INLINEFORM2 and INLINEFORM3 are the dimensions of transformed user and topic embeddings respectively  We use INLINEFORM4 to model semantically what each user prefers to read and/or write, and use INLINEFORM5 to model the semantics of each topic  The dot operation of INLINEFORM6 and INLINEFORM7 transforms the global representation INLINEFORM8 to a user-dependent representation  Likewise, the dot operation of INLINEFORM9 and INLINEFORM10 transforms INLINEFORM11 to a topic-dependent representation 
After the two dot operations on INLINEFORM0 , we have user-dependent and topic-dependent word vectors INLINEFORM1 and INLINEFORM2 , which are concatenated to form a user- and topic-dependent word vector INLINEFORM3   Then the transformed word embeddings INLINEFORM4 are used as the CNN input  Here we apply three convolutional layers on the concatenated transformed word embeddings INLINEFORM5 : DISPLAYFORM0
where INLINEFORM0 is the index of words; INLINEFORM1 is a non-linear activation function (we use INLINEFORM2 ); INLINEFORM5 is the convolutional filter with input length INLINEFORM6 and output length INLINEFORM7 , where INLINEFORM8 is the window size of the convolutional operation; and INLINEFORM9 and INLINEFORM10 are the output and bias of the convolution layer INLINEFORM11 , respectively  In our experiments, the three window sizes INLINEFORM12 in the three convolution layers are one, two, and three, encoding unigram, bigram, and trigram semantics accordingly 
After the convolutional layer, we add a maximum pooling layer among convolutional outputs to obtain the unigram, bigram, and trigram n-gram representations  This is succeeded by an average pooling layer for an element-wise average of the three maximized convolution outputs 
UTCNN Model Description
Figure FIGREF10 illustrates the UTCNN model  As more than one user may interact with a given post, we first add a maximum pooling layer after the user matrix embedding layer and user vector embedding layer to form a moderator matrix embedding INLINEFORM0 and a moderator vector embedding INLINEFORM1 for moderator INLINEFORM2 respectively, where INLINEFORM3 is used for the semantic transformation in the document composition process, as mentioned in the previous section  The term moderator here is to denote the pseudo user who provides the overall semantic/sentiment of all the engaged users for one document  The embedding INLINEFORM4 models the moderator stance preference, that is, the pattern of the revealed user stance: whether a user is willing to show his preference, whether a user likes to show impartiality with neutral statements and reasonable arguments, or just wants to show strong support for one stance  Ideally, the latent user stance is modeled by INLINEFORM5 for each user  Likewise, for topic information, a maximum pooling layer is added after the topic matrix embedding layer and topic vector embedding layer to form a joint topic matrix embedding INLINEFORM6 and a joint topic vector embedding INLINEFORM7 for topic INLINEFORM8 respectively, where INLINEFORM9 models the semantic transformation of topic INLINEFORM10 as in users and INLINEFORM11 models the topic stance tendency  The latent topic stance is also modeled by INLINEFORM12 for each topic 
As for comments, we view them as short documents with authors only but without likers nor their own comments  Therefore we apply document composition on comments although here users are commenters (users who comment)  It is noticed that the word embeddings INLINEFORM0 for the same word in the posts and comments are the same, but after being transformed to INLINEFORM1 in the document composition process shown in Figure FIGREF4 , they might become different because of their different engaged users  The output comment representation together with the commenter vector embedding INLINEFORM2 and topic vector embedding INLINEFORM3 are concatenated and a maximum pooling layer is added to select the most important feature for comments  Instead of requiring that the comment stance agree with the post, UTCNN simply extracts the most important features of the comment contents; they could be helpful, whether they show obvious agreement or disagreement  Therefore when combining comment information here, the maximum pooling layer is more appropriate than other pooling or merging layers  Indeed, we believe this is one reason for UTCNN's performance gains 
Finally, the pooled comment representation, together with user vector embedding INLINEFORM0 , topic vector embedding INLINEFORM1 , and document representation are fed to a fully connected network, and softmax is applied to yield the final stance label prediction for the post 
Experiment
We start with the experimental dataset and then describe the training process as well as the implementation of the baselines  We also implement several variations to reveal the effects of features: authors, likers, comment, and commenters  In the results section we compare our model with related work 
Dataset
We tested the proposed UTCNN on two different datasets: FBFans and CreateDebate  FBFans is a privately-owned, single-topic, Chinese, unbalanced, social media dataset, and CreateDebate is a public, multiple-topic, English, balanced, forum dataset  Results using these two datasets show the applicability and superiority for different topics, languages, data distributions, and platforms 
The FBFans dataset contains data from anti-nuclear-power Chinese Facebook fan groups from September 2013 to August 2014, including posts and their author and liker IDs  There are a total of 2,496 authors, 505,137 likers, 33,686 commenters, and 505,412 unique users  Two annotators were asked to take into account only the post content to label the stance of the posts in the whole dataset as supportive, neutral, or unsupportive (hereafter denoted as Sup, Neu, and Uns)  Sup/Uns posts were those in support of or against anti-reconstruction; Neu posts were those evincing a neutral standpoint on the topic, or were irrelevant  Raw agreement between annotators is 0 91, indicating high agreement  Specifically, Cohen’s Kappa for Neu and not Neu labeling is 0 58 (moderate), and for Sup or Uns labeling is 0 84 (almost perfect)  Posts with inconsistent labels were filtered out, and the development and testing sets were randomly selected from what was left  Posts in the development and testing sets involved at least one user who appeared in the training set  The number of posts for each stance is shown on the left-hand side of Table TABREF12   About twenty percent of the posts were labeled with a stance, and the number of supportive (Sup) posts was much larger than that of the unsupportive (Uns) ones: this is thus highly skewed data, which complicates stance classification  On average, 161

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the baselines?
Context 2: 1 users were involved in one post  The maximum was 23,297 and the minimum was one (the author)  For comments, on average there were 3 comments per post  The maximum was 1,092 and the minimum was zero 
To test whether the assumption of this paper – posts attract users who hold the same stance to like them – is reliable, we examine the likes from authors of different stances  Posts in FBFans dataset are used for this analysis  We calculate the like statistics of each distinct author from these 32,595 posts  As the numbers of authors in the Sup, Neu and Uns stances are largely imbalanced, these numbers are normalized by the number of users of each stance  Table TABREF13 shows the results  Posts with stances (i e , not neutral) attract users of the same stance  Neutral posts also attract both supportive and neutral users, like what we observe in supportive posts, but just the neutral posts can attract even more neutral likers  These results do suggest that users prefer posts of the same stance, or at least posts of no obvious stance which might cause annoyance when reading, and hence support the user modeling in our approach 
The CreateDebate dataset was collected from an English online debate forum discussing four topics: abortion (ABO), gay rights (GAY), Obama (OBA), and marijuana (MAR)  The posts are annotated as for (F) and against (A)  Replies to posts in this dataset are also labeled with stance and hence use the same data format as posts  The labeling results are shown in the right-hand side of Table TABREF12   We observe that the dataset is more balanced than the FBFans dataset  In addition, there are 977 unique users in the dataset  To compare with Hasan and Ng's work, we conducted five-fold cross-validation and present the annotation results as the average number of all folds BIBREF9 , BIBREF5  
The FBFans dataset has more integrated functions than the CreateDebate dataset; thus our model can utilize all linguistic and extra-linguistic features  For the CreateDebate dataset, on the other hand, the like and comment features are not available (as there is a stance label for each reply, replies are evaluated as posts as other previous work) but we still implemented our model using the content, author, and topic information 
Settings
In the UTCNN training process, cross-entropy was used as the loss function and AdaGrad as the optimizer  For FBFans dataset, we learned the 50-dimensional word embeddings on the whole dataset using GloVe BIBREF21 to capture the word semantics; for CreateDebate dataset we used the publicly available English 50-dimensional word embeddings, pre-trained also using GloVe  These word embeddings were fixed in the training process  The learning rate was set to 0 03  All user and topic embeddings were randomly initialized in the range of [-0 1 0 1]  Matrix embeddings for users and topics were sized at 250 ( INLINEFORM0 ); vector embeddings for users and topics were set to length 10 
We applied the LDA topic model BIBREF22 on the FBFans dataset to determine the latent topics with which to build topic embeddings, as there is only one general known topic: nuclear power plants  We learned 100 latent topics and assigned the top three topics for each post  For the CreateDebate dataset, which itself constitutes four topics, the topic labels for posts were used directly without additionally applying LDA 
For the FBFans data we report class-based f-scores as well as the macro-average f-score ( INLINEFORM0 ) shown in equation EQREF19   DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are the average precision and recall of the three class  We adopted the macro-average f-score as the evaluation metric for the overall performance because (1) the experimental dataset is severely imbalanced, which is common for contentious issues; and (2) for stance classification, content in minor-class posts is usually more important for further applications  For the CreateDebate dataset, accuracy was adopted as the evaluation metric to compare the results with related work BIBREF7 , BIBREF9 , BIBREF12  
Baselines
We pit our model against the following baselines: 1) SVM with unigram, bigram, and trigram features, which is a standard yet rather strong classifier for text features; 2) SVM with average word embedding, where a document is represented as a continuous representation by averaging the embeddings of the composite words; 3) SVM with average transformed word embeddings (the INLINEFORM0 in equation EQREF6 ), where a document is represented as a continuous representation by averaging the transformed embeddings of the composite words; 4) two mature deep learning models on text classification, CNN BIBREF3 and Recurrent Convolutional Neural Networks (RCNN) BIBREF0 , where the hyperparameters are based on their work; 5) the above SVM and deep learning models with comment information; 6) UTCNN without user information, representing a pure-text CNN model where we use the same user matrix and user embeddings INLINEFORM1 and INLINEFORM2 for each user; 7) UTCNN without the LDA model, representing how UTCNN works with a single-topic dataset; 8) UTCNN without comments, in which the model predicts the stance label given only user and topic information  All these models were trained on the training set, and parameters as well as the SVM kernel selections (linear or RBF) were fine-tuned on the development set  Also, we adopt oversampling on SVMs, CNN and RCNN because the FBFans dataset is highly imbalanced 
Results on FBFans Dataset
In Table TABREF22 we show the results of UTCNN and the baselines on the FBFans dataset  Here Majority yields good performance on Neu since FBFans is highly biased to the neutral class  The SVM models perform well on Sup and Neu but perform poorly for Uns, showing that content information in itself is insufficient to predict stance labels, especially for the minor class  With the transformed word embedding feature, SVM can achieve comparable performance as SVM with n-gram feature  However, the much fewer feature dimension of the transformed word embedding makes SVM with word embeddings a more efficient choice for modeling the large scale social media dataset  For the CNN and RCNN models, they perform slightly better than most of the SVM models but still, the content information is insufficient to achieve a good performance on the Uns posts  As to adding comment information to these models, since the commenters do not always hold the same stance as the author, simply adding comments and post contents together merely adds noise to the model 
Among all UTCNN variations, we find that user information is most important, followed by topic and comment information  UTCNN without user information shows results similar to SVMs — it does well for Sup and Neu but detects no Uns  Its best f-scores on both Sup and Neu among all methods show that with enough training data, content-based models can perform well; at the same time, the lack of user information results in too few clues for minor-class posts to either predict their stance directly or link them to other users and posts for improved performance  The 17 5% improvement when adding user information suggests that user information is especially useful when the dataset is highly imbalanced  All models that consider user information predict the minority class successfully  UCTNN without topic information works well but achieves lower performance than the full UTCNN model  The 4 9% performance gain brought by LDA shows that although it is satisfactory for single topic datasets, adding that latent topics still benefits performance: even when we are discussing the same topic, we use different arguments and supporting evidence  Lastly, we get 4 8% improvement when adding comment information and it achieves comparable performance to UTCNN without topic information, which shows that comments also benefit performance  For platforms where user IDs are pixelated or otherwise hidden, adding comments to a text model still improves performance  In its integration of user, content, and comment information, the full UTCNN produces the highest f-scores on all Sup, Neu, and Uns stances among models that predict the Uns class, and the highest macro-average f-score overall  This shows its ability to balance a biased dataset and supports our claim that UTCNN successfully bridges content and user, topic, and comment information for stance classification on social media text  Another merit of UTCNN is that it does not require a balanced training data  This is supported by its outperforming other models though no oversampling technique is applied to the UTCNN related experiments as shown in this paper  Thus we can conclude that the user information provides strong clues and it is still rich even in the minority class 
We also investigate the semantic difference when a user acts as an author/liker or a commenter  We evaluated a variation in which all embeddings from the same user were forced to be identical (this is the UTCNN shared user embedding setting in Table TABREF22 )  This setting yielded only a 2 5% improvement over the model without comments, which is not statistically significant  However, when separating authors/likers and commenters embeddings (i e , the UTCNN full model), we achieved much greater improvements (4 8%)  We attribute this result to the tendency of users to use different wording for different roles (for instance author vs commenter)  This is observed when the user, acting as an author, attempts to support her argument against nuclear power by using improvements in solar power; when acting as a commenter, though, she interacts with post contents by criticizing past politicians who supported nuclear power or by arguing that the proposed evacuation plan in case of a nuclear accident is ridiculous  Based on this finding, in the final UTCNN setting we train two user matrix embeddings for one user: one for the author/liker role and the other for the commenter role 
Results on CreateDebate Dataset
Table TABREF24 shows the results of UTCNN, baselines as we implemented on the FBFans datset and related work on the CreateDebate dataset  We do not adopt oversampling on these models because the CreateDebate dataset is almost balanced  In previous work, integer linear programming (ILP) or linear-chain conditional random fields (CRFs) were proposed to integrate text features, author, ideology, and user-interaction constraints, where text features are unigram, bigram, and POS-dependencies; the author constraint tends to require that posts from the same author for the same topic hold the same stance; the ideology constraint aims to capture inferences between topics for the same author; the user-interaction constraint models relationships among posts via user interactions such as replies BIBREF7 , BIBREF9  
The SVM with n-gram or average word embedding feature performs just similar to the majority  However, with the transformed word embedding, it achieves superior results  It shows that the learned user and topic embeddings really capture the user and topic semantics  This finding is not so obvious in the FBFans dataset and it might be due to the unfavorable data skewness for SVM  As for CNN and RCNN, they perform slightly better than most SVMs as we found in Table TABREF22 for FBFans 
Compared to the ILP BIBREF7 and CRF BIBREF9 methods, the UTCNN user embeddings encode author and user-interaction constraints, where the ideology constraint is modeled by the topic embeddings and text features are modeled by the CNN  The significant improvement achieved by UTCNN suggests the latent representations are more effective than overt model constraints 
The PSL model BIBREF12 jointly labels both author and post stance using probabilistic soft logic (PSL) BIBREF23 by considering text features and reply links between authors and posts as in Hasan and Ng's work  Table TABREF24 reports the result of their best AD setting, which represents the full joint stance/disagreement collective model on posts and is hence more relevant to UTCNN  In contrast to their model, the UTCNN user embeddings represent relationships between authors, but UTCNN models do not utilize link information between posts  Though the PSL model has the advantage of being able to jointly label the stances of authors and posts, its performance on posts is lower than the that for the ILP or CRF models  UTCNN significantly outperforms these models on posts and has the potential to predict user stances through the generated user embeddings 
For the CreateDebate dataset, we also evaluated performance when not using topic embeddings or user embeddings; as replies in this dataset are viewed as posts, the setting without comment embeddings is not available  Table TABREF24 shows the same findings as Table TABREF22 : the 21% improvement in accuracy demonstrates that user information is the most vital  This finding also supports the results in the related work: user constraints are useful and can yield 11 2% improvement in accuracy BIBREF7   Further considering topic information yields 3 4% improvement, suggesting that knowing the subject of debates provides useful information  In sum, Table TABREF22 together with Table TABREF24 show that UTCNN achieves promising performance regardless of topic, language, data distribution, and platform 
Conclusion
We have proposed UTCNN, a neural network model that incorporates user, topic, content and comment information for stance classification on social media texts  UTCNN learns user embeddings for all users with minimum active degree, i e , one post or one like  Topic information obtained from the topic model or the pre-defined labels further improves the UTCNN model  In addition, comment information provides additional clues for stance classification  We have shown that UTCNN achieves promising and balanced results  In the future we plan to explore the effectiveness of the UTCNN user embeddings for author stance classification 
Acknowledgements
Research of this paper was partially supported by Ministry of Science and Technology, Taiwan, under the contract MOST 104-2221-E-001-024-MY2 
Context 3: Winograd Schemas
A Winograd schema (Levesque, Davis, and Morgenstern 2012) is a pair of sentences, or of short texts, called the elements of the schema, that satisfy the following constraints:
The following is an example of a Winograd schema:
Here, the two sentences differ only in the last word: `large' vs  `small'  The ambiguous pronoun is `it'  The two antecedents are `trophy' and `brown suitcase'  A human reader will naturally interpret `it' as referring to the trophy in the first sentence and to the suitcase in the second sentence, using the world knowledge that a small object can fit in a large container, but a large object cannot fit in a small container (Davis 2013)  Condition 4 is satisfied because either a trophy or a suitcase can be either large or small, and there is no reason to suppose that there would be a meaningful correlation of `small' or `large' with `trophy' or `suitcase' in a typical corpus 
We will say that an element of a Winograd schema is “solved” if the referent of the pronoun is identified 
An example of a pair of sentences satisfying conditions 1-3 but not 4 would be
Since women cannot be carcinogenic and pills cannot be pregnant, the pronoun `they' in these sentences is easily disambiguated using selectional restrictions  This pair is therefore not a valid Winograd schema 
The Winograd Schema Challenge (WSC) is a challenge for AI programs  The program is presented with a collection of sentences, each of which is one element of a Winograd schema, and is required to find the correct referent for the ambiguous pronoun  An AI program passes the challenge if its success rate is comparable to a human reader  The challenge is administered by commonsensereasoning org and sponsored by Nuance Inc  It was offered for the first time at IJCAI-2016 (Morgenstern, Davis, and Ortiz, in preparation); the organizers plan to continue to offer it roughly once a year 
Winograd schemas as the basis for challenges for machine translation programs
In many cases, the identification of the referent of the prounoun in a Winograd schema is critical for finding the correct translation of that pronoun in a different language  Therefore, Winograd schemas can be used as a very difficult challenge for the depth of understanding achieved by a machine translation program 
The third person plural pronoun `they' has no gender in English and most other languages (unlike the third person singular pronouns `he', `she', and `it')  However Romance languages such as French, Italian, and Spanish, and Semitic languages such as Hebrew and Arabic distinguish between the masculine and the feminine third-person plural pronouns, at least in some grammatical cases  For instance in French, the masculine pronoun is `ils'; the feminine pronoun is `elles'  In all of these cases, the masculine pronoun is standardly used for groups of mixed or unknown gender 
In order to correctly translate a sentence in English containing the word `they' into one of these languages, it is necessary to determine whether or not the referent is a group of females  If it is, then the translation must be the feminine pronoun; otherwise, it must be the masculine pronoun  Therefore, if one can create a Winograd schema in English where the ambiguous pronoun is `they' and the correct referent for one element is a collection of men and for the other is a collection of women, then to translate both elements correctly requires solving the Winograd schema 
As an example, consider the Winograd schema:
If these sentences are translated into French, then `they' in the first sentence should be translated `elles', as referring to Jane and Susan, and `they' in the second sentence should be translated `ils', as referring to Fred and George 
A number of the Winograd schemas already published can be “converted” quite easily to this form  Indeed, the above example was constructed in this way; the original form was “Jane knocked on Susan's door, but she did not [answer/get an answer] ” Of the 144 schemas in the collection at http://www cs nyu edu/faculty/davise/papers/WinogradSchemas/WSCollection html there are 33 that can plausibly be translated this way  (#'s 3, 4, 12, 15, 17, 18, 23, 25, 26, 27, 42, 43, 44, 45, 46, 57, 58, 60, 69, 70, 71, 77, 78, 79, 86, 97, 98, 106, 111, 114, 127, 132, 144) — essentially any schema that involves two people as possible referents where the content of the sentences makes sense as applied to groups of people rather than individuals 
A similar device, in the opposite direction, relies on the fact that French does not distinguish between the possessive pronouns `his' and `her'  The pronouns `son' and `sa' are gendered, but the gender agrees with the possession, not the possessor  Therefore a Winograd schema that relies on finding a referent for a possessive pronoun can be turned into a hard pair of French-to-English translation problems by making one possible referent male and the other one female  For example, schema #124 in the online collection reads “The man lifted the boy onto his [bunk bed/shoulders] ” Changing the boy to a girl and translating into French gives “L'homme leva la fille sur [ses épaules/son lit superposé]”  In the first sentence “ses” is translated “his”, in the second “son” is translated “her”  The eleven Winograd schemas #'s 112, 117, 118, 119, 124, 125, 127, 129, 130, 131, and 143 can be converted this way 
Care must be taken to avoid relying on, or seeming to rely on, objectionable stereotypes about men and women  One mechanism that can sometimes be used is to include a potentially problematic sentence in both directions  For instance, schema 23 from the WSC collection can be translated into both “The girls were bullying the boys so we [punished/rescued] them” and “The boys were bullying the girls, so we [punished/rescued] them,” thus avoiding any presupposition of whether girls are more likely to bully boys or vice versa 
A historical note: Winograd schemas were named after Terry Winograd because of a well-known example in his doctoral thesis (Winograd 1970)  In his thesis, this example followed the above form, and the importance of the example was justified in terms of machine translation  Winograd's original schema was: “The city councilmen refused to give the women a permit for a demonstration because they [feared/advocated] violence", and Winograd explained that, in the sentence with `feared', `they' would refer to the councilmen and would be translated as `ils' in French, whereas in the sentence with `advocated', `they' would refer to the women and would be translated as `elles'  In the later versions of the thesis, published as (Winograd 1972), he changed `women' to `demonstrators'; this made the disambiguation clearer, but lost the point about translation 
Current state of the art
No one familiar with the state of the art in machine translation technology or the state of the art of artificial intelligence generally will be surprised to learn that currently machine translation program are unable to solve these Winograd schema challenge problems 
What may be more surprising is that currently (July 2016), machine translation programs are unable to choose the feminine plural pronoun even when the only possible antecedent is a group of women  For instance, the sentence “The girls sang a song and they danced” is translated into French as “Les filles ont chanté une chanson et ils ont dansé” by Google Translate (GT), Bing Translate, and Yandex  In fact, I have been unable to construct any sentence in English that is translated into any language using the feminine plural pronun  Note that, since the masculine plural pronoun is used for groups of mixed gender in all these languages, it is almost certainly more common in text than the feminine plural; hence this strategy is reasonable faute de mieux  (It also seems likely that the erroneous use of a feminine plural for a masculine antecedent sounds even more jarring than the reverse )
The same thing sometimes occurs in translating the feminine plural pronoun between languages that have it  GT translates the French word `elles' into Spanish as `ellos' (the masculine form)  Curiously, in the opposite direction, it gets the right answer; the Spanish `ellas' (fem ) is translated into French as `elles' 
Language-specific issues
The masculine and feminine plural pronouns are distinguished in the Romance languages (French, Spanish, Italian, Portuguese etc ) and in Semitic languages (Arabic, Hebrew, etc ) I have consulted with native speakers and experts in these languages about the degree to which the gender distinction is observed in practice  The experts say that in French, Spanish, Italian, and Portuguese, the distinction is very strictly observed; the use of a masculine pronoun for a feminine antecedent is jarringly wrong to a native or fluent speaker  “Les filles ont chanté une chanson et ils ont dansé” sounds as wrong to a French speaker as “The girl sang a song and he danced” sounds to an English speaker; in both cases, the hearer will interpret the pronoun as referrinig to some other persons or person, who is male  In Hebrew and Arabic, this is much less true; in speech, and even, increasingly, in writing, the masculine pronoun is often used for a feminine antecedent 
Looking further ahead, it is certainly possible that gender distinctions will be abandoned in the Romance languages, or even that English will have driven all other languages out of existence, sooner than AI systems will be able to do pronoun resolution in Winograd schemas; at that point, this test will no longer be useful 
In some cases, a translation program can side-step the issue by omitting the pronoun altogether  For example, GT translates the above sentence “The girls sang a song and they danced” into Spanish as “Las chicas cantaron una canción y bailaban” and into Italian as “Le ragazze hanno cantato una canzone e ballavano ” However, with the more complex sentences of the Winograd Schemas, this strategy will rarely give a plausible translation for both elements of the schema 
Other languages, other ambiguities
Broadly speaking, whenever a target language INLINEFORM0 requires some distinction that is optional or non-existent in source language INLINEFORM1 , it is possible to create a sentence INLINEFORM2 in INLINEFORM3 where the missing information is not explicit but can be inferred from background knowledge  Translating INLINEFORM4 from INLINEFORM5 to INLINEFORM6 thus requires using the background knowledge to resolve the ambiguity, and will therefore be challenging for automatic machine translation 
A couple of examples: The word `sie' in German serves as both the formal second person prounoun (always capitalized), the third person feminine singular, and the third person plural  Therefore, it can be translated into English as either “you”, “she”, “it”, or “they”; and into French as either `vous', `il', `elle', `ils', or `elles'  (The feminine third-person singular German `sie' can be translated as neuter in English and as masculine in French because the three languages do not slice up the worlds into genders in the same way ) Likewise, the possessive pronoun `ihr' in all its declensions can mean either `her' or `their'  In some cases, the disambiguation can be carried out on purely syntactic ground; e g  if `sie' is the subject of a third-person singular verb, it must mean `she'  However, in many case, the disambiguation requires a deeper level of understanding  Thus, it should be possible to construct German Winograd schemas based on the words `sie' or `ihr' that have to be solved in order to translate them into English  For example,
Marie fand fünf verlassene Kätzchen im Keller  Ihre Mutter war gestorben 
(Marie found five abandoned kittens in the cellar  Their mother was dead )
vs 
Marie fand fünf verlassene Kätzchen im Keller  Ihre Mutter war unzufrieden 
(Marie found five abandoned kittens in the cellar  Her mother was displeased )
Another example: French distinguishes between a male friend `ami' and a female friend `amie'  Therefore, in translating the word “friend” into French, it is necessary, if possible, to determine the sex of the friend; and the clue for that can involve an inference that, as far as AI programs are concerned, is quite remote  Human readers are awfully good at picking up on those, of course  In general, with this kind of thing, if you want to break GT or another translation program, the trick is to place a large separation between the evidence and the word being disambiguated  In this case, at the present time, the separation can be pretty small  GT correctly translates “my friend Pierre” and “my friend Marie” as “mon ami Pierre” and “mon amie Marie” and it translates “She is my friend” as “Elle est mon amie”, but rather surpringly it breaks down at “Marie is my friend,” which it translates “Marie est mon ami ” As for something like “Jacques said to Marie, `You have always been a true friend,' ” that is quite hopeless  GT can surprise one, though, in both directions; sometimes it misses a very close clue, as in “Marie is my friend”, but other times it can carry a clue further than one would have guessed 
Acknowledgements
Thanks to Arianna Bisazza, Gerhard Brewka, Antoine Cerfon, Joseph Davis, Gigi Dopico-Black, Nizar Habash, Leora Morgenstern, Oded Regev, Francesca Rossi, Vesna Sabljakovic-Fritz, and Manuela Veloso for help with the various languages and discussion of gendered pronouns 
References
E  Davis, “Qualitative Spatial Reasoning in Interpreting Text and Narrative” Spatial Cognition and Computation, 13:4, 2013, 264-294 
H  Levesque, E  Davis, and L  Morgenstern, “The Winograd Schema Challenge,” KR 2012 
L  Morgenstern, E  Davis, and C  Ortiz, “Report on the Winograd Schema Challenge, 2016”, in preparation 
T  Winograd, “Procedures as a Representation for Data in a Computer Program for Understanding Natural Language," Ph D  thesis, Department of Mathematics, MIT, August 1970  Published as MIT AITR-235, January 1971 
T  Winograd, Understanding Natural Language, Academic Press, 1972 

ground_truth: SVM with unigram, bigram, and trigram features, SVM with average word embedding, SVM with average transformed word embeddings, CNN, ecurrent Convolutional Neural Networks, SVM and deep learning models with comment information

SVM with unigram, bigram, trigram features, with average word embedding, with average transformed word embeddings, CNN and RCNN, SVM, CNN, RCNN with comment information

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What language do they explore?
Context 4:  Since IWSLT participators use different data pre-processing methods, we reproduce several competitive baselines based on the ESPnet BIBREF12 for a fair comparison 
Experiments ::: Dataset ::: Speech translation data:
The organizer provides a speech translation corpus extracting from the TED talk (ST-TED), which consists of raw English wave files, English transcriptions, and aligned German translations  The corpus contains 272 hours of English speech with 171k segments  We split 2k segments from the corpus as dev set and tst2010, tst2013, tst2014, tst2015 are used as test sets 
Speech recognition data: Aside from ST-TED, TED-LIUM2 corpus BIBREF13 is provided as speech recognition data, which contains 207 hours of English speech and 93k transcript sentences 
Text translation data: We use transcription and translation pairs in the ST-TED corpus and WIT3 as in-domain MT data, which contains 130k and 200k sentence pairs respectively  WMT2018 is used as out-of-domain training data which consists of 41M sentence pairs 
Data preprocessing: For speech data, the utterances are segmented into multiple frames with a 25 ms window size and a 10 ms step size  Then we extract 80-channel log-Mel filter bank and 3-dimensional pitch features using Kaldi BIBREF14, resulting in 83-dimensional input features  We normalize them by the mean and the standard deviation on the whole training set  The utterances with more than 3000 frames are discarded  The transcripts in ST-TED are in true-case with punctuation while in TED-LIUM2, transcripts are in lower-case and unpunctuated  Thus, we lowercase all the sentences and remove the punctuation to keep consistent  To increase the amount of training data, we perform speed perturbation on the raw signals with speed factors 0 9 and 1 1  For the text translation data, sentences longer than 80 words or shorter than 10 words are removed  Besides, we discard pairs whose length ratio between source and target sentence is smaller than 0 5 or larger than 2 0  Word tokenization is performed using the Moses scripts and both English and German words are in lower-case 
We use two different sets of vocabulary for our experiments  For the subword experiments, both English and German vocabularies are generated using sentencepiece BIBREF15 with a fixed size of 5k tokens  BIBREF9 inaguma2018speech show that increasing the vocabulary size is not helpful for ST task  For the character experiments, both English and German sentences are represented in the character level 
For evaluation, we segment each audio with the LIUM SpkDiarization tool BIBREF16 and then perform MWER segmentation with RWTH toolkit BIBREF17  We use lowercase BLEU as evaluation metric 
Experiments ::: Baseline Models and Implementation
We compare our method with following baselines 
Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder  It is trained from scratch on the ST-TED corpus 
Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained  The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus  The MT model has a text encoder and decoder with the same architecture of which in TCEN  It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data 
Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting  In the first two settings, we train the model with $\alpha _{st}=0 75$ while $\alpha _{asr}=0 25$ or $\alpha _{mt}=0 25$  For many-to-many setting, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$   For MT task, we use only in-domain data 
Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models  Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation  Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model  Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder  For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model  The Triangle model is fine-tuned under their multi-task manner 
All our baselines as well as TCEN are implemented based on ESPnet BIBREF12, the RNN size is set as $d=1024$ for all models  We use a dropout of 0 3 for embeddings and encoders, and train using Adadelta with initial learning rate of 1 0 for a maximum of 10 epochs 
For training of TCEN, we set $\alpha _{asr}=0 2$ and $\alpha _{mt}=0 8$ in the pre-training stage, since the MT dataset is much larger than ASR dataset  For fine-tune, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$, same as the `many-to-many' baseline 
For testing, we select the model with the best accuracy on speech translation task on dev set  At inference time, we use a beam size of 10, and the beam scores include length normalization with a weight of 0 2 
Experiments ::: Experimental Results
Table TABREF29 shows the results on four test sets as well as the average performance  Our method significantly outperforms the strong `many-to-many+pretrain' baseline by 3 6 and 2 2 BLEU scores respectively, indicating the proposed method is very effective that substantially improves the translation quality  Besides, both pre-training and multi-task learning can improve translation quality, and the pre-training settings (2nd-4th rows) are more effective compared to multi-task settings (5th-8th rows)  We observe a performance degradation in the `triangle+pretrain' baseline  Compared to our method, where the decoder receives higher-level syntactic and semantic linguistic knowledge extracted from text encoder, their ASR decoder can only provide lower word-level linguistic information  Besides, since their model lacks text encoder and the architecture of ST decoder is different from MT decoder, their model cannot utilize the large-scale MT data in all the training stages  Interestingly, we find that the char-level models outperform the subword-level models in all settings, especially in vanilla baseline  A similar phenomenon is observed by BIBREF6 berard2018end  A possible explanation is that learning the alignments between speech frames and subword units in another language is notoriously difficult  Our method can bring more gains in the subword setting since our model is good at learning the text-to-text alignment and the subword-level alignment is more helpful to the translation quality 
Experiments ::: Discussion ::: Ablation Study
To better understand the contribution of each component, we perform an ablation study on subword-level experiments  The results are shown in Table TABREF37  In `-MT noise' setting, we do not add noise to source sentences for MT  In `-weight sharing' setting, we use different parameters in CTC classification layer and source embedding layer  These two experiments prove that both weight sharing and using noisy MT input benefit to the final translation quality  Performance degrades more in `-weight sharing', indicating the semantic consistency contributes more to our model  In the `-pretrain' experiment, we remove the pre-training stage and directly update the model on three tasks, leading to a dramatic decrease on BLEU score, indicating the pre-training is an indispensable step for end-to-end ST 
Experiments ::: Discussion ::: Learning Curve
It is interesting to investigate why our method is superior to baselines  We find that TCEN achieves a higher final result owing to a better start-point in fine-tuning  Figure FIGREF39 provides learning curves of subword accuracy on validation set  The x-axis denotes the fine-tuning training steps  The vanilla model starts at a low accuracy, because its networks are not pre-trained on the ASR and MT data  The trends of our model and `many-to-many+pretrain' are similar, but our model outperforms it about five points in the whole fine-tuning process  It indicates that the gain comes from bridging the gap between pre-training and fine-tuning rather than a better fine-tuning process 
Experiments ::: Discussion ::: Compared with a Cascaded System
Table TABREF29 compares our model with end-to-end baselines  Here, we compare our model with cascaded systems  We build a cascaded system by combining the ASR model and MT model used in pre-training baseline  Word error rate (WER) of the ASR system and BLEU score of the MT system are reported in the supplementary material  In addition to a simple combination of the ASR and MT systems, we also re-segment the ASR outputs before feeding to the MT system, denoted as cascaded+re-seg  Specifically, we train a seq2seq model BIBREF19 on the MT dataset, where the source side is a no punctuation sentence and the target side is a natural sentence  After that, we use the seq2seq model to add sentence boundaries and punctuation on ASR outputs  Experimental results are shown in Table TABREF41  Our end-to-end model outperforms the simple cascaded model over 2 BLEU scores, and it achieves a comparable performance with the cascaded model combining with a sentence re-segment model 
Related Work
Early works conduct speech translation in a pipeline manner BIBREF2, BIBREF20, where the ASR output lattices are fed into an MT system to generate target sentences  HMM BIBREF21, DenseNet BIBREF22, TDNN BIBREF23 are commonly used ASR systems, while RNN with attention BIBREF19 and Transformer BIBREF10 are top choices for MT  To enhance the robustness of the NMT model towards ASR errors, BIBREF24 DBLP:conf/eacl/TsvetkovMD14 and BIBREF25 DBLP:conf/asru/ChenHHL17 propose to simulate the noise in training and inference 
To avoid error propagation and high latency issues, recent works propose translating the acoustic speech into text in target language without yielding the source transcription BIBREF4  Since ST data is scarce, pre-training BIBREF7, multi-task learning BIBREF4, BIBREF6, curriculum learning BIBREF26, attention-passing BIBREF27, and knowledge distillation BIBREF28, BIBREF29 strategies have been explored to utilize ASR data and MT data  Specifically, BIBREF5 DBLP:conf/interspeech/WeissCJWC17 show improvements of performance by training the ST model jointly with the ASR and the MT model  BIBREF6 berard2018end observe faster convergence and better results due to pre-training and multi-task learning on a larger dataset  BIBREF7 DBLP:conf/naacl/BansalKLLG19 show that pre-training a speech encoder on one language can improve ST quality on a different source language  All of them follow the traditional multi-task training strategies  BIBREF26 DBLP:journals/corr/abs-1802-06003 propose to use curriculum learning to improve ST performance on syntactically distant language pairs  To effectively leverage transcriptions in ST data, BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 augment the multi-task model where the target decoder receives information from the source decoder and they show improvements on low-resource speech translation  Their model just consumes ASR and ST data, in contrast, our work sufficiently utilizes the large-scale MT data to capture the rich semantic knowledge  BIBREF30 DBLP:conf/icassp/JiaJMWCCALW19 use pre-trained MT and text-to-speech (TTS) synthesis models to convert weakly supervised data into ST pairs and demonstrate that an end-to-end MT model can be trained using only synthesised data 
Conclusion
This paper has investigated the end-to-end method for ST  It has discussed why there is a huge gap between pre-training and fine-tuning in previous methods  To alleviate these issues, we have proposed a method, which is capable of reusing every sub-net and keeping the role of sub-net consistent between pre-training and fine-tuning  Empirical studies have demonstrated that our model significantly outperforms baselines 

ground_truth: English, French, German 

French, English, Spanish, Italian, Portuguese, Hebrew, Arabic