dataesr/scientific-paragraphs-categorization huggingface dataset:
Development and Test Data Data released for the shared task comes from different datasets. We re-used the whole SEN-TIPOLC 2014 dataset, and also added new tweets derived from different datasets previously developed for Italian. The dataset composition has been designed in cooperation with other Evalita 2016 tasks, in particular the Named Entity rEcognition and Linking in Italian Tweets shared task (NEEL-IT,  Basile et al. (2016a) ). The multiple layers of annotation are intended as a first step towards the long-term goal of enabling participants to develop end-to-end systems from entity linking to entitybased sentiment analysis  (Basile et al. , 2015). A portion of the data overlaps with data from NEEL-IT  (Basile et al. , 2016a) , PoSTWITA  (Bosco et al. , 2016)  and FacTA  (Minard et al.
(2016a) ). The subsets of data extracted from existing corpora (TW-SENTIPOLC14 and TW-BS) have been revised according to the new annotation guidelines specifically devised for this task (see Section 3. 3 for details). Tweets in the datasets are marked with a "topic" tag. The training data includes both a political collection of tweets and a generic collection of tweets. The former has been extracted exploiting specific keywords and hashtags marking political topics (topic = 1 in the dataset), while the latter is composed of random tweets on any topic (topic = 0). The test material includes tweets from the TW-BS corpus, that were extracted with a specific socio-political topic (via hashtags and keywords related to #labuonascuola, different from the ones used to collect the training material).
The final label was assigned by majority vote on each field independently. With three annotators, this procedure ensures an unambiguous result for every tweet. TW-BS The TW-BS section of the dataset had been previously annotated for polarity and irony 2. The original TW-BS annotation scheme, however, did not provide any separate annotation for overall and literal polarity. The tags POS, NEG, MIXED and NONE, HUMPOS, HUMNEG in TW-BS were automatically mapped in the following values for the SENTIPOLC's subj, opos, oneg, iro, lpos and lneg annotation fields: POS ⇒ 110010; NEG ⇒ 101001; MIXED ⇒ 111011; NONE ⇒ 000000 3 ; HUMPOS ⇒ 1101??; HUM-NEG ⇒ 1011??. For the last two cases, i. e. where iro=1, the same manual annotation procedure 2 For the annotation process and inter-annotator agreement see  (Stranisci et al.
, 2016)  3 Two independent annotators reconsidered the set of tweets tagged by NONE in order to distinguish the few cases of subjective, neutral, not-ironic tweets, i. e. 100000, as the original TW-BS scheme did not allow such finer distinction. The inter-annotator agreement on this task was measured as κ = 0. 841 and a third independent annotation was used to solve the few cases of disagreement. described above was applied to obtain literal polarity values: two independent annotations were provided (inter-annotator agreement κ = 0. 605), and a third annotation was added in a second round in cases of disagreement. Just as with the TW-SENTIPOLC14 set, the final label assignment was done by majority vote on each field. TW-TWITA15 and TW-NEEL-IT For these new datasets, all fields were annotated from scratch using CrowdFlower (CF) 4  , a crowdsourcing platform which has also been recently used for a similar annotation task  (Nakov et al.
, 2016). CF enables quality control of the annotations across a number of dimensions, also by employing test questions to find and exclude unreliable annotators. We gave the users a series of guidelines in Italian, including a list of examples of tweets and their annotation according to the SENTIPOLC scheme. The guidelines also contained an explanation of the rules we followed for the annotation of the rest of the dataset, although in practice these constraints were not enforced in the CF interface. As requested by the platform, we provided a restricted set of "correct" answers to test the reliability of the users. This step proved to be challenging, since in many cases the annotation of at least one dimension is not clear cut. We required to collect at least three independent judgments for each tweet.
Crowdsourced data: consolidation of annotations Despite having provided the workers with guidelines, we identified a few cases of value combinations that were not allowed in our annotation scheme, e. g. , ironic or polarised tweets (positive, negative or mixed) which were not marked as subjective. We automatically fixed the annotation for such cases, in order to release datasets of only tweets annotated with labels consistent with the SENTIPOLC's annotation scheme. 6 oreover, we applied a further manual check of crowdsourced data stimulated by the following observations. When comparing the distributions of values (0,1) for each label in both training and crowdsourced test data, we observed, as mentioned above, that while the assignment of 1s constituted from 28 to 40% of all assignments for the opos/pos/ oneg/neg labels, and about 68% for the subjectivity label, figures were much lower for the crowdsourced data, with percentages as low as 6 (neg), 9 (pos), 11 (oneg), and 17 (opos), and under 50% for subj.
If we measure the number of correctly predicted examples instead of the average of the two classes, the systems perform well (micro F1 of best system is 0. 82). Moreover, performance for irony detection drops significantly compared to SENTIPOLC 2014. An explanation for this could be that unlike SENTIPOLC 2014, at this edition the topics in the train and in the test sets are different, and it has been shown that systems might be modelling topic rather than irony  (Barbieri et al. , 2015). This evidence suggests that examples are probably not sufficient to generalise over the structure of ironic tweets. We plan to run further experiments on this issue, including a larger and more balanced dataset of ironic tweets in future campaigns.
Distant supervision The frequency distribution of classes in the dataset, as shown in Table  1 , seems skewed and not fully representative of the distribution in a statistical sample of tweets: negative tweets are normally much less frequent than positive or neutral ones  (Bravo-Marquez, 2015). To reduce this bias and to increase the size of the training set, we selected additional tweets from our corpus of Italian tweets by means distant supervision. In the first step we selected the tweets belonging to a class (positive, negative, neutral, mixed) via regular expressions. In the second step, the selected tweets are classified by the classifier trained using the task trainset. The silver corpus is built taking the tweets with the matched class between the regular expression system and the classifier.
g. tears, angry and sad). Integris srl cooperated to the task providing a set of 1. 3 million tweets, selected by relying on a lexicon of handcrafted polarized words. This resource is also added to the corpus. We split the training set provided for the Evalita 2016 SentiPolc Task into a train set (5335 tweets), a validation set (592 tweets) and a test set (1482 tweets). This dataset was tokenized and normalized as described in Section 2. 1. For the take of participating to subtask 2, polarity classification, the 13-value annotations present in the datasets were converted into four values: "neutral", "positive", "negative" and "mixed" depending on the values of the fields "opos" and "oneg", which express the tweet polarity, according to the task guidelines 3. We did not take into account the values for "lpos" and "lneg".
The frequency distribution of these classes turns out to be quite unbalanced, as shown in The training set is still fairly small, compared for example to the size of the corpus used in SemEval 2106. The "mixed" class in particular is small in absolute numbers, even though not in percentage value, which makes hard to properly train a ML classifier. Therefore we tried to increase the training set by means of the distant supervision as described above: we selected a maximum of 10,000 tweets for class via regular expressions, then we classified them with the classifier trained with the gold training set. We chose for addition into a silver training set, the tweets which were assigned by the classifier the same class of the regular expression. As reported in Table  2 , the silver dataset remains unbalanced; in particular, no "mixed" example was added to the original train set.
Results and Discussion We evaluated our approach on the dataset provided by the organizers of SENTIPOLC 2016. This dataset is composed by up to 10,000 tweets distributed in training set and test set. Both datasets contain tweets related to political and socio-political domains, as well as some generic tweets 3. We experimented with different configurations for assessing subjectivity, polarity and irony. We sent two runs for evaluation purposes in SENTIPOLC-2016: • run 1. For assessign the subjectivity label a Tensorflow 4  implementation of Deep Neural Ngetwork (DNN) was applied, with 2 hidden layers with 1024 and 512 states, respectively. Then, the polarity and irony labels were determined by exploiting a SVM 5. • run 2. In this run, the bag-of-words were revised to remove words that may have a differ-ent polarity depending on the context (.
, 2009)  heuristic. The labeled dataset is projected in the embedding space by applying a simple but effective linear combination of the word vectors composing each sentence. Then, a polarity classifier is trained over these sentences in order to emphasize those dimensions of the space more related to the polarity classes. The DPL is generated by classifying each word (represented in the embedding through a vector) with respect to each targeted class, using the confidence level of the classification to derive a word polarity signature. For example, in a DPL the word ottimo is 0. 89 positive, 0. 04 negative and 0. 07 neutral (see Table  1 ). For more details, please refer to  (Castellucci et al. , 2015a). This method has two main advantages: first, it allows deriving a signature for each word in the embedding to be used in the CNN; second, this method allows assigning sentiment information to words by observing their usage.
The former (conversational context) is made of the stream of messages belonging to the same conversation in Twitter, while the latter (hashtag context) is composed by tweets preceding a target message and sharing at least one hashtag with it. In  (Vanzo et al. , 2014) , these messages are first classified through a context-unaware SVM classifier. Here, we are going to leverage on contextual information for the selection of pre-training material for the CNN. We select the messages both in the conversation context, and we classify them with a context-unaware classifier to produce the pre-training dataset.
In the case the message is first classified as objective (subjective=False), the remaining classifiers are not invoked. Algorithm 1 Unitor classification workflow. The same CNN architecture is adopted to implement all the three classifiers and tweets are modeled in the same way for the three sub-tasks. Each classifier has been specialized to the corresponding sub-task by adopting different selection policies of the training material and adapting the output layer of the CNN to the sub-task specific classes. In detail, the Subjectivity CNN is trained over the whole training dataset with respect to the classes subjective and objective. The Polarity CNN is trained over the subset of subjec-tive tweets, with respect to the classes neutral, positive, negative and conflict. The Irony CNN is trained over the subset of subjective tweets, with respect to the classes ironic and not-ironic.
As suggested in  (Croce et al. , 2016) , the contextual pre-training (see Section 2. 2) is obtained by considering the conversational contexts of the provided training data. This dataset is made of about 2, 200 new messages, that have been classified with the Unitor-U1 system. This set of messages is adopted to initialize the network parameters. In the following, the system adopting the pre-trained CNNs is called Unitor-U2. The CNNs have a number of hyper-parameters that should be fine-tuned. The parameters we investigated are: size of filters, i. e. , capturing 2/3/4/5-grams. We combined together multiple filter sizes in the same run. The number of filters for each size: we selected this parameter among 50, 100 and 200. The dropout keep probability has been selected among 0. 5, 0. 8 and 1.
Moreover, the pre-training material has been classified with the Unitor-U1 system. It could be the case that the adoption of such added material was not so effective, as instead demonstrated in  (Croce et al. , 2016). In fact, in that work the pre-training material was classified with a totally different algorithm (Support Vector Machine) and a totally different representation  (kernel-based). In this setting, the different algorithm and representation produced a better and substantially different dataset, in terms of covered linguistic phenomena and their relationships with the target classes. Finally, the constrained version of our system, obtained a remarkable score of. 7134, demonstrating that the random initialization of the input vectors can be also adopted for the classification of the subjectivity of a message.
In Table  3  the Polarity Classification results are reported. Also in this task, the performances of the unconstrained systems are higher with respect to the constrained one (. 662 against. 6382). It demonstrates the usefulness of acquiring lexical representations and use them as inputs for the CNNs. Notice that the performances of the Unitor classifiers are remarkable, as the two unconstrained systems rank in 2 nd and 3 rd position. The contribution of the pre-training is not positive, as instead measured in  (Croce et al. , 2016). Again, we believe that the problem resides in the size and quality of the pre-training dataset. In Table  4   Irony, is not a word-level phenomenon but, instead, it is related to sentence or even social aspects. For example, the best performing system in Irony Detection in SENTIPOLC 2014  (Castellucci et al.
We present here only the results of the constrained runs in which only the provided training data were used to develop the systems. The first system was entirely developed by the LACAM research group (all the classes used in the pipeline). After studying the effect of different combinations of features and algorithms on the automatic learning of sentiment polarity classifiers in Italian based on the EVALITA SENTIPOLC 2014 dataset, we applied the best one to the training set of EVALITA 2016 in order to participate to the challenge. The second system was developed using the scikit-learn  (Pedregosa et al. , 2011)  and NLTK libraries  (Bird et al. , 2009)  for building the pipeline and in order to optimize the performance on the provided training set, classifications algorithms and feature sets, different from those used in SentimentWS, were tested.
Experiments Both systems were tested on other domains before applying them to the SENTIPOLC subtask. In the results tables, for each class (positive and negative) 0 represents the value "False" used in the dataset annotations for the specific tweet and class, 1 represents "True", following the task guidelines of Sentipolc 2016. Thus the cell identified by the row positive and the column prec. 0 shows the precision related to the tweets with positive polarity annotations set to False. The meaning of the other cells can be obtained analogously.
SentimentWS Results SentimentWS was tested initially on a dataset of 2000 reviews in Italian language, concerning 558 movies, taken from http://filmup. leonardo. it/. In this case, classification performance was evaluated on 17 different feature settings using a 5fold cross-validation procedure. Equal weight was assigned to all classifiers in the Senti-mentWS committee. Overall accuracy reported in  (Ferilli et al. , 2015)  was always above 81%. When Rocchio outperformed Naive Bayes, accuracy of the committee was greater than that of the components; in the other cases, corresponding to settings that used n-grams, Naive Bayes alone was the winner. Before tackling the EVALITA 2016 SENTI-POLC task, in order to tune the system on a (hopefully) similar environment, we tested our system on the EVALITA 2014 dataset and determined in this way the combination of features that had a better accuracy on this dataset.
Conclusions Looking at the results of the Sentiment Polarity detection subtask we were surprised of the overall performance of the systems presented in this paper since they were simply Text Categorization systems. The only integrations to the original systems, in order to tune their performance on the sentiment polarity detection task, concerned emoticons. In SentimentWS these were included in the feature set and SentiPy was enriched with a classifier created for handling emoticons. Besides the experiments that were executed on the SENTIPOLC dataset, we tested both systems on a dataset of Facebook posts in Italian collected and annotated by a group of researchers in our laboratories. This experiment was important in order to understand whether their performance was comparable to the one obtained in the SENTIPOLC challenge.
, 2013b; Severyn and Moschitti, 2015a). Particularly successful for sentiment classification were Convolutional Neural Networks (CNN)  (Kim, 2014; Kalchbrenner et al. , 2014; Severyn and Moschitti, 2015b; Johnson and Zhang, 2015) , on which our work builds upon. These networks typically have a large number of parameters and are especially effective when trained on large amounts of data. In this work, we use a distant supervision approach to leverage large amounts of data in order to train a 2-layer CNN 2. More specifically, we train a neural network using the following threephase procedure: P 1 : creation of word embeddings for the initialization of the first layer based on an unsupervised corpus of 300M Italian tweets; P 2 : distant supervised phase, where the network is pre-trained on a weakly labelled dataset of 40M tweets where the network weights and word embeddings are trained to capture aspects related to sentiment; and P 3 : supervised phase, where the network is trained on the provided supervised training data consisting of 7410 manually labelled tweets.
2M out of 46. 6M parameters) and are updated during the next two phases to introduce sentiment specific information into the word embeddings and create a good initialization for the CNN. Distant Supervised Phase We pre-train the CNN for 1 epoch on an weakly labelled dataset of 40M Italian tweets where each tweet contains an emoticon. The label is inferred by the emoticons inside the tweet, where we ignore tweets with opposite emoticons. This results in 30M positive tweets and 10M negative tweets. Thus, the classifier is trained on a binary classification task. Supervised Phase During the supervised phase we train the pre-trained CNN with the provided annotated data. The CNN is trained jointly on all tasks of EvalItalia. There are four different binary labels as well as some restrictions which result in 9 possible joint labels (for more details, see Section 3.
2). The multi-task classifier is trained to predict the most likely joint-label. We apply 9-fold cross-validation on the dataset generating 9 equally sized buckets. In each round we train the CNN using early stopping on the heldout set, i. e. we train it as long as the score improves on the held-out set. For the multi-task training we monitor the scores for all 4 subtasks simultaneously and store the best model for each subtask. The training stops if there is no improvement of any of the 4 monitored scores. Meta Classifier We train the CNN using 9-fold cross-validation, which results in 9 different models. Each model outputs nine real-value numbers ŷ corresponding to the probabilities for each of the nine classes. To increase the robustness of the system we train a random forest which takes the outputs of the 9 models as its input.
Finally, we show that Tweet2Check unconstrained runs are overall always better (or almost equal) than the constrained ones. To support our hypothesis, we provide an evaluation of Tweet2Check also on the Sentipolc 2014  (Basile et al. , 2014)  datasets. This is very important for an industrial tool, since it allows to potentially predict well tweets coming from new domains, by keeping in the training set a higher number of examples discussing different topics, and thus to generalize well from the perspective of the final user.
Text modeling with B4MSA B4MSA is a system for multilingual polarity classification that can serve as a baseline as well as a framework to build sophisticated sentiment analysis systems due to its simplicity. The source code of B4MSA can be downloaded freely  1. We used our previous work, B4MSA, to tackle the SENTIPOLC challenge. Our approach learns based on training examples, avoiding any digested knowledge as dictionaries or ontologies. This scheme allows us to address the problem without caring about the particular language being tackled. The dataset is converted to a vector space using a standard procedure: the text is normalized, tokenized and weighted. The weighting process is fixed to be performed by TFIDF (Baeza-Yates and  Ribeiro-Neto, 2011). After that process, a linear SVM (Support Vector Machines) is trained using  10-fold cross-validation (Burges, 1998).
B4MSA + EvoDAG In the polarity task besides submitting B4MSA which is a constrained approach, we decided to generate an unconstrained submission by performing the following approach. The idea is to provide an additional dataset that it is automatically label with positive and negative polarity using the Distant Supervision approach  (Snow et al. , 2005; Morgan et al. , 2004). We start collecting tweets (using Twitter stream) written in Italian. In total, we collect more than 10, 000, 000 tweets. From these tweets, we kept only those that were consistent with the emoticon's polarity used, e. g. , the tweet only contains consistently emoticons with positive polarity. Then, the polarity of the whole tweet was set to the polarity of the emoticons, and we only used positive and negative polarities.
Furthermore, we decided to balance the set, and then we remove a lot of positive tweets. At the end, this external dataset contains 4, 550, 000 tweets, half of them are positive and the another half are negative. Once this external dataset was created, we decided to split it in batches of 50, 000 tweets half of them positive and the other half negative. This decision was taken in order to optimize the time needed to train a SVM and also around this number the Macro F1 metric is closed to its maximum value. That is, this number of tweets gives a good trade-off between time needed and classifier performance. In total there are 91 batches. For each batch, we train a SVM at the end of this process we have 91 predictions (it is use the decision function). Besides these 91 predictions, it is also predicted (using as well the decision function) each tweet with B4MSA.
Results and Discussion This Section presents the results of the IN-GEOTEC team. In this participation we did two runs, a constrained and an unconstrained run with B4MSA system, and only a constrained run with B4MSA + EvoDAG. The constrained run was conducted only with the dataset provided by SEN-TIPOLC'16 competition. For more technical details from the database and the competition in general see  (Barbieri et al. , 2016). The unconstrained run was developed with an additional dataset of 4, 550, 000 of tweets labeled with Distant Supervision approach. The Distant Supervision is an extension of the paradigm used in  (Snow et al. , 2005)  and nearest to the use of weakly labeled data in  (Morgan et al. , 2004). In this case, we consider the emoticons as key for automatic labeling. Hence, a tweet with a high level of positive emoticons is labeled as positive class and a tweet with a clear presence of negative emoticons is labeled as negative class.