dataesr/scientific-paragraphs-categorization huggingface dataset:
In general, it could be possible to train and fine-tune these custom embeddings on specific datasets to let the model learn the usage of words in specific cases. The information extracted from the OpeNER Sen-Figure  1 : Model architecture timent Lexicon Italian are the word polarity with its confidence and they are concatenated in a vector of length 2 that is one of the input to the first layer of the network. The NER tags and POS tags instead are mapped to randomly initialized embeddings of dimensionality respectively 2 and 4 that are not trained during the model training for the task submission. With more data available it would probably be beneficial to train all the NER, POS and custom embeddings but for this specific dataset the results were comparable and slightly better when not training the embeddings.
This form of integration is similar to the "global temporal pooling" described in (Sander 2014), which is based on the "global average pooling" technique of  (Min et al. 2014). The non linear activations used in the model are Rectified Linear Units (ReLU) for the internal dense layers, hyperbolic tangent (tanh) in the recurrent layers and sigmoid in the output dense layer. In order to contrast overfitting the dropout mechanism has been used after the Fastext embedding dimensionality reduction with rate 0. 5, in both the recurrent layers between each timestep with rate 0. 5 and on the output of the recurrent layers with rate 0. 3. The model has 61,368 trainable parameters and a total of 45,233,366 parameters, the majority of them representing the Fastext embedding matrix (45,000,300). Compared to many NLP models used today the number of trainable parameters is quite small to reduce the possibility of overfitting the training dataset (6,337 examples is small compared to many English sentiment datasets) and also because is compensated by the addition of en-gineered features like polarity dictionary, NER tag and POS tag that help in classifying the examples.
In addition the NER-tagging and POStagging models for Italian included in the Spacy library are applied to the text to compute the additional NER-tags and POS-tags features for each word. To train the model and generate the challenge submission a k-fold cross validation strategy has been applied. The dataset has been divided in 5 folds and 5 different instantiations of the same model (with the same architecture) have been trained picking each time a different fold as validation set (20%) and the remaining 4 folds as training set (80%). The number of training epochs is defined with the early stopping technique with patience parameter equal to 7. Once the training epochs are completed, the model snapshot that achieved the best validation loss is loaded. At the end the predictions from the 5 models have been averaged together and thresholded at 0.
Xileny Seijas Portocarrero You Are My GUide xileny@youaremyguide. com Abstract English. In identifying the different emotions present in a review, it is necessary to distinguish the single entities present and the specific semantic relations. The number of reviews needed to have a complete dataset for every single possible option is not predictable. The approach described starts from the possibility to study the aspect and later the polarity and to create an ensemble of the two models to provide a better understanding of the dataset. Italiano. Nell'identificazione delle diverse emozioni presenti in una recensione è necessario distinguere le singole entità presenti e le singole relazioni semantiche. Il numero di recensioni necessarie per avere un dataset completo per ogni singola opzione possibile non è predicibile.
Introduction With the increase in interactions between users and businesses across different channels and different languages, it becomes increasingly difficult for businesses to respond promptly and effectively in an effective manner. Not all activities can have a team dedicated to public relations and often rely on external agencies that do not know the internal operations of the company. Automating the correct recognition of the various problems can lead to the timely addressing of the same to the persons appointed to solve them. The research was carried out with the dataset provided within the task called ABSITA, Aspectbased Sentiment Analysis at EVALITA 2018 1   (Basile et al. , 2018). The task was a combination of two tasks, Aspect Category Detection (ACD) and Aspect Category Polarity (ACP).
The number of leaves is 250, the learner set as 'Feature', and a the learning rate at 0. 04. The result of the union between the three models could not be submitted to the final evaluation, due to the limit of 2 possible submissions, but reported results higher than 83% in the tests car-ried out after the release of the complete dataset for ASPECT and 75% for POLARITY. Also, the inference is faster than the RNN models.
Aspect Category Detection (ACD) Runs micro precision micro recall micro F1 In the evaluation phase, we can see how the results have given reason to the ensemble of the two results. It is clear that the ACP task (table 4) is the beneficiary of this process, instead of the ACD one (table  3 ) that lost more than one point. The study of the dataset is influenced by the little extension of the training dataset and by the specificity of some terms that could refer to different categories such as the comfort of the room and the quality/price ratio. Various types of data preparation have also been used, including the preservation of special characters, the shape of words (to better identify cities or places written in capital letters), and some SMOTE functions to increase the number of entries but with poor results and noticeable overfitting.
ITAmoji Task This model was initially designed for the ITAmoji task in EVALITA 2018  (Ronzano et al. , 2018). The goal of this task is to predict which of 25 emojis (shown in Table  1 ) is most likely to be in a given Italian tweet. The provided dataset is 250,000 Italian tweets with one emoji label per tweet, and no additional data is allowed for training the models. However, it is allowed to use additional data to train unsupervised systems like word embeddings. All results in the coming subsections were tested on the dataset of 25,000 Italian tweets provided by the organizers. Table  1 : Each of the 25 different emojis used in the ITAmoji task, their labels, and the corresponding percent of samples in the test dataset.
07% 1. 06% Figure  1 : Emojis under consideration and their frequency within the dataset. trees in particular, with Tree Echo State Networks  (Gallicchio and Micheli, 2013) , also referred to as TreeESNs. We follow this approach for solving the ITAmoji task in the EVALITA 2018 competition  (Ronzano et al. , 2018). In particular, we parse the input texts into trees resembling the grammatical structure of the sentences, and then we use multiple TreeESN models to process the parse trees and make predictions. We then merge these models by using an ensemble to make our final predictions.
Preprocessing The provided dataset has been shuffled and split into a training set (80%) and a validation set (20%). We preprocessed the data by first removing any URL from the sentences, as most of them did not contain any informative content (e. g. "https://t. co/M3StiVOzKC"). We then parsed the sentences by using two different parsers for the Italian language: Tint 2  (Palmero Aprosio and Moretti, 2016) and spaCy  (Honnibal and Johnson, 2015). This produced two sets of trees, both including information about the dependency relations between the nodes of each tree. We finally replace each word with its corresponding pretrained FastText embedding  (Joulin et al. , 2016).
Training The training algorithm differs based on the kind of model taken under consideration. We address each of them in the following paragraphs. Models 1-6 The first six models are TreeESNs using a multilayer perceptron as readout. Given the fact that the main evaluation metric for the competition is the Macro F-score, each of the models has been trained by rebalancing the frequencies of the different target classes. In particular, the sampling probability for each input tree has been skewed so that the data extracted during training follows a uniform distribution with respect to the target class. For the readout part we use the Adam algorithm  (Kingma and Ba, 2015)  for the stochastic optimization of the multi-class cross entropy loss function. Models 7-10 Models from 7 to 10 are again TreeESNs, but with a Ridge Regression readout.
The second block of features consists of sentiment lexicon features. As Italian lexicon database, we used MultiWordNet  (Pianta et al. , 2002) , where at each lemma is assigned a positive, negative and neutral score. In particular, we include features based on the prior polarity of words in the tweets. To deal with mixed polarity cases we defined two sentiment variation features so as to capture the simultaneous expression of positive and negative sentiment. We decided to include features related to the polarity of the tweets since emoji could be intuitively categorized into positive and negative and are usually used to enforce the sentiment expressed. The list of sentiment lexicon features is reported in 2. The last group of features is the semantic one, which exploits a Distributional Semantic Model.
Evaluation The goal of the ITAmoji challenge is to evaluate the capability of each system to predict the right emoji associated with a tweet, regardless of its position in the text. Organizers selected a subset of 25 emojis and provided 250,000 tweets for training, each tweet contains only one emoji which is extracted from text and given as a target feature. The training set is very unbalanced since three emojis (i. e. : read heart , face with tears of joy , and smiling face with heart eyes ) represent almost 50% of the whole dataset. For the evaluation instead, the organizers created a test set made up of 25,000 tweets, keeping unchanged the ratio of the different classes over the whole set. The prediction for each tweet is composed by the list of all the 25 emojis ordered by their probability to be associated to the tweet: in this way, it is possible to evaluate the systems according to their accuracy up to a certain position in the rank.
For this particular task, where both training and testing dataset are heavily unbalanced, we think that the micro-F1 score is more suited to capture the performance of the submitted systems since it takes into account the support of each class. There is a result which is particularly interesting that is, the value for the 5-fold using only the logistic regression as a classifier which is particularly high (0,358) and is opposing to the final score. This aspect surely needs further investiga-Semantic Description vec the sum of the vector representations of each word in the tweet simNeg the similarity between t and the negative prototype vector p s simPos the similarity between t and the positive prototype vector p s simSubj the similarity between t and the subjective prototype vector p s simObj the similarity between t and the objective prototype vector p s vecMin the element-wise minimum of the vectors representations of each word in the tweet vecMax the element-wise maximum of the vectors representations of each word in the tweet Table  3 : Semantic Features.
This yielded 1. 1M 100-dimensional word vectors. Finally, we have also conducted experiments with word vectors suggested by the task organizers  (Barbieri et al. , 2016). 2 http://commoncrawl. org/ 3 https://github. com/facebookresearch/ fastText/blob/master/docs/crawl-vectors. md 4 https://doi. org/10. 5281/zenodo. 1467220.
During our experiments with the SemEval 2018 task as part of the NL4AI workshop  (Coman et al. , 2018) , we have found the CNN-based architecture to perform better, while here the RNN was a clear winner. Such discrepancy might suggest that even within the emoji prediction task the effectiveness of different approaches may significantly vary based either on a language of the tweets or based on a way the dataset was constructed. In the future, we would like to investigate this topic further by trying to study differences in.
Abstract English. This document describes the results of our system in the evaluation campaign on the prediction of Emoji in Italian, organized in the context of EVALITA 2018 1   (Ronzano et al. , 2018). Given the text of a tweet in Italian, the task is to predict the emoji most likely associated with that tweet among the 25 emojis selected by the organizers. In this report, we describe the three proposed systems for evaluation. The approach described starts from the possibility of creating two different models, one for the part of categorization, and the other for the part of polarity. And to combine the two models to get a better understanding of the dataset.
Description of the system We first started by cleaning the given data from all the noise information. All the punctuation marks were removed from the text of tweets, and we focused on cleaning the text and removing ambiguities such as shortened words and abbreviations. We substituted all the hyperlinks with a more generic word "LINK" and we did the same with the usernames preceded by '@' (users' tags), after seeing that it was not relevant in the prediction of the most likely emoji for the tweet. We tried removing the stop words from the tweets' text to leave only the words with relevant meaning in it, but the results were poor. It is important to note that despite the significant presence of the dataset the has a meager final F1 score. On the other hand, the has a high F1 score even if only present in 319 items.
, 2015). This method leverages on word embeddings to model lexical polarity by transferring it from entire sentences whose polarity is known. The process is based on the capability of word embeddings to represent both sentences and single words in the same space  (Landauer and Dumais, 1997). First, sentences (here tweets) are labeled with some polarity classes: in  (Castellucci et al. , 2015)  this labeling is achieved by applying simple heuristics, e. g. Distant Supervision  (Go et al. , 2009). The labeled dataset is projected in the embedding space by applying a simple but effective linear combination of the word vectors composing each sentence. Then, a polarity classifier is trained over these sentences in order to emphasize dimensions of the space that are more related to the polarity classes.
Each SVM classifier is built by using the KeLP framework 8  (Filice et al. , 2018). Figure  1  reflects also the learning strategy that has been set up during the training phase: the Irony Classifier was trained on the complete training dataset composed by the entire training set (made of 3, 977 tweets) while the Sarcasm Classifier is trained only on the ironic tweets in the training dataset (made of 2, 023 tweets). A 10fold cross validation strategy was applied to optimize the SVM parameters, while the linear combination of the kernel assigns the same weights to each kernel function. In Table  1  the performances of the Irony Classification task are reported: in the constrained run the UNITOR system ranks 7 th , while in 6 st position in the unconstrained one. For this task the adopted representations were able to correctly determine whether a message is ironic with good precision.
Systems description The dataset provided by Ironita organizers has been split into training set (80% of the documents) and development set (the remaining 20%). We randomly sampled the examples for each category, thus obtaining different sets for training/test set, by keeping the distribution of ironic and nonironic samples through the two sets. We submitted two runs, as the results of the two different systems we developed for each category, called X2C-A and X2C-B. The former has been developed on top of the Scikit-learn library in Python language  (Pedregosa et al. , 2011) , and the latter on top of the WEKA library  (Frank et al. , 2016)  in JAVA language. In both cases, input text has been cleaned with a typical NLP pipeline, involving punctuation (with the exclusion of question/exclamation mark), numbers and stopwords removal.
The training dataset was provided by the organizers: within it, the annotations specify, for each tweet, if it is ironic and if it is sarcastic, marking no nonironic tweet as sarcastic. The non-annotated test dataset was given to the teams to annotate. Each team was allowed a maximum of 4 runs (attempts to annotating the dataset) for either task, each of which was ranked in the leaderboard. Taking part in subtask B implied taking part in subtask A with the same run annotations. Of the maximum of 4 runs for each task, each team was allowed 2 constrained runs (using no other dataset containing irony or sarcasm annotations of tweets or sentences but the provided one) and 2 unconstrained runs (where teams were allowed to use other training data), and taking part in a subtask with an unconstrained run implied taking part in the same subtask with a constrained run as well.
In order to have regularization, dropout layers between the layers of the network of above and a gaussian noise layer applied to the input are used. Their purpose is similar and is to make the neural network better able to generalize, even without a very big training dataset. Particularly, thanks to the gaussian noise applied to the input, during training, the same tweet, read multiple times, does not constitute the same input data and the alteration in the data is propagated through the network. A visualization of the model is given in fig. 1. The depth of the data at every layer isn't shown for simplicity, thus each box is meant to represent a vector (rather than an individual value). The length of the input sequence would actually be larger, but only a part is shown for simplicity. The regularization layers aren't shown.
The two identical models were trained individually by using only the data provided by the task organizers for the training phase: all data was used to train the model for the subtask A and only the ironic tweets were used to train the model for the second part of subtask B (detection of sarcasm in ironic examples). The testing dataset was, therefore, annotated as follows: every tweet which was recognized by the first copy of the model as belonging to the negative class was labeled as non-ironic and non-sarcastic. Every tweet recognized in the positive class was labeled as ironic and also evaluated using the second copy of the model, marking it as non sarcastic when it was recognized in the negative class and as sarcastic otherwise. Therefore, no tweet was marked as sarcastic without being also marked as ironic.
Results After training on the dataset provided by the organizers, an individual run was generated from the testing dataset and used for both subtask B and subtask A. The proposed model ranked 9th among 17 runs (ignoring the baseline ones) in subtask A, as the 5th among 7 teams, considering the best run for each team. The F1-score for the negative class (non-ironic) was 0. 668 (precision 0. 742 and recall 0. 606) and the F1-score for the positive class (ironic) was 0. 722 (precision 0. 666 and recall 0. 789). The score was, thus, 0. 695. This is consistent with the results obtained during the testing of the model (before the actual submission of the results), with a relatively low drop in the F1-scores. As a comparison, the first ranked team, with two runs, ranking first and second, got a score of 0.
The resulting vector was the input of a fully connected layer producing the output of the neural network. The model produced results better of those of the state of the art at the time on 4 out of 7 tasks. In 2015  (Zhang et al. , 2015) , a model more similar to the one proposed was presented. The model used a character-level convolutional neural network for text classification, achieving competitive results. However, it did not use a recurrent layer (and was, in fact, compared with recurrent neural networks) and represented each input character as a one-hot encoded vector (without any additional flags). The model was trained using very big datasets (hundreds of thousands of instances, whereas only 3977 tweets were provided for IronITA) as this works better for character-level neural networks (because other kind of model often depend on pretrained layers and can, thus, use knowledge, for instance that related to the meaning of words, which couldn't be derived from the training dataset alone).
, 2018) : this way we want to investigate if surface features can be used to mitigate topic and genre effects (RQ2). We organise this work as follows: in Section 2 we present an overview of the data set released by the organisers; in Section 3 we describe the exper-imental set up, the model and the features used; in Section 4 we give an overview of the results and finally we conclude our work in Section 5. The contributions of this work for the GxG task for EVALITA 2018 are the following: • we test if a gender prediction model achieving state-of-the-art performances when trained and tested on the same domain can also achieve good results when tested in a cross-domain setting • we experiment with abstract features in order to factor out domain-dependent effects • we release all the code for further reproducibility at https://github.
The subset trees satisfy the constraint that grammatical rules cannot be broken and every tree exhaustively represents a CFG rule. Partial Tree Kernel (PTK)  (Moschitti, 2006)  relaxes this constraint considering partial trees, i. e. , fragments generated by the application of partial production rules (e. g. sequences of non terminal with gaps). The strict constraint imposed by the STK may be problematic especially when the training dataset is small and only few syntactic tree configurations can be observed. The Partial Tree Kernel (PTK) overcomes this limitation, and usually leads to higher accuracy, as shown in  (Moschitti, 2006).
It is the cosine similarity between the word vectors v n 1 and v n 2 associated with the labels of n 1 and n 2 , respectively. τ is called terminal factor and weighs the contribution of the lexical similarity to the overall kernel computation. • else if n 1 and n 2 are nodes sharing the same label, then σ(n 1 , n 2 ) = 1. • else σ(n 1 , n 2 ) = 0. In the challenge we adopt the SPTK in order to implement the K obs function used in the Markovian SVM. This kernel in fact has been showed very robust in the classification of (possibly short) sentences, such as in Question Classification  (Croce et al. , 2011)  or Semantic Role Labeling  (Croce et al. , 2011; Croce et al. , 2012). Dataset.
Experimental Results In iLISTEN, the reference dataset includes the transcriptions of 60 dialogues amounting to about 22, 000 words. The detailed statistics regarding dialogues and turns in the train and test dataset are reported in Table  2. Table  3 : Results of the UNITOR system In the proposed classification workflow each utterance from the training/test material is processed through the SpaCy 3  dependency parser whose outputs are automatically converted into GRCT structures 4  discussed in Section 2. These structures are used within the Markovian SVM implemented in KeLP 5   (Filice et al. , 2015; Filice et al. , 2018). The learning algorithm is based on a SPTK combined with a One-Vs-All multi-classification schema adopted to assign individual utterances to the targeted classes. All the parameters of the specific kernel (i.