Package: uinauil
UINAUILUINAUIL(Unified Interactive Natural Understanding of the Italian Language) is a Python module for downloading language resources in order to train, evaluate, and analyze natural language understanding systems for the Italian Laguage, inspired by theGLUEandSuperGLUEbenchmarks. It is based on theEuropean Language Grid(ELG) platform, that provides access to Language Technology services and resources from all over Europe.InstallationTheuinauilmodule can be installed via pip / PyPI with:pip install uinauilOtherwise, you can directly download thesource fileand save it on your working folder.DependenciesThe project dependencies are:python 3.6.7elg 0.4.22scikit_learn 1.2.1You can install the project requirements with:pip install -r requirement.txtQuick startHere thebasic functionalitiesof theuinauilpackage:select a task;download training e test set of the task;evaluate the predictions of a model on the standard metrics of the task.Note: theuinauilpackagedoes not contain models, but only datasets fortrainingyour models on common tasks.import uinauil as ul

# load a task, for example 'facta'
task = ul.Task('facta')

# get training and test set of the task
train = task.data.training_set   # training set
test = task.data.test_set        # test set

# train your model on the training set and make prediction on test set
...
predctions = <make predictions on test set>

# evaluate your model on the standard metrics of the task
scores = task.evaluate(predictions)
print(scores)You can see an example of this code on theQuick Startnotebook.UsageHere we introduce all the functionalities of theuinauilpackage.First of all, import the package with:import uinauil as ulFor getting thelist of available tasksuse:ul.taskstaskscontains a dictionary of the available tasks (herethe list with a brief presentation). Eachkeyis the name of the task, while thevaluecontains itsidentifieron theELG platformand thetypeof task.Select a taskYou can select a task (for example'facta') using theTaskmethod:task = ul.Task('facta')This instruction downloads the task data on your local computer into the./datafolder. In order to download the resources you need to use yourELGaccount.Thetaskobject contains 4 instance variables:task_name: name of the task (the same used as key inul.tasks);desc: description of the task from theELG platform;link: link to the original resource on theELG platform;data: dataset of the task, the main content described in the next section.ThedatavariableThedatavariable of Task object contains thedataset, already divided into training and test sets:train = task.data.training_set   # training set
test = task.data.test_set        # test setBoth training and test sets are retrieved inJSON format(a dictionary in Python). Each task uses a different set of keys, then thedatavariable is also provided with several variables containing useful metadata:data.keys: list of all the keys in the dictionary;data.feature_keys: list of keys of the features;data.target_key: key of the target;data.feature_dim: list of dimensionality of each feature (single or multiple), in the order ofdata.feature_keys;data.target_dim: dimensionality of target (single or multiple);data.feature_info: list of brief descriptions of the meaning of each feature, in the order ofdata.feature_keys;data.target_values: list of possible values of the target;data.target_desc: list of brief descriptions of the meaning of each possible value of the target, in the order ofdata.target_values.You can see a sample of all the available metadata or each task on theMetadatanotebook.Train a modelSorry,uinaiuldoes not contain any pre-trained model. You have to use anexternal packageforbuildinga model, then you cantrainit using the training set of a task.Evaluate a modelTheevaluatemethod of the task object takes a list of predictions on the test set and calculates the standard performance metrics on the task.scores = task.evaluate(predictions)
print(scores)The metrics have been chosen according to thetypeof task (describedhere):forsequencetasks, the method returns theaccuracy, as the ratio of hits over all the tokens;forclassificationtasks, the method returnsaccuracyon all classes, andprecision,recallandF1for each single class and their macro average;forpairstasks, the method returns theclassification reportfromscikit-learn.Details on tasksTheuinauilmodule contains6 tasks, divides into the following3 types:sequencetype, where the features set is composed by alist of tokens. It consists of the following2 tasks:facta: theFACTAdataset consists of 169 news stories selected from the Ita-TimeBank, 120 Wikinews articles, and 301 tweets, annotated with eventfactuality information.eventi: theEVENTIcorpus collects news articles and stories annotated withtemporal informationat different levels (i.e. events, temporal expressions, signals and temporal relations) following theIt-TimeML Annotation Guidelines.classificationtype, where the features set is composed by asingle text. It consists of the following3 tasks:haspeede: theHaSpeeDe2dataset collects 8,012 tweets and 500 news headlines annotated for the presence ofhate speech, stereotypes and nominal utterance.ironita: theIronITAdataset collects 4,849 tweets annotated forirony and sarcasm.sentipolc: TheSENTIPOLC 2016dataset contains 9410 tweets annotated forsubjectivity, overall and literal polarity, and irony.pairstype, where the features set is composed by apair of texts. It consists of the followingtask:textualentailment: theTextual Entailmentdataset contains 800 pairs of Italian sentences, extracted from Wikipedia, and annotated for the presence oftextual entailment.Working examplesTheexamplesfolder in this project contains several examples of use of theuinauilmodule:Quick Start: presents the main feature ofuinauil;Metadata: shows the values of the metadata for each task.LeaderboardAs a baseline we used the training and test sets of each task to train and evaluatecommon ML models. The results of the best models are summarized in the following table:RankCreatorModel NameAVGfactaeventihaspeedeironitasentipolctextualentailment1Italian BERT XXL.769.908.936.791.765.675.5412Italian BERT.755.907.916.785.736.646.5383ALBERTO.744.909.925.741.742.621.5294Multilingual BERT.731.909.925.739.709.559.544The leaderboards of the models on each sigle task are available on theLeaderboardsfile.Include a new modelIf you want to add your model to the ranking, please contactValerio Basile.AuthorsTheuinauilpackage has been created byValerio Basile, with the collaboration ofAlessio BoscaandLivio Bioglio.CitationIf you useuinauilin a scientific publication, we would appreciate citations of ourpaper:@inproceedings{basile-etal-2023-uinauil,title="{UINAUIL}: A Unified Benchmark for {I}talian Natural Language Understanding",author="Basile, Valerio  andBioglio, Livio  andBosca, Alessio  andBosco, Cristina  andPatti, Viviana",editor="Bollegala, Danushka  andHuang, Ruihong  andRitter, Alan",booktitle="Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",month=jul,year="2023",address="Toronto, Canada",publisher="Association for Computational Linguistics",url="https://aclanthology.org/2023.acl-demo.33",doi="10.18653/v1/2023.acl-demo.33",pages="348--356"}LicenseGNU General Public License v3.0