## 4 Natural Language Understanding
### RoBERTa  
Developed by Facebook, RoBERTa, the robust and optimised BERT method, is a model based on BERT using an improved methodology for the training process, using 10 times the data used in BERT and improved computational power [59]. It is an optimised method for pretraining NLP systems. RoBERTa is a model in which the system learns to predict sections of hidden text within previously unlabelled linguistic examples using BERT's masking strategy [97]. For RoBERTa training, the BERT hyperparameters are modified and the training target of the next phrase and minibatch training is lined and the learning rates are modified. Meta's RoBERTa model has been used in relevant proposals, to identify patterns in text. RoBERTa uses large amounts of text data for training, approximately 160 GB, including data from books and Wikipedia used by BERT. Sources supplementing the training data include news data (CommonCrawl news, 76 GB), text from websites (38 GB), and stories(31 GB). The training of this data is performed over the course of a day using the latest generation graphics card. This model outperforms its predecessor BERT and XLNet in the GLUE benchmark data. An important point to consider about the RoBERTa model is that if an attempt was made to reduce the model to a smaller model, using techniques such as pouring, quantisation, or distillation, the model would have lower prediction metrics. Currently there are many jobs that use RoBERTa to perform tasks in different areas. For example, in [98] the model is used for the analysis of English language tweets for the COVID-19 classification of adverse pregnancy outcomes and contingency cases, resulting in F1 scores of 93% and 75%, respectively. Furthermore, in the area of sentiment analysis and classification, in [99] ASK-RoBERTa is presented, a pretraining model that allows the prediction of sentiment inclinations from sentences or documents based on adaptive pretraining of the model to sentiment knowledge. In the paper, some rules are developed to perform term and sentiment mining based on grammar and parts of speech. ASK-RoBERTa performs better than previously proposed sentiment analysis models based on BERT and current deep learning models. On the other hand, a model that makes use of RoBERTa and interactive attention networks (IAN) for sentiment analysis for product reviews is proposed in [100], called RoBERTa-IAN. This work proposes the use of low-dimensional vectors to serve as input to a model for the extraction of semantic features of the text and obtaining the hidden representation. In [101], a solution is presented that, by using RoBERTa and conditional random fields (CRF), is used to detect offensive language on a social media post. The model takes into account the existing differences with the general language, in addition to the jargon used, the proposed solution obtained a 66.34% F1 score. Further, in [102] RoBERTa is used to detect sarcastic comments in tweets posted in English. The proposed model is based on a model pretrained with Twitter data and uses a three-layer forward connected neural network. The model obtains an F1 score of 52.6%. Furthermore, in [103] a model based on RoBERTa trained only on Czech language data is presented. This proposal improves on existing multilingual contextualised linguistic representation models, such as multilingual XLM-RoBERTa [104] and SlavicBERT [105], in NLP tasks such as tagging and lemmatisation, dependency parsing, and semantics. In addition, some proposals allow the categorisation of large numbers of digital documents by automatically assigning unlabelled text documents to predefined categories, providing a solution to the growing demand to organise, store, and retrieve these documents accurately and efficiently. For example, in [106] EVI-IBLMM (extended variational inference for inverted Beta-Liouville mixture model) is proposed for text categorisation. Two datasets are used in the proposal: WebKB and 20 Newsgroups, which have four and twenty categories, respectively.