Azaad@BND at SemEval-2023 Task 2: How to Go from a Simple Transformer Model to a Better Model to Get Better Results in Natural Language Processing
Year: 2023

Abstract:
The purpose of this competition is to extract the named entity (Named-Entity Recognition) within a text. NER is basically a token classification task where each token is classified into one or more predefined categories. For example, persons, locations, corporations, etc. should be extracted and classified from within the text. [1] In this competition, the basis of classification is divided into two categories: general classification and partial classification. For example, each Location category has 4 sub-categories (subclassifications) including: Facility, OtherLOC, HumanSettlement, Station, and all classifications are available and accessible on the competition dataset page [2]. The total classifications are as follows: 2 Related Work

heading: 1 Introduction
text: The purpose of this competition is to extract the named entity (Named-Entity Recognition) within a text.
NER is basically a token classification task
where each token is classified into one or more predefined categories. For example, persons, locations, corporations, etc. should be extracted and classified from within the text. [1]
In this competition, the basis of classification is divided into two categories: general classification and partial classification. For example, each Location category has 4 sub-categories (subclassifications) including: Facility, OtherLOC, HumanSettlement, Station, and all classifications are available and accessible on the competition dataset page [2]. The total classifications are as follows:

heading: 2 Related Work
text: This article is related to the previous year's SemEval-2022 Task 11 article [4], which is to identify the named entity in the general category [PER, LOC, CORP, GRP, PROD, CW] .
About 55 teams have participated in this task and the best team achieved the score of F1=0.91 in the English language that we want. According to what is mentioned in the article, "most of the teams have used external databases such as Wikipedia, Gazetteer. Also, they have been more interested in the pre-trained XLM-RoBERTa model ."
2184

heading: 3 Data
text: The data set used for training and developing the model is the same data that is provided in the Dataset section [1] of the competition, which are labeled with the CoNLL format and the entities are labeled with the IOB method .
IOB: Inside–outside–beginning (tagging) is a common tagging for tagging tokens in computational linguistics .The B- prefix before a tag specify that the tag is the first of a chunk, and an I- prefix before a tag specify that the tag is inside a chunk. The B- tag specify that a tag is followed by a tag of the same type without O tokens between them. An O tag shows that a token belongs to no entity chunk .
After checking, due to the imbalance of the classes used in the training data, in order for the machine to understand more about the data and classifications, techniques such as Oversampling (increasing data with less density) should be used on the data in the pre-processing stage of the data. The balance of the categories could be maintained .
Also, in order to improve machine learning, we first calculated the ratio of the number of famous words to the length of the sentence, and according to the obtained ratio, we added the sentence to the dataset several time . Here, experimentally, if obtained ratio was more than 90%, 6 times were repeated, if the ratio was more than 80%, 3 times were repeated, and for a ratio greater than 65%, one repetition was sufficient .

heading: 4 Methodology

heading: 4.1 Transformers
text: Transformers are in many cases replacing convolutional and recurrent neural networks (CNN and RNN), which were the most popular types of deep learning models until five years ago. Like most neural networks, transformer models are essentially large encoder/decoder blocks that process the data .
The structure of an encoder layer in a transformer layer is such that each encoder consists of two separate sub-layers, the first layer is the attention layer and the second layer is a feedforward neural network .
The structure of a decoder class, like the encoder, consists of two layers: self-attention and FNN, with the difference that in the decoder there is an intermediate layer called encoder-decoder attention, which helps the machine not focus on the word being learned. Pay attention to related words .
The output of the decoder is a vector, so the last layer of the transformer needs to be a softmax layer, because this layer divides the values into the probability distribution that the output of each element of the vector is in the range of 0 to 1, and the sum of all these elements must be one .
Transformers use positional encoders to label data elements entering and leaving the network. Attention units follow these labels and compute some sort of algebraic map of how each element relates to the other elements. Attention queries are usually executed in parallel by computing a matrix of equations in what is called multi-headed attention .
In this project, a simple transformer model was first used for training based on general classification, which by increasing the number of layers of the transformer model and also better setting things such as the length of tokens, the number of attention heads, and the number of FFN layers, led to an F1 score of about 65%. became. Of course, this model did not give us good results for partial classification where the number of categories was about 36 categories. A better solution is to use a pre-trained esophageal transfer model such as BERT, RoBERTa, ALBERT, …

heading: 4.2 BERT
text: Bidirectional Encoder Representations from Transformers (Bidirectional Encoder Representations from Transformers) or BERT, which uses transformers, is an attention mechanism that learns the textual relationships between words (or subwords) in the text [3] .
The BERT model is actually a group of transformer model encoders that have been studied. In the BERT base model, there are 12 like transformer blocks and 12 attention layers, and in this base model there are 768 hidden nodes in FFN. Fine-tuning method is used here to use this model. In this method, the input of the model is a
list of tokens with a length of 512 tokens. These tokens are passed through the 12 mentioned layers and at the end a vector It is returned as output with a length of 768 .

heading: 4.3 Used in this project
text: In this project, the training was done using the TFAutoModelForTokenClassification model, and the keras library and the Adam method with a learning rate of 0.0001, as well as the loss function from the SparseCategoricalCrossentropy method available in the keras library's loss methods, were used to adjust the optimizer, which is in the next part of the output results. is brought
During the training learning process, the batch_size value was set to 40, that is, the machine divides the data set into 40 groups and updates the weights after learning each of these groups. Considering that the total length of the categories was 17760, the number of categories was equal to 444.
Also, here the number of rounds of the learning process was chosen as 3. The duration of each training process was about 1500 seconds, the specifications of a system on a virtual machine (VMware Workstation Pro) are: • OS: Fedora Linux 36 (Workstation Edition) • Memory: 12 GB • Processor :11th Gen Intel® Core™ i7-1165G74 • Graphics: SVGA3D; build: RELEASE; LLVM;

heading: 5 Results

heading: 5.1 Dev data results
text: The results obtained for the fine-grained section are with precision=0.64, recall=0.54, F1=0.57, and also these results for the coarse-grained section are with precision=0.76, recall=0.68, F1=0.72 .

heading: 5.2 Test data results
text: The results obtained for the fine-grained section are with precision=0.53, recall=0.44, F1=0.47, and also these results for the coarse-grained section are with precision=0.73, recall=0.62, F1=0.67 .
The results obtained according to the match log show that both on the Dev data and on the Test data, the best performance of the machine is PER and LOC classification, and the highest score is on the Medicine and PROD categories .

heading: 6 Conclusion
text: Conclusion Using pre-trained models such as BERT can give us much better output (prediction) about an NLP problem, you just need to fine-tune the model and do a good pre-processing on the data to be closer to the final goal. Let's become and the work will go better. Of course, in the future, we plan to implement these tests on the BERT-large model, which requires more computing time .

heading: Future Work
text: Of course, in the future, we plan to implement these tests on the BERT-large model, which requires more computing time