# A C-BiLSTM Approach to Classify Construction Accident Reports
## 2 Related Works  
### Text Mining and Machine Learning  
Text mining refers to obtain valuable information and knowledge from text data, which is a method in data mining [17]. With the fast coming of big data, the use of massive data has been reforming every industry and business, becoming an important production factor [18]. In many cases, the text data is one of the most easily generated data forms, with typical features of unstructured data. Although the unstructured text is easily perceived and handled by humans, it is hard to be understood by machines. The most basic yet important application of text mining is to achieve automatic classification based on text content [19]. Text classification is one of the common tasks in NLP which concerns the method to program computers to process and analyze large amounts of natural language data. In-text classification, a mathematical model is trained by a set of input texts with associated classification tags to have certain generalization ability so that the model can perform a good prediction on the category of other texts in the same domain. It is essential to measure and calculate the similarity of texts.  
Currently, as an effective method for text information management, text classification has been widely applied to multiple fields such as information classification [20], recommendation system [21], and sentiment analysis [22]. Traditional text classification methods generally adopt a machine learning method [23]. Machine learning is the scientific study of algorithms and statistical models to perform a specific task without using explicit instructions but relying on patterns and inference. The mathematical model is built based on sample data (as known as training data) to make predictions. Various types of models have been used and researched for machine learning systems, including Support Vector Machine (SVM) [24], Naive Bayes (NB) [25], K neighbors [26], etc.  
DL is a class of machine learning algorithms that uses multiple layers to progressively extract higher-level features from the raw input [27]. For example, in image processing, lower layers may identify edges, while higher layers may identify the concepts relevant to a human such as faces. DL method has been proved to be effective for feature extraction [28]. A series of DL algorithms such as Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) have been extensively used by researchers in various fields for text classification [29; 30].  
The transformer architecture is proposed to deal with the difficulty of parallel training related to BiLSTMs. It completely replaces LSTMs by the so-called attention mechanism [31]. With attention, an entire sequence is treated as a whole, therefore it is much easier to train in parallel. There are many variants of attention mechanisms, such as co-attention mechanism [32] and self-attention [33]. As a transformer-based approach, BERT (Bidirectional Encoder Representations from Transformers) [34] has achieved amazing results in many language understanding tasks, including the tasks of text classification. However, those advanced models usually have a large size with many parameters, making a higher cost in training.