저작자표시-비영리-변경금지 2.0 대한민국이용자는 아래의 조건을 따르는 경우에 한하여 자유롭게l이 저작물을 복제, 배포, 전송, 전시, 공연 및 방송할 수 있습니다.

다음과 같은 조건을 따라야 합니다:저작자표시. 귀하는 원저작자를 표시하여야 합니다.

비영리. 귀하는 이 저작물을 영리 목적으로 이용할 수 없습니다.

변경금지. 귀하는 이 저작물을 개작, 변형 또는 가공할 수 없습니다.

ll

귀하는, 이 저작물의 재이용이나 배포의 경우, 이 저작물에 적용된 이용허락조건을 명확하게 나타내어야 합니다.
저작권자로부터 별도의 허가를 받으면 이러한 조건들은 적용되지 않습니다.

저작권법에 따른 이용자의 권리는 위의 내용에 의하여 영향을 받지 않습니다.
이것은 이용허락규약(Legal Code)을 이해하기 쉽게 요약한 것입니다.
Disclaimer

Master of Science

Development of a Construction Specialized
Pretrained Language Model

February 2022

Department of Civil & Environmental Engineering
The Graduate School
Seoul National University

Geonwoo Kim

Development of a Construction Specialized
Pretrained Language Model지도교수 지 석 호이 논문을 공학석사학위논문으로 제출함
2022년 2월서울대학교 대학원건설환경공학부김 건 우김건우의 석사학위논문을 인준함
2022년 2월

위 원 장

(인)

부위원장

(인)

위

(인)

원

Abstract

Development of aConstruction Specialized
Pretrained Language Model

Geonwoo Kim
Department of Civil & Environmental Engineering
The Graduate School
Seoul National University

Due to the nature of construction fields, various irregular text data are
generated, and natural language processing is being used in many studies toanalyze these data. However, previous studies have limitations that individualmodels should be created for the study to utilize models that have not been
pretrained and lots of labeled data to learn each model is required. On the other
hand, there is a difference in the case of pretrained language model that
pretraining using unlabeled data in the early days makes a basic model, andthen various tasks can be performed only with simple finetuning without
creating individual models.
In recent years, some studies have used the pretrained language model, butthe pretrained language model used was taught based on general terms, not theterm used mainly in the construction field, so there was a limitation in terms ofaccuracy in analyzing terms of construction.
In order to solve these limitations, this research collected text data used in
the construction field and built a construction corpus, and developed andverified a construction specialized pretrained language model by pretraining it.
This research consists of two main stages. First, develop a pretrained language
model for construction specialization through data collection and comparison
between pretrained language models according to corpus. Second, thesuperiority of the developed model was verified through experiments andcomparisons in terms of accuracy, efficiency, and adaptability between thedeveloped pretrained language model and the previously un-pretrained
language model.
The results of these experiments show that the pretrained language model
developed in this research is superior in terms of accuracy, efficiency, andadaptability compared to the language model that has not been pretrained, andthe accuracy is higher than that of the language model that has been pretrainedin general corpus. It is expected that the developed construction specialized
pretrained language model can be used to perform various natural language
processing tasks in the construction field.

Keywords:

Pretrained Language Model, Natural Language Processing,
Corpus, ELECTRA

Student Number: 2020-20589

Contents

Chapter 1. Introduction........................................................1
1.1 Research Background............................................................... 1
1.2 Problem Statement ................................................................... 3
1.3 Research Objectives and Scope ............................................... 4
1.4 Dissertation Outline ................................................................. 6

Chapter 2. Theoretical Background and Related Works...............................................................................8
2.1 Pretrained Language Model ...................................................... 9
2.2 Natural Language Processing in Construction ........................ 14

Chapter 3. Construction Specialized Pretrained
Language Model ..................................................................18
3.1 Pretrained Language Model Building ..................................... 20
3.2 Model Comparison and Validation .......................................... 27
3.3 Summary of the methodology ................................................. 38

Chapter 4. Results and Discussions ...................................39
4.1 Construction specialized pretrained language model .............. 40
4.2 Comparison and validation results .......................................... 45
4.3 Discussion ............................................................................... 52i

Chapter 5. Conclusion ........................................................53
5.1 Summary and Contributions.................................................... 53
5.2 Limitation and Future Study ................................................... 55

Bibliography ......................................................................... 56
Abstract (Korean) ................................................................. 64ii

List of Tables
Table 3.1

Summary of collected text data .................................................. 23

Table 3.2

Hyperparameter of the ELECTRA model .................................. 24

Table 3.3

Byte Pair Encoding (BPE) example ........................................... 25

Table 3.4

Python Library list ...................................................................... 26

Table 3.5

Accident case data example ....................................................... 28

Table 4.1

Embedding example ................................................................... 42

Table 4.2

Task#1 results by Corpus ........................................................... 43

Table 4.3

Task#2 results by Corpus ........................................................... 43

Table 4.4

Hyperparameters of the Word2vec model .................................. 45

Table 4.5

Accuracy experiment Task #1 result .......................................... 46

Table 4.6

Accuracy experiment Task #2 result .......................................... 47

Table 4.7

Accuracy experiment result comparison .................................... 47

Table 4.8

Efficiency experiment Task #1 result ......................................... 50

Table 4.9

Efficiency experiment Task #1 result ......................................... 50

Table 4.10 Adaptability experiment Task #1 result ...................................... 51
Table 4.11 Adaptability experiment Task #2 result ...................................... 51iii

List of Figures
Figure 2.1

Process of pretraining and fine-tuning ......................................... 9

Figure 2.2

ELECTRA model performance .................................................. 11

Figure 3.1

Overview of research methodology ........................................... 19

Figure 3.2

General text example .................................................................. 21

Figure 3.3

Construction text example .......................................................... 22

Figure 3.4

Model loss graph ........................................................................ 26

Figure 3.5 NER labeling example................................................................ 30
Figure 3.6 Previous model structure ............................................................ 31
Figure 3.7 Word2vec example ..................................................................... 32
Figure 3.8

Confusion matrix ........................................................................ 35

Figure 3.9

Experiment #1 process ............................................................... 35

Figure 3.10 Experiment #2 process ............................................................... 36
Figure 3.11 Experiment #3 process ............................................................... 37
Figure 4.1

General corpus vocab ................................................................. 40

Figure 4.2

Construction corpus vocab ......................................................... 41

Figure 4.1

General corpus vocab ................................................................. 40

List of Equations
Equation 1. Accuracy..................................................................................... 34
Equation 2. Precision ..................................................................................... 34
Equation 3. Recall .......................................................................................... 34
Equation 4. F1 score ...................................................................................... 34
Equation 5. Micro F1 score............................................................................ 35

iv

Chapter 1. Introduction

1.1 Research Background
Most of the data in the construction field is text data, and various results
can be obtained through analysis of these text data. However, the amount ofthese text data is too large and costly to handle manually, so natural language
processing is essential for analyzing text data in the construction field (Jeongand Kim, 2012; Lee et al., 2016; Liu and El-Gohary 2017). In the meantime,
various studies have been conducted to conduct construction-related tasks using
natural language processing (M.-Y. Cheng et al., 2020; S. Chi & Han, 2013;
Gibb et al., 2014; Rowlinson & Jia, 2015). However, most studies have used amodel rather than a pretrained language model, so it is necessary to develop
individual models to perform each task, and there is a limit to the need for alarge amount of learning data to develop each model.
To solve these limitations, natural language processing techniques using
pre-trained language models have emerged.(Devlin et al., 2018; Howard &Ruder, 2018; Peters et al., 2018) The pretrained language model is a techniquethat uses unsupervised learning from a large-scale corpus that is not labeled tolearn the language model first, and performs additional supervised learning byusing labeled data according to the task that is performed afterwards. Pretrained
language model learns contextual meaning from large capacity corpus, so it

1

shows high learning performance with little additional data. In addition, the task
can be performed only by finetuning without the need to generate an individualmodel according to the task, thereby reducing time and costs required forbuilding the individual model. In various fields such as economy, bio, and law
build their own corpus from text data composed of terms mainly used in eachspecialized field and used it to make pretrained language model (Chalkidis etal., 2020; Clark et al., 2020; Lee et al., 2020). Previous studies havedemonstrated that in specialized fields where general terms and mainly used
terms are different, the performance of the pretrained language model
constructed as a corpus suitable for the specialized field is higher when
performing tasks in each field than in the pretrained language model learned ingeneral terms.
In the construction field, recent research using the pretrained language
model have been conducted. (Li et al., 2021; Xiong et al., 2021; Xue & Zhang,2021). However, there is a limitation that the analysis accuracy of construction
terminology is low because many technical terms that are very different fromthe commonly used terminology are used in the task mainly performed in theconstruction field (Kim & Chi, 2019; Zhong, He, et al., 2020)2

1.2 Problem Statement
Previous studies on using natural language processing for construction
fields mainly focused on un-pretrained model. Therefore, Each task had to be
developed separately to suit the task, and in the process, various labeled data
were required to suit the task. In order to obtain labeled data, it is expensive andtime-consuming to mobilize personnel with expertise in the construction field,
and its applicability is low because only one task can be performed through thecompleted labeled data. Although these problems have been solved in studies
using some pretrained language models, there is a limitation in that the model'sunderstanding of construction terms is poor due to the characteristics that
construction terms are different from general terms. To overcome theselimitations, it is necessary to develop a pretrained language model specialized
in the construction field.

3

1.3 Research Objectives and Scope
This research aims to develop construction specialized pretrained language
model for various kinds of construction related natural language processing
tasks. In addition, the developed construction specialized pretrained language
model is compared with the natural language processing model, which was
mainly used in the construction field, in terms of (1) accuracy (2) efficiency (3)adaptability. And this research set the scope of the research as data related tothe construction of Korean words.

A research methodology and the specific objectives to achieve the primaryobjective are as follows:

1) Objective 1: Text data in the construction field is collected and corpus
in the construction field is constructed using the data. In this process,text data in general fields also enables comparative experiments
between models learned with corpus in the construction field andmodels learned with general corpus through collection and corpus
creation.

2) Objective 2: By training the ELECTRA model based on the corpus ofeach construction field and general field, and comparing eachperformance, proving that the pretrained language model based on the
corpus in the construction field is more suitable for natural language
treatment tasks.

43) Objective 3: The developed construction specialized pretrained
language model is compared with the natural language processing
model, which was mainly used in the construction field, to prove its
accuracy performance.
4) Objective 4: The developed construction specialized pretrained
language model is compared with the natural language processing
model, which was mainly used in the construction field, to prove its
efficiency performance.
5) Objective 5: The developed construction specialized pretrained
language model is compared with the natural language processing
model, which was mainly used in the construction field, to prove its
adaptability performance.

51.4 Dissertation Outline
This dissertation is composed of five chapters and the details for eachchapter are as below.

Chapter 1. Introduction: This chapter covers the backgrounds, problemsof the research, objectives, and scope of the research.
Chapter 2. Theoretical Background and Related Works: This chapter
provides a comprehensive review of using pretrained language model in natural
language processing, studies in which natural language processing was used in
the construction field, and studies in which a pretrained language model wasused in the construction field.
Chapter 3. Construction Specialized Pretrained Language Model:This chapter presents a process of build construction specialized pretrained
language model and describe the process of comparing and verifying thedeveloped model with previous model.
Chapter 4. Results and Discussions: This chapter covers the result ofcomparison of the pretrained language model learned with the construction
corpus and the pre-trained language model learned with the general language
corpus and the results of comparison and verification in three aspects (accuracy,efficiency, adaptability) with the developed construction specialized pretrained
language model and the previous model mainly used in the construction field
are explained.

6

Chapter 5. Conclusions: This chapter summarize achievements,
contributions, limitations of this research, and describe the contents of future
study.

7

Chapter 2. Theoretical Background andRelated Works

This chapter describes a comprehensive review of what is and why using
pretrained language model in natural language processing. In addition, theresearch that developed and used a pretrained language model specialized ineach domain area in domain areas where there is a large difference between
general and mainly used languages is described. It is also described various
studies using natural language processing models that were not pretrained in
the construction field and studies using pretrained language models.

8

2.1 Pretrained Language Model

2.1.1 pretrained language model description
Pretrained language model is a technique that applies transfer learning,which is a method of performing supervised learning for labeled data suitable
for the desired task after performing unsupervised learning for large amount of
unlabeled data (Figure 2.1).

Figure 2.1 process of pretraining and fine-tuning

Before the emergence of the Pretrained language model, the previous
models did not include information according to context, and the same word
was expressed in the same embedding. Peters et al. (2018) developed ‘ELMO(Embedding from Language Model)’ and this pretrained language model
implies the contextual meaning of the whole word sequence, not the individual
word, and it showed excellent results in the processing of homonyms, whichwere the limitations of the existing model. After the development of ELMO,

9

transfer learning has spread in the field of natural language processing.
ULMFiT (Universal Language Model Fine-Tuning) showed that the pretrainedmodel weighted value can be applied to new task, and the data with much lessamount can show the same or superior performance as the existing algorithm(Howard & Ruder, 2018.). Vaswani et al. (2017) developed a Transformer
structure. Transformer model introduced an attachment mechanism toovercome the limitations caused by the serial connection of recurrent models(RNN, LSTM, GRU), and it dramatically improved the performance of theexisting pretrained language model by calculating word representation basedon the whole sequence. Recently, various pretrained language models using
Transformer's Encoder stack and Decoder stack have appeared in the field of
natural language processing. BERT (Bidirectional Encoder Representationfrom Transformers) improved the inefficiency of learning caused by the left-toright form of the existing pretrained language model by using MLM (Masked
Language Model) in the encoder stack of Transformers (Devlin et al., 2018).
Then, based on BERT, Clark et al. (2020) proposed a new pre-training task
called Replaced Token Detection (RTD) to develop ELECTRA. In the case of
ELECTRA model, it has been developed that are much more economical incomputing resources and faster and more effective than existing models, suchas Figure 2.2, compared to BERT and existing language models.
In this research, we collected text data in the construction field and decided
to use ELECTRA, which can learn quickly with less computing resources andrelatively little data, to derive the most efficient results based on the collected
text data.

10

Figure 2.2 ELECTRA model performance112.1.2 Domain specialized pretrained language modelWith the emergence of various pretrained language models, natural
language processing studies using the models have been conducted in various
domain fields. In the field of using general language, the pretrained language
model learned by the existing general text data was used to obtain excellent
performance, but some domain term was relatively low in general term andother fields. Many studies have been made to solve these problems and it isvery time and cost to build this corpus (Roziewski & Kozłowski, 2021).
In some studies, BERT was learned according to domain specific corpus,
which resulted in higher performance when performing domain natural
language tasks. Araci, (2019) developed FinBERT for financial sentiment
analysis. The existing financial sector analysis has a problem of ‘no specialized
language and available data, and the general-purpose model is not effective
cause of the specialized language used in domain’, so to solve this problem,
TRC2-financial, Financial Phrasebank, and FiQA Sentiment dataset was
pretrained to BERT to develop FinBERT. The FinBERT was obtained from thefinancial domain task with SOTA (State-of-the-art) and proved its excellence.
Lee et al., (2020) developed BioBERT for biomedical natural language
processing tasks. BioBERT was pretrained by biomedical domain corpus suchas PubMed, and PMC. Developed BioBERT recorded SOTA in 12 biomedical
natural language processing tasks and performed best average score in 15 tasks.
Chalkidis et al., (2020) developed LEGAL-BERT and It has good performancein the legal domain field. In addition, the training domain corpus from scratch

12

in the process of LEGAL-BERT was demonstrated to achieve better
performance during the adapt domain corpus from pretraining.

13

2.2 Natural Language Processing in Construction

2.2.1 Using Un-pretrained language model for Construction tasks
In the construction field, there were various studies using un-pretrained
language models, not pretrained language models. Most previous studies using
natural language processing tasks were about information extraction (e.g.,
Identify keywords, identify similar documentation), classification (documentclassification, construction risk classification), Named entity recognition(extract risk factor, predict classes of words), Question answering (extract relate
regulation, chatbot).
Liu & El-Gohary, (2017) proposed ontology-based, semi-supervised
conditional random fields (CRF) based information extraction methodology
from bridge inspection reports. Tian et al., (2021) used convolutional neuralnetworks (CNN) for text classification and used term frequency-inverse
document frequency (TF-IDF) for extract construction knowledge. Ren &Zhang, (2021) proposed a semantic rule-based information extraction (IE)
methodology to extract construction execution steps from construction
procedural documents automatically. Goh & Ubeynarayana, (2017) usednatural language processing with various kinds of models (e.g., Support Vector
Machine (SVM), Linear Regression (LR), Random Forest (RF), etc.) for
classify the accident causes and reasons. M. Y. Cheng et al., (2020) developed
hybrid model incorporating Gated Recurrent Unit (GRU) and Symbiotic
Organisms Search (SOS) and named Symbiotic Gated Recurrent Unit (SGRU).

14

In addition, various studies for classification were conducted (Ayhan et al.,2019; N. W. Chi et al., 2016; Fang et al., 2020; Mo et al., 2020; Zhang, 2019;
Zhong, Pan, et al., 2020). Moon et al., (2020) used NER for extract mainelement from bridge inspection report and proposed active learning to reducetime and cost for labeling. H. Liu et al., (2021) used natural language processing
for data preprocessing and encoded to tokenize item descriptions and link pay
items across different catalogs.

However, for most studies that did not use the pretrained language model, alarge amount of labeled data is essential for the task, and there is a limit that thedeveloped model cannot be used for other models.

15

2.2.2 Using pretrained Language Model for Construction tasks

In the construction field, there were several studies using pretrained
language models for natural language processing tasks.
Amer et al., (2021) proposed the first attempt to automate linking lookahead planning tasks to master-schedule activities following an natural
language processing-based multi-stage ranking formulation. This study used a
distance-based matching for candidate generation and a transformer
architecture for final matching. By presenting a list of the top five games with
a 76.5% precision, it proved that the look-ahead planning task helps match
master scheduling activities.
Xue & Zhang, (2021) proposed automated code compliance checking
systems to enable an automated regulatory rule conversion. Accurate Part-ofSpeech (POS) tagging of building code texts is crucial to this conversion.
Therefore, this study used BERT_Cased_Base pretrained language model. Thismodel outperformed the previous SOTA POS taggers.
Zhong, He, et al., (2020) develops a robust end-to-end methodology to
improve the efficiency and effectiveness of retrieving queries pertaining tobuilding regulations. The developed methodology integrates information searchwith a deep learning model of natural language processing to provide accurate
and fast answers to user questions in the building regulation collection.
Li et al., (2021) proposed a new NER neural network model using
pretrained BERT model based on vocabulary enhancement machine reading to
identify plane and overlapped entities in Chinese bridge inspection texts.

16

In the case of these studies, the pretrained language model was used toobtain superior performance compared to the existing previous model.
However, since the pretrained language model was not pretrained based on
construction corpus, there is a limit of accuracy in construction task.
Therefore, this study is to develop a model optimized for natural language
processing in construction field by creating a pretrained language model
learned by corpus in construction field.

17

Chapter 3. Construction Specialized Pretrained
Language Model

This chapter describes the data collection, development of a pretrained
language model, and verification of the developed model. First, in the case oflanguage model development, divided into two steps: (1) collect data and makecorpus, (2) pretraining model.
Collect data and make corpus data is the process of collecting text data related
to construction, and making construction corpus through preprocessing andintegration of the collected data. In this process, a total of 6.6GB of
construction-related text data was collected from various sources such as bridge
precise safety inspection report, construction standard specification,
construction information related laws and enforcement regulations, andconstruction related articles.
For the verification of the developed language model, divided into three steps:(1) Accuracy, (2) Efficiency (3) Adaptability Validation
To compare accuracy, we studied the model developed with the same epoch andthe natural language processing model used mainly in the construction field,
and then experimented to compare the F1 score of the two models. To verify
the efficiency, the amount of data was gradually increased from 10% to 100%of the total data 10 times, and the F1 score was compared according to theamount of data. In order to verify the applicability, the accuracy of the existing18

model was calculated by F1 Score by cross-verification method of introducing
the results from one task to the embedding stage to another task.

Overview of research methodology is like Figure 3.1.

Figure 3.1 Overview of research methodology

193.1 Pretrained Language Model Building

3.1.1 Collect data and make corpus
It is a step of constructing corpus based on data collected and collected tolearn language model in advance. The entire step for Collect data and makecorpus consists of (1) collecting text data for each field (2) preprocessing anddata extraction (3) constructing three different corpus.
(1) Collecting text data for each field.
Text data consisting of general language are NSMC(Naver Sentiment
Movie Corpus), Korean corpus dataset(Korean spoken language, octopus,
newspaper, hot speech, etc), ‘CheongWaDae’ national petition comment data,
chatbot question and answer fair, KcBERT learning comment. All of the data
are open source and data can be provided through a simple data utilization plansubmission. An example of text data in a general language is the same as Figure
3.2.

20

Figure 3.2 General text example

The collected construction text data consists of safety diagnosis report,
construction standard specification, construction information related laws andenforcement regulations, safety newspaper, construction related articles,
construction field papers, construction term dictionary. Among them, data werecollected through web crawling in the case of construction information related
laws and enforcement regulations, safety newspapers, and construction related
articles, and data were collected through cooperation of related organizationsin the remaining data. In the case of web crawling, we used the ‘BeautifulSoup’and ‘Selenium’ of Python Library. We used the news search query of Naver
portal, which is the most used search site in Korea. Based on the 'construction'
keyword, we collected news data for 15 years from 2005 to 2020. The
construction text data collected example is equal to Figure 3.3.

21

Figure 3.3 Construction text example(2) Preprocessing and data extraction
For preprocessing text data, ‘soynlp’ and ‘kss (Korean Sentence splitter)’,which are Python libraries, were used. Preprocessing carried out the removal of
Chinese characters, special characters and open characters, the removal of URL
patterns, and the removal of repeated letters.
In order to reduce the variables according to the amount of data, the total
amount of general text data and construction text data was unified to 6.6GB
through Random sampling. The summary of the collected general text data andconstruction text data is equal to Table 3.1

22

Table 3.1 Summary of collected text data일반 데이터(General)건설 데이터(Construction)

Naver 영화감성분석(NSMC)

0.1GB안전진단보고서

0.5GB

모두의 말뭉치 한국어 데이터셋4.7GB

건설공사기준 시방서

0.9GB

청와대 국민청원 댓글 데이터

0.5GB건설정보 관련 법령 및 시행규칙0.1GB

챗봇 문답페어

0.4GB

안전신문, 건설관련 기사

3.3GB

기타(KcBERT 학습용 댓글 등)

0.9GB

기타(건설 논문, 건설용어사전 등)

1.8GB

총합

6.6GB

총합

6.6GB

(3) Corpus creation
In order to derive the optimal model according to the change in the corpus
composition ratio, three corpus were constructed by extracting 0%, 50%, and100% of plain text data and construction text data. In the case of constructing
a corpus that combines 50% general data and 50% construction data, random
sampling was extracted from each data.

23

3.1.2 Pretraining modelThis step is a process of pretraining based on the completed corpus. Pretraining was conducted for a week using four Google TPU V3-8 from TFRC
(TensorFlow Research Cloud), and the basic model performed pretraining bylearning three corpus built ahead of the two types of ELECTRA models
developed by the Google Research Team (small, base). The entire step for the
pretraining model consists of (1) selection of a basic model (2) generation of
Vocab (3) model learning.

(1) Two types of ELECTRA model description
The small model consists of a hidden size 256, an embedding size 128, and abatch size 128. In the case of the base model, it consists of a hidden size 768,
an embedding size 768, and a batch size 256, etc. The parameters of the small
model and the base model are shown in Table 3.2.

Table 3.2 Hyperparameter of the ELECTRA model

24

(2) Vocab creation
To learn a language model based on the corpus, an embedding process thatconverts the corpus into a vector is needed. Vocab was created for thisembedding process.
Vocab for each corpus was established to learn the configured corpus. Vocab
consists of 32000 words each, and Wordpiece, a basic compatible tokenizer
method of the ELECTRA model, was used. Wordpiece is a representative
method used in subword tokenizer and is an algorithm that performed Byte Pair
Encoding (BPE) based on likelihood of the corpus. An example of how BPE
obtains vocab given a sequence ‘abcabc’ is as shown in Table 3.3.

Table 3.3 Byte Pair Encoding (BPE) example

Iteration

Sequence

Vocabulary0a b a b c a b c{a, b, c}1

ab ab c ab c{a, b, c, ab}2

ab abc abc{a, b, c, ab, abc}

3

ababc abc{a, b, c, ab, abc, ababc}

4

ababcabc{a, b, c, ab, abc, ababc, ababcabc}(3) Model training
After constructing vocab by corpus, TPU V3-8 and N1-standard-1 VM(Virtual Machine) were used to study. The Python Library used in the learning
process is the same as Table 3.4.

25

Hyperparameter for model learning is the same as Vocab_Size: 32000,
Num_train_steps: 700,000, Train_batch_size: 256 Learning_rate: 2e-4,
Max_seq_length: 512, No_lower_case. When 100% corpus of construction
text data was trained as the ELECTRA-Base model, it was learned byconverging to 10.5331 loss like Figure 3.4.

Table 3.4 Python Library list

Number

Library version

Number

Library version

1

huggingface-hub==0.0.12

6

soynlp==0.0.493

2

Korpora==0.2.0

7

tensorflow==1.15.0

3

kss==2.5.0

8

tokenizers==0.9.3

4regex==2020.11.13

9torch==1.6.0

5sentencepiece==0.1.91

10transformers==3.5.1

Figure 3.4 Model loss graph26

3.2 Model Comparison and Validation

3.2.1 Task Selection and Data Collection for Model Verification

The step of collecting the task selection and related data for verifying thedeveloped pretrained language model is. In this study, Classification andNamed Entity Recognition (NER) were selected among natural language
processing tasks mainly performed in construction field.

(1) Classification
Text classification is the process of classifying text into word groups. Using
NLP, text classification can automatically analyze text and then assign apredefined set of labels or categories according to the context.
In this experiment, Classification performed a task to classify accident types
based on construction accident case data. The construction accident case data
was provided from Construction Safety Management Integrated Information(CSI) of the National Land Safety Management Agency under the Ministry ofLand, Infrastructure and Transport. The data is accident case data for one year
from July 2019 to July 2020, and consists of a total of 3,719 cases, excluding
data without accident details, which is the main information. Among the total
data, a column on how to conduct text analysis and a column on the type of
personal accident that will provide label information were extracted, and the27extracted data is shown in Table 3.5. There are six types of labels: 'hit object','fall down', 'fall', ‘get stuck’, 'cut', and 'others'.

Table 3.5 Accident case data example
Content

Labeltext

Label

물체에맞음

4

물체에맞음

4

넘어짐

2

기타

0

넘어짐

2

떨어짐

3

끼임

1

···

···

신당교 A2 시스템비계 하부 베이스 플레이트 해체 작업을위한 비계 인상 작업 중에 작업발판과 작업자와의
부딪힘으로인한찰과상이동식 고소작업대 이동 중 작업대 상부가 지붕에 걸려이동이 불가하였으나 이를 인지하지 못하고 더 세게작업대를밀다작업대가넘어져그밑에깔린사고임
일반공안왕근씨가시멘트를어깨에메고가설계단을이동중
뒤로넘어지며허리와늑골을가설계단에부딪히며넘어짐
현장식당 앞 이동통로에 재해자가 의식을 잃고 앉아있는상태에서입에서거품이나오는것을보건관리자가목격하여
119 신고조치
펌프카차량의 아웃트리거 설치를 위해 지반상태 및 여유폭확인중도로에서미끄러져손목골절
상수도공사사진촬영중실족으로인한낙상사고굴삭기 인양 중인 복공판이 흔들리며 장비 몸체와 복공판사이손가락끼임(왼손중지)···(2) Named entity recognition (NER)
Named entity recognition (NER) is a sub-task of information extraction (IE)
that finds and classifies a particular entity in the body or body. NER is also
known simply as entity identification, entity chunking, and entity extraction.
NER is used in many areas of artificial intelligence (AI), including natural
language processing (NLP) and machine learning. Information extraction is28based on NER and uses a model that operates based on grammar or statisticalmodel to find target information. The NER first recognizes the entity as one ofseveral categories: people, location, organization, expression, percentage, andmonetary value. The category is abbreviated to location (LOC), person (PER),
and organization (ORG). When the information category is recognized, theinformation extraction utility extracts the relevant information of the namedentity and constructs a document that the machine can read from theinformation so that other tools can process further to extract meaning.

In NER, we performed task to derive each element based on the bridge safety
inspection report. The bridge safety inspection diagnosis report was provided
by the Bridge Management System (BMS) of the Korea Institute ofConstruction Management and Technology under the Ministry of Land,
Transport and Maritime Affairs. Data extracted a total of 1,650 paragraphs from10 of the bridge safety inspection reports conducted in 2014. The labels toperform NER were divided into four categories: ELEMENT, FACTOR,DAMAGE, and NONE. Labeling software Prodigy was used to label NER data,
and the example labeled on the actual UI is the same as Figure 3.5.

29

Figure 3.5 NER labeling example

30

3.2.2 Set up a previous model for comparative verification

To compare with the developed pretrained language model, the model usedmainly in the natural language processing research in the construction field wasselected. The basic framework of the selected model consists of Bi-LSTM +CRF structure, and word embedding uses word2vec. The detailed structure of
the model is like Figure 3.6.

Figure 3.6 Pervious model structure

Text tokens that enter the input are embed using word2vec. The word2vec
algorithm is based on the distribution hypothesis that word sets with similar
relationships exist in similar vector spaces and that relationships also have31

constant vector values. For example Since the relationship between ‘MAN’ and‘WOMAN’ is similar to that between ‘UNCLE’ and ‘AUNT’, and ‘KING’ and‘QUEEN’, it can be seen that they are distributed with the same vector
difference (Figure 3.7). The word2vec algorithm uses a simple two-tier neuralnetwork to insert words into a numeric vector. Neural networks are trained to
predict specific words when other adjacent words for target words are provided
to the input layer of the network. In particular, the input layer provides a onehot vector of context words for the target word; the value of the input layer ispassed to a hidden layer as much as the size specified by the user. When thevalue of the hidden layer is finally provided to the output layer and the context
of the target word is provided to the network, the weight of the neural network
is adjusted so that the network can successfully predict the target word. After
network training, certain words are mapped to the values of the hidden layer,
which is ultimately defined as the word vector for certain words.

Figure 3.7 Wod2vec example

32

The two hidden layers, the forward layer, and the other reverse layer, havethe same number of LSTM cells as the length of the input sequence or thenumber of words in the input sentence. Each cell of one hidden layer isconnected in one direction so that the model can learn language patterns to
express causality. The output of the hidden layer is a vector sequence of size 4.
Each element of a vector corresponds to each class in which a word is classified
by a model. By interpreting the vector of the 3-output layer as the logit ofprediction for each word, the model classifies each word in the input sentence
as the logit largest class among the four classes. The model's loss function usesa softmax cross entropy function normalized to the length of the input statement.
After classification, go through the Conditional Random Field (CRF) layer.
Adding a CRF layer allows the model to consider the dependence between
predictive object names, that is, labels. The output value past the activation
function for all words is the input of the CRF layer, and the CRF layer predicts
the sequence with the highest score for the label sequence. This reflects the twoway context of the output label.

33

3.2.3 Verification of Accuracy

The developed pretrained language model and the previous model arecompared and verified in terms of accuracy in this part. In this step, two tasks
are performed for the pretrained language model and the previous model, andthe results are compared. The process for the accuracy comparison experimentis the same as Figure 3.9. In this experiment, four types of evaluation metrics
were used to evaluate the performance of the pretrained language model andthe previous model: accuracy, precision, recall, F1 score.
Accuracy is the most intuitive performance measure, and it is simply aratio of correctly predicted observation to the total observations. (Eq. 1)
Precision is the ratio of the correctly predicted instances among the retrieved
instances (Eq. 2). Recall is the ratio of the total amount of relevant instancesamong the instances actually retrieved (Eq. 3). Lastly, F1 score is a harmonicmean of precision and recall, and this metric is used due to the trade-off thatexists between precision and recall (Eq. 4).

𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =𝑇𝑃 + 𝑇𝑁𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝑅𝑒𝑐𝑎𝑙𝑙 =𝐹1 − 𝑠𝑐𝑜𝑟𝑒 =𝑇𝑃
𝑇𝑃 + 𝐹𝑃𝑇𝑃
𝑇𝑃 + 𝐹𝑁

𝐸𝑞. 1
𝐸𝑞. 2
𝐸𝑞. 3

2 ∗ 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∗ 𝑅𝑒𝑐𝑎𝑙𝑙
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙

34

𝐸𝑞. 4

2 ∗ 𝑀𝑖𝑐𝑟𝑜 − 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∗ 𝑀𝑖𝑐𝑟𝑜 − 𝑅𝑒𝑐𝑎𝑙𝑙
𝑀𝑖𝑐𝑟𝑜 − 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑀𝑖𝑐𝑟𝑜 − 𝑅𝑒𝑐𝑎𝑙𝑙

𝑀𝑖𝑐𝑟𝑜 𝐹1 − 𝑠𝑐𝑜𝑟𝑒 =

𝐸𝑞. 5

TRUE

FALSE

TRUE

TP

FP

FASLE

Predicted Value

Actual ValueFN

TN

Figure 3.8 Confusion matrix

In these equations (Eq.1~Eq.5, Figure 3.8), True Positive (TP) refers to thecases where the model get the answer right. False Positive (FP) refers to thecases where the model incorrectly get the answer, i.e. model predicted theanswer is ‘Fall off’ but the real answer is not ‘Fall off’. False Negative (FN)
refers to the cases model predicted the answer except ‘Fall off’ but the realanswer is ‘Fall off’. Cause these experiments have various label class and ourlabels are imbalanced, we use micro F1 score for measurement.

Figure 3.9 Experiment #1 process353.2.4 Verification of Efficiency

The developed pretrained language model is a step to compare and verify
the developed pretrained language model in terms of previous model andefficiency. In this step, we increase the amount of learning data from 10% to
100%, and perform two tasks for the pretrained language model and theprevious model developed in 10 experiments. By the F1 score according toamount of learning data being recorded respectively and comparing thecomparison about the efficiency part of the pretrained language model andprevious model are proceed and the result is compared. The process for theefficiency comparison experiment is the same as Figure 3.10.

Figure 3.10 Experiment #2 process363.2.5 Verification of Adaptability

The developed pretrained language model is a step to compare and verify
the existing natural language processing model and efficiency. In order tocompare the application of the pretrained language model to the existing natural
language processing techniques to several Tasks, the results of the text
preprocessing of one Task, the creation of a tokenizer, and the embedding
learning are applied to the model learning of the other Tasks (4) model
application (5). The application of F1 score is compared by cross-application to
two types of tasks. The process of the application comparison experiment isequal to Figure 3.11.

Figure 3.11 Experiment #3 process373.3 Summary of the methodology
In summary, for developing construction specialized pretrained language
model, we collected text data consisting of general language and text dataconsisting of construction language and built three different corpus based on
this. Based on the three corpus, we learned six models for ELECTRA-Smalland ELECTRA-Base models, respectively.
To verify the model, two NLP tasks were set up to find out each elementin the classification and the bridge precise safety inspection report that fit theaccident type based on the construction accident case, and data were collected
and labeled according to each task.
In addition, the model used mainly in the natural language processing of
the existing construction field was selected through word2vec-based
embedding and Bi-LSTM + CRF was selected as the basic framework. The
model selected is set up as the control group. The comparison with thedeveloped pretrained language model is performed. The comparison andverification of the pretrained language model with the existing model are
performed by experimenting with three things: accuracy, efficiency, andapplicability.

38Chapter 4. Results and Discussions

This chapter covers the results of building construction specialized
pretrained language model and comparing and validation the developed modelwith the previous model. The data used in all the processes are the abovementioned construction accident case classification and bridge safety
inspection report NER, and the effectiveness of the construction specialized
pretrained language model is proved based on the F1 score obtained from theexperiments.

39

4.1 Construction specialized pretrained language model

4.1.1 Embedding according to learning by corpus

In order to derive the optimal model according to the change in the corpus
composition ratio, three corpus were constructed by extracting 0%, 50%, and100% of plain text data and construction text data.
Based on the three corpus, vocab with 32,000 words each was createdusing Wordpiece tokenizer. An example of Vocab consisting of 100% generaltext data is shown in Figure 4.1. Since it is based on Wordpiece, Token, which
has acquired a likelihood of less than threshold within the corpus, is connectedto the '##' mark.

Figure 4.1 General corpus vocab

40

The vocab of the corpus consisting of 100% of construction-related text data is
the same as Figure 4.2.

Figure 4.2 Construction corpus vocab

The corpus consisting of construction-related text data shows that the words
used mainly in the construction field such as ‘##플랜트’, ‘##시공’,‘철근콘크리트’, ‘##공법’, ‘##균열’ have been learned.
Based on the trained tokenizer, the result of embedding when you put an
example sentence "소성수축균열은 경화하는 과정에서 외부에 수분을41빼앗기면서 발생하는 초기재령균열을 의미하는데, 비표면적이 큰방호벽이나 슬래브에서 크게 발생한다." is like Table 4.1.

Table 4.1 Embedding example

Embedding example"소성수축균열은

Text초기재령균열을
Example경화하는의미하는데,

과정에서

외부에

비표면적이

큰

수분을

빼앗기면서

방호벽이나

발생하는

슬래브에서

크게

발생한다."['[CLS]', '소', '##성', '##수', '##축', '##균', '##열', '##은', '경', '##화하는', '과정에서', '외부GeneralCorpus에', '수분', '##을', '빼앗', '##기', '##면서', '발생하는', '초기', '##재', '##령', '##균', '##열을
', '의미하는', '##데', ',', '비', '##표', '##면적', '##이', '큰', '방', '##호', '##벽', '##이나', '슬','##래', '##브', '##에서', '크게', '발생한다', '[SEP]’]['[CLS]', '소성', '##수축', '##균열', '##은', '경화', '##하는', '과정에서', '외부에', '수', '##분Construction 을', '빼앗', '##기', '##면서', '발생하는', '초기', '##재', '##령', '##균열', '##을', '의미하는',Corpus '##데', ',', '비', '##표', '##면적이', '큰', '방호벽', '##이나', '슬래브', '##에서', '크게', '발생한다', '[SEP]’]

According to the example, words such as ‘소성’, ‘수축’, ‘균열’, ‘경화’,‘방호벽’, '슬래브' are trained when the corpus was trained from construction
text data, but they are not trained when they are based on corpus trained fromgeneral text data.

424.1.2 The Results of Task Performance and Optimal Model
Selection according to the Type of Corpus

As a result of experiment on Task#1 – classification and Task#2-NER according
to corpus, it was confirmed that F1 score increased as corpus with highproportion of construction data such as Table 4.2 and Table 4.3, and F1 score
was higher than ELECTRA-Base model when ELECTRA-Base model wastaught. I confirmed the appearance.

Table 4.2 Task #1 results by Corpus

CorpusGeneral 100%

General 50% +Construction 50%

Construction 100%

Small

73.34

74.16

75.31

Base

77.53

78.18

78.92

Model

Table 4.3 Task #2 results by Corpus

CorpusGeneral 100%

General 50% +Construction 50%

Construction 100%

Small

90.04

91.16

91.34

Base

91.33

91.42

92.24

Model

The results of the experiment on two types of tasks showed that the performanceof the model, which was studied with 100% Construction Corpus in
ELECTRA-Base, was the best with the F1 score of 78.92 in Task#1 and 92.24

43

in Task#2, respectively. Therefore, this model was selected as a construction
specialized pretrained language model.

444.2 Comparison and validation results

4.2.1 Set up a previous model

The description of the previous model set for comparison is as follows.
First, the hyperparameters of the Word2vec model applied for word embedding
are the same as Table 4.4.

Table 4.4 Hyperparameters of the Word2vec model

Hyperparameter

Value

Description

Vector Size

200

The size of word vector

Window Size

10

The number of neighboring words used totrain the word distribution

Minimum Count30

The minimum threshold for each word to
train

Epochs100

The number of iterations

Both task models were used in the same structure, and only the learning of
input-related embedding and tokenizer was different according to each task.
The basic skeleton Bi-LSTM consists of 128 units, and Dropout is set at 0.2. In
addition, the Softmax function was used for activation function. Optimzier usedadam, and loss function used categorical_crossentropy. The overall model
consisted of 128 for batch_size and 30 for epochs. The idealization was stopped
when the optimal model was obtained within the epochs by applying early
stopping.

45

4.2.2 Verification of Accuracy result

To compare the construction specificized pretrained language model with
the Accuracy of the previous model, the experiment for Task #1-classification,
Task #2-NER was performed. For the experiment, the data for each task weredivided into 8:1:1 for the training: test: Validation, which is the ratio usedmainly in the natural language processing field. For task #1-classification,
2,976 data out of a total of 3,719 data were divided into a train set, 371 data
into a test set, and 372 data into a validation set. For task #2-classification,
1,320 data out of 1,650 data were divided into train set, 165 data were divided
into test set, and 165 data were divided into validation set. The test results ofthe pretrained language model are the same as Table 4.5 and Table 4.6.

Table 4.5 Accuracy experiment Task #1 result

Task#1

Precision

RecallF1 score

SupportOthers

0.6786

0.59380.633364

Get stuck

0.80590.80590.805936

Fall down

0.75930.89510.8217

95

Fall

0.89390.831

0.861371

Hit object

0.76910.78970.778871

Cut

0.86240.71470.781735

0.7892372

accuracymacro avg

0.79640.77430.7841372

weighted avg

0.78870.78590.7892372

46

Table 4.6 Accuracy experiment Task #2 result

Task#2

Precision

RecallF1 score

SupportDamage

0.94

0.94

0.94

745

Element

0.93

0.94

0.93

425

Factor

0.84

0.85

0.84

235

micro avg

0.92

0.93

0.92

1405

macro avg

0.90

0.91

0.91

1405

weighted avg

0.92

0.93

0.92

1405

The comparison result of the Pretrained language model and the Previous
model is the same as the Table 4.7.

Table 4.7 Accuracy experiment result comparison

Pretrained language modelPrevious model

Task#1
Classification

78.92

67.56

Task#2
NER

92.24

76.23

In task #1-Classification, the pretrained language model was 11.36 points
higher than the previous model, and the task #2-NER was 16.01 points higher.
In the case of actual classification example, it was confirmed that the previous
model was mispredicted as 'Get stick' in case of 'the moment the connection
part is broken when removing the bottom connection pin for the crawler craneboom extension in the field', while the previous model was well judged as 'Hit
object'. In the case of the example, there are a lot of construction-related terms,

47and the actual example sentence itself does not have words related to 'Hit object',
so it was difficult to predict the correct answer in the previous model. However,in the case of the pretrained language model, contextual information was
grasped and correct answers were predicted.

48

4.2.3 Verification of Efficiency result

To compare the efficiency of the construction specialized pretrained
language model with the previous model, the experiment was performed ontask #1-classification, task #2-NER. In this experiment, the total data wasdivided into 10 parts and experimented with increasing the amount of data used
for learning from 10% to 100% 10 times. The experimental results according
to the change of data amount by each task are the same as Table 4.8 and Table
4.9.
Previous model showed that the accuracy of data increased as the amountof data increased in both task #1-classification and task #2-NER. Especially, it
was found that the amount of data was very low in 10% and 20%, and the resultof learning with 100% data was very low compared to the result of learning,which means that there is a lot of labeled data needed for learning of theprevious model. On the other hand, the Pretrained language model showed
good performance in both Tasks despite the fact that the amount of data was
10%. In case of Task#2, the increase of F1 score as the data increases is thesame as the previous model, but the width is small, and it is possible to learn
enough through a small amount of labeled data. The results of this study showedthat the effectiveness of the pretrained language model was higher than that ofthe previous model

49

Table 4.8 Efficiency experiment Task #1 result

Model
Pretrained language model
Data size

Previous model

10%

78.56

52.34

20%

78.52

53.62

30%

78.44

54.11

40%

78.20

58.00

50%

77.97

60.23

60%

78.35

62.41

70%

78.88

66.79

80%

77.92

67.51

90%

78.16

67.69

100%

78.92

67.56

Table 4.9 Efficiency experiment Task #2 result

Model
Pretrained language model
Data size

Previous model

10%

90.83

60.69

20%

88.88

61.21

30%

88.99

68.93

40%

92.13

73.84

50%

92.06

70.65

60%

91.52

75.44

70%

90.23

76.17

80%

89.83

75.24

90%

91.27

76.84

100%

92.24

76.23

.

50

4.2.2 Verification of Adaptability result

To compare the adaptability of the construction specialized pretrained
language model with the previous model, the experiment was performed ontask #1-classification, task #2-NER.
In this experiment, the results of (1) text preprocessing, (2) tokenizer generation,and (3) embedding learning of one task are applied to (4) model application (5)model learning of another task. Cross-applied to two tasks and compared
adaptability with F1 score. In the case of the pretrained language model, theresults are the same as the results of the accuracy experiment performed earlier
because new embedding is not trained and applied as the task changes. On theother hand, in the case of the previous model, embedding training according totask is performed separately, and as a result, F1 score fell significantly
compared to the existing cross-applied objective result (Table 4.10, Table 4.11).
Table 4.10 Adaptability experiment Task #1 result

Cross-apply

Task#2 → Task#1

Adaptability experimentresultPretrained language model

78.92

78.92

Previous model

61.45

67.56

Model

Table 4.11 Adaptability experiment Task #2 result

Cross-apply

Task#1 → Task#2

Adaptability experimentresultPretrained language model

92.24

92.24

Previous model

47.63

76.23

Model

51

4.3 Discussion
First, it was proved through experiments that the accuracy of the NLP task
in the domain-knowledge part of the pretrained language model was changed
according to the corpus pretraining. The higher the proportion of construction
text data, the better the learning of the language used in the construction field,
and the embedding result according to the example sentence was confirmed.
The results of this study also led to the comparison of the accuracy of two actualtasks. This study proved the necessity of pretrained language model based on
corpus composed of text data in the field and built construction specialized
pretrained language model.
The developed construction specialized pretrained language model and theprevious three experiments on the previous model, which was mainly used in
the construction field natural language treatment, proved that the pretrained
language model is superior in terms of accuracy, efficiency and adaptability
compared to the previous model. In addition, the pretrained language model
was more effective in grasping contextual information through example data.

52

Chapter 5. Conclusion

5.1 Summary and Contributions
In the construction industry, a vast amount of text data such as safety
accident cases, specifications, construction laws, and construction contract
documents are accumulated, and the processing and analysis of these data
consumes a lot of time and money. To solve this problem, various natural
language processing techniques have been used in many previous studies.
However, in the case of natural language processing techniques used in theconstruction field, there is a limit that data analysis is still inefficient because alarge amount of labeled learning data is needed and individual models must be
created according to the task. Although there have been studies on construction
using pretrained language model to overcome the limitations of existing natural
language processing techniques, there are limitations that the accuracy ofconstruction term analysis is not high because it was pretrained based on corpus
composed from text data of general terms, not the term used mainly inconstruction. To overcome these problems, this research develops a
construction specialized pretrained language model. In addition, the developedmodel was compared with the existing natural language processing techniquesin terms of accuracy, efficiency, and applicability.
This research has the following contributions. First, based on the data
related to Korean construction, we have built a Korean construction corpus, and53

confirmed that the corpus is suitable for the natural language processing task inthe construction field compared to the corpus composed of other generallanguages. Second, it proposed a method applicable to natural language
processing in construction fields that solved the limitations of previous studies(e.g., The need for individual models for each task, the need for large amountsof labeled data, and the low accuracy in analyzing construction terminology
because they are not learned by corpus in construction field due to the use oflanguage models that have not been pretrained.). Finally, the developed
pretrained language model can be used in various studies in the future because
it can be used directly through fine adjustment in industrial part.

54

5.2 Limitation and Future StudyIn this research, two different natural language processing tasks relatedwith construction were experimented, but in addition to the two tasks that havebeen tested by but, various natural language processing tasks are being used in
the construction field. In some cases, question answering (QA) related toconstruction regulations is created and used, and similar documents areautomatically found and compared and contrasted. There is a limit that the test
for these various tasks has not been applied. However, given previous studies
in other research fields, it is found that if the performance of both the basic
classification and the NER is excellent, the overall high index is obtained inother tasks, so it is possible to solve it if only the supplementary experiment isperformed.
As a future study, various kinds of NLP tasks such as QA, Text
summarization, chatbot, etc. will be proved that the developed pretrained
language model can be applied to other tasks in addition to the classification
and NER performed earlier and is superior to previous NLP models. And in
addition to Korean construction data, it is also intended to apply to construction
text data in various languages by learning based on English or other languagebased data. Also, In the construction field, such as GLUE test, which is
generally performed in computer science, and KLUE, which is an evaluationindex of Korean natural language processing model, It could be possible to
make test datasets for performance verification of natural language processing
model.

55

Bibliography
Amer, F., Jung, Y., & Golparvar-Fard, M. (2021). Transformer machinelearning language model for auto-alignment of long-term and shortterm plans in construction. Automation in Construction, 132.
https://doi.org/10.1016/j.autcon.2021.103929
Araci, D. T. (2019). FinBERT: Financial Sentiment Analysis with Pre-trained
Language Models. https://arxiv.org/abs/1908.10063v1
Ayhan, B. U., Tokdemir, O. B., & Asce, M. (2019). Accident Analysis forConstruction Safety Using Latent Class Clustering and Artificial Neural
Networks. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001762
Baker, H., Hallowell, M. R., & Tixier, A. J. P. (2020). AI-based prediction of
independent construction safety outcomes from universal attributes.
Automation in Construction, 118.
https://doi.org/10.1016/J.AUTCON.2020.103146
Baker, H., Hallowell, M. R., & Tixier, A. J. P. (2020). Automatically learning
construction injury precursors from text. Automation in Construction,
118. https://doi.org/10.1016/j.autcon.2020.103145
Bilge, E. Ç ., & Yaman, H. (2021). Research trends analysis using text mining
in construction management: 2000–2020. Engineering, Construction and56

Architectural Management. https://doi.org/10.1108/ECAM-02-20210107
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., &Androutsopoulos, I. (2020). LEGAL-BERT: The Muppets straight out of
Law School. 2898–2904. https://doi.org/10.18653/v1/2020.findingsemnlp.261
Cheng, M. Y., Kusoemo, D., & Gosno, R. A. (2020). Text mining-based
construction site accident classification using hybrid supervised machinelearning. Automation in Construction, 118, 103265.
https://doi.org/10.1016/j.autcon.2020.103265
Chi, N. W., Lin, K. Y., El-Gohary, N., & Hsieh, S. H. (2016). Evaluating thestrength of text classification categories for supporting construction field
inspection. Automation in Construction, 64, 78–88.
https://doi.org/10.1016/j.autcon.2016.01.001
Chi, S., & Han, S. (2013). Analyses of systems theory for construction
accident prevention with specific reference to OSHA accident reports.
International Journal of Project Management, 31(7), 1027–1041.
https://doi.org/10.1016/j.ijproman.2012.12.004

57

Chou, C.-L., Chang, C.-H., Lin, Y.-H., Chien, K.-C., Chang, -H, Lin, Y.-H.,& Chien, K.-C. (2020). On the Construction of Web NER Model
Training Tool based on Distant Supervision. ACM Trans. Asian LowResour. Lang. Inf. Process, 19(6), 87. https://doi.org/10.1145/3422817
Clark, K., Luong, M.-T., Brain, G., Le Google Brain, Q. V, & Manning, C. D.
(2020). ELECTRA: Pre-training Text Encoders as Discriminators Rather
Than Generators. https://arxiv.org/abs/2003.10555v1
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pretraining of Deep Bidirectional Transformers for Language
Understanding. NAACL HLT 2019 - 2019 Conference of the North
American Chapter of the Association for Computational Linguistics:
Human Language Technologies - Proceedings of the Conference, 1,4171–4186. https://arxiv.org/abs/1810.04805v2
Elgibreen, H., Faisal, M., Sulaiman, M. Al, Abdou, S., Mekhtiche, M. A.,
Moussa, A. M., Alohali, Y. A., Abdul, W., Muhammad, G., Rashwan,M., & Algabri, M. (2021). An Incremental Approach to Corpus Design
and Construction: Application to a Large Contemporary Saudi Corpus.
IEEE Access, 9, 88405–88428.
https://doi.org/10.1109/ACCESS.2021.3089924

58

Fang, W., Luo, H., Xu, S., Love, P. E. D., Lu, Z., & Ye, C. (2020). Automatedtext classification of near-misses from safety reports: An improved deeplearning approach. Advanced Engineering Informatics, 44, 101060.
https://doi.org/10.1016/J.AEI.2020.101060
Gibb, A., Lingard, H., Behm, M., & Cooke, T. (2014). Construction accident
causality: Learning from different countries and differing consequences.
Construction Management and Economics, 32(5), 446–459.
https://doi.org/10.1080/01446193.2014.907498
Goh, Y. M., & Ubeynarayana, C. U. (2017). Construction accident narrativeclassification: An evaluation of text mining techniques. Accident
Analysis and Prevention, 108, 122–130.
https://doi.org/10.1016/j.aap.2017.08.026
Howard, J., & Ruder, S. (2018). Universal language model fine-tuning for textclassification. In ACL 2018 - 56th Annual Meeting of the Association
for Computational Linguistics, Proceedings of the Conference (LongPapers) (Vol. 1). https://doi.org/10.18653/v1/p18-1031
Kim, T., & Chi, S. (2019). Accident Case Retrieval and Analyses: Using
Natural Language Processing in the Construction Industry. Journal ofConstruction Engineering and Management, 145(3), 04019004.
https://doi.org/10.1061/(asce)co.1943-7862.0001625

59

Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020).
BioBERT: a pre-trained biomedical language representation model forbiomedical text mining. Bioinformatics, 36(4), 1234–1240.
https://doi.org/10.1093/BIOINFORMATICS/BTZ682
Li, R., Mo, T., Yang, J., Li, D., Jiang, S., & Wang, D. (2021). Bridge
inspection named entity recognition via BERT and lexicon augmented
machine reading comprehension neural model. Advanced Engineering
Informatics, 50. https://doi.org/10.1016/j.aei.2021.101416
Liu, H., Kwigizile, V., & Huang, W.-C. (2021). Holistic Framework for
Highway Construction Cost Index Development Based on Inconsistent
Pay Items. Journal of Construction Engineering and Management,
147(7). https://doi.org/10.1061/(ASCE)CO.1943-7862.0002080
Liu, K., & El-Gohary, N. (2017). Ontology-based semi-supervised conditionalrandom fields for automated information extraction from bridge
inspection reports. Automation in Construction, 81, 313–327.
https://doi.org/10.1016/j.autcon.2017.02.003
Mo, Y., Zhao, D., Du, J., Syal, M., Aziz, A., & Li, H. (2020). Automated staff
assignment for building maintenance using natural language processing.
Automation in Construction, 113.
https://doi.org/10.1016/J.AUTCON.2020.103150

60

Mo, Y., Zhao, D., Du, J., Syal, M., Aziz, A., & Li, H. (2020). Automated staff
assignment for building maintenance using natural language processing.
Automation in Construction, 113.
https://doi.org/10.1016/J.AUTCON.2020.103150
Moon, S., Chung, S., & Chi, S. (2020). Bridge Damage Recognition from
Inspection Reports Using NER Based on Recurrent Neural Networkwith Active Learning. Journal of Performance of Constructed Facilities,
34(6), 04020119. https://doi.org/10.1061/(asce)cf.1943-5509.0001530
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., &Zettlemoyer, L. (2018). Deep contextualized word representations.
NAACL HLT 2018 - 2018 Conference of the North American Chapterof the Association for Computational Linguistics: Human Language
Technologies - Proceedings of the Conference, 1, 2227–2237.
https://doi.org/10.18653/v1/n18-1202
Ren, R., & Zhang, J. (2021). Semantic Rule-Based Construction Procedural
Information Extraction to Guide Jobsite Sensing and Monitoring.
Journal of Computing in Civil Engineering, 35(6), 04021026.
https://doi.org/10.1061/(asce)cp.1943-5487.0000971

61

Rowlinson, S., & Jia, Y. A. (2015). Construction accident causality: An
institutional analysis of heat illness incidents on site. Safety Science, 78,
179–189. https://doi.org/10.1016/j.ssci.2015.04.021
Roziewski, S., & Kozłowski, M. (2021). LanguageCrawl: a generic tool forbuilding language models upon common Crawl. Language Resources
and Evaluation, 55(4), 1047–1075. https://doi.org/10.1007/s10579-02109551-7
Suh, Y. (2021). Sectoral patterns of accident process for occupational safety
using narrative texts of OSHA database. Safety Science, 142.
https://doi.org/10.1016/j.ssci.2021.105363
Tian, D., Li, M., Shi, J., Shen, Y., & Han, S. (2021). On-site text classificationand knowledge mining for large-scale projects construction by
integrated intelligent approach. Advanced Engineering Informatics, 49.
https://doi.org/10.1016/j.aei.2021.101355
Toosi, H., Ghaaderi, M. A., & Shokrani, Z. (2021). Comparative study ofacademic research on project management in Iran and the World withtext mining approach and TF–IDF method. Engineering, Construction
and Architectural Management. https://doi.org/10.1108/ECAM-052020-0325

62

Wu, H., Shen, G., Lin, X., Li, M., Zhang, B., & Li, C. Z. (2020). Screening
patents of ICT in construction using deep learning and NLP techniques.
Engineering, Construction and Architectural Management, 27(8), 1891–
1912. https://doi.org/10.1108/ECAM-09-2019-0480
Xue, X., & Zhang, J. (2021). Part-of-speech tagging of building codes
empowered by deep learning and transformational rules. Advanced
Engineering Informatics, 47. https://doi.org/10.1016/j.aei.2020.101235
Zhang, F. (2019). A hybrid structured deep neural network with Word2Vec
for construction accident causes classification. International Journal ofConstruction Management, 0(0), 1–21.
https://doi.org/10.1080/15623599.2019.1683692
Zhong, B., He, W., Huang, Z., Love, P. E. D., Tang, J., & Luo, H. (2020). A
building regulation question answering system: A deep learning
methodology. Advanced Engineering Informatics, 46(October), 101195.
https://doi.org/10.1016/j.aei.2020.101195
Zhong, B., Pan, X., Love, P. E. D., Ding, L., & Fang, W. (2020). Deeplearning and network analysis: Classifying and visualizing accident
narratives in construction. Automation in Construction, 113.
https://doi.org/10.1016/J.AUTCON.2020.103089

63

초

록

건설분야 특성상 다양한 비정형 텍스트 데이터가 발생하고 있으며,
이러한

데이터를

분석하기

위해

자연어처리가

많은연구에서활용되고 있다. 그러나 이전의 연구들은 주로 사전학습 되지 않은언어모델을 활용하기에 연구 수행을 위해 개별 모델을 만들어야하고, 각 모델을 학습시키기 위한 라벨링된 데이터를 많이 필요로한다는 한계점이 있었다. 반면에 사전학습 언어모델의 경우 초기에라벨링 되지 않은 데이터를 이용해서 사전학습시켜 기본 모델을만들고, 이후 개별 모델을 만들 필요없이 간단한 미세조정 만으로다양한 과업을 수행할 수 있다는 차이점이 있다.
최근에는 일부 연구에서 사전학습 언어모델을 활용한 사례도있었으나 사용한 사전학습 언어모델이 건설분야에서 주로 사용하는용어가 아닌 일반적인 용어를 기준으로 학습되었기에 건설분야의용어를 분석하는데 정확도 측면에서 한계가 있었다.
본 연구는 이러한 한계를 해결하기 위해 건설분야에서 사용되는
텍스트

데이터를

수집하여

건설분야

코퍼스를

구축하고,

이를

사전학습 시켜서 건설특화 사전학습 언어모델을 개발 및 검증했다.
연구는 크게 두 가지 단계로 구성되어 있다. 첫째로, 데이터 수집및 코퍼스에 따른 사전학습 언어모델간 비교를 통한 건설특화
사전학습

언어모델을

개발하였다.

둘째로,

개발된

건설특화

사전학습 언어모델과 기존에 주로 사용하던 사전학습 되지 않은언어모델과의 정확성, 효율성, 적용성 측면에서의 실험 및 비교를통해 개발된 모델의 우월성을 검증하였다.

64그 결과 본 연구에서 개발한 사전학습 언어모델이 사전학습되지 않은 언어모델에 비해 정확성, 효율성, 적용성 측면에서 모두
우수함을

보였으며,

일반적인

비해서도

정확도가

더

언어로

높음을

사전학습된

확인하였다.

언어모델에

개발된

건설특화

사전학습 언어모델을 활용하여 건설분야 다양한 자연어처리 과업수행에 활용할 수 있으리라 기대된다.

주요어: 사전학습 언어모델, 자연어처리, 코퍼스, ELECTRA학번: 2020-20589

65