# Construction of Cultural Heritage Knowledge Graph Based on Graph Attention Neural Network
## 4 Data Model Construction
### Entity-Relationship Extraction Model Construction
#### 4.1.1 Embedding Layer  
The embedding layer encodes words from natural language into dense mathematical vectors, converting the input text into high-dimensional space vectors that a computer can recognize. This helps the model understand the basic meanings of words. The core structure of the BERT model consists of the stacking of multiple transformer modules, forming a bidirectional self-attention representation model. Each transformer module primarily includes two submodules: the multihead attention mechanism submodule and the feed-forward neural network submodule. These submodules incorporate residual connections and layer normalization. After processing the text through the BERT model, an output vector sequence of the same length as the input text is obtained. These output vectors contain the semantic information of the input text, providing representations for downstream tasks such as entity recognition and relationship extraction. Given the input text, the sentence is encoded as \(U=\{u_{0},u_{1},u_{1},\cdots,u_{n}\}\), where \(u_{v}\) is the \(v\)-th character in the sentence \(U\) and \(n\) is the length of the sequence. Here, \(u_{0}=[\text{CLS}]\) is the start of the sentence; \(u_{n}=[\text{SEP}]\) is the separator of the sentence. This step converts the text sequence into a fixed-length vector, representing the semantic information of the text. After the model input, position embedding and sentence embedding are also required. Position embedding indicates the position of a word in the sentence and sentence embedding represents the vector of the entire sentence or text segment, with markers used to separate and segment the sentence. Through BERT encoding, the sentence's semantic feature vector \(R=\{r_{0},r_{1},r_{1},\cdots,r_{n}\}\) is obtained. After the BERT model assigns word vector encoding, semantic dependency relations are introduced to strengthen the feature representation between word vectors. Semantic dependency relations refer to the grammatical relationships within a sentence, which are used to understand the sentence's structure and meaning, thereby enhancing the expressiveness of word vectors. This paper employs the dependency parser of Stanford CoreNLP [27] to obtain the dependency relationship matrix based on the input sentence. Considering that different dependency relations and dependent words have varying impacts on the grammatical features of a sentence, a graph attention neural network (GAT) model [28] is introduced. In this model, the input sentence is treated as a graph where words are nodes and dependency relations are the connections between these nodes. The graph attention neural network dynamically adjusts the attention weights between different words to extract more comprehensive word and syntactic connections. By using a graph attention neural network (GAT), the dependency tree of a sentence is encoded to extract syntactic information of the words in the sentence, thereby enhancing the word vectors. The dependency parser analyzes the sentence to identify the dependency relations between words in the sentence, such as subject-verb relationships, verb-object relationships, and prepositional-object relationships. Based on the identified dependency relations, a matrix \(M_{n\times n}\) is constructed. According to the input text \(T\) and the relationship matrix \(M_{n\times n}\), a graph with \(n\) nodes is built. The graph attention neural network is defined as \(L\) layers. In the \(l\)-th layer, \(t_{i}^{l}\) represents the vector of node \(i\). The update vector of node \(i\) is represented as \(t_{i}^{(l+1)}\), which is calculated based on the vectors of the connected nodes. The calculation formulas are shown in Equations (1) and (2).