# Design of a Semantic Understanding System for Optical Staff Symbols
## 3 Materials and Methods
### High-Level Semantic Understanding Stage
#### 3.3.2 Note Reconstructing  
In the process of constructing the staff symbol relationship structure, the understanding of the semantic information of the symbols and the interrelationship between the symbols are what we should focus on. We define \(e(m,j)\) as the semantics of the element \(j\) in the \(m\)th vector, \(e(m,j)\subseteq\mathbb{E}\). As the symbol acts directly or indirectly on the natural scale, it affects the played pitch and duration. As for the entire staff image, we define the global variables \(e(m,0)\) and \(e(m,1)\) for the \(m\)th line, where \(e(m,0)\subseteq\mathbb{C}\), \(e(m,1)\subseteq\mathbb{T}\). In addition, we define the local variables \(v_{1 mn}\) and \(v_{2 mn}\) that affect the \(n\)th note in the \(m\)th line. The variable \(v_{1 mn}\) indicates whether the note is transposed or not and how it should be transposed, i.e., sharp, flat, or natural. The variable \(v_{2 mn}\) indicates whether the note's duration is extended to 1.5 times. In this context, when YOLOv5 outputs the symbol class \({}^{\prime}cls^{\prime}\) as a note, the duration and pitch information of the symbol are expressed as \((p_{mn},du_{mn})\). Thus, the note information in line \(m\) is represented as \([\mathbf{p_{m}},\mathbf{du_{m}}]\), where vectors \(\mathbf{p_{m}}=[p_{m1},p_{m2},p_{m3},\cdots,\)\(p_{mNm}]^{T}\) and \(\mathbf{du_{m}}=[du_{m1},du_{m2},du_{m3},\cdots,du_{mNm}]^{T}\), and where \(Nm\), the number of notes in each line, varies depending on the line. The control information for pitch and duration is expressed as \([\mathbf{v_{1m}},\mathbf{v_{2m}}]\), where vector \(\mathbf{v_{1m}}=[v_{1m1},v_{1m2},v_{1m3},\cdots,v_{1 mNm}]^{T}\) and vector \(\mathbf{v_{2m}}=[v_{2m1},v_{2m2},v_{2m3},\cdots,v_{2 mNm}]^{T}\). The variables \(v_{1 mn}\) and \(v_{2 mn}\) are calculated as shown in Equations (2) and (3):  
\[v_{1m(n+1)}= \begin{cases}0&e(m,j)\
otin\{sharp,flat,natural\}\\ 1&e(m,j)=sharp\\ -1&e(m,j)=flat\\ -e(m,1)&e(m,j)=natural\\ \end{cases}  \tag{2}\]  
\[v_{2 mn}= \begin{cases}0&e(m,j)\
eq dot\\ 1/2&e(m,j)=dot\end{cases}  \tag{3}\]  
In Equation (2), \(n+1\) is the update of the note index. If \(e(m,j)\) is natural, the corresponding note performs the opposite control of the key signature, e.g., F in G major has been raised a semitone, but when there is a natural before an F, \(v_{1 mn}\) controls the note to perform a descending semitone operation. In Equation (3), if \(e(m,j)\) is a dot, the duration of the corresponding note is extended by \(1/2\) of the original duration; otherwise, it is not extended. The specific implementation of the note reconstructing part is shown in Algorithm 2 ([[https://github.com/[PERSON]/Semantic-Understanding-System-for-Optical-Staff-Symbols/tree/main/result/HSUS](https://github.com/[PERSON]/Semantic-Understanding-System-for-Optical-Staff-Symbols/tree/main/result/HSUS)]([https://github.com/[PERSON]/Semantic-Understanding-System-for-Optical-Staff-Symbols/tree/main/result/HSUS](https://github.com/[PERSON]/Semantic-Understanding-System-for-Optical-Staff-Symbols/tree/main/result/HSUS)), accessed on 18 November 2023). ```
0: Vector data. 0: Staff notation relationship structure. 1: Initialize:\(M\gets len(\mathbb{C}\in\mathbb{E})\); \(m\gets 0\);\(j_{m}\gets len(e(m,\,))\); //Maximum value of the line index according to the clef number; initializes the row index and symbolic index. 2:while (\(m\leq M\)) do
3:\(m\gets m+1\);\(j\gets 0\);\(n\gets 0\); // Initializes index \(j\) for symbols and index \(n\) for notes
4:\(e(m,0),e(m,1),j\gets 1\); //Gets the value of the line clef and key signature. 5:while (\(j\leq J_{m}-1\)) do
6:\(j\gets j+1\); //Loop through all valid symbols in the \(m\)th line. 7:if (\(e(m,j)\in\mathbb{L}\)then
8:\(v_{1m(n+1)}\gets value\); //Assign pitch-shifting notation to the \(v_{1 mn}\) of the next note. 9:else
10:if (\(e(m,j)\in\mathbb{S}\)) then
11:\(n\gets n+1\);\((p_{mn},du_{mn})\); //If it is a note, then calculate its value of pitch and duration. 12:\(v_{2 mn}\gets 0\); //Assign the note duration.