# Design of a Semantic Understanding System for Optical Staff Symbols
## 3 Materials and Methods
### High-Level Semantic Understanding Stage
#### 3.3.1 Data Preprocessing  
Data preprocessing using the preprocessing part of the NERA for numeric encoding set \(\mathcal{Y}\) has the purpose and functions as follows:  
* Removal of invalid symbols. The task of this paper is to implement the encoding of the pitch and duration of staff notes during the performance. Among the numerous symbols that affect the pitch and duration of notes are the clefs, the key signatures, the accidentals, and the natural scales, while other symbols are considered invalid symbols within this article. In the preprocessing stage, invalid symbols are removed, and valid symbols are retained. We define the set of valid symbols as \(\mathbb{E}\). The relationships among clefs, key signatures, accidentals, natural scales, the valid symbol set, and the dataset are shown in Equation (1): \[\mathbb{C},\mathbb{L},\mathbb{T},\mathbb{S}\subseteq\mathbb{E}\subseteq \mathcal{Y},\] (1) where clef, key signature, accidental, and natural scale are denoted by \(\mathbb{C}\)={\(Gclef,Fclef,\)\(\ldots,Cclef\)}, \(\mathbb{T}=\{0,D\_S,A\_S,\ldots,C\_F\)}, \(\mathbb{L}=\{sharp,flat,natural,dot\}\), and set \(\mathbb{S}\in[P,Du]\), respectively. Specifically, \(P\) is the space spanned by the natural scale \((C,D,E,F,G,A,B)\), and \(Du\) is spanned by the duration \((1,1/2,1/4,1/8,1/16,1/32,\)\(1/64)\). Further, the element \(0\) in set \(\mathbb{T}\) means there is no key signature and implies that the signature in this line is C major. Each natural scale \(s\in\mathbb{S}\) has two pieces of information that indicate the pitch and duration, respectively. Table 2 shows the relationship between categories and sets. \begin{table}
\begin{tabular}{l l l l} \hline \hline \multicolumn{2}{c}{**Sets**} & \multicolumn{2}{c}{**Categories**} \\ \hline \multirow{6}{*}{\(\mathcal{Y}\)} & & \multirow{6}{*}{\(\mathbb{C}\)} & Gclef, High\_Gclef, DHigh\_Gclef, Lower\_Gclef, DLower\_Gclef, \\  & & & Soprano\_Cclef, M-soprano\_Cclef, Cclef, Tenor\_Cclef, Baritone\_Cclef, \\  & & & Fclef, High\_Fclef, DHigh\_Fclef, Lower\_Fclef, DLower\_Fclef \\  & & & \(\mathbb{T}\) & 0, G\_S, D\_S, A\_S, E\_S, B\_S, F\_S, C\_S, C\_F, G\_F, D\_F, A\_F, E\_F, B\_F, F\_F \\  & & & \(\mathbb{L}\) & Sharp, Flat, Natural, Dot \\  & & & \(\mathbb{S}\) & \(\mathbb{P}\) & \(C,D,E,F,G,A,B\) \\  & & & & **Du** & \(1,1/2,1/4,1/8,1/16,1/32,1/6\) \\ \cline{2-4}  & & & & Rest1, Rest2, Rest4, Rest8, Rest16, Rest32, Rest64 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The relationship between categories and sets. * Sorting of valid symbols. The YOLOv5 algorithm in the LSUS outputs the objects, and each object \(y_{i}\) is unordered with the information \((cls,X,Y,W,H)\), where \({}^{\prime}cls^{\prime}\) denotes the symbol's class; \(X,Y\) denote the Cartesian coordinate values of the center point of the object bounding box; and \(W\), \(H\) denote the width and height. The clef is the first element of each row in the staff. Let its center point coordinate be \((X_{C},Y_{C})\). Denote \(\Delta=D/2\), where \(D\) is the distance between two adjacent clefs' center points. If the symbol \(y_{i}\) is \(Y\in[Y_{C}-\Delta,Y_{C}+\Delta]\), then it goes to the same line. Next, the symbols in the same row are sorted in order by \(X\) from small to large. By this method, all valid symbols are rearranged in a new order, which is the exact order of the symbols when reading the staff. After the preprocessing, the digital information of the staff with \(M\) lines is represented as \(M\) vectors, and each vector has \(J_{m}\) elements.