# Neural Network-Based Bilingual Lexicon Induction for Indonesian Ethnic Languages
## 3 Materials and Methods
### SentencePiece Sequence-to-Sequence with Byte Pair Encoding (BPE)  
The second method we present is SentencePiece (as subword tokenization). According to [PERSON] [22], subword tokenization implements SentencePiece, subword-nmt, and WordPiece model features. Subword vocabulary is built by using the BPE segmentation method to train a SentencePiece tokenization model, which divides words into chunks of characters based on vocabulary size to make pattern detection easier. BPE was added to our research methodology because Indonesian ethnic languages now utilize an alphabet script established by the Dutch, despite having original traditional scripts. Dutch people appeared to assign chunks of alphabets to phonemes of Indonesian ethnic languages when teaching the alphabet to them [1]. As a result, most Indonesian ethnic languages can use the same or similar tokens. Furthermore, with each phonetic development, languages belonging to the same language family descended from the same proto-language. As a result, we assume a phonetic-based strategy is preferable to a character-based method. The number of words to be processed into tokenization is known as the vocabulary size, which in this case, refers to the number of most frequently occurring characters, including symbols like \(<\)/unk\(>\), and whitespace. We employed a wide range of vocabulary sizes. In the character-level sequence-to-sequence model shown in Figure 3, both input and output are split, character by character. As for the SentencePiece sequence-to-sequence model with BPE, as shown in Figure 4, both input and output are split by the BPE method, after initially setting the vocabulary size for each language. BPE constructs a base vocabulary comprising all symbols found in the set of unique words, and then learns merge rules to combine two symbols from the base vocabulary to create a new symbol. It continues to do this until the vocabulary has grown to the required size. The BPE algorithm replaces the data byte pairs that occur most frequently with a new byte until the data can no longer be compressed because no byte pair occurs most frequently. The steps in the training procedure are as follows [23]:  
1. [label=(0)]
2. Gather a huge amount of training data. 3. Determine the vocabulary size. 4. Identify the end of a word, add an identifier (\(<\)/w\(>\)) to the end of each word, and then calculate the word frequency in the text. 5. Calculate the character frequency after dividing the word into characters. Figure 3: Character-level sequence-to-sequence model. 5. Count the frequency of consecutive byte pairs from the character tokens for a predetermined number of rounds and combine the most frequently occurring byte pairings. 6. Repeat step 5 until the required number of merging operations has been performed or the specified vocabulary size is reached. The input text is treated as a sequence of Unicode characters by SentencePiece. Whitespace is also treated like any other symbol. SentencePiece expressly handles whitespace as a fundamental token by first escaping it with the meta symbol \"\(\_\) \" (U + 2581) [22]. Meanwhile, the \"\(\backslash\)n' symbol is the end of a string. The results of the chunks of characters from the BPE will vary when utilizing a larger vocab size. Excluding alphabets, the vocabularies obtained from BPE 40 and 100 are summarized in Tables 1 and 2. Overall, the vocabulary numbers in Indonesia and Minangkabau are the same (7 and 68, respectively). As shown in Table 1, more character pieces are obtained if larger vocabulary sizes are used. The alphabet following the \"\(\_\)\" symbol is a portion of the characters from the beginning of the term in the vocabulary. For example, as indicated in bold in Table 1, the difference between the Minangkabau character pieces, **sa** and _sa**, is that **sa** indicates that the character is not at the beginning of the word. Tokenization results are presented in Table 2, which shows the words in Minangkabau and Indonesian turned into pieces of characters from BPE.