## 2 Materials and Methods
### Loss Function for the Multi-Hot Language Model  
The multi-hot representation language model proposed in this paper uses BERT's training method. The multi-hot LM uses the same masked LM process as BERT, but more than one correct token can exist. Because the multi-hot LM can have more than one answer, the existing loss function was modified. In addition, the final output value was obtained by adding one linear layer without applying the embedding table used for the input to the final layer. The existing masked LM uses Softmax cross entropy (SCE), which is a function that is used for problems with a single correct answer. However, this function is difficult to use in the multi-hot LM proposed in this paper. \[SCE=-\sum_{i}\log(S(O_{i}))y_{i} \tag{3}\]  
Figure 1: The multi-hot representation language model acting on an input value (on the example: “Staycation is trending”). The multi-hot LM can have several correct answers for one masking position. When multiple answers exist, the \(y_{i}\) value increase with the number of correct answers. Additionally, since the model is trained so that the probability of the correct answer is 1, when the position of a correct answer increases, the loss value of another true position increases and does not fall below a specific loss value when multiple answers exist. Another loss function is binary cross entropy (BCE), which finds a loss value for multiple correct answers. The average of the loss value for each output position is used. \[BCE=-\frac{1}{N}\sum_{i}^{N}\updownarrow_{i},\updownarrow_{i}=y_{i}(\log(p(y_ {i}))+(1-y_{i})\log(1-p(y_{i}))) \tag{4}\]  
This poses a problem for masked LM, as outputs are produced according to the list of tokens, and, with most answers being zero, the model is trained to elicit a probability value of zero for each output. For the multi-hot LM, we implemented a function to obtain loss values for problems with more than one answer by modifying the SCE. We changed the probability of a correct answer from 1 to \(y_{i}/n\). Here, n is the total number of correct answers, and \(y_{i}\) is the same number of tokens included in the correct answer. By adjusting the probability value of the correct answer, the sum of the Softmax function is designed to not exceed 1 and to prevent excessive loss values. For example, for the masked word '[PERSON] (a person's name, with multi-hot token representation: [\"@A,\"@r,\"@c', \"@h', \"@i', \"@m', \"@e', \"@d', \"@e', \"@s\"]), the resulting probability value for '@A' is 1/10, and the total probability value for all ten letters is 1. The \"@' symbol was used to distinguish it from tokens created via the existing BPE. The problem with using Softmax is that the probability value of the correct answer of the tokens at each position becomes 1, resulting in a total of 10. In addition, if one of the three answers' loss decreases, the other answer increases the loss value. Therefore, the model cannot be trained to converge. We calculate a loss value for \(O_{i}\), which is the output value \(\updownarrow_{i}\):  
\[\updownarrow_{i}=-(\log(S(O_{i}))-\log(y_{i}/n)) \tag{5}\]  
Here, \(S\) is the Softmax function. The above equation is the existing SCE, minus \(\log(y_{i}/n)\). Since the multi-hot representation-based loss function gives an answer probability of \(y_{i}/n\), we infer that the log-Softmax value of the output value at that position converges to zero using \(\log(y_{i}/n)\). For duplicate characters, \(y_{i}\) increases, so the probability of that character increases with the number of characters. For example, for a set of three different tokens, the maximum value of \(\log(S(O_{i}))\) cannot be less than \(\log(1/3)\) as the probability of each answer position is 1/3. So, \(\log(1/3)\) is subtracted to derive a loss value such that the existing probability value is 1. After obtaining \(\updownarrow_{i}\)--the loss value for each output neuron--the correct answer for the current training data is redefined using this value.