# CSRLoan: Cold Start Loan Recommendation with Semantic-Enhanced Neural Matrix Factorization
## 4 Methodology
### Statement Encoding  
The statements of user applications play a key role in solving cold start recommendation problems. The semantics in these statements reflect the user demographic and could be used to find similar users. In this section, we present a statement-encoding module to convert these statement texts into semantic representations. A statement can be represented as a sequence of words. Given a statement \(s\) of user \(u\), i.e., \(s=[w_{1},w_{2},\ldots,w_{K}]\), CSRLoan first embeds each word into a vector with an embedding matrix, \(\mathbf{M}\):  
\[[\mathbf{e}_{1},\mathbf{e}_{2},\ldots,\mathbf{e}_{K}]=Embedding(\mathbf{M},s) \tag{2}\]  
where \(\mathbf{M}\in\mathbb{R}^{M\times d}\) is the word embedding's matrix. To model the sequential information of \(s\), we employ the sinusoidal values proposed by [PERSON] et al. [36] for encoding the  
Figure 1: The architecture of CSRLoan. position's information. This approach is an absolute position encoding called sinusoidal position embeddings. We construct sequential position embeddings as follows:  
\[\lambda_{kj}=\begin{\{} \begin{matrix}sin(10,000^{\hat{l}}),&\text{ if $k$ is even}\\ cos(10,000^{\frac{|-1}{k}}k),&\text{ if $k$ is odd}\end{matrix} \end{matrix} \tag{3}\]  
where \(k=[1,\dots,K]\) and \(j=[1,\dots,d]\). For the sequential position encoding of \(\mathbf{e}_{k}\), we denote it as \(\lambda_{k}\in\mathbb{R}^{d}\). The input vector can be represented as follows. \[\mathbf{i}_{k}=\mathbf{e}_{k}+\lambda_{k} \tag{4}\]  
After the generation of input vectors, we feed them into a transformer network to obtain semantic embeddings:  
\[\mathbf{h}^{\prime\ell+1}_{i}=\mathbf{O}_{h}^{\ell}\ \text{\bf concat}_{k=1}^{H }\Bigg{(}\sum_{j=1:K}\mathbf{w}_{ij}^{k,\ell}\mathbf{V}^{k,\ell}\mathbf{h}_{j} ^{\ell}\Bigg{)}, \tag{5}\] \[\text{\bf where, }\mathbf{w}_{i,j}^{k,\ell}=\text{\bf softmax }_{j}\Bigg{(}\frac{\mathbf{Q}^{k,\ell}\mathbf{h}_{i}^{\ell}\cdot\mathbf{K}^{ k,\ell}\mathbf{h}_{j}^{\ell}}{\sqrt{d_{k}}}\Bigg{)},\]  
where \(\mathbf{Q}^{k,\ell},\mathbf{K}^{k,\ell},\mathbf{V}^{k,\ell}\in\mathbb{R}^{K \times d},\mathbf{O}_{h}^{\ell}\in\mathbb{R}^{d\times d},k=1\) to \(H\) all indicate the number of attention heads. For numerical stability, the outputs after taking exponents of the terms inside \(softmax\) are limited to the range of \((-5,5)\). Then, the attention output \(\mathbf{h}^{\prime\ell+1}_{i}\) are fed into a feed-forward network (FFN), which contains the residual connections and batch normalization modules. For conciseness, we omit the equations of identical operations with transformer. After the \(P\) layers of transformer, we obtain the hidden representations of all words, i.e., \([\mathbf{h}_{1}^{P},\dots,\mathbf{h}_{K}^{P}]\). To generate the final embedding of \(s\), we employ the mean readout function on the node's embedding. The embedding, \(\mathbf{e}_{s}\), of trajectory \(s\) is computed as \(\mathbf{e}_{s}=\sum_{i=1}^{K}\mathbf{h}_{i}^{P}/K\). For conciseness, we represent the above-mentioned operations as follows:  
\[\mathbf{e}_{s}=Encoder_{trans}(s) \tag{6}\]  
Instead of initializing the model parameters randomly, we design a pre-training strategy to obtain the initial model's parameters. Traditional pre-training utilizes an encoder-decoder framework. It masks 20% percentage of tokens and ties to predict the masked tokens in the decoded sequence. Similarly to the encoder, we can also build a decoder to reconstruct \(s\):  
\[ \begin{split}[\hat{w}_{1},\hat{w}_{2},\dots,\hat{w}_{K}]& =Decoder_{trans}([\mathbf{h}_{1}^{P},\dots,\mathbf{h}_{K}^{P}])\\ \mathcal{L}_{mask}&=\sum_{i\in MP}CrossEntropy(w_{i },\hat{w}_{i})\end{split}  \tag{7}\]  
where \(MP\) is the masked position. For loan applications, there is an \"activity\" label \(y\) along with the statement indicating whether the loan is defaulted or not. It consists of three states, i.e., default, active, and completed. Thus, we propose a new pre-training task, which predicts the activity label based on the representation.