# Knowledge Distillation-Based Multilingual Code Retrieval
## 2 Multi-Programming Language Code Search
### Teacher Model  
We follow [PERSON] et al's study and use the same monolingual model structure, i.e., 1 dCNN, NBOW, and self-attention, these are commonly used methods based on token sequences. In this paper, we will only introduce the model structure of self-attention and the subsequent experiments will be based on this model, since it performed best among the 3 methods. We embedded the code in the same way as embedding the natural language--an encoding of their token sequences with an added attention vector. In general, the model needs to be trained for the following parts as shown in Figure 2: code_vocab and query_vocab of the code and the description of the embedding layers, the fully connected layers of the corresponding code and description, and the attention vector. Here, we define two embedding lexicons: code_vocab and query_vocab, each line corresponds to a specific code token or objects of the description token:  
\[code\_vocab\in\mathbb{R}^{|X|\times d} \tag{1}\]  
Figure 2: Details in Teacher Models whose encoder is self-attentions. \[query\_vocab\in\mathbb{R}^{|Q|\times d} \tag{2}\]  
\(X\) is the set of code token dictionaries, \(Q\) is the set of natural language description dictionaries, and \(d\) is the dimension of the embedding, which we set to 128 in our experiments. Finding the embedding of a code or query is a simple matter of finding the corresponding line. For a line of code \(C=ctoken_{1},ctoken_{2},,ctoken_{n}\) and its corresponding natural language description \(Q=qtoken_{1},qtoken_{2},\cdot\cdot\cdot,qtoken_{m}\),with the \(ctoken_{i}(i=1,\cdot\cdot\cdot,n)\) as code token and \(qtoken_{i}(i=1,\cdot\cdot\cdot,m)\) as description token. After embedding (random initialization) we can obtain:  
\[c_{i}=embedding(ctoken_{i}), \tag{3}\] \[q_{i}=embedding(qtoken_{i})\in\mathbb{R}^{d}\]  
And after going through a full connected layer, we have:  
\[\tilde{c}_{i}=tanh(W_{c}\cdot c_{i}),\tilde{q}_{i}=tanh(W_{q}\cdot q_{i}) \tag{4}\]  
In which \(W_{c},W_{q}\in\mathbb{R}^{d\times d}\), \(tanh\) is the hyperbolic tangent function, which is a common monotonous nonlinear activation function, taking the value from \((-1,1)\). Finally, we use the attention vector to aggregate these token vectors, it is essentially a weighted average aggregation. The main process is to calculate the weight of each token in the current block of code or natural language sentence, in terms of goals, we naturally hope that the token that can represent the code or sentence will occupy a greater weight, and this is where \"attention\" comes into play. In the beginning, the attention vector \(a\in\mathbb{R}^{d}\) will be randomly initialized and study the model simultaneously during the training process. The weight of each token is calculated by dotting the vector of the token with the attention vector and then normalizing it to ensure that the weights sum up to 1, the weights of each corresponding code token vector \(\tilde{c}_{i}\) are calculated as follow:  
\[\alpha_{i}=\frac{exp(\tilde{c}_{i}^{T}\cdot a)}{\sum_{j=1}^{n}exp(\tilde{c}_{ j}^{T}\cdot a)} \tag{5}\]  
The purpose of using \(\exp\) is to ensure that the weights are positive, as in the standard softmax function, and we divide by the sum of all terms to ensure that the sum of the weights is 1, the calculation of natural language description token vector follows a similar pattern. Once \(\alpha_{i}(i=1,\cdot\cdot\cdot,n)\) is calculated, the final code vector can be obtained by summing the linear weights of the code token vectors \(\{\tilde{c}_{1},\tilde{c}_{2},\cdot\cdot\cdot,\tilde{c}_{n}\}\). Code Vector represents the entire code segment, and it's expressed as follow:  
\[v_{c}=\sum_{i=1}^{n}\alpha_{i}\cdot\tilde{c}_{i} \tag{6}\]  
**Similarity Model**  
After obtaining the code vector _wait_ and the description vector, we want the code and description vectors to be co-embedding, so we measure the similarity between these two vectors.