# TEDL: A Text Encryption Method Based on Deep Learning
## 8 Interpretable Word Embedding by Matrix Decomposition  
Given that the feasibility of TEDL is based on the fact that the distributed representations of all the words change after adding a small amount of incremental corpus to the original corpus, it is necessary to understand why the training process can achieve the desired effect. As discribed in Section 6.2, the distributed representations refer to dense word vectors. Two Densification Methods are offered here to generate them instead of sparse vectors. Figure 8: The update of \(C_{\beta}\) as \(R\) increases. 1. Apply truncated SVD to the sparse matrix derived from a sparse word embedding model. 2. Use a dense word embedding model. As for Densification Method 1, since the matrix \(\mathbf{X}^{\prime}\) calculated after adding a small amount of corpus can be regarded as the original matrix \(\mathbf{X}\) is locally perturbed. If the location of the disturbance is appropriate, the effect is global and each component of word vectors change under finite precision conditions. As for Densification Method 2, actually, training can be interpreted as matrix decomposition. It has been proved that the embedding process of skip-gram negative sampling (SGNS) and noise-contrastive estimation (NCE) is an implicit matrix decomposition [31], while GloVe [17] is an explicit matrix decomposition [32]. Furthermore, [PERSON] derived and explained the parameter update equations of the Word2 vec models [30]. Now, we give a theorem about the essence of the process of skip-gram hierarchical softmax. Theorem 1: _The process of skip-gram hierarchical softmax (SGHS) is an implicit matrix decomposition._  
Proof: In a corpus, words \(w\in V_{w}\) and their contexts \(c\in V_{c}\), where \(V_{w}\) and \(V_{c}\) are the word and context vocabularies. The vector representation for word \(w_{i}\) is \(\mathbf{v}_{w_{i}}\), while for \(c_{i}\) is \(\mathbf{v}_{c_{i}}\). For word \(w_{i}\), the contexts are the words surrounding it in an \(L\)-sized window \(w_{i-L},\cdots,w_{i-1},w_{i+1},\cdots,w_{i+L}\), one of which is \(c_{i}\). We denote the collection of observed word-context pairs as \(\mathbb{S}\). We use \(\#(w_{i},c_{i})\) to denote the number of times the pair \((w_{i},c_{i})\) appears in \(\mathbb{S}\). Similarly, \(\#(w_{i})\) is the number of times \(w_{i}\) in \(\mathbb{S}\) and \(\#(c_{i})\) shows the number of times \(c_{i}\) occurred in \(\mathbb{S}\). They are defined as:  
\[\#(w_{i})=\sum_{c_{i}^{\prime}\in V_{c}}\#\big{(}w_{i},c_{i}^{ \prime}\big{)} \tag{21}\] \[\#(c_{i})=\sum_{w_{i}^{\prime}\in V_{w}}\#\big{(}w_{i}^{\prime},c_ {i}\big{)} \tag{22}\]  
Consider a word-context pair \((w_{i},c_{i})\). In the hierarchical softmax model, no output vector representation exists for context words. In other words, the vector \(\mathbf{v}_{c_{i}}\) is untrained. Instead, there is an an output vector \(\mathbf{v}_{n(c_{i},j)}^{\prime}\), which is trained during the training process, for each of the \(|V_{c}|-1\) inner units. Furthermore, the probability of a word being context \(c_{i}\), the output word, is defined as  
\[P(w_{out}\!=\!c_{  
Obviously, the following equation is true.  
\[\sum_{i=1}^{|V_{c}|}P(w_{out}=c_{i}\mid w_{i})=1 \tag{27}\]  
The probability of going left at an inner unit (including the root unit) \(n\) is defined as  
\[P(n,left)=\sigma\Big{(}\mathbf{v}_{n}^{\prime\ T}\cdot\mathbf{v}_{w_{i}}\Big{)} \tag{28}\]  
which is determined by both the output vector of the inner unit and the hidden layer. Similarly, the probability of going from unit \(n\) to right is  
\[P(n,right)=1-\sigma\Big{(}\mathbf{v}_{n}^{\prime\ T}\cdot\mathbf{v}_{w_{i}} \Big{)}=\sigma\Big{(}-\mathbf{v}_{n}^{\prime\ T}\cdot\mathbf{v}_{w_{i}}\Big{)} \tag{29}\]  
The parameter update process is derived in the following part. For simplicity, we consider the situation of one-word context models, which are easily extended to skip-gram models.