## 3 Model Abstracting and Explaining Framework
### Generating Explanation
\[\mathit{hol}\Big{(}w_{i}^{j}\Big{)}=o_{i}^{j}(\alpha)-o_{i-1}^{j}(\alpha) \tag{4}\]  
\[\mathit{rem}\Big{(}w_{i}^{j}\Big{)}=p\big{(}\mathit{sen}_{j},\alpha\big{)}-p \Big{(}\mathit{sen}_{j}\backslash\Big{\{}w_{i}^{j}\Big{\}},\alpha\Big{)} \tag{5}\]  
They can be normalized as  
\[\mathit{hol}^{\prime}\Big{(}w_{i}^{j}\Big{)}=\frac{\mathit{hol}\Big{(}w_{i}^ {j}\Big{)}}{\sum_{w_{i}^{j}\in\mathit{set}_{j}}\big{|}\mathit{hol}\Big{(}w_{i }^{j}\Big{)}\big{|}} \tag{6}\]  
\[\mathit{rem}^{\prime}\Big{(}w_{i}^{j}\Big{)}=\frac{\mathit{rem}\Big{(}w_{i}^ {j}\Big{)}}{\sum_{w_{i}^{j}\in\mathit{sen}_{j}}\big{|}\mathit{rem}\Big{(}w_{ i}^{j}\Big{)}\big{|}} \tag{7}\]  
where \(\mathit{set}_{j}\backslash\Big{\{}w_{i}^{j}\Big{\}}\) indicates removing words \(w_{i}^{j}\) from the sentence \(\mathit{sen}_{j}\). Finally, based on the weight coefficients corresponding to the words and the two influence scores above, the weight score of the words is calculated as in Equations (8) and (9), where \(\mathit{sco}_{i}^{j}\) normalizes \(\mathit{sco}_{i}^{j}\). If the weight score of each word in the sentence \(\mathit{sen}_{j}\) reaches a certain threshold, the high explanation-related words are selected for the \(a-th\) output label and stored in \(\mathit{itp}(\mathit{sen}_{j},\alpha)\). This can be obtained from the sequence of high explanation-related words in the sentence. Algorithm 2 presents the specific process of explanation generation. \[\mathit{sco}_{i}^{j}=\theta\cdot\Big{(}\mathit{hol}^{\prime}\Big{(}w_{i}^{j} \Big{)}+\mathit{rem}^{\prime}\Big{(}w_{i}^{j}\Big{)}\Big{)} \tag{8}\]  
\[\mathit{sco}_{i}^{j}=\frac{e_{i}^{j}}{\sum_{w_{i}^{j}\in\mathit{sen}_{j}} \big{|}e_{i}^{j}\big{|}} \tag{9}\]  
```
Input: PFA \(\mathcal{A}\), text dataset \(\mathcal{D}_{t}\). Output: high-related explanation words \(\mathit{Itp}\), explanation scores \(\mathit{Sco}\). 1. For each sentence \(\mathit{sen}_{j}\) in the text dataset \(\mathcal{D}_{t}\) do
2. Obtaining word embeddings on \(\mathit{sen}_{j}\) to work out the input word vector for PFA;
3. Determine feature word classification based on behavior analysis on state \(\mathit{diagram}\) in terms of PFA
4. Calculate the weight score of explanation words using Equations (4)-(9);
5. If (weight score > preset threshold value) then:
6. Add corresponding words to the set of explanation words;
7. Record the corresponding weight score as an explanation result;
8. EndIf
9. End for
10. Return \(\mathit{Itp}\), \(\mathit{Sco}\). ```  
**Algorithm 2** Explanation-Generating Algorithm. To understand the generation process, we give Example 4. **Example 4**. : _Explanation generating of sentiments classification for RNN movie reviews._Take the sentence \"This movie is nice!\" as a demonstration example. The way in which it is classified as a positive label \"P\" will be introduced. The word embedding vector is \(emb_{sen}=(emb_{1},emb_{2},emb_{3},emb_{4})\), and output labels are \(lb~{}=~{}P\). We obtain its hidden state \((h_{1},h_{2},h_{3},h_{4})\) and output state \((o_{1},o_{2},o_{3},o_{4})\) in the RNN. We then list all the trace paths \(tr\) from the input node to the mapped \"positive\" label node on PFA. The current impact probability \(hol\) and the removal impact probability \(rem\) are calculated based on Equations (4) and (5). After normalization, based on Equation (9), we can synthesize them to explain which words in the sentence have a significant impact on classification. These words, in this example, are {movie, exciting}, and then we calculate their influence score list as \(Sco=\{(sco_{2},sco_{4})\}\).