Improving Interpretability of Deep Neural Networks with Semantic Information
Yinpeng Dong, Hang Su, Jun Zhu, Bo Zhang
Department of Computer Science and Technology, Tsinghua University, Beijing, China
We also show the neuron activations with respect to relevant (R) and irrelevant (I) topics in sampled videos. We plot
the activations of one neuron related to each topic throughtime.

Introduction
Interpretability of deep neural networks (DNNs) is essential
since it enables users to understand the overall strengths
and weaknesses of the models, conveys an understandingof how the models will behave in the future, and how todiagnose and correct potential problems. However, it ischallenging to reason about what a DNN actually does due
to its opaque or black-box nature.

• Retrieve a set of neurons associated with t;• Calculate the average activations of these neurons of thetraining videos with t;• Enhance the activations by adding the average activations
from v to v∗;

2. Interpretive Loss
• Topics extracted from descriptions by LDAnX1• LI(V, s) = kf( nvi) − sk22
3. TrainingN X• Correction propagation: generalize the error tofuture unseen data and diagnose potential problems
by finetuning the model:

i=1
n !
XkNsX
11k , xk)

λkflog p(ytk|y<t− skk22 −
L=N
nt=1
k=1
i=1
4. Interpretation• Each neuron can be represented by a semantic topic• Find the associations by the prediction difference
maximization algorithmj∗i = arg max([f(v)]i − [f(v\j)]i)j

In this work, we• Incorporates the interpretability of hidden featuresduring training;• Leverage the semantic information embedded inhuman descriptions;• Interpret the neurons by a prediction difference
maximization algorithm;• Introduce a human-in-the-loop learning procedure.

Results
We show the average activations of the representation for
the videos which include a given topic (dog, girl, walk,dance). The first row shows the results of the proposedmodel LSTM-I and the second row shows the baseline model
LSTM-B (without interpretive loss).

Lhuman = kv0 − v∗k22 + µkθ0 − θk22
We compare the performance of the proposed model withother state-of-the-art models of description generation on
the YouTubeClips dataset measured by BLEU and METEOR.
Model (CNN features)
BLEU-4 METEOR
LSTM-B (GoogLeNet)
0.416
0.295
LSTM-I (GoogLeNet)
0.446 0.297
LSTM-YT (AlexNet)
0.333
0.291
S2VT (RGB + Optical Flow)
0.298
SA (GoogLeNet)
0.403
0.290
LSTM-E (VGG)
0.402
0.295
h-RNN (VGG)
0.443
0.311
SA (GoogLeNet + C3D)
0.419
0.296
LSTM-E (VGG + C3D)
0.453
0.310
h-RNN (VGG + C3D)
0.499
0.326

Human-in-the-Loop Learning
Methodology
1. Attentive encoder-decoder framework• Encoder: GoogLeNet + Bi-directional LSTM
• Decoder: Attentive LSTM

Procedure:• For an inaccurate prediction, users provide a missing
topic t;• Activation enhancement:

Motivation:• Get human users involved in the learning process;• Embed human knowledge into the model;• Correct model‘s mistakes.

Demonstration:

Conclusions• A novel technique to improving interpretability during

training;• We simultaneously improve the interpretability of thelearned features in deep neural networks and achieve
better performance;• Propose a prediction difference
maximization algorithm;• Introduce a human-in-the-loop
learning procedure.