# Design and Investigation of Capsule Networks for Sentence Classification
## 4 Experiments, Results and Discussion
### Looking inside Internal Layers  
For a better understanding of what is happening inside the network, we visualized the output of the network on a number of internal layers. Looking into the Class Caps Layer, we found that the value of a certain capsule in a class is higher than all the other values if the input belongs to that class.  
Figure 5: Effect of kernel sizes and the number of parallel convolution layers.  
We present the output of the Class Caps Layer for three negative and three positive sentiments in Figure 6a. Looking at Figure 6a closely, we can observe that there is a pattern. For a negative sentiment, the brightest pixel, or the longest capsule in capsule notation, resides in the second row. For a positive sentiment, the brightest pixel is found in the first row. By looking at this figure, we can easily deduce that the longest capsule determines the class of our input data.  
So as to examine how the change of sequential order of words affects the capsule information, we shuffled words in the input test data, keeping track of the location of strong words that are indicative of the polarity of the given test input. The outcomes of this experiment are given in Figure 6b. As it can be seen from this figure, when shuffling words, the output of the Class Caps Layer also changes. Even though we can say that shuffling words results in change of the position of the longest capsule, we cannot clearly conclude which shuffled word caused which change.  
In order to further investigate how CapsNets handle spatial features of text at the phrase level, we did a visualization of the output at the first convolution layer with kernel size 7 and stride 1. By keeping the stride 1, a single output of this convolution layer maps to a single 7-gram phrase. To ensure that the CapsNet captures the order of words, we repeated the experiment by shuffling a given sentence in the test set, the results of which are given in Figure 7. When given normal test input (see Figure 7a, the CapsNet responds with a large output for 7-grams which contain higher polarity information than the other 7-grams. However, the outputs for the shuffled case usually have much lower values even for polarity indicating phrases and no significant difference in the output values among 7-gram phrases can be observed (see Figure 7b. We performed several experiments with different shuffling and the outputs for those shuffled cases were almost similar to the one given in  
Figure 6: Visualization of the Class Caps Layer. (**a**) class identification based on longest capsule in Class Caps Layer; (**b**) output of Class Caps Layer before and after shuffling (left: negative sentiment, right: positive sentiment).  
Figure 7b. From this experiment, we argue that the CapsNet learns the spatial information presented in the input sentences.  
Figure 7: Visualization of the first convolution layer for normal and reshuffled test inputs. (**a**) convolution layer 1 output: Normal test input; 7-grams that contain much polarity information have higher outputs; (**b**) convolution layer 1 output: shuffled test input. No significant difference in the output between 7-grams that contain enough polarity information and those which do not.