Analysis of Video Feature Learning in Two-Stream CNNs on the Example of Zebrafish Swim Bout Classification
Authors: Bennet Breier
Arno Onken
Venue:  (2020)
Keywords: convolutional neural networks
neural network transparency
AI explainability
deep Taylor decomposition
supervised classification
zebrafish
transparency
behavioral research
optical flow

Abstract: Semmelhack et al. (2014) have achieved high classification accuracy in distinguishing swim bouts of zebrafish using a Support Vector Machine (SVM). Convolutional Neural Networks (CNNs) have reached superior performance in various image recognition tasks over SVMs, but these powerful networks remain a black box. Reaching better transparency helps to build trust in their classifications and makes learned features interpretable to experts. Using a recently developed technique called Deep Taylor Decomposition, we generated heatmaps to highlight input regions of high relevance for predictions. We find that our CNN makes predictions by analyzing the steadiness of the tail's trunk, which markedly differs from the manually extracted features used by Semmelhack et al. (2014). We further uncovered that the network paid attention to experimental artifacts. Removing these artifacts ensured the validity of predictions. After correction, our best CNN beats the SVM by 6.12%, achieving a classification accuracy of 96.32%. Our work thus demonstrates the utility of AI explainability for CNNs.

Published as a conference paper at ICLR 2020
ANALYSIS OF VIDEO FEATURE LEARNING IN TWo- STREAM CNNS ON THE EXAMPLE OF ZEBRAFISH SWIM BOUT CLASSIFICATION
Bennet Breier & Arno Onken
School of Informatics
The University of Edinburgh
Edinburgh, EH8 9AB, UK
b. b.breiers sms. ed. ac. uk, aonken an..ee.c...ddu.
ABSTRACT
Semmelhack et al. (2014) have achieved high classification accuracy in distin- guishing swim bouts of zebrafish using a Support Vector Machine (SVM). Con- volutional Neural Networks (CNNs) have reached superior performance in various image recognition tasks over SVMs, but their learnt features are not immediately visible. Reaching better transparency helps to build trust in their classifications and makes learned features interpretable to experts. Using a recently developed technique called Deep Taylor Decomposition, input regions of high relevance for predictions. We find that our CNN makes pre- dictions by analyzing the steadiness of the tail's trunk, which markedly differs from the manually extracted features used by Semmelhack et al. (2014). We fur- ther uncovered that the network paid attention to experimental artifacts. Removing these artifacts ensured the validity of predictions. After correction. our best CNN beats the SVM by 6.1%, achieving classification accuracy of 96.32%. Our work thus demonstrates the utility of AI explainability for CNNs.
generated heatmaps to highlight
INTRODUCTION
In the study by Semmelhack et al. (2014), well-performing classifier allowed to correlate neural interventions with behavioral changes. Support Vector Machines (SVMs) were commonly applied to such classification tasks, relying feature engineering by domain experts. In recent years, Con- volutional Neural Networks (CNNs) have proven to reach high accuracies in classification tasks on images and videos reducing the need for manual feature engineering. After Lecun & Bengio (1995) introduced them in the 90s, CNNs had their break-through in the competition ILSVRC2012 with the architecture of Krizhevsky al. (2012). Since then, more and more sophisticated architectures have been designed enabling them to identify increasingly abstract features. This development has become possible due to the availability of larger training sets, computing resources, GPU training implementations, and better regularization techniques, such as Dropout (Hinton et al. (2012); Zeiler & Fergus (2014)).
While these more complex deep neural network architectures achieved better results, they also kept their learnt features hidden not further analyzed. This caused CNNs come with significant drawbacks: a lack of trust in their classifications, missing interpretability of learned features in the application domain, and the absence of hints as to what data could enhance performance (Molnar (2019)). Explaining the decisions made by CNNs might even become legal requirement in certain applications (Alber et al. (2018)).
In order to overcome these drawbacks, subsequent research has developed approaches to shed light on the inner workings of CNNs. These approaches have been successfully used for uncovering how CNNs might learn unintended spurious correlations, termed "Clever Hans" predictions (Lapuschkin et al. (2019)). Such predictions could even become harmful the predictions entailed decisions with severe consequences (Leslie (2019)). Also, since deep neural networks have become a popular

---

Published as a conference paper at ICLR 2020
machine learning technique in applied domains, spurious correlations would undermine scientific discoveries.
This paper focuses zebrafish research as an applied domain of Al explainability, considering that the research community around this organism has grown immensely. The zebrafish is an ex- cellent model organism for vertebrates, including humans, due the following four reasons: The genetic codes of humans and zebrafish are about 70% orthologue (Howe et al. (2013)). The fish are translucent which allows non-invasive observation of changes in the organism (Bianco et al. (2011)). Furthermore, zebrafish are relatively cheap to maintain, produce plenty of offspring, and develop rapidly. Finally, they are capable of recovering their brain structures within days after brain injury (Kishimoto et al. (2011); 1); Kizil et al. (2012)).
In this paper. we adapt CNNs work on highly controlled zebrafish video recordings and show the utility of a recently developed Al explainability technique on this task. We train the network on optical flow for binary classifying swim bouts and achieve superior performance when compared to the current state-of-the-art in bout classification (Semmelhack et al. (2014)). We then create heatmaps over the videos with the "INNvestigate" toolbox (Alber et al. (2018)) which highlight the areas that our CNN pays attention to when making a prediction. The resulting heatmaps show that our CNN learns reasonable features which are very different from those manually composed by Semmelhack et al. (2014).
RELATED WORK
In the following, we will give an overview over relevant CNN architectures and approaches. Then, we will summarize existing AI explainability approaches focusing on attribution techniques. Finally, we highlight important studies of behavioral zebrafish research and give details of the study by Semmelhack et al. (2014).
CNN architectures. Carreira Zisserman (2017) identified five relevant types of video architec- tures: CNN LSTM LLong-Short-TTem Memory), 3D-CNN (Ji et al. (2013)). Two-Stream (Si- monyan & Zisserman (2014)). 3D-Fused Two-Stream, and Two-Stream 3D-CNN. They differ in whether the convolutions are based on 2D or 3D kernels, whether optical flow is added, and how consecutive frames exchange information. Optical flow can be described as the horizontal and ver- tical displacement of pixel from one frame to the next (Farnebäck (2003)). Several algorithms for flow calculation exist, such as TV-LI (Zach et al. (2007)), Brox (Brox et al. (2004)), and Farneback (Farnebäck (2003)). Even novel deep learning approaches have been developed, e.g. FlowNet (Dosovisskiy et al. (2015); llg et al. (2017)).
Carreira & Zisserman (2017) initialized several CNN architectures with pre-trained weights and trained them on human action recognition datasets to show the benefits of transfer learning in video classifiers. Previous studies had shown this for images (Shin al. (2016)). Also, they found that adding a temporal stream based on optical flow always improved performance. This idea of a net- work with a spatial stream for appearance and temporal stream for motion had been first developed by Simonyan & Zisserman (2014).
AI explainability techniques. Current AI explainability techniques on images can be largely cate- gorized into two types: attribution and feature visualization (Olah et al. (2017)). Attribution relates regions of sample image to activations of units in the CNN, while feature visualization uncovers what kinds of inputs strongly activate a particular unit.
One of the earlier and quite successful attribution approaches is called Sensitivity Analysis (Si- monyan et al. (2014)). It is based on the idea that if single pixel were marginally changed, the prediction would worsen significantly for an important pixel, but only slightly for less important ones. Furthermore, Zeiler & Fergus (2014) and Ribeiro et al. (2016) showed simple ways of pro- ducing approximate relevance heatmaps. By occluding parts of an image one by one and observing the change in activation, they could measure the influence of different image regions. This allowed them to check whether the CNN focused on the important objects of the image or performed its classification only based contextual information.
The technique applied in this paper is called Deep Taylor Decomposition (Montavon et al. (2017)), which arose from Layer-Wise Relevance Propagation (Bach al. (2015)). It has been put into
2

---

Published as a conference paper at ICLR 2020
with text (Arras et al. (2017)), speech (Becker al. (2018)), and only once with video data (Anders et al. (2018)). It highlights the areas in a sample image which the CNN deems most relevant for making correct classification. We assume that the relevance of a pixel is determined by how much the classification accuracy would deteriorate we altered this pixel. DTD distributes relevance from the output layer the input layer by applying specific propagation rules for each layer. This approach equals a first-order Taylor decomposition of the output function. The authors argue that yields better approximation of relevance than Sensitivity Analysis. Alber et al. (2018) included this technique in their "iNNvestigate" toolbox, which we used in our work.
Apart from attribution techniques, CNNs can be explained using direct feature visualization ap- proaches. They try to find intuitive representations of the patterns a given unit responds to partieu- larly strongly or weakly. Among these are deconvolution networks (Zeiler & Fergus (2014)), which are closely related to Sensitivity Analysis, and optimization approaches in the input domain (Erhan al. (2009); Olah et al. (2018)). Also, instead of creating an abstract representation of features, one can select specific samples which highly activate or suppress particular unit. Bau et al. (2017) even went a step further by hiding irrelevant parts of the input image, similar to LIME (Ribeiro et al. (2016)).
Behavioral zebrafish research. As we argue in Section 1, zebrafish are a highly suitable model organism for the human body. They serve the object of study in many fields, such as wound repair (Kishimoto et al. (2011); )); Kizil et al. (2012)), visual processing in the brain (Roeser & Baier (2003); Gahtan (2005); Semmelhack al. (2014); Temizer al. (2015)), cancer research (White et al. (2013),, and genetic modifications (Hwang et al. (2013)). Especially in neuroscientific re- search, understanding behavior and behavioral changes in response to cerebral interventions of high importance (Krakauer et al. (2017)).
Previous studies therefore closely investigated the motion patterns of zebrafish (Borla et al. (2002); McElligott & O' O'Malley (2005)). Borla et al. (2002) described characteristic motion patterns during prey bouts, such as the precise bending of the very tip of the tail bring the head into position and the subsequent strong arching of the tail's center. Based on their observations, they concluded that zebrafish must possess fine axial motor control in all parts of their tail.
Given that prey movements can be clearly distinguished from other types of movements, Semmel- hack et al. (2014) hypothesized that there must be a dedicated circuitry in the zebrafish brain. They found pathway from retinal ganglion cells to an area called AF7 projecting to the optic tectum, the nucleus of the medial longitudinal fasciculus, and the hindbrain. which in turn produces the charac- teristic motor output. They verified their findings by ablating the AF7 neuropil and observing that lesioned fish failed to respond to prey stimuli with a movement that a trained SVM would classify as prey. They have identified the following five features as the most discriminating ones, ordered by descending importance:
1. Maximum tail curvature (maximum over the bout)
2. Number of peaks in tail angle
3. Mean tip angle (absolute value of tip angle in each frame, average over the bout)
4. Maximum tail angle (maximum over the bout)
5. Mean tip position (average position of last eight points in tail, with horizontal deflection as a fraction of the tail length)
In our work, we make use of the dataset they gathered for training their SVM and compare our CNN to the features above.
METHODS
We trained a Two-Stream CNN distinguish prey and spontaneous swim bouts of larval zebrafish with short video samples. For data preparation, we first extracted standardized snippets from raw videos and then performed augmentation by subsampling, flipping, and cropping. After training, we computed heatmaps showing the regions of high relevance within frames!.
The code can be found at wwtttttt///// / //gi http:/ggttttttttce thub. con/ Henji /eeeri/i//////iiii h- learning.git
3

---

Published as a conference paper at ICLR 2020
Detected Contours
Cropped Frame
Flow Frame
100
150
200
250
100
300
400
100
200
100 200
Figure I: Detected contours (black) in the original frame, resulting cropped frame after normaliza- tion, and corresponding optical flow frame.
Data pre-processing. We used the raw video files recorded by Semmelhack et al. (2014) with a high-speed camera at 300 frames per second and labeled as either spontaneous (0/negative, 56. %) or prey (1/positive. 43,9%) bout. The heads of the fish were embedded in substance called agarose to keep them steady. We turned the videos into grayscale, normalized them, and kept crop of size 256x256 pixels from each frame, such that the right-most part of the bladder was central on the left, shown in Figure We did not include the head of the fish in the crop, because the eyes would give away information about the type of bout (Bianco et al. (2011)). More details in Appendix B.
For correct centering we implemented gamma correction with 7 exp --skewaess/paan /pram)) (where param was tweaked to maximize detection performance (here param = 4.3)) and applied binary threshold at value 3 to separate the bladder from the rest of the fish. Since the eyes might also fall under this threshold, we declared the right-most contour the bladder. Each raw video contained several bout events which we extracted as sequences of 150 frames resulting in 1,214 video snippets. Each sequence started 15 frames before the actual bout motion was detected.
Extension to the pre-processing procedure. After training and heatmap analysis, we found that the trained CNN had found a shortcut for classification by using agarose motion instead of tail features. We therefore extended our pre-processing procedure by setting squares of 85x85 pixels the upper and lower left-hand corners of all frames plain white. While this cut away some of the tail tip in rare occasions, ensured that bouts could not be classified based on the agarose motion anymore.
Data augmentation. Our data augmentation procedure worked with batches of 32 videos with the original data randomly shuffled, allowing a decent approximation of the gradient during training with some inherent noise, which is generally desirable to avoid falling into sharp minima (Keskar et al. (2016)). The procedure performed three augmentation steps: first subsampling, then flipping, and cropping, which achieved augmentation factors of 8, 2, and 9 respectively, totaling 174,816 augmented samples. We decided to compute optical flow after subsampling. not before, in order to create differing augmented samples. All samples of augmented batch were derived from different original vidcos to ensure good gradient approximation during training.
In the subsampling step, our algorithm randomly selected 86 of 150 frames under the constraint that between two frames, no more than 2 frames may be omitted. This was to ensure meaningful flow calculation in the subsequent step, because the tail could move quite fast. After subsampling, for each video our procedure selected one of the 86 frames to be the input for the spatial network. Then we computed the optical flow resulting in 85 flow frames with one and y component each. They were stacked to 170 channels, alternating æ and y. We used the optical flow algorithm by Farnebäck (2003) with parameters? detecting flow even when the tail moved fast. The procedure then generated 18 augmented batches from each subsampled batch by randomly flipping vertically
"pyraiid scale 0.8, pyramid levels 10, window size 10, iterations per pyramid level = 10, pixel neighborhood for polynomial expansion 13, std dev of the Gaussian for polynomial expansion 1.8.
4

---

Published as a conference paper at ICLR 2020
Spotial Stream
conv1
conv2
conv3
conv4
conv5
fe6 4096 drapout
fc7
fc8 2
log softmax
7x7x96
5x5x256
3x3x512 stride
3x3x512 stride
3x3x512
2048
stride 2
stride
stride 1
dropout
LAN norm
LRN norm
pool 2x2
pool
pool
grayscale channel
fusion averaging
170 channels
optical flow
Temporal Stream
conv1
conv2
conv3
convd 3x3x512 stride
conv5
fc8
log- softmax
7x7x96
5x5x256 stride 2
3x3x512 stride
3x3x512 stride 1 pool
4096
2048 dropout
stride 2
dropout
pool 2x2
pool 2x2
Figure 2: Full Two-Stream CNN architecture.
and cropping into crops of size 224x224 with the upper left corner at coordinates (8,8), (8.16), (8,24), (16,8), (16,16), (16,24), (24.8), (24,16), and (24,24).
Since flow calculated floats, minimized storage space significantly by rescaling each frame to range 0-255, compressing them with lossy JPEG-compressio at level 40, and turning them into unsigned integers, similar what Simonyan & Zisserman (2014) had done. Frames were rescaled later for training.
CNN architecture and framework. Just like Simonyan & Zisserman (2014), used two- stream network with an adapted CNN-M-2048 network (Chatfield et al. (2014)) for each stream, as depicted in Figure 2. As shown by Carreira & Zisserman (2017), this network deal with small sample size and learns quickly. The spatial stream had one gray-scale channel and the temporal stream 170 flow channels in the first layer. After obtaining the predicted probabilities of each stream by calculating the log-softmax of the individual two outputs, they were fused by averaging. We computed the negative log-likel loss of this average, which can be interpreted as the joint log- bog-probability of both streams, assuming their statistical independence.
Our dataset was made up of 38 files, with 28, and files for training, validation, and test sets respectively. The individual sets were not shuffled during training in order to allow sequential reads, which might have decreased training time. Notwithstanding. batches consisted of random samples due to augmentation procedure explained above.
Initialization weights. We initialized both streams with weights pre-trained on ImageNet'. This has become common initialization strategy in various classification tasks (Shin et al. (2016); Lee et al. (2016); Van Horn & Perona (2017)), due to the utility of general features learnt from natural images, such edge detection. While Simonyan & Zisserman (2014) did not pre-train flow, Carreira & Zisserman (2017) found a pre-trained temporal stream to reach superior performance. With this initialization hoped that training would require only fine-tuning and therefore less data and fewer epochs.
Specifically, for the weights of the input layer of the spatial stream took the average of the pre- trained weights of 3 RGB-channels to get the weights for grayscale-channel. For the temporal stream we copied the RGB-channels 56 times get 170 channels, and added uniform random noise to all of them. This was to ensure that the channels evolved differently during training and should aid learning. Regarding outputs, we averaged the pre-trained weights of 500 units on the output layer to obtain the weights of two output neurons, because we dealt with 2 classes instead of 1,000 as in ImageNet.
Training procedure. We made use of the Adam optimizer with standard settings and tuned its learning rate and weight decay coefficient - the neural network equivalent of L2-regullarzzatio (Loshchilov & Hutter (2019)). Furthermore, we added gamma learning rate scheduler which

---

Published as a conference paper at ICLR 2020
Table I: Results on training, validation, and test (split by stream) sets for bascline (B), analyzed
CNN (0), best CNN before (1) and after (2) correction.
ID
LR
WD
TRAIN
VALID
SPATIAL
TEMPORAL
FULL TEST
B
.946
.946
.902
0
le-4
le-3
9962
9489
.2216
.9441
.9566
le-5
le-3
.9899
.9722
.8138
.9660
.9669
2
le-5
1c-3
.9977
.9577
.8213
.9581
.9632
updated the learning rate by a multiplicattive factor of y = 1/Vepach (Kingma & Ba (2014)) every epoch. Our training framework computed accuracy on the validation set after each epoch to estimate generalization performance.
Since were initializing our CNN with weights learned from quite different domain, fine-tuning was crucial. In particular, we performed a hyperparameter search over learning rate ((1e-3, le-4, le-5)) and weight decay (fle-2, le-3, le-4):) using a smaller dataset and training for about 8 epochs. With the best hyperparameters we trained the CNN on the full dataset for 5 epochs.
Relevance analysis with heatmaps. Making our CNN more transparent required Al explainabil- ity technique which would be well interpretable in the optical flow domain of our temporal stream. We expected feature visualization approaches (Olah et al. (2018); Bau et al. (2017); Erhan et al. (2009)) and deconvolutions (Zeiler & Fergus (2014)) to yield less interpretable outputs than attri- bution techniques. In particular, we chose to analyze our CNN with Deep Taylor Decomposition (DTD), because it had been applied successfully before (Lapuschkin et al. (2019); Van Molle et al. (2018); Anders et al. (2018)) and was conveniently accessible in the "iNNvestigate" toolbox (Alber et al. (2018)). For validation purposes, we additionally generated saliency maps (Simonyan et al. (2014)) and heatmaps from Guided BackProp (Springenberg et al. (2015)). The produced relevance heatmaps could be expected give clues about what specific regions of optical flow, within frame and across frames, the network was paying attention to.
Also, simplified the analysis by splitting the network into its individual streams. This pos- sible because no weights were learned after the final layer of each stream. Once the network was initialized correctly, "iNNvestigate" made the generation of heatmaps surprisingly simple. Also, with about 50 minutes for the whole analysis it was quite fast even on CPU, because it effectively only needed one forward and backward pass per sample. We used small dataset of 3,420 samples for analysis by setting the subsampling factor from 8 to 1, in order to simplify the process.
4 RESULTS
We fine-tuned the training of our CNN reach high accuracies in distinguishing prey bouts of larval zebrafish from spontaneous swims. Subsequently, we analyzed the learned weights by computing and averaging the relevance heatmaps of all samples grouped by class. We further define prey bouts as positive and spontaneous swims as negative.
CNN test results. We performed small hyperparameter search over learning rate and weight decay, which proved sufficient because all models were initialized with pre-trained weights from ImageNet. For our baseline SVM detailed
- in Appendix A we report the 5-fold cross-validated accuracy and the final accuracy on the held-out test set. The hyperparameters' agree with the ones found by Semmelhack et al. (2014). We further present the accuracies of the CNN used for heatmap analysis, as well as the best CNNs before and after removal of experimental artifacts training, validation, and test sets in Table I. For the test set we additionally report individual accuracies of the spatial and temporal streams. We highlight that the final CNN attains test accuracy of 96.32%. which is 6.12% points better than the baseline.
Relevance heatmaps. We used relevance heatmaps visualize the regions the CNN pays most attention to when classifying specific sample. We computed relevance averages across samples and
FRBF-kemnel with y 0.001 and C=

---

Published as a conference paper at ICLR 2020
Single Frame
Ang Only Peoltires
Aug Negathess (Baferc)
Aug Negazivos (Afte))
Avg All Samples
Temppon
Spattil
Figure 3: Relevance heatmaps of the CNN. High relevance is shown in dark red color, while light red stands for low relevance. Left: example temporal heatmap over a single frame (true pos- itive, optical flow in grayscale, relevance heatmap overlaid in red). Middle: averages of temporal heatmaps split by class; negatives before and after correction. Right: average spatial heatmaps over all samples.
1e-5
Average Relevance
1.0
mean
0.5
0.0
10
15
20
25
30
35
40
50
55
60
65
70
75
80
Frame
Figure 4: Distribution of relevance the flow frames of the averaged sample.
frames, as well as split by class in Figure 3 for more comprehensive insights into the features learned by our CNN. Similar results from other explainability techniques can be found in Appendix C. As expected, the heatmaps exhibit the checkerboard artifacts typical of kemnlls with stride 2 in the first convolutional layer (Odena al. (2016): Montavon al. (2017)).
Steadiness of the fish's 's trunk as differentiating feature. First and foremost, the heatmaps show that our CNN able differentiate the movements of zebrafish based on their characteristic motion. Relevance is highly concentrated at the trunk, i.e. the rostral part of the tail, for both temporal and spatial stream. We observe very sharp relevance pattern along the edges the tail. This indicates that the pre-trained weights helped the network to look for edges. The CNN pays little no attention to the end of the tail even though is visible in most frames. Instead, it makes positive classifications by looking at the very start of the trunk. The heatmaps suggest that calm and steady trunk indicates a prey bout. As for the negatives, the spread out relevance pattern reflects the high frequency of tail deflections typical of spontaneous bouts, which had been identified by Semmelhack al. (2014) before. This makes clear that the network is able to differentiate the movements of zebrafish to high accuracy based on their characteristic motion.
"Clever Hans" predictions. CNNs are incredibly powerful at finding any kinds of correlations in the input data even if they are not related the object of interest. Lapuschkin al. (2019) have termed such spurious correlations "Clever Hans" predictions, because the model bases its prediction on what we want it focus on, but some unintended artifacts in the data. Figure shows clearly that our CNN bases significant number of its negative responses mainly motion in the top left corner and focuses little on tail appearance and motion. This plays a role only negative classifications and only in the temporal stream. While the heatmaps vertically symmetric, would expect due to vertical flipping during augmentation, this not true for the peculiar region in the top left corner. Figure 3 depicts the averaged heatmap after removing the artifacts in the top and bottom left hand corners and retraining our CNN. Relevance is now entirely focused on the tail.
Relevance distribution across frames. Most relevance is concentrated the frames in the range 7-46, as depicted in Figure The first seven frames are of least importance. This is very likely because pre-processing procedure added buffer of 15 frames before each bout, suggesting that

---

Published as a conference paper at ICLR 2020
the network focuses on the range of frames which are in fact the most relevant ones. This further supports the hypothesis that our CNN is able to differentiate zebrafish movements based on their characteristic motion patterns.
5 CONCLUSION AND FUTURE WORK
We trained a two-stream Convolutional Neural Network (CNN) on recordings of larval zebrafish to classify prey and spontaneous swim bouts. We then visualized the learned weights by generating relevance heatmaps showing which regions of the input the network focuses on while performing its classifications. We find that our CNN is capable of learning highly discriminating tail features. These features seem to be quite different from the ones used in the SVM classification by Sem- melhack et al. (2014)- the previous state-of-the-art in bout classification. The heatmaps further uncovered a "Clever Hans" type of correlation. After removing this spurious correlation and re- training the network, the network reached test accuracy of 96.32%, which is 6.12% points better than the accuracy achieved by Semmelhack et al. (2014). Judging from the test accuracy, our CNN has learned better discriminating features than those used for the SVM by Semmelhack et al. (2014), and has thus beaten manual feature engineering in this application domain.
Steadiness of the fish's trunk as differentiating feature. The relevance heatmaps and high ac- curacy show that the network achieves correct classifications by looking for salient features in the trunk of the tail while largely disregarding the tip. A sharp and clear relevance profile confined to the edges of the trunk gives clear sign of a prey bout. The opposite speaks for spontancous bout. Here, attention spreads out to capture the strong vertical oscillation of the trunk. For this reason we conclude that the CNN makes its predictions based on the steadiness of the trunk. We believe interpretation of learned features to be in line with existing research on the kinematics of prey bouts. As shown by Borla et al. (2002) and McElligott & O' Malley (2005), prey bouts require fine control of the tail's 's axial kinematics to perform precise swim movements. Zebrafish noticeably reduce their yaw rotation and stabilize the positioning of their head to make a targeted move at their prey. Such precise movements are not required in spontaneous swim bouts. The heatmaps indicate that the network has found clear evidence for these kinds of motion in the trunk of the tail.
Furthermore, we argue that the CNN has learned features which are very different from the ones identified by Semmelhack et al. (2014). All of their features as outlined in Section -, except the second one, rely on information from the tip of the tail and a complete sequence of frames. However, many optical flow frames do not depict the tip of the tail because of its small size and high speed. This might have happened due to suboptimal parameter settings which could not handle the sometimes long distances which the tip traveled between frames. Also, subsamples include only 85 of the original 150 frames for each video. Due to its higher performance, conclude not only that the CNN has learned a different set of features, but also that these features must bear higher discriminative power.
Origin of the "Clever Hans" correlation. The telltale motion in the top left corner stems from substance called agarose, which the fish's head was embedded in to keep it steady. It is quite curious that, while not visible to human eyes, the agarose seems to be moving each time the fish performed spontancous swim bout, but not so for a prey bout. We speculate that this correlation was unintentionally introduced by the experimenters who might have tapped the petri dish to induce the fish to perform a spontaneous swim bout.
Future work. Calculating and storing optical flow is expensive. If we attained similar performance on original frames, training would be considerably cheaper. While we can confirm the findings by Simonyan & Zisserman (2014) 4) that the spatial stream by itself reaches fairly competitive accuracy, provides only very minor improvement to the overall network. Yet, this stream is probably looking for very similar features as the temporal stream, because it focuses largely the upper half of the tail, just like the temporal stream. If that is the case, we should see improved performance when giving the spatial stream sequence of frames. It should be interesting to probe whether the spatial stream could then match or even surpass the performance of the temporal stream.
Furthermore, CNNs such the one used in this paper could be used to investigate brain recovery in larval zebrafish. It has been shown on a cellular level that zebrafish can heal their brain within days after lesion. However, this needs to be proven on a behavioral level (Krakauer et al. (2017)).
8

---

Published as a conference paper at ICLR 2020
Future work could perform a lesion study the optic tectum in zebrafish (McDowell et al. (2004); Roeser & Baier (2003)), a brain region responsible for translating visual input into motor output. CNNs could then assess swim bouts of recovered fish and give measure for potential behavioral changes. Insights from relevance heatmaps would be required the CNN were not able not distin- guish recovered fish from healthy ones.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, and Others. TensorFlow: Large-scale machine learning on heterogeneous .rgg/ Software available from tensorflow.org.
systems, 2015. URL
Maximilian Alber, Sebastian Lapuschkin, Philipp Seegerer, Miriam Hägele, Kristof T. Schütt, Grégoire Montavon, Wojciech Samek, Klaus-Robert Müller, Sven Dähne, and Pieter-Jan Kin- dermans. iNNvestigate neural networks! aug 2018. URL. hRtphtt:://axxiii aas/g/////////// /108 04260.
Christopher Anders, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Understand- ing Patch-Based Learning by Explaining Predictions. jun 2018. URL http:://rxxvv..rgg abs/1806 06926.
Leila Arras, Franziska Horn, Grégoire Montavon, Klaus Robert Müller, and Wojciech Samek. "What is relevant in a text document?": An interpretable machine learning approach. PLoS ONE, 12(8)=e0181142, aug 2017. URL https: //dx. plos org/ 1371/ /orr///oourall...aa cournal.pone. 0181142.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer- wise relevance propagation, PLoS ONE, 10(7):e0130140, jul 2015. URL ht /ttp///xxppppppp dp.oos.. http///ooonaaa..ooom 0130140
org/10 10.1111.11///11 ott//10.337/jourmall.o
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantifying Interpretability of Deep Visual Representations, 2017. URL
http:://opeaccees thecvf. com/ http///oneet_eteeeeeeee Dissectioon_uuunttiyyyngg... 201_p__2rttppp..hhtmmm...
wwwpc_ppppppheeeeeeeeee (001//htllBBu__Netoork, Network_
Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Interpreting and Explaining Deep Neural Networks for Classification of Audio Signals.
jul 2018. URL http: www.iiiwrgg.......aa.o.a...ooort abs//bb///// 180...10...0 1807.03418......
Isaac H. Bianco, Adam R. Kampff, and Florian Engert. Prey Capture Behavior Evoked by Simple Visual Stimuli Larval Zebrafish. Frontiers Systems Neuroscience, 5:101, dec 2011. URL http: /tftrra///oraall.. wwwntiersin. wtt.://rwii..oog/ttteeeeeeeeeeee http://......////////// 2011. 00100011101111 absttrac....
Melissa A Borla, Betsy Palecek, Seth Budick, and Donald M. O'Malley. Prey capture by larval zebrafish: Evidence for fine axial motor control. Brain, Behavior and Evolution, 60(4):207-229, 2002. URL www. karger. comwww. homg/er.combbbe.
G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert. High Accuracy Optical Flow Estimation Based on Theory for Warping. pp. 25-36. Springer, Berlin, Heidelberg, 2004. URL pttt:://inng springer. com/ 10. 1007 9788 540-24473---2-4
João Carreira and Andrew Zisserman. Quo Vadis, action recognition? A new model and the kinetics dataset. In Proceedings 30th IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, volume 2017-Janua, pp. 4724 4733, 2017. URL https: //rxivv. .org/ pdf/ 1705. 07750.pds..

---

Published as a conference paper at ICLR 2020
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the Devil in the Details: Delving Deep into Convolutional Nets. 2014. URL http: http:://ww /ttp://wwwww..... robots ox.
ac.uk/http:: http://///.aaa....o.////////../.
François Chollet et al. Keras, https:://itth// http://hwab..ccccoooeeee////eeee 2015.
Andrew Collette. Python and HDF5. O'Reilly, 2013.
Casper O. da Costa-Luis. tqdm: Fast, Extensible Progress Meter for Python and CLL. Journal of Open Source Software, 4(37):1277, 2019, doi: (0.21105/jos.01277 URL https: /hub. docker. com/.
Alexey Dosovitskiy, Philipp Fischer, Eddy llg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning Optical Flow With
Convolutional Networks, 2015. URL http: http::///pp/ccccceeeeeeeee thecvf com/ content_ ICCV_2015
icev Accv_20151 ht15/ttm//ossvitttyyy. wtwpovvvttsiiyylloeeeeeeeeeeeeee Lww.eiaaaannnnneeeeeeeeeeeeeeeeo http:///.//.//////aa///////////r paper. html.
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Visualizing Higher-Layer Features of a Deep Network Oracle Performance for Visual Captioning View project Semantic
View project Visualizing Higher-Layer Features of Deep Network Département d' Informatique et Recherche Opérationnelle. (August 2014). 2009, URL https: hwtp///wwww///..o//////////ww www.w.eessee researchgate.
nat/publicate wtt.lb/iaaiion///50//
Gunnar Famebück. Two-Frame Motion Estimation Based on Polynomial Expansion. pp. 363-370. 2003. URL http: /www isy. 1iu. se/cv//.
Ethan Gahtan. Visual Prey Capture in Larval Zebrafish Is Controlled by Identified Reticu- lospinal Neurons Downstream of the Tectum. Journal of Neuroscience, 25(40)99944-9333 2005. URL http::/rsstbbt info...iiiif...... nih
/ttp:///www.. http://wwww...rroc....o///ogg/g
doi /001/10.0 523 111.11////EERS.I.22222 /EEEOOSEEUOOIIOOO 2678-6678880 2005.
Geoffrey F. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. jul 2012. URL http: /aaxiiiiiiv.//.ii//aaaa/aaaiavva org/orgg//...o 1207 1207.0580. 0580.
Kerstin Howe, Matthew D Clark, Carlos F Torroja, James Torrance, Camille Berthelor, Matthieu Muffato, John Collins, Sean Humphray, Karen McLaren, Lucy Matthews, and And Others. The zebrafish reference genome sequence and its relationship to the human genome. Nature, 496(7446):498-503, apr 2013. URL http: wwww nature. wot./ooifinder/11.10000 nature12111.
John D. Hunter. Matplotlib: A 2D Graphics Environment. Computing in Science & Engi- neering, 9(3):90-55, 2007. ISSN 1521-9615. doi: 10.100/MCSE.2007.55 URL http:
//ieeexplore. ieee. org/ /dc/udonttttttmmm /1160665/.
Woong Y Hwang, Yanfang Fu, Deepak Reyon, Morgan L Maeder, Shengdar Q Tsai, Jeffry D Sander, Randall T Peterson, J-R Joanna Yeh, and Keith Joung. Efficient genome editing in zebrafish using CRISPR-Cas system. Nature biotechnology. 31(3):227-9, 2013. URL http: /www. nature.. www doi wwcm//ooffinee/////ccooo www.ff/inndn..1100
nbt 25C1.
Eddy 1lg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. FlowNet 2.0: Evolution of Optical Flow Estimation With Deep Networks, 2017. URL http:// openaccess. thecvf. com cwwm//onteet_ccpp_211111
1_ Fww.Floowet____ html.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3D Convolutional neural networks for human action recognition. IEEE Transactions Patlern Analysis and Machine Intelligence, 35(1):221-231, 2013.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. sep 2016. URL http: /ttp:::////xvv URt.tttps///rxwwwoorr////////// 60 04836.
10

---

John W. Krakauer, Asif A. Ghazanfar, Alex Gomez-Marin Malcolm A. Maclver, and David Poep- pel. Neuroscience Needs Behavior: Correcting Reductionist Bias, feb 2017. URL https: // www. sc sciencediiiicce rect cwwmcoooc//oo.ooo. /ttpee/ne/eettte/epp//////// Published as a conference paper at ICLR 2020
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. dec 2014. URL http://arxiv. aariv..iii.v///.. org/ bbrg/bba abs 1412.4222 6980.
N. Kishimoto, K. Shimizu, and K. Sawamoto. Neuronal regeneration in a zebrafish model of adult brain injury. Disease Models Mechanisms, 5(2):200-25) 209, 2011, URL http://dmn ddmm. biologists, org/ content / dmm/ 5 2/200/22000 ful..pdf.
Caghan Kizil, Jan Kaslin, Volker Krochne, and Michael Brand. Adult neurogenesis and brain re- generation in zebrafish. Developmental Neurobiology, 72(3))429-466) 2012. URL https: // //onlineeibraree wiley. com /wi/pdf//C..cco//oo wtt.f//................ http://wc...////mu..2222 ddw...uuu..........
Ron Kohavi. A Study Cross- Validation and Bootstrap for Accuracy Estimation and Model Selec- tion. International Joint Conference of Artificial Intelligence, 1995. Utthttp: rrrboticccccs stanford. edu/ ronnyk.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances In Neural Information Pro- cexsing Systems, pp. 1-9, 2012. URL https: /ttps://pppprra/ee nips cc cc/paper///pccccccccccoccecce aee--iageeee--laasiiieeee pdf.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus Robert Müller. Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communicaations, 10(1):1096, dec 2019. URL http:// /www nature. com/ articles ////441460000 01077---19-0887777
Lecun and Y Bengio. Convolutional networks for images, speech, and time-series. Technical report, 1995.
http::/p//wwww.eeeaccceeeeeeeeee / 24539999
Sue Han Lee, Yang Loong Chang, Chee Seng Chan, and Paolo Remagnino. Plant identification system based on convolutional neural network for the lifeclef 2016 plant classification task. In CEUR Workshop Proceedings, volume 1609, pp. 502-510, 2016
David Leslie. Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of systems the public sector. 2019. URL htths:::: /do. .org/ 10. 5281 /ww/odzoooooo//ooo////oooooooo.. 3240529.
Ilya Loshchilov and Frank Hutter. Fixing Weight Decay Regularization in Adam. In Proceedings of the 2019 International Conference on Learning Representations (ICLR' 19), 2019. URL http: // //arxiv.cr. org/ abs 11111011111
Angela L. McDowell, Lee J Dixon, Jennifer D Houchins, and Joseph Bilotta. Visual processing of the zebrafish optic tectum before and after optic nerve damage. Visual Neuroscience, 21(2): 97-106, 2004. URL https: / doi. tttp://...1111000222233333333330
Melissa B. McElligott and Donald M. O'Malley. Prey tracking by larval zebrafish: Axial kinematics and visual control. Brain, Behavior and Evolution, 66(3):177-196, 2005. URL www. karger.
com.
Christoph Molnar. Interpretable Machine Learning, 2019. URL https://christopg
github. / interpretaablle -ml-book/
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder. Wojciech Samek, and Klaus Robert Müller. Explaining nonlinear classification decisions with deep Taylor decomposition. Par- tern Recogmition, 65:21,-222, may 2017. URL https: //wwww wtwww/wwiiiiccc. www.sssiincedieee..o com/
science/articler http:/rrrcccee/pp//0000112110100 66035822.
Augustus Odena, Vincent Dumoulin, and Chris Olah.
Deconvolution and Checker-
board Artifacts. Distill, 1(10):e3, oct 2016. URL
deconv-checkerboard.
11

---

Published as a conference paper at ICLR 2020
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature Visualization. Distill, 2(11) nov 2017. URL https: /ttpsiii///iiillll. /ttp://pp////aaaaa.pp//////aaaaa
Chris Olah, Arvind Saryanarayan, lan Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The Building Blocks of Interpretability. Distill, 3(3)e10, mar 2018.
URL htttps://distinn pub/ b blocks.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Fabian Pedregosa, Gaël Varoquaux. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg. Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duch- esnay. Scikit-leam:: Machine Learning in Python. Journal of Machine Learning Research, 12 (0ct):2825-288) 2011. URL http: //ww:/ jmlr. org/ https/ppprrs/vlle http//////peeeggsaaaalaa
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?". In Pro- ceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD '16 pp. 1135-1144, New York, New York, USA, 2016. ACM Press. URL http::/dll acm. org/ ei at wtt.::/c......oo...9991......33.
Tobias Roeser and Herwig Baier. Visuomotor Behaviors in Larval Zebrafish after GFP-Guided Laser Ablation of the Optic Tectum. 2003. URL http: jneurosci. wwtpsuroosii.og//oooooooee corgen/t
jneuro/23/9 3726. full.pdl.
Julia I. Semmelhack, Joseph C Donovan, Tod R Thiele, Enrico Kuehn, Eva Laurell, and Herwig Baier. A dedicated visual pathway for prey detection in larval zebrafish. 3:4878, 2014. URL -087884888888 0r3.pff
https: //cpnn./////ee/////// el
Hoo-Chang Shin, Holger R. Roth, Mingchen Gao, Le Lu, Ziyue Xu, Isabella Nogues, Jianhua Yao. Daniel Mollura, and Ronald M. Summers. Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristies and Transfer Learn- ing. IEEE Transactions on Medical Imuging, 1285)11285- may 2016. URL http: //ieeexplore ieee. org/ ottpeeo///oooueete////////////// document/74040117
Karen Simonyan and Andrew Zisserman. Action Recognition Videos, 2014. URL http://papers: nips. cc/paper/ 5353-twa streemt convolltinn................
Two-Stream Convolutional Networks for
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. 2014. URL http: //axiiv/rrxxv..
brg/abs/1311 6034.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for Simplicity: The All Convolutional Net. 2015. WtLhhtp::/ /ttttp:///rxi....// wwtp:/ww..rr/a///////b/ba/e abs/1412 6806.
Incinur Temizer, Joseph C. Donovan, Herwig Baier, and Julia L. Semmelhack. A Visual Pathway for Looming-Evoked Escape in Larval Zebrafish. Current Biology, 25(14);1823- 1834, jul 2015. URL https:://// bttccp:://www.e www..ciiencedii..co rect. $0960982215006677
Stéfan van der Walt, S Chris Colbert, and Gaël Varoquaux. The NumPy Array: A Structure for Efficient Numerical Computation. Computing Science Engineering, 13(2):22-30, mar 2011. ISSN 1521-9615. doi: 10.1 1091MMC.E222011..57 URL http:// www.eorreeeeeeeeee.ee.ee...eeeee ieee. document /572523//
Grant Van Horn and Pietro Perona. The Devil in the Tails: Fine-grained Classification in the Wild. 2017. URL www. inaturall st otghttt http:://aaww./eeee.ee.ee /10/ 01 450..
12

---

Published as a conference paper at ICLR 2020
Pieter Van Molle, Miguel De Strooper, Tim Verbelen, Bert Vankeirsbilck,, Pieter Simoens, and Bart Dhoedt. Visualizing convolutional neural networks to improve decision support for skin lesion classification. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artifi- cial Intelligence and Lecture Notes in Bioinformaties,, volume 1038 LNCS, pp. 115-123, 2018, URL http::ttt////..t...tt
/ttp:::://dd.. /doi .rgg 10. 1007 0000007978--003-02222222
Richard White, Kristin Rose, and Leonard Zon. Zebrafish cancer: The state of the art and the path forward, 2013. URL www nat ure wttp://www////////////ccccc/c
C. Zach, T. Pock, and H. Bischof. A Duality Based Approach for Realtime TV-L 1 Optical Flow. In Pattern Recognition, pp. 214-223. Springer Berlin Heidelberg. Berlin, Heidelberg, 2007. URL
http:// /tt:::///nnkkk//// springerrnnger com 10. 10000099777 540- 74936-3-26666-
Matthew D Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. 2014.
URL https://css nyu. www.nn..uuuueeeee
A FITTING THE BASELINE SVM
For each frame in the 1,214 videos, applied the tail-fitting code developed by Semmelhack et al. (2014) to compute points along the tail, as depicted in Figure S5. We initialized their procedure central and 8 pixels from the left edge, because after pre-procsssing could assume this to be just next to the right end the bladder. Some of the videos contained frames which the tail-fitting code had problems processing, possibly because the pre-processing procedure off the tip of the tail in these instances. This resulted in 953 correctly processed videos, including 482 (50.6%) spontaneous and 471 (49.4%) prey bouts. We performed augmentation here because this would not have benefited the SVM.
150
200
Figure S5: Distribution of points along the tail after running the tail-fitting procedure.
The feature extraction and model fitting algorithm then split the set into 85% training and 15% held-out test sets. Semmelhack et al. (2014) have identified 5 key features which allowed their SVM to achieve a cross-validane vasssvvaiiaiinnn accuracy of 96%. They did not report results on held-out test set. We used their provided code to extract these features. Then we performed grid-search to tune SVM-kernel, and C Just like Semmelhack et al. (2014) we used stratified 5-fold cross-validation (Kohavi (1995)).
B PROJECT PIPELINE
Figure S6 gives an overview over the whole project pipeline. The figure includes a depiction of the raw data input, pre-processing, augmentation, CNN and SVM training, and heatmap analy- sis. Figure S7 summarizes the data augmentation procedure. All scripts worked with a seed of 462019. We used the openly available distributions of NumPy 1.16.4 (van der Walt et al. (2011)),
RBFF-kemel, 7 (1c-1, le-2, le-3, 1c-4), E (0.01, 0.1, 1, 10); linear-kernel, C (0.01, 0.1, 1, 10).
13

---

Published as a conference paper at ICLR 2020
Matplotlib 3.1.1 (Hunter (2007)), tqdm 4.32.2 (da Costa-Luis (2019)), OpenCV 4.1.0.25 (Bradski (2000)), scikit-learn 0.21.2 (Pedregosa et al. (2011)), PyTorch 1.1.0 (Paszke et al. (2017)), h5py 2.9.0 (Collette (2013)), TensorFlow 1.14.0 (Abadi et al. (2015)), Keras 2.2.4 (Chollet et al. (2015)), and iNNvestigate 1.0.8 (Alber et al. (2018)).
ImageNet weights
videos (avi) + labels
train
weights (pth)
output
accuracy
augment
samples (hdfs)
CNN
(PyTorch) trans form
scores
pre- process
weights (hdfS) (Keras)
visualize
videos (npz)
heatmaps
(n=1,214)
tailfit points (pickle)
fit SVM (baseline)
accuracy scores
tailfit
Figure S6: Overview over the whole project pipeline.
1214 videos segments
videos per segment
has shape (32,38,150)
(num__maks, segment_size, num_frames)
for in range(32):
(randomized) combinations of flip//ropp
apply masks[il[O] and compute flow
flip/crop
save batches subsequently compressed
videos
disk
apply masks[i][37]] and computte flow
flip/crop
Figure S7: Data augmentation procedure.
Pre-processing, We aimed to center the fish's bladder the left of each cropped frame. To achieve this, we applied a binary threshold at value 3, because after normalization and gamma correction the pixels of the bladder as the second darkest part of the fish might still have fallen under this threshold. Therefore, we first detected all contours discard tiny contours (< 0.01% of the whole frame) and then kept only the right-most contour. Since this had to be the bladder now, we could get the crop dimensions using the right-most pixel of that contour.
separated mostly below this value. While the fish was quite light, the eyes
Each raw video mainly consisted of still fish interspersed with a few short bouts. These were the events we extracted into consecutive 150 frames each. The idea was detect motion by checking the percentage of pixel value changes from one frame to the next, considering only the tail. We omitted pixels other than the tail with simple binary threshold value 200. The pixels had to change in case at least 0.38% of the entire pixel range (height width x 255) in order for motion to be detected. If the algorithm detected motion for a certain number of consecutive frames, it set this the start of event. Also, it added preceding buffer of 15 frames. The end set 150 frames from the start. no motion was detected, we automatically took the first frame as a start.
We had to take care that extracted videos did not overlap, i.e. in part contained identical frames, even there were two or more distinct movements contained within 150 frames. This might have otherwise lead to train/test-cntaminnat Therefore, discarded any detected motions which fell
14

---

Published as a conference paper at ICLR 2020
in the range of previous video. One special case did not take into account was when the start was less than 150 frames from the end of the file. We shifted the start back by just enough frames to fit it in, but this might have made it overlap with previous video. Since this case probably very rare and not detrimental, we have kept the code as it was.
Data augmentation. We parallelized data augmentation with 38 workers, because optical flow cal- culation took roughly 14 seconds per subsample. We used a Dell PowerEdge R815 with four 16 core Opteron CPUs, 256 GB memory, and 4 TBB of disk space (HDD). Furthermore, the resulting hdf5- files were compressed with gzip-compression at maximum compression level. This outperformed lrf-copmression by factor of 1.76 when comparing training times. This advantage can be ascribed to heavy parallelization of decompression and relatively low transfer speeds between hard drive and memory.
Training procedure. We implemented PyTorch's Dataset module to make use of the multi- processing capabilities of the DataLoader class on servers with 16-32 CPU cores and 4 GPUs (NVIDIA GTX 1060 GB) each. This was necessary to achieve manageable epoch times.
Relevance analysis with heatmaps. The toolbox "INNvestigate" (Montavon et (2017)) for an- alyzing the trained weights only supported Keras with the GensorFlow-backende Therefore, we re-implemented the exact structure of our CNN and initialized with the extracted weights from PyTorch. While the conversion could have been done with tools like ONNX, after few unsuccess- ful attempts we transported the weights with a custom Python script.
A caveat to the "iNNvestigate" toolbox emerged after heatmap generation: it had problems analyz- ing 1,578 of the 3,420 samples. These produced an empty output. We made sure the problematic samples did not follow any systematic pattern by checking the indices of correctly analyzed samples, the ratio of true positives and negatives and false positives and negatives, as well as the distribution of classification confidence after the analysis. Since all numbers the same before the analysis, we continued with 1,842 samples for further investigation.
C OTHER EXPLAINABILITY TECHNIQUES
We generated saliency maps (Simonyan et al. (2014)) and Guided BackProp heatmaps (Springenberg al. (2015)) analogous to the relevance heatmaps in Figures S8 and S9 for comparison with the more recent technique of DTD. becomes apparent that these other two techniques allow similar insights, although slightly fuzzier. Importantly, they also uncover the "Clever Hans" prediction.
Avg Only Rosttives
Aug Nogatives (Befora)
Avg Negatives (Aitor)
Avg Samples
20
250
200
Figure S8: Averaged saliency maps, analogous to Figure 3.
Tomoora
200
250
132
250
Figure $9: Averaged heatmaps from Guided BackProp, analogous to Figure
15

---

Published as a conference paper at ICLR 2020
D SELECTION OF ADDITIONAL HEATMAPS
Here, we depict the ten most informative consecutive flow frames of the single most confident true positive (Figure S10), true negative (Figure S11), false positive (Figure S12), and false negative (Figure S13) sample. Figure S15 summarizes the spatial heatmaps of the same samples. More- over, we gather five particularly informative flow frames in Figure $14 and four spatial heatmaps in Figure S15.
40
41
43
44
45
46
48
49
Figure $10: Selection of flow frames of the most confident true positive sample.
Figure $11: Selection of flow frames of the most confident true negative sample.
E CONFIDENCE ANALYSIS ON HEATMAPS
We performed confidence analysis on the heatmaps which depicted the "Clever Hans" feature of agarose motion in the top left corner. We sorted all negative classifications by increasing confidence calculated (og(log(PU))//oo(PP)) for each sample. We then grouped and averaged the heatmaps over windows of 104 samples, as shown Figure $16. The analysis uncovered that the more confident a negative classification, the more the CNN relied on tail features. This in turn indicated that the CNN was able to learn actual features on the tail and did not entirely rely on agarose motion. Also, it suggested that tail features were better predictor than agarose motion.
16

---

Published as a conference paper at ICLR 2020
25
Figure S12: Selection of flow frames of the most confident false positive sample.
11
14
10
Figure S13: Selection of flow frames of the most confident false negative sample.
TP
TN
Figure S14: Selection of flow frames of selection of true positive (TP) and true negative samples (TN).
17

---

Published as a conference paper at ICLR 2020
TN
TP
Figure S15: Upper row: spatial input of the most confident samples of each category. Lower row: spatial input a selection samples.
0-104
104-208
208-312
312-416
416-520
520-624
624-728
728-832
832-936
936-1040
Figure $16: Averaged heatmaps of 104 samples per subfigure, sorted by increasing confidence for responding negative.
18


Reviews:
[{"review_id": "r1gLu9xO5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY: explore the use of CNN in a binary task on images of zebrafish\n\nIt is important to note that researchers in the field of AI and deep learning are themselves aware of the fallacies of deep learning, and are striving everyday to overcome these themselves. The hype over deep learning has caused certain disdain among a section of the research community over the workings of deep neural networks. This is evident in this paper with the authors calling CNNs \"black box\" and the learnings of a neural network \"cheating\". Perhaps the authors are not aware that CNNs are hardly black boxes, their inner workings quite transparent in mathematical terms, which the submitted paper itself explores. Perhaps the authors are also not aware that the fallacies that causes CNNs to overfit on some characteristics in the input data are also present in other machine learning tools such as SVMs. Perhaps the intention of the authors is to bring more relevance to the dangers of spurious correlations, especially when the applications are critical. I hope the community can work together in improving the state of the art while improving transparency and explainability.\n\nBACKGROUND:\nHypothesis: Prey movements in zebrafish are characterized by specific motions, that are triggered by a specific pathway involving an area called AF7.\nValidation: Semmelhack et. al. (2014) removed the AF7 neuropil and observed that they failed to respond to prey stimuli\nAI: the prey stimuli was a characteristic movement that an SVM was trained to detect.\n\nGood to know that the authors will share their code.\n\nIt is not explained why the authors chose to pretrain on ImageNet, since ImageNet does not have any image classes that are comparable to the dataset the authors use. They then proceed to do a hyperparameter sweep to fine-tune on their dataset.\n\nIt is also not explained why they chose to average outputs of 500 output nodes to get two outputs, instead of simply replacing the last layer with a 2-neuron layer and finetune, as is general practice.\n\nThe level of detail in the training procedure is very helpful to reproduce the setting as well as establish a reference for any future work in this direction. This itself is a notable achievement and a good use of significant research time. Furthermore, the authors conduct one good analysis (DTD) to explain the results of their CNN. The conclusion has many repeated points from previous sections, but it is a good summary.\n\nGiven their premise about explainability in machine learning, perhaps more significance was to be given to DTD, and other methods also performed to check if the results coincide with those from DTD.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SkxFdq2Y9r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a case study of training a video classifier using convolutional networks, and on how the learned features related to previous, hand-designed ones. The particular domain considered is of importance for biologists/medical doctors/neuroscientists: zebra fish swim bout classification.\n\nIn order to identify which particular features the neural networks are paying attention to, the paper used Deep Taylor Decomposition, which allowed the authors to identify \"clever-hans\"-type phenomena (network attending to meaningless experimental setup differences that actually gave a way the ground truth classes). This allowed for the authors to mask out such features and make the network attend to more meaningful ones. In particular observations like \" looking for salient features in the trunk of the tail while largely disregarding the tip\", are typically absent from most deep learning studies and it's quite interesting.\n\nOverall the paper is well written, the experiments are well designed; everything seems very rigorous and  well executed. It makes for a very good quality practitioner-level case study on video understanding, which may also be useful for people studying zebra fish or related simple life forms. My main concern with the paper is whether ICLR is an appropriate venue, as it does not provide pure machine learning contributions in the form of new techniques of generally applicable insights. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Skl0dCZk5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "the paper uses model interpretation techniques to understand blackbox CNN fit of zebrafish videos. they show that, relying on the technique of deep taylor decomposition, their CNN relies its prediction on a different part of zebra fish than existing understanding. it is also able to detect the use of experimental artifacts, whose removal improves predictive performance. \n\nthe idea of a case study about the usefulness of model interpretation techniques is interesting. while the experimental studies rely on our belief that the interpretation technique indeed interprets, the result that removing experimental features and improving predictive performance is convincing and interesting. it illustrates how model interpretability and human intuition and domain knowledge can be useful.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a case study of training a video classifier and subsequently analyzing the features to reduce reliance on spurious artifacts. The supervised learning task is zebrafish bout classification which is relevant for biological experiments. The paper analyzed the image support for the learned neural net features using a previously developed technique called Deep Taylor Decomposition. This analysis showed that the CNNs when applied to the raw video were relying on artifacts of the data collection process, which spuriously increased classification accuracies by a "clever Hans" mechanism. By identifying and removing these artifacts, a retrained CNN classifier was able to outperform an older SVM classifier. More importantly, the analysis of the network features enabled the researchers to isolate which parts of the zebrafish motion were relevant for the classification.

The reviewers found the paper to be well-written and the experiments to be well-designed. The reviewers suggested a some changes to the phrasing in the document, which the authors adopted. In response to the reviewers, the authors also clarified their use of ImageNet for pre-training and examined alternative approaches for building saliency maps.

This paper should be published as the reviewers found the paper to be a good case study of how model interpretability can be useful in practice. 
Decision: Accept (Poster)