dataesr/scientific-paragraphs-categorization huggingface dataset:
1, we used the CLASH regions to evaluate whether the occurrences of a motif are globally well placed or not. These results are presented in the Supplementary Tables  S2  and S3  for each miRNA with at least 2 interaction sites and for each miRNA with at least 10 interaction sites. The results are shown when considering all classes together (first row), and then each one separately. In the Seq W i datasets, however, the values of the strong version of the F-score decrease rather importantly, and this called for a refinement of the analysis carried out. Additionally, we computed the F-score for each miRNA individually to verify the distribution of these metrics (Supplementary Figure  S9 ). We checked the F-score computed on the occurrences of a motif (instead of all occurrences of all motifs in the table), which emphasizes the fact that the majority of the motifs, as expected, do not correspond to the interaction sites but to other conserved part of the sequences.
There are, however, a few motifs that have near perfect F-scores, which means that they correspond at least partially to interaction sites. These conserved motifs are as expected once again to be found more in the first three classes of the human dataset (because of the seed) and less so in relation to Classes 1 and 2 in the mouse dataset. We then verified if the best motifs detected by MEME matched the seed region or the miRNA sequence by comparing the matrices of sequences with TOMTOM  [27]  from the MEME suite. For this purpose, the seed regions or miRNA sequences were transformed into a matrix with, for each position, a 1 for the nucleotide present at that position and 0 for the others. Figure  5  shows the boxplot of the p-values given by TOMTOM when comparing either the seed or the complete miRNA sequence to the motifs predicted by MEME with the site distribution oops (one or plus) when the input contains less noise and only with miRNAs having more than 10 targets.
When using only the region detected by CLASH relative to the miRNAs having more than 10 targets, the classes in which the seed is very well defined (I, II, and III in the human dataset and 1, 2, 3, and 6 in the mouse dataset) showed a better correspondence between the best motif detected and both the seed and the miRNA. Although there is still a correlation between seeds and motifs in Class IV, this trend is lost when using the complete miRNA sequence and not seen at all in both cases for Class V. The same is not observed in the mouse dataset. Nevertheless, it is interesting to note that by adding more noise to the dataset by either using miRNAs with 2 or more targets or by extending the regions with 200 nucleotides down-and upstream (Supplementary Figures  S10C  and S11C ), the correlation between motifs and seed, or between motifs and miRNA, decreases considerably.
Conclusions and Perspectives In this paper, we revisited the importance of two specific features to predict putative miRNA targets, namely accessibility of the mRNA and seed anchoring, by comparing three state-of-the-art methods (MIRANDA, PITA, and INTARNA). To this end, we used datasets based on the CLASH technique, which allowed us to obtain information of both the miRNA along with its mRNA interaction sites. While the three selected methods rely on seed matches (although calculated differently and not used in the same order among the different features), MIRANDA is the only method that does not take into account accessibility. INTARNA was the only method in which we were able to fully exclude both features separately to more fully understand the impact of their absence in the results. As a perspective, it would be very interesting to be able to exclude these two features in other methods in order to investigate whether these and possibly other features are essential to or do not affect the overall results obtained.
This information is rather important to keep in mind since the largest portion of the datasets used to predict miRNA targets are rather noisy. In some cases, although we can predict regulatory couples by detecting a correlation of expression between a given miRNA and its putative mRNA targets, most sequencing experiments do not provide the exact site of interaction between a miRNA and its mRNA target. Moreover, despite the results obtained in this first analysis on the possible usefulness of exploiting intra-species conservation and the associated over-representation of motifs to infer new interaction sites and thus also new miRNAs, it might be interesting to continue investigating this idea in the future, perhaps with a refinement on how such motifs are inferred. One path that might be worth exploring would be limiting such motif inference to mRNA regions with high accessibility, thereby potentially reducing the amount of noise in the input given to tools such as MEME.
Number of interactions for the Seq C dataset Total UTR CDS 3 UTR Other mRNA regions Class I Class II Class III Class IV Class V 18,497 868 11,112 6093 424 3590 3290 4629 3382 3606 Number of interactions for the Seq W dataset Other Total UTR CDS 3 UTR mRNA Class I Class II Class III Class IV Class V regions 15,754 291 10,304 4916 243 3095 2833 3970 2890 2966 2. 1. 2. Mouse CLASH Data.
The output of a model from a sample is called a prediction. To solve more and more complex tasks and to obtain state-of-the-art accuracies (percentages of correct predictions), new models are built with more depth and parameters  (Simonyan and Zisserman, 2014; He et al. , 2016). They raise new challenges: First, as the set of filters that forms a model is learned by the machine from datasets, it is arduous for a human to understand the inner working of a neural network  (Cashman et al. , 2019). One of the challenges in eXplainable Artificial Intelligence (XAI)  (Gilpin et al. , 2018)  is to understand how decisions are made in neural networks. That implies to understand the role of the layers and their filters. The more filters a model has, the more filters need to be analyzed to understand a decision.
Secondly, the computational cost and memory requirements of these neural networks complicate their use on small IoT devices. A belief in AI is that the more an architecture contains filters, the more it will succeed in learning from a dataset. For example, the uses of artificial intelligence (AI) design patterns such as residual blocks make architectures increasingly deep  (He et al. , 2016). If a model does not achieve to solve a complex task, a new architecture is built with more repetitions of the same pattern  (He et al. , 2016) , so that the final architecture is oversized  (Liu et al. , 2017). During the learning phase, the assignment of the parameters is computationally expensive and requires high memory capacities as well as a long training period. At prediction time, even though the number of operations per instance in a dataset is constant for a model, it can easily reach the order of the million or the billion (665 million operations per small-sized image on  VGG-16 LeCun et al.
(f) The Sparklines Matrix view, an important feature of our tool, allows to estimate the discriminative power of the selected filters. A dark (resp. light) color indicates a high (resp. low) discriminative power. Luo et al. , 2017; He et al. , 2017; Yu et al. , 2018; Zhou et al. , 2019; He et al. , 2019; Wang et al. , 2020; Li et al. , 2020). In software engineering, especially in eXtreme-Programming, one first develops a working solution and then refactors and fine-tunes that solution  (Martin et al. , 2003)  to optimize it. We argue that AIbased solutions development should follow the same practices: AI software developers need development environments to ease this refactoring and fine-tuning. Our objective in this paper is to provide tools to analyze and guide the refactoring and fine-tuning of an existing neural network (DNN) on a classification task, to reduce its computational cost with a minimal negative impact on its performance.
Both approaches are useful, we argue that automatic method could leverage fine-tuned strategies found by the AI designer to obtain better results. Our contributions contain: • The definition of functions to aggregate the filters outputs (i. e. , activation maps) that can be used to guide the pruning. • A novel local Class Pairwise Difference (CPD) measure of the importance of the filters in the discrimination of two classes. • The definition of a complete visual pruning method based on a novel sparklines matrix view. • Two case studies and a user study with several experts on well-known network architectures and datasets that show the efficiency of the approach. In the following, we first present a brief state of the art of the automatic approaches to prune a network (Section 2), and then the most common visualization techniques to analyze DNNs.
We follow by establishing the goals (Section 3) a pruning tool should address. Then, we describe step by step our methodology (Section 4) to prune a network and continue by the description of the components we developed (Section 5) within our tool to meet our objectives. Then, we present two case studies on convolutional networks for well-known datasets (Section 6). We also discuss the strengths, the weaknesses and future improvements (Section 7) of our method, and finish with a conclusion.
Concerning the pruning criteria, data-agnostic criteria can be distinguished from data-aware criteria  (Liu et al. , 2020). Dataagnostic criteria evaluate the importance of weights or filters, regardless any dataset. The most important weights can be the ones with the strongest values  (Han et al. , 2015b)  or the ones that have the strongest impact on the objective function  (LeCun et al. , 1989). Removing individual weights has the disadvantage to create tensors sparsity, which does not directly speedup the execution. Instead, several filters (i. e. , weight tensor slices) can be removed at a time. The filters to remove can be those having the lowest L1-norm values  (Li et al. , 2016a)  or the filters close to the geometric median  (He et al. , 2019)  to remove redundancies. Meanwhile, the data-aware category considers the filters through their outputs on data.
In that category, the filters to remove can be those producing the highest Average Percentage of Zeros (APoZ) on data  (Hu et al. , 2016) , those having the least impact on the activation maps  (He et al. , 2017; Luo et al. , 2017) , on the objective function  (Molchanov et al. , 2017) , the redundant filters  (Zhou et al. , 2019)  or the ones that do not discriminate the classes  (Yu et al. , 2018). Even though, both categories yield good results and allow model reductions, we believe data-aware criteria enable a better analysis of the filters because the filters of a model are the result of a learning phase on a specific dataset. At the frontier between KD and pruning techniques,  Liu et al. (2019)  presented a method involving a pruning neural network that learns to generate a pruned network with its weights, from a dataset made of encoded subnetworks architectures.
This approach is interesting as it tries to automatically find the best subnetwork within the wide space of all subnetworks. However, the performance of the subnetwork strongly depends on the dataset of randomly chosen encoded architectures. This dataset may not be representative enough of the subnetworks space to reach an optimal solution. As  Cashman et al. (2019)  suggested, a user may intuitively find strategies to reach better solutions within the space of subnetwork architectures if given visual tools to guide her exploration. More recently,  Li et al. (2020)  developed an interactive visualization tool, CNNPruner to make neural networks pruning an interpretable process. They combine an automatic approach with a method to quantify and visualize filters importance within all layers at once, that gives users the opportunity to manually refine a pruning strategy.
In their paper, a pruning strategy refers to a set of pruning steps; a pruning step starts from a pre-trained architecture, reduces the number of its parameters and finishes with a new training phase to try to recover initial performances. This method neither assesses if the pruning process overfits the dataset used for testing on each iteration, nor specifies whether the pruning iterations preserve the local performance (performance for individual classes). In our work, we want to explore more in-depth this idea of manual pruning, providing tools to a user to determine the important parts of a network and assess the generalization power of a pruned network.
, 2020). The comparison at a sample level includes the identification of similarities or differences on the classification of an instance between two models. For instance,  Li et al. (2020)  suggested analyzing improved and degenerated instances to get further insights on the role of the removed filters. T4: Assess the generalization power of a pruned model. Removing filters to get a smaller submodel is commonly done through several iterations of pruning  (Molchanov et al. , 2017). At each pruning iteration, the same test dataset is used to compute metrics and help the user decide which filters to keep in the new submodel. To detect any overfitting phenomenon to the test dataset during this process, an unseen dataset has to be preserved since the beginning, to evaluate the resulting submodel.
Pruning process overview As illustrated in Fig. 2 , we use the pruning process presented by  Molchanov et al. (2017) , as a guideline. The process starts with a base model, trained until decent accuracy is reached. In this network, each filter is given a score with several metrics that estimate its importance from different perspectives. The least important filters are pruned which may imply a drop on performances. To try to recover this performances loss, the pruned network is fine-tuned (i. e. , trained with a couple of epochs). The resulting model is analyzed and can be the starting point of a new pruning iteration. The number of pruning iterations depends on criteria that can be a performance limit or a minimum pruning factor. The training dataset (see Fig. 2 ) is used to train the base model.
At each epoch of the training phase, a second dataset (i. e. , validation dataset) is used to evaluate and select the best network that does not overfit the training set. The test dataset is used in the end of the training phase of any network to evaluate its final performance on unseen data. With this approach, after several iterations, the final pruned model risk to overfit the test dataset. To tackle this problem, we use the evaluation dataset in the end of the pruning process to evaluate the pruned models performances on unseen data (T4).
To compute these metrics for each filter, at test time, all samples of the test dataset are fed to the model that produces one feature map for each filter of each layer and each sample. We consider the filters outputs after the activation function rectifies their values. The outputs of this function are called activation maps (AM). All the AM of layer L have a dimension of: [N, w, h, m L ] for 2D-convolutional layers, N being the number of input samples considered, w and h being respectively the width and height of the activation maps and m L being the number of channels (or filters) in the layer L. We consider fully-connected layers similarly, with w = h = 1. To make meaningful comparisons, the raw AM must be normalized, since the filters that produce them do not have automatically either the same size or the same level of abstraction.
Performances view The performances view (Fig. 1b ) displays the performance of the selected model (R1), or its comparison to another model (R8) at global scale (Performance summary), at class or at a sample scale (Class performance). Performance summary: A table provides information about the model general performance (Fig. 1b1 ). It displays the model global accuracy, its total number of parameters and the number of FLOPs(FLoating-point OPerations) needed to process a single sample (R1). An extra row displays the performance on the evaluation dataset, for the last step of the pruning process (Fig. 2 ). If the selected model is not the base model, it also displays the variation rates between the base model and this selected model (R8). The accuracy obtained at each epoch of the training/fine-tuning is plotted for both training and validation datasets to show how the selected model converges (Fig.
4b ) designed by  Pommé et al. (2022)  to efficiently display these differences of performance at a class level (R8). In each cell of this matrix, a + sign (resp. -) represents an improvement (resp. a deterioration) of model B compared to model A for the corresponding pair (ground truth, prediction). The white-to-blue gradient represents the intensity of the improvement or of the deterioration. For example, the cell (3, 3) indicates a strong deterioration (model B correctly classified fewer samples of class 3 than model A). The cell (4, 9) indicates a lower deterioration (model B misclassified more samples of class 4 as being of class 9). The confusion matrix cells are clickable to show well-classified or misclassified samples of the test dataset that contribute to the selected cell (Fig. 1b4 ).
To select the filters, the global metrics may be used first as they were shown to detect the global importance of filters in previous works  (Zhong et al. , 2017; Li et al. , 2016a; Hu et al. , 2016). Though, to focus on the least discriminated pairs of classes, one may need to focus on the associated sparklines to keep the locally important filters. For datasets like Cifar-100  (Krizhevsky, 2009)  or ImageNet  (Russakovsky et al. , 2015) , the number of classes hinders a display of the entire matrix with all the sparklines. Instead, as hierarchical confusion matrices  (Görtler et al. , 2022) , the sparklines visualization profits from the hierarchical structure of the datasets to adopt a hierarchical structure. The average activations (5) need to be computed at each level of the hierarchy, the notion of class depending on the target level.
Experiments This section first presents a study we conducted on several cases of various difficulties (pairs dataset/network). Then, we present a user study conducted with four machine learning experts to verify if our approach allows identifying important filters and prune the models with minor performance losses. We consider three datasets: Dogs vs. Cats  (Dataset, 2022) ,  Cifar-10 (Krizhevsky, 2009)  and MNIST (LeCun et al. , 2010) (see Table  1 ).
Case studies Two case studies are presented in this section to illustrate how NetPrune helps the user to freely explore the submodels space and define her own pruning strategies. The first one illustrates several pruning strategies and their impact on a binary classification problem with a small-sized network. The second one illustrates how the knowledge acquired in the first case study about pruning strategies can be reused and adapted on a more complex problem. 6. 1. 1. Dogs vs. Cats, hand-crafted CNN Dogs vs. Cats (Dataset, 2022)  is a dataset of images categorized in two balanced classes: cats and dogs. The dataset is originally composed of 25 000 labeled images. We first randomly extracted 1250 images of cats and 1250 images of dogs from this dataset to generate each subset  (validation, test, evaluation) , as shown in Table  1.
We replicated the seven-layer-architecture  Li et al. (2020)  used in CNNPruner for this dataset. The initial model achieves 93. 14% of accuracy (94. 08% on cats and 92. 40% on dogs) for 40 epochs of training under the Adamax optimizer with a learning rate of 0. 001. The model has 6. 89M parameters and needs to process 4. 6G FLOPs for the prediction of a single instance. To prune the neural network with NetPrune, we first needed to choose which layer(s) to prune, and then choose which filters to keep in the target layer(s). We consider two distinct strategies to choose the layer(s) to prune. The first one (S all ), inspired by the literature  (Li et al. , 2020) , consists in pruning all layers at once, except the prediction layer. We first pruned drastically all layers of the base model at once, with the objective to quicky and drastically reduce the complexity of the model.
The model dropped the performance by 1. 76% after removing 89. 38% of the parameters and 92. 43% of the FLOPs at once. Another model dropped less accuracy (-0. 54%) after removing also less parameters (-65. 40%) and less FLOPs (-65. 72%) in one step. Repeating that manipulation several times from the same model while keeping high compression rates did not give any better result for this dataset. On the opposite, removing only 13. 23% of the parameters and 22. 10% of the FLOPs lead to an increase of 0. 06% in the accuracy with a faster recovery within the first epochs, which sped-up the finetuning phase. Using a binary search, one can try and find a balance between pruning as much filters as possible in all layers and keeping an acceptable accuracy after the fine-tuning phase. Even if our results tend to demonstrate that this strategy may not be precise enough to reach state-of-the-art results alone, it seems to be a good entry point to start pruning models.
(2020). The measures are presented for three pruned models obtained by combining our strategies. Our reference model reaches 93. 14% of accuracy on the test dataset after 40 epochs of training. Li et al. (2020)  fixed a threshold to 92. 5% of accuracy under which a pruned model is to be discarded. For fairness, since the accuracy of our Base model is higher by 0. 38% than the CNNPruner Base model, we chose our threshold to 93. 00%. The first pruned model we report outperforms our Base model by 1. 64% with -81. 90% of the initial parameters, while having -78. 16% of the initial computing operations per input data. The final evaluation revealed that this model is also performing well on the unseen evaluation dataset with an increase of 0. 70% compared to the Base model. The second pruned model (top of #Params) is the model containing the fewest number of parameters (-99.
63%) and FLOPs (-98. 23%). Its accuracy on the test dataset equals the threshold. It could have been considered the best trade-off model if its accuracy on the evaluation dataset had not been below the threshold. The last reported pruned model in Table  2  has more remaining parameters but shows a better accuracy on both test and evaluation datasets than the second pruned model. The last reported model offers the best trade-off between accuracy, number of parameters and FLOPs.
Cifar-10, VGG-16 For the second case study, we considered Cifar-10 (Krizhevsky, 2009), a dataset of images categorized in 10 classes. The dataset is originally composed of 50 000 labeled train images and 10 000 labeled test images. We randomly extracted 500 images per class for both validation and evaluation sets (see Table  1 ). We replicated a modified architecture of VGG-16 1  with batch normalization layers. We trained it from scratch for 250 epochs under the SGD optimizer, with an initial learning rate of 0. 1. Our reference model reaches 88. 50% of accuracy on the test dataset. The model has 15. 00M parameters and needs to process 627. 91M FLOPs for the prediction of each input data. For this network, we applied the same strategies found in the previous case study to select the layers to prune.
Since VGG-16 is deeper than the previous network, we first pruned several layers at a time within the same convolutional block to accelerate the process. We stopped when the overall accuracy began to drop too much. Unlike the previous scenario, this case is a dataset of 10 classes. Using the CPD metric only can be tedious because of the number of combinations to take into account (here, 90 combinations). To avoid considering all combinations, we defined a new strategy (S bottleneck ). With this strategy, we first consider all the classes globally, making a pre-selection of filters with global metrics. Then, we focus on the problematic classes, which can be the classes with the worst performance in the confusion matrix view or the least discriminated classes in the Sparklines view, by completing the selection of filters with CPD.