Coarse-to-Fine Semi-Structured Pruning of Graph Convolutional Networks for Skeleton-based Recognition

Authors: hichem  sahbi
Venue: ICML_cc_2024_Workshop_WANT
Published: 2024-06-02 21:59:52

Abstract: Deep neural networks (DNNs) are nowadays witnessing a major success in solving many pattern recognition tasks including skeleton-based classification. The deployment of DNNs on edge-devices, endowed with limited time and memory resources, requires designing lightweight and efficient variants of these networks. Pruning is one of the lightweight network design techniques that operate by removing unnecessary network parts, in a structured or an unstructured manner, including individual weights, neurons or even entire channels. Nonetheless, structured and unstructured pruning methods, when applied separately, may either be inefficient or ineffective.
In this paper,  we devise a novel semi-structured method that discards the downsides of structured and unstructured pruning while gathering their upsides to some extent. The proposed solution is based on a differentiable cascaded parametrization which combines (i) a band-stop mechanism that prunes weights depending on their magnitudes, (ii) a weight-sharing parametrization that prunes connections  either individually or group-wise, and (iii) a gating mechanism which arbitrates between different group-wise and entry-wise pruning. All these cascaded parametrizations are built upon a common latent tensor which is trained end-to-end  by minimizing a classification loss and a surrogate tensor rank regularizer. Extensive experiments, conducted on the challenging tasks of action and hand-gesture recognition, show the clear advantage of our proposed semi-structured pruning approach against both structured and unstructured pruning, when taken separately, as well as the related work.

Main paper: 
 introduction: 
 Deep neural networks (DNNs) are nowadays becoming a hotspot in machine learning with increasingly performant models used to approach eclectic pattern recognition tasks (Krizhevsky et al., 2017; He et al., 2016; 2017; Jian et al., 2020; Ronneberger et al., 2015; Jiu & Sahbi, 2017; 2019) . These models are also steadily oversized and this makes their deployment on cheap devices, endowed with limited hardware resources, very challenging. In particular, hand-gesture recognition and human computer interaction tasks require fast and lightweight DNNs with high recognition performances. However, DNNs are currently showing some saturated improvement in accuracy while their computational efficiency remains a major issue. Among these DNN models, graph convolutional networks (GCNs) are deemed effective especially on non-euclidean domains including skeleton-data (Zhu et al., 2016b) . Two families of GCNs exist in the literature: spectral and spatial. Spectral methods project graph signals from the input to the Fourier domain, achieve convolution, prior to back-project the convolved signals in the input domain (Kipf & Welling, 2016; Levie et al., 2018; Li et al., 2018b; Defferrard et al., 2016; Bruna et al., 2013; Henaff et al., 2015; Chung, 1997; Sahbi, 2021c; Mazari & Sahbi, 2019b) . Spatial methods proceed differently by aggregating signals through neighboring nodes, using multi-head attention, prior to achieve convolutions (as inner products) on the resulting node aggregates (Gori et al., 2005; Micheli, 2009; Wu et al., 2020; Hamilton et al., 2017; Knyazev et al., 2019; Sahbi et al., 2011; Sahbi, 2021b; a) . Spatial GCNs are known to be more effective compared to spectral ones. Nonetheless, with multi-head attention, spatial GCNs become oversized, computationally overwhelming, and their deployment of cheap devices requires making them lightweight and still effective (Huang et al., 2018a; Sandler et al., 2018; Howard et al., 2017; Tan & Le, 2019; Cai et al., 2019; He et al., 2018a; b; Sahbi, 2021d; 2023) . Several existing works address the issue of lightweight network design, including tensor decomposition (Howard et al., 2019) , quantization (Han et al., 2015a) , distillation (Hinton et al., 2015; Mirzadeh et al., 2020; Zhang et al., 2018; Ahn et al., 2019; Sahbi & Geman, 2006) , neural architecture search (Li et al., 2022) and pruning (LeCun et al., 1989; Has-sibi & Stork, 1992; Han et al., 2015b; Sahbi, 2022) . Pruning methods are particularly effective, and their general recipe consists in removing connections in order to enable reduced storage and faster inference with a minimal impact on classification performances. One of the mainstream methods is magnitude pruning (MP) (Han et al., 2015a) which removes the smallest weight connections before retraining the pruned networks. Two categories of MP techniques exist in the literature: unstructured (Han et al., 2015b; a) and structured (Li et al., 2016; Liu et al., 2017d) . Unstructured methods remove weights individually by ranking them according to the importance of their magnitudes whilst structured approaches zero-out groups of weights (belonging to entire rows, columns, filters or channels) according to the importance of their aggregated magnitudes. Unstructured MP results into more flexible, accurate networks, and allows reaching any fine-grained targeted pruning rate but requires dedicated hardware to actually achieve efficient computation. In contrast, structured MP offers a more practical advantage by making the resulting DNNs compatible with standard hardware for efficient computation. However, this comes at the expense of a reduced classification performance and coarse-grained pruning rates. In order to fully exhibit the potential of these two pruning categories, a more suitable framework should gather the upsides of both structured and unstructured pruning while discarding their downsides to some extent. In this paper, we introduce a novel variational MP approach that leverages both structured and unstructured pruning. This method dubbed as semi-structured is based on a differentiable cascaded weight parametrization composed of (i) a band-stop mechanism enforcing the prior that the smallest weights should be removed, (ii) a weight-sharing that groups mask entries belonging to the same rows, columns, or channels in a given tensor, and (iii) a gating mechanism which arbitrates between different mask group assignments while maximizing the accuracy of the trained lightweight networks. We also consider a budget loss that allows implementing any targeted fine-grained pruning rate and reducing further the rank of the pruned tensors, resulting into more efficient networks while being closely accurate as shown later in experiments. 
related work: 
 The following review discusses the related work in variational pruning and skeleton-based recognition, highlighting the limitations that motivate our contributions.Variational Pruning. The general concept behind variational pruning is to learn weights and binary masks that capture the topology of pruned networks. This is achieved by minimizing a global loss that combines a classification error and a regularizer that controls the sparsity (or the cost) of the resulting networks (Liu et al., 2017d; Wen et al., 2016; Louizos et al., 2017) . However, these approaches are powerless to implement any given targeted pruning rate without overtrying multiple weighting of the regularizers. Alternative methods explicitly model the network cost using ℓ 0 -based criteria (Louizos et al., 2017; Pan et al., 2016) in order to minimize the discrepancy between the observed and the targeted costs. Existing solutions rely on sampling heuristics or relaxation, which promote sparsity -using different regularizers (ℓ 1 /ℓ 2 -based, entropy, etc. ) (Gordon et al., 2018; Carreira-Perpinán & Idelbayev, 2018; Koneru & Vasudevan, 2019; Wiedemann et al., 2019) -but are powerless to implement any given targeted cost exactly and result in overpruning effects leading to disconnected subnetworks. Furthermore, most of the existing solutions, including magnitude pruning (Han et al., 2015a) , decouple the training of network topology from weights, making the learning of pruned networks suboptimal. On another hand, existing pruning methods are either structured (Li et al., 2016; Liu et al., 2017d) or unstructured (Han et al., 2015b ;a) so their benefit is not fully explored. In contrast to the aforementioned related work, our contribution in this paper seeks to leverage the advantage of both structured and unstructured pruning where the training of masks and weights are coupled on top of shared latent parameters.Skeleton-based recognition. This task has gained increasing interest due to the emergence of sensors like Intel Re-alSense and Microsoft Kinect. Early methods for handgesture and action recognition used RGB (Liu & Yuan, 2018; Hu et al., 2015; Wang & Sahbi, 2013; Yuan et al., 2012; Wang & Sahbi, 2014) , depth (Ohn-Bar & Trivedi, 2014; Wang et al., 2015) , shape / normals (Oreifej & Liu, 2013; Rahmani & Mian, 2016; Yun et al., 2012; Ji et al., 2014; Li et al., 2015; Zanfir et al., 2013; Sahbi, 2007; Sahbi & Fleuret, 2004) , and skeleton-based techniques (Wang et al., 2018) . These methods were based on modeling human motions using handcrafted features (Xia et al., 2012; Yang & Tian, 2014) , dynamic time warping (Vemulapalli et al., 2014) , temporal information (Zhang et al., 2016; Garcia-Hernando & Kim, 2017) , and temporal pyramids (Zhu et al., 2016a; Q. De Smedt & Vandeborre, 2016) . However, with the resurgence of deep learning, these methods have been quickly overtaken by 2D/3D Convolutional Neural Networks (CNNs) (Feichtenhofer et al., 2016; Nunez et al., 2018a; Mazari & Sahbi, 2019a) , Recurrent Neural Networks (RNNs) (Zhu et al., 2016b; Chen et al., 2017; Ke et al., 2017; Liu et al., 2017c; Liu & Yuan, 2018; Wang et al., 2016; Du et al., 2015; Wang & Wang, 2017; Liu et al., 2017b; Nunez et al., 2018b; Shahroudy et al., 2016; Zhang et al., 2017b; Lee et al., 2017; Liu et al., 2016; Maghoumi & LaViola, 2019; Zhang et al., 2017a; Liu et al., 2017a) , manifold learn-ing (Huang & Van Gool, 2017; Huang et al., 2018b; Kacem et al., 2018) , attention-based networks (Liu et al., 2021; Weng et al., 2018; Hou et al., 2018; Chen et al., 2019; Song et al., 2017) , and GCNs (Huang et al., 2017; Li et al., 2018a; Yan et al., 2018; Wen et al., 2019; Shi et al., 2018; Nguyen et al., 2019; Li et al., 2019; 2020) . The recent emergence of GCNs, in particular, has led to their increased use in skeleton-based recognition (Li et al., 2018b) . These models capture spatial and temporal attention among skeleton joints with better interpretability. However, when tasks involve relatively large input graphs, GCNs (particularly with multi-head attention) become computationally inefficient and require lightweight design techniques. In this paper, we design efficient GCNs that make skeleton-based recognition highly efficient while also being effective. 
a glimpse on graph convolutional networks: 
 Let S = {G i = (V i , E i )} i denote a collection of graphs with V i , E i being respectively the nodes and the edges of G i . Each graph G i (denoted for short as G = (V, E)) is endowed with a signal {ϕ(u) ∈ R s : u ∈ V} and associated with an adjacency matrix A. GCNs aim at learning a set of C filters F that define convolution on n nodes of G (withn = |V|) as (G ⋆ F) V = f A U ⊤ W , here ⊤ stands for transpose, U ∈ R s×n is the graph signal, W ∈ R s×C isthe matrix of convolutional parameters corresponding to the C filters and f (.) is a nonlinear activation applied entrywise. In (G ⋆ F) V , the input signal U is projected using A and this provides for each node u, the aggregate set of its neighbors. Entries of A could be handcrafted or learned so (G ⋆ F) V corresponds to a convolutional block with two layers; the first one aggregates signals in N (V) (sets of node neighbors) by multiplying U with A while the second layer achieves convolution by multiplying the resulting aggregates with the C filters in W. Learning multiple adjacency (also referred to as attention) matrices (denoted as {A k } K k=1 ) allows us to capture different contexts and graph topologies when achieving aggregation and convolution. With multiple matrices {A k } k (and associated convolutional filter parameters{W k } k ), (G⋆F) V is updated as f K k=1 A k U ⊤ W k .Stacking aggregation and convolutional layers, with multiple matrices {A k } k , makes GCNs accurate but heavy. We propose, in what follows, a method that makes our networks lightweight and still effective. 
proposed method: semi-structured magnitude pruning: 
 In what follows, we formally subsume a given GCN as a multi-layered neural network g θ whose weights are defined as θ = W 1 , . . . , W L , being L its depth, W ℓ ∈ R d ℓ-1 ×d ℓ its ℓ th layer weight tensor, and d ℓ the dimension of ℓ. The output of a given layer ℓ is defined asϕ ℓ = f ℓ (W ℓ ⊤ ϕ ℓ-1), ℓ ∈ {2, . . . , L}, with f ℓ an activation function; without a loss of generality, we omit the bias in the definition of ϕ ℓ . Pruning consists in zeroing-out a subset of weights in θ by multiplying W ℓ with a binary maskM ℓ ∈ {0, 1} d ℓ-1 ×d ℓ .The binary entries of M ℓ are set depending on whether the underlying layer connections are pruned, soϕ ℓ = f ℓ ((M ℓ ⊙ W ℓ ) ⊤ ϕ ℓ-1), here ⊙ stands for the elementwise matrix product. In our definition of semi-structured pruning, entries of the tensor {M ℓ } ℓ are set depending on the prominence and also on how the underlying connections in g θ are grouped; pruning that removes the entire connections individually (resp. jointly) is referred to as unstructured (resp. structured) whereas pruning that removes some connections independently and others jointly is dubbed as semi-structured. However, such pruning (with {M ℓ } ℓ ) suffers from several drawbacks. In the one hand, optimizing the discrete set of variables {M ℓ } ℓ is deemed highly combinatorial and intractable especially on large networks. In the other hand, the total number of parameters {M ℓ } ℓ , {W ℓ } ℓ is twice the number of connections in g θ and this increases training complexity and may also lead to overfitting.In order to overcome the aforementioned issues, we consider an alternative parametrization that allows finding both the topology of the pruned networks together with their weights, without doubling the size of the training parameters, while making magnitude pruning semi-structured and learning still effective. This parametrization corresponds to the Hadamard product involving a weight tensor and a cascaded function applied to the same tensor asEQUATIONbeing Ŵℓ a latent tensor and ψ( Ŵℓ ) (withψ = ψ 3 •ψ 2 •ψ 1 )a continuous relaxation of M ℓ which enforces the prior that (i) smallest weights Ŵℓ should be removed from the network, (ii) the underlying mask entries ψ( Ŵℓ ) are shared (across tensor rows, columns, channels, etc.) when pruning is structured, and (iii) any given mask entry in ψ( Ŵℓ ) is either unstructurally or structurally pruned. In what follows, we detail the different parametrizations used to define ψ( Ŵℓ ); unless explicitly mentioned, we omit ℓ in the definition of Ŵℓ and we rewrite it simply as Ŵ.Band-stop Parametrization (ψ 1 ). This parametrization ψ 1 is entry-wise applied to the tensor Ŵ and enforces the prior that smallest weights should be removed from the network.In order to achieve this goal, ψ 1 must be (i) bounded in [0, 1],(ii) differentiable, (iii) symmetric, and (iv) ψ 1 (ω) ⇝ 1 when |ω| is sufficiently large and ψ 1 (ω) ⇝ 0 otherwise. The first and the fourth properties ensure that the parametrization is neither acting as a scaling factor greater than one nor changing the sign of the latent weight, and also acts as the identity for sufficiently large weights, and as a contraction factor for small ones. The second property is necessary to ensure that ψ 1 has computable gradient while the third condition guarantees that only the magnitudes of the latent weights matter. A choice, used in practice, that satisfies these four conditions isEQUATIONbeing σ a scaling factor that controls the crispness (binarization) of mask entries in ψ 1 ( Ŵ). According to Eq. 2, σ controls the smoothness of ψ 1 around the support of the latent weights. This allows implementing an annealed (soft) thresholding function that cuts-off all the connections in a smooth and differentiable manner as training of the latent parameters evolves. The asymptotic behavior of ψ 1 -that allows selecting the topology of the pruned subnetworksis obtained as training reaches the latest epochs, and this makes mask entries, in ψ 1 ( Ŵ), crisp and (almost) binary. This mask ψ 1 ( Ŵ) (rewritten for short as ψ 1 ) is used as input to the subsequent parameterizations ψ 2 and ψ 3 as shown below.Weight-sharing Parametrization (ψ 2 ). This parametrization ψ 2 implements semi-structured pruning by tying mask entries belonging to the same rows, columns or channels in the tensor ψ 1 . More precisely, each mask entry in ψ 2 (ψ 1 ) will either be (i) entry-wise evaluated (dependent only on its underlying weight), or (ii) shared through multiple latent weights belonging to the same row, column or channel of ψ 1 resulting into the following multi-head parametrization (see Fig. 1 )ψ 2 (ψ 1 ) =                              ψ u 2 (ψ 1 ) = ψ 1 unstructured (entry-wise) ψ r 2 (ψ 1 ) = vec -1 (P r vec(ψ 1 )) structured (row-wise) ψ c 2 (ψ 1 ) = vec -1 (vec(ψ 1 ) ⊤ P c ) structured (column-wise) ψ b 2 (ψ 1 ) = vec -1 (P r P ⊤ c vec(ψ 1 )) structured (block/channel-wise),(3) here vec (resp. vec -1 ) reshapes a matrix into a vector (resp. vice-versa), andP r ∈ {0, 1} (d ℓ-1 ×d ℓ ) 2 , P c ∈ {0, 1} (d ℓ-1 ×d ℓ ) 2are two (layer-wise fixed) adjacency matrices that model the neighborhood system across respectively the rows and the columns of ψ 1 (in other words, [P r ] ij,pq ̸ = 0 iff the two connections (i, j), (p, q), in a given layer, share the same neuron, i.e., i = p) whilst the product P r P ⊤ c ∈ {0, 1} (d ℓ-1 ×d ℓ ) 2 models this neighborhood through blocks/channels of ψ 1 . When composed (with ψ 1 ), the mask ψ 2 inherits all the aforementioned fourth properties: mask entries in ψ 2 (ψ 1 ) remain bounded in [0, 1], differentiable, symmetric, and close to 1 when entries of the latent tensor Ŵ (i.e., inputs of ψ 1 ) are sufficiently large and 0 otherwise.Gating Parametrization (ψ 3 ). As each connection in g θ is endowed with a multi-head parametrization ψ 2 , we define ψ 3 as a gating mechanism that selects only one of them. More precisely, each mask entry can either be (i) entrywise pruned, i.e., untied, or (ii) tied to its row, column or block/channel. Again with ψ 3 , the composed parametrization ψ 3 (ψ 2 ) is bounded in [0, 1], differentiable, symmetric and reaches 1 if the entries of the latent tensor Ŵ are sufficiently large, and 0 otherwise. Formally, the gating mechanism ψ 3 is defined asEQUATIONbeing ψb 2 = U -ψ b 2 , and U a tensor of ones with the same dimensions as ψ b 2 (and ψr 2 , ψc 2 , ψu 2 are similarly defined). It is easy to see that when entries in ψ 1 (and hence ψ 2 ) are crisp, at most one of these four terms is activated (i.e., equal to one) for each connection in g θ . From Eq. 4, block-wise pruning has the highest priority, followed by column-wise, row-wise and then entry-wise pruning respectively. This priority allows designing highly efficient lightweight networks with a coarse-granularity budget implementation for block/column/row-wise (structured) pruning, while entrywise (unstructured) pruning is less computationally efficient but allows reaching any targeted budget with a finer granularity, and thereby with a better accuracy. Note that this parametrization acts as a weight regularizer which not only improves the lightweightness of the pruned networks but also their generalization performances (as shown later in experiments). Note also that ψ 1 and ψ 2 are commutable in the cascaded parameterization ψ = ψ 3 • ψ 2 • ψ 1 but ψ 3 should be applied at the end of the cascade.By considering Eq. 1, we define our semi-structured pruning loss as EQUATIONψ 1 w i,j w i,j w i,j w i,j w i,j Ŵ w i,j ψ b 2 ψ c 2 ψ r 2 ψ u 2 w i,j ψ 3Figure 1 : This figure shows the three stages of the cascaded parametrization including (i) band-stop, (ii) weight-sharing and (iii) gating. Cyan stands for shared connections, and the triangle for the "not gate" operator. For ease of visualization, only 4 connections are shown during the whole evaluation of the parameterization, and only the outcome (1 or 0) of w i,j is shown in the final mask tensor.being 1 d ℓ a vector of d ℓ ones, and the left-hand side term is the cross entropy loss that measures the discrepancy between predicted and ground-truth labels. The right-hand side term is a budget loss that allows reaching any targeted pruning cost c. Nonetheless, it's worth noticing that actual efficiency is not only related to the pruning rate but also to the actual dimensionality of the tensors. In order to take full advantage of the semi-structured setting of our method, we complement the aforementioned budget function with another one that minimizes the rank of the pruned tensors {ψ 3 • ψ 2 • ψ 1 ( Ŵℓ } ℓ . However, as the rank is not differentiable, we consider a surrogate function (as an upper bound) of the rank 1 . Hence, Eq. 5 becomesEQUATIONbeing r[W] a surrogate differentiable rank function set in practice toEQUATIONbeing γ an annealed temperature and exp(.) is entry-wise applied. Eq. 7 seeks to minimize the number of non-null rows/columns in a given tensor W, and this allows achieving higher speedup compared to when only the budget loss is minimized (see experiments). In Eq. 6, β controls the 1 This function is an upper bound of the rank as it corresponds to the sum of the number of non-zero rows and non-zero columns."structureness" of pruning; large β favors stringent tensors first through blocks, columns and then through rows, while smaller β leads to mixed structured and unstructured pruning. Once the above loss optimized, actual rank minimization requires reordering dimensions layer-wise in order to fully benefit from compact tensors and eliminate fragmentation; this is achievable as only outward connections, from unpruned neurons in each layer, are actually pruned during optimization.Let L denote the global loss in Eq. 6, the update of { Ŵℓ } ℓ is achieved using the gradient of L obtained by propagating the gradients through g θ . More precisely, considering the parametrization in Eq. 1, the gradient of the global loss w.r.t. Ŵℓ is obtained asEQUATIONhere the original gradient ∂L/∂ψ( Ŵℓ ) is obtained from layer-wise backpropagation, and ∂L ∂ Ŵℓ is obtained by multiplying the original gradient by the three rightmost Jacobians (whose matrix forms are shown in Table 1 ). In the above objective function, β = 0.1 and λ is overestimated (to 1000 in practice) in order to make Eq. 6 focusing on the implementation of the budget. As training reaches its final epochs, the budget loss reaches its minimum and the gradient of the global objective function will be dominated by the gradient of L e (and to some extent by the gradient of the surrogate rank function); this allows improving both classification performances and efficiency as shown subsequently.Row-wise Column-wise Block-wise[J 1 ] ij,pq = 1 {ij=pq} ψ ′ 1 ( Ŵpq ) NA NA NA [J u 2 ] ij,pq = 1 {ij=pq} [J r 2 ] ij,pq = [P r ] ij,pq [J c 2 ] ij,pq = [P ′ c ] ij,pq [J b 2 ] ij,pq = [P r P ′ c ] ij,pq [J u 3 ] ij,pq = 1 {ij=pq} [J r 3 ] ij,pq = 1 {ij=pq} [J c 3 ] ij,pq = 1 {ij=pq} [J b 3 ] ij,pq = 1 {ij=pq} ×[ ψb 2 ⊙ ψc 2 ⊙ ψr 2 ] pq ×[ ψb 2 ⊙ ψc 2 ⊙ ψu 2 ] pq ×[ ψb 2 ⊙ ψr 2 ⊙ ψu 2 ] pq ×[ ψc 2 ⊙ ψr 2 ⊙ ψu 2 ] pqTable 1 : Jacobians of different parametrizations w.r.t. different settings; here[J 1 ] ij,pq = [∂ψ 1 /∂ Ŵ] ij,pq , [J x 2 ] ij,pq = [∂ψ x 2 / ∂ψ 1 ] ij,pq and [J x 3 ] ij,pq = [∂ψ 3 /∂ψ x 2] ij,pq with x ∈ {u, r, c, b}; here u, r, c and b stand for entry-wise, row-wise, columnwise and block-wise respectively. It is easy to see that all these Jacobians are extremely sparse and their evaluation is highly efficient. In this table, NA stands for not applicable as the Jacobian of the parametrization ψ 1 is necessarily entry-wise. 
experiments: 
 This section assesses baseline and pruned GCNs' performance in skeleton-based recognition using SBU Interaction (Yun et al., 2012) and the First Person Hand Action (FPHA) (Garcia-Hernando et al., 2018) datasets, comparing our lightweight GCNs against related pruning techniques. SBU is an interaction dataset acquired using the Microsoft Kinect sensor, it contains 282 moving skeleton sequences performed by two interacting individuals and belonging to 8 categories. Each pair of interacting individuals corresponds to two 15 joint skeletons, each one encoded with a sequence of its 3D coordinates across video frames. The evaluation protocol follows the train-test split as in the original dataset release (Yun et al., 2012) . The FPHA dataset includes 1175 skeletons belonging to 45 action categories performed by 6 different individuals in 3 scenarios. Action categories are highly variable, including various styles, speed, scale, and viewpoint. Each skeleton includes 21 hand joints, each one again encoded with a sequence of its 3D coordinates across video frames. The performances of different methods are evaluated using the 1:1 setting proposed in (Garcia-Hernando et al., 2018) with 600 action sequences for training and 575 for testing. The average accuracy over all classes of actions is reported in all experiments.Input graphs. Let's consider a sequence of skeletons {S t } t with S t = {p t 1 , . . . , pt n } being the 3D skeleton coordinates at frame t, and {p t j } t a joint trajectory through successive frames. We define an input graph G = (V, E) as a finite collection of trajectories, with each node v j ∈ V in G being a trajectory {p t j } t , and an edge (v j , v i ) ∈ E exists between two nodes if the underlying trajectories are spatially neighbors. Each trajectory is processed using temporal chunking, which splits the total duration of a sequence into M evenly-sized temporal chunks (with M = 4 in practice). Then, joint coordinates {p t j } t of the trajectory are assigned to these chunks, based on their time stamps. The averages of these chunks are concatenated in order to create the raw description of the trajectory (denoted as ϕ(v j ) ∈ R s with s = 3 × M ). This process preserves the temporal structureAccuracy (%) Raw Position (Yun et al., 2012) 49.7 Joint feature (Ji et al., 2014) 86.9 CHARM (Li et al., 2015) 86.9 H-RNN (Du et al., 2015) 80.4 ST-LSTM (Liu et al., 2016) 88.6 Co-occurrence-LSTM (Zhu et al., 2016b) 90.4 STA-LSTM (Song et al., 2017) 91.5 ST-LSTM + Trust Gate (Liu et al., 2016) 93.3 VA-LSTM (Zhang et al., 2017a) 97.6 GCA-LSTM (Liu et al., 2017a) 94.9 Riemannian manifold. traj (Kacem et al., 2018) 93.7 DeepGRU (Maghoumi & LaViola, 2019) 95.7 RHCN + ACSC + STUFE (Li et al., 2020) 98.7 Our baseline (unpruned) GCN 98.4Table 2 : Comparison of our baseline GCN against related work on the SBU database.Color Depth Pose Accuracy (%) 2-stream-color (Feichtenhofer et al., 2016) ✓ ✗ ✗ 61.56 2-stream-flow (Feichtenhofer et al., 2016) ✓ ✗ ✗ 69.91 2-stream-all (Feichtenhofer et al., 2016 (Hu et al., 2015) ✓ ✗ ✗ 66.78 JOULE-depth (Hu et al., 2015) ✗ ✓ ✗ 60.17 JOULE-pose (Hu et al., 2015) ✗ ✗ ✓ 74.60 JOULE-all (Hu et al., 2015) ✓ ✓ ✓ 78.78 Huang et al. (Huang & Van Gool, 2017 4 5 show a comparison and an ablation study of our method both on the SBU and the FPHA datasets. First, according to the observed results, when only the cross entropy loss is used without budget (i.e., λ = β = 0 in Eq. 6), performances are close to the initial heavy GCNs (particularly on FPHA), with less parameters2 as this produces a regularization effect similar to (Wan et al., 2013) . Then, when pruning is structured, the accuracy is relatively low but the speedup is important particularly for high pruning regimes. When pruning is unstructured, the accuracy reaches its highest value, but no actual speedup is observed as the architecture of the pruned networks remains unchanged (i.e., not compact). When pruning is semi-structured, we observe the best trade-off between accuracy and speedup; in other words, coarsely pruned parts of the network (related to entire block/column/row connections) lead to high speedup and efficient computation, whereas finely pruned parts (related to individual connections) lead to a better accuracy with a contained marginal impact on computation, so speedup is still globally observed with a significant amount. Extra comparison of our method against other regularizers shows a substantial gain. Indeed, our method is compared against different variational pruning with regularizers plugged in Eq. 6 (instead of our proposed budget and rank regularizers), namely ℓ 0 (Louizos et al., 2017) , ℓ 1 (Koneru & Vasudevan, 2019) , entropy (Wiedemann et al., 2019) and ℓ 2 -based cost (Lemaire et al., 2019) , all without our parametrization. From the observed results, the impact of our method is substantial for different settings and for equivalent pruning rate (namely 98%). Note that when alternative regularizers are used, multiple settings (trials) of the underlying mixing hyperparameters (in Eq. 6) are considered prior to reach the targeted pruning rate, and this makes the whole training and pruning process overwhelming. While cost-aware regularization makes training more affordable, its downside resides in the observed collapse of trained masks; this is a well known effect that affects performances at high pruning rates. Finally, Fig. 2 shows examples of obtained mask tensors taken from the second (attention) layer of the pruned GCN.For semi-structured pruning, we observe a compact tensor layer with some individually pruned connections whereas structured and unstructured pruning -when applied separately -either produce compact or spread tensors, with a negative impact on respectively accuracy or speed.In sum, semi-structured pruning gathers the advantages of both while discarding their inconveniences. 
conclusion: 
 This paper introduces a novel magnitude pruning approach that combines both the strengths of structured and unstructured pruning methods while discarding their drawbacks. The proposed method, dubbed as semi-structured, is based on a novel cascaded weight parametrization including bandstop, weight-sharing, and gating mechanisms. Our pruning method also relies on a budget loss that allows implementing fine-grained targeted pruning rates while also reducing the rank of the pruned tensors resulting in more efficient and still effective networks. Extensive experiments, conducted on the challenging task of skeleton-based recognition, corroborate all these findings. 
 
 Appendix: Pruning rate does not exceed 70% and no control on this rate is achievable when λ = 0. 


Decision: ## Strengths
* The paper is well written and organized
* The approach is novel and interesting
* The approach provides good performance results by combining the best of both worlds

## Weaknesess
* The paper would benefit from a wider range of experiments (accross larger classes of neural
  networks beyong GCN) to assess the generalizability of the approach
* More comparison with other approaches at the same problem like HAN are needed to assess the
  benefit compared the state-of-the-art solutions

The general sentiment about this paper is rather positive, I recommend acceptance as an oral presentation.