CALANet: Cheap All-Layer Aggregation for Human Activity Recognition

Authors: jaegyun  park
dae-won  kim
jaesung  lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:14:53

Abstract: With the steady growth of sensing technology and wearable devices, sensor-based human activity recognition has become essential in widespread applications, such as healthcare monitoring and fitness tracking, where accurate and real-time systems are required. 
To achieve real-time response, recent studies have focused on lightweight neural network models.
Specifically, they designed the network architectures by restricting the number of layers shallowly or connections of each layer.
However, these approaches suffer from limited accuracy because the classifier only uses the features at the last layer.
In this study, we propose a cheap all-layer aggregation network, CALANet, for accuracy improvement while maintaining the efficiency of existing real-time HAR models.
Specifically, CALANet allows the classifier to aggregate the features for all layers, resulting in a performance gain.
In addition, this work proves that the theoretical computation cost of CALANet is equivalent to that of conventional networks. 
Evaluated on seven publicly available datasets, CALANet outperformed existing methods, achieving state-of-the-art performance. 
The source codes of the CALANet are publicly available at https://github.com/jgpark92/CALANet.

Main paper: 
 introduction: 
 Human activity recognition (HAR) is a fundamental technique in healthcare [28, 53] , fitness tracking [5, 23] , and surveillance [21, 41] . Wearable sensor-based HAR has drawn attention in pervasive computing applications due to the popularity of smart wearable devices in recent years [7, 53] . Specifically, it aims to identify motion details of users or activity tracks from sensor signal patterns [10] . To this end, neural networks (NNs) have been widely used to achieve a superior learning performance without handcrafted feature engineering [7, 52] . Besides, with advances in microelectronics and inertial sensor-based wearable devices, recent researchers have focused on achieving real-time systems [4, 43] . Especially, a recent trend across most studies has become increasingly to train NNs on a resource-rich computing device and then deploy them to resource-limited wearable devices, where inference is executed [44, 60, 61] .Recent real-time HAR studies have focused on one-dimensional (1D) convolutional neural networks (CNNs) compatible with various hardware accelerators and deployment frameworks [14, 20, 23, 36, 37, 48, 51, 60] . CNNs include convolution and pooling operations, which allow them to extract more abstracted high-level features as input signals pass from early to later layers. Specifically, the pooling operation abstracts signals by reducing the feature size (temporal resolution), which can be regarded as a sampling of signal. As a result, the final classifier predicts an activity class based on the abstracted or sampled features at the last layer. The sampled features are more semantic and global Figure 1 : Analysis of representations in our experiments on KU-HAR dataset [47] . In a conventional CNN, the classifier predicts activities only using the feature representations at the last layer. Features at the early layer include the detailed information of original signals that may confound the classifier. In comparison, features at the later layer are more semantic, but the features (with more compact and short waveforms) make it challenging to classify activities that share similar semantics. Our goal is to design a CALANet that allows the classifier to use features for all layers while maintaining the inference time of conventional CNNs.than those of prior layers, but it has not been proven that the last layer is the optimal representation [58] . Although the high-level features sampled by the pooling operation can avoid over-fitting of the classifier [59] , for the HAR dataset, the loss of some detailed information makes it challenging to classify activities that share similar semantics, such as "Sit" and "Talk-Sit." Figure 1 illustrates intermediate features in the forward pass of conventional CNNs. Conventional CNNs classify "Jump" and "Sit" well but tend to misclassify "Talk-Sit" as another activity, as shown in Figure 1 . These experimental observations have also been reported in existing studies [24, 47] . Specifically, the features at the eighth layer (with more compact and short waveforms) make it easy for the classifier to discriminate "Jump" and "Sit," compared with the ones at the third layer. On the contrary, the features at the third layer (with more detailed information) can be more suitable than the ones at the eighth layer when classifying "Sit" and "Talk-Sit" that have similar vibrations in signal waveforms. Although which layer has the best features depends on the activity, conventional CNNs classify multiple activities only using the features at the last layer.The objective of real-time HAR is to maximize accuracy under real-time constraints. To achieve real-time response, prior HAR studies designed the network architectures of CNNs by restricting the number of layers shallowly [20, 23, 51, 60] or reducing connections of each layer [14, 36, 37, 48] . However, these approaches suffer from limited accuracy because their classifier only uses the features at the last layer. A straightforward approach to address this issue is to allow the classifier to use the features for all layers [19, 27] , but this leads to a substantial increase in computational cost, particularly as the number of layers deepens. Therefore, our goal is to design a novel network architecture that allows the network to aggregate the features for all layers while maintaining the computational cost of the conventional CNNs regardless of network depth, as shown in Figure 1 .In this paper, we propose a novel network, CALANet, with a cheap all-layer aggregation (CALA) structure. To achieve our goal, CALANet includes (1) learnable channel-wise transformation matrices and (2) scalable layer aggregation pool. First, we introduce new learnable channel-wise transformation matrices (LCTMs) to minimize an increase in computational cost due to all-layer aggregation. Given intermediate features at a specific layer (with temporal resolution T and the number of channels M ), M LCTMs generate a vector with N ≪ T elements based on linear transformation and combination without increasing the theoretical computation cost. Second, we improve the effectiveness of all-layer aggregation by introducing a scalable layer aggregation pool (SLAP) that allows CALANet to stack layers without significantly increasing computational costs. As a result, the main contributions of this paper are as follows:• We proposed CALANet with a CALA structure that allows the network to aggregate the features for all layers while maintaining the efficiency of CNNs regardless of network depth based on (1) LCTMs and (2) SLAP. • We theoretically proved that the computational cost of CALANet is equivalent to that of conventional CNNs, including even shallow networks. • We empirically demonstrated the effectiveness of CALANet in achieving superior performance compared to 11 state-of-the-art methods on seven public benchmark datasets. 
related work: 
 Many comprehensive surveys in HAR literature have highlighted the importance of NN-based models and real-time applications [7, 10, 34, 42-44, 52, 61] . For instance, early real-time HAR studies adopted two-dimensional (2D) convolutional NNs (CNNs) with shallow architectures [9, 25, 33, 45] . Specifically, they transformed the sensor signal patterns to 2D spectral images as an input of 2D CNNs. However, these approaches require complex preprocessing, such as discrete or short-time Fourier transform, which increases the overhead during continuous processing for real-time HAR. Meanwhile, Ignatov [23] proposed a one-dimensional (1D) CNN architecture using basic statistical features to encode global temporal information. Although this approach demonstrated the potential of 1D CNN, it still requires several data preprocessing like calculating the histogram of input signals.Recent real-time HAR studies focus on 1D CNNs without any complex data preprocessing. For example, Zebin et al. [60] proposed a CNN architecture comprising four convolution layers. Furthermore, they showed the efficiency of parameter quantization as post-processing for further optimization. In another study, Wan et al. [51] adopted a CNN architecture including three convolution layers. They demonstrated the superiority of CNNs compared with recurrent NN variants on HAR datasets with basic activities. To enhance shallow CNNs, Huang et al. [20] introduced a cross-channel communication that exchanges information among channels within the same layer. These models achieved real-time HAR by shallowly restricting the number of layers, resulting in limited accuracy.To alleviate the issue, some studies have considered efficient variants of convolution at each layer instead of reducing the number of layers. For example, Gao et al. [14] proposed a selective kernel module that divides the convolution into split, fuse, and select steps to adjust receptive field size adaptively. Similarly, Tang et al. [48] designed a hierarchical-split block to enhance multiscale temporal features by composing channel groups hierarchically. In another study, Teng et al. [49] proposed RepHAR, which re-parameterizes a pretrained multibranch CNN to a plain CNN before deploying it into resource-limited devices. However, these approaches still are insufficient to classify activities that have similar vibrations in signal waveforms because the classifier only uses the features at the last layer. Meanwhile, Park et al. [37] introduced a grouped temporal shift network that can flexibly re-design a network architecture to support various hardware specifications. Although this network can derive layer-specific structures suitable for a given computational budget, its performance is limited according to an initial network architecture. Therefore, its performance can be improved by using our CALANet as the initial network, as will be described in Section 4.2.Besides, the classifier requires both local and global temporal representations to achieve high HAR accuracy [64] . Specifically, the locality of CNNs improves accuracy due to their translational invariance concerning the precise location of activity within a segment of time-series data [17] .On the other hand, recurrent layers or attention mechanisms have an advantage for global feature extraction because they can model long-term dependencies. In this regard, many studies have attempted to integrate recurrent layers [8, 24, 35, 50, 56] or attention mechanisms [40, 63] into CNNs, which has increased both accuracy and inference times. The increase in inference time is primarily because of the lack of device-level optimizations compared with CNNs [15, 29, 60] . These accuracy-oriented networks will be compared with our CALANet in Section 4.2.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: This paper does not consider various computational budgets that may be changed according to the specific devices. This limitation must be improved via future works such as neural architecture search, runtime optimization, and parallel computation. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
cheap all-layer aggregation network: 
 In this section, our goal is to design a CNN architecture that can aggregate features for all layers into the final classifier without increasing the computational cost of CNNs. Furthermore, we prove that the theoretical computation cost of the proposed CALANet is equivalent to that of conventional CNNs, including even shallow networks.Our goal is to improve the HAR accuracy while maintaining the efficiency of CNNs. Therefore, before deriving a novel network structure and proving its theoretical efficiency, we formalize the computational cost of the conventional CNNs in a generalized form. Note that we only consider the computational cost in the feed-forward step, not the training step, which is unrelated to inference time. Let K (l) be the l-th layer of a network, where K (l) and K (l+1) are calculated sequentially and independently. Therefore, the theoretical computation cost of the CNN can be defined as a summation for each computation of layers, as described in Definition 1. Definition 1. Let α(•) be the computational cost of calculating the output of each layer. Given a network architecture A with L layers, the input X (l) is fed into the l-th layer with trainable parameters θ (l) to calculate the output X (l+1) . Due to this layer composition, its computational cost is formalized asEQUATIONTo formalize the computation cost of CNNs, we borrow the concept of time complexity as the upper bound of the computational cost. Similar to Proposition 3 of [37] , Eq. ( 1) is simplified by Proposition 1. Proposition 1. The time complexity of CNNs is formalized as:EQUATION)where M and T are the number of channels and temporal resolution for input data, and N and D k are average numbers for output channels and kernel sizes across a network, respectively.The proof is given in Appendix A. Because we do not restrict the number of layers shallowly, we will assume that the condition of Eq. ( 2) is always true in this paper.In this section, we introduce a learnable channel-wise transformation matrix (LCTM) that allows our CALANet to aggregate features for all layers without increasing the theoretical computation cost. Figure 2 shows the network architecture of CALANet. Given input signals, the convolution and pooling layers extract the high-level sampled features by calculating temporal correlations and reducing the feature resolution. After that, the features for all layers are connected with the classifier via the cheap all-layer aggregation module based on the LCTMs.Let x m be a feature vector of m-th channel with temporal resolution T = |x m | at any layer. Because T varies with the layer, we define a mapping function f : x m → y ∈ R N , where a constant value N ≪ T . After that, we calculate f via a transformation matrix A ∈ R N,T . As the mapping function is calculated for each channel of features, m-th feature vector is transformed to y m ∈ R N as follows:EQUATIONwhere this transformation can be interpreted as a compression of global temporal information. After that, a linear combination is conducted to calculate relations between channels as follows: where ŷ ∈ R N is a vector and a m is m-th coefficient of the linear combination.EQUATIONWe replace a m A m with B m ∈ R N,T . As a result, Eq. ( 4) is simplified as follows:EQUATIONwhere elements of B m can directly be optimized by stochastic gradient descent because these matrix multiplications can be implemented by dense layers. Therefore, we define B m as LCTM. Herein, the time complexity of calculating M LCTM operations is O(T M N ). Finally, ŷ for all layers are concatenated and then fed into the classifier.To investigate whether our CALANet can maintain the efficiency of conventional CNNs, we analyze the time complexity of CALANet. Based on Proposition 1, the time complexity of CALANet is formalized in Lemma 1. Lemma 1. The time complexity of CALANet is equivalent to:EQUATION)The proof is given in Appendix B. According to Proposition 1 and Lemma 2, our CALANet can aggregate features for all layers while maintaining the efficiency of conventional CNNs.The effectiveness of all-layer aggregation depends on a layer aggregation pool, i.e., the number of layers, as shown in Table 3 . To improve the accuracy of CALANet further, we also introduce a scalable layer aggregation pool (SLAP) that allows CALANet to stack layers without significantly increasing computational cost. To this end, we, in this section, aim to omit L in Eq. (2) . Inspired by ShuffleNet [62] , we first use the grouped convolution and channel shuffle to reduce the time complexity of the standard convolutions. Precisely, the M input channels are evenly divided into G channel groups. After that, the standard convolution generates ⌊N/G⌋ output channels for each channel group. Subsequently, the channel shuffle operation is executed.The entire channels are fully related by the channel shuffle operations if and only if (the number of layers) × (the number of channels within each channel group) ≥ (the number of channel groups) [62] . Therefore, G is set into a value satisfying L × N ≥ G 2 . As N ≥ G and G are inversely proportional to the computational cost, we set G into L without any loss of information for channel correlations. Consequently, the time complexity of the stack of the standard convolutions is reduced in Lemma 2. Lemma 2. The time complexity of calculating the standard convolutions is reduced to:EQUATION)The proof is given in Appendix C. To reduce the time complexity of all-layer aggregation, we focus on the norm of vectors extracted from LCTMs, i.e., |ŷ| ≈ N from Eq. ( 5). The LCTMs for all layers generate a vector with L × N elements fed into the Softmax layer to classify the activities. The large number of units in the Softmax layer may incur overfitting [57] . Therefore, we fix the number of features fed into the Softmax layer to N by dividing the number of rows of LCTM in Eq. ( 5) by L, resulting in B m ∈ R N/L,M . Consequently, the time complexity of all-layer aggregation is reduced in Lemma 3. Lemma 3. The time complexity of calculating all-layer aggregation is formalized as:EQUATION)The proof is given in Appendix D. Finally, we introduce CALANet with the SLAP by omitting the factor L from its time complexity. Consistent with Lemma 2 and Lemma 3, the time complexity of CALANet is reduced in Theorem 1.Theorem 1. The time complexity of CALANet is reduced to:EQUATIONThe proof is given in Appendix E. From Theorem 1, we crosscheck the efficiency of our CALANet by making comparisons with the time complexity of shallow CNNs in Corollary 1 and Corollary 2. Corollary 1. The time complexity of CALANet is equivalent to the shallow CNNs with L ≥ 2.L = 2 is O(TD k MN) + O(TD k N 2 ) = O(TD k N 2 ).It is equivalent to Eq. ( 9).L = 1 if M ≈ N.Proof. The time complexity of shallow CNNs withL = 1 is O(TD k MN). If M ≈ N, then it is equivalent to Eq. (9).In conclusion, our CALANet has a computation cost comparable to shallow CNNs. Especially from Corollary 2, the time complexity of CALANet becomes equivalent to the shallow CNNs even with L = 1 as the number of sensors increases. 
experiments: 
 In this section, we evaluate the superiority of CALANet. In Section 4.1, we describe the experimental setup. Section 4.2 presents the compared results of CALANet and other networks on seven HAR datasets. Section 4.3 provides an in-depth analysis via an ablation study. Lastly, Section 4.4 measures the actual inference time of CALANet.Dataset. We used seven public benchmark datasets, including various sampling frequencies, the number of activities, and sensors. They include UCI-HAR [1] , UniMiB-SHAR [30] , DSADS [3] , OPPORTUNITY [6] , KU-HAR [47] , PAMAP2 [46] , and REALDISP [2] . The details for each dataset are described in Appendix F. Baseline. We compared CALANet with 11 baseline networks. To evaluate the efficiency of CALANet, we used Shallow ConvNet [23] , RepHAR [49] , and Res-GTSNet [37] as state-of-the-art models in real-time HAR. Meanwhile, we adopted four CNNs with recurrent layers or attention mechanisms, including DeepConvLSTM [35] , Bi-GRU-I [50] , RevAttNet [40] , and IF-ConvTransformer [63] , to verify the effectiveness of our all-layer aggregation. In addition, we used four networks, T-ResNet [54, 12] , T-FCN [54, 12] , MILLET [11] , and DSN [55] that achieved substantial success in the time-series classification (TSC), which is more general-purpose than HAR. The details for the models and hyperparameters are described in Appendix G.To evaluate the performance of CALANet, we used two metrics: F1-score and floating-point operations (FLOPs). Because the HAR datasets inherently involve a class imbalance, the F1-score has been commonly used as an alternative for accuracy. In particular, FLOPs have been widely used to describe how many operations a given model requires to run a single pattern. In addition, we investigate the change in performance according to L, as will be described in Section 4.3. Meanwhile, Res-GTSNet [37] can derive layer-specific structures suitable for a given computational budget, and the original paper adopted T-ResNet as an initial network. To improve the efficiency of CALANet further, we also designed the CALA-GTSNet by replacing T-ResNet with our CALANet.Table 1 shows the results of comparing CALANet and the baseline networks. The experiments ran ten times, and the average values were recorded on all the datasets. In addition, we performed a paired t-test at the 95% significance level on each dataset. In Table 1 , ▼/△ indicates that the compared network was significantly worse/better than CALALet regarding the F1-score.Comparison with real-time CNNs. In Table 1 , the F1-scores of CALANet were statistically superior to real-time HAR models on all datasets. In particular, CALANet has the lowest FLOPs compared to other real-time HAR models with standard convolution layers on seven datasets. Meanwhile, Res-GTSNet, with an efficient variant of the convolution, exhibited significantly low FLOPs. This variant can be easily integrated with our CALANet to reduce its FLOPs further. As shown in Table 1 , CALA-GTSNet outperformed Res-GTSNet on 86% of the datasets. Also, CALA-GTSNet has lower FLOPs than Res-GTSNet on 71% of the datasets. As a result, CALANet and GTSNet can complement each other to improve the accuracy or reduce computations. These results demonstrated that our cheap all-layer aggregation can maintain a low computational cost.Comparison with accuracy-oriented networks. We noted that real-time or efficient HAR models using wearable sensors process the input signals with short segmentation lengths for rapid response. If CNNs are sufficient to extract meaningful information from the short-term signals, unnecessary increases in inference time due to integration with recurrent layers or attention mechanisms can be avoided. In Table 1 , CALANet outperformed two CNNs with recurrent layers, i.e., DeepConvLSTM and Bi-GRU-I, on all datasets. Compared with RevAttNet and IF-ConvTransformer, CALANet exhibited a comparable F1-score despite its significantly low FLOPs. These results indicate that CNNs are sufficient to model the temporal information for the real-time HAR dataset. Compared with TSC models, CALANet showed comparable performance despite its significantly low FLOPs. These results demonstrated that our cheap all-layer aggregation can significantly improve HAR accuracy while maintaining low FLOPs.The breakdown effect of CALANet. We conducted an ablation study to investigate the effectiveness and efficiency of our CALANet. The key components of CALANet are LCTMs and SLAP. Therefore, we compared the performance of our CALANet with that of its two variants, which were obtained by removing each component. The first variant removes the SLAP described in Section 3.3. The second variant replaces the LCTMs with fully-connected layers that have the same number of units as the input size. Table 2 shows that our LCTMs substantially reduced the FLOPs for calculating all-layer aggregation without losing the F1-score. In addition, the SLAP enhanced the effectiveness of all-layer aggregation even while reducing FLOPs. Especially, Figure 3 shows the tradeoff between the FLOPs and F1-score with varying numbers of layers in CALANet with/without LCTMs and SLAP. The tradeoff curves closer to the top-left are more efficient, with a higher F1-score per FLOPs. As shown in Figure 3 , CALANet with LCTMs and SLAP achieved a higher F1-score in similar computational cost than one without LCTMs and SLAP. Effect of scalable layer aggregation pool.We investigated the layer aggregation pool at which the best F1-score of CALANet is achieved on seven datasets. Table 3 shows the change in F1-score of CALANet as the layer aggregation pool L increased; herein, the best F1-score is indicated by the bold font on each dataset. As shown in Table 3, the layer aggregation pool and F1score tend to be proportional. In Figure 4 , CALANet with SLAP (red line) exhibited a negligible increase in FLOPs compared with CALANet only with LCTMs. As a result, SLAP allows CALANet to stack layers without significantly increasing FLOPs.Performance analysis on similar activities. To verify the performance of CALANet, we investigated the confusion matrices (see Appendix H). Prior works [47, 24] suffered from activities that have similar vibrations in signal waveforms, such as "Sit" and "Talk-Sit," as described in Section 1.Compared with these works, our CALANet significantly improved the accuracy of those activities on the KU-HAR dataset. For other examples, these activities include ("rope jumping" and "waking") [20, 14, 48, 49] and ("knees bending crouching" and "reach heels backwards") [8] . On the other hand, our CALANet correctly classified "rope jumping" as "waking" compared to RepHAR that misclassified "rope jumping" as "walking" 20 times [49] on the PAMAP dataset. Compared to MG-WHAR [8] misclassified "knees bending crouching" by approximately 20% as "reach heels backwards", our CALANet misclassified "knees bending crouching" as "reach heels backwards" only two times on the REALDISP dataset.Applicability of CALA structure. Our CALA structure can effectively be applied to existing CNNs if the following constraints are satisfied: (1) the layers of a network architecture should be calculated sequentially and independently; (2) the output of each layer should be able to be expressed as a (temporal length × channel size) matrix. To the best of our knowledge, most wearable sensor-based human activity recognition models can satisfy the above constraints. In Table 4 , we applied our LCTMs and SLAP to SqueezeNet [22] . Specifically, the output of a squeeze convolution layer in each fire module is fed into LCTMs and connected to the last layer. As a result, our modification significantly improved the F1-score of SqueezeNet on 71% of all datasets while maintaining its FLOPs. In addition, we applied CALANet to the ECG heartbeat classification problem using the MIT-BIH arrhythmia dataset [16] . CALANet exhibited comparable performance with other networks To estimate the actual response time of our CALANet, we used the AMD Ryzen 7 5800X 8-Core Processor without the support of graphics processing units. Particularly, we compared the inference time of CALANet with Shallow ConvNet. Similar to the conventional real-time HAR studies, the measurements were repeated 1,000 times, and the minimum, maximum, and mean values were recorded. Table 5 shows the inference time of the two networks, where the window length was set to 300 (3 s) to slide one instance at a time. CALANet exhibited a response time similar to Shallow ConvNet, even though its depth is nine times deeper than that of Shallow ConvNet. Consequently, these measurements show that our model is sufficient to meet the real-time requirements. 
conclusion: 
 In this article, we proposed an effective neural network called CALANet for real-time HAR from wearable sensors. In particular, our CALANet has an all-layer aggregation structure that can aggregate features for all layers based on the learnable channel-wise transformation matrix and scalable layer aggregation pool. As a result, CALANet improved HAR accuracy while maintaining the efficiency of existing real-time HAR models. In addition, we proved that the computational cost of CALANet is equivalent to that of shallow CNNs. Our experiments demonstrated that CALANet could achieve state-of-the-art performance on the HAR datasets under low latency.Future studies can be conducted to overcome the limitations of the proposed method. CALANet does not consider the various computational budgets that can be changed according to the specific devices and the runtime optimizations of actual devices, such as memory access costs and parallel computations. For example, future studies may further improve CALANet by introducing a new operator designed to match the target device.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We will release our code by including the GitHub link in the paper if finishing the blind review process. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We will include the usage guidelines or restrictions in code release.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited the original paper that produced the datasets and our source code release follows GPL-3.0 license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Given the original signals X (0) ∈ R T,M , the first convolution layer generates new feature signals X (1) ∈ R T,N 1 , resulting in a time complexity of O(TMN (1) Dk ). Suppose that the l-th intermediate convolution layer includes a kernel K (l) ∈ R D (l) k ,N (l-1) ,N (l) and N (l) is a positive-integer multiple of N (l-1) , that is, N (l) = c (l) N (l-1) . Generally, c ≥ 1 and the pooling layer adjusts the temporal resolution T to T /c if c ≥ 2. Therefore, its time complexity isO(T (l) D (l) k M (l) N (l) )) = O(T (l-1) D (l)k (N (l-1) ) 2 ). Given the stack of L convolution layers, their time complexity can be simplified as follows:EQUATIONwhere N and D k are the average of the number of output features and the kernel sizes across the layers, respectively. If M ≤ N(L -1), the time complexity of convolution layers becomes O(TD k N 2 L).Consistent with Proposition 1, the time complexity of calculating the features is O(TD k N 2 L). Since the number of output channels across the standard convolution layers is N, the time complexity of calculating LCTMs for all layers is O(TN 2 L). Given L × N aggregated features and V activities, the Softmax layer predicts an activity with the time complexity of O(LNV ). Consequently, the time complexity of our CALANet is formalized as:EQUATIONBecause V commonly is less than T × D k × N, Eq. ( 11) can be rewritten as O(TD k N 2 L).Given an input X ∈ R T,M , M channels are evenly divided into L channel groups. More precisely, for each channel group, the convolution layer generates ⌊N/L⌋ output channels, resulting in the time complexity of O(T D k (M N/L 2 )). Since each convolution layer generates the output Y ∈ R T,N across L channel groups, the time complexity of each layer is O(T D k (M N/L)). Because our CALANet includes L standard convolution layers to calculate the local temporal correlations, part 2 of Eq. ( 11) is reduced to O(TD k N 2 ), consistent with Proposition 1.Given an input X ∈ R T,M , LCTMs at each layer generate a vector with ⌊N/L⌋ elements, resulting in the time complexity of O(T M (N/L)). As shown in Figure 2 , the number of output channels for each layer is fixed as N across layers. Therefore, the time complexity of the all-layer aggregation is O(T (N 2 /L)). Similar to Lemma 2, part 3 of Eq. ( 11) is reduced to O(TN 2 ).Consistent with Lemma 2 and Lemma 3, part 2 and part 3 of Eq. ( 11) is simplified as:EQUATIONIn addition, we fixed the number of features fed into the Softmax layer to N. Therefore, part 4 of Eq. ( 11) is reduced to O(NV ). Because V commonly is less than T × D k × N, Eq. ( 6) is reduced O(TD k N 2 ).The benchmark datasets used in our experiment follow the following setup:• The UCI-HAR dataset [1] was recorded at a sampling frequency of 50 Hz. Precisely, 30 subjects performed six basic activities (e.g., walking, upstairs, sitting) using accelerometers and gyroscopes embedded in Android smartphones. As the authors recommended, each segment's length is set to 128, and 70% and 30% of the dataset were used as the training and test sets, respectively.• The UniMiB-SHAR dataset [30] was recorded at a sampling frequency of 50 Hz, where the length of each segment is 151. The 30 subjects performed 17 activities, including nine activities of daily living (e.g., walking and standing) and eight fall activities (e.g., forward and syncope), using an accelerometer in Android smartphones. Precisely, 70% and 30% of the dataset were used as the training and test sets, respectively.• The DSADS dataset [3] was recorded at a sampling frequency of 25 Hz, where the length of each segment is 125. Eight subjects performed 19 daily and sports activities (e.g., exercising on a stepper and rowing) using accelerometers, gyroscopes, and magnetometers embedded in five MTx trackers. More precisely, the MTx units measured the sensor signals on the torso, right arm, left arm, right leg, and left leg. We split the dataset into 80% training and 20% test sets based on the subject's identification (ID).• The OPPORTUNITY dataset [6] was recorded at a sampling frequency of 30 Hz in a sensor-rich environment with wearable, object, and ambient sensors, where the length of each segment is 90. We only considered wearable sensors for real-time HAR, including accelerometers and inertial measurement units (IMUs); the number of input channels is 113 in our experiments. The four subjects performed 17 activities, including complicated activities such as "drink from cup" and "open door," categorized into ADL 1-5 and Drill. We used the ADL 5 data for subject 1; the Drill data for subject 2; the ADL 1 and 4 data for subject 3; and the ADL 4 data for subject 4 as the test set [32] . The remaining data were used as the training set.• The KU-HAR dataset [47] was recorded at a sampling frequency of 100 Hz, where the length of each segment is 300. Precisely, 90 subjects performed 18 daily activities (e.g., talking with hand movements and picking up an abject) using accelerometers and gyroscopes embedded in smartphones. The 80% and 20% of the dataset were used as the training and test sets, respectively.• The PAMAP2 dataset [46] was recorded at 100 and 9 Hz sampling frequencies for IMUs and a heart rate monitor, respectively. Precisely, nine subjects performed 18 activities, including basic activities (e.g., sitting and running) and complicated activities (e.g., watching TV and folding laundry), where the length of each segment is 512. We used the data aggregated from subjects 102 and 106 as the test set. In addition, we added 30% of the data for "watching TV", "car driving," and "playing soccer" into the test set because they were performed by only one subject. The remaining data were used as the training set.• The REALDISP dataset [2] was recorded at a sampling frequency of 50 Hz from nine IMUs, where the length of each segment is 250. More precisely, 17 subjects performed 33 fitness activities (e.g., lateral bend arm up and upper trunk and lower body opposite twist) using accelerometers, gyroscopes, and magnetometers. In addition, the authors provided orientation estimates in quaternion format. We split the dataset into 70% training and 30% test sets based on subject ID.We summarize the models used in our experiments, as follows:• CNNs for real-time HAR. We adopted three CNNs as state-of-the-art models in realtime HAR. Shallow ConvNet [23] comprises a single convolution layer and two fullyconnected layers, where the basic statistical features to encode global temporal information are concatenated with outputs of the convolution layer. RepHAR [49] comprises three convolution layers with re-parameterization and a Softmax layer. Res-GTSNet [37] contains nine grouped temporal shift module layers and a Softmax layer.• CNNs with recurrent layers or attention mechanisms. We adopted two CNNs with recurrent layers and two CNNs with attention mechanisms to verify the effectiveness of our all-layer aggregation. DeepConvLSTM [35] contains four convolution layers and two recurrent LSTM layers with a Softmax classifier. Bi-GRU-I [50] is composed of two bi-directional GRU layers, three inception layers, and a Softmax layer. Compared with DeepConvLSTM, its architecture has a reverse order of convolution and recurrent layers. RevAttNet [40] contains six convolutional layers, two recurrent layers, and two reverse attention modules, including LSTM, deconvolution, and multi-head attention. IF-ConvTransformer [63] is composed of IMU fusion blocks, four convolutional layers, and two self-attention layers, where the IMU fusion blocks are used to fuse the features from multiple sensor modalities based on sensor-wise convolutional layers.• Time-Series Classification (TSC) models. In addition, we adopted two CNNs that achieved substantial success in the TSC, which is more general-purpose than HAR. Specifically, T-ResNet [54, 12] consists of three residual blocks, each comprising three convolution layers with a residual connection, and a Softmax layer. T-FCN [54, 12] consists of three convolution layers and a Softmax layer. Compared with the real-time models, it has more output channels. MILLET [11] comprises five identical networks with conjunctive pooling layers, where each network contains six convolutional layers with multiple kernel sizes. DSN [55] contains three sparse CNN module, each of which includes a dynamic sparse convolution and point-wise convolution.For fairness, we re-implemented all the baseline networks in PyTorch [38] and excluded sophisticated tricks of each original setting, such as a gradient clipping [31] and learning rate warmup [18] . Precisely, they were trained for 300 epochs with a batch size of 128 using a 2080Ti graphicsprocessing unit. We used the Adam optimizer [26] with β 1 = 0.9, β 2 = 0.999 and ϵ = 10 -8 , where the learning rate and weight decay were set to 0.0005. For the CALALet, we set D k , N, and L to 5, 128, and 9, respectively. On the OPPORTUNITY and REALDISP datasets with many input channels (113 and 117), we doubled the number of filters for the first convolution of CALANet.The hyper-parameters of the existing networks are set according to the values recommended in the original papers.Figures 5 6 7 show the confusion matrices of CALANet on three datasets. Their diagonal elements represent the number of instances correctly classified to the related activities, with the color darkening as the number grows.A 1 A 2 A 3 A 4 A 5 A 6 A 7 A 8 A 9 A 1 0 A 1 1 A 1 2 A 1 3 A 1 4 A 1 5 A 1 6 A 1 7 A 1 8 Predicted A 1 A 2 A 3 A 4 A 5 A 6 A 7 A 8 A 9 A 1 0 A 1 1 A 1 2 A 1 3 A 1 4 A 1 5 A 1 6 A 1 7 A 1 8 A 1 A 2 A 3 A 4 A 5 A 6 A 7 A 8 A 9 A 1 0 A 1 1 A 1 2 A 1 3 A 1 4 A 1 5 A 1 6 A 1 7 A 1 8 A 1 9 A 2 0 A 2 1 A 2 2 A 2 3 A 2 4 A 2 5 A 2 6 A 2 7 A 2 8 A 2 9 A 3 0 A 3 1 A 3 2 A 3 3 Predicted A 1 A 2 A 3 A 4 A 5 A 6 A 7 A 8 A 9 A 1 0 A 1 1 A 1 2 A 1 3 A 1 4 A 1 5 A 1 6 A 1 7 A 1 8 A 1 9 A 2 0 A 2 1 A 2 2 A 2 3 A 2 4 A 2 5 A 2 6 A 2 7 A 2 8 A 2 9 A 3 0 A 3 1 A 3 2 A 3 3Actual 244 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 162 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 204 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 55 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 54 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 78 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 129 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 108 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 107 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 88 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 68 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 53 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 47 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 53 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 49 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 57 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 69 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 3 27 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 5 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 113 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 65 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 174 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 226 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 175 0 [39] 98.5 Ganguly et al. [13] 97.3We applied CALANet to the ECG heartbeat classification problem using the MIT-BIH arrhythmia dataset [16] , which includes 24-hour ambulatory ECG recordings collected from inpatients and outpatients at Boston's Beth Israel Hospital. The dataset has 21,892 heartbeats, each with a signal length of 187. In Table 6 , CALANet exhibited comparable performance with other networks designed to process ECG signals. This result shows that CALANet has promising applicability to other ML applications.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: To achieve real-time HAR, prior HAR studies designed the network architectures of CNNs by restricting the number of layers shallowly or reducing connections of each layer, resulting in the significant loss of accuracy. The proposed method addressed this problem by improving the accuracy based on cheap all-layer aggregation while maintaining the computational cost of the existing real-time HAR networks. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper considers human activity recognition (HAR) from wearable IMU sensor measurements which has been very active research area 
more than a decade. Paper is focusing on improving deep learning -based HAR by proposing convolutional neural network utilizing novel 
architecture to improve the recognition accuracy and decreasing the computational complexity and size of the network for real-time small resource (on the edge/mobile devices) inference. Proposed approach is based on all-layer feature aggregation with learnable channel weight matrices and fast fusion. Proposed methodologies are empirical evaluated against several related methods on publicly available HAR benchmark datasets, showing improvement of recognition accuracy meanwhile reducing the number of computational operations.

Although in borderline, this paper has received positive feedback, and based on scores most of the reviewers are towards accepting the paper. The strengths of the paper are related to motivated and interesting topic with clear research goal (Reviewers QGq8, 1Z8p, and  2n6z), good empirical evaluation (Reviewers 1Z8p and 2n6z) with proper theoretical analysis of background (Reviewers VmYr and 2n6z), and promising results against other models on benchmark datasets. Weaknesses/limitations are related to focusing only on narrow area of HAR (Reviewers QGqB and VmYr), some limitations on evaluation and ablation study (Reviewers VmYr and 2n6z), and not well-justified usage of CNN architecture (e.g., compared to RNN/transformers, pointed by Reviewer 2n6z). 

Authors have provided response to critics in rebuttal by showing some additional experimental result, including: 1) additional ECG-classification task with comparable accuracy against baselines (in discussion comments with Reviewer QGqB), 2) additional ablation study showing usefulness of LCTM and SLAP (in global rebuttal file, Fig 1/ Table 1), and 3) additional experiments against transformer based architectures (in discussion comment with Reviewer 2n6z). Although somewhat narrow application area where the method is proposed to, it has few important merits (described above) and brings interesting knowledge to NeurIPS community (in addition to Ubicomp/wearable computing domain). To keep conference diverse, I am recommending the paper to be accepted as a poster. Authors are encouraged to revise the final paper, including all the improvements and suggestion raised during the rebuttal discussion.

Reviews: review: paper_summary: paper_summary: The problem of Human Activity Recognition (HAR) is considered in this paper where the border between different activities can differ depending on the type of activity. For example, one activity can be “just sitting”, and another can be “sitting while speaking”. To this end, the authors argue that we need to leverage the features extracted in all the layers of a neural net. Thus, the authors propose a modification to ConvNet models where (called “all-layer aggregation”) is added to the model which takes its input from all the previous conv layers. To do so, the authors design learnable channel-wise transformation matrices that can be added to the model and provide fast aggregation. Evaluation results on seven HAR dataset shows that the proposed modification can improve the classification accuracy of HAR, compared to other alternatives.
strength_weakness: strength_weakness: The weakness of this paper is its relevance to the NeurIPS. This work gets more attention and appreciated by people in the Mobile or UbiComp community as the novelty and contribution is not much in the ML part but in systemic modification of the architecture.
questions: questions: 1. It seems that such a modification is only applicable to ConvNets. It is not clear how similar things can be done to other architectures.
 
2. I could not understand whether this CALANet is only useful to HAR or if it can also help with other data types like audio data or bio signals. 

3. It would be interesting and useful if the authors could show how this modification can be applied to some benchmark architectures that are built for mobile or wearable devices such as MobileNet or EfficientNet, and similar models.
limitations: limitations: The main limitation is that the solution is very specific to HAR datasets and it is only applied to basic ConvNets and not other benchmark models.
paper_summary: paper_summary: This paper proposes an all-layer aggregation network, CALANet, to improve model accuracy while maintaining the efficiency of lightweight models. Specifically, the authors have exploited the features from all layers for classification, as a kind of aggregation.
strength_weakness: strength_weakness: 1. My major concern is about the aggregation. When I first read the title of this paper, I assumed that the authors have trained a neural network and then compress its multiple layers via aggregation. However, it seems that the authors just add several components (LCTMS, SLAP) to a complete CNN. It looks like an advanced version of residual networks with channel weights.
2. The authors claimed that they wanted to improve model performance while keeping model efficiency. However, in Table, it seems that, in some cases (KU-HAR, PAMAP2), the proposed method did not have obvious F1 improvements but resulted in highly increased FLOPs.
questions: questions: 1. How to prove that the performance improvement is because of the aggregation, but not because of the more complicated network architecture? To be more specific, what if we do not use the proposed LCTMs and SLAP but directly increase the layers of CNNs? It would be convincing if the authors provide results with varying numbers of layers with/without LCTMs and SLAP.
2. The authors have used FLOPs to measure model efficiency, which may be inadequate, because the best hyperparameters for different models are different. If the authors directly use the same hyperparameters for all models, we may see a fair comparison in terms of efficiency, but it is not fair to compare their accuracy/F1. Can the authors provide the results in terms of training/testing time for intuitive efficiency comparison (in Table 1)?
limitations: limitations: 1. This is not a model compression work but just proposed some aggregation modules. The authors should discuss the difference between their work and model compression works.
2. This work mainly focuses on CNNs. The authors can further discuss other network architectures.
paper_summary: paper_summary: The CALANet describes a technique to aggregate the features from all the neural network layers for human activity recognition (HAR).
Because HAR is a common application for wearable devices, the model needs to be lightweight to be deployed on the edge for example on an Apple Watch.
Existing studies are often limited by shallow networks. The activity prediction is done on the final layer without using the features from previous layers, 
which may contain key information.  Working with computational constraints, CALANet proposes to aggregate features from all the layers to improve the model performance.

The CALANet consists of two modifications: (1) a channel-wide transformation matrix to condense features from each layer (2) A ShuffleNet-like convolution to
aggregate all the layer features with computational efficiency. The authors also presented proof results on why CALANet is within the same computational complexity as a shallow network. The empirical results on 7 benchmarks against the state-of-the-art models supported the authors' claim.
strength_weakness: strength_weakness: 1. The notations of the proofs can be better clarified so that the readers don't need to refer to the appendix. For example, D is not explained in eq. 2 during its first occurrence.
2. This is a limitation of the field of HAR in general. Existing benchmarks are small usually with the number of participants under 100. It is fairly easy to overfit on the test set. I can see that you are just doing a simple train/test split. I know that some of the older benchmark recommends this but would be interesting to see cross-fold validation results. It is probably ok if you don't have the time to do this. Furthermore, could you clarify how you selected the hyper-parameters for all the models? Again, I suspect that the results reported could overfit your current test set given your evaluation framework. 
3. In your ablation study you are trying to show the effectiveness of your layer aggregation technique (Table 2), there are several changing variables including network depths and FLOPs in addition to the network tricks you introduced. I don't think we can conclude that the layer aggregation trick worked. To test this properly, we should do the ablation studies using the same network L and probably similar FLOPs across different baselines.
questions: questions: 1. L60: what do you mean by temporal resolution T? 
2. The majority if not all of the benchmark datasets you used are lab-based, which is really not a realistic assessment. Could consider adding one of the free-living datasets to the baselines if time allows. But you don't have to do this. 
    1. Logacjov, Aleksej, et al. "HARTH: a human activity recognition dataset for machine learning." Sensors 21.23 (2021): 7853.
    2. Chan, Shing, et al. "CAPTURE-24: A large dataset of wrist-worn activity tracker data collected in the wild for human activity recognition." arXiv preprint arXiv:2402.19229 (2024).
limitations: limitations: NA
paper_summary: paper_summary: This paper designs CALANet, a cheap all-layer aggregation network designed for real-time sensor-based  HAR. The main objective of CALANet is to improve the accuracy of HAR while maintaining a low computational costs on edge devices for real-time applications. The authors argue that existing CNN models for HAR often suffer from limited accuracy because they only use features from the last layer of the network. In contrast, CALANet allows the classifier to aggregate features from all layers. The authors theoretically prove that the computational cost of CALANet is equivalent to that of conventional CNNs. The authors utilize 7 datasets to demonstrate the effectiveness of CALANet. The results show that CALANet outperforms seven state-of-the-art methods, achieving superior performance on 7 datasets.
strength_weakness: strength_weakness: -  One of my concerns is the baselines. Why choose these models as baselines? I believe there are many more advanced models in sensor-based HAR. I suggest the authors go through some papers in AAAI, IMWUT, Sensys, Mobicom, where most HAR papers are published. Otherwise, current improvement might not justify the advantage of CALANet. Also, there are more advanced time series classification models from NIPS, ICML, etc. 
-  The goal of this study is improving the accuracy of HAR with the computational costs during the inference phase as the constraint. Therefore, the fundamental thing is the accuracy. However, why CNN become the choice for this study? While authors mention that CNN is popular for HAR recently in Line 27, this argument does not support the study of accelerating CNN for HAR in this paper. 
-  Also curious about the dataset partition, some are 7:3 and others are 8:2. 
-  More references added in the content will improve the manuscript further, e.g., from line 51 to 55, references can be added to support the argument of the “” straightforward approach” and the “”computational costs”. 
-  In related work section, it seems the discussion of accelerating CNN or neural network is missing, which is a popular topic and has been widely studied. 
-  Writing: Writing can be further improved. For example, in Table 1, authors can put their own models at the same location (e.g., at the bottom). I believe this will lower the cognitive load of readers. In line 47, “under real-time response”. 
-  I suggest the authors to re-design Fig 2 for better readability. Fig 2 takes a lot of spaces but it is challenging to understand why the proposed model can utilize all features with low computational costs.
questions: questions: Please see above.
limitations: limitations: Please see above.

Comments: comment: I agree with the authors that the work presented is of relevance to the NeurIPS community. This is well-justified by the relevant work that you've have cited.

I would like to thank the authors for your timely and clear responses to my comments. 

My rating stands as 6 because the proposed work is specific to bio-signals instead of time series modelling in general.

Good luck :D
Thank you for your recommendation and encouragement.

We applied CALANet to the ECG heartbeat classification problem using the MIT-BIH arrhythmia dataset [1], which includes 24-hour ambulatory ECG recordings collected from inpatients and outpatients at Boston's Beth Israel Hospital. The dataset has 21,892 heartbeats, each with a signal length of 187. In the following Table, CALANet exhibited comparable performance with other networks designed to process ECG signals. This result shows that CALANet has promising applicability to other ML applications.

||CALANet|Pham et al. [2] |Ganguly et al. [3]|
|:---|:---:|:---:|:---:|
|Average Accuracy|98.2|98.5|97.3    |

In addition, we elaborate on how CALANet can be applied to other architectures.
  1. Define the smallest unit of a set of adjacent operators repeated across the network architecture as the "layer," such as the fire module of SqueezeNet and the residual module of ResNet.
  2. Check if the output of each layer can be expressed as a (temporal length $\times$ channel size) matrix.
  3. Multiply the output matrix and LCTM for each layer, and concatenate the output vectors across layers.
  4. Feed the concatenated features into the classifier to predict activity. 

> [1] Goldberger, Ary L., et al. "PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals." circulation 101.23 (2000): e215-e220.

> [2] Pham, Bach-Tung, et al. "Electrocardiogram heartbeat classification for arrhythmias and myocardial infarction." Sensors 23.6 (2023): 2993.

> [3] Ganguly, Biswarup, et al. "Automated detection and classification of arrhythmia from ECG signals using feature-induced long short-term memory network." IEEE Sensors Letters 4.8 (2020): 1-4.
Thank you for your constructive comments.

In this study, one of our primary missions is to achieve an accurate HAR without exceeding the model complexity of existing NNs for HAR. 
Thus, increasing the complexity of the model is not an option in our study because a larger model typically yields better accuracy with increasing inference time. 
As a result, we had to choose CNNs because CNNs are much lighter than RNNs or transformers, and CNNs for HAR already exist. 

Please see the following detailed explanation:

Sensor-based HAR can be defined as a multivariate time series classification task.
To solve this problem, the classifier requires both local (via CNNs) and global (via RNNs or transformers) temporal representations [1].
Specifically, the locality of CNNs improves accuracy due to their translational invariance concerning the precise location of activity within a segment of time-series data [2].
On the other hand, RNNs or transformers have an advantage for global feature extraction because they can model long-term dependencies.
In this regard, many studies have attempted to integrate RNNs or transformers into CNNs [1,3-6], which has increased both accuracy and inference times.
The increase in inference time is primarily because of the lack of device-level optimizations compared with CNNs [7].

We noted that real-time or efficient HAR models using wearable sensors processes the input signals with short segmentation lengths for rapid response.
If CNNs are sufficient to extract meaningful information from the short-term signals, unnecessary increase of inference time due to integration with RNNs or transformers can be avoided.
In Table 1 of the manuscript, CALANet outperformed two CNNs with RNNs, i.e., Bi-GRU-I [5] and DeepConvLSTM [6], on all datasets.
In addition, we compare CALANet with RevAttNet [3] and IF-ConvTransformer [4], hybridizations of CNNs and transformers.
In the below Table, CALANet exhibited comparable performance despite its significantly low FLOPs.
These results indicate that CNNs are sufficient to model the temporal information for the real-time HAR dataset.

|(F1-score / FLOPs)|UCI-HAR|UniMiB-SHAR|DSADS|OPPORTUNITY| KU-HAR|PAMAP2|
|:---|:---|:---|:---|:---|:---|:---|
|CALANet|96.1 / 7.6M|78.3 / 8.8M|90.0 / 8.5M|81.6 / 19.3M|97.5 / 29.6M|79.4 / 74.9M|98.2 / 56.7M|
|RevAttNet|95.1 / 143.1M|76.7 / 168.7M|87.6 / 140.2M|78.6 / 101.5M|97.7 / 335.3M|79.7 / 573.5M|98.5 / 282.1M|
|IF-ConvTransformer|95.4 / 209.8M|77.0 / 183.5M|87.5 / 628.4M|82.2 / 986.2M|96.4 / 491.7M|80.1 / 1.7G|97.4 / 3.0G|


> [1] Zhao, Bowen, et al. "Rethinking attention mechanism in time series classification." Information Sciences 627 (2023): 97-114.

> [2] Hammerla, Nils Y., Shane Halloran, and Thomas Plötz. "Deep, convolutional, and recurrent models for human activity recognition using wearables." Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. 2016.

> [3] Pramanik, Rishav, Ritodeep Sikdar, and Ram Sarkar. "Transformer-based deep reverse attention network for multi-sensory human activity recognition." Engineering Applications of Artificial Intelligence 122 (2023): 106150.

> [4] Zhang, Ye, et al. "IF-ConvTransformer: A framework for human activity recognition using IMU fusion and ConvTransformer." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6.2 (2022): 1-26.

> [5] Tong, Lina, et al. "A novel deep learning Bi-GRU-I model for real-time human activity recognition using inertial sensors." IEEE Sensors Journal 22.6 (2022): 6164-6174.

> [6] Ordóñez, Francisco Javier, and Daniel Roggen. "Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition." Sensors 16.1 (2016): 115.

> [7] Mehta, Sachin, and Mohammad Rastegari. "MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer." International Conference on Learning Representations. 2022.
Thanks for your detailed response. Good luck.
I appreciate the detailed feedback from the authors, which addressed some concerns. 

One thing about "real-time HAR" is that some studies might not exactly highlight "real-time" in the title or abstract, however, they are still highly efficient. Also, in (w2), there are multiple claims about the efficiency of RNN and transformers. I suggest that the authors provide better intuition or motivation for choosing CNN. Also, some studies or tutorials focusing on the computational complexities of different model architectures might also be helpful.
Thank you for your response and enthusiasm for improving this work. Considering additional datasets, such as biosignals similar to motion sensors, can improve the demonstration of this work's applicability to other ML applications. Also, a proper discussion on how the method presented in this work can be applied to different architectures is necessary. The new results should also be appropriately integrated into the paper.
Thank you for your recommendation and encouragement.

Human Activity Recognition is a research topic of interest to the NeurIPS conference from the past to the present. Below are representative works presented at the NeurIPS conference.

* DelPreto, Joseph, et al. "ActionSense: A multimodal dataset and recording framework for human activities using wearable sensors in a kitchen environment." Advances in Neural Information Processing Systems 35 (2022).

* Cheng, Ricson, Ziyan Wang, and Katerina Fragkiadaki. "Geometry-aware recurrent neural networks for active visual recognition." Advances in Neural Information Processing Systems 31 (2018).

* Mahdaviani, Maryam, and Tanzeem Choudhury. "Fast and scalable training of semi-supervised CRFs with application to activity recognition." Advances in Neural Information Processing Systems 20 (2007).

Also, studies regarding the model's complexity concerning inference time on resource-limited devices have recently attracted attention.

* Kuzmin, Andrey, et al. "Pruning vs quantization: which is better?." Advances in neural information processing systems 36 (2024).

* Zheng, Hong-Sheng, et al. "StreamNet: memory-efficient streaming tiny deep learning inference on the microcontroller." Advances in Neural Information Processing Systems 36 (2024).

These studies redesign models primarily based on FLOPs or inference time in seconds, so their analysis depends on the device's choice.

In this regard, one of this study's key contributions and differences from previous works is improving model performance while maintaining the model's complexity based on theoretical time complexity.

Because these aforementioned works are closely related to our work and were presented at the NeurIPS, we believe that our work is within the scope of the NeurIPS conference.
Thank you very much for addressing all of my concerns.

I can't upgrade my current rating from 6 to 7 because of the limited relevance of this manuscript because model architecture proposed are mostly applicable for bio-signals rather than time series in general.  However, a rating of 7 would require moderate impact on several domains. Perhaps, the ubiquitous computing community will find this work more relevant.