CALANet: Cheap All-Layer Aggregation for Human Activity Recognition
Authors: Jaegyun Park
Dae-Won Kim
Jaesung Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Human activity recognition
Wearable sensors
Neural networks
Real-time systems
Primary area: deep_learning_architectures
TL;DR: We proposed CALANet that allows the classifier to aggregate features for all layer while maintaining the efficiency of existing real-time HAR models.

Abstract: With the steady growth of sensing technology and wearable devices, sensor-based human activity recognition has become essential in widespread applications, such as healthcare monitoring and fitness tracking, where accurate and real-time systems are required. 
To achieve real-time response, recent studies have focused on lightweight neural network models.
Specifically, they designed the network architectures by restricting the number of layers shallowly or connections of each layer.
However, these approaches suffer from limited accuracy because the classifier only uses the features at the last layer.
In this study, we propose a cheap all-layer aggregation network, CALANet, for accuracy improvement while maintaining the efficiency of existing real-time HAR models.
Specifically, CALANet allows the classifier to aggregate the features for all layers, resulting in a performance gain.
In addition, this work proves that the theoretical computation cost of CALANet is equivalent to that of conventional networks. 
Evaluated on seven publicly available datasets, CALANet outperformed existing methods, achieving state-of-the-art performance. 
The source codes of the CALANet are publicly available at https://github.com/jgpark92/CALANet.

CALANet: Cheap All-Layer Aggregation for Human Activity Recognition
Jaegyun Park', Dae-Won Kim I+, Jaesung Lee
School of Computer Science and Engineering, Chung-Ang University, Republic of Korea 2 Department of Artificial Intelligence, Chung-Ang University, Republic of Korea jgp0566. cau@gmail com, {dwkim, curseor)&cau, ac.kr
Abstract
With the steady growth of sensing technology and wearable devices, sensor- sensor-based human activity recognition has become essential in widespread applications, such as healthcare monitoring and fitness tracking, where accurate and real-time sys- tems are required. To achieve real-time response, recent studies have focused on lightweight neural network models. Specifically, they designed the network architectures by restricting the number of layers shallowly connections of each layer. However, these approaches suffer from limited accuracy because the clas- sifier only uses the features at the last layer. In this study, propose cheap all-layer aggregation network, CALANet, for accuracy improvement while main- taining the efficiency of existing real-time HAR models. Specifically, CALANet allows the classifier to aggregate the features for all layers, resulting in a perfor- mance gain. In addition, this work proves that the theoretical computation cost of CALANet is equivalent to that of conventional networks. Evaluated on seven publicly available datasets, CALANet outperformed existing methods, achieving state-of-the-arr performance. The source codes of the CALANet are publicly available Ittps: // gigggtttbbb.. wttptt/..ooo/gggrr/////........e CALAAee
Introduction
Human activity recognition (HAR) fundamental technique healthcare 23 53
fitness tracking
[3]. and surveillance [21] 41 Wearable sensor-based HAR has drawn attention in pervasive computing applications due to the popularity smart wearable devices in recent years 7 53). Specifically, it aims to identify motion details of users or activity tracks from sensor signal patterns [10]. To this end, neural networks (NNs) have been widely used achieve superior learning perfor- mance without handcrafted feature engineering 7 52 Besides, with advances in microelectronics and inertial sensor-based wearable devices, recent researchers have focused on achieving real-time systems
Especially, a recent trend across most studies has become increasingly to train NNs on resource-rich computing device and then deploy them to resource-limited wearable devices, where inference is executed [44 60 61.
Recent real-time HAR studies have focused one-dimensional (ID) convolutional neural networks (CNNs) compatible with various hardware accelerators and deployment frameworks [14 20 23 36
37
60). CNNs include convolution and pooling operations, which allow them to extract more abstracted high-level features as input signals pass from early to later layers. Specifically, the pooling operation abstracts signals by reducing the feature size (temporal resolution), which can be regarded as sampling of signal. As a result, the final classifier predicts an activity class based on the abstracted or sampled features the last layer. The sampled features are more semantic and global
48
51
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input Signals
Conventional CNNs
3rd layer
8th layer The better features.
Predicted
Jump
Ynu
Convolutional and Pooling layers
Convolutional Pooling layers
Fully conneced layer (Clssiffee))
"Jump"
(Correct)
Sit
The better Teatures.
Mahhlh M
"Sit" (Correct)
Talk-Sit
Add Avvlyy Mr Aam 1us
"Sit"
(Incorrect) Features with signifieantly similar waveforms
Segment dice (Timen)
200
Design goal of CALANet
Convo lutional and pooling
Convolutional
Convolutional
and pooling
Convolutional
All-layer
apgregation
Classifier
layer
layers
"Talk-Sit"
(Correct)
How can CALANet aggregate all-layer features cheapty?
Figure I:
Analysis of representations in our experiments on KU-HAR dataset [47] In conventional CNN, the classifier predicts activities only using the feature representations at the last layer. Features the early layer include the detailed information of original signals that may confound the classifier. comparison, features at the later layer are more semantic, but the features (with more compact and short waveforms) make challenging classify activities that share similar semantics. Our goal is to design a CALANet that allows the classifier to use features for layers while maintaining the inference time of conventional CNNs.
has not been proven that the last layer is the optimal representation (88) Although the high-level features sampled by the pooling operation can avoid over-fitting of the classifier [59], for the HAR dataset, the loss of some detailed information makes it challenging to classify activities that share similar semantics, such as "Sit" and "Talk-Sit."
than those of prior layers,
Figure Iillustrates intermediate features in the forward pass of conventional CNNs. Conventional CNNs classify "Jump" and "Sit" well but tend to misclassify "Talk-Sit" as another activity, as shown in Figure T These experimental observations have also been reported existing studies 24 47 Specifically, the features at the eighth layer (with more compact and short waveforms) make it easy for the classifier to discriminate "Jump" and "Sit," compared with the ones at the third layer. On the contrary, the features at the third layer (with more detailed information) can be more suitable than the ones eighth layer when classifying "Sit" and "Talk-Sit" that have similar vibrations signal waveforms. Although which layer has the best features depends on the activity, conventional CNNs classify multiple activitics only using the features at the last layer.
The objective of real-time HAR is to maximize accuracy under real-time constraints. To achieve real-time response, prior HAR studies designed the network architectures of CNNs by restricting the number of layers shallowly (20 23 51 60 or reducing connections of each layer [14] 36 37 48)
However, these approaches suffer from limited accuracy because their classifier only uses the features at the last layer. A straightforward approach to address this issue is to allow the classifier to use computational cost, particularly as the number of layers deepens. Therefore, our goal is to design novel network architecture that allows the network to aggregate the features for all layers while maintaining the computational cost of the conventional CNNs regardless of network depth, shown in Figure
the features for layers [19 27), but this leads substantial increase

---

In this paper, we propose a novel network, CALANet, with a cheap all-layer aggregation (CALA) structure. To achieve our goal, CALANet includes (1) learnable channel-wise transformation matrices and (2) scalable layer aggregation pool. First, we introduce new learnable channel-wise transformation matrices (ICCTMs) to minimize an increase in computational cost due to all-layer aggregation. Given intermediate features at a specific layer (with temporal resolution T and the number of channels M), M LCTMs generate a vector with N < ' elements based on linear transformation and combination without increasing the theoretical computation cost. Second, we improve the effectiveness of all-layer aggregation by introducing a scalable layer aggregation pool (SLAP) that allows CALANet to stack layers without significantly increasing computational costs. As result, the main contributions of this paper are as follows:
We proposed CALANet with a CALA structure that allows the network to aggregate the features for all layers while maintaining the efficiency of CNNs regardless of network depth based on (1) LCTMs and (2) SLAP.
We theoretically proved that the computational cost of CALANet is equivalent to that of conventional CNNs, including even shallow networks.
We empirically demonstrated the effectiveness of CALANet in achieving superior perfor- mance compared to 11 state-of-the-art methods on seven public benchmark datasets.
2
Related Work
Many comprehensive surveys in HAR literature have highlighted the importance of NN-based models
and real-time applications 77 I0 34
44 52 61]. For instance, early real-time HAR studies adopted two-dimensional (2D) convolutional NNs (CNNs) with shallow architectures
42
25 45
Specifically, they transformed the sensor signal patterns to 2D spectral images as an input of 2D CNNs. However, these approaches require complex preprocessing, such as discrete or short-time Fourier transform, which increases the overhead during continuous processing for real-time HAR. Meanwhile, Ignatov 23 proposed a one-dimensional (1D) CNN architecture using basic statistical features to encode global temporal information. Although this approach demonstrated the potential of ID CNN, it still requires several data preprocessing like calculating the histogram of input signals.
Recent real-time HAR studies focus on ID CNNs without any complex data preprocessing, For exam- ple, Zebin al. 60 proposed CNN architecture comprising four convolution layers. Furthermore. they showed the efficiency of parameter quantization
pot-processing for further optimization. In another study, Wan et al. [51) adopted a CNN architecture including three convolution layers. They demonstrated the superiority of CNNs compared with recurrent NN variants on HAR datasets with basic activities. To enhance shallow CNNs, Huang et al. 20 introduced cross-channel communi- cation that exchanges information among channels within the same layer. These models achieved real-time HAR by shallowly restricting the number of layers, resulting in limited accuracy.
To alleviate the issue, some studies have considered efficient variants of convolution at each layer instead of reducing the number of layers. For example, Gao et al. (14) proposed selective kernel module that divides the convolution into split, fuse, and select steps to adjust receptive field size adaptively. Similarly, Tang et al. designed a hierarchical-split block to enhance multiscale temporal features by composing channel groups hierarchically. In another study, Teng et al. [49] proposed RepHAR, which re- p-arametterizss pretrained multibranch CNN deploying it into resource- resitede-lmmited devices. However, these approaches still are insufficient to classify activities that have similar vibrations in signal waveforms because the classifier only uses the features at the last layer. Meanwhile, Park et al. 337 introduced grouped temporal shift network that can flexibly re-design a network architecture to support various hardware specifications. Although this network can derive layer-specific structures suitable for a given computational budget, its performance is limited according to an initial network architecture. Therefore, its performance can be improved by using our CALANet as the initial network, will be described in Section 4.2
plain CNN before
Besides, the classifier requires both local and global temporal representations to achieve high HAR accuracy [64]. Specifically, the locality of CNNs improves accuracy due to their translational invariance concerning the precise location of activity within a segment of time-series data [17]
On the other hand, recurrent layers or attention mechanisms have an advantage for global feature extraction because they can model long-term dependencies. In this regard, many studies have
attempted to integrate recurrent layers [8] 24
35 50, 56) attention mechanisms [40] 63 into
3

---

CNNs, which has increased both accuracy and inference times. The increase in inference time is primarily because of the lack device-level optimizations compared with CNNs [3] 29 60 These
accuracy-oriented networks will be compared with our CALANet in Section 4.2
3 Cheap All-Layer Aggregation Network
In this section, our goal is to design CNN architecture that aggregate features for all layers into the final classifier without increasing the computational cost of CNNs. Furthermore, we prove that the theoretical computation cost of the proposed CALANet is equivalent to that of conventional CNNs, including even shallow networks.
3.1
Computational cost of convolutional neural networks
Our goal is to improve the HAR accuracy while maintaining the efficiency of CNNs. Therefore, before deriving novel network structure and proving theoretical efficiency, we formalize the computational cost of the conventional CNNs in a generalized form. Note that we only consider the computational cost the feed-forward step, not the training step, which unrelated to inference time. Let KC()) be the I-th layer of a network, where KC() and K((+1) are calculated sequentially and independently. Therefore, the theoretical computation cost of the CNN can be defined as a summation for each computation of layers, as described in Definition]] [
Definition 1. Let ex(-) be the computational cost of calculating the output of each layer. Given a network architecture A with layers, the input X() is fed into the 1-th layer with trainable parameters ( to calculate the output ((I-1), Due this layer composition, its computational cost is formalized as
L (A) = Σ (X 1=1
To formalize the computation cost of CNNs, we borrow the concept of time complexity as the upper bound of the computational cost. Similar to Proposition 3 of [37] Eq. is simplified by Proposition] !
Proposition 1. The time complexity of CNNs is formalized as:
M S - -
(2)
where IM and T are the number of channels and temporal resolution, for input data, and IN and DR are average numbers for output channels and kernel sizes across a network, respectively.
The proof is given in Appendix A Because we do not restrict the number of layers shallowly, we will assume that the condition of Eq. is always true in this paper.
3.2 Learnable channel-wise transformation matrix
In this section, we introduce learnable channel-wise transformation matrix (LCTM) that allows our CALANet to aggregate features for all layers without increasing the theoretical computation cost. Figure shows the network architecture of CALANet. Given input signals, the convolution and pooling layers extract the high-level sampled features by calculating temporal correlations and reducing the feature resolution. After that, the features for all layers are connected with the classifier via the cheap all-layer aggregation module based on the LCTMs.
Let Xm be feature vector of m-th channel with temporal resolution T [xm] at any layer. Because T varies with the layer, we define a mapping function f Xm + y E R. where a constant value N & T. After that, we calculate f via a transformation matrix E R.N.T As the mapping function is calculated for each channel of features, m-th feature vector is transformed to ym E R as follows:
Ym Am Xm
(3)
where this transformation can be interpreted as a compression of global temporal information. After that, linear combination is conducted calculate relations between channels follows:
Σ
a Yu
(4)
m

---

Input Signal
layer
layer
3rd Seyer
layar
Channel
Saandrr
Conv
LCTMs
Uheme
LCTMs
Split at
Cheap All-layer Aggregation
LCTMs
Leurnable Chunnel-wise Trunsformation Matrices (LCTMs)
A
channel
A2
2nd channel
A
3rd channel
A
4th chnanol
OOOOO OOOOO OOoCO
N
OC0OO poooo
X
X
X
OOODO
COoOO
a3
a4
T
A can be simplitied as Bm
Bz
B3 OCoCO
Bg OOOOO
LCTMss leamed by stochastic gradient desent.
OOOOO
Figure 2: Network architecture of CALANet. Convolution and pooling layers extract the sampled features by reducing the temporal resolution. CALANet aggregates the features for all layers based on the linear transformation and combination.
where ý E RN is vector and am is m-th coefficient of the linear combination. We replace am Am with Bm RRN. As result, Eq. simplified follows:
Bmxm
(5)
where elements of Bm can directly be optimized by stochastic gradient descent because these matrix multiplications can be implemented by dense layers. Therefore, we define Bm, as LCTM. Herein, the time complexity of calculating M LCTM operations is O(TMN). Finally, ý for all layers are concatenated and then fed into the classifier.
To investigate whether our CALANet can maintain the efficiency conventional CNNs, we analyze the time complexity of CALANet. Based on Proposition [ the time complexity of CALANet is formalized L
Lemma 1. The time complexity of CALANet is equivalent lo:
O(TD,N*L).
The proof given in Appendix B According to Proposition and Lemma 2 our CALANet can aggregate features for all layers while maintaining the efficiency of conventional CNNs.
3.3 Scalable layer aggregation pool
The effectiveness of all-layer aggregation depends layer aggregation pool, i.c., the number of layers, shown in Table 3. To improve the accuracy of CALANet further, we also introduce scalable layer aggregation pool (SLAP) that allows CÁLANet stack layers without significantly increasing computational cost. To this end, we, in this section, aim to omit L in Eq. (2). Inspired by ShuffleNet [62), we first use the grouped convolution and channel shuffle to reduce the time complexity of the standard convolutions. Precisely, the M input channels are evenly divided into G channel groups. After that, the standard convolution generates [N/G] output channels for each channel group. Subsequently, the channel shuffle operation is executed.
5

---

The entire channels are fully related by the channel shuffle operations if and only if (the number of layers) x (the number of channels within each channel group) (the number of channel groups) [62] Therefore, G is set into value satisfying I. x N 2 G². As N 2 G and G are inversely proportional to the computational cost, we set G into I. without any loss of information for channel correlations. Consequently, the time complexity of the stack of the standard convolutions is reduced in Lemma
Lemma 2. The time complexity of calculating the standard convolutions is reduced to:
O(TD&N
The proof is given in Appendix C To reduce the time complexity of all-layer aggregation, we focus on the norm of vectors extracted from LCTMs, i... |y| = IN from Eq. (5) The LCTMs for all layers generate a vector with IN elements fed into the Softmax layer to classify the activities. The large number of units in the Softmax layer may incur overfitting [57]. Therefore, we fix the number of features fed into Softmax layer to IN by dividing the number of rows of LCTM in Eq. (5) by L,
resulting in Bm E RN Consequently, the time complexity of all-layer aggregation is reduced in Lemma 3
Lemma 3. The time complexity of calculating all-layer aggregation is formalized as:
O
(8)
The proof is given in Appendix D Finally, we introduce CALANet with the SLAP by omitting the factor L from its time complexity. Consistent with Lemma 2 and Lemma] the time complexity of CALANet is reduced in Theorem
Theorem 1. The time complexity of CALANet is reduced
O(TD,N²).
(9)
The proof is given in Appendix E From Theorem 1 we crosscheck the efficiency of our CALANet
by making comparisons with the time complexity of shallow CNNs in Corollary and Corollary
Corollary 1. The time complexity of CALANet is equivalent to the shallow CNNs with L 2 2.
Proof. The time complexity of shallow CNNs with I. 2 is M(TD,M + O(TD, O(TD,N2). It is equivalent to Eq. (9)
Corollary 2. The time complexity of CALANet is equivalent to the shallow CNNs with L 1 if M&
Proof. The time complexity of shallow CNNs with L 1 is M(T,, I & N, then it is equivalent to Eq.
In conclusion, our CALANet has a computation cost comparable to shallow CNNs. Especially from Corollary 2 the time complexity of CALANet becomes equivalent to the shallow CNNs even with L 1 as the number sensors increases.
Experiments
In this section, we evaluate the superiority of CALANet. In Section 4.1 we describe the experimental setup. Section 4.2 presents the compared results of CALANet and other networks on seven HAR datasets. Section 4.3 provides an in-depth analysis via an ablation study. Lastly, Section 4.4 measures the actual inference time of CALANet.
4.1 Experimental Settings
Dataset. We used seven public benchmark datasets, including various sampling frequencies, the number of activities, and sensors. They include UCI-HAR [.. UniMiB-SHAR [30], DSADS K
OPPORTUNITY (6) KU-HAR 47]. PAMAP2 46 and REALDISP The details for each
dataset are described in Appendix

---

Table 1: Comparison results on seven datasets.
indicates that the corresponding model is
significantly worse/better than CALANet according to a paired t-test at 95% significance level.
UCI-HAR
UniMiB-SHAR
DSADS
OPPORTUNITY
Model
FLOPs
FLOPs
FLOPs
FI
FLOPs
CALANet (Ours)
96.1
7.6M
78.3
8.8M
8.5M
81.6
19.3M
CALA-GTSNel (Ours)
94.7V 4
3.3M
74.1
4.8M
87.2
5.4M
78.4
15.0M
Shallow ConvNet 23
92.5
17.9M
72.2
18.2M
85.6
48.5M
79.5
74.3M
RepHAR 49
95.1V
31.8M
71.6
37.3M
85.5
32.9M
80.0
26.0M
Res- GTSNet
94.5
6.4M
77.2
7.51M
84.4
7.3M
76.0
6.4M
DeepConvLSSII
85
91.4
67.2M
71.6
80.4M
85.5
68.3M
62.0
50.4M
Bi-GRU-I 50
94.6
46.1M
75.2
54.0M
85.6
48.7M
77.2
39.8M
RevAuNet 40
95.1
143.IM
76.7V
168.7M
87.6
140.2M
78.6
101.5M
[F-Conv Toansforrrrem
63
95.4
209.8M
77.0V
183.5M
87,5
628.4M
82.2
986.2M
T-ResNet [54 12
95.3
123.2M
76.5
145.5M
87.3 V
125.8M
80.9
96.9M
T-FCN 54 12
95.8
68.9M
76.9
80.6M
86.7V
76.1M
76.2
65.8M
MILLET 11
94.7V
111.6M
81.44
129.9M
84.3
132.8M
82.3
125.0M
DSN [55]
95.4
270.8M
79.8
320.0M
86.4
265.7M
71.8
192.1M IM
KU-HAR
PAMAP2
REALDISP
Model
F1
FLOPS
F1
FL.PPs
F1
FL.OPs
CALANet (Ours)
97.5
29.6M
79.4
74.9M
98.2
56.7M
CALA-GTSNet (Ours)
96.1 4
12.2M
76.3
23.9M
95.4
43.3M
Shallow NonyNet 23
77.9
41.6M
67.44
151.8M
95.9
209.9M
RepHAR 40
93.4
74.4M
73.0
131.9M
94.7V
72.7M
Res-GTSNet 37
94.5
15.3M
76.2
28.6M
94.9
18.0M
DeepConyLSTM
93.5
169.1M
77.3
303.9M
91.7w
156.9M
Bi-GRU-I [50
94.9 4
108.0M
71.0
194.1M
96.1
111.3M
RevAuNet 40
97.7
335.3M
79.7
573.5M
98.5
282.1M
63
96.4
491.7M
80.1
1.7G
97.4
3.0G
T-ResNet [54 [2]
95.0
290.0M
71.4
506.1M
96.0w
270.1M
T-PCN [4 12
92.5
161.7M
72.5
298.5M
95.9
184.5M
MILLET
97.8
262.5M
80.2
509.5M
4
352.9M
DSN 35
97.1
634.8M
68.8W
1,08G
97.5
532.7M
Baseline. We compared CALANet with baseline networks. To evaluate the efficiency of CALANet, we used Shallow ConvNet 23 RepHAR [49] and Res-GTSNet [77 as state-of ttate-of-the-art models in real-time HAR. Meanwhile, we adopted four CNNs with recurrent layers attention mechanisms, including DeTPConvLSTM [35], Bi- Bi-GRU-I [50), RevAttNet [40]. and IF-Conv Con-Transformer 63] to verify the effectiveness of our all-layer aggregation. In addition, we used four networks, T-ResNet
(4 T-FCN [54 12, MILLET [[T], and DSN 55 that achieved substantial success in the time-series classification (TSC), which is more general-purpose than HAR. The details for the models and hyperparameters are described in Appendix
To evaluate the performance of CALANet, we used two metrics: Fl-score and floating-point oper- ations (FLOPs). Because the HAR datasets inherently involve a class imbalance, the Fl-score has been commonly used as alternative for accuracy. In particular, FLOPs have been widely used describe how many operations a given model requires to run single pattern. addition, we investigate the change in performance according to L, as will be described in Section 4.3 Meanwhile, Res-GTSNet [37] can derive layer-speeific structures suitable for a given computational budget, and the original paper adopted T-ResNet as an initial network. To improve the efficiency of CALANet further, we also designed the CALA-GTSNet by replacing T-ResNet with our CALANet.
4.2 Comparison results
Table shows the results of comparing CALANet and the baseline networks. The experiments ran ten times, and the average values were recorded on all the datasets. In addition, we performed paired
7

---

Table Ablation study of CALANet on two datasets: LCTMs: Learnable channel wise transforma-
tion matrices, SLAP: Scalable layer aggregation pool, ALA: All-layer aggregation
KU-HAR
PAMAP2
Networks
L
FLOPs
FLOPs
CALANet LCTMs SLAP
29.7M
79.4
74.9M
CALANet with LCTMs only
4
93.8
60.0M
73.1
113.3M
CALANet with ALA only
4
95.0
577.9M
72.8V
1.7G
100.0
97.5
95.0
02.5
90.0
87.5
85.0
02.5
CALANet
72
CALANet
LCTNs
without
80.0
16
48
64
96
70
64
96
128
192
FLOPs (millen)
FLOPs (millon)
(a) KU-HAR dataset
(b) PAMAP2 dataset
Figure 3: Tradeoff between the FLOPs and Fl-score.
I-test at the 95% significance level on each dataset. Table I network was significantly worse/better than CALALet regarding the Fl-score.
/ indicates that the compared
Comparison with real-time CNNs. Table the Fi-scores of CALANet were statistically superior to real-time HAR models on all datasets. In particular, CALANet has the lowest FLOPs compared to other real-time HAR models with standard convolution layers on seven datasets. Meanwhile, Res-GTSNet, with an efficient variant of the convolution, exhibited significantly low FLOPs. This variant can easily integrated with our CALANet to reduce its FLOPs further. As shown Table [ CALA-GTSNet outperformed Res-GTSNet on 86% of the datasets, Also, CALA-GTSNG has lower FLOPs than Res-GTSNet on 71 1% of the datasets. As a result, CALANet and GTSNet can complement each other to improve the accuracy reduce computations. These results demonstrated that our cheap all-r aggregation can maintain a low computational cost.
Comparison with accuracy- -oriented networks. We noted that real-time or efficient HAR models using wearable sensors process the input signals with short segmentation lengths for rapid response. If CNNs are sufficient to extract meaningful information from the short-term signals, unnecessary increases inference time due to integration with recurrent layers or attention mechanisms can be avoided. In Table D CALANet outperformed two CNNs with recurrent layers, i... DeepConVLSTM and Bi-GRU-I, on all datasets. Compared with RevAttNet and IF-Conv Transforme, CALANet exhibited comparable Fl-score despite its significantly low FLOPs. These results indicate that CNNs are sufficient to model the temporal information for the real-time HAR dataset. Compared with TSC models, CALANet showed comparable performance despite its significantly low FLOPs. These results demonstrated that our cheap all-layer aggregation can significantly improve HAR accuracy while maintaining low FLOPs.
4.3
Ablation study
The breakdown effect of CALANet. We conducted an ablation study to investigate the effectiveness and efficiency of our CALANet. The key components of CALANet are LCTMs and SLAP. Therefore, we compared the performance of our CALANet with that of its two variants, which were obtained by removing each component. The first variant removes the SLAP described in Section 3.3 The second variant replaces the LCTMs with fully-connected layers that have the same number of units as the input size. Table 2 shows that our LCTMs substantially reduced FLOPPs for calculating all-layer aggregation without losing the Fl Fl-sccre. In addition, the SLAP enhanced the effectiveness of all-layer

---

SLAP
with LCTMs
CALANet with LCTMMs
CALANet with LCTMs
(millon)
R.0% (milion)
400
150
300
100
200
50
100
Depth Network
Depth Network
(a) KU-HAR dataset
(b) PAMAP2 dataset
Figure 4: Comparison among two networks with regard the impact of on the FLOPs.
aggregation even while reducing FLOPs. Especially, Figure shows the tradeoff between the FLOPs and F1-score with varying numbers of layers in CALANet with/without LCTMs and SLAP. The tradeoff curves closer to the top-left are more efficient, with a higher F1-score per FLOPs. As shown in Figure 3 CALANet with LCTMs and SLAP achieved higher Fi-score in similar computational cost than one without LCTMs and SLAP.
Effect of scalable layer aggregation pool. We investigated the layer aggregation pool which the best F1-score of CALANet is achieved on seven datasets. Table B shows the change in Fl-score of CALANet as the layer aggregation pool L increased; herein, the best F1-score indicated by the bold font on each dataset. As shown in Ta- ble 3 the layer aggregation pool and F1- score tend to be proportional. In Figure 4 CALANet with SLAP (red line) exhibited negligible increase in FLOPs compared with CALANet only with LCTMs. As re- sult, SLAP allows CALANet to stack layers without significantly increasing FLOPs.
Table Fi-score of CALANet on different layer ag- gregation pool, i.e., network depth I.
L
Datascts
2
3
17
UCI-HAR
93.2
93.9
94.8
96.1
95.7
UniMiB-SHAR
72.8
76.0
78.0
78.3
77.5
DSADS
87.2
84.5
86.0
90.0
89.4
OPPORTUNITY
78.9
80.2
79.0
81.6
80.3
KU-HAR
90.0
94.7
92.7
97.5
97.7
PAMAP2
72.5
73.4
76.5
79.4
79.0
REALDISP
92.9
96.7
96.9
98.2
97.7
Performance analysis on similar activities. To verify the performance of CALANet, we investigated the confusion matrices (see Appendix HD. Prior works [47 24 suffered from activities that have similar vibrations in signal waveforms, such as "Sit" and "Talk-Sit." as described in Section [ Compared with these works, our CALANet significantly improved the accuracy of those activities on the KU-HAR dataset. For other examples, these activities include ("rope jumping" and "waking") and ("knees bending crouching" and "reach heels backwards")
[ On the other hand, our CALANet correctly classified "rope jumping" as "waking" compared to RepHAR that
20
14
4%
49
misclassified "rope jumping" as "walking" 20 times [49] on the PAMAP dataset. Compared to
MG-WHAR [ misclassified "knees bending crouching" by approximately 20% as "reach heels backwards", our CALANet misclassified "knees bending crouching" as "reach heels backwards" only two times on the REALDISP dataset.
Applicability of CALA structure. Our CALA structure can effectively be applied existing CNNs if the following constraints are satisfied: (1) the layers of a network architecture should be calculated sequentially and independently; (2) the output of each layer should be able to be expressed (temporal length x channel size) matrix. To the best of our knowledge, most wearable sensor-based human activity recognition models can satisfy the above constraints.
Table 4 we applied our LCTMs and SLAP to SqueezeNet (22]. Specifically, the output of a squeeze convolution layer in each fire module fed into LCTMs and connected to the last layer. As a result, our modification significantly improved the F1-score of SqueezeNet on 71% of all datasets while maintaining its FL.PPs. In addition, we applied CALANet to the ECG heartbeat classification problem using the MIT- MIT-BIH arrhythmia dataset [16]. CALANet exhibited comparable performance with other networks

---

Table 4: Comparison results of SqueezeNet with/without CALA structure on seven datasets.
UCI-HAR
UniMiB-SHLAR
DSADS
OPPORTUNITY
Model
F1
FLOPs
F1
FLOPs
F1
FLOPs
F1
FLOPs
SqueezeNet + CALA (Ours)
92.4
8.2M
75.8
9.5M
87.3
11.4M
68.4
12.3M
SqueezeNet
92.1
10.4M
74.9
12.4M
84.7
13.7M
59.7
13.2M
KU-HAR
PAMAP2
REALDISP
Model
FLOPs
F1
FLOPs
F1
FLOPs
SqueezeNet + CALA (Ours)
96.7
19.5M
70.4
41,5M
93.7
34.8M
SqueezeNet
94.5
25.5M
68.0
52.9M
85.6
39.7M
13 39 designed process ECG signals (see Appendix D. This result shows that CALANet has promising applicability to other ML applications.
4.4 Real-Time Activity Prediction
To estimate the actual response time of our CALANet, used the AMD Ryzen 7 5800X 8-Core Processor without the support of graph- compared
Table 5: Actual inference time of CALANet
ics processing units. Particularly, the inference time of CALANct with Shallow ConyNet. Similar the conventional real-time HAR studies, the measurements were repeated 1,000 times, and the minimum, maximum, and mean values were recorded. Table shows the inference time of the two networks, where the window length was set to 300 (3 s) to slide one instance at a time. CALANet exhibited a response time similar Shallow ConvNet, even though its depth is nine times deeper than that of Shallow ConvNet. Consequently, these measurements show that our model is sufficient to meet the real-time requirements.
Inference Time (ms / window)
Model
Min
Mean
Max
CALANet Shallow ConvNet
1.59ms
2.25ms
3.40ms
1.57ms
2.15ms
3.48ms
5 Conclusion
In this article, we proposed an effective neural network called CALANet for real-time HAR from wearable sensors. In particular, CALANet has all-layer aggregation structure that can aggregate features for all layers based on the learnable channel-wist transformation matrix and scalable layer aggregation pool. As result, CALANet improved HAR accuracy while maintaining the efficiency of existing real-time HAR models. In addition, we proved that the computational cost of CALANet is equivalent to that of shallow CNNs. Our experiments demonstrated that CALANet could achieve state-of-the-are performance on the HAR datasets under low latency.
Future studies can be conducted to overcome the limitations of the proposed method. CALANet does not consider the various computational budgets that can changed according to the specific devices and the runtime optimizations of actual devices, such as memory access costs and parallel computations. For example, future studies may further improve CALANet by introducing new operator designed match the target device.
Acknowledgement. This research was supported in part by the Institute of Information & Communi- cations Technology Planning & Evaluation (IITP) grant funded by the Korean Government (MSIT) (2021-0-01341, Artificial Intelligence Graduate School Program (Chung-Aiy Aaaugg-Ang University)). in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean Government (MSIT) (2021-0-00766 Development of Integrated Development Framework that supports Automatic Neural Network Generation and Deployment optimized for Runtime Environment), and in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.
10

---

References
[1] D Anguita, A Ghio, L. Oneto, X Parra, and JL Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In Proc. 21st Int. Eur. Symp. Artif. Neural Netw. Comput. Intell. Mach. Learn. (ESANN), pages 437-442, Apr. 2013.
[2] Oresti Baños, Miguel Damas, Héctor Pomares, Ignacio Rojas, Máté Attila Tóth, and Oliver Amft. A benchmark dataset to evaluate sensor displacement in activity recognition. In Proc. ACM Conf. Ubiquitous Comput., pages 1026-1035, Sep. 2012.
[3] Billur Barshan and Murat Cihan Yüksek. Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units. Comput. J., 57(11): 1649-1667, Nov. 2014.
[4] Valentina Bianchi, Marco Bassoli, Gianfranco Lombardo, Paolo Fornacciari, Monica Mordonini, and Ilaria De Munari. IoT wearable sensor and deep learning: An integrated approach for personalized human activity recognition in a smart home environment. IEEE Internet Things J., 6(5):8553-856) Oct 2019.
[5] Andreas Bulling, Ulf Blanke, and Bernt Schiele. A tutorial on human activity recognition using body-wom inertial sensors. ACM Comput.Suru. (CSUR), 46(3):1-33 Jan. 2014.
[6] Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R Millán, and Daniel Roggen. The opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognit. Lett., 34(15):2033- 2042, Nov. 2013.
[7] Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. Deep learning for sensor- sensor-based human activity recognition: Overview, challenges, and opportunities. ACM Comput. Surv. (CSUR), 54(4):1-40) May 2021.
[8] Ling Chen, Yingsong Luo, Liangying Peng, Rong Hu, Yi Zhang, and Shenghuan Miao. A multi-graph convolutional network based wearable human activity recognition method using multi-sensors, Appl. Intell., 23(23))281699 169118185 2023.
[9] Yuqing Chen and Yang Xue. A deep learning approach to human activity recognition based on single accelerometer. In Proc. IEEE Int. Conf. Syst., Man, Cybern., pages Oct. 2015.
IEEE,
[10] L Minh Dang, Kyungbok Min, Hanxiang Wang. Md Jalil Piran, Cheol Hee Lee, and Hyeonjoon Moon. Sensor-based and vision-based human activity recognition: A comprehensive survey. Pattern Recognit. Ar8:Art. no. 107561, Dec. 2020.
[11] Joseph Early, Gavin Cheung, Kurt Cutajar, Hanting Xie, Jas Kandola, and Niall Twomey, Inherently interpretable time series classification via multiple instance learning. In Proc. Int. Conf. Learn. Repr. (ICLR), 2024.
[12] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: a review. Data Mining Knowl. Discov, 33(4):917-963 Mar. 2019.
[13] Biswarup Ganguly. Avishek Ghosal, Anirbed Das, Debanjan Das, Debanjan Chatterjee, and Debmalya Rakshit, Automated detection and classification of arrhythmia from eeg signals using feature-inducet long short-term memory network. IEEE Sens. Lett., 4(8):1-4, 2020.
[14] Wenbin Gao, Lei Zhang, Wenbo Huang, Fuhong Min, Jun He, and Aiguo Song. Deep neural networks for sensor-based human activity recognition using selective kemel convolution. IEEE Truns. Instrum. Meas., 70:1-13, Aug. 2021.
[15] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pages 1243- 1252. PMLR, 2017.

---

[16] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus. George B Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. Circulation, 101((3))e21--222) e220, 2000.
[17] Nils Y Hammerla, Shane Halloran, and Thomas Plötz. Deep, convolutional, and recurrent models for human activity recognition using wearables. In Int. J1. Conf. Artif. Intell. (IJCAI), pages 1533-1540, 2016.
[18] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proc. IEEE/CVF Conf. Comput. Vis. Pattem Recognit. (CVPR), pages 558-567, Jun. 2019.
[19] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 4700-4708, Jul. 2017.
[20] Wenbo Huang, Lei Zhang, Wenbin Gao, Fuhong Min, and Jun He. Shallow convolutional neural networks for human activity recognition using wearable sensors. IEEE Trans. Instrum. Meas., 70:1-11, Jun. 2021.
[21] Altaf Hussain, Tanveer Hussain, Waseem Ullah, and Sung Wook Baik. Vision transformer and deep sequence learning for human activity recognition in surveillance videos. Comput. Intell. Neurosci., A022:Art. no. 3454167, Apr. 2022.
[22] Forrest N. landola, Song Han, Matthew W. Moskewicz. Khalid Ashraf, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <005mb model size. arXiv:1602.07360 2016.
[23] Andrey Ignatov. Real-time human activity recognition from accelerometer data using convolu- tional neural networks. Appl. Soft Comput., 62:915-922, Jan. 2018.
[24] Shaik Jameer and Hussain Syed. Deep se-bilstm with ifpoa fine-tuning for human activity recognition using mobile and wearable sensors. Sensors, 23(9):4319, 2023.
[25] Wenchao Jiang and Zhaozheng Yin. Human activity recognition using wearable sensors by deep convolutional neural networks. In Proc. 23rd ACM Int. Conf. Multimedia, pages 1307-1310, Oct. 2015.
[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Int. Conf. Learn. Repr. (ICLR), pages 1-15, Jan. 2015.
[27] Chen Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In Proc. Artif. intell. Statist. (AISTATS), pages 562-570. PMLR, May 2015.
[28) Athanasios Lentzas and Dimitris Vrakas. Non-intrusive human activity recognition and abnor- mal behavior detection on elderly people: A review. Artif. Intell. Rev., 5373):1975-201,, Jun. 2020.
[29] Sachin Mchta and Mohammad Rastegari. Mobilevit: Light- general-purpose, and mobile-friendly vision transformer. In Proc. Int. Conf. Learn. Repr. (ICLR), 2022.
[30] Daniela Micucci, Marco Mobilio, and Paolo Napoletano. Unimib shar: A dataset for human activity recognition using accelcration data from smartphones. Appl. Sci., 7(10):Art. no. 1101, Oct. 2017.
[31] Tomás Mikolov et al. Statistical language models based on neural networks. Presented at Google, Mountain View, 2nd Apr., 80(26), 2012.
[32] Taima Rahman Mim, Maliha Amatullah, Sadia Afreen, Mohammad Abu Yousuf, Shahadat Uddin, Salem A Alyami, Khondokar Fida Hasan, and Mohammad Ali Moni. GRU-INC: An inception-attention based approach using gru for human activity recognition. Expert Syst. Appl., Arr:Art no. 119419, Apr. 2023.
12

---

[33] Mark Nutter, Catherine H Crawford, and Jorge Ortiz. Design of novel deep learning models for real-time human activity recognition with mobile phones. Proc. IEEE Int. Joint Conf. Neural Netw: (IJCNN), pages 1-8. IEEE, Jul. 2018.
[34] Henry Friday Nweke, Ying Wah Teh, Mohammed Ali Al-Garadi, and Uzoma Rita Alo. Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges. Expert Syst. Appl., 105:233 22112,1,160 Sep. 2018.
[35] Francisco Javier Ordóñez and Daniel Roggen. Deep convolutional and Istm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1))Art. 115, Jan. 2016.
[36] Jaegyun Park, Won-Seon Lim, Dae-Won Kim, and Jaesung Lee. Multitemporal sampling module for real-time human activity recognition. IEEE Access, 10:54507-54515, May 2022.
[37] Jaegyun Park, Won-Seon Lim, Dae-Wor Won Kim, and Jaesung Lee. GTSNet: Flexible architecture under budget constraint for real-time human activity recognition from wearable sensor. Eng. Appl. Artif. Intell., 124:Art. no. 106543, Sep. 2023.
[38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, pegeorerform deep learning library. In Proc. Adv. Neural Inf. Proces. Syst. (NIPS), volume 32, pages 8026-8037, 2019.
[39] Bach- Tung Pham, Phuong Thi Le, Tzu-Chiang Tai, Yi-Chiung Hsu, Yung- Hui Li, and fia-Ching Wang. Electrocardiogram heartbeat classification for arrhythmias and myocardial infarction. Sensors, 23(6),9993, 2023.
[40] Rishav Pramanik, Ritodeep Sikdar, and Ram Sarkar. Transformer-based deep reverse attention network for multi multi-sensory human activity recognition. Eng. Appl. Artif. Intell., 122:106150,
Jun. 2023.
[41] Andrea Prati, Caifeng Shan, and Kevin I-Kai Wang. Sensors, vision and networks: From video surveillance to activity recognition and health monitoring. J. Ambient Intell. Smart Environ., 11 (1):5-22, Jan. 2019.
[42] Jun Qi, Po Yang, Atif Waraich, Zhikun Deng, Youbing Zhao, and Yun Yang. Examining sensor-based physical activity recogmition and monitoring for healthcare using internet of things: systematic review. J. Biomed. Informat., 87:138-153, 2018
[43] Sen Qiu, Hongkai Zhao, Nan Jiang, Zhelong Wang, Long Liu, Yi An, Hongyu Zhao, Xin Miao, Ruichen Liu, and Giancarlo Fortino. Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges. Inf. Fusion, 80:241-265. Apr. 2022.
[44] Ramanujam, Thinagaran Perumal, and S Padmavathi. Human activity recognition with smartphone and wearable sensors using deep learning techniques: A review. IEEE Sensors J., 21(12): 13129))34339914440 Mar. 2021.
[45] Daniele Ravi, Charence Wong, Benny Lo, and Guang-Zhong Yang. deep learning approach to on-node sensor data analytics for mobile or wearable devices. IEEE J. Biomed. Health Informat., (1())566-64, Dec. 2016.
[46] Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In Proc. IEEE 16th Int. Symp. Wearable Comput. (ISWC), pages 108-109. IEEE, Jun. 2012.
[47] Niloy Sikder and Abdullah-Al Nahid. Ku-har: An open dataset for heterogeneous human activity recognition. Pattern Recognit. Lett., 146:46-54, Jun. 2021.
[48] Yin Tang, Lei Zhang, Fuhong Min, and Jun He. Multiscale deep feature learning for human activity recognition using wearable sensors. IEEE Trans. Ind. Electron., 70(2):2106-2116, Mar. 2022.
[49] Qi Teng, Yin Tang, and Guangwei Hu. Rephar: Decoupling networks with accuracy-speed tradeoff for sensor-based human activity recognition. IEFE Trans. Instrum. Meas., 72:1 1-11, Feb. 2023,
13

---

[50] Lina Tong, Hanghang Ma, Qianzhi Lin, Jiaji He, and Liang Peng. A novel deep learning bi bi-gru-i model for real-time human activity recognition using inertial sensors. IEEE Sensors J., 22(6): 6164-611 6174, Feb. 2022.
[51] Shaohua Wan, Lianyong Qi, Xiaolong Xu, Chao Tong, and Zonghua Gu. Deep learning models for real-time human activity recognition with smartphones. Mob. Netw. Appl., 25(2):743-755, Dec. 2020.
[52] Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. Deep learning for sensor-based activity recognition: A survey. Pultern Recognit. Lett., 119:3-11, Mar. 2019.
[53] Yan Wang, Shuang Cang, and Hongnian Yu. A survey on wearable sensor modality centred human activity recognition in health care. Expert Syst. Appl., 137:167-190, Dec. 2019.
[54] Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from seratch with deep neural networks: A strong baseline. In Proc. IEEE Iut. Joint Conf. Neural Netw. (IJCNN), pages 1578-1585 IEEE, May 2017.
[55] Qiao Xiao, Boqian Wu, Yu Zhang. Shiwei Liu, Mykola Pechenizkiy, Elena Mocanu, and Decebal Constantin Mocanu. Dynamic sparse network for time series classification: Learning what to "see". Proc. Adv. Neural Inf. Proces. Syst. (NeurIPS), 15:16849-16862 2022.
[56] Cheng Xu, Duo Chai, Jie He, Xiaotong Zhang, and Shihong Duan. Innohar: A deep neural network for complex human activity recognition. IEEE Access, 7:9893-9902, Jan. 2019.
[57] Qi Xu, Ming Zhang, Zonghua Gu, and Gang Pan. Overfitting remedy by sparsifying regulariza- tion on fully-connected layers of cnns. Neurocomputing, 328:69 74, Feb. 2019.
(58) Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proc. IEEE Conf. Comput. vis. Pattern Recognit. (CVPR), pages 2403-2412, 2018.
[59] Afia Zafar, Muhammad Aamir, Nazri Mohd Nawi, Ali Arshad, Saman Riaz, Abdulrahman Alruban, Ashit Kumar Dutta, and Sultan Almotairi. A comparison of pooling methods for convolutional neural networks. Appl. Sci., 12(17):8643, 2022.
[60] Tahmina Zebin, Patricia J Scully, Niels Peek, Alexander Casson, and Krikor B Ozanyan. Design and implementation of convolutional neural network on edge computing smartphone for human activity recognition. IEEE Access, 7:133509-133520, Sep. 2019.
[61] Shibo Zhang, Yaxuan Li, Shen Zhang. Farzad Shahabi, Stephen Xia, Yu Deng, and Nabil Alshurafa. Deep learning in human activity recognition with wearable sensors: review on advances. Sensors, A2(4):Art. no. 1476, Feb. 2022.
[62] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages
Jun. 2018.
[63] Ye Zhang, Longguang Wang, Huiling Chen, Aosheng Tian, Shilin Zhou, and Yulan Guo. ff-convtransformmr:: A framework for human activity recognition using imu fusion and con- vtransformer. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 6(2):1-26, Jul. 2022.
[64] Bowen Zhao, Huanlai Xing, Xinhan Wang, Fuhong Song, and Zhiwen Xiao. Rethinking attention mechanism in time series classification. Inf. Sci., 627:97-114, 2023.
14

---

Appendix
A Proof of Proposition 1
Given the original signals X RTMM the first convolution layer generates new feature sig- nals ( E RT.N" resulting in time complexity of O(TMNU) Suppose that the -P(-1) ,y0 and NO is positive-integer multiple of N(l-1), that is, N c(0) N(--), Generally, 2 1 and the pool- ing layer adjusts the temporal resolution T to T/c if 2 2. Therefore, its time complexity is O(T() D M N()) O(T(-1) (N)())))2) Given the stack of L convolution layers, their time complexity can be simplified as follows:
I-th intermediate convolution layer includes kernel K(O)
OTTD;MMN LO(TD,NLL),
(10)
where IN and D2 are the average of the number of output features and the kernel sizes across the layers, respectively. If M < N(L - 1), the time complexity of convolution layers becomes OTTD, L.L.
B Proof of Lemma 1
Consistent with Proposition I the time complexity of calculating the features is (TT;,N' L). Since the number of output channels across the standard convolution layers is N, the time complexity of calculating LCTMs for all layers is L). Given x IN aggregated features and V activities, the Softmax layer predicts an activity with the time complexity of O(LVV) Consequently, the time complexity of our CALANet is formalized as:
O(TD&NN + O(TIN2L) + +OLNN)
(11)
Because V commonly is less than T x D2 x IN, Eq. 11 can be rewritten as O(TD,NN
C Proof of Lemma 2
Given an input X R.T.M M channels are evenly divided into I. channel groups. More precisely, for each channel group, the convolution layer generates [N/T.] output channels, resulting in the time complcxity of (MTD((NNLL))
Since each convolution layer generates the output E RT across L channel groups, the time complexity of each layer is DM (MN/L)). Because CALANet includes L standard convolution layers to calculate the local temporal correlations, part of Eq. 11 is reduced to OTD,,N"" consistent with Proposition
D Proof of Lemma 3
Given an input X E RT. LCTMs at each layer generate a vector with [N/L] elements, resulting in the time complexity of (T) As shown in Figure 2 the number of output channels for each layer is fixed as NN across layers. Therefore, the time complexity of the all-rayer aggregation is (T /)). Similar to Lemma 2 part 3 of Eq. [11b is reduced to O(TN?).
E
Proof of Theorem I
Consistent with Lemma 2 and Lemma 3, part 2 and part 3 of Eq. 11 is simplified as:
N²) OTTN2)
(12)
In addition, we fixed the number of features fed into the Softmax layer to IN. Therefore, part 4 of Eq. (T) is reduced to O(NV). Because V commonly less than T x Dg x NN Eq. 6 is reduced OTTD,N N2).
15

---

F Details of datasets
The benchmark datasets used in our experiment follow the following setup:
The UCI-HAR dataset [ was recorded at a sampling frequency of 50 Hz. Precisely, 30 subjects performed six basic activities (e.g., walking, upstairs, sitting) using accelerometers and gyroscopes embedded in Android smartphones. As the authors recommended, each segment's ' length is set to 128, and 70% and 30% of the dataset were used as the training and test sets, respectively.
The UniMiB-SHAR dataset 30 was recorded at a sampling frequency of 50 Hz, where the length of each segment is 151. The 30 subjects performed 17 activities, including nine activities of daily living (e.g., walking and standing) and eight fall activities (e.g., forward and syncope), using an accelerometer Android smartphones. Precisely, 70% and 30% of the dataset were used as the training and test sets, respectively.
The DSADS dataset (3] was recorded at a sampling frequency of 25 Hz, where the length of each segment is 125. Eight subjects performed 19 daily and sports activities (e.g.. exercising on a stepper and rowing) using accelerometers, gyroscopes, and magnetometers embedded in five MTx trackers. More precisely, the MT' units measured the sensor signals on the torso, right arm, left arm, right leg. and left leg. We split the dataset into 80% training and 20% test sets based on the subject's identification (ID).
The OPPORTUNITY dataset [6] was recorded at a sampling frequency of 30 Hz a sensor-rich environment with wearable, object, and ambient sensors, where the length of each segment is 90. We only considered wearable sensors for real-time HAR, including accelerometers and inertial measurement units (IMUs); the number of input channels is 113 in our experiments. The four subjects performed 17 activities, including complicated activities such as "drink from cup" and "open door," " categorized into ADL 1-5 and Drill. We used the ADL 5 data for subject I; the Drill data for subject 2; the ADL 1 and 4 data for subject and the ADL data for subject 4 as the test 332 The remaining data were used as the training set.
The KU-HAR dataset [7 was recorded at a sampling frequency of 100 Hz, where the length of each segment is 300. Precisely, 90 subjects performed 18 daily activities (c.g.. talking with hand movements and picking up an abject) using accelerometers and gyroscopes embedded in smartphones. The 80% and 20% of the dataset were used as the training and test sets, respectively.
The PAMAP2 dataset 46 was recorded at 100 and 9 Hz sampling frequencies for IMUs and heart rate monitor, respectively. Precisely, nine subjects performed 18 activities, including basic activities (e.g., sitting and running) and complicated activities (e.g., watching TV and folding laundry), where the length of each segment 512. We used the data aggregated from subjects 102 and 106 as the test set. In addition, we added 30% of the data for "watching TV", "car driving," and "playing soccer" into the test set because they were performed by only one subject. The remaining data were used as the training set.
The REALDISP dataset [2] was recorded
sampling frequeney of 50 Hz from nine IMUs, where the length of each segment is 250. More precisely, 17 subjects performed 33 fitness activities (e.g., lateral bend arm up and upper trunk and lower body opposite twist) using accelerometers, gyroscopes, and magnetometers. In addition, the authors provided orientation estimates in quaternion format. We split the dataset into 70% training and 30% test sets based on subject ID.
G Details of baselines
We summarize the models used in our experiments, as follows:
CNNS for real-time HAR. We adopted three CNNs as state-of-the-art models real- time HAR. Shallow ConvNet [3] comprises single convolution layer and two fully- connected layers, where the basic statistical features to encode global temporal information are concatenated with outputs of the convolution layer. RepHAR [9] comprises three
16

---

convolution layers with re-peer parameterization and a Softmax layer. Res-GTSNet 337 contains nine grouped temporal shift module layers and Softmax layer.
CNNs with recurrent layers or attention mechanisms. We adopted two CNNs with recurrent layers and two CNNs with attention mechanisms to verify the effectiveness of our all-layer aggregation. DeepConvLSTM [35] contains four convolution layers and two recurrent LSTM layers with a Softmax classifier. Bi-GRU-I (50) is composed of two bi-directional GRU layers. three inception layers, and Softmax layer. Compared with LSTT its architecture has reverse order of convolution and recurrent layers. RevAttNet [40 contains six convolutional layers, two recurrent layers, and two reverse attention modules, including LSTM, deconvolution, and multi-head attention. IF- Conv ConyTransformer (33 is composed of IMU fusion blocks, four convolutional layers, and two self-attention layers, where the IMU fusion blocks are used to fuse the features from multiple sensor modalities based sensor-wise convolutional layers.
Time-Series Classification (TSC) models. In addition, we adopted CNNs that achieved substantial success in the TSC, which is more general-purpost than HAR. Specifically, T-ResNet [54] [2] consists of three residual blocks, each comprising three convolution layers with residual connection, and Softmax layer. T-FCN [34] [2] consists of three convolution layers and a Softmax layer. Compared with the real-time models, it has more output channels. MILLET III comprises five identical networks with conjunctive pooling layers, where each network contains six convolutional layers with multiple kernel sizes. DSN (55) contains three sparse CNN module, each of which includes dynamic sparse convolution and point-wise convolution.
For fairness, we re-implemented all the baseline networks in PyTTrch 38 and excluded sophisticated
tricks of each original setting, such as gradient clipping 31 and learning rate warmup [18] Precisely, they were trained for 300 epochs with batch size of 128 using 2080Ti graphics- processing unit. We used the Adam optimizer (26] with 3 = 0.9, 82 = 0.999 and 10 where the learning rate and weight decay were set to 0.0005. For the CALALet, we set IDx, IN, and to 5, 128, and 9, respectively. On the OPPORTUNITY and REALDISP datasets with many input channels (113 and 117), we doubled the number of filters for the first convolution of CALANet The irper-parameters of the existing networks are according to the values recommended in the original papers.
H
Confusion matrices
Figures 5 show the confusion matrices of CALANet on three datasets. Their diagonal elements represent the number of instances correctly classified to the related activities, with the color darkening as the number grows.
17

---

30
200
250
200
310
Predkted
Figure 5: Confusion matrix of CALANet on the KU-HAR dataset. A1, stand; A2, sit; A3, talk-sit; A4, talk-stand; A5, stand-sit; A6, lay; A7, lay-stand; A8, pick; A9. A10, push-up; A11, sit-up: A12, walk; A13, walk-backward; A14, walk-circle; A15, run; A16, stair-up; A17, stair-down; A18, table-tennis.
290
391
500
400
S67
200
204
320
326
122
Predicted
Figure 6: Confusion matrix of CALANet on the PAMAP2 dataset. A1, lying: A2, sitting; A3, standing; A4, walking; A5, running: A6, cycling: A7, Nordic-walking: A8, watching-TV: A9, work; A10, car-driving: A11, ascending-staiin stairs; A12, descending-stairs, A13, vacuum- cleaning; A14, ironing; A15, oading-laundyy A A16, house-cleaning; A17, playing-soccer, A18, rope-jumping.
18

---

BZM MMY D
BEM DE RE
0
0
A74
221
%
4,
4/2
4/-
4
A22
42
Figure 7: Confusion matrix of CALANet on the REALDISP dataset. AI, walking; A2, jogging: A3, running; A4, jump-up; A5, jump-front-bacee A6, jump-sideways: A7, haamppllg/aammsoop//lloseeee A8, jump-rope; A9, trunk-twist-army A10, runk-twiit-ellows, A11, waist-rotation; A13, waist-bends; A A14, maachhhelss-bacwware A15, lateral-bend; A16, lateral-bend- arm-up; A17, repetttiveei foweee-a-orrddtttttieieee A18, upper- upper-trunk-aane hodd-lowrrbboy-oopooee
A12,
A19, arms-lateral-elleatton:: A20, arms-fionttllcceevtton; A21, frontal-hand-llaps: A22, arms-frontal- A25, arms-inner-rotation; A26, knnes-altermatiiely-bread A27, hhels-aleemaaively-bekkiiee A28, knees- bending-crouching: A29, mmees-alternaiivey--ee---oooo A30, rotation--ontthe-knees; A31, rowing;
crossing: A23, houulderr-high--aplldd--ooooo
A24, pouulles-loo-ampiiuud-oootatt
A32, elliptical-bike. A33, cycling.
ECG heartbeat classification
We applied CALANet to the ECG heartbeat clas- sification problem using the MIT-BIH arrhyth- mia dataset (16). which includes 24-hour ambu- latory ECG recordings collected from inpatients and outpatients Boston's Beth Israel Hospi- tal. The dataset has 21,892 heartbeats, each with signal length of 187. In Table 6 CALANet exhibited comparable performance with other networks designed to process ECG signals. This result shows that CALANet has promising ap- plicability other MI applications.
Table 6: Comparison results on the MIT-BIH ar- rhythmia dataset.
Model
Average Accuracy
CALANet
98.2
Pham et al.
39
98.5
Ganguly al.
13
97.3
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: To achieve real-time HAR, prior HAR studies designed the network architec- tures of CNNs by restricting the number of layers shallowly or reducing connections of each layer, resulting in the significant loss of accuracy. The proposed method addressed this problem by improving the accuracy based on cheap all-layer aggregation while maintaining the computational cost of the existing real-time HAR networks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This paper does not consider various computational budgets that may be changed according to the specific devices. This limitation must improved via future works such as neural architecture search, runtime optimization, and parallel computation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
20

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We proved that the computational cost of CALANet is equivalent to that of conventional CNNs, including shallow networks.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We will release code by including the GitHub link in the paper if finishing the blind review process.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
21

---

reproducible, they In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release our code by including the GitHub link in the paper finishing the blind review process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
htcc:/uulll///uuuded///ceeuuaaaa Sww.//ooodSsssss.eoooea wttpsuummmsssssnniii ssi.iioiccciiiiiiicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The full details are provided in the code and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We performed a paired /-test at the 95% significance level on each dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The appendix includes the details on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps:/neurr
Answer: [Yes]
Justification:
This research does NOT violate the the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: As HAR is a fundamental technique in healthcare, fitness tracking, and surveillance, our accurate and real-time model can reduce the negative societal impacts due false predictions.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will include the usage guidelines or restrictions in code release.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited the original paper that produced the datasets and our source code release follows GPL-3,0 license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper and the code release include the details about training. license, usage guidelines, etc.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We used only public datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We used only public datasets.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "jFc1LqJuqg", "reviewer": "Reviewer_QGqB", "summary": "The problem of Human Activity Recognition (HAR) is considered in this paper where the border between different activities can differ depending on the type of activity. For example, one activity can be “just sitting”, and another can be “sitting while speaking”. To this end, the authors argue that we need to leverage the features extracted in all the layers of a neural net. Thus, the authors propose a modification to ConvNet models where (called “all-layer aggregation”) is added to the model which takes its input from all the previous conv layers. To do so, the authors design learnable channel-wise transformation matrices that can be added to the model and provide fast aggregation. Evaluation results on seven HAR dataset shows that the proposed modification can improve the classification accuracy of HAR, compared to other alternatives.", "questions": "1. It seems that such a modification is only applicable to ConvNets. It is not clear how similar things can be done to other architectures.\n \n2. I could not understand whether this CALANet is only useful to HAR or if it can also help with other data types like audio data or bio signals. \n\n3. It would be interesting and useful if the authors could show how this modification can be applied to some benchmark architectures that are built for mobile or wearable devices such as MobileNet or EfficientNet, and similar models.", "limitations": "The main limitation is that the solution is very specific to HAR datasets and it is only applied to basic ConvNets and not other benchmark models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper spots a very interesting problem in HAR and offers an effective solution with a nice architectural modification. The paper is written well and easily understandable.", "weaknesses": "The weakness of this paper is its relevance to the NeurIPS. This work gets more attention and appreciated by people in the Mobile or UbiComp community as the novelty and contribution is not much in the ML part but in systemic modification of the architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the constructive comments. Below, we provide specific answers and explanations regarding those comments.\n\n**(W1) The weakness of this paper is its relevance to the NeurIPS. This work gets more attention and appreciated by people in the Mobile or UbiComp community as the novelty and contribution is not much in the ML part but in systemic modification of the architecture.**\n\nThank you for your constructive comments. We respect good opinions, but we believe that the discovery of new structures in architecture can also serve as a foundation for important research in the field of ML.\n\n**(Q1) It seems that such a modification is only applicable to ConvNets. It is not clear how similar things can be done to other architectures.**\n\n**(Q3) It would be interesting and useful if the authors could show how this modification can be applied to some benchmark architectures that are built for mobile or wearable devices such as MobileNet or EfficientNet, and similar models.**\n\nThank you for your constructive comments. Some constraints must be satisfied to apply our method to other architectures effectively: 1) the layers of a network architecture should be calculated sequentially and independently; 2) the forward pass should include the resolution reduction operation like the pooling layer. 3) the output of each layer should be able to be expressed as a (temporal length * channel size) matrix.\nTo the best of our knowledge, most wearable sensor-based human activity recognition models can satisfy the above constraints. For example, in Inception-like models, LCTMs and SLAP can be effectively applied for each module rather than each operator.\nPlease check the attached .pdf file in “Global(Author) Rebuttal”. In Table 1, we applied our LCTMs and SLAP to SqueezeNet. Specifically, the output of a squeeze convolution layer in each fire module is fed into LCTMs and connected to the last layer.\nAs a result, our modification significantly improved F1-score of SqueezeNet on 71% of the all datasets while maintaining its FLOPs.\n\n**(Q2) I could not understand whether this CALANet is only useful to HAR or if it can also help with other data types like audio data or bio signals.**\n\nThank you for your constructive comments. CALANet is useful when information lost in the intermediate layer affects classification accuracy. In this regard, our CALANet can be useful to some applications using bio signals. On the other hand, most audio data is collected in high sampling frequency. Therefore, these applications require the capability modeling long-term temporal dependency. In this regard, CALANet may not be appropriate.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ppqjnUf4y", "reviewer": "Reviewer_1Z8p", "summary": "This paper proposes an all-layer aggregation network, CALANet, to improve model accuracy while maintaining the efficiency of lightweight models. Specifically, the authors have exploited the features from all layers for classification, as a kind of aggregation.", "questions": "1. How to prove that the performance improvement is because of the aggregation, but not because of the more complicated network architecture? To be more specific, what if we do not use the proposed LCTMs and SLAP but directly increase the layers of CNNs? It would be convincing if the authors provide results with varying numbers of layers with/without LCTMs and SLAP.\n2. The authors have used FLOPs to measure model efficiency, which may be inadequate, because the best hyperparameters for different models are different. If the authors directly use the same hyperparameters for all models, we may see a fair comparison in terms of efficiency, but it is not fair to compare their accuracy/F1. Can the authors provide the results in terms of training/testing time for intuitive efficiency comparison (in Table 1)?", "limitations": "1. This is not a model compression work but just proposed some aggregation modules. The authors should discuss the difference between their work and model compression works.\n2. This work mainly focuses on CNNs. The authors can further discuss other network architectures.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of this work is clear. It sounds reasonable that the features from all layers can provide more information for classification, compared to merely using the features from the last layer.\n2. Extensive experiments have been conducted, including comparison results and ablation study.\n3. The organization of paper is clear and the writing is easy to understand.", "weaknesses": "1. My major concern is about the aggregation. When I first read the title of this paper, I assumed that the authors have trained a neural network and then compress its multiple layers via aggregation. However, it seems that the authors just add several components (LCTMS, SLAP) to a complete CNN. It looks like an advanced version of residual networks with channel weights.\n2. The authors claimed that they wanted to improve model performance while keeping model efficiency. However, in Table, it seems that, in some cases (KU-HAR, PAMAP2), the proposed method did not have obvious F1 improvements but resulted in highly increased FLOPs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the constructive comments of the reviewer. Below, we provide specific answers and explanations regarding those comments.\n\n**(W1) My major concern is about the aggregation. When I first read the title of this paper, I assumed that the authors have trained a neural network and then compress its multiple layers via aggregation. However, it seems that the authors just add several components (LCTMS, SLAP) to a complete CNN. It looks like an advanced version of residual networks with channel weights.**\n\n**(L1) This is not a model compression work but just proposed some aggregation modules. The authors should discuss the difference between their work and model compression works.**\n\nThank you for your constructive comments. There is an important difference between our study and model compression works. The objective of model compression is to reduce model size and computations while minimizing loss of accuracy. Conversely, our goal is to improve accuracy while maintaining computations. Thus, we did not focus on compressing the model. Instead, we fixed its complexity because the computational capability of wearable devices is fixed and hard to change. \n\n**(W2) The authors claimed that they wanted to improve model performance while keeping model efficiency. However, in Table, it seems that, in some cases (KU-HAR, PAMAP2), the proposed method did not have obvious F1 improvements but resulted in highly increased FLOPs.**\n\n**(Q1) How to prove that the performance improvement is because of the aggregation, but not because of the more complicated network architecture? To be more specific, what if we do not use the proposed LCTMs and SLAP but directly increase the layers of CNNs? It would be convincing if the authors provide results with varying numbers of layers with/without LCTMs and SLAP.**\n\nThank you for your constructive comments. If what we have figured out is correct, we apologize for the confusion caused by removing a version without layer aggregation from our CALANet in Table 2 of the submitted paper. To avoid these confusions, we will add Figure 1 and Table 2 in the attached .pdf file to the paper. \nPlease check the attached .pdf file in “Global(Author) Rebuttal”. Figure 1 shows the tradeoff between the FLOPs and F1-score with varying numbers of layers in CALANet with/without LCTMs and SLAP.\nThe tradeoff curves closer to the top-left are more efficient, with a higher F1-score per FLOPs. As shown in Figure 1, CALANet with LCTMs and SLAP the higher F1-score in similar computational cost than one without LCTMs and SLAP.\n\n**(Q2) The authors have used FLOPs to measure model efficiency, which may be inadequate, because the best hyperparameters for different models are different. If the authors directly use the same hyperparameters for all models, we may see a fair comparison in terms of efficiency, but it is not fair to compare their accuracy/F1. Can the authors provide the results in terms of training/testing time for intuitive efficiency comparison (in Table 1)?**\n\nThank you for your constructive comments. We are measuring the train and test times for all baselines and datasets in Table 1. In our measurements, we found that the testing time of the network architecture depends on not only FLOPs but also various environments, including devices, memory available space, and background apps. Interestingly, in most cases, it was confirmed that applying LCTMs and SLAP to a given specific architecture hardly increased the time.\nAlso, the training time tended to be almost proportional to FLOPs because we used the same epochs for all models, and larger models used more memory accesses.\n\n**(L2) This work mainly focuses on CNNs. The authors can further discuss other network architectures.**\n\nThank you for your constructive comments. In sensor-based human activity recognition (HAR) literature, popular architectures include CNNs, RNNs, and Transformer. \nCompared with CNNs, RNNs in terms of real-time HAR have some weaknesses as follows: \n\n1) poor parallelization due to dependency of computations, \n2) the lack of hardware accelerators for edge device deployment.\n\nMeanwhile, Transformer needs to calculate the relationship between timestamp of which each is meaningless.\nFurthermore, the hybridization of CNNs and Transformer can be promising approach. We want to emphasize strengths of CALANet because the self-attentions commonly are calculated for the output of convolutional layers due to their local temporal modeling capability. Therefore, our LCTMs and SLAP can improve the hybridization of CNNs and Transformer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rxowRhkZaB", "reviewer": "Reviewer_VmYr", "summary": "The CALANet describes a technique to aggregate the features from all the neural network layers for human activity recognition (HAR).\nBecause HAR is a common application for wearable devices, the model needs to be lightweight to be deployed on the edge for example on an Apple Watch.\nExisting studies are often limited by shallow networks. The activity prediction is done on the final layer without using the features from previous layers, \nwhich may contain key information.  Working with computational constraints, CALANet proposes to aggregate features from all the layers to improve the model performance.\n\nThe CALANet consists of two modifications: (1) a channel-wide transformation matrix to condense features from each layer (2) A ShuffleNet-like convolution to\naggregate all the layer features with computational efficiency. The authors also presented proof results on why CALANet is within the same computational complexity as a shallow network. The empirical results on 7 benchmarks against the state-of-the-art models supported the authors' claim.", "questions": "1. L60: what do you mean by temporal resolution T? \n2. The majority if not all of the benchmark datasets you used are lab-based, which is really not a realistic assessment. Could consider adding one of the free-living datasets to the baselines if time allows. But you don't have to do this. \n    1. Logacjov, Aleksej, et al. \"HARTH: a human activity recognition dataset for machine learning.\" Sensors 21.23 (2021): 7853.\n    2. Chan, Shing, et al. \"CAPTURE-24: A large dataset of wrist-worn activity tracker data collected in the wild for human activity recognition.\" arXiv preprint arXiv:2402.19229 (2024).", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presented both theoretical guarantees and empirical evidence to validate the computational efficiency of the proposed model.\n2. It was cool to see the authors start the paper with an empirical observation on how the features at different layers might differ to motivate the work.\n3. CALANet addresses an important question for efficient mobile computing in the HAR space. \n4. The manuscript reads well.", "weaknesses": "1. The notations of the proofs can be better clarified so that the readers don't need to refer to the appendix. For example, D is not explained in eq. 2 during its first occurrence.\n2. This is a limitation of the field of HAR in general. Existing benchmarks are small usually with the number of participants under 100. It is fairly easy to overfit on the test set. I can see that you are just doing a simple train/test split. I know that some of the older benchmark recommends this but would be interesting to see cross-fold validation results. It is probably ok if you don't have the time to do this. Furthermore, could you clarify how you selected the hyper-parameters for all the models? Again, I suspect that the results reported could overfit your current test set given your evaluation framework. \n3. In your ablation study you are trying to show the effectiveness of your layer aggregation technique (Table 2), there are several changing variables including network depths and FLOPs in addition to the network tricks you introduced. I don't think we can conclude that the layer aggregation trick worked. To test this properly, we should do the ablation studies using the same network L and probably similar FLOPs across different baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the constructive comments of the reviewer. Below, we provide specific answers and explanations regarding those comments.\n\n**(W1) The notations of the proofs can be better clarified so that the readers don't need to refer to the appendix. For example, $D$ is not explained in eq. 2 during its first occurrence.**\n\n**(Q1) L60: what do you mean by temporal resolution $T$?**\n\nThank you for your constructive comments. We will modify or add some sentences that help understanding of notation, including a kernel size $D_k$ and a length of sequence $T$.\n\n**(W2) This is a limitation of the field of HAR in general. Existing benchmarks are small usually with the number of participants under 100. It is fairly easy to overfit on the test set. I can see that you are just doing a simple train/test split. I know that some of the older benchmark recommends this but would be interesting to see cross-fold validation results. It is probably ok if you don't have the time to do this. Furthermore, could you clarify how you selected the hyper-parameters for all the models? Again, I suspect that the results reported could overfit your current test set given your evaluation framework.**\n\n**(Q2) The majority if not all of the benchmark datasets you used are lab-based, which is really not a realistic assessment. Could consider adding one of the free-living datasets to the baselines if time allows. But you don't have to do this.**\n\nThank you for your constructive comments. We agree that the recommendations of the older benchmark may cause an overfitting on the test set. We are conducting additional experiments and the results of CALANet's 5-fold cross-validation for KU-HAR dataset are as follows:\n\n> KU-HAR dataset (5-fold)\n\n> F1-score: 92.1  95.7  94.7  97.3  94.1\n\nAlso, we almost used the hyper-parameters recommended from original paper except for epochs, batch size, and optimizer.\n\n**(W3) In your ablation study you are trying to show the effectiveness of your layer aggregation technique (Table 2), there are several changing variables including network depths and FLOPs in addition to the network tricks you introduced. I don't think we can conclude that the layer aggregation trick worked. To test this properly, we should do the ablation studies using the same network L and probably similar FLOPs across different baselines.**\n\nThank you for your constructive comments. Please check the attached .pdf file in “Global(Author) Rebuttal”. In Table 1 we compared two networks with similar FLOPs and with/without our cheap all-layer aggregation. Despite applying the layer aggregation, its FLOPs were slightly reduced while improving F1-score. Especially, Figure 1 shows the tradeoff between the FLOPs and F1-score with varying numbers of layers in CALANet with/without LCTMs and SLAP. The tradeoff curves closer to the top-left are more efficient, with a higher F1-score per FLOPs. As a result, we can conclude that the layer aggregation trick worked.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C0eVSMHdx7", "reviewer": "Reviewer_2n6z", "summary": "This paper designs CALANet, a cheap all-layer aggregation network designed for real-time sensor-based  HAR. The main objective of CALANet is to improve the accuracy of HAR while maintaining a low computational costs on edge devices for real-time applications. The authors argue that existing CNN models for HAR often suffer from limited accuracy because they only use features from the last layer of the network. In contrast, CALANet allows the classifier to aggregate features from all layers. The authors theoretically prove that the computational cost of CALANet is equivalent to that of conventional CNNs. The authors utilize 7 datasets to demonstrate the effectiveness of CALANet. The results show that CALANet outperforms seven state-of-the-art methods, achieving superior performance on 7 datasets.", "questions": "Please see above.", "limitations": "Please see above.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "[1] The topic of sensor-based HAR is interesting and important, aligning with the scope of the ML community. \n\n[2] I appreciate authors for providing theoretical proofs, for the complexity and others. \n\n[3] 7 datasets are used in evaluation, providing sufficient evaluation results with detailed analysis. Both efficiency and effectiveness are evaluated. \n\n[4] It is good to see future studies are also discussed.", "weaknesses": "-  One of my concerns is the baselines. Why choose these models as baselines? I believe there are many more advanced models in sensor-based HAR. I suggest the authors go through some papers in AAAI, IMWUT, Sensys, Mobicom, where most HAR papers are published. Otherwise, current improvement might not justify the advantage of CALANet. Also, there are more advanced time series classification models from NIPS, ICML, etc. \n-  The goal of this study is improving the accuracy of HAR with the computational costs during the inference phase as the constraint. Therefore, the fundamental thing is the accuracy. However, why CNN become the choice for this study? While authors mention that CNN is popular for HAR recently in Line 27, this argument does not support the study of accelerating CNN for HAR in this paper. \n-  Also curious about the dataset partition, some are 7:3 and others are 8:2. \n-  More references added in the content will improve the manuscript further, e.g., from line 51 to 55, references can be added to support the argument of the “” straightforward approach” and the “”computational costs”. \n-  In related work section, it seems the discussion of accelerating CNN or neural network is missing, which is a popular topic and has been widely studied. \n-  Writing: Writing can be further improved. For example, in Table 1, authors can put their own models at the same location (e.g., at the bottom). I believe this will lower the cognitive load of readers. In line 47, “under real-time response”. \n-  I suggest the authors to re-design Fig 2 for better readability. Fig 2 takes a lot of spaces but it is challenging to understand why the proposed model can utilize all features with low computational costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the constructive comments of the reviewer. Below, we provide specific answers and explanations regarding those comments.\n\n**(W1) One of my concerns is the baselines. Why choose these models as baselines? I believe there are many more advanced models in sensor-based HAR. I suggest the authors go through some papers in AAAI, IMWUT, Sensys, Mobicom, where most HAR papers are published. Otherwise, current improvement might not justify the advantage of CALANet. Also, there are more advanced time series classification models from NIPS, ICML, etc.**\n\nThank you for your constructive comments. Although there are many advanced models in sensor-based HAR, unfortunately, we could not find the advanced models in “real-time HAR.”\nFor example, generative models or seme-supervised HAR studies are difficult to compare fairly with our CALANet due to different evaluation scenarios. \nInstead, we experimented with two models in time series classification. Please check the attached .pdf file in “Global(Author) Rebuttal”. Table 2 shows the comparison results for CALANet, MILLET, and DSN. MILLET is a TSC framework designed to provide inherent interpretability. Meanwhile, DSN was proposed so that the temporal receptive field is trained sparsely and selectively.\nAs shown in Table 2, CALANet exhibited comparable performance despite its significantly low FLOPs.\n\n**(W2) The goal of this study is improving the accuracy of HAR with the computational costs during the inference phase as the constraint. Therefore, the fundamental thing is the accuracy. However, why CNN become the choice for this study? While authors mention that CNN is popular for HAR recently in Line 27, this argument does not support the study of accelerating CNN for HAR in this paper.**\n\nThank you for your constructive comments. Popular architectures in sensor-based human activity recognition (HAR) literature include CNNs, RNNs, and Transformers. \nHowever, our goal is to provide accurate feedback while satisfying real-time constraints. In real-time HAR, RNNs have some weaknesses, including poor parallelization and the lack of hardware accelerators compared to CNNs.\nIn addition, Transformer needs to calculate the relationship between timestamp of which each is meaningless. Specifically, its search space is larger than CNNs, resulting in slow training time.\nAlthough the fundamental thing in this study is accuracy, other architectures find it difficult to satisfy the real-time constraint. Therefore, we adopted CNN. We apologize for the confusing sentences. To avoid these confusions, we will modify the corresponding sentences.\n\n**(W3) Also curious about the dataset partition, some are 7:3 and others are 8:2.**\n\nThank you for your constructive comments. We used the values recommended from original paper or followed the setting that the prior studies adopted.\n\n**(W4) More references added in the content will improve the manuscript further, e.g., from line 51 to 55, references can be added to support the argument of the “” straightforward approach” and the “”computational costs”.**\n\nThank you for your constructive comments. We will add not only two references [1], [2] but also more references to improve the manuscript further.\n\n**(W5) In related work section, it seems the discussion of accelerating CNN or neural network is missing, which is a popular topic and has been widely studied.**\n\nThank you for your constructive comments. We will add some sentences. For example, furthermore, model compression technology is commonly used to accelerate inference time further on wearable or mobile devices. Especially recent studies preferred quantization methods over other compression methods, such as pruning [3]. This postprocessing can further accelerate and optimize CALANet.\n\n**(W6) Writing: Writing can be further improved. For example, in Table 1, authors can put their own models at the same location (e.g., at the bottom). I believe this will lower the cognitive load of readers. In line 47, “under real-time response”.**\n\nThe entire sentence will be reviewed and modified to improve readability.\n\n**(W7) I suggest the authors to re-design Fig 2 for better readability. Fig 2 takes a lot of spaces but it is challenging to understand why the proposed model can utilize all features with low computational costs.**\n\nThank you for your constructive comments. Although the figure still takes up a lot of space and cannot be attached, we hope it can be shown after further development.\n\n> [1] Lee, Chen-Yu, et al. \"Deeply-supervised nets.\" Artificial intelligence and statistics. Pmlr, 2015.\n\n> [2] Yu, Fisher, et al. \"Deep layer aggregation.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n> [3] Kuzmin, Andrey, et al. \"Pruning vs quantization: which is better?.\" Advances in neural information processing systems 36 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We thank the reviewers for their detailed feedback and valuable comments. We are glad the reviewers find that
- Our paper deals with a novel and interesting question and has a clear motivation.
  - "This paper spots a very interesting problem in HAR and offers an effective solution with a nice architectural modification" – QGqB
  - "The motivation of this work is clear" - 1Z8p
  - "CALANet addresses an important question for efficient mobile computing in the HAR space" – VmYr
  - "It was cool to see the authors start the paper with an empirical observation on how the features at different layers might differ to motivate the work" – VmYr
  - "The topic of sensor-based HAR is interesting and important, aligning with the scope of the ML community" - 2n6z
  - "It is good to see future studies are also discussed" - 2n6z
- Our claim and approach are theoretically reasonable.
  - "It sounds reasonable that the features from all layers can provide more information for classification, compared to merely using the features from the last layer" - 1Z8p
  - "This paper presented both theoretical guarantees and empirical evidence to validate the computational efficiency of the proposed model" – VmYr
  - "I appreciate authors for providing theoretical proofs, for the complexity and others" - 2n6z
- Our paper is well-organized and easily understandable.
  - "The paper is written well and easily understandable" – QGqB
  - "The organization of paper is clear and the writing is easy to understand" -1Z8p
  - "The manuscript reads well" – VmYr

We agree that some aspects of the paper can be improved, and many suggestions will be incorporated in the paper. We respond to individual comments below but briefly provide some common responses here. If any questions are unanswered or our responses need clarification, we would appreciate the chance to engage further with our reviewers.

One of the key concerns that multiple reviewers raised during the review process was that the experiments and performance analysis needed to be improved. Attached is a file containing the additional experiments suggested by the reviewers.
1. Reviewer QGqB asked about whether our architectural modification can be applied to some benchmark models that are built for mobile devices. Table 1 in .pdf file shows the comparison results of SqueezeNets with/without our cheap all-layer aggregation (LCTMs and SLAP). We explain in detail in our response below.
2. Reviewer 1Z8p asked if the authors provide results with varying numbers of layers with/without LCTMs and SLAP. Figure 1 in .pdf file shows the tradeoff between the FLOPs and F1-score with varying number of layers; here, tradeoff curves closer to the top-left are more efficient, with a higher F1-score per FLOPs. We explain in detail in our response below.
3. Reviewer VmYr suggested the ablation studies using the same depth of networks and probably similar FLOPs across different baselines. Table 1 in .pdf file compared two networks with similar FLOPs and with/without our cheap all-layer aggregation. In addition, Figure 1 shows an efficiency of our layer aggregation. We explain in detail in our response below.
4. Reviewer 2n6z suggests that we compare CALANet and more advanced models. We conducted an additional experiment with two larger models in Table 2. We explain in detail in our response below.

Once again, we are grateful for the time and effort put into reviewing this submission, and we firmly believe that these comments will strengthen the clarity of our manuscript.

Meta-review: The paper considers human activity recognition (HAR) from wearable IMU sensor measurements which has been very active research area 
more than a decade. Paper is focusing on improving deep learning -based HAR by proposing convolutional neural network utilizing novel 
architecture to improve the recognition accuracy and decreasing the computational complexity and size of the network for real-time small resource (on the edge/mobile devices) inference. Proposed approach is based on all-layer feature aggregation with learnable channel weight matrices and fast fusion. Proposed methodologies are empirical evaluated against several related methods on publicly available HAR benchmark datasets, showing improvement of recognition accuracy meanwhile reducing the number of computational operations.

Although in borderline, this paper has received positive feedback, and based on scores most of the reviewers are towards accepting the paper. The strengths of the paper are related to motivated and interesting topic with clear research goal (Reviewers QGq8, 1Z8p, and  2n6z), good empirical evaluation (Reviewers 1Z8p and 2n6z) with proper theoretical analysis of background (Reviewers VmYr and 2n6z), and promising results against other models on benchmark datasets. Weaknesses/limitations are related to focusing only on narrow area of HAR (Reviewers QGqB and VmYr), some limitations on evaluation and ablation study (Reviewers VmYr and 2n6z), and not well-justified usage of CNN architecture (e.g., compared to RNN/transformers, pointed by Reviewer 2n6z). 

Authors have provided response to critics in rebuttal by showing some additional experimental result, including: 1) additional ECG-classification task with comparable accuracy against baselines (in discussion comments with Reviewer QGqB), 2) additional ablation study showing usefulness of LCTM and SLAP (in global rebuttal file, Fig 1/ Table 1), and 3) additional experiments against transformer based architectures (in discussion comment with Reviewer 2n6z). Although somewhat narrow application area where the method is proposed to, it has few important merits (described above) and brings interesting knowledge to NeurIPS community (in addition to Ubicomp/wearable computing domain). To keep conference diverse, I am recommending the paper to be accepted as a poster. Authors are encouraged to revise the final paper, including all the improvements and suggestion raised during the rebuttal discussion.
Decision: Accept (poster)