Omni-Scale CNNs: a simple and effective kernel size configuration for time series classification

1 INTRODUCTION . One of the most challenging problems for Time Series Classification ( TSC ) tasks is how to tell models in what time scales 1 to extract features . Time series ( TS ) data is a series of data points ordered by time or other meaningful sequences such as frequency . Due to the variety of information sources ( e.g. , medical sensors , economic indicators , and logs ) and record settings ( e.g. , sampling rate , record length , and bandwidth ) , TS data is naturally composed of various types of signals on various time scales ( Hills et al. , 2014 ; Schäfer , 2015 ; Dau et al. , 2018 ) . Thus , in what time scales can a model “ see ” from the TS input data has been a key for the performance of TS classification . Traditional machine learning methods have taken huge efforts to capture important time scales , and the computational resource consumption increase exponentially with the length of TS increase . For example , for shapelet methods ( Hills et al. , 2014 ; Lines et al. , 2012 ) , whose discriminatory feature is obtained via finding sub-sequences from TS that can be representative of class membership , the time scale capture work is finding the proper sub-sequences length . To obtain the proper length , even for a dataset with length 512 , ( Hills et al. , 2014 ) has to try 71 different sub-sequence lengths . For other methods , such as ( Berndt & Clifford , 1994 ; Schäfer , 2015 ; Lucas et al. , 2019 ) , despite the time scale capture might be called by different names such as finding warping size or window length . They all need searching works to identify those important time scales . More recent deep learning based methods also showed that they had to pay a lot of attention to this time scale problem . MCNN ( Cui et al. , 2016 ) searches the kernel size to find the best RF of a 1D-CNN for every dataset . Tapnet ( Zhang et al. , 2020 ) additionally considers the dilation steps . Chen & Shi ( 2021 ) also take 1It has different names for different methods . Generally , it refers to the length of the time series subsequence for feature extraction . the number of layers into considerations . These are all important factors for the RF of CNNs and the performance for TSC . Although a number of researchers have searched for the best RF of 1D-CNNs for TSC , there is still no agreed answer to 1 ) what size of the RF is the best ? And 2 ) how many different RFs should be used ? Models need to be equipped with different sizes and different numbers of RFs for a specific dataset . Using the same setup for every dataset can lead to a significant performance drop for some datasets . For example , as shown by the statistics on the University of California Riverside ( UCR ) 85 “ bake off ” datasets in Figure 1a , the accuracy of most datasets can have a variance of more than 5 % just by changing the RF sizes of their model while keeping the rest of the configurations the same . As also shown in Figure 1b , no RF can consistently perform the best over different datasets . To avoid those complicated and resource-consuming searching work , we propose Omni-Scale block ( OS-block ) , where the kernel choices for 1D-CNNs are automatically set through a simple and universal rule that can cover the RF of all scales . The rule is inspired by Goldbach ’ s conjecture , where any positive even number can be written as the sum of two prime numbers . Therefore , the OS-block uses a set of prime numbers as the kernel sizes except for the last layer whose kernel sizes are 1 and 2 . In this way , a 1D-CNN with these kernel sizes can cover the RF of all scales by transforming TS through different combinations of these prime size kernels . What ’ s more , the OS-block is easy to implement to various TS datasets via selecting the maximum prime number according to the length of the TS . In experiments , we show consistent state-of-the-art performance on four TSC benchmarks . These benchmarks contain datasets from different domains , i.e. , healthcare , human activity recognition , speech recognition , and spectrum analysis . Despite the dynamic patterns of these datasets , 1DCNNs with our OS-block robustly outperform previous baselines with the unified training hyperparameters for all datasets such as learning rate , batch size , and iteration numbers . We also did a comprehensive study to show our OS-block , the no time scale search solution , always matches the performance with the best RF size for different datasets . 2 MOTIVATIONS . Two phenomena of 1D-CNNs inspire the design of the OS-block . In this section , we will introduce the two phenomena with examples in Figure 2 and more discussions can be found in Section 4.6 . Firstly , we found that , although the RF size is important , the 1D-CNNs are not sensitive to the specific kernel size configurations that we take to compose that RF size . An example is given in the right image of the Figure 2 Secondly , the performance of 1D-CNNs is mainly determined by the best RF size it has . To be specific , supposing we have multiple single-RF-size-models which are of similar model size and layer numbers , but each of them has a unique RF size . Let ’ s denote the set of those RF sizes as S. When testing those models on a dataset , we will have a set of accuracy results A . Then , supposing we have a multi-kernel model which has multiple RF sizes2 and set of those sizes is also S. Then , 2A detailed discussion about how to calculate RF sizes for 1D-CNNs with multiple kernels in each layer can be found in Section 3.2 the accuracy of the multiple-RF-sizes-model will be similar to the highest value of A . An example is given in the left image of Figure 2 . Specifically , when testing single-RF-size-models on the Google Speechcommands dataset , the model ’ s performance is positive correlation with the model ’ s RF size . For example , the light blue line whose set of RF size is { 99 } outperforms the light green line { 39 } and light red line { 9 } . For those multiple-RF-sizes-models which has more than one element in their set of RF sizes , their performance are determined by the best ( also the largest because of the positive correlation ) RF size it has . Having more worse ( smaller ) RF sizes will not have much influence on the performance . The second phenomenon means that , instead of searching for the best time scales , if the model covers all RF sizes , its performance will be similar to that of a model with the best RF size . However , there are many designs that can cover all RF sizes . Which one should be preferred ? Based on the first phenomenon , from the performance perspective , we could choose any design that we want . However , as we will show in Section 3.3 , those candidate designs are not of the same characteristics such as the model size or the expandability for long TS data . Therefore , the design of the OS-block that we propose aims at covering all RF sizes in an efficient manner . 3 METHOD . The section is organized as follows : Firstly , we give the problem definition in Section 3.1 . Then , we will explain how to construct the Omni-scale block ( OS-block ) which covers all receptive field sizes in Section 3.2 . Section 3.3 will explain the reason why OS-block can cover RF of all sizes in an efficient manner . In Section 3.4 , we will introduce how to apply the OS-block on TSC tasks . 3.1 PROBLEM DEFINITION . TS data is denoted as X = [ x1 , x2 , ... , xm ] , where m is the number of variates . For univariate TS data , m = 1 and for m > 1 , the TS are multivariate . Each variate is a vector of length l. A TS dataset , which has n data and label pairs , can be denoted as : D = { ( X1 , y1 ) , ( X2 , y2 ) , ... , ( Xn , yn ) } , where ( X∗ , y∗ ) denotes the TS data X∗ belongs to the class y∗ . The task of TSC is to predict the class label y∗ when given a TS X∗ . 3.2 ARCHITECTURE OF OS-BLOCK . The architecture of the OS-block is shown in Figure 3 . It is a three-layer multi-kernel structure , and each kernel does the same padding convolution with input . For the kernel size configuration , we use P ( i ) to denote the kernel size set of the i-th layer : P ( i ) = { { 1 , 2 , 3 , 5 , ... , pk } { 1 , 2 } , i ∈ { 1 , 2 } , i = 3 ( 1 ) Where { 1 , 2 , 3 , 5 , 7 , ... , pk } is a set of prime numbers from 1 to pk . The value of pk is the smallest prime number that can cover all sizes of RF in a range . Here , the range that we mentioned is all meaningful scales . For example , since the TS length is l , we don ’ t need to cover RFs that are larger than l or smaller than 1 . Therefore , the pk is the smallest prime number that can cover the RF size from 1 to l. If we have prior knowledge , such as that we know there are cycles in the TS , or we know the length range of the hidden representative pattern . We could change the RF size range of the OS-block by simply changing the prime number list . An example is given in the left image in Figure 3 , which uses the prime number list in the blue block to cover the RF size range from 10 to 26 . RF sizes of the OS-block : The RF is defined as the size of the region in the input that produces the feature . Because each layer of the OS-block has more than one convolution kernel , there will be several different paths from the input signal to the final output feature ( Araujo et al. , 2019 ; Luo et al. , 2016 ) , and each path will have a RF size . For the 3-layer OS-block , which has no pooling layer and the stride size is 1 , the set of RF sizes S is the set of RF size of all paths , and it can be described as : S = { p ( 1 ) + p ( 2 ) + p ( 3 ) − 2 | p ( i ) ∈ P ( i ) , i ∈ { 1 , 2 , 3 } } . ( 2 ) For the reasons that P ( i ) are prime number list when i ∈ { 1 , 2 } , the set { p ( 1 ) + p ( 2 ) |p ( i ) ∈ P ( i ) , i ∈ { 1 , 2 } } is the set of all even numbers E.3 Thus , we have S = { e+ p ( 3 ) − 2 | p ( 3 ) ∈ P ( 3 ) , e ∈ E } . ( 3 ) With Equation 3 and Equation 1 , we have S = { e|e ∈ E } ∪ { e− 1|e ∈ E } ≡ N+ . ( 4 ) Where N+ is the set of all integer numbers in the range . Specifically , the S ≡ N+ is because a real number must be an odd number or an even number , while E is the even number set , { e−1|e ∈ E } is 3This is according to Goldbach ’ s conjecture . Specifically , the conjecture states that any positive even number can be composed of two prime numbers . For example , 8 = 5 + 3 , 12 = 7 + 5 , and more examples can be found in the left image of Figure 3 . Despite that the conjecture is yet unproven in theory , but its correctness has been validated up to 4× 1014 ( Richstein , 2001 ) , which is larger than the length of all available TS data . OS-block with residual connection OS-block on each variate for multi variate time series dataOS-block with ensemble learning the odd number set . Therefore , with the proper selection of pk , we could cover any integer RF size in a range . It should be noticed that , there might be many options to cover all RF sizes , we use the Godlach ’ s conjecture to make sure that we could all scales .

Summary:  The paper is on the receptive field of CNNs for 1D time series classification. It proposes an elegant decomposition of receptive fields based on the Goldbach conjecture that any number can be represented by a sum of primes. The paper thus puts RFs with prime numbers in CNN layers, and by having multiple layers, these RFs will sum, and thus allowing the proposed network to cover all RF sizes. Experiments on several datasets shows that the method gives good results.