Data Continuity Matters: Improving Sequence Modeling with Lipschitz Regularizer
Authors: Eric Qu
Xufang Luo
Dongsheng Li
Venue: ICLR 2023 notable top 25% (2023)
Keywords: deep learning
data continuity
sequence modeling

Abstract: Sequence modeling is a core problem in machine learning, and various neural networks have been designed to process different types of sequence data. However, few attempts have been made to understand the inherent data property of sequence data, neglecting the critical factor that may significantly affect the performance of sequence modeling. In this paper, we theoretically and empirically analyze a generic property of sequence data, i.e., continuity, and connect this property with the performance of deep models. First, we empirically observe that different kinds of models for sequence modeling prefer data with different continuity. Then, we theoretically analyze the continuity preference of different models in both time and frequency domains. To further utilize continuity to improve sequence modeling, we propose a simple yet effective Lipschitz Regularizer, that can flexibly adjust data continuity according to model preferences, and bring very little extra computational cost. Extensive experiments on various tasks demonstrate that altering data continuity via Lipschitz Regularizer can largely improve the performance of many deep models for sequence modeling.

Published as a conference paper at ICLR 2023
DATA CONTINUITY MATTERS: IMPROVING SEQUENCE MODELING WITH LIPSCHITZ REGULARIZER
Eric Qu"
Xufang Luo'& Dongsheng Li Microsoft Research Asia Shanghai 200232, China
Duke Kunshan University Kunshan, Jiangsu 215316, China zhonghang. ww@uuke.eeeeeeeeeeoooo
(xufluo, dongsheng. 1 wtw.://g.....mmmm.ommsoommmmmooe /emicrosoft.com
ABSTRACT
Sequence modeling is problem in machine learning, and various neural net- works have been designed to process different types of sequence data. However, few attempts have been made understand the inherent data property of sequence data, neglecting the critical factor that may significantly affect the performance of sequence modeling. In this paper, we theoretically and empirically analyze a generic property of sequence data, i.e., continuity, and connect this property with the performance of deep models. First, we empirically observe that different kinds of models for sequence modeling prefer data with different continuity. Then, we theoretically analyze the continuity preference of different models in both time and frequency domains. To further utilize continuity to improve sequence mod- eling, we propose a simple yet effective Lipschitz Regularizer, that can flexibly adjust data continuity according to model preferences, and bring very little ex- tra computational cost. Extensive experiments on various tasks demonstrate that altering data continuity via Lipschitz Regularizer can largely improve the perfor- mance of many deep models for sequence modeling
INTRODUCTION
Sequence modeling is central problem in many machine learning tasks, ranging from natural lan- guage processing (Kenton & Toutanova 2019) to time-series forecasting (Li et al., 2019). Although simple deep models, like MLPs, be used for this problem, various model architectures have been specially designed to process different types of real-world sequence data, achieving vastly superior performance to simple models. For instance, the vanilla Transformer shows great power in natural language processing (Wolf et al. 2020). and its variant Informer (Zhou et al. 2021 is more efficient in time-series forecasting tasks. And a recent work Structured State Space sequence model (S4) (Gu et al. 202T) reaches SoTA in handling data with long-range dependencies. However, few attempts have been made to understand the inherent property of sequence data in various tasks, neglecting the critical factor which could largely influence the performance of different types of deep models. Such investigations can help us to understand the question that what kind deep model suitable for specific tasks, and is essential for improving deep sequence modeling.
In this paper, we study a generic property of sequence data, i.e., continuity, and investigate how this property connects with the performance of different deep models. Naturally, all sequence data can be treated as discrete samples from an underlying continuous function with time as the hidden axis. Based on this view, we apply continuity to describe the smoothness of the underlying function, and further quantify it with Lipschitz continuity. Then, it can be noticed that different data types have different continuity. For instance, time-series audio data more continuous than language sequences, since they are sampled from physical continuous signals evolved through time.
Furthermore, empirically observe that different deep models prefer data with different continu- ity. We design a sequence-to-sequene task to show this phenomenon. Specifically, we generate
"Work done during an intersship in Microsoft Research Asia.
1 CCoresponding author.
"ode is available at http:://rriieuu.siicl...pgg

---

Published as a conference paper at ICLR 2023
1.0
Output
0.8
Inout
0.6
0.4
Dutput np.t
0.2
0.0
25
73
100
125
150
175
200
125
250
175
200
Index
In dedex
0.75
0.70
0.65
Value
0:55
0.50
0.45
Ground Iruth
0.40
Index
200
50
75
Index
300
125
150
175
200
0.55
Ground
0.50
0.4
0.40
Panndormer with Up Regubrizer
100
200
300
Index
Index
(a) High Continuity
(b) Low Continuity
Figure 1: A Scquence-to-sequees task to show that different deep models prefer data with differ- ent continuity. The first row shows input and output sequences. We generate input sequences with different continuities (left column: high continuity; right column: low continuity) and learn map- ping function using different models (second row: S4; third row: Transformer). We can see that S4 prefers more continuous sequences, while Transformer prefers more discrete sequences. And adjusting continuity according to the preferences of models with Lipschitz Regularizer can largely improve their performances. More details of this experiment are in Appendix A
two kinds of input sequences with different continuity, and map them output sequences using exponential moving average. Then, we use two different deep models to learn this mapping. Each model has an identical ID convolution embedding layer, and a separate sequence processing mod- ule. One uses the S4 model (Gu et al. (02I)) and the other uses vanilla Transformer (Vaswani al.) 2017) (with the same number of layers and hidden dimensions). The results of this experiment are shown in Figure []] It can be observed that the S4 model achieves significantly better performance with more continuous inputs, and the Transformer performs better with more discrete inputs. Note that, essentially, they are learning the same mapping only with different data continuity. This clearly shows different models prefer different data continuity.
Inspired by the above observation, we hypothesize that it is possible to enhance model performance by changing the data continuity according to their preferences. To make the proposed method simple derive surrogate that can be directly optimized for the
and applicable for different deep models, Lipschitz continuity, and use it as regularizer in the loss function. We call the proposed surrogate Lipschitz Regularizer, which depicts the data continuity and can also be used adjust it.
Then, we investigate the data continuity property for different models and how to use Lipschitz Reg- ularizer to change data continuity according to the model preference. We provide in-depth analyses in both time and frequency domains. On the one hand, Lipschitz continuity describes the continuity of sequences over time, which feature in the time domain. Here, we investigate two models. One is a continuous-time model S4, and the other model Informer is based self-attention. As for S4, since the fitting eтror of the S4 model is bounded by the Lipschitz constant, S4 prefers smoother input sequences with smaller Lipschitz constant. Hence, ers more continuous by adding the Lipschitz Regularizer to the loss function. Experiment results on the Long Range Arena benchmark demonstrate that Lipschitz Regularizer can largely improve
make the inputs of S4 lay-
2

---

Published as a conference paper at ICLR 2023
the performance of the S4 model, especially for tasks with discrete inputs. Conversely, Informer is built upon self-attention, which is designed to process some tokenized discrete data, e.g., text, so In- former prefers less continuous sequences. Therefore, by subtracting the Lipschitz Regularizer from the loss function. Prediction performance and empiri- cal analyses on many time-series tasks well prove the superiority of the Lipschitz Regularizer. Also, we observe the same results on the task mentioned above as shown in Figure
decrease the continuity of input sequences
On the other hand, for the frequency domain, we find that Lipschitz Regularizer represents the expectation of the frequency of data's underlying function. Here, we take the ReLU network as the studied case, and theoretically justify that Lipschitz Regularizer is related to the spectral bias a phenomenon that neural networks tend to prioritize learning the low-frequency modes. We then propose to use Lipschitz, Regularizer by subtracting from the loss function to mitigate the spectral bias. In this way, neural networks are forced to learn high-frequency parts, and convergence can be accelerated since information in different frequency bands can be learned simultaneously.
In summary, Lipschitz Regularizer can be used to flexibly adjust data continuity for wide range of deep models which have a preference for data continuity. It improves various models with very little extra computational cost, shedding light on inherent data property analyses for sequence modeling.
2 RELATED WORK
Deep Neural Networks for Sequence Modeling Sequence modeling plays a critical role in many machine learning problems. Many general architectures, including MLPs (Rahaman et al. 2019). RNNs (Mikolov et al.] 2010), and CNNs (Bai et al.) 2018), can all be used for sequence model- ing. And recently, two types of models show great power in addressing challenges for sequence modeling, such as handling complex interactions and long-range dependencies. The first type is self-attention-based models. For instance, the vanilla Transformer (Vaswani et al., (017), Informer (Zhou et al.] (02T], and Performer (Choromanski et al.] 2020) all show great performance on natu- ral language processing, time-series forecasting, and speech processing, respectively. Another type is continuous-time models, which are built upon the view that inputs are sampled from continuous functions. They include but not limited to Neural ODE (Chen al. 2018), Lipschitz RNN (Erichson et al. 2020), State-space model (Gu et al., 2021). In this paper, we do not aim at proposing novel models like previous works, but we focus on understanding intrinsic preference for input sequences. We show these two types of models both have preference for the data continuity property, and we utilize it to promote their performance.
Lipschitz Continuity of Neural Networks The Lipschitz continuity is a general property for any function, and is widely used for analyzing different kinds of neural networks, including MLPs (hang et al., 202T Gouk et al.) 202T), CNNs (Zou et al., 2019), self-attention-basel networks (Dasoulas et al. 2021), graph neural networks (Gama et al., (020)) and GANs (Gulrajani et al.) 2017]. It becomes an essential property of neural networks, and can be used in various ways, such as improving adversarial robustness (Meunier et al., 2022) and proving generalization bounds (Sokolić) et al. 2017). In this paper, we focus on the Lipschitz continuity of the underlying function of sequence data, and use as a data property, but not a property of models.
3 LIPSCHITZ CONTINUITY OF SEQUENCE DATA
In this section, we first show the measure for continuity of sequence data, and how it can be used as a regularizer. We give a definition for Lipschitz. Regularizer here, and leave detailed analyzes and usages of it in specific models in the rest of the sections. Then, we provide views for Lipschitz Regularizer in both time and frequency domains.
To define the measure for the continuity of sequence data, we view inputs signals, and data points in the sequence are discrete samples of an underlying continuous function with certain time steps. Next, calculate the Lipschitz constant of the underlying function, which is widely used as the measure for continuity. Specifically, suppose the sequence is Tr Ti, Fns and the underlying function is defined as (1a) = Zo. [(h) = æı,-... s(I,) Fne where lo, l,..., ln are time steps.
3

---

Published as a conference paper at ICLR 2023
Then, if we let to 0, 1, I,....tt, n. the Lipschitz constant Ly of function f
lf(tt) f(tz)]
D3 2t i
L
max (0
max LE (0,),........
(1)
|1 16-t [/g
By Mcan Value Theorem, for i,j E {0, 1, (0,1,...,77) n and - 1, we could always find an interval
[k, k + 1] of time step such that i < k S
x-1 < |2++1 -xkl Therefore, we have
L
max LJE (U,
max (E,
TR+1 -xk/-
|i j
However, since we would like to adjust this continuity according to the preferences of different models, this measure should be easy to be optimized. but is hard to pass gradients due to the max operator. To help with the optimization process, we design surrogate by taking the average over all terms and changing the norm to &g. Moreover, since we simply use this surrogate as a regularizer in the loss function to flexibly adjust the continuity for various models, name this term Lipschitz Regularizer, and its formal definition is given as follows.
Definition 3.1. (Lipschitz Regularizer) Suppose the sequence is To, T1, Fn- We define the Lip- schitz Regularizer in the following equation:
LLip
(ai+1 (1i+1-2) x;)²
(3)
1=0
3.1 VIEW LIPSCHITZ REGULARIZER IN TIME AND FREQUENCY DOMAINS
We then provide two views for Lipschitz Regularizer. On the one hand, Lipschitz Regularizer is a feature for sequence data in the time domain, representing the continuity of sequences over time. Thus, it can be used to alter the continuity of input sequences to specific models. As shown in Figure [ different deep models have different preferences for data continuity. We can use the Lipschitz Regularizer to manually make the sequences more or less continuous, and therefore improve the performance of the model. An example of increasing the continuity to improve the performance of continuous-time model is described in 1 A converse example of decreasing the continuity to improve the performance of an attention- based model described in $4.2
On the other hand, from the frequency perspective, Lipschitz Regularizer directly relates to the frequency the function, and can be used to alter modes with different frequencies. Specifically,
(xi41 xz1²
/ (as(!)) dt
di (2miE)) (2(E((-EE) (())))) 4T="CE
(4)
where & is the frequency the Fourier transform f. p(E) j(()/C is the normalized squared Fourier transform of f. where f(E) == (x)ee=2nnt
Details of the derivation are presented in Appendix G.I] Essentially, the Lipschitz Regularizer of sequence data represents the exception of the frequency of the data's underlying function. Besides, previous literature shows that neural networks tend prioritize the learning of low-frequency parts of the target function (Rahaman) et al., 2019]. We find that Lipschitz Regularizer can be utilized to mitigate this phenomenon by emphasizing high-frequency parts, which allows the network results in faster convergence rate. The details of this discussion are in $.1
all spectra simultaneously and
4 TIME DOMAIN
In this section, we view Lipschitz Regularizer in the time domain, and show how it can be used to make the sequence more discrete or continuous over time, catering to the preference of different models. Generally, to change the continuity of the input sequence to different models with the Lipschitz Regularizer, we apply it on the output of the embedding layer before the sequence is sent to different models. We describe the details of two different models in the following sections.
4

---

Published as a conference paper at ICLR 2023
Table 1: Accuracy of the S4 model and its variant with our proposed Lipschitz Regularizer (S4
Emb + Lip) in LRA. S4 + Emb is set to ablate the effect of the extra embedding layer.
ListOps
Text
Retrieval
Image
Image-c
Path
Path-c
PathX
PathX-c
S4
59.53
86.51
91.07
88.54
84.27
94.02
89.11
96.03
92.41
S4 Emb
58.94
87.12
90.28
87.25
85.13
92.37
90.32
93.87
92.81
S4 Emb Lip
61.37
89.74
93.83
89.19
88.43
93.52
91.39
95.72
94.36
4.1 STATE SPACE MODEL
The State Space Model is a classic model in control engineering. Gu et al. (2021) extended it to the deep sequence model, and proposed the S4 model. S4 is continuous-timm sequence model. advances SoTA on long-range sequence modeling tasks by a large margin. An S4 layer can be denoted follows:
i Ar + Bu
(5)
= Cx + Du,
where u the input function, x the hidden state y is the output. A, B, C, are trainable matrices.
The critical and essential design in the S4 layer is the transition matrix A, which is initialized with the HiPPO matrix. The HiPPO matrix makes the S4 layer optimally remember the history of the imput's underlying function, so the S4 model can substantially outperform previous methods on long- range sequence modeling tasks. Particularly, the HiPPO matrix is designed to find the best polynomial approximation of the input's underlying function given a measure that defines the optimal history, and memory budget which is the hidden dimension in the model. Each measure corresponds an optimal HiPPO matrix.
Theoretical Analyses To connect the continuity property with the S4 model, we provide the fol- lowing intuition here while formal proposition along with its proof in Appendix G.2 Generally, the eтror rate of HiPPO-LegS projection decreases when the sequence is more continuous/smooth (Gu et al. 2021). Here, LegS denotes the scaled Legendre measure, which assigns uniform weights to all history. This is also true for S4 layers, since the HiPPO matrix is the most critical design in the S4 layer. However, in many tasks, such as natural language processing, the input sequence not particularly smooth. This will deteriorate the performance of the S4 model.
Lipschitz Regularizer be used to solve the above problem, because it can adjust the continuity of sequences. Specifically, since we cannot directly manipulate the underlying function of the input sequence, add a ID convolutional layer that does not change the sequence length an embedding layer before the S4 layer, and then apply Lipschitz Regularizer to the output of the embedding layer as follows:
Cly. ú. i) = (s. 0) (l),
(6)
where y is the ground-truth, and ÿ is the output of the S4 model. i is the output of the embedding layer, and Lsı is the original loss of the S4 model. X is hyperparameter to control the magnitude of the Lipschitz Regularizer. By using Equation (6) as the loss function, the input of the S4 layers becomes more continuous, the error of the HiPPO-LegS projection and S4 layer be reduced, leading to better model performance.
Experiments To demonstrate the effectiveness of the Lipschitz Regularizer, we use a modified version of the Long Range Arena (LRA) (Tay et al. 2020) benchmark with harder tasks. The descriptions of the original LRA are in Appendix In addition to the original IRA,, we create harder tasks with more discrete sequences. Particularly, them use pixels as inputs (i.c., Image and Pathfinder), which could be more continuous than texts in the other 3 tasks. So we design Image-c, Path-c, and PathX-c, in which the contrast of images is increased, and the increasing degree is randomly sampled from 50% to 100% for each image.
notice that among these 6 tasks, 3 of
We test three methods the modified LRA. The first is the original S4 model (denoted as S4). The second one is the S4 model with ID convolutional layer as the embedding layer, and Lipschitz Regularizer is applied to the outputs of the embedding layer (denoted as S4 + Emb
5

---

Published as a conference paper at ICLR 2023
Lip). Furthermore, we also design the third model to ablate the effect of the extra embedding layer. Here, we use the S4 model with the same embedding layer as the previous method, and Lipschitz Regularizer is not applied (denoted as S4 Emb). Hyperparameter À is chosen from {1, 4, when the model performs best on the validation set.
Results are listed in Table I and have the following observations. (1) It is obvious that method (i.e., S4 + Emb Lip) significantly outperforms other methods in almost all tasks, especially in tasks with discrete inputs, such as Text and Retrieval. Improved performance in Image-c, Path- c, and PathX-c shows that Lipschitz Regularizer can mitigate the influence of increased contrasts. These results well demonstrate the effectiveness of the Lipschitz Regularizer, indicating that it can make input sequences of the S4 layer more continuous, and better cater to the preference of the S4 model. (2) Comparing the results of S4 on Image/Path(X) and Image-c/Path(X)-<
can be observed that the performance of the S4 model degenerates with the increasing contrasts of images. The cause the deceased continuity against the preference of the S4 model, verifying that the S4 model indeed prefers continuous inputs. (3) Only adding the extra embedding layer (S4 Emb) makes the accuracy decrease in out of tasks, indicating that improvements come from the effect of the Lipschitz Regularizer, but not the extra layer. Besides, this extra embedding layer is also the main reason causing the performance drop in Path and PathX dataset. In Appendix [B.2] the visualization for the output vector of the embedding layer shows that this embedding layer may overly and incorrectly blur or even erase some informative shapes in the original picture, causing some necessary information lost, and the model confused.
4.2 TRANSFORMER-BAN
MODELS
In this section, we show that Lipschitz Regularizer can improve the performance of Transformer- based models when inputs are continuous. In particular, we choose time-series forecasting tasks whose inputs are highly continuous, and we use three Transformer-basee models, i.e. vanilla Trans- former (Vaswani et al.] 2017), Informer (Zhou et al.] (202T) and Autoformer (Wu et al.) [02T] to evaluate the effectiveness of Lipschitz Regularizer. Although these models already have good per- formance on time-series forecasting tasks, due to the preference of Transformer-baser models for discrete sequences (shown in Figure T and highly continuous inputs, we can still apply Lipschitz Regularizer to further improve the model by decreasing the continuity of input sequences. Specifi- cally, since all three models have an embedding layer, we directly apply Lipschitz Regularizer the output of the embedding layer as follows:
Cly, 9. i) = Crransoormr 0) ALLip()),
where y the ground-truth, and ý is the output of the respective model. i is the output of the embedding layer, and Transformer is the original loss of the Transformer-base based model. X controls the magnitude of the Lipschitz Regularizer. Note that different from the usage in the S4 model, here subtract Lipschitz Regularizer to make the input discrete, and cater to the model preference.
Experiments We use 5 datasets in this experiment and their descriptions are in Appendix C.I Evaluation metrics are Mean Square Error (MSE) and Mean Absolute Error (MAE). Hyperparameter À is chosen from {1, 21,2,3 3, 4, 5, 6, 7, 8} when the model performs best on the validation set.
Results of Transformer, Informer, Autoformer, and these models with Lipsehitz Regularizer (de- noted as Transformer + Lip, Informer + Lip and Autoformer + Lip) are shown in Table 2 and results of multivariate experiments are Appendix C.2 We can see that the models with Lips- chitz Regularizer generally outperform the original models on most of the tasks. This indicates that Transformer- banssd models prefer discrete sequences and reducing input continuity with Lipschitz Regularizer can be helpful for them. We also note that the Lipschitz. Regularizer is more effective on vanilla Transformer than the models with specialized designs for time series forecasting. This in- dicates the vanilla Transformer is more sensitive to data continuity, and special designs in Informer and Autoformer may mitigate it.
Besides, to investigate whether Lipschitz Regularizer changes the data continuity, we also show curves tracking the Lipsehitz constant of the output of the embedding layer during training in Fig- ure 2 Curves show that with the Regularizer, the Lipschitz constant becomes larger than it is in the original model, and continuously increases during training. Results demonstrate that Lipschitz Regularizer indeed change the continuity, and thus improve the model.

---

Published as a conference paper at ICLR 2023
Table Univariate time-series forecasting results of 3 Transformer-bait -based models and training them with Lipschitz Regularizer (indicated by + Lip). Note in this table, prediction window sizes are converted to lengths of sequences used in the model.
Methods Metric
Transformer
Transformer
Lip
Informer
Informer
Lip
Autoformer
Autoformer
Lip
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
24
0.07047
0,20586
0.07019
0.20570
0.09842
0.24947
0.08882
0.23674
0.05667
0.18596
0.05504
0.18495
48
0.18902
0.37046
0.16716
0,34974
0,15845
0.31907
0.12615
0.28333
0.07860
0.22324
0.07422
0.21398
0.39773
0.55569
0,30811
0,48183
0.18314
0.34619
0.10579
0.25552
0.09232
0.24037
0.08983
0.23544
336
0.41523
0.56902
0.413324
0.56402
0.22166
0.38720
0.11810
0.26959
0.10462
0.25484
0.10461
0.25483
720
0.65586
0,75324
0.62233
0.73160
0,26883
0.43506
0.13131
0.28731
0,12069
0.27791
0.12394
0.27833
24
0.09449
0,24259
0.07560
0.20989
0,09309
0.24015
0.08626
0.22559
0.11136
0.26315
0.09345
0.25515
E7
48
15016
0.30996
0.13222
0.29278
0,15483
0.31445
0.13684
0.289.36
0.15137
0.30316
0.14945
0.30129 29
0.25197
0,41087
0.21046
0.37453
0.23193
0,38947
0.30071
0,20400
0.35646
0,18370
0.33714
336
0,22258
0
0.20867
0.37298
0.26321
0.41659
0,24875
0.40827
0.22188
0.37417
0.21195
0.36425
720
0.21932
0,38844
0,18445
0.35793
0,27722
0.43063
0.23646
0.39648
0,25612
0.40089
0,25604
0,40085
24
0.01279
0.084100
0.01210
0.08312
0
0.13717
0.01815
0,09147
0,02517
0.11778
0.02300
0.10107
ET1mm
48
0.08974
0,25869
0.02872
0.12820
0.06944
0.20255
0.05848
0.19686
0.04130
0.15783
0.03931
0.15601
0.05341
0.1
0,05182
0.15017
0.19414
0.37236
0.13336
0
0,05432
0.18033
0,05258
288
0,22354
0,40455
0.13780
0,29825
0,40140
0.55355
0.30266
0.46864
0.11893
0.27181
0.07521
0.21728
672
0.40726
0,58824
0,40726
0.55826
0,511664
0.64300
0.27543
0.48377
0.09156
0.23690
0.09280
0.23621
24
0.00223
0.03468
0.00154
0.02497
0.11676
0.25142
0.11256
0,23844
0.00740
0.06422
0.00736
0.06329
Weatber
48
0.00422
0,04106
0,00292
0.03022
0,17822
0,31846
0.19134
0.32408
0.01002
0,07648
0.00978
0.07727
0.00537
0.05975
0,00319
0.04464
0,26585
0,39764
0,25138
0.37400
0.01038
0.07082
0.00528
0.05638
336
0.00524
0,05772
0.00417
0,03673
0,29713
0.41571
0,24748
0.37725
0.00729
0.06492
0.00566
0.05888
720
0.00933
0.07630
0.00272
0.03823
0.35875
0.46647
0.26479
0.39214
0.00960
0.08758
0.00925
0.07136
48
0.26161
0.37762
0.24277
0.36460
0,23894
0.35891
0.17306
0.30488
0.57845
0.55653
0.52893
0.54479
168
0.32283
0,41766
0.29221
0.39677
0,44680
0,50315
0.18765
0,31497
0,52339
0.53251
0,42961
0,48926
0.47213
0,50381
0.37916
0.45903
0,48892
0.52840
0.37758
0,43689
0.53511
0.54472
0,51762
0.52391
720
0.48477
0.52413
0.46524
0,51649
0.54026
0.57059
0. 0,44496
0.50387
0.80028
0.66756
0,91144
0.69692
0,46930
0,51409
0.43643
0.49698
0,58225
0.60782
0.37740
0.45817
0.76603
0.65389
0.64859
0.62054
Count
2
49
46
44
0.45
Lipschitz Regularizer
Lipschitz Constant
0.40
Original
Wilk
0.35
0.3.5
0.30
336
0.30
720
0.25
0.25
0.20
0.20
0.15
0.15
0.10
2
3
5
0.10
Epoch
À
Figure 2: The Lipschitz constant of the output of the embedding layer during the training process of Informer + Lip. The experiment is the uni- variate ETTh, with the prediction window size of 24h.
Figure MSE of Informer + Lip on univariate EETh, with different X. Colors represent differ- ent window sizes.
We also show an ablation study for the hyperparameter À in Figure 3 We can observe that (1) MSE increases when À < O, while decreases when > O. Since positive À reduces the data continuity, can conclude that Informer prefers discrete sequences, and Lipschitz Regularizer can reduce the continuity and cater to the preference; (2) MSEs do not have a large variance for different positive X, indicating that the performance improvement is not sensitive to hyperparameter changes.
5 FREQUENCY DOMAIN
In this section, we study how continuity affects the performance of deep models from the frequency perspective. We take the ReLU network as the study case, and provide theoretical analyses and ex- periment results to show the effectiveness of applying the Lipschitz Regularizer on ReLUU networks.
7

---

Published as a conference paper at ICLR 2023
5.1 RELU NETWORK
A ReLU network Rd + R with L hidden layers of width di,... , dr. is defined as:
g(x) (I((e+)) oo o T(T) .... T(1) (x), .0
(8)
where T(*), I. R is an affine function (dg d. dr+1 1 = 1) and or is the ReLU function.
Theoretical Analyses In the previous literature, Rahaman et al. (2019) showed that the low- frequency part of the sequence data learned faster by the ReLU network, and such phenomenon is called the "spectral bias". We claim that the Lipschitz Regularizer could help mitigate the spectral bias. Intuitively, when the Lipschitz constant of the ReLU network increases, we expect that the model can learn more information in high- high-frequency parts. We provide formal proposition and its proof on this intuition in Appendix G.3 This inspires us to balance frequency modes via changing the Lipschitz continuity of functions. Besides, suppose use ReLU network to learn a sequence- to io-sequence mapping, where values of data in the input sequence (length n) increase lincarly in the interval (0, 1) with step size and the output generated by the mapping function h(t). Note that since values of data in the input sequence increase linearly, the Lipschitz, constant of the ReLU network is the same as the output sequence. Therefore, we design the decayed Lipschitz Regularizer as follows:
Cly, 5) LMSE(Y, 1) -e CL.p
where y is the ground-truth generated by h(t) and ÿ is the prediction. AMSE is the MSE Loss. and e are hyperparameters that control the magnitude and decay rate of the Lipschitz Regularizer, respectively.
We further explain the reasons why this regularizer can mitigate spectral bias in two aspects. First, by Equation (4). the added term could be seen as direct penalty to the low-frequency part of the output sequence. Since the value of data in the input sequence increases linearly, this is equivalent to penalizing the low-frequency part of the ReLU network, and prioritizing the learning of the high- frequency part.
In another perspective, Rahhmman et al. (2019) claimed that the origin of the spectral bias is the gradually increasing parameter norm, and Lipschitz Regularizer can intentionally relieve it. Specif- ically, Fourier components of the ReLU network (E(E) is bounded by O(Lg), and Lg is bounded by the parameter norm, which can only increase by a small step during the optimization step. Hence, gradually increasing parameter norms can hinder the learning of high-frequency parts at the early optimization stage. Besides, due the fact that Lipschitz Regularizer can intentionally change Lg subtracting Lipschitz Regularizer as Equation ( can enlarge the parameter norm, making it possi- ble for optimizing both high and low-frequency parts. This can be seen as a warm-up process for the network where the parameter norm increases at the beginning of optimization, and then the conver- gence can be significantly accelerated, since modes of all frequencies can be learned simultaneously after the warm-up.
Experiments We choose a mapping task to evaluate the proposed Lipschitz Regularizer. Specif- ically, we try to learn the mapping function whose input is the sequence with linearly increasing values, and output is a highly periodic sequence. Given frequencies K {kk kg...., k. ampli- tudes A {ai, al...... ... a,,,), and phases + = {0i, .2,...,,,, the mapping function is defined as h(x) = E. A, sin(2k,,r + di..
In this experiment, we take {0.1,
10 and frequencies K = (5, 10.....45 45, 50), amplitudes A 1). The phases are uniformly sampled from 0 to 2n, i.e., di ~ (0, 2T). The input samples in the sequence are uniformly placed over (0, 1) with the number of samples N 100, and the output is generated by h(x).
As for the model, we use 6-layer deep ReLU network with the hidden dimension set to 256 for all layers. To verify the effectiveness of the proposed Lipschitz Regularizer, we train two identical networks with the same training procedure. One is trained with the decayed Lips- chitz Regularizer and the other without it. We set hyperparameter À E {1, 3, 3,4, 4,4,5] 5) and c E {0.00001, 0.0001, 0.001, 0.01, 0.1) when the model performs best on the validation set.
We show the frequency and MSE of ReLU networks during the training process in Figure 4 From Figure 4 (a) and (b), notice that low-frequency parts learned first in both networks, but
8

---

Published as a conference paper at ICLR 2023
10
3001
2.0
1.75
with Lipeests Requenton
300% 080)
130
123
3390 500
1.00
10000 3000
250
123
10030 22006 30000 43026 50000 60003 Training Beeation
Froguani D60
(a) Frequency of ReLU network
(b) Frequency of an identical ReLU (c) MSE of the ReLU network network with Lipschitz Regularizer and w/o Lipschitz Regularizer
Figure 4: Evolution of the frequeney and MSE of ReLU networks during the training process. In (a) and (b), color indicates the normalized amplitude of the Fourier component at the corresponding frequency, i.e., lãe (Aa) Lipschitz Regularizer enables faster learning of high frequencies and faster convergence.
Iteration 100
Iteration 5000
Iteration 10000
Iteration 60000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
06
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 5: Predictions of ReLU networks (first without Lipschitz Regularizer; second row: with Lipschitz Regularizer) during the training process. With Lipschitz. Regularizer, high-frequency parts can be learned faster.
with the decayed Lipschitz Regularizer, high frequencies can be learned significantly faster. Figure (c) shows that Lipschitz Regularizer can accelerate convergence. We also show predictions of two models during the training process in Figure 5 which gives more intuitive result, indicating that high frequencies can be learned faster when we use decayed Lipschitz Regularizer to warm- up optimization. All results demonstrate that Lipschitz Regularizer enables almost simultaneous learning for all frequencies, so spectral bias can be relieved this way, and the convergence is accelerated.
SUMMARY
We investigate a generic property of sequence data, i.e., continuity, which is closely related the performance of different models, and propose Lipschitz Regularizer to flexibly adjust the continuity for various models. We first empirically observe that the different deep models prefer different data continuity. Then, from both time and frequency domains, we provide in-depth theoretical and experimental studies for specific models. For the time domain, we show that the continuous- time model S4 prefers continuous sequences, while the Transformer-basee discrete inputs. We use Lipschitz Regularizer to adjust the data continuity for both of them and largely improve their performance by catering to their preference. For the frequency domain, show that Lipschitz Regularizer can help mitigate the spectral bias, and accelerate convergence for ReLU networks. In general, Lipschitz Regularizer is available for any sequence modeling tasks and models which have preference for data continuity, and can accordingly facilitate learning for various models with very little computational cost.
model Informer prefers

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors would like to thank Yifei Shen and Yansen Wang for their helpful discussions and insights. The authors also want to thank our reviewers for providing all the valuable feedback and suggestions.
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprini arXiv: 100.:0000....0001111
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.
George Dasoulas, Kevin Scaman, and Aladin Virmaux. Lipschitz normalization for self-attention layers with application to graph neural networks. In International Conference Machine Learn- ing, pp. 2456-2466. PMLR, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Ma- honey. Lipschitz recurrent neural networks. In International Conference on Learning Represen- tations, 2020.
Fernando Gama, Joan Bruna, and Alejandro Ribeiro. Stability properties of graph neural networks. IEEE Transactions Signal Processing, 68:5680-5695. 2020.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree. Regularisation of neural net- works by enforcing lipschitz continuity. Machine Learning, 110(2):393-41 2021.
Albert Gu, Tri Dao. Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances Neural Information Processing Systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representatiinss 2021.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
Makerere AI Lab. Bean disease dataset, January 2020. URL https://github.
AI-Lab- Makerere/ ibean/
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IFFF/CVF International Conference Computer Vision, pp. 10012-10022, 2021.
10

---

Published as a conference paper at ICLR 2023
Laurent Meunier, Blaise Delattre, Alexandre Araujo, and Alexandre Allauzen. A dynamical system perspective for lipschitz neural networks. In International Conference on Machine Learning, pp- 15484-15500. PMLR, 2022.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2. pp. 1045-1048, Makuhari, 2010.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning, pp. 5301-5310. PMLR, 2019.
Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing, 65(16):4265 4204442266880000 2017.
Yi Tay, Mostafa Dehghani. Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: benchmark for efficient transformers. In International Conference on Learning Representations, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention you need. Advances in neural informa- fion processing systems, 30, 2017.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv: 180400290 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing:
system demonstrations, pp. 38 45, 2020.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with curo-coorllation for long-tern series forecasting. Advances in Neural Information Processing Systems, 34:22419 04-22411-22430,, 2021.
Bohang Zhang, Tianle Cai, Zhou Lu, Di He, and Liwei Wang. Towards certifying l-infinity ro- bustness using neural networks with l-inf-dist neurons. In International Conference on Machine Learning. pp. 12368 12368-12379. PMLR, 2021.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11106-11118 2021.
Dongmian Zou, Radu Balan, and Maneesh Singh. On lipschitz bounds of general convolutional neural networks. IEEE Transactions on Information Theory, 66(3):1738-11599 2019.

---

Published as a conference paper at ICLR 2023
A APPENDIX
TABLE OF CONTENTS
DETAILS OF EXPERIMENT IN THE INTRODUCTION
12
A.1
UNIVARIATE
12
A.2
MULTIVARIATE
12
B
DETAILS OF EXPERIMENT FOR THE S4 MODEL
15
B.1
DATASET DESCRIPTION
15
B.2
ANALYSIS FOR THE PATHFINDER TASK
15
DETAILS EXPERIMENT FOR THE TRANSFORMER BASED MODEL
16
C.1
DATASET DESCRIPTION
16
C.2
RESULTS MULTIVARIATE
FORECASTING
16
C.3
CASES IN TIME-SERIES FORECASTING TASKS
16
D
FINE-TUNE SWIN TRANSFORMER WITH LIPSCHITZ REGULARIZER
19
APPLY LIPSCHITZ REGULARIZER TO THE SPEECH CLASSIFICATION TASK
20
F
NEURAL ODE WITH LIPSCHITZ REGULARIZER
20
G
MATUEMATICAL DERIVATIONS
21
G.I
DERIVATION OF EQUATION (4)
21
G.2
CONTINUITY AND THE S4 MODEL
21
G.3
CONTINUITY AND THE RELU NETWORK
22
A DETAILS OF EXPERIMENT IN THE INTRODUCTION
A.1 UNIVARIATE
We present a this experiment, we generate two types of input sequences with different continuity (each has 1000 samples), and map them to output with the exponential moving average l(()::
task in the Introduction section, and show more details here. In
2 Je + 11-1++ h(t) = 1+ w
h(t 1)
where ø1, ..,,,.., TN is the input sequence, is the window size (set to 50). We choose the ponential moving average because is a sequence-to- tooeencee---eeuenee mapping that makes use of the con- textual information. The Lipschitz constant of the input and output sequence is shown in Table 3 Note that the high Lipschitz constant represents low continuity, while the low Lipschitz constant represents high continuity. Then, train the $4 model and the Transformer model with generated input and output sequences. Each model has a ID convention embedding layer with kernel size 5, stride and padding Both Transformer and $4 have separated layer with the hidden dimension set to 16. We also apply Lipschitz Regularizer to the output of the embedding layer and train models again. MSE of these 4 models is shown in Table 3 We could observe that S4 performs better with continuous inputs and the Transformer is better with discrete inputs. Also, Lipschitz Regularizer can improve the performance S4 and Transformer by changing the data continuity into their prefers ones.
A.2 MUL.IVVRRATE
We repeat the above experiment with multivariate data. Specifically, we also generate high and low continuity input sequences with dimension 16 (each has 1000 samples). The input sequences
12

---

Published as a conference paper at ICLR 2023
Table Results of the experiment in the Introduction.
High Continuity
Low Continuity
Input Output
0.0543
0.2706
Lipschitz Constant
0.0107
0.0142
S4
0,00014
0.00567
S4 + Lip
0.00045
MSE
Transformer
0,00003
0.00222
Transformer + Lip
0.00038
mapped output with exponential moving average. Same models are trained the generated data.
Results are shown in Table 4 We also randomly sample four dimensions and corresponding curves
are shown in Figure 6 7
Our findings and conclusions in the univariate experiment also hold
in the multivariate case.
Table 4: Results of the experiment in the Introduction running with multivariate data.
High Continuity
Low Continuity
Input Output
0.0708
0,3592
Lipschitz Constant
0.0138
0.0171
S4
0.00072
0.00963
S4 + Lip
0.00141
Average MSE over all variates
Transformer
0.00014 14
0.01368
Transformer + Lip
0.00418
0.7
0.6
Value
0.5
0.4
0.3
0.2
0.7
0.6
Value
0.5
0.4
0.3
Ground Truth
0.2
25
100 Index
125
150
175
200
25
50
100 Index
125
150
175
200
Figure 6: Results of the S4 model with high continuity multivariate data for the experiment in the Introduction.
13

---

Published as a conference paper at ICLR 2023
0.7
0.6
Value
0.5
0.4
0.3
0.2
0.7
0.6
Value
0.5
0.4
0.3
Transformer
Regularizer
0.2
25
100 Index
125
200
25
50
125
150
200
Index
Figure 7: Results the Transformer model with high continuity multivariate data for the experiment in the Introduction.
0.70
0.65
0.60
A AMA M
Mdd
0.55
0.50
0.46
0.35
8.98
0.60
0.66
0.50
Ground
0.40
0.35
with Lip Regu/brizer
0.30
25
50
100 Index
125
150
175
200
50
75
100 Index
125
150
175
200
Figure 8: Results of the S4 model with low continuity multivariate data for experiment in Introduction.
0.70
Value
0.60
0.55
0.45
0.40
0.70
0.65
0.60
0.56
0.50
0.45
Transformer
25
50
75
100 Index
125
150
175
200
0
25
50
75
100 Index
125
150
175
200
Figure 9: Results of the Transformer model with low continuity multivariate data for the experiment in the Introduction.
14

---

Published as a conference paper at ICLR 2023
B DETAILS OF EXPERIMENT FOR THE S4 MODEL
B.I DATASET DESCRIPTION
In this experiment, test the Lipschitz Regularizer the S4 model. The testing dataset is the Long Range Arena (LRA) benchmark. Specifically, LRA contains various long sequence modeling tasks with sequence lengths ranging from IK to 16K, which very challenging for deep models. Tasks in LRA include (1) ListOps, in which the model needs to calculate the answer with a list of algebraic operations. (2) Text, a binary classification task with byte-level texts from IMDB reviews, (3) Retrieval, document retrieval task with byte-level texts and documents from ACL Anthology Network, (4) Image, an image classification task with the image in CIFAR- I- dataset flattened into the pixel sequence, (5) Path (referred as Pathfinder in Gu et al. (202T)), , in which the model needs to deduce whether two points in the image are connected by dashed lines, and the image is also flattened into the pixel sequence, (6) PathX, harder version of Path, where the dimension of input images increased from 32 x 32 to 128 x 128.
B.2 ANALYSIS FOR THE PATHFINDER TASK
We notice that the Lipschitz Regularizer causes deteriorated performance the Pathfinder dataset, so here, we provide detailed analyzes to explore the reason. We show a case in the experiment of the S4 model on the Path dataset in Figure I0 We can see that the performance drop is mainly caused by the embedding layer. As explained in 4. since we cannot directly manipulate the underlying function of the input sequence, we add an extra embedding layer before the S4 layer. However, changes from figure al to a2 show that this embedding layer may overly and incorrectly blur or even erase some informative shapes the original picture, causing some necessary information lost, and making the model confused. Although Lipschitz Regularizer can slightly relieve this issue, the necessary path information is not as obvious as is in the original image. The performance of these 3 models (i.c., S4, S4 Emb, S4 + Emb Lip) in Table I alsoo matches this finding. Moreover, Figure bl, 62, and b3 show that when the contrast increased, these shapes are not likely to be erased since their pixels all have high gray values. Hence, Lipschitz Regularizer can improve model performance on the Path-c task.
(al)
(a2)
(a3)
(bl)
(b2)
(b3)
Figure 10: A case in the experiment of the S4 model on the Path dataset. In this task, the model needs to deduce whether two points in the image are connected by dashed line. (al) An image randomly sampled from the Path dataset. (bl) The image in (al) with 100% contrast increased. (a2, b2) Average of the output vector the embedding layer in a trained S4 Emb model, with al and bl as the input, respectively. (a3, b3) Average of the output vector of the embedding layer in trained $4 Emb + Lip model, with and bl the input, respectively.
15

---

Published as a conference paper at ICLR 2023
C DETAILS OF EXPERIMENT FOR THE TRANSFORMER-BASE MODEL
C.I DATASET DESCRIPTION
In this experiment, we show that the Lipschitz Regularizer can improve the performance of Transformer-basee models, including Transformer, Informer, and Autoformer. We use 3 real- world datasets: ETT (Electricity Transformer Temperature), ECL (Electricity Consuming Load), and Weather. ETT has separate datasets, i.e., {ETTh,, ETTh₂] } for 1-hour-level with 2 separated countries, and ETTm; for 15-minute-level. And we use multiple prediction window sizes, including (24h, 48h, 168h, 336h, 720h, 960h} for ETTh, ECL, Weather, and {6h, 12h, 24h, 72h, 168h} for ETTm.
C.2 RESULTS OF MULTIVARIATE TIME-SERIES FORECASTING
Results of multivariate time-series forecasting of three Transformer- models are presented in 5 We can see that Lipschitz Regularizer can improve the model performance in most cases, showing that altering data continuity is also helpful in multivariate time-series forecasting tasks. Besides, we can also observe that the improvement by Lipschitz Regularizer is slightly less signif- icant than which in univariate time-series forecasting. The reason may be that we apply the same regularizer to input sequences of all variates. However, these input sequences may have different continuities and need regularizers with different weights. Future work might be adding trainable weights to the regularizer of different input sequences.
Table 5: Multivariate time-series forecasting results of 3 Transformer-basee models and training them with Lipschitz Regularizer (indicated by + Lip). Note in this table, prediction window sizes converted to lengths of sequences used in the model.
Methods
Transformrr
Transformer Lip
Informer
Informer Lip
Autoformer
Autoformer Lip
Metric
MSE
MAE
MSB
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.59446
0.57293
0.57268
0,56055
0,57727
0.54945
0.53483
0.51756
0.39562
0.43466
0.37109
0.40143
48
0.82961
0.70984
0.71223
0.65060
0.68461
0.62487
0.68292
0.61618
0.41434
0.44104
0.38277
0.41764
168
1.05050
0,83959
104581
0.84110
0,93119
0,75159
1.02153
0,79883
0.46037
0.46466
0.46020
0.47129
1.40297
0.99042
L04308
0.804337
1.12811
0.87302
0.99066
0.75376
0.51710
0.49936
0.50278
0.48853
720
1.05294
0,80128
104462
0,50122
1,21454
0.89606
1.20265
0.88891
0.47690
0.49172
0,50977
0.50879
0.83025
0.71355
0.42514
0.72025
0.66539
0.33658
0,43113
0.26384
0.34252
0.25465
0.33327
ETThs
48
1,28728
0.9288
0.65413
0.64771
1,45708
1.00622
1.81144
1.08344
0.31247
0.37143
0.31090
0.37014
168
3.66520
1.91462
3,56351
1.43387
3.48945
1.51457
3,05333
1.44302
0.46657
0.46249
0,46982
0.46203
5.11314
1,81923
2.95696
1.37183
2,72290
1,33987
2.35546
1,28684
0.47924
0.48042
0,49315
0,48750
720
3.00465
1.44033
2.529334
1.26787
3.46729
1.47321
3.72312
1.66671
0.47352
0.48400
0.46775
0,48066
0.28475
0.35111
0,28900
0.34211
0.32315
0
0.35075
0
0.35616
0.40320
0.35807
ET
48
0.45711
0.45664
0.43928
0.44483
0.49426
0
0.44407
0.45275
0.44404
0. 0.44776
0.43239
0.44631
0.68831
0,60144
0,53434
0,51490
0.67758
0.61533
0.47372
0,47680
0,55732
0.49872
0.55357
0.4970
288
0.88883
0.71069
0.83600
0.66682
1,05643
0.78565
L03285
0
0.58302
0.50759
0.56224
0.50469
672
1.4777
0,83787
0.92590
0.73367
1,19203
0.92626
0.90468
0.72072
0.56261
0.50822
0.56208
0,50788
0.14897
0.23293
0.21319
0,28302
0.33501
0
0.32660
0.38280
0.15944
0.24224
0.16573
0.24907
Weather
48
0. 0.26360
0.33987
0.22625
0,31913
0.39546
0.45890
0.38138
0.43057
0.21638
0. 0.29847
0.21580
0.29788
0.49273
0,48412
0.42120
0.41574
0.60843
0.56714
0.60603
0.5640
0.30901
0.37048
0.31191
0.37294
336
0.66636
0.58604
0.60194
0.57205
0.70204
0.61955
0.67344
0.62094
0.35603
0.39652
0,34620
0.38616
720
0.89504
0.69045
0.50826
0.52560
0,83106
0.73079
0.65801
0.60332
0.42178
0.43347
0.42083
0,431 25
0.24832
0.35046
0.23187
0.33168
0.34399
0.39289
0,25829
0.35700
0.18320
0.29892
0.18281
0.29758
168
0,26446
0.36343
0,23862
0.33770
0.36820
0.42427
0.37272
0.43410
0.22662
0.33740
0.21555
0.32441
BCL
336
0.27420
0.37078
0.25691
0.36375
0,3806
0.43101
0.34641
0.42144
0.21740
0.33667
0,23392
0.34069
720
0.28641
0.37610
0.27986
0.3451
0.406 16
0.44335
0.35887
0.43815
0.29826
0. 0.38835
0.36776
0.33395
960
0.328
0.39887
0.30984
0,38930
0,45993
0,54774
0.44522
0,50400
0.27856
0,37727
0,23146
0,31658
Count
4
46
12
38
13
38
C.3 CASES IN
FORECASTING TASKS
We present some cases to provide some deeper analyses for Lipschitz Regularizer. Here, we plot the last dimension of forecasting results for qualitative comparison. In all figures of this section, the first 96 data points are inputs of the model, and others are forecasting results.
Univariate Forecasting We show three examples of univariate forecasting in Figure II 12 L3
Figure 11 and 12 show how the Lipschitz Regularizer improves the performance of the model, while
16

---

Published as a conference paper at ICLR 2023
Figure 13 shows negative case. In Figure 13 the main problem is that the model trained with
Lipschitz Regularizer captures a larger decrease than the original data when the time is in 100-200.
Ground rush Prodiction
0.10
0.10
0.08
0.00
0.02
0.c0
Figure 11: Univariate forecasting example of Transformer on the Weather dataset with the prediction window size set to 720. Left figure shows the result of the original Transformer (MSE: 0.00933, MAE: 0.07630). Right figure shows the result of the Transformer trained with Lipschitz Regularizer (A 1, MSE: 0.00272, MAE: 0.03823).
0.-
GroundTuun
GroundThuz
Rediitioon
Prediction
-25
-03
My M
20
2.
Figure 12: Univariate forecasting example of Autoformer on the ETTh2 dataset with the prediction window size set to 168. Left figure shows the result of the original Autoformer (MSE: 0.20403, MAE: 0.35646). Right figure shows the result of the Autoformer trained with Lipschitz Regularizer (A 5, MSE: 0818370, MAE: 0.33714).
GreundTuuth
Prediction
My Wloh Mobs WM0WyooWh
-1
Prediciton
Figure 13: Univariate forecasting example of Autoformer on the ECL dataset with the prediction window size set to 720. Left figure shows the result the original Autoformer (MSE: 0.80028, MAE: 0.66756). Right figure shows the result of the Autoformer trained with Lipschitz Regularizer (A 5, MSE: 0.91144, MAE: 0.69692).
Multivariate Forecasting We show four examples of multivariate forecasting Figure 14 15
16 17 Figure 14 16 the where the Lipschitz Regularizer improves the performance of the model, while Figure 15 and show two negative cases. For Figure [3] the sudden change in the input is abnormal, which influences the model with Lipschitz Regularizer more than the original
17

---

Published as a conference paper at ICLR 2023
model. As for Figure 17 both models did not capture the pattern in the input data. This figure shows that Lipschitz Regularizer makes the output slightly more continuous, therefore causing the increased error.
0.
GroandPuthh Trsiicionn
Croundi uth Thodiction
-0.2
0.0
-0.4
-044
-06-
AMN
08-
-1.0
-1.
-1.4
100 420
$00
1000
1000
203
420
$50
Figure 14: Multivariate forecasting example of Transformer the ETTml dataset with the predic- tion window size set to 672. Left figure shows the result of the original Transformer (MSE: 1.17478, MAE: 0.83787). Right figure shows the result of the Transformer trained with Lipschitz Regularizer (A = 1, MSE: 0.92590, MAE: 0.73367).
Groandruuhh Prai o1
0.35
0.55
0.05
0.65
-0.05
Geunt Buth
Redd on
42
40
Figure 15: Multivariate forecasting example of Transformer on the Weather dataset with the predic- tion window size set to 24. Left figure shows the result of the original Transformer (MSE: 0.14897, MAE: 0.23293), Right figure shows the result of the Transformer trained with Lipschitz Regularizer (A 1, MSE: 0.21319, MAE: 0.28302).
Groundiluut
2.0
220
320
400
200
600
7OC
200
2.0 220
320
400
200
600
700
502
Figure 16: Multivariate forecasting example of Autoformer the ECL dataset with the prediction window size set to 720. Left figure shows the result of the original Autoformer (MSE: 0,28641, MAE: 0.37610). Right figure shows the result of the Autoformer trained with Lipschitz Regularizer (A 1, MSE: 0.25691, MAE: 0.36375).
18

---

Published as a conference paper at ICLR 2023
-1.0
-100
redicction
prediction
-12
de My Mn Rooo po
-1.6
-18
-1.0
200
200
300
400
300
300
710
802
200
700
300
400
300
5000
710
D00
Figure 17: Multivariate forecasting example of Autoformer on the ETThl dataset with the prediction window size set to 720. Left figure shows the result of the original Autoformer (MSE: 0.47690, MAE: 0.49172). Right figure shows the result of the Autoformer trained with Lipschitz Regularizer (A 1, MSE: 0.50977, MAF: 0.50879).
D FINE-TUNE SWIN TRANSFORMER WITH LIPSCHITZ REGULARIZER
In this section, we investigate whether the proposed Lipschitz Regularizer can be used to improve large pre-trained models by fine-tuning its embedding layer down-stream tasks. Here, we use a regular setting that the model is pre-trained on large image dataset, and then fine-tuned it on the down-stream image classification task. Moreover, considering that Transformer-based models have shown great power in computer vision domains (Dosovitskiy et al. [020) Liu et al.] 2021), and these models process images by splitting them into patches and feeding the model a sequence
of patch embeddings, we choose a typical model, i.c., Swin Transformer (Liu et al. 2021). for experiments in this section.
Generally, because input tokens of the Transformer are local image patches, they tend to be con- tinuous, which might not match the preference of the Transformer model. Therefore, we apply the Lipschitz Regularizer to make them more discrete. Specifically, we use the Lipschitz Regularizer for outputs of the embedding layer, and change the loss as follows:
Cly. 0) Lswin 9) Al))
(10)
where is the ground-truth, and ý is the output of the Swin Transformer. is the output of the embedding layer, and Lswin is the original loss of the Swin Transformer. À controls the magnitude of the Lipschitz Regularizer.
We use the pre-trained Swin Transformer on ImageNe and fine-tune on the image classification task with the Beans dataset (Lab) 2020), containing bean leaf images of diseased and healthy leaves. We only fine-tune the embedding layer and the last linear layer, and freeze other parts of the model.
Table 6: Results of fine-tuning Swin Transformer with Lipschitz Regularizer on an image classifi- cation task. Test accuracy with different À reported.
0
-1
-5
Test Accuracy 0.9398
0.9549
0.9248
0.9098
0.8947
We show the validation accuracy during training in Figure 18 and list the testing accuracy in Table By setting À to zero, we obtain the result of the baseline. We can see that the model performance can be improved by the Lipschitz Regularizer, showing great potential for changing data continuity the fine-tuning setting and vision tasks. Besides, results also show that setting À to positive values (i.e., and 1) benefits model performance, while setting to negative values (i.e., -5 and -1) degenerates the performance. This verifies our intuition that the Transformer-based model prefers more discrete inputs.
"The pre-trained model is acquired http:///wggggiiiooooooooooooooom
19

---

Published as a conference paper at ICLR 2023
1.00
0.95
Val Accuracy
0.90
0.85
À -
0.80
À = -1
À = 0
0.75
À = 1
À 5
0.70
40
80
100
Epochs
Figure 18: Validation accuracy of fine-tuning Swin Transformer in each epoch with different values
of X.
E APPLY LIPSCHITZ REGULARIZER TO THE SPEECH CLASSIFICATION TASK
In this section, we show the effectiveness of the Lipschitz Regularizer on the speech classification task by applying it to a Transformer-base model. As we discussed in & [ the Transformer-batt model prefers discrete inputs. However, voice signals are highly continuous since they are sampled from a continuous physical process with a high sample rate. This inspires us to use the Lipschitz Regularizer make it more discrete and therefore more preferable for Transformer-basee models.
Specifically, following the settings in Gu et al. (202), ), we investigate the Performer model (Choro- manski et al. 2020) the Speech Commands (SC) dataset (Warden 2018). We test the Performer model on two versions of SC. One is MFCC, where the sequence is pre-processed into standard MFCC features (length 161). Another is Raw, which contains unprocessed signals (length 16000). The Lipschitz Regularizer is applied after the embedding layer, and changes the loss as follows:
Cly. 0. Leer
XLLip()),
(11)
where is the ground-truth, and ÿ is the output of the Performer model. i the output of the embedding layer, and Cer is the original loss of the Performer model. controls the magnitude of the Lipschitz Regularizer.
Table 7: Results of the Performer model and Lipschitz Regularizer on the Speech Commands dataset. Test accuracy for MFCC and Raw speech data is reported.
3
MFCC
80.63
81.13
83.21
Raw
30.89
35.72
33.86
Results are shown in Table 7 The column with À = 0 represents the baseline. The performance of the Performer model is increased by the Lipschitz Regularizer, which further verifies our claim that Transformer-bast models prefer discrete inputs.
F NEURAL ODE WITH LIPSCHITZ REGULARIZER
In this section, we apply the Lipschitz Regularizer to the Neural ODE model (Chen et al. 2018) to the effect of the regularizer the model. Similar the state-space model, Neural ODE is also a continuous- time model, which treats the input as samples from a continuous function. We expect that the Neural ODE will perform better when the input is more continuous.
We adopt the experiment fitting time series using the latent ODE in its original paper (Chen et al. 2018). Essentially, the neural network is generative latent function time-series model, predicting
20

---

Published as a conference paper at ICLR 2023
the solution an ODE, and the imput data of this experiment is sampled from randomly generated ODE with the same generation process as Chen et al. (2018). The network is a variational autoen- coder, which consists of an RNN encoder and a Neural ODE decoder. To alter the continuity of the input to the Neural ODE, we directly apply Lipschitz. Regularizer to the output of the RNN encoder as follows:
Cly 9. 1) = LODE(Y, 0) ALLip (?)
(12)
where is the ground-truth, and ğ is the output of the model. is the output of the RNN encoder, and LODE is the original loss of the Neural ODE. controls the magnitude the Lipschitz Regularizer.
The MSE during training is shown in Figure 19
We can observe that Neural ODE performs better when data is more continuous. Predictions of independent runs are presented in Figure[0) We can see that the model has better fitting results when we use Lipschitz Regularizer to make inputs more continuous.
400
A=1
A=
300
MSE
200
100
0
2500
5000
7500
10000 12500 15000 17500 20000 Epochs
Figure 19: The MSE of the Neural ODE model with Lipschitz Regularizer during training.
G MATHEMATICAL DERIVATIONS
G.1 DERIVATION OF EQUATION
n-1
f(t+1) - f(ti)) 2
[1i41 - xz)) = Σ
li+1 - li
1=0
df(t))
dl
dt
(2mig)² (2(E)(-dg)
(13)
4m E2 (2))(E)
2 / $ 19 de
C
4m CIE
G.2 CONTINUITY AND THE S4 MODEL
Proposition G.I. Suppose J1, R+ R are two differentiable functions of input sequences, and their Lipschitz constant Lıf, and Liz- The HiPPO matrix with scaled Legendre measure (LegS) is denoted as HiPPO-LegS. Let the error of the HiPPO-LegS projection of f1, f2 at time t be
(t), (t),,, respectively. Let 81 tL,, 82(t) tL For any time t, suppose L S L we have & (1) = (1), (2(1) (()(1)), and 8n (1) (2))..
21

---

Published as a conference paper at ICLR 2023
Proof. By Gu al. (2020) Proposition 6), the LegS measure, which uniformly weighs all his- tory, has the following property. Suppose the HiPPO-LegS projection for the target function f(t) at time is p(o) = proje (f), then the error (t) = || - - = O /VN), where L the Lipschitz constant ft), and the maximum polynomial degree N - So, we have (t) = (() (t) (t)( and (t) < 62(t). Therefore, the error rate of HiPPO- LegS projection decreases with the Lipschitz constant, so with smaller Lipschitz constant, we expect smaller projection error.
G.3 CONTINUITY AND THE RELU NETWORK
Proposition G.2. Suppose there are two ReLU networks ge,, 99, with identical architecture, and the
Lipschitz constant of them are L1, Lz respectively. Let h1(E) = LI Ia/l|[]]*+1 h2(E) L2/ [
where & is the frequency, and (e( is the Fourier component of go- Suppose L1 S Lz, we have Go, (5) ( 90. (5) (E)((E)), and (g) h2(E)-
Proof. By Rahaman et al. component is,
2019 Theorem 1), for a ReLU network do with parameter 0, its Fourier
(o(E) G (0, ||
(14)
where the numerator (0,-) Rd 7 C is bounded by O(Lg). So, we have (E) (E)) (E) (()2(E)). and (E) S h2(E). Therefore, with smaller Lipschitz constant, we expect smaller (E)
22

---

Published as a conference paper at ICLR 2023
(a) Apply Lipschitz Regularizer with À 1.
22
15
(b) The original Neural ODE model (A = 0).
36 45 15 15
14917
43 28 21
32 43 00
10-
45
11 21 00
12
13
20 on 16
Apply Lipschitz Regularizer with ) 11
10
Figure 20: Results generative latent function with Neural ODE and Lipschitz Regularizer. Points are input sequences, and the color of points indicates their time. The blue line is the ground truth and the orange line is the prediction. The gap between points and the orange line indicates training loss, and the gap between the orange line and the blue line indicates the test loss.
23


Reviews:
[{"review_id": "zwbbRGBNu6d", "reviewer": "Reviewer_cBYY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Lipschitz regulariser to adjust to the continuity of certain type of data.", "summary_of_the_review": "Paper seems interesting but I cannot evaluate its relevance to the field.", "strength_and_weaknesses": "Strengths:\n- The methodology seems to be novel.\n- Experimental evaluation looks extensive and quite thorough with different models and data types.\n\nWeaknesses:\n- The regulariser seems to be quite simple, it is just the square difference between two sequence values.\n- Adding the regulariser did not always improve results but made them worse. I would have liked to see a detailed analysis on those situations, why that happens. E.g. For Pathfinder in table 1 and the quite a few cases (especially multivariate) in table 2.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a6GzQO0-y9", "reviewer": "Reviewer_EzfA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes the impact of data continuity on deep learning models, and propose Lipschitz regularizer which can improve performance of different models based on their preferred data continuity.", "summary_of_the_review": "Based on  data continuity, the authors analyze the continuity preference of different models, and propose a simple yet effective regularizer to improve performance of different models. I lean towards acceptance of the paper.", "strength_and_weaknesses": "The paper has a good motivation, which is reasonable and straightforward;\n\nThe proposed regularizer is simple and general, can be used to improve performance of different models;\n\nGood experimental results are obtained across different datasets on different models;\n\nSome theoretical analysis and clear explanations are provided;\n\nSome experiments can be added to strengthen the paper. Specifically, instead of training from scratch, I'm curious of whether we can use the proposed method to improve large pre-trained models by fine-tuning its embedding layer and freeze other layers on down-stream tasks, instead of fine-tuning the whole model or several selected layers. If the claims are correct, the proposed method should lead to good results. I think this experiment is very important, given the fact that many pre-trained transformer-like models are very important in real-world applications.\n\nIs the finding in Figure 1 always holds across different dimensionalities?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NNcvK3jUQ55", "reviewer": "Reviewer_Tz2y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper comprehensively investigates the continuity of sequential data from both theoretical and empirical perspectives. The authors observe that different model structures may prefer data with different continuity. Then, they give a theoretical analysis in both time and frequency domains. To further improve sequence modeling, they propose Lipschitz Regularizer which can flexibly adjusts data continuity based on the model preferences. Experimental results on different model structures and tasks demonstrate the effectiveness of Lipschitz Regularizer for many deep models in sequence modeling.", "summary_of_the_review": "In my view, this paper is of high quality. I will surely recommend acceptance. But if other reviewers raise serious problems in this paper, I will be still open to change my mind.", "strength_and_weaknesses": "Strengths:\n\n1) In my view, this paper is both comprehensive and deep. Now, there are many effective model structures such as Transformer and state space models for sequence modeling. But few works try to investigate the reasons why they work well in specific tasks. This paper starts from continuity, which is an intrinsic property of data, and clearly reveals the relationship between task data and model structures with theoretical supports. The authors provide meaningful insights into different tasks and model designs, which may guide the future research of sequence modeling.\n\n2) In addition to analyzing the phenomenon, the authors also propose a simple and effective method by adjusting the data continuity according to model preferences. Empirical results show its superior performance on challenging tasks like LRA, which are convincing for me.\n\n3) This paper is well-organized and easy to follow. I really enjoy reading this paper and learn a lot from it.\n\nWeaknesses:\n\n1) The authors only conduct experiments on LRA and time-series forecasting tasks in the analysis of Section 4. I wonder whether the authors try to use Lipschitz Regularizer in NLP or audio tasks such as language modeling because they are also attractive sequence modeling tasks.\n\n2) Typo: Preformer –> Performer in line 7 of Section 2; Removing ) in line 1 of Section 5.1.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Jfc15wWOKib", "reviewer": "Reviewer_L7Qs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work studies the connection between sequence data continuity with the performance of deep sequential models and proposes a regularizer that adjusts the Lipschitz continuity of the model to improve its performance. It analyzes the property of the regularize from both time and frequency domains. Experiment results conform with the analysis and show the effectiveness of the proposed regularizer.", "summary_of_the_review": "The perspective of studying the continuity property of sequential model is interesting and novel. The proposed Lipschitz regularizer is well supported by theoretical analysis and strong performance in experiment results. However, I'm concerned that the set of models studied in the paper, despite being exemplary, are limited.", "strength_and_weaknesses": "Strength:\n\n1. The main idea of the work is simple and clear. It is strongly motivated by empirical observations.\n2. The analysis from both the time and frequency perspectives provides deeper insight and a solid basis for the proposed Lipschitz continuity regularizer.\n3. The experiment results are coherent with the motivation and theoretical analysis.\n\n\nWeakness:\n\nThe work only studied S4, informer, and ReLU networks. The theoretical results on the Lipschitz continuity property of S4 and ReLU network which in turn support the use of Lipschitz regularizer are model-specific. I’m concerned about the general applicability of the proposed Lipschitz regularizer to similar sequential models without more general theoretical or experiment results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%