Predicting subscriber usage: Analyzing multi-dimensional time-series using Convolutional Neural Networks
Authors: Benjamin Azaria
Lee-Ad Gottlieb
Venue: ICLR 2022 Submitted (2022)
Keywords: Convolutional Neural Network
time series
usage prediction

Abstract: Companies operating under the subscription model typically invest significant resources attempting to predict customer's feature usage. These predictions can be used to fuel growth: It may allow these companies to target individual customers -- for example to convert non-paying consumers to begin paying for for enhanced services -- or to identify customers not maximizing their subscription product.
This assistance can avoid an increase in the churn rate, and for some consumers may increase their usage.

In this work, we develop a deep learning model to predict the product usage of a given consumer, based on historical usage. We adapt a Convolutional Neural Network to time-series data followed by Auxiliary Output, and demonstrate that this enhanced model effectively predicts future change in usage.

Under review as a conference paper at ICLR 2022
PREDICTING SUBSCRIBER USAGE: ANALYZING MULTI- DIMENSIONAL TIME-SERIES USING CONVOLUTIONAL NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Companies operating under the subscription model typically invest significant re- sources attempting to predict customers' future usage. These predictions can be used to fuel growth: Companies example to convert non-paying consumers to begin paying for enhanced services or to identify customers not maximizing their subscription product. This al- low the company to avoid an increase in the churn rate, and to increase the usage of some customers.
them to target individual customers for
this work, we develop a deep learning model to predict the product usage of a given consumer, based historical usage. We adapt Convolutional Neural Net- work with auxiliary input to time-series data, and demonstrate that this enhanced model effectively predicts future change in usage.
INTRODUCTION
Historically, companies interested in usage prediction have retained the services of expert analysts. These experts may examine past behavior multiple features, estimate future usage in the short- term, and translate this knowledge into actionable tasks. Yet for companies which boast thousands of customers and have an increased need for accurate research forecasting, this non-scalable approach quickly becomes infeasible. Hence, the development of an automated model is an obvious necessity (Miller al., 2021).
Future usage prediction is especially crucial for subscriber- companies, for which churn rate has especially sigmificant impact on profitability. crucial as well for companies valuated based on Annual Recurring Revenue (ARR) which reflects income derived from customers. For these companies, future usage prediction is prerequisite for predicting the future valuation of the com- pany.
Usage data sets are often represented as multi-dimensional time-series data, and indeed we have at our disposal two large multi-dimensional time-series, which constitute high quality labeled data sets, each produced in a controlled environment. However, successfully predicting multi siltiidimensional time-series values is itself non-trivial task (Cerqueira al., 2019). Indeed, automated multi- dimensional time series prediction a relatively new field, and although different strategies have been proposed in the literature, eurrently there seems to be consensus approach to the problem. Here, machine learning and deep learning algorithms are typically used for time-series data classifi- cation, which produces binary decision value (for example, whether the customer usage will grow or not). The problem of regression, which produces a continuous value describing customer growth the extent of customer growth, clearly much more challenging accurate accurately predict.
Our contribution. For the regression problem of multi-dimennional time-series prediction, we propose to utilize both common machine learning techniques and novel methods. Specifically, we shall leverage one-dimensional convolutional neural network (CNN), in conjunction with auxiliary output, to predict future utilization based on historical usage. In this model, an auxiliary layer pro- duces prediction for the future in the short-term. This prediction is fed into the next auxiliary layer, which produces a prediction for future prediction for a slightly later period, until the final auxiliary layer produces prediction for the desired longer-term period. We demonstrate experimentally that

---

Under review as a conference paper at ICLR 2022
this enhanced model effectively predicts future usage, and gives superior results when compared to other commonly utilized models in the field.
We present the data sets in Section 3.1, describe our learning model in detail in Section 4, and demonstrate promising empirical results in Section 5. In Section 6, we give concluding remarks and discuss future work.
1.1 RELATED WORK
Here we give an overview of some traditional machine learning approaches for time-series predic- tion (for both classification and regression), as well as more recent CNN-base approaches for this problem.
Traditional learning approaches. Tan et al. (2020) aimed to learn the relationship between a time-series and continuous scalar variable, by identifying the right coefficients best predict the future variable. Time-series regression models have been applied to various problems such as predicting annual rainfall from observed temperature, or predicting fat content in meat from near-infrared spectrum Goldsmith & Scheipl (2014). That study compared different time series regression models, such as gradient boosting and random forest. It demonstrated that functional regression models usually require an in-depth understanding of the data in order to apply the right basis function fit the model.
In Pimentel et al. (2015) it was shown that many cutting-edge regression methods fail to distinguish between periods of high and low quality data, and further are not able to generalise well other data-sets. They attempted to estimate respiratory rates (RR) from a photoplethysmogram (PPG) time-series using sliding window. Producing an estimation for RR per window consists of four key components: (I) extracting respiratory signals; (2) estimating respiratory rates; (3) fusing the estimates and (4) executing quality assessments. The authors applied a Gaussian process regression framework extract RR from the different sources of modulation in the PPG signal, and also tried fitting multiple auto-regressive models to the extracted respiratory signals.
In Zheng et al. (2014), it was shown that k-Nearest Neighbor (KNN) combined with Dynamic Time Warping (DTW) achieves state of the art performance for classification. However, although the time complexity of I-NN with DTW grows linearly with the data size, their performance dependent on the quality of hand-crafted features, and these are not scalable.
In Samsudin et al. (2010), Support Vector Machine (SVM) were compared with Artificial Neural Network ANN on five distinct one-dimensional time-series for the task of regression. They uti- lized grid-scarch technique to identify the optimal cross validation while training. They indicated that SVM was more efficient than ANN on single- dimensional time-series series.
in the data, and used 10-fold
CNN-based approaches. In Yang et al. (2015) it was shown that classification using CNN multi-channel human activity recognition time-series outperformed all other algorithms tested. They proposed systematic feature learning method, adopting CNN automate feature learning from raw inputs. In deep architecture, the learned features are the high level abstract representations of low level raw time series signals. By leveraging the labelled information through supervised learning, the learned features are presented with more discriminatory power. LeCun et al. (1995) suggested a potentially more interesting scheme (for classification as well). by eliminating the feature extractor, and feeding the network with raw inputs (e.g. normalized images). This relies on back-propagation to turn the first few layers of the neural net into an appropriate feature extractor.
According to Okita & Inoue (2017), the combination of RNN (such as LSTM) and CNN for clas- sification problems such as activity recognition yielded improved accuracy on the range of 27% to 43% when compared to the other popular algorithms. Wang et al. (2019) also considered classi- fication for Human Action Recognition (HAR), and evaluated multiple models, including LSTM, CNN, Random Forest, J48 Decision Tree, and SVM. They found that the superior model was Bidir- LSTM-CNN, combination of the outputs of a bidirectional LSTM and CNN connected to three fully connected hidden layers.
2

---

Under review as a conference paper at ICLR 2022
There has been significant recent interest in analyzing time-series in order to effectively allocate company resources. Snow (2020) undertook an extensive review of popular algorithmic solutions for regression problems, such Facebook's s Prophet. This tool is given list of dated events, and takes into consideration holidays, weekends and special occasions, before outputting predictions for future usage. Amazon's Gluon'TSS and Microsoft's ForecastTCN attempt to do the same. This problem has also received attention from academia and independent developers such as N-Beats, Auto-Arima, and TBATS. One caveat is that all of these algorithm deal with only one-dimensional time-series input data.
Using CNN to address multivariate time-series regression has became increasingly popular. Mode & Hoque (2020) compared CNN, LSTM, and Gated Recurrent Unit, showing that the results of the CNN model were more transferable when compared to the others. According to experimental analysis undertaken by Antsfeld et al. (2020), utilizing LSTM layers along with additional convolu- tional layers, for regression can provide significant boost in forecasting performance. And Mehtab et al. (2020) compared three models of CNN and four models of LSTM (regression problem), and concluded that the execution time of the CNN models were 3-10 times faster than the others, while the best accuracy was achieved by of the CNN models.
PROBLEM STATEMENT
We are given a set of multi-dimensional time-series, each with continuous label-vector representing usage. Our goal is predict the label of a query time-series, that its future increase or decrease in usage. There is an imbalance in this model, in that the maximum possible decrease is bounded by 100%, while the maximum possible increase is unbounded. To address this, let Pa be the average usage in the observed Lime-frame and PB the average usage in the time-frame to be predicted. Setting 8 Pe 1, we define our usage increase function as:
PA
[ 88
if 0
f())
145
(1)
otherwise
The above function maps the increase decrease tending to -1 imply significant decrease in usage, values close to 0 imply no change in usage, and values close to I imply extreme growth. (See also its use Section 3.1 below.)
real value in the range [-1, 1]. Values
3 DATA AND PROCESSING
Before presenting the model, we describe our data sets and their select features, and detail our data processing. The data processing includes the creation modified lahel-vector for the time-series, which will be necessary for running the model.
3.1 DATA SETS AND FEATURES
Our main data-set is a proprietary set provided by Cloudinary, which is a softwarreas----ervicc (SAAS) company providing media end-to-end solutions in the cloud. The data contains daily records of the media transformations undertaken by each customer, as well delivery over the network, recorded as delivery requests and utilized bandwidth. Another important feature is the total data storage used by each customer through uploading media to Cloudinary's 's distributed shards; this includes original assets and assets derived via transformations. The data-set also records each cus- tomer's subscription plan these can be standard or custom, and monthly or annual. Note that customer in this data set is often company consisting of many distinct users.
We also utilized the public Bike-Share Usage in London and Taipei Network data set (link), which spatio-temporal urban transport data set from network of bike stations. London and Taipei are both very large cities with large bike-share systems. At each location where bikes are picked up, the individual bikes tracked, that each rental generates digital footprint: which bike, from where, to where, for how long, at what date and time, and by whom. The data collects events from 2016 6 until the first half of 2020.
3

---

Under review as a conference paper at ICLR 2022
For both data sets, care was taken to determine which features should be extracted. We avoided sparse features, as well as pairs of correlated features, or features unrelated to usage. A detailed list of features extracted from the Cloudinary data set is found in the Appendix. From the Bike Sharing data set we extracted the following fields: Start date, bike ID, duration, and startStation ID. We have also filtered out customers who are outliers, meaning their usage of the product is atypically small or large.
3.2 DATA PROCESSING
For each data-set we computed the time-series associated with each individual user, and further spawned new user time-series at every one-week interval. While this could cause individual cus- tomers to appear as multiple time-series in the data set, we ensured that the same customer can appear (possibly multiple times) in only one of the train, validation or test groups.
Having segmented the data into individual time-series, we normalize each one: We first log-scale the time-series features, and then normalize each feature twice:
1. Self-comparison: Each customer feature Ij is mapped to mean and a is the variance of this feature over this time-series.
[ where j is the
2. Global comparison: Each customer feature 2j is mapped to w/ mean and a is the variance of this feature over this time-series.
where is the
Labels. It remains calculate the usage label, as previously diseussed in Section 2. We preprocess the Cloudinary data set as follows: Recall that PA is the average usage in the observation period, which we now define be the first ten weeks of the series, that is days 1-70. For the i-th time-series, we create a label vector Yi as follows: For entry Mi.j. we define Pn.j to be the average usage in the Pa.; and set Vi. f(5,) (as defined
30 day period ending at day 70+ We then compute 6j PA in Section 2), and obtain the distribution of our label. See Figures 1 and 2 for illustration of the computation of the label for j = 60 and the distribution of these labels in the Cloudinary data set.
The Bike-share data set is preprocessed in the same way, except that we take PB.j to be the average usage in the 21 day period (instead of 30 day period) ending at day 70 + j. Our choice for this period length reflects the fact that the behavior captured in the Bike-share data set is strongly influ- enced by the weekly cycle, as typically usage is determined by the week day. While the Cloudinary data set is also influenced by a weekly cycle, the standard approach in the corporate world is to compute usage in monthly blocks, making this the more appropriate measure for this data set.
Period usage during learning period
Period usage 31-60 days after learning period
Day
Day 70
Day 101
Day 130
Figure 1: Label of data
4 CNN CONSTRUCTION
We model our data using Convolutional Neural Networks. These are known to return superior results for time-series classification when compared to other models (Canizo, 2019; Okita & Inoue, 2017; Yang et al., 2015), and when combined with auxiliary output known to give predictions with 50% improved accuracy compared to state of the art models (Zhang et al., 2018).
Our exact model depends on the period to be predicted. In what follows, we initially describe the model for the Cloudinary data set and predicting future usage of 60 days, meaning predicting the average usage in the 30-day period including days -1-60 after the initial 70 days of the time-series. Hence we stipulated above (Sections 2,3.2) that the time-series in the training set are labelled with a scalaria in [-1, 1) representing the average of this period.
4

---

Under review as a conference paper at ICLR 2022
Frequency
50k
Label
Figure 2: Label frequency
Considering a single feature in sequence (as in Figure 3), the I-dimensional convolution uses kernel that weights adjacent observations, and by using pooling and second convolution, enables the model learn the trend of this specific feature. In our case we have multiple features, so we combine their output into fully connected layer.
Figure 3: One dimensional convolution on a single feature
The structure of model is follows: It is composed of convolutions (with 40 and 50 filters, respectively). followed by fully connected (dense) layer of 150 neurons, using RELU as the activation. (see Figure 4). Following this unit are an additional six fully connected layers of auxiliary output (see Figure 5; the first auxiliary layer is also illustrated in Figure 4). The goal of each auxiliary layer predict futurre usage the short term, and to pass this prediction to the next auxiliary layer. This next auxiliary layer will predict future usage for slightly later period, until the final auxiliary layer yields long-term prediction. In particular, we view each layer as predicting an additional ten days in advance, leading to final prediction of sixty days. For each of the six auxiliary layers we utilize a dropout of 50% to avoid over-fitting.
We used grid search on the validation data set to determine the optimal number of convolutions, number of filters each convolution, size of the fully connected layer, and the drop-out rate; the best results determined the parameters adopted in the above model. We also tried reducing the drop-out rate for later auxiliary level, but found that this did not yield appreciably better results. For computing the optimal number of auxiliary layers, added layers until the improvement was negligible.
To each auxiliary layer E [1, 2, 4,5,6) 6) associate mean-squared error (MSE) loss func- tion (as Figure 5). Recall that the motivation behind auxiliary layer was to produce a future prediction for 10k days ahead. Then
loss:
Σ (Vi tanh(Gr.uuu
(2)
5

---

Under review as a conference paper at ICLR 2022
4087axx0
4007Bxs
50840x5
5049403
1x150
1x30
Convolution
Max-Pool
Convolution
Max-Pool
Dense
Figure 4: The first part of the model includes two layers of 1 1-dimensional CNN followed by a fully connected layer. At the far right of the figure is the first auxiliary layer.
loss
loss_
loss
losa_6
Figure 5: The second part of the model includes 6 auxiliary layers.
where Vi, is the true label of future day 10k time-series (as defined in Section 3.2), and Vi, denotes the k-th auxiliary layer's prediction for this value. The tanh function is used restrict the range of Dr, 10s to [-1, 1]. while having only minor effect on values already within this range.
The final model attempts to minimize sum of the six individual loss functions. However, because each coordinate in label-vector y/s is determined by averaging time-series entries over the previous would overvalue the time-series entries in the earlier part of the evalu-
30-day period, simple ation period, and undervalue the later time-series entries that period. For example, the time-series value of day 31 after the observation period figures into all labels yi,; for j E [31, 60), while the time series value of day 60 only figures only into the value of yi.60 in this range. To address this issue, take weighted instead, and the goal becomes minimize
1
loss 8y.
joint-loss
k=1
for moderately decreasing values dg chosen via experimentation.

---

Under review as a conference paper at ICLR 2022
We stop the training after encountering 20 consecutive epochs without improvement in validation loss, as presented in the following algorithm:
best cos__validaiion.ooo
cectaainnnngepppclee t- 20;
while remaining-epochs
do Train model one epoch and calculate current validation loss score; ; if best validation loss score 2 current validation loss score then
best dist_validation Joss_ossssssss _ccree +- ourrent_ualidation..no loss_scor
cennainingeepeelee 20;
else
ceatuiiinng.ppchhee t- remaining epochs 1;
end end
Algorithm 1: Stopping the training process
Extension to other prediction periods. The above description gives our model for 60-day pre- diction. this model, each auxiliary layer corresponded to a forward prediction days. Hence, to adapt this model to different forward value predictions specifically, 30 day and 90 day predic- tions - we simply modify the model to contain three or nine auxiliary layers, respectively, instead of the original six.
For the Bike-share data set, the experiment below attempts a forward prediction of nine weeks. Recall that for this data view each auxiliary layer as giving forward prediction of one week (instead of 10 days). It follows that for forward prediction of nine weeks can a model of nine auxiliary layers.
average periods of 21 days (instead of 30), and therefore will
EXPERIMENTS
We run the following common machine learning models on our data sets: Random forest, Fully Connected Artificial Neural Network (ANN), Recurring Neural Network (RNN), and our augmented Conventional Neural Network (CNN) model. For both data sets, we divided the data as follows: 70% of the customers in the training set, 20% of the customers in the validation set, and 10% of the customers the test set.
5.1 CLOUDINARY DATA SET
Here attempted to predict future usage of 30 days. 60 days and 90 days (meaning the respective averages of the three 30-day periods [1, 30), [31, 60), [61, 90)), and reported the success rates for each period in designated table. The computed error values for each model is
(4)
M F.,, [>d
where Vi, is the true label (of Section 3.2), Vi, the output prediction of the model, 1 is the indica- tor function, and j takes values in (30, 60, 90). The value d a threshold on the difference between these two models; if the difference exceeds the threshold. the prediction is taken to be error, and otherwise the prediction is viewed as successful. This is a common regression measure, and es- pecially appropriate for setting due to commercial rationale behind the model: Companies seek to maximize the customers correctly served. We tried values d E {0.05, 0.1, 0.15, 0.2, 0.25, 0.3) Comparisons for 30, 60, and 90 days presented in Tables 1, and 3.
The results show that our method almost always gives improved results when compared to the others. This is especially true the forward prediction period is increased, making predition more difficult.

---

Under review as a conference paper at ICLR 2022
Value of d
Random Forest
ANN
RNN (LSTM)
Our model (CNN)
5%
21.62%
21.52%
26.66%
28.33 %
10%
39.78%
43.22%
47.51%
50.40
15%
55.56%
59.25%
63.41%
66.40 %
20%
67.20%
70.25%
74.80%
76.93%
25%
76.29%
78.21%
82.17%
84.09
30%
82.16%
84.37%
87.04%
88.90 %
Table
1: Performance of predicting behavior at 30 days in Cloudinary data
Value of
Random Forest
ANN
RNN (LSTM)
Our model (CNN)
5%
16.42%
18.01%
18.27%
19.09
10%
32.73%
34.66%
35.52%
36.17% %
15%
47.10%
48.45%
50.96
50.88%
20%
59.77%
59.92%
63.29%
63.73%
25%
69.44%
70.77%
73.34%
74.57 %
30%
77.92%
80.13%
80.84%
81.71 %
Table Performance of predicting behavior 60 days Cloudinary data
Value of d
Random Forest
ANN
RNN (LSTM)
Our model (CNN)
5%
15.20%
15.68%
14.27%
17.09%
10%
28.37%
29.95%
28.27%
31.56 %
15%
40.16%
42.31%
41.03%
45.42 %
20%
50.42%
52.87%
51.40%
55.90 %
25%
59.54%
61.52%
60.81%
63.65 %
30%
67.16%
68.06%
68.40%
70.28 %
Table 3: Performance of predicting behavior at 90 days in Cloudinary data
8

---

Under review as a conference paper at ICLR 2022
Value of
Random Forest
ANN
RNN (LSTM)
Our model (CNN)
1%
35.14%
45.68%
53.87%
59.50%
2.5%
74.79%
82.31%
88.22%
92.20 %
5%
82. 14%
87.41%
95.81%
98,49 %
7.5%
95.03%
96.52%
98.61%
99.18% %
10%
97.87%
98.06%
99.47%
99.86 %
Table 4: Performance of predicting behavior at weeks in Bike-share data
5.2 BIKE RENTAL DATA-SET
For this data set, we attempted to predict the total duration of usage in each station. As be- fore the training period was 10 weeks (weeks 1-10), but now attempt to predict usage at weeks (week 19). As above, the computed enror is taken to be 1 Σi=o 1 1n.. -b,.s- but with d E {0.01, 0.025, 0.05, 0.075, 0.1). The lower values of d taken here in comparison to those taken for the Cloudinary data set are due to this data set being easier to predict indeed, the practices of individuals recorded in the bike-share data set simpler and more predictable than those of the clients in the Cloudinary data set, which are usually large companies of many users. This is also the reason sufficed here with only single experiment, utilizing the most advances (nine layer) model. The results are reported in Table 4.
While all methods performed well for high values of d. our method returned better results. For low values of d our method is appreciably better.
6 CONCLUSIONS
We attempted to predict future customer behavior using multi-dimensions time-series our input. We utilized two data-sets: The first was the Cloudinary data set, where customer are often large companies. Our second data set was public data set containing behavior of individuals using the Bike-share in London and Taipei. We suggested a CNN-based model with auxiliary output, and compared this model to other standard approaches. Our approach achieved the best results in almost all of the comparisons undertaken. Our model performed better the Bike-share data, and we believe this was because individual behavior easier to predict than that of large companies.
We note that we also ran our model on less structured data, such as the Exchange rate per country data set (link). Our model returned poor results for this set, and we believe this is because its behavior erratic, see Figure 6.
20-
Figure 6: Exchange rate per country. Our model was unable to predict the erratic behaviour of this data set.
Future research. One shortcoming of our model is that it must be modified and re-trained for different forward prediction periods. Some other models (such RNN), do not have this issue. In future work, we would like to modify the structure of our model to allow changing the forward prediction period without needing retrain the model.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Leonid Antsfeld, Boris Chidlovskii, and Dmitrii Borisov. Magnetic sensor based indoor positioning by multi-channel deep regression. In Proceedings of the 18th Conference on Embedded Net- worked Sensor Systems, pp. 707-708, 2020.
Mikel Canizo. Multi-head cnn-rnn for multi-time series anomaly detection: An industrial case study. 363:246-260, 2019.
Vitor Cerqueira, Luis Torgo, and Igor Mozetic. Evaluating time series forecasting models: An empirical study on performance estimation methods. arXiv preprint arXiv:1905. //74, 2019.
Jeff Goldsmith and Fabian Scheipl. Estimator selection and combination in gression. Computational Statistics & Data Analysis, 70:362-372, 2014.
re-
Yann LeCun, Yoshua Bengio, al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, (16) 1995.
Sidra Mehtab, Jaydip Sen, and Subhasis Dasgupta. Analysis and forecasting of financial time series using and lstm- based deep learning models. arXiv preprint arXiv:201 1,08011, 2020.
Allan Miller, Ben Vonwiller, and Peter Weed. ing on customer success to drive growth.
Grow fast die slow: Focus-
https://www.mckiit com/ cust -ot--oon-ccuuuuueeee omer- success- -uccess-ooddivv-grooo
grow- orrst-or-aee die- slow- focusing- on- on- 2021. Accessed: 2021-10-5.
Gautam Raj Mode and Khaza Anuarul Hoque. Adversarial examples in deep learning for multivari- ate time series regression. arXiv preprint arXiv:2009. 2020.
Tsuyoshi Okita and Sozo Inoue. Recognition of multiple overlapping activities using compositional cnn-lstm model. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wear- able Computers, pp. 165-168, 2017.
Marco AF Pimentel, Peter H Charlton, and David A Clifton. Probabilistic estimation of respiratory rate from wearable sensors. Wearable electronics sensors, pp. 241-262. Springer, 2015.
Samsudin, A Shabri, and P Saad. A comparison of time series forecasting using support vector machine and artificial neural network model. Journal of applied sciences, 10(11):950-958, 2010.
Derek Snow. Atspy: Automated time series forecasting in python. Available at SSRN, 2020.
Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey Webb. Time series regres- sion. arXiv preprint arXiv:2006. 12672, 2020.
Jiahao Wang, Qiuling Long, Kexuan Liu, Yingzi Xie, et al. Human action recognition cellphone using compositional bidir-lstm-enn networks. In 2019 International Conference on Computer, Network, Communication and Information Systems (CNCI 2019). Atlantis Press, 2019.
Jianbo Yang. Minh Nhut Nguyen, Phyo Phyo San, Xiao Li Li, and Shonali Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In Twwenty-Forth International Joint Conference on Artificial Intelligence, 2015.
Youmei Zhang, Faliang Chang, Mengdi Wang, Fulei Zhang, and Chao Han. Auxiliary learning for crowd counting via count-net. Neurocomputing, 273:190-198. 2018.
Yi Zheng, Qi Liu, Enhong Chen, Yong Ge, and J Leon Zhao. Time series classification using multi-channels deep convolutional neural networks. In International Conference on Web-Age Information Management, pp. 298-310. Springer, 2014.
10

---

Under review as a conference paper at ICLR 2022
7 CLOUDINARY DATA SET FEATURES
The following features were extracted from the data:
(1) Day of the week: It is evident from the data that customers have weekly consumption cycle. Some customers have usage that peaks during workdays, while others peak during weekends.
(2) Months since customer registered: Seasoned customers are more stable and predictable, while newer customers may still be learning and experimenting with the functionality of the system.
This category includes five different features, meant give the model the ability to differentiate between different subscription plans. (3.1) Normalized consumption: This is the cost the customer computed as if the customer were enrolled in the default plan. The default plan is generally suited to small customers, while large customers typically have custom (and heavily discounted) plans. (3.2) Consumption utilization numerator: The unit consumption of each cus- tomer based on the customer's s plan, and including the consumption over the entire payment period (month or year). (3.3) Consumption utilization plan denominator: The number of expected con- sumption units based on the customer's s plan. For monthly plans this is simply the number of units of the plan, and for annual plans this is the number of units used to date. (3.4) Monthly consumption amount: The number of units consumed by the customer in the last month. (3.5) Consumption utilization denominator: This is the normalized available units in the plan for the relevant month.
(3) Utilization:
For monthly plans this is simply the number of units included in the plan, and for annual plans this is the number of units divided by 12.
(4) Storage: In this category we created five different features to allow the model adjust to the storage requirements of various file types. (4.1) Resources: A count of the number of media objects the customer has uploaded. (4.2) Derived resources: A count of the number of transformed (derived) media objects the customer holds. (4.3) Resource storage: The size of the uploaded resources (in Terabytes). (4.4) Derived resource storage: The size of transformed resources (int Terabytes). (4.5) Backup storage TB: Total storage backup.
(5) Requests: In this category we created six different features allow the model to adapt to delivery consumption. (5.1) Image requests: This is a count of the number of image requests made by the customer over the Content Delivery Network (CDN). (5.2) Video requests: This is the same as image requests, but for video. (5.3) Total requests: This includes requests for all file types. (5.4) Image bandwidth: The size of all delivered images (in Terabytes). (5.5) Video bandwidth: The size of all delivered video assets (in Terabytes). (5.6) Total bandwidth: This includes the total sizes of image, video and other file formats (in Terabytes).
(6) Transformations: Under transformations understand the core usage of our customer during the last month. As usage of transformation is the strongest indication of our customers experience. (6.1) Image transformations: Amount of transformations that were done on images in the last months, a customer that doesn't 't do any trans- formations is strong indication that will later decrease his requests and later his storage. (6.2) Video transformations: Amount of video transformations done in the last month by the customer (6.3) Video transformations: Total amount of transformations (image, video and other)
created 3 different features to help the model
(7) Monetization: Under monetization we created different features that reflect the commercial affect (7.1) Company cost: Total company cost based on the customer usage, this includes the storage, bandwidth and transformations. Some customers the cost may be correlated to their plan, yet some may be abusing or un-abusing. (7.2) Monthly Recurring Revenue (MRR): The current MRR of the customer, no matter if he is in a monthly plan or annually plan. This value represents the recurring revenue the customer pays for the all of the services he gets from the company. (7.3) Overages dollar: In case customer exceeded his plan he starts to pay overages, most customers are fine with paying overages especially when they are small, yet some can get a bill shock case the amount is very high which may lead to future chum.
11


Reviews:
[{"review_id": "Q8jOIK9aD3y", "reviewer": "Reviewer_ANiP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is focused on an interesting topic of multivariate time-series prediction, which is a vital area surrounding many real-world application domains (besides the consumer behaviour forecasting in this paper, the applications can also pertain to a multitude of very different tasks such as wind power forecasting, weather forecasting, stock price predictions etc.). The proposed model in this paper can easily be adapted to other tasks besides the ones discussed in this paper in the opinion of this reviewer, which can be beneficial to other domains.  The paper is well written and easy to follow.\n\nHowever, this reviewer believes that the proposed methodology in this paper has limited novelty - as multi-dimensional time-series prediction for long has been an area of active research in the AI community. Additionally, only utilising auxiliary inputs to make the conventional CNN architecture deeper for which the authors add 6 auxiliary layers) does not yield major  performance improvements in multiple cases as per the results discussed in the paper in Tables 2 and 3 (for the Cloudinary data) and Table 4 (for the bike-sharing data).  However, the reviewer notes that in case of the 30 day predictions with the Cloudinary dataset and when low values of ‘d’ were utilised with the bike sharing data, there were significant performance improvements. For a performance improvement in accuracy as described in the paper which mostly levels around 2-3% in most of the cases (and maximum around 6%) - this reviewer does not see any key improvements compared to the LSTM architecture the authors mention in the baseline. Also, there are no details of results obtained with the conventional CNN architecture (without using auxiliary information) in the paper - which places uncertainty into the suitability of using auxiliary information for the learning task discussed in this paper. As the authors are not comparing their proposed model with a baseline CNN, there is an uncertainty in their claim that the performance boost can only arise from incorporation of the auxiliary layers. It may well be possible that appropriate hyper-parameter optimisation and tuning of a baseline CNN  (which is generally simpler and less computationally expensive) could yield similar (or even better) results, but since it was not compared to in this study, the methodology pertaining to development of an improved model architecture in Section 4 (in comparison to conventional CNNs) in the paper is not well-supported in terms of the results.  Additionally, the literature review in the paper is very limited and mostly does not cover broader literature in multi-dimensional time-series prediction beyond the CNN utilised in the paper. This includes aspects concerning prediction of bike sharing demand by other related studies which have used very similar datasets and applied AI models (including RNNs and simpler models such as ANNs) - such as :\n\n- Pan, Yan & Zheng, Ray & Zhang, Jiaxi & Yao, Xin. (2019). Predicting bike sharing demand using recurrent neural networks. Procedia Computer Science. 147. 562-566. 10.1016/j.procs.2019.01.217. \n- Thirumalai et al. (2017). Bike Sharing Prediction using Deep Neural Networks. JOIV : International Journal on Informatics Visualization. 1. 83. 10.30630/joiv.1.3.30.  \n\netc .  Similarly,  for the Cloudinary case study - there is no related work covered to provide an overview of related studies utilised in datasets from software-as-a-service (SAAS) companies and the past literature in applying AI for predicting subscriber usage with closely related datasets. This limits the literature review’s scope to be very broad and not particularly domain-specific in relation to the two case studies the paper focuses on. \n\nA comment which the reviewer would like to make is regarding the computational cost (this is not critical to the paper’s argument though) - When making the network deeper by incorporating 6 auxiliary layers - it would have been useful to see information regarding the computational time (across the baselines as well as the proposed approach), which the paper does not discuss. If the computational time of the model is significantly higher than the baselines (particularly the conventional LSTM), it seems unclear as to whether the limited performance gain at the cost of high computational time (which likely may be the case on making the network deeper in the opinion of this reviewer), is justified in terms of the contribution of the approach. While this reviewer understands that in certain application domains, even a small improvement in accuracy could be potentially significantly important, the paper does not discuss the role of the ‘accuracy gain’ in relation to the two case studies, and without a clear summary of the importance of this performance boost to the domain-specific problem, which may have been an interesting result to see.  \n\nThis paper would have benefited from a clearer description of the datasets, its potential challenges and the key drawbacks of the baselines  in specific cases of experiments when the performance gain in the proposed approach is minimal (particularly with the Cloudinary dataset). Specifically, a discussion of the potential applications of the specific model architecture proposed by the authors on beyond the two datasets and its usage outside these domains would have made this paper’s novelties much clearer.  Additionally, the paper should provide a summary of the key limitations of the approach and any concerns it raises in real-world deployment of the model - in comparison to simpler, easier to deploy and train, and generally computationally less expensive architectures like conventional CNNs. \n\nIt would have been useful to see a clearer conclusion in the paper on why the authors consider the proposed methodology to be well suited for the specific problem (and datasets) under consideration. Additionally, it would be nice if the authors can comment on other potential use cases of the methodology.\n", "summary_of_the_paper": "This paper focuses on a multi-dimensional time-series prediction task through deep learning - particularly towards predicting consumer behaviour. The authors utilised two real-world datasets (Cloudinary and bike-sharing dataset, with the bike-sharing data being publicly available) for predicting subscriber usage to identify consumer behaviour. Additionally, the authors also mention that experiments were also conducted on a publicly available exchange-rate dataset for time-series forecasting. The key theme in the proposed approach is the incorporation of auxiliary inputs to a CNN model (by making the conventional CNN deeper with multiple auxiliary layers), and ultimately utilising the model for time-series prediction. The paper, with multiple experiments conducted across baselines (such as ANNs, Random Forest and LSTMs), highlights that forecasting individual consumer behaviour (based on the bike-share dataset) yields better results when utilising auxiliary layers in the CNN model.  \n", "summary_of_the_review": "Overall, while the paper’s topic is interesting - given that multi-dimensional time-series prediction is an active area of research interest in the AI community, as well as focuses on predicting subscriber usage based on two real-world datasets (of which one is publicly available),  the paper’s methodology has limited novelty. Also, the literature review is not thorough, and the paper is not well-supported in terms of clearer description of results beyond the models’ accuracies as described in the main review. \n\nThis reviewer believes that this paper would be better suitable for publication in a more applied venue - as the contribution to technical novelty, particularly the methodology proposed and the application of the study is very limited.  Additionally, the results should be incorporated with additional details on computation time and metrics beyond accuracy, which is integral for comparison of (deep) ML architectures with more traditional baselines, particularly as the paper aims to make the conventional CNN architecture deeper by adding additional auxiliary layers. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "OodaRcqcL_L", "reviewer": "Reviewer_4yDn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is an application paper and thus lacks methodology and theory strengths.\n\nIn the contribution part of the introduction section, it shows the proposed method highly resembles the well-known iterative method for multi-step predictions. The proposed idea of using auxiliary layers is straightforward and lacks technical motivation and insights.  \n\nIn the related work section, the listed papers are less representative and many relevant and technically advanced work are missing, for example (but not limited to), \n\n2019, NeurIPS, Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting\n2019, ICML, Deep Factors for Forecasting\n2021, ICLR, Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows\n2021, ICML, Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting\n2021, ICML, RNN with Particle Flow for Probabilistic Spatio-temporal Forecasting\n\nThe presentation of this paper needs to be re-structured for better highlighting the methodology and theoretical contribution. \nThe author is supposed to clarify the special characteristics of the data if any, given that the data processing is in one of the main sections in the paper, while the content is ordinary and less interesting. The section presenting the models unnecessarily includes too many implementation details, e.g. network setups, hyper-parameter search, which are rather trivial and should be put in the experiment or appendix sections. \n\nIn the experiment, the author used a less common metric involving an error threshold parameter, while the commonly used point and probabilistic errors are missing, e.g. mean squared error, quantile error, test likelihood, etc (refer to the papers listed above). The authors are expected to elaborate more on the motivation of the chosen metric. It would be better if the authors present the results of the common metrics as well, because existing methods are mostly evaluated on these metrics.\n\nEven for the chosen metric, the result is confusing in that based on the error definition Eq.(4), for an error threshold, the interpretation would be the less the metric value, the better the performance, because it means there is less predictions different from the true values beyond the error threshold. In this sense, the proposed method is performing the worst. \n\nThe baselines are rather naïve and insufficient to justify the advantages of the proposed method. \n", "summary_of_the_paper": "This is an application paper about utilizing CNN for multi-dimensional time series from subscriber usage. The authors proposed to use auxiliary layers to produce multi-step predictions. Overall, this paper lacks technical and theoretical novelty. The presentation needs significant improvements and the experiment section is unsolid to justify the proposed methods.  ", "summary_of_the_review": "On the methodology side, the proposed method is a stack of ordinary convolutional layer and auxiliary output layers and thus falls short of strong motivation, novelty and insights. On the presentation side, it is more a technical report and lacks the differentiating elements from existing works. The experimental evaluation is weak, because of the less common error metrics, naïve baselines, and unconvincing result analysis. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EVDLeUtT6XS", "reviewer": "Reviewer_Cb5i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents an empirically effective methodology to predict the product usage of a given consumer using multivariate time series.\nThere are two main problems that make the paper not ready.\nFirst, the paper would have been more suitable for an industrial track or an applied analytics tracks rather than for a research track.\nIndeed, in my opinion, the paper lacks novelty as it can be reduced to applying a CNN-based approach to the problem of time series regression. \nSecond, the paper lacks a formalism that prevents clarifying the data dimensionality. Indeed, there is a continuous discrepancy between the terms multivariate time series for the data and one-dimensional CNN for the model. Perhaps a formalization of the data and of the model would have clarified this ambiguity and helped in better understanding the data description and data preprocessing steps. Also, the overall presentation would benefit from a toy example.\n", "summary_of_the_paper": "The paper presents a deep learning model to predict the product usage of a given consumer based on historical usage.", "summary_of_the_review": "Probably the wrong venue.\nLack of formalism in the presentation creates confusion between data and model dimensionality.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D7tV-EgOTT", "reviewer": "Reviewer_Y6ay", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the proposed techniques aren't identical to methods in the literature (to the best of my knowledge), there is significant overlap with well-known methods that lack citation. Notably, the paper doesn't cite Temporal Convolutional Networks and its variations (other than briefly alluding to commercial implementations), which has been used extensively in similar scenarios. TCNs share many of the properties of the proposed method, including the multi-scale temporal resolution.\n\nFor reference, here is a paper applying TCNs to a comparable time series forecasting task: https://www.nature.com/articles/s41598-020-65070-5\n\nWith the lack of contextualization in the relevant literature, the paper also misses important baselines for its two experiments. The proposed method beats vanilla techniques (namely, vanilla RF, ANN, CNN, and RNN), but it's unclear if the method is competitive with state-of-the-art techniques (like TCN and others). Additionally, for such a task, it's helpful to understand how much the additional features help beyond an autoregressive effect; it'd be helpful to also have included a simple autoregressive model in the baseline.\n\nThe paper transforms the labels to a number in [-1, 1] based on rates of change, which it reasonably justifies for modeling. However, the metric (|y - y'| < threshold, after the transformation) used in the experiments is not very intuitive and could be misleading, especially considering the paper focuses on business applications. For example, the real-world difference for transformed values 0.999 and 0.9999 is massive, but the error metric will be minor. It'd be helpful to include some standard metrics (e.g. MAE and RMSE) on the original label values (before the transformation), for contextualization.\n\nThe paper is otherwise clear and generally well-motivated, and covers an important business use case. I appreciate that the authors used a public dataset in addition to their private dataset.  \n\n\nOther thoughts:\n\n- The paper mentions multiple categorical features (e.g. plan type), but does not specify if there was any special treatment for them. Are they one-hot encoded, treated as integers, ...?\n- [nit] Typo: Section 3 -- \"and their select[ED] features,\"\n- [nit] Style: mixed usage of \"data sets\" and \"data-sets\" (hyphen)\n\n", "summary_of_the_paper": "The paper presents a model architecture for time series forecasting based on CNNs. The proposed method incorporates multi-dimensional inputs as well as auxiliary outputs for different time scales. Compared to the demonstrated baselines, the paper's method generally outperformed across two datasets for product usage (cloud SaaS and bike rentals).\n", "summary_of_the_review": "My primary concerns are missing literature (and therefore baselines), and unintutive evaluation metrics.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1wFNxBYNw3T", "reviewer": "Reviewer_BJUT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Details have been given for datasets, data processing. Improvement has been shown on real datasets. \n\nHere are a few weakness points to the paper:\n\n1. The novelty of this paper is limited. The proposed method is essentially a variant of convolutional nets on time series data. The major difference is that it introduces multiple losses to incrementally predict multiple variables from short term to long term. \n\n2. In the result section, it would be more convincing if the authors could provide more analysis on the improvement. Also, the authors could consider studying the effect of multiple losses, e.g., by comparing to a CNN without (or with less) auxiliary layers\n\n3. Some figures are not informative. For example, more details should be added to Figure 3 and its caption to make it clear. \n\n4. The authors compared the proposed method to standard ML approaches, but missed some state-of-the-art baselines. Also, it would be better to discuss/compare with other approaches used in weather forecasting, which is known to be a challenging forecasting problem with chaotic dynamics nature and long-term patterns. \n\n5. The presentation needs to be polished. Typos: Page 1, “to accurate to accurately predict”\n", "summary_of_the_paper": "This paper is focused on the development of an automated model for predicting multi-variate future usage. This is an important problem for industrial companies that rely on subscriber systems. Long-term future prediction/forecasting is known to be a challenging problem, especially for systems with complex dynamics and high dimensional variables. In this paper, the authors propose a CNN-based model with multiple outputs/losses. Multiple auxiliary layers are constructed to incrementally produce predictions from short-term future to long-term future. Results have shown some improvement on two datasets. ", "summary_of_the_review": "Overall, this paper studies an important problem, but the novelty is limited. The experiments are also insufficient. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ICLR is selective and reviewers are not sufficiently enthusiastic about this paper. In particular, they point out closely related methods that should be cited and compared to as baselines. The reviews are of good quality, and the authors did not respond.
Decision: Reject