Predicting subscriber usage: Analyzing multi-dimensional time-series using Convolutional Neural Networks

1 INTRODUCTION . Historically , companies interested in usage prediction have retained the services of expert analysts . These experts may examine past behavior over multiple features , estimate future usage in the shortterm , and translate this knowledge into actionable tasks . Yet for companies which boast thousands of customers and have an increased need for accurate research forecasting , this non-scalable approach quickly becomes infeasible . Hence , the development of an automated model is an obvious necessity ( Miller et al. , 2021 ) . Future usage prediction is especially crucial for subscriber-based companies , for which churn rate has especially significant impact on profitability . It is crucial as well for companies valuated based on Annual Recurring Revenue ( ARR ) which reflects income derived from customers . For these companies , future usage prediction is a prerequisite for predicting the future valuation of the company . Usage data sets are often represented as multi-dimensional time-series data , and indeed we have at our disposal two large multi-dimensional time-series , which constitute high quality labeled data sets , each produced in a controlled environment . However , successfully predicting multi-dimensional time-series values is itself a non-trivial task ( Cerqueira et al. , 2019 ) . Indeed , automated multidimensional time series prediction is a relatively new field , and although different strategies have been proposed in the literature , currently there seems to be no consensus approach to the problem . Here , machine learning and deep learning algorithms are typically used for time-series data classification , which produces a binary decision value ( for example , whether the customer usage will grow or not ) . The problem of regression , which produces a continuous value describing customer growth the extent of customer growth , is clearly much more challenging to accurate to accurately predict . Our contribution . For the regression problem of multi-dimensional time-series prediction , we propose to utilize both common machine learning techniques and novel methods . Specifically , we shall leverage one-dimensional convolutional neural network ( CNN ) , in conjunction with auxiliary output , to predict future utilization based on historical usage . In this model , an auxiliary layer produces a prediction for the future in the short-term . This prediction is fed into the next auxiliary layer , which produces a prediction for a future prediction for a slightly later period , until the final auxiliary layer produces a prediction for the desired longer-term period . We demonstrate experimentally that this enhanced model effectively predicts future usage , and gives superior results when compared to other commonly utilized models in the field . We present the data sets in Section 3.1 , describe our learning model in detail in Section 4 , and demonstrate promising empirical results in Section 5 . In Section 6 , we give concluding remarks and discuss future work . 1.1 RELATED WORK . Here we give an overview of some traditional machine learning approaches for time-series prediction ( for both classification and regression ) , as well as more recent CNN-based approaches for this problem . Traditional learning approaches . Tan et al . ( 2020 ) aimed to learn the relationship between a time-series and a continuous scalar variable , by identifying the right coefficients to best predict the future variable . Time-series regression models have been applied to various problems such as predicting annual rainfall from observed temperature , or predicting fat content in meat from near-infrared spectrum Goldsmith & Scheipl ( 2014 ) . That study compared different time series regression models , such as gradient boosting and random forest . It demonstrated that functional regression models usually require an in-depth understanding of the data in order to apply the right basis function to fit the model . In Pimentel et al . ( 2015 ) it was shown that many cutting-edge regression methods fail to distinguish between periods of high and low quality data , and further are not able to generalise well to other data-sets . They attempted to estimate respiratory rates ( RR ) from a photoplethysmogram ( PPG ) time-series using a sliding window . Producing an estimation for RR per window consists of four key components : ( 1 ) extracting respiratory signals ; ( 2 ) estimating respiratory rates ; ( 3 ) fusing the estimates and ( 4 ) executing a quality assessments . The authors applied a Gaussian process regression framework to extract RR from the different sources of modulation in the PPG signal , and also tried fitting multiple auto-regressive models to the extracted respiratory signals . In Zheng et al . ( 2014 ) , it was shown that k-Nearest Neighbor ( KNN ) combined with Dynamic Time Warping ( DTW ) achieves state of the art performance for classification . However , although the time complexity of 1-NN with DTW grows linearly with the data size , their performance dependent on the quality of hand-crafted features , and these are not scalable . In Samsudin et al . ( 2010 ) , Support Vector Machine ( SVM ) were compared with Artificial Neural Network ANN on five distinct one-dimensional time-series for the task of regression . They utilized a grid-search technique to identify the optimal hyper-parameters in the data , and used 10-fold cross validation while training . They indicated that SVM was more efficient than ANN on singledimensional time-series . CNN-based approaches . In Yang et al . ( 2015 ) it was shown that classification using CNN on multi-channel human activity recognition time-series outperformed all other algorithms tested . They proposed a systematic feature learning method , adopting CNN to automate feature learning from raw inputs . In deep architecture , the learned features are the high level abstract representations of low level raw time series signals . By leveraging the labelled information through supervised learning , the learned features are presented with more discriminatory power . LeCun et al . ( 1995 ) suggested a potentially more interesting scheme ( for classification as well ) , by eliminating the feature extractor , and feeding the network with raw inputs ( e.g . normalized images ) . This relies on back-propagation to turn the first few layers of the neural net into an appropriate feature extractor . According to Okita & Inoue ( 2017 ) , the combination of RNN ( such as LSTM ) and CNN for classification problems such as activity recognition yielded improved accuracy on the range of 27 % to 43 % when compared to the other popular algorithms . Wang et al . ( 2019 ) also considered classification for Human Action Recognition ( HAR ) , and evaluated multiple models , including LSTM , CNN , Random Forest , J48 Decision Tree , and SVM . They found that the superior model was BidirLSTM-CNN , a combination of the outputs of a bidirectional LSTM and CNN connected to three fully connected hidden layers . There has been significant recent interest in analyzing time-series in order to effectively allocate company resources . Snow ( 2020 ) undertook an extensive review of popular algorithmic solutions for regression problems , such as Facebook ’ s Prophet . This tool is given a list of dated events , and takes into consideration holidays , weekends and special occasions , before outputting predictions for future usage . Amazon ’ s GluonTS and Microsoft ’ s ForecastTCN attempt to do the same . This problem has also received attention from academia and independent developers such as N-Beats , Auto-Arima , and TBATS . One caveat is that all of these algorithm deal with only one-dimensional time-series input data . Using CNN to address multivariate time-series regression has became increasingly popular . Mode & Hoque ( 2020 ) compared CNN , LSTM , and Gated Recurrent Unit , showing that the results of the CNN model were more transferable when compared to the others . According to experimental analysis undertaken by Antsfeld et al . ( 2020 ) , utilizing LSTM layers along with additional convolutional layers , for regression can provide a significant boost in forecasting performance . And Mehtab et al . ( 2020 ) compared three models of CNN and four models of LSTM ( regression problem ) , and concluded that the execution time of the CNN models were 3-10 times faster than the others , while the best accuracy was achieved by one of the CNN models . 2 PROBLEM STATEMENT . We are given a set of multi-dimensional time-series , each with a continuous label-vector representing usage . Our goal is to predict the label of a query time-series , that is its future increase or decrease in usage . There is an imbalance in this model , in that the maximum possible decrease is bounded by 100 % , while the maximum possible increase is unbounded . To address this , let PA be the average usage in the observed time-frame , and PB the average usage in the time-frame to be predicted . Setting δ = PBPA − 1 , we define our usage increase function as : f ( δ ) = { 1− 11+δ , if δ > 0 δ , otherwise ( 1 ) The above function maps the increase or decrease onto a real value in the range [ −1 , 1 ] . Values tending to -1 imply significant decrease in usage , values close to 0 imply no change in usage , and values close to 1 imply extreme growth . ( See also its use in Section 3.1 below . ) 3 DATA AND PROCESSING . Before presenting the model , we describe our data sets and their select features , and detail our data processing . The data processing includes the creation of a modified label-vector for the time-series , which will be necessary for running the model . 3.1 DATA SETS AND FEATURES . Our main data-set is a proprietary set provided by Cloudinary , which is a software-as-a-service ( SAAS ) company providing media end-to-end solutions in the cloud . The data contains daily records of the media transformations undertaken by each customer , as well as delivery over the network , recorded as delivery requests and utilized bandwidth . Another important feature is the total data storage used by each customer through uploading media to Cloudinary ’ s distributed shards ; this includes original assets and assets derived via transformations . The data-set also records each customer ’ s subscription plan – these can be standard or custom , and monthly or annual . Note that a customer in this data set is often a company consisting of many distinct users . We also utilized the public Bike-Share Usage in London and Taipei Network data set ( link ) , which is a spatio-temporal urban transport data set from a network of bike stations . London and Taipei are both very large cities with large bike-share systems . At each location where bikes are picked up , the individual bikes are tracked , so that each rental generates a digital footprint : which bike , from where , to where , for how long , at what date and time , and by whom . The data collects events from 2016 until the first half of 2020 . For both data sets , care was taken to determine which features should be extracted . We avoided sparse features , as well as pairs of correlated features , or features unrelated to usage . A detailed list of features extracted from the Cloudinary data set is found in the Appendix . From the Bike Sharing data set we extracted the following fields : Start date , bike ID , duration , and startStation ID . We have also filtered out customers who are outliers , meaning their usage of the product is atypically small or large .

Summary: The paper presents a model architecture for time series forecasting based on CNNs. The proposed method incorporates multi-dimensional inputs as well as auxiliary outputs for different time scales. Compared to the demonstrated baselines, the paper's method generally outperformed across two datasets for product usage (cloud SaaS and bike rentals).