Predicting subscriber usage: Analyzing multi-dimensional time-series using Convolutional Neural Networks

1 INTRODUCTION . Historically , companies interested in usage prediction have retained the services of expert analysts . These experts may examine past behavior over multiple features , estimate future usage in the shortterm , and translate this knowledge into actionable tasks . Yet for companies which boast thousands of customers and have an increased need for accurate research forecasting , this non-scalable approach quickly becomes infeasible . Hence , the development of an automated model is an obvious necessity ( Miller et al. , 2021 ) . Future usage prediction is especially crucial for subscriber-based companies , for which churn rate has especially significant impact on profitability . It is crucial as well for companies valuated based on Annual Recurring Revenue ( ARR ) which reflects income derived from customers . For these companies , future usage prediction is a prerequisite for predicting the future valuation of the company . Usage data sets are often represented as multi-dimensional time-series data , and indeed we have at our disposal two large multi-dimensional time-series , which constitute high quality labeled data sets , each produced in a controlled environment . However , successfully predicting multi-dimensional time-series values is itself a non-trivial task ( Cerqueira et al. , 2019 ) . Indeed , automated multidimensional time series prediction is a relatively new field , and although different strategies have been proposed in the literature , currently there seems to be no consensus approach to the problem . Here , machine learning and deep learning algorithms are typically used for time-series data classification , which produces a binary decision value ( for example , whether the customer usage will grow or not ) . The problem of regression , which produces a continuous value describing customer growth the extent of customer growth , is clearly much more challenging to accurate to accurately predict . Our contribution . For the regression problem of multi-dimensional time-series prediction , we propose to utilize both common machine learning techniques and novel methods . Specifically , we shall leverage one-dimensional convolutional neural network ( CNN ) , in conjunction with auxiliary output , to predict future utilization based on historical usage . In this model , an auxiliary layer produces a prediction for the future in the short-term . This prediction is fed into the next auxiliary layer , which produces a prediction for a future prediction for a slightly later period , until the final auxiliary layer produces a prediction for the desired longer-term period . We demonstrate experimentally that this enhanced model effectively predicts future usage , and gives superior results when compared to other commonly utilized models in the field . We present the data sets in Section 3.1 , describe our learning model in detail in Section 4 , and demonstrate promising empirical results in Section 5 . In Section 6 , we give concluding remarks and discuss future work . 1.1 RELATED WORK . Here we give an overview of some traditional machine learning approaches for time-series prediction ( for both classification and regression ) , as well as more recent CNN-based approaches for this problem . Traditional learning approaches . Tan et al . ( 2020 ) aimed to learn the relationship between a time-series and a continuous scalar variable , by identifying the right coefficients to best predict the future variable . Time-series regression models have been applied to various problems such as predicting annual rainfall from observed temperature , or predicting fat content in meat from near-infrared spectrum Goldsmith & Scheipl ( 2014 ) . That study compared different time series regression models , such as gradient boosting and random forest . It demonstrated that functional regression models usually require an in-depth understanding of the data in order to apply the right basis function to fit the model . In Pimentel et al . ( 2015 ) it was shown that many cutting-edge regression methods fail to distinguish between periods of high and low quality data , and further are not able to generalise well to other data-sets . They attempted to estimate respiratory rates ( RR ) from a photoplethysmogram ( PPG ) time-series using a sliding window . Producing an estimation for RR per window consists of four key components : ( 1 ) extracting respiratory signals ; ( 2 ) estimating respiratory rates ; ( 3 ) fusing the estimates and ( 4 ) executing a quality assessments . The authors applied a Gaussian process regression framework to extract RR from the different sources of modulation in the PPG signal , and also tried fitting multiple auto-regressive models to the extracted respiratory signals . In Zheng et al . ( 2014 ) , it was shown that k-Nearest Neighbor ( KNN ) combined with Dynamic Time Warping ( DTW ) achieves state of the art performance for classification . However , although the time complexity of 1-NN with DTW grows linearly with the data size , their performance dependent on the quality of hand-crafted features , and these are not scalable . In Samsudin et al . ( 2010 ) , Support Vector Machine ( SVM ) were compared with Artificial Neural Network ANN on five distinct one-dimensional time-series for the task of regression . They utilized a grid-search technique to identify the optimal hyper-parameters in the data , and used 10-fold cross validation while training . They indicated that SVM was more efficient than ANN on singledimensional time-series . CNN-based approaches . In Yang et al . ( 2015 ) it was shown that classification using CNN on multi-channel human activity recognition time-series outperformed all other algorithms tested . They proposed a systematic feature learning method , adopting CNN to automate feature learning from raw inputs . In deep architecture , the learned features are the high level abstract representations of low level raw time series signals . By leveraging the labelled information through supervised learning , the learned features are presented with more discriminatory power . LeCun et al . ( 1995 ) suggested a potentially more interesting scheme ( for classification as well ) , by eliminating the feature extractor , and feeding the network with raw inputs ( e.g . normalized images ) . This relies on back-propagation to turn the first few layers of the neural net into an appropriate feature extractor . According to Okita & Inoue ( 2017 ) , the combination of RNN ( such as LSTM ) and CNN for classification problems such as activity recognition yielded improved accuracy on the range of 27 % to 43 % when compared to the other popular algorithms . Wang et al . ( 2019 ) also considered classification for Human Action Recognition ( HAR ) , and evaluated multiple models , including LSTM , CNN , Random Forest , J48 Decision Tree , and SVM . They found that the superior model was BidirLSTM-CNN , a combination of the outputs of a bidirectional LSTM and CNN connected to three fully connected hidden layers . There has been significant recent interest in analyzing time-series in order to effectively allocate company resources . Snow ( 2020 ) undertook an extensive review of popular algorithmic solutions for regression problems , such as Facebook ’ s Prophet . This tool is given a list of dated events , and takes into consideration holidays , weekends and special occasions , before outputting predictions for future usage . Amazon ’ s GluonTS and Microsoft ’ s ForecastTCN attempt to do the same . This problem has also received attention from academia and independent developers such as N-Beats , Auto-Arima , and TBATS . One caveat is that all of these algorithm deal with only one-dimensional time-series input data . Using CNN to address multivariate time-series regression has became increasingly popular . Mode & Hoque ( 2020 ) compared CNN , LSTM , and Gated Recurrent Unit , showing that the results of the CNN model were more transferable when compared to the others . According to experimental analysis undertaken by Antsfeld et al . ( 2020 ) , utilizing LSTM layers along with additional convolutional layers , for regression can provide a significant boost in forecasting performance . And Mehtab et al . ( 2020 ) compared three models of CNN and four models of LSTM ( regression problem ) , and concluded that the execution time of the CNN models were 3-10 times faster than the others , while the best accuracy was achieved by one of the CNN models . 2 PROBLEM STATEMENT . We are given a set of multi-dimensional time-series , each with a continuous label-vector representing usage . Our goal is to predict the label of a query time-series , that is its future increase or decrease in usage . There is an imbalance in this model , in that the maximum possible decrease is bounded by 100 % , while the maximum possible increase is unbounded . To address this , let PA be the average usage in the observed time-frame , and PB the average usage in the time-frame to be predicted . Setting δ = PBPA − 1 , we define our usage increase function as : f ( δ ) = { 1− 11+δ , if δ > 0 δ , otherwise ( 1 ) The above function maps the increase or decrease onto a real value in the range [ −1 , 1 ] . Values tending to -1 imply significant decrease in usage , values close to 0 imply no change in usage , and values close to 1 imply extreme growth . ( See also its use in Section 3.1 below . ) 3 DATA AND PROCESSING . Before presenting the model , we describe our data sets and their select features , and detail our data processing . The data processing includes the creation of a modified label-vector for the time-series , which will be necessary for running the model . 3.1 DATA SETS AND FEATURES . Our main data-set is a proprietary set provided by Cloudinary , which is a software-as-a-service ( SAAS ) company providing media end-to-end solutions in the cloud . The data contains daily records of the media transformations undertaken by each customer , as well as delivery over the network , recorded as delivery requests and utilized bandwidth . Another important feature is the total data storage used by each customer through uploading media to Cloudinary ’ s distributed shards ; this includes original assets and assets derived via transformations . The data-set also records each customer ’ s subscription plan – these can be standard or custom , and monthly or annual . Note that a customer in this data set is often a company consisting of many distinct users . We also utilized the public Bike-Share Usage in London and Taipei Network data set ( link ) , which is a spatio-temporal urban transport data set from a network of bike stations . London and Taipei are both very large cities with large bike-share systems . At each location where bikes are picked up , the individual bikes are tracked , so that each rental generates a digital footprint : which bike , from where , to where , for how long , at what date and time , and by whom . The data collects events from 2016 until the first half of 2020 . For both data sets , care was taken to determine which features should be extracted . We avoided sparse features , as well as pairs of correlated features , or features unrelated to usage . A detailed list of features extracted from the Cloudinary data set is found in the Appendix . From the Bike Sharing data set we extracted the following fields : Start date , bike ID , duration , and startStation ID . We have also filtered out customers who are outliers , meaning their usage of the product is atypically small or large .

Summary: This paper is focused on the development of an automated model for predicting multi-variate future usage. This is an important problem for industrial companies that rely on subscriber systems. Long-term future prediction/forecasting is known to be a challenging problem, especially for systems with complex dynamics and high dimensional variables. In this paper, the authors propose a CNN-based model with multiple outputs/losses. Multiple auxiliary layers are constructed to incrementally produce predictions from short-term future to long-term future. Results have shown some improvement on two datasets.