SYSTEMS AND METHOD FOR MASKED MULTI-STEP MULTIVARIATE TIME SERIES POWER FORCASTING AND ESTIMATION

A system includes a computing device including at least one processor in communication with at least one memory. The at least one processor is programmed to (a) store a plurality of historical time series data; (b) randomly select a sequence; (c) randomly select a mask length for a mask for the selected sequence; (d) apply the mask to the selected sequence, wherein the mask is applied to the plurality of forecast variables in the selected sequence; (e) execute a model with the masked selected sequence to generate predictions for the masked forecast variables; (f) compare the predictions for the masked forecast variables to the actual forecast variables in the selected sequence; (g) determine if convergence occurs based upon the comparison; and (h) if convergence has not occurred, update one or more parameters of the model and return to step b.

BACKGROUND

The field of the invention relates generally to predicting further performance of systems, such as, but not limited to, electric power generation and delivery systems and, more particularly, to systems and methods for training machine learning to predict future outputs using multi-step multivariate time series power forecasting.

Accurate short to mid-term forecasting is critical for grid planning and operation. In many cases, time series forecasting that requires multi-step predictions has become an important part of many real-world applications in areas such as electricity demand modeling, air traffic volume prediction, stock prices forecasting, and crop yields estimation. Specifically, there is often some future information available, such as the weather information for short-to-mid-term electricity demand modeling, and jet fuel price for air traffic volume prediction, which is not fully leveraged in the existing forecasting frameworks.

In general, the multi-step time series forecasting problem can be categorized into two kinds of approaches: recursive methods and direct methods. Recursive methods typically use an autoregressive approach (one-step-ahead prediction) and produce multi-step forecasts by recursively feeding samples into the future time steps. However, as an error is often at each step, the recursive structure tends to accumulate large errors over long forecasting horizons. Direct methods, on the other hand, directly map all available inputs to multi-step forecasts and typically use a sequence-to-sequence (seq2seq) structure. The disadvantage of this method is that it is harder to train, especially when the forecast horizon is large.

Accordingly, there is a need for an improved system for multi-step multivariate time series forecasting, for electricity demand modeling, air traffic volume prediction, stock prices forecasting, and crop yields estimation, for example.

BRIEF DESCRIPTION

In one aspect, a system is provided. The system includes a computing device including at least one processor in communication with at least one memory device. The at least one processor is programmed to store a plurality of historical time series data including a plurality of predictor variables and a plurality of forecast variables. The at least one processor is also programmed to randomly select a sequence including a subset of continuous data points in the plurality of historical time series data. The at least one processor is further programmed to randomly select a mask length for a mask for the selected sequence. In addition, the at least one processor is programmed to apply the mask to the selected sequence, wherein the mask is applied to the plurality of forecast variables in the selected sequence. Moreover, the at least one processor is programmed to execute a model with the masked selected sequence to generate predictions for the masked forecast variables. Furthermore, the at least one processor is programmed to compare the predictions for the masked forecast variables to the actual forecast variables in the selected sequence. In addition, the at least one processor is also programmed to determine if convergence occurs based upon the comparison. If convergence has not occurred, the at least one processor is programmed to update one or more parameters of the model and return to step b. The system may have additional, less, or alternate functionality, including that discussed elsewhere herein.

In another aspect, a computer-implemented method is provided. The method is implemented by a computing device including at least one processor in communication with at least one memory device. The method includes storing a plurality of historical time series data including a plurality of predictor variables and a plurality of forecast variables. The method also includes randomly selecting a sequence including a subset of continuous data points in the plurality of historical time series data. The method further includes randomly selecting a mask length for a mask for the selected sequence. In addition, the method includes applying the mask to the selected sequence, wherein the mask is applied to the plurality of forecast variables in the selected sequence. Moreover, the method includes executing a model with the masked selected sequence to generate predictions for the masked forecast variables. Furthermore, the method comparing the predictions for the masked forecast variables to the actual forecast variables in the selected sequence. In addition, the method also includes determining if convergence occurs based upon the comparison. If convergence has not occurred, the method include updating one or more parameters of the model and return to step b. The method may have additional, less, or alternate functionality, including that discussed elsewhere herein.

DETAILED DESCRIPTION

The field of the invention relates generally to predicting further performance of systems, such as, but not limited to, electric power generation and delivery systems and, more particularly, to systems and methods for training machine learning to predict future outputs using multi-step multivariate time series power forecasting. The systems and methods described herein describe a masked multi-step multivariate forecasting (MMMF) system. The MMMF system is a novel and self-supervised learning framework for time series forecasting with known future information. In many real-world forecasting scenarios, some future information is known, e.g., the weather information when making a short-to-mid-term electricity demand forecast, or the future oil prices when making an airplane departure forecast. Existing machine learning forecasting frameworks can be categorized into (1) sample-based approaches where each forecast is made independently, and (2) time series regression approaches where the future information is not fully incorporated. To overcome the limitations of existing approaches, the MMMF system is configured to train any neural network model capable of generating a sequence of outputs, that combines both the temporal information from the past and the known information about the future to make better predictions. Furthermore, once a neural network model is trained with the MMMF system, its inference speed is similar to that of the same model trained with traditional regression formulations, thus making the MMMF system an improvement over existing regression-trained time series forecasting models if there is some available future information.

The MMMF system incorporates known future information directly during training. The MMMF system uses the future information with recursion when making iterative forecasts. The MMMF system integrates a general self-supervised learning task for training time series models (including Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and attention-based methods) to make multi-step forecasts with known future information. The MMMF system provides a flexible learning framework that improves upon existing methods by taking into account both recent history and known future information.

In the exemplary embodiment, the MMMF system is used for training neural network (NN)-based multi-step time series forecasting models with known future information. The MMMF system uses a masking technique that is flexible and can generate forecasts of different lengths. The MMMF system improves over existing methods by combining both recent history and known future information.

The Masked Multi-Step Multivariate Forecasting (MMMF) system is configured for training and inference of machine learning models for multi-step multivariate time series forecasting with future information. The MMMF system provides a framework to accommodate any underlying time series models, including recurrent neural networks, transformers, temporal convolutional networks, etc. The MMMF system provides a masked training scheme to combine past information on predictor variables and forecast variables, and future information on predictor variables, to generate all predictions on future forecast variables at once. Furthermore, the MMMF system incorporates historical information and also incorporates future information about the predictor variables. The MMMF system provides a flexible framework that once trained, generates forecasts for both short-term (1-step) predictions and multi-step predictions.

The training method for Masked Multi-Step Multivariate time series Forecasting includes, but is not limited to: Model Initialization; Data Preprocessing; Data Masking; and Model Updating. The MMMF system performs model initialization, which includes specifying a time series model ƒθwith trainable parameters θ, maximum forecasting horizon k, maximum history length T, and loss function. Then the MMMF system performs data preprocessing, which includes partitioning the dataset S={zi}={(xi, yi)} into length (T+k+1) sequences {zt−T, . . . , zt−1, zt, zt+1, . . . , zt+k}, where t is the current step, xiare the predictor variables, yiare the forecast variables.

The MMMF system next performs data masking by randomly choosing B sequences, for each sequence, randomly choose a mask length of 0<lm≤k+1, and masking the last lmsteps of forecast variables y. Additionally, the MMMF system performs model updating including providing masked sequences to model ƒθand generate forecasts ŷ for the masked outputs. The MMMF system sggregates the total loss(y, ŷ) for B sequences, update model parameters θ. The MMMF system repeats the Data Masking and Model Updating for n epochs or until convergence.

In at least one embodiment, the loss function could be means square error (MSE), means absolute percentage error (MAPE), means absolute percentage deviation (MAPD), mean absolute scale error (MASE), symmetric mean absolute percentage error (sMAPE), Mean Directional Accuracy (MDA), and/or any other error or loss function needed.

In at least one embodiment, the model may include, but is not limited to: long short-term memory networks (LSTM), transformer, temporal convolution networks, and/or any other sequence to sequence model.

The Masked Multi-Step Multivariate Forecasting (MMMF) system described herein is a new self-supervised learning framework for multi-step time series forecasting with known future information. One of the advantages of this MMMF system is that it provides more than just a new model and a set of hyperparameters for a particular problem. The MMMF system provides a general training task can outperform existing time series forecasting approaches, including recursive methods and direct methods while using the same base model. Once trained with MMMF, a time series model can generate any length forecasts below the maximum forecast length during training, and the inference speed, as well as the memory usage, are similar to those of traditional methods. Accordingly, the MMMF system is an upgrade to existing deep learning-based multi-step time series forecasting models for real-world forecasting applications where some future information is available.

Further, as used herein, the terms “software” and “firmware” are interchangeable and include any computer program storage in memory for execution by personal computers, workstations, clients, servers, and respective processing elements thereof.

FIGS.1A-1Dillustrates a plurality of graphs different types of forecasting system in accordance with at least one embodiment. For each of these graphs, the variables in the darker shade are used to predict those in the lighter shade. The darker variables are the predictor variables x. The lighter variables are the forecast variables y. The graphs illustrate time t as going in a downward direction, where t is the current time.

Graph100illustrates sample-based forecasting (SBF). This is a non-time series SBF regression method, which treats each future prediction separately. SBF makes forecasts with only predictor variables at each step and then moves to the next step.

Graph105illustrates recursive single-step forecasting (RSF). RSF makes a single-step prediction on current forecast variables using past information, then advances the time window and makes predictions recursively.

Graph110illustrates direct multi-step forecasting (DMF). DMF directly maps past information to multi-step future predictor variables.

As shown in graphs105and110, RSF and DMF do not utilize the knowledge of some future information for making forecasts.

Graph115illustrates Masked Multi-Step Multivariate Forecasting (MMMF). MMMF directly uses all available past and future information to predict all forecast variables. In MMMF, predictor variables x from both past and future are known, while forecast variables y are only known in the past. The time series data of known past predictor variables x and the known past forecast variables y are used to train the MMMF model. The time series data used in MMMF is often continuous. In training, MMMF replaces all masked variables with random values within the ranges of those variables. As explained further herein MMMF calculates the loss on only the masked outputs. In inference or prediction, MMMF uses the known future predictor variables x to determine the forecast variables y.

Graphs100,105,110, and115illustrate various solutions for a multivariate time series forecasting problem. In the multivariate time series forecasting problem, let xt∈nbe a sample of predictor variables x with dimension n at time t and the j-th dimension is denoted as xtj(i.e., xt=[xt1, xt2, . . . , xtn]). Let yt∈nbe a sample of forecast variables y with dimension m at time t (i.e., yt=[yt1, yt2, . . . , ytn]). The task of process200is to predict up to (k+1) steps (k>0) of forecast variables yt, yt+i, . . . , yt+kfrom past T-step information and some knowledge about the future predictor variables x up to time t+k.

A distinct feature of this problem formulation is the need to incorporate future information into the predictions directly. For example, when forecasting electric demand for a particular region over the next month, the calendar variables (date, month, day of week, etc.) and weather forecasts are known.

Formally, the MMMF method directly models the following relationships:

Traditionally, there are three most common machine learning formulations for modeling such a multi-step multivariate forecasting problem.

First is the sample-based forecasting (SBF) approach shown in graph100. This formulation treats each step as a distinct sample, and learns a function that maps the predictor variables to forecast variables directly without considering the temporal de-pendency, i.e., they model the following relationship:

This non-time series direct mapping from input to output could use any traditional regression models, e.g., Linear Regression, fully connected neural networks, etc. How-ever, it falls apart if there are no predictor variables but only forecast variables. Another disadvantage of this approach is that the temporal information is lost and recent history would not affect the forecasts.

Second, is the recursive single-step forecasting (RSF) approach shown in graph105. This formulation is the standard next step prediction (NSP) task for a time series, where during training a one-step forward prediction model is learned, i.e., the loss is only calculated on the next step. That learned model is then being applied recursively during inference, i.e.:

RSF does use all future information during training because the task is simply NSP. The major disadvantage of this formulation is that it makes predictions based on previous predictions, thus compounding errors will grow with the increasing number of steps.

Third is the direct multi-step forecasting (DMF) approach shown in graph110. This formulation directly generates multiple outputs for all future steps of forecast variables in a time series, given past information, i.e.:

DMF does not utilize the known future information and they simply map the past information to future predictions. Many base models for RSF, such as recurrent neural networks, could be reused for DMF. The difference is how the outputs of those models are mapped, i.e., to 1-step future versus multi-step future forecast variables.

These traditional techniques mainly suffer from two categories of issues. On one hand, SBF does not consider the temporal components and thus could perform poorly when the forecasting horizon is short. On the other hand, RSF and DMF do not utilize the knowledge of some future information for making forecasts. MMMF is proposed to take advantage of both information from the past and the future to make better forecasts.

It should be noted that the goal of this formulation is not to evaluate how good the future predictor variables are, but instead to develop a framework that could assimilate them regardless of how they are generated and use them to predict forecast variables. In real-world scenarios, some predictor variables are clearly defined and deterministic, like day of week, while others come with some uncertainty, like weather forecasts for the next month. The gap in existing formulations that MMMF addresses is that they cannot properly incorporate the known future information. Therefore, MMMF is a more general time series modeling framework than traditional regression models, and if there is no known future information, MMMF reduces to an autoregressive-like masked DMF model.

To solve this multi-step multivariate forecasting problem, the MMMF system600(as shown inFIG.6) uses process200for MMMF training as described below.

FIG.2illustrates block diagram of a process200for training a model for masked multi-step multivariate forecasting in accordance with at least one embodiment. In the exemplary embodiment, process200is performed by the Masked Multi-Step Multivariate Forecasting (MMMF) server610(shown inFIG.6).

The MMMF server610receives a plurality of time-series data205. In the exemplary embodiment, the time-series data is continuous over a significant period of time. Examples of the time-series data205include, but are not limited to, daily weather readings, fuel prices, stock values, economic indicators, and/or other daily information covering a significant period of time, such as one to two years. The plurality of time-series data205includes both predictor variables x and forecast variables y for

In other embodiments, the time-series data205may include, but is not limited to, sensor reading being made on a periodic basis, including, but not limited to, once a day, once an hour, once a minute, once a second, and/or any other periodic basis.

The MMMF server610selects a sliding window210of the data205to analyze. In the exemplary embodiment, the MMMF server610selects a window210of90continuous readings from the data205. The data205from that window210is considered the active time series dataset215. In some further embodiments, the MMMF server610acts in batches and selects a plurality of sliding windows210to determine a plurality of time series datasets215.

The MMMF server610takes the dataset(s)215and divides them into past information220and future information225. The division into past information220and future information225is randomized in that the MMMF servers610determines a length of the future information225between one and 60 readings, while the past information220is the remaining information. For example, the MMMF server610may determine a length of 30 readings for the future information225, which will be the last 30 readings and the past information220could be the 60 readings before the future information. In at least one embodiment, the size of the future information225is limited to two-third of the total time series dataset215. In the embodiments, where the MMMF server610is dealing with multiple datasets215at once, each dataset215may have the same size for future information225. In other embodiments, each dataset215may have different sizes for the future information225. In the exemplary embodiment, for each pass of process200, the size of the future information225changes between passes.

The MMMF server610applies a mask230to the future information225. The mask230is applied to the forecast variables y for all of the data points in the future information225. In some embodiments, masking techniques may include, but are not limited to, (i) replace with random numbers, (ii) replace with all zeros, (iii) replace with all ones, and/or any other values as appropriate.

The MMMF server610applies the past information220, which includes predictor variables x and forecast variables y, and masked future information220, which includes predictor variables x and the forecast variables y have been masked, into a time series model235that is being trained to determine the forecast variable y. The MMMF server230has the time series model235generate predictions240based on the past information220and the masked future information225.

The MMMF server610compares the predictions240for the forecast variables y and the forecast variables y for the same data points in the actual time series dataset215to calculate losses245. Based on the differences, the MMMF server610trains the model parameters250. In the exemplary embodiment, the MMMF server610adjusts the weights of one or more of the model parameters250based on the differences.

The MMMF server610then restarts process200by selecting a new sliding window210for the data205to execute on the updated time series model235. In some embodiments, the MMMF server610continues training the time series model235with time series datasets215from windows210of data205until one or more ending conditions occur. One example ending condition is that the calculated losses245are below a threshold. In some embodiments, the MMMF server610ends the training and process200when the calculated losses245stay below the threshold for a predetermined number of passes of process200. In other embodiments, the MMMF server610ends the training and process200when the calculated losses245do not change for a predetermined number of passes of process200. In still further embodiments, the MMMF server610ends the training and process200when the calculated losses245do not change by more than a predetermined threshold amount for a predetermined number of passes of process200.

Another view of process200is a solution for a multivariate time series forecasting problem. Let xt∈nbe a sample of predictor variables x with dimension n at time t and the j-th dimension is denoted as xtj(i.e., xt=[xt1, xt2, . . . , xtn]). Let yt∈nbe a sample of forecast variables y with dimension m at time t (i.e., yt=[yt1, yt2, . . . , ytn]). The task of process200is to predict up to (k+1) steps (k>0) of forecast variables yt, yt+i, . . . , yt+kfrom past T-step information and some knowledge about the future predictor variables x up to time t+k.

A distinct feature of this problem formulation is the need to incorporate future information into the predictions directly. For example, when forecasting electric demand for a particular region over the next month, the calendar variables (date, month, day of week, etc.) and weather forecasts are known.

Formally, the MMMF method directly models the following relationships:

To solve this multi-step multivariate forecasting problem, the MMMF system600(as shown inFIG.6) uses process200for MMMF training as described below.

The method for MMMF training is given in Algorithm 1 below. The time series model ƒθin this algorithm (or the base model for MMMF shown in process200) can be any neural network model that generates a sequence of outputs. Therefore, one having skill in the art would understand that MMMF is not limited to one model but is a general learning task for all-time series NN (neural network) models.

Algorithm 1 for MMMF training may be used as shown in process200. The input includes time series model ƒθwith a set of trainable parameters 0, maximum forecasting horizon k, maximum history length T, loss function. The data includes time series dataset S={zi}={(xi, yi)}, where i represents the i-th time step, xiare the predictor variables, yiare the forecast variables. Step one of Algorithm 1 includes preprocessing dataset with a sliding window of length (T+k+1) to {zt−T, . . . , zt−1, zt, zt+1, . . . , zt+k} sequences, where ztis the sample at current step. The second step of Algorithm 1 includes initializing model parameters θ. While not at the end of training epochs and while not at the end of all mini-batches, the MMMF system600randomly choses a batch of sequence and then randomly chooses an integer mask length lmin for this current batch where 0<lm≤k+1. For each sequence in the mini-batch, the MMMF system600masks the last lmsteps of forecast variables ŷ. The MMMF system feeds the masked sequences to the model ƒθto generate estimations ŷ using information of x from both the past and future, and unmasked y. The MMMF system calculates loss only on the masked outputs for future predictions, such that Σi=k−lmi=k(yi, ŷi). The MMMF system600backpropogates and updates model parameters θ based on the calculated losses. The MMMF system600then returns to randomly choose a batch of sequences and repeats the subsequent steps. The steps repeats until at the end of the training epochs or when convergence occurs.

Algorithm 1 outputs a trained model ƒθ, which may be similar to trained time series model420(shown inFIG.4).

One of the key steps for Algorithm 1 is the random masking of the last lmsteps of the forecast variables y in the randomly chosen sequence. Because the sequence is chosen randomly for each mini-batch of data, this essentially creates many forecasting sub-tasks where at each iteration the base model ƒθis trying to forecast different length outputs. In one extreme, when lm=1, the MMMF system600reduces to a similar formulation as RSF, with the exception that the information of predictor variables y at time step t is also used. In the other extreme, when lm=k+1, the MMMF system600reduces to a similar formulation as DMF, with the exception that the information of predictor variables y from time step t to t+k is also used. From this perspective, the learning task of MMMF is more comprehensive than the traditional time series regression tasks, as well as the non-time series SBF regression task.

Different from Masked Language Models (MLMs) such as BERT (Bidirectional Encoder Representations from Transformers) where the tokens are discrete, the time series data is often continuous. Therefore, the MMMF system600replaces all masked variables with random values within the ranges of those variables. Different from autoencoders, the MMMF system600calculates the loss on only the masked outputs, which is similar to other masked approaches such as BERT, instead of the full reconstruction loss.

Because the MMMF-trained model has learned to generate different lengths of forecasts during its training process, it is very flexible during inference and could generate any length of forecasts from 1 to the maximum forecast horizon k. Fundamentally, the self-supervised learning approach learns a representation of the data by being able to fill in the blanks when some forecast variables are masked. This leads to the flexibility of MMMF-trained models during inference. This includes that they are not restricted to making fixed-length forecasts. This could potentially be useful in some real-world applications, e.g., when an electricity load demand forecast model is trained, it needs to be able to make both short-term forecasts for unit commitment and mid-term forecasts for fuel planning and maintenance planning. Instead of having multiple models for each application, an MMMF-trained model could do all of them.

Furthermore, since masking requires very little additional computational time, MMMF-trained models could generate forecasts at a similar speed as RSF and DMF approaches if they are using the same base model. Given the more complicated learning task, the training time is generally longer for MMMF, but in practice, the inference time is more important for real-world applications. That is to say, MMMF could generate better forecasts at the same speed and memory usage as RSF and DMF models, at the expense of a more difficult training task and longer training time.

FIG.3illustrates a computer-implemented process300for training a model for masked multi-step multivariate forecasting using the process200(shown inFIG.2). In the exemplary embodiment, process300is performed by the Masked Multi-Step Multivariate Forecasting (MMMF) server610(shown inFIG.6).

In the exemplary embodiment, the MMMF server610stores305a plurality of historical time series data205(shown inFIG.2) including a plurality of predictor variables and a plurality of forecast variables.

The MMMF server610randomly selects310a sequence including a subset of continuous data points in the plurality of historical time series data205. In some embodiments, the sequence is similar to the time series dataset215(shown inFIG.2). The MMMF server610randomly selects the sequence including a subset of continuous data points in the plurality of historical time series data. The each randomly selected sequence is different, such that a first selected sequence in a first pass is different than a second selected sequence in a second pass. In the exemplary embodiment, the plurality of historical time series data205is significantly larger than the selected sequence or time series dataset215. For example, the selected sequence may include 90 days worth of data points, while the plurality of historical time series data205includes one or more years of data points.

The MMMF server610randomly selects315a mask length for a mask230(shown inFIG.2) for the selected sequence. The mask length determines how many data points in the selected sequence will be masked out. The MMMF server610applies320the mask230to the selected sequence. The mask230is applied to the end of the selected sequence. The masked selected sequence includes unmasked forecast variables followed by masked forecast variables.

The mask230is applied320to the end of the plurality of forecast variables in the selected sequence. For example, if the sequence has 60 data points and the mask length is 30, then the forecast variables y associated with the last 30 data points in the sequence will be masked. The forecast variables associated with the first 30 data points and all of the predictor variables x are not masked.

The MMMF server610executes325a model235(shown inFIG.2) with the masked selected sequence to generate predictions240(shown inFIG.2) for the masked forecast variables. The model235takes the predictor variables and the unmasked forecast variables and generates values for the masked predictor values.

The MMMF server610compares330the predictions for the masked forecast variables to the actual forecast variables in the selected sequence. In some embodiments, the MMMF server610determines a difference between the masked forecast variable and the forecast variable prior to masking for each masked forecast variable. In at least one embodiment, the MMMF server610calculates a loss function245(shown inFIG.2) based on the plurality of differences. In some embodiments, the loss function245includes at least one of, but is not limited to, means square error (MSE), means absolute percentage error (MAPE), means absolute percentage deviation (MAPD), mean absolute scale error (MASE), symmetric mean absolute percentage error (sMAPE), Mean Directional Accuracy (MDA), and/or any other error or loss function needed.

The MMMF server610determines335if convergence occurs based upon the comparison, In some additional embodiments, the MMMF server610determines that convergence has occurred if the loss function245is below a threshold. The MMMF server610can also determine that convergence has occurred if a value of the loss function has not changed in a predetermined number of passes. The MMMF server610can further determine that convergence has occurred if an amount of change of the loss function245has not exceeded a threshold or if an amount of change of the loss function245has not exceeded a threshold for a predetermined number of passes. In still additional embodiments, the MMMF server610determines that convergence has occurred after a predetermined plurality of passes through the algorithm.

If convergence has not occurred, the MMMF server610updates640one or more parameters250(shown inFIG.2) of the model235and returns to Step310for another pass through process300.

In some embodiments, the predictor variables include at least one of date, time, weather conditions. In some further embodiments, the forecast variables include electricity demand.

In the exemplary embodiment, when convergence has occurred and the model235has been trained. The model235may be used for inference predictions, where the model235predicts future values. The MMMF server610determine a future period of time to predict. This future period of time may be a number of days, hours, minutes, weeks, or any other period of time. The limitation on the period of time is that its maximum is the maximum amount of time in the future that the MMMF system600has predictor variable data for.

The MMMF server610selects a plurality of historical data points that precede the future period of time to predict. For example, if the future period of time is 30 days, the historical data points may go back 60 days. The plurality of historical data points includes predictor variables and forecast variables for those 60 days. The MMMF server610determines predictor variables for the future period of time to predict. In at least one example, the predictor variables include weather information. The MMMF server610executes the model235with the plurality of historical data points and the predictor variables for the future period of time to generate forecast variables for the future period of time. In some embodiments, the MMMF server610masks the forecast variables for the future period of time.

While processes200and300are described in regards to energy grid predictions based on weather, one having skill in the art would understand that the systems and methods described herein can also be applied to other types of prediction.

FIG.4illustrates block diagram of a process for prediction using a masked multi-step multivariate forecasting model in accordance with at least one embodiment. In the exemplary embodiment, process400is performed by the Masked Multi-Step Multivariate Forecasting (MMMF) server610(shown inFIG.6).

Another major advantage of the MMMF learning task, which uses all the rest information to forecast the variable-length masked variables, is that once trained a base neural network model can generate forecasts for any forecast length lffor 0<lƒ<k+1, by simply masking the last lƒsteps of the desired forecast variables.

In process400, the MMMF server610combines known past information405with partially known future information410. The known past information405includes predictor variables x and forecast variables y for a period of time before the present time t. The partially known future information410includes predictor variables x for a period of time after the present time t. For example, the past known information405could include weather and date information for the predictor variables x and electrical grid usage for the forecast variables y for a period of time, such as 60 days, prior to the present day t. The partially known future information410includes weather forecasts and date information for the predictor variables x for a second period of time, such as 30 days, subsequent to the present day t, where the predictor variables y are unknown. The MMMF server610fills415the unknown predictor variables y in the partially known future information410with a mask, which may be similar to the mask230(shown inFIG.2). The mask may include all ones, all zeros, random values, and/or any other values as appropriate.

The MMMF server610provides the known past information405and the masked partially known information410are provided to the trained time series model420. The trained time series model420uses the known past information405and the masked partially known information410as inputs to execute. The trained time series model420executes and generates multi-step forecasts425as outputs. The multi-step forecasts425include the predictor variables y for the second period of time subsequent to the present time t. In the example case, the predictor variables y include the predicted values for the electrical grid usage for the 30 days after the present time t.

In the exemplary embodiment, the trained time series model420may be used for generating multi-step forecasts for time periods where some information is known, where the trained time series model420is trained to fill in the unknown information based on the past information and the partially known information.

The method for MMMF inference is given in Algorithm 2 below. The trained time series model ƒθin this algorithm (or the base model for MMMF shown in process400) can be any neural network model that generates a sequence of outputs. Therefore, one having skill in the art would understand that MMMF is not limited to one model but is a general learning task for all-time series NN (neural network) models.

Algorithm 2 for MMMF inference may be used as shown in process400. The input includes MMMF-trained time series model ƒθ, forecast horizon lƒwhere 0<lƒ<k+1. The trained time series model ƒθis similar to trained time series model420. The data includes a sequence of length (T+k+1), with all predictor variables x known, and the last lƒ-step forecast variables y unknown. In the first step of Algorithm 2, the MMMF system600fills the last lƒ-step forecast variables y with a mask, such as mask230. Then the MMMF system600provides the masked sequence to the trained time series model ƒθ. The MMMF system600executes the trained time series model ƒθto generate forecasts ŷ for the last lƒsteps.

The MMMF system600and the trained time series trained time series model ƒθoutput a multi-step multivariate forecast for forecast variables y of length lƒ, which is similar to multi-step forecasts425.

FIG.5illustrates a process for training a model for masked multi-step multivariate forecasting using the process shown inFIG.4. In the exemplary embodiment, process500is performed by the Masked Multi-Step Multivariate Forecasting (MMMF) server610(shown inFIG.6).

In the exemplary embodiment, the MMMF server610determines505a future period of time to predict. In some embodiments, the future period of time to predict is based on a quantity of predictor variables x available for the future period of time. For example, where the predictor variables x are weather forecasts, and the quality of weather forecasting may decrease the further away in time that it is. Therefore, the future period of time may be limited, such as to 30 days and/or any other period of time that the user desires.

In the exemplary embodiment, the MMMF server610selects510a plurality of historical data points that precede the further period of time to predict. In some embodiments, the plurality of historical data points are from a period of time before the current time that is greater than the future period of time. In one example, if the future period of time is 30 days, the plurality of historical data points covers 60 days prior to the current time. In at least one embodiment, the plurality of historical data points are similar to the known past information405(shown inFIG.4).

In the exemplary embodiment, the MMMF server610determines515predictor variables x for the future period of time. The predictor variables x may be weather forecasts as described herein. The predictor variables x are filled in for each of the days or data points to be analyzed. In at least one embodiment, the predictor variables x for the future period of time are similar to the partially known future information410(shown inFIG.4).

In the exemplary embodiment, the MMMF server610executes520the trained time series model420with the plurality of historical data points and the predictor variables x for the future period of time to generate forecast variables y for the future period of time.

In some embodiments, the MMMF server610masks320the forecast variables y for the future period of time.

While processes400and500are described in regards to energy grid predictions based on weather, one having skill in the art would understand that the systems and methods described herein can also be applied to other types of prediction.

FIG.6depicts a simplified block diagram of an exemplary computer system600for implementing processes200,300,400, and500shown inFIGS.2,3,4, and5. In the exemplary embodiment, system600may be used for predicting future performance of systems, such as power generation systems. As described below in more detail, a masked multi-step multivariate forecasting (MMMF) computer device610(also known as MMMF server610) may be configured to (a) store a plurality of historical time series data205(shown inFIG.2) including a plurality of predictor variables and a plurality of forecast variables; (b) randomly select a sequence including a subset of continuous data points in the plurality of historical time series data205; (c) randomly select a mask length for a mask230(shown inFIG.2) for the selected sequence; (d) apply the mask230to the selected sequence, wherein the mask230is applied to the plurality of forecast variables in the selected sequence; (e) execute a model235(shown inFIG.2) with the masked selected sequence to generate predictions240(shown inFIG.2) for the masked forecast variables; (f) compare the predictions240for the masked forecast variables to the actual forecast variables in the selected sequence; (g) determine if convergence occurs based upon the comparison; and (h) if convergence has not occurred, update one or more parameters250(shown inFIG.2) of the model235and return to step b.

In the exemplary embodiment, client computer devices605are computers that include a web browser or a software application, which enables client computer devices605to access MMMF server610using the Internet. More specifically, client computer devices605are communicatively coupled to the Internet through many interfaces including, but not limited to, at least one of a network, such as the Internet, a local area network (LAN), a wide area network (WAN), or an integrated services digital network (ISDN), a dial-up-connection, a digital subscriber line (DSL), a cellular phone connection, and a cable modem. Client computer devices605may be any device capable of accessing the Internet including, but not limited to, a mobile device, a desktop computer, a laptop computer, a personal digital assistant (PDA), a cellular phone, a smartphone, a tablet, a phablet, wearable electronics, smart watch, virtual headsets or glasses (e.g., AR (augmented reality), VR (virtual reality), or XR (extended reality) headsets or glasses), chat bots, or other web-based connectable equipment or mobile devices.

A database server615may be communicatively coupled to a database620that stores data. In one embodiment, database620may include historical data, further information, models, model parameters, and other information. In the exemplary embodiment, database620may be stored remotely from MMMF server610. In some embodiments, database620may be decentralized. In the exemplary embodiment, a person may access database620via client computer devices605by logging onto MMMF server610, as described herein.

MMMF server610may be communicatively coupled with one or more the client computer devices605. In some embodiments, MMMF server610may be associated with, or is part of a computer network associated with grid operation, or in communication with the grid operation's computer network (not shown). In other embodiments, MMMF server610may be associated with a third party and is merely in communication with the grid operation's computer network.

One or more future information servers625may be communicatively coupled with MMMF server610via the Internet or a local network. More specifically, future information servers625are communicatively coupled to the Internet through many interfaces including, but not limited to, at least one of a network, such as the Internet, a local area network (LAN), a wide area network (WAN), or an integrated services digital network (ISDN), a dial-up-connection, a digital subscriber line (DSL), a cellular phone connection, and a cable modem. Future information servers625may be any device capable of accessing the Internet including, but not limited to, a mobile device, a desktop computer, a laptop computer, a personal digital assistant (PDA), a cellular phone, a smartphone, a tablet, a phablet, wearable electronics, smart watch, virtual headsets or glasses (e.g., AR (augmented reality), VR (virtual reality), or XR (extended reality) headsets or glasses), chat bots, or other web-based connectable equipment or mobile devices. In the exemplary embodiments, future information servers625provide information about future values for x, such as, but not limited to, weather information, gas prices, economic indicators, stock values, population information, demand information, and/or any other information used for modeling future performance.

FIG.7depicts an exemplary configuration of client computer devices, in accordance with one embodiment of the present disclosure. User computer device702may be operated by a user701. User computer device702may include, but is not limited to, client computer device605and MMMF computer device610(shown inFIG.6).

User computer device702may include a processor705for executing instructions. In some embodiments, executable instructions are stored in a memory area710. Processor705may include one or more processing units (e.g., in a multi-core configuration). Memory area710may be any device allowing information such as executable instructions and/or transaction data to be stored and retrieved. Memory area710may include one or more computer readable media.

User computer device702may also include at least one media output component715for presenting information to user701. Media output component715may be any component capable of conveying information to user701. In some embodiments, media output component715may include an output adapter (not shown) such as a video adapter and/or an audio adapter. An output adapter may be operatively coupled to processor705and operatively coupleable to an output device such as a display device (e.g., a cathode ray tube (CRT), liquid crystal display (LCD), light emitting diode (LED) display, or “electronic ink” display) or an audio output device (e.g., a speaker or headphones).

In some embodiments, media output component715may be configured to present a graphical user interface (e.g., a web browser and/or a client application) to user701. A graphical user interface may include, for example, results of forecasting. In some embodiments, user computer device702may include an input device720for receiving input from user701. User701may use input device720to, without limitation, select a forecast variable to analyze and/or select a time frame to analyze.

Input device720may include, for example, a keyboard, a pointing device, a mouse, a stylus, a touch sensitive panel (e.g., a touch pad or a touch screen), a gyroscope, an accelerometer, a position detector, a biometric input device, and/or an audio input device. A single component such as a touch screen may function as both an output device of media output component715and input device720.

User computer device702may also include a communication interface725, communicatively coupled to a remote device such as MMMF server610(shown inFIG.6). Communication interface725may include, for example, a wired or wireless network adapter and/or a wireless data transceiver for use with a mobile telecommunications network.

Stored in memory area710are, for example, computer readable instructions for providing a user interface to user701via media output component715and, optionally, receiving and processing input from input device720. A user interface may include, among other possibilities, a web browser and/or a client application. Web browsers enable users, such as user701, to display and interact with media and other information typically embedded on a web page or a website from MMMF server610. A client application allows user701to interact with, for example, time frames and forecasting results. For example, instructions may be stored by a cloud service, and the output of the execution of the instructions sent to the media output component715.

Processor705executes computer-executable instructions for implementing aspects of the disclosure. In some embodiments, the processor705is transformed into a special purpose microprocessor by executing computer-executable instructions or by otherwise being programmed. For example, the processor705may be programmed with the instructions such as processes200,300,400, and500(shown inFIGS.2,3,4, and5, respectively).

FIG.8illustrates an example configuration of the server system, in accordance with one embodiment of the present disclosure. Server computer device801may include, but is not limited to, MMMF server610, database server615, and future information server625(all shown inFIG.6). Server computer device801also includes a processor805for executing instructions. Instructions may be stored in a memory area810. Processor805may include one or more processing units (e.g., in a multi-core configuration).

Processor805is operatively coupled to a communication interface815such that server computer device801is capable of communicating with a remote device such as another server computer device801, MMMF server610, client computer device605(shown inFIG.6), or future information server625. For example, communication interface815may receive requests from client computer devices605via the Internet.

Processor805may also be operatively coupled to a storage device834. Storage device834is any computer-operated hardware suitable for storing and/or retrieving data, such as, but not limited to, data associated with database620(shown inFIG.6). In some embodiments, storage device834is integrated in server computer device801. For example, server computer device801may include one or more hard disk drives as storage device834. In other embodiments, storage device834is external to server computer device801and may be accessed by a plurality of server computer devices801. For example, storage device834may include a storage area network (SAN), a network attached storage (NAS) system, and/or multiple storage units such as hard disks and/or solid state disks in a redundant array of inexpensive disks (RAID) configuration.

In some embodiments, processor805is operatively coupled to storage device834via a storage interface820. Storage interface820is any component capable of providing processor805with access to storage device834. Storage interface820may include, for example, an Advanced Technology Attachment (ATA) adapter, a Serial ATA (SATA) adapter, a Small Computer System Interface (SCSI) adapter, a RAID controller, a SAN adapter, a network adapter, and/or any component providing processor805with access to storage device834.

Processor805executes computer-executable instructions for implementing aspects of the disclosure. In some embodiments, the processor805is transformed into a special purpose microprocessor by executing computer-executable instructions or by otherwise being programmed. For example, the processor805is programmed with the instructions such as processes200,300,400, and500(shown inFIGS.2,3,4, and5, respectively).

FIG.9illustrates a graph900of comparing different forecasting techniques in accordance with at least one embodiment. Graph900illustrates mid-Term Forecasting Results. Graph900illustrates the means absolute percentage error (MAPE) for different forecasting styles based on the number of days into the future that are being forecast. Line905illustrates the MAPE for short-term forecasting (STF). Line910illustrates the MAPE for long-term forecasting (LTF). And line915illustrates the MAPe for the MMMF system600described herein.

FIG.10illustrates another graph1000of comparing different forecasting techniques in accordance with at least one embodiment. Graph1000illustrates the means absolute percentage error (MAPE) for a single step prediction. Graph100shows significantly more error for short-term forecasting (STF) than for the MMMF system600described herein.

In at least one embodiment, the MMMF system600is provided. The MMMF system600includes a MMMF computing device610including at least one processor805in communication with at least one memory device810. The at least one processor805is programmed to perform a plurality of steps. The MMMF computing device610is programmed to store305a plurality of historical time series data205including a plurality of predictor variables and a plurality of forecast variables. The MMMF computing device610is also programmed to randomly select310a sequence including a subset215of continuous data points in the plurality of historical time series data205. The MMMF computing device610is further programmed to randomly select315a mask length for a mask230for the selected sequence215. In addition, the MMMF computing device610is programmed to apply320the mask230to the selected sequence215. The mask230is applied320to the plurality of forecast variables in the selected sequence215. Furthermore, the MMMF computing device610is programmed to execute325a model235with the masked selected sequence215to generate predictions240for the masked forecast variables. Moreover, the MMMF computing device610is programmed to compare330the predictions240for the masked forecast variables to the actual forecast variables in the selected sequence215. In addition, the MMMF computing device610also is programmed to determine335if convergence occurs based upon the comparison. If convergence has not occurred, the MMMF computing device610is programmed to update340one or more parameters250of the model235and return to step310.

In an embodiment, the MMMF computing device610compares the predictions240for the masked forecast variables to the actual forecast variables in the selected sequence by determining a difference between the masked forecast variable and the forecast variable prior to masking for each masked forecast variable.

In another embodiment, the MMMF computing device610is further programmed to calculate a loss function245based on the plurality of differences. The loss function includes at least one of means square error (MSE) and means absolute percentage error (MAPE). The MMMF computing device610is programmed to determine that convergence has occurred if the loss function245is below a threshold. The MMMF computing device610is further programmed to determine that convergence has occurred if a value of the loss function245has not changed in a predetermined number of passes. In addition, the MMMF computing device610is programmed to determine that convergence has occurred if an amount of change of the loss function245has not exceeded a threshold. Furthermore, the MMMF computing device610is programmed to determine that convergence has occurred if an amount of change of the loss function245has not exceeded a threshold for a predetermined number of passes. Moreover, the MMMF computing device610is programmed to determine that convergence has occurred after a predetermined plurality of passes through the algorithm.

The MMMF computing device610is further programmed to randomly select the sequence215including a subset of continuous data points in the plurality of historical time series data205. A first selected sequence in a first pass is different than a second selected sequence in a second pass. The plurality of historical time series data205is significantly larger than the selected sequence205.

In further embodiments, the masked selected sequence includes unmasked forecast variables followed by masked forecast variables. In some embodiments, the predictor variables include at least one of date, time, weather conditions. In further embodiments, the forecast variables include electricity demand.

In a further embodiment, the MMMF computing device610is programmed to determine505a future period of time to predict. The MMMF computing device610is programmed to select510a plurality of historical data points405that precede the future period of time to predict. The plurality of historical data points405includes predictor variables and forecast variables. The MMMF computing device610determines515predictor variables for the future period of time to predict. The MMMF computing device610executes520the model420with the plurality of historical data points405and the predictor variables for the future period of time to generate forecast variables425for the future period of time. The MMMF computing device610is also programmed to mask415the forecast variables for the future period of time. The mask230is applied to the end of the selected sequence215.

In another embodiment, a computer-implemented method300is implemented by the MMMF computing device610including at least one processor805in communication with at least one memory device810. The method300includes storing305a plurality of historical time series data205including a plurality of predictor variables and a plurality of forecast variables. The method300also includes randomly selecting310a sequence215including a subset of continuous data points in the plurality of historical time series data205. The method300further includes randomly selecting315a mask length for a mask230for the selected sequence. In addition, the method300includes applying320the mask230to the selected sequence215. The mask230is applied to the plurality of forecast variables in the selected sequence215. Moreover, the method300includes executing325a model235with the masked selected sequence215to generate predictions240for the masked forecast variables. Furthermore, the method300includes comparing330the predictions for the masked forecast variables to the actual forecast variables in the selected sequence215. In addition, the method300includes determining335if convergence occurs based upon the comparison. If convergence has not occurred, the method300includes updating340one or more parameters250of the model235and return to step310.

In a further embodiment, the method300includes determining a difference between the masked forecast variable and the forecast variable prior to masking for each masked forecast variable. The method300also includes calculating a loss function245based on the plurality of differences. The method300further includes determining that convergence has occurred if the loss function245is below a threshold, if a value of the loss function245has not changed in a predetermined number of passes, if an amount of change of the loss function245has not exceeded a threshold, if an amount of change of the loss function245has not exceeded a threshold for a predetermined number of passes, or after a predetermined plurality of passes through the algorithm300.

In still a further embodiment, a method500includes determining505a future period of time to predict. The method500also includes selecting510a plurality of historical data points405that precede the future period of time to predict. The plurality of historical data points405includes predictor variables and forecast variables. The method further includes determining515predictor variables for the future period of time to predict. In addition, the method includes executing520the model420with the plurality of historical data points405and the predictor variables for the future period of time to generate forecast variables425for the future period of time.

At least one of the technical solutions to the technical problems provided by this system may include: (i) improved accuracy in short and mid-term forecasting; (ii) significantly reduced error in forecasting; (iii) ability to forecast based on historical and future data; (iv) forecasting not limited based on neural network used; and (v) able to be applied to multiple different models for training.

The methods and systems described herein may be implemented using computer programming or engineering techniques including computer software, firmware, hardware, or any combination or subset thereof, wherein the technical effects may be achieved by performing at least one of the following steps: a) store a plurality of historical time series data including a plurality of predictor variables and a plurality of forecast variables, wherein the predictor variables include at least one of date, time, weather conditions, wherein the forecast variables include electricity demand; b) randomly select a sequence including a subset of continuous data points in the plurality of historical time series data; c) randomly select a mask length for a mask for the selected sequence; d) apply the mask to the selected sequence, wherein the mask is applied to the plurality of forecast variables in the selected sequence, wherein the mask is applied to the end of the selected sequence, wherein the masked selected sequence includes unmasked forecast variables followed by masked forecast variables; e) execute a model with the masked selected sequence to generate predictions for the masked forecast variables; f) compare the predictions for the masked forecast variables to the actual forecast variables in the selected sequence; g) determine if convergence occurs based upon the comparison; h) if convergence has not occurred, update one or more parameters of the model and return to step b; i) for each masked forecast variable, determine a difference between the masked forecast variable and the forecast variable prior to masking; j) calculate a loss function based on the plurality of differences, wherein the loss function includes at least one of means square error (MSE) and means absolute percentage error (MAPE); k) determine that convergence has occurred if the loss function is below a threshold; l) determine that convergence has occurred if a value of the loss function has not changed in a predetermined number of passes; m) determine that convergence has occurred if an amount of change of the loss function has not exceeded a threshold; n) determine that convergence has occurred if an amount of change of the loss function has not exceeded a threshold for a predetermined number of passes; o) determine that convergence has occurred after a predetermined plurality of passes through the algorithm; p) determine a future period of time to predict; q) select a plurality of historical data points that precede the future period of time to predict, wherein the plurality of historical data points includes predictor variables and forecast variables; r) determine predictor variables for the future period of time to predict, wherein the at least one processor is further programmed to mask the forecast variables for the future period of time; s) execute the model with the plurality of historical data points and the predictor variables for the future period of time to generate forecast variables for the future period of time; and t) randomly select the sequence including a subset of continuous data points in the plurality of historical time series data, wherein a first selected sequence in a first pass is different than a second selected sequence in a second pass, wherein the plurality of historical time series data is significantly larger than the selected sequence.

The computer-implemented methods and processes described herein may include additional, fewer, or alternate actions, including those discussed elsewhere herein. The present systems and methods may be implemented using one or more local or remote processors, transceivers, and/or sensors (such as processors, transceivers, and/or sensors mounted on computer systems or mobile devices, or associated with or remote servers), and/or through implementation of computer-executable instructions stored on non-transitory computer-readable media or medium. Unless described herein to the contrary, the various steps of the several processes may be performed in a different order, or simultaneously in some instances.

Additionally, the computer systems discussed herein may include additional, less, or alternate functionality, including that discussed elsewhere herein. The computer systems discussed herein may include or be implemented via computer-executable instructions stored on non-transitory computer-readable media or medium.

A processor or a processing element may employ artificial intelligence and/or be trained using supervised or unsupervised machine learning, and the machine learning program may employ a neural network, which may be a convolutional neural network, a deep learning neural network, or a combined learning module or program that learns in two or more fields or areas of interest. Machine learning may involve identifying and recognizing patterns in existing data in order to facilitate making predictions for subsequent data. Models may be created based upon example inputs in order to make valid and reliable predictions for novel inputs.

In supervised machine learning, a processing element may be provided with example inputs and their associated outputs, and may seek to discover a general rule that maps inputs to outputs, so that when subsequent novel inputs are provided the processing element may, based upon the discovered rule, accurately predict the correct output. In unsupervised machine learning, the processing element may be required to find its own structure in unlabeled example inputs. In one embodiment, machine learning techniques may be used to extract data about the computer device, the user of the computer device, the computer network hosting the computer device, services executing on the computer device, and/or other data.

Based upon these analyses, the processing element may learn how to identify characteristics and patterns that may then be applied to training models, analyzing sensor data, and detecting abnormalities.

In another embodiment, a computer program is provided, and the program is embodied on a computer-readable medium. In an example embodiment, the system is executed on a single computer system, without requiring a connection to a server computer. In a further example embodiment, the system is being run in a Windows® environment (Windows is a registered trademark of Microsoft Corporation, Redmond, Washington). In yet another embodiment, the system is run on a mainframe environment and a UNIX® server environment (UNIX is a registered trademark of X/Open Company Limited located in Reading, Berkshire, United Kingdom). In a further embodiment, the system is run on an iOS® environment (iOS is a registered trademark of Cisco Systems, Inc. located in San Jose, CA). In yet a further embodiment, the system is run on a Mac OS® environment (Mac OS is a registered trademark of Apple Inc. located in Cupertino, CA). In still yet a further embodiment, the system is run on Android® OS (Android is a registered trademark of Google, Inc. of Mountain View, CA). In another embodiment, the system is run on Linux® OS (Linux is a registered trademark of Linus Torvalds of Boston, MA). The application is flexible and designed to run in various different environments without compromising any major functionality.