DETERMINING STRATEGIC DIGITAL CONTENT TRANSMISSION TIME UTILIZING RECURRENT NEURAL NETWORKS AND SURVIVAL ANALYSIS

Methods, systems, and non-transitory computer readable storage media are disclosed for determining and applying digital content transmission times using machine-learning. For example, in one or more embodiments, the disclosed system trains a recurrent neural network based on past electronic messages for a user that have been partitioned into a plurality of time bins. Additionally, in one or more embodiments, the system utilizes the trained recurrent neural network to generate predictions of engagement metrics (e.g., a hazard metric based on survival analysis or interaction probability metric) for sending a new electronic message within the plurality of time bins. The system then executes the digital content campaign by selecting a time bin based on the predicted engagement metrics and then sending the new electronic message at a send time corresponding to the selected time bin.

BACKGROUND AND RELEVANT ART

Recent years have seen significant improvement in computer systems for generating and executing digital content campaigns across computer networks to deliver digital content to client devices. Indeed, publishers now utilize various hardware and software platforms to generate digital content campaigns (e.g., campaigns comprising one or more digital design assets such as digital images, videos, and/or audio) and then implement the campaigns by distributing digital content to client computing devices. For example, publishers can utilize digital content systems to generate digital content campaigns for the distribution of digital media via email messaging to targeted computing devices of particular users.

Although conventional email digital content campaign systems can generate and disseminate digital content across computer networks, these systems have a number of shortcomings. For example, conventional email digital content campaign systems struggle to identify and implement accurate and efficient digital content send times (i.e., time of transmission). Inasmuch as digital content campaign systems provide digital content to computing devices instantaneously (or near-instantaneously) the digital content send time can have significant impacts on computing systems implementing digital content campaigns. For example, some conventional email digital content campaign systems transmit digital content to client devices at times when the digital content is unlikely to be accessed or acted upon thus resulting in transmission and storage of large volumes of digital content that users fail to open, access, or utilize.

Accordingly, implementation of inaccurate digital content transmission times can lead to unnecessary and inefficient waste of computing and networking resources. In particular, transmitting digital content at incorrect or inaccurate times often leads to duplication in digital content generation, transmission, and storage. Indeed, as mentioned, incorrect transmission times often leads to reduced access rates at client devices, resulting in unnecessary transmission and storage of digital content. Moreover, when digital content access rates are reduced, publishers often re-transmit digital content and/or expand the number of targeted computing devices to satisfy digital campaign requirements. This results in additional burdens on computer processing and networking assets for publisher and recipient computing devices/networks.

In addition, conventional email digital content campaign systems are also inflexible. For example, some digital content campaign systems predict a transmission time for digital messages based on factors determined by a campaign designer based on analysis of historical data. Although such systems can utilize these determined factors to select a transmission time for digital content, they are inflexible and still suffer from inaccuracies. Indeed, such systems are rigidly bound to selected factors generally identified by campaign designers (based on limited historical data at the time the campaign designer generates the campaign) and, therefore, are generally incapable of flexibly applying individualized features in deciding when to provide digital content to client devices.

These and other disadvantages exist with respect to conventional digital email content campaign systems.

SUMMARY

One or more embodiments provide benefits and/or solve one or more of the foregoing or other problems in the art with systems, methods, and non-transitory computer readable storage media for determining and applying strategic digital content transmission times using a recurrent neural network. In particular, the disclosed systems can utilize a recurrent neural network to flexibly and accurately determine and execute digital content transmission times within a digital content campaign based on one or more predicted engagement metrics for particular transmission time bins. For example, the disclosed systems can utilize the recurrent neural network to generate a predicted engagement metric that reflects a predicted time for user interaction, such as a hazard metric resulting from a survival analysis. Further, the disclosed systems can provide an option to flexibly optimize for various objectives, such as increasing open probability or decreasing time to open sent messages. Moreover, the disclosed systems can utilize the recurrent neural network to flexibility, and automatically, learn significant features, including features only apparent from sequential analysis of data from prior messages transmitted to a particular user.

To illustrate, in one or more embodiments, the disclosed system trains a recurrent neural network for a user using past electronic messages for the user that have been partitioned into a plurality of time bins based on corresponding transmission timestamps. The disclosed system uses the recurrent neural network to predict an engagement metric in connection with sending a new electronic message to a client device of the user. For example, the disclosed system can use the recurrent neural network to predict an interaction probability with the new electronic message for the user. Alternatively, the disclosed system can use the recurrent neural network with a survival function to predict a time-to-interact (e.g., hazard metric) with the new electronic message for the user. In one or more embodiments, the disclosed system uses the predicted engagement metric to determine a time bin and then executes a digital content campaign by sending the new electronic message during the determined time bin. By using a recurrent neural network trained for the user based on past electronic messages for the user to predict an engagement metric for a new electronic message, the disclosed system is able to determine a transmission time that optimizes the engagement metric, improves accuracy, reduces wasted resources, and increases flexibility.

DETAILED DESCRIPTION

One or more embodiments of the present disclosure include a campaign management system that determines transmission times for electronic messages in a digital content campaign utilizing a recurrent neural network. Specifically, the campaign management system utilizes a recurrent neural network to predict transmission times for new electronic messages by analyzing past electronic messages for a user in sequential order to generate a predicted engagement metric. By utilizing a recurrent neural network to predict engagement metrics for new electronic messages based on past electronic messages for the user, the campaign management system can improve the accuracy, flexibility, and efficiency of computing systems implementing digital content campaigns.

To illustrate, in one or more embodiments, the campaign management system trains a recurrent neural network for a group of users (e.g., a target audience) utilizing a plurality of past electronic messages that have been partitioned into a plurality of time bins. Using the recurrent neural network, the campaign management system generates a prediction of an engagement metric. Specifically, the campaign management system can predict, for each time bin, a hazard metric (e.g., a hazard metric from a survival analysis) or an interaction probability metric in connection with a new electronic message for a user in the group of users. Furthermore, the campaign management system can select a time bin from the time bins according to the predicted engagement metrics. The campaign management system can execute a digital content campaign by sending the new electronic message to a client device of the user based on the selected time bin.

As mentioned, the campaign management system trains a recurrent neural network to generate predicted engagement metrics for a user. In particular, in one or more embodiments, the campaign management system first partitions a plurality of past electronic messages into a plurality of time bins based on timestamps corresponding to the past electronic messages. Specifically, the campaign management system can partition past electronic messages into time bins (and adjust the size or time range for the time bins) to achieve a particular grouping proportion of the past electronic messages (e.g., time bins with an equal number of past electronic messages). In this manner, the campaign management system can control for publisher preferences in send times in determining an engagement metric for a particular user.

As just mentioned, the plurality of past electronic messages includes past electronic messages sent to a particular user (i.e., to a client device of the user). The campaign management system can use the past electronic messages sent to the user to train the recurrent neural network to determine important features specific to the user and the past messages sent to the user. For example, utilizing the recurrent neural network, the campaign management system can identify significant features based on sequential data from past campaigns and prior messages to the user.

Additionally, the campaign management system trains the recurrent neural network based on different engagement metrics. For instance, the campaign management system can train the recurrent neural network to output predictions for time-to-interact metric (e.g., hazard metric) or interaction probability metrics with respect to a given electronic message. When training for a time-to-interact with electronic messages, the campaign management system can use a recurrent neural network with a survival algorithm for outputting a hazard metric that indicates a measure of time for a user interaction. Similarly, when training for an interaction probability with electronic messages, the campaign management system can use a recurrent neural network without a survival algorithm (e.g., trained for interaction classification) that outputs an interaction probability metric that indicates a probability of a user interaction with an electronic message.

Upon training the recurrent neural network for the user and for a specific engagement metric, in one or more embodiments, the campaign management system uses the trained recurrent neural network to automatically generate predicted engagement metrics for a new electronic message to a particular user in a digital content campaign. In particular, the campaign management system can analyze each time bin and generate a predicted engagement metric corresponding to each time bin. Specifically, the campaign management can determine an engagement metric that indicates a predicted performance of the new electronic message if sent within the time bin. The campaign management system can rank the time bins based on the generated engagement metrics and then select one or more time bins for sending the new electronic message.

In one or more embodiments, the campaign management system executes a digital content campaign that includes a plurality of electronic messages based on one or more campaign parameters. For example, the campaign management system can use campaign parameters that limit the number of messages or frequency of messages to determine the transmission times for the electronic messages. Furthermore, the campaign management system can execute the campaign by scheduling and sending the messages according to the determined transmission times.

As mentioned, the campaign management system provides a number of advantages over conventional systems. For example, the campaign management system can improve the accuracy and efficiency of computing systems implementing digital design campaigns utilizing transmission times for digital content. In particular, the campaign management system can more accurately predict engagement metrics, such as a hazard metric (e.g., that indicates an instantaneous probability that a user will interact with digital content, allowing for transmission times that correspond with opening times) or an interaction probability metric (e.g., that indicates a probability of interaction with digital content, allowing for transmission times that will increase response). Increasing the accuracy of transmission times in relation to one or more engagement metrics results in more efficient use of computing resources in conducting digital content campaigns. In particular, the campaign management system can reduce the amount of digital content transmitted by a computing device across a computing network (e.g., the digital content required to satisfy digital content campaign requirements) and the amount of computer storage required to manage and store such digital content.

In addition, the campaign management system further improves flexibility of computing devices implementing digital content campaigns. For example, the campaign management system can utilize a recurrent neural network to automatically identify significant features for individual users. Indeed, the campaign management system can train a recurrent neural network to identify the unique features for modeling user responses for a specific user or group of users in selecting and implementing a transmission time as part of a digital content campaign. This includes features gleaned from sequential analysis of prior messages sent to the user (or group of users).

In addition, the campaign management system can automatically incorporate up-to-date (e.g., real-time) digital data regarding individual users in identifying significant features. For example, as the campaign management system detects individual user interactions with digital content, the campaign management system can update training of the recurrent neural network to identify any additional features indicated by the detected interactions. Thus, for instance, the campaign management system can analyze user responses to an electronic message sent the previous day before executing a digital content campaign. Conventional systems tend to ignore such data points because not much time has elapsed for customers to respond. Accordingly, the campaign management system can not only automatically identify significant features for modeling responses for individual users (or groups of users), but can automatically modify which features the system determines to be significant based on additional user interactions with electronic messages sent to the user(s).

Furthermore, the campaign management system also improves flexibility by providing for options to optimize various types of engagement metrics. For instance, as mentioned above, the campaign management system can allow for selection among various engagement metric types. For example, the campaign management system can allow for optimization based on increasing open probability or decreasing time to open.

As illustrated by the foregoing discussion, the present disclosure utilizes a variety of terms to describe features and advantages of the campaign management system. Additional detail is now provided regarding the meaning of such terms. For example, as used herein, the terms “electronic message” and “message” refer to an electronic communication that includes digital content. For example, an electronic message can include an email. An electronic message can include digital content such as digital text, digital images, or digital video associated with a digital content campaign. In addition, as used herein, the term “new electronic message” refers to a generated, proposed, future, or unsent electronic message for one or more recipient users. A new electronic message may include content that has previously been used, new content, or a combination of previously used/new content. Thus, a new electronic message can be a newly generated message or an already existing message (or previously sent message) to be re-sent as part of a digital content campaign.

Additionally, the terms “digital content campaign” and “campaign” refer to a series of activities, parameters, and/or rules used in connection with providing digital content corresponding to a product, service, group, individual, entity, or other category. For instance, a digital content campaign can include electronic messages comprising digital content about a product, service, group, individual, entity, or other category. Moreover, a digital content campaign can include campaign parameters such as a target audience, budget, or constraints for delivering digital content. Executing the digital content campaign, therefore, can include sending the electronic messages to a target audience that can include recipients that share one or more characteristics according to one or more campaign parameters.

As used herein, the term “recurrent neural network” refers to an artificial neural network (e.g., a neural network that performs analytical tasks for elements in a sequential order based on computations from previous elements). In particular, a recurrent neural network includes an artificial neural network that uses sequential information associated with a plurality of electronic messages within a chronological timeline, and in which an output of a current step is dependent on computations for previous steps. For example, the campaign management system trains and uses a recurrent neural network for a user to predict user interactions with electronic messages based on transmission times of the electronic messages and learned behaviors of the user. Specifically, a trained recurrent neural network sequentially analyzes electronic messages to learn important features that indicate a pattern of user interactions with the electronic messages (e.g., based on when the electronic messages were sent and preferences or tendencies of the user).

As used herein, the term “input attributes” refer to characteristics or traits of electronic messages that a recurrent neural network uses in predicting performance of electronic messages. In particular, input attributes can include information regarding electronic messages provided as input to a recurrent neural network. Specifically, input attributes for a message can include user interactions or send characteristics associated with one or more electronic messages message (e.g., a current electronic message or previously transmitted electronic messages). For example, input attributes can include analytics data including, but not limited to, user interactions with past electronic messages, user preferences, detected user actions, or message details or characteristics. The recurrent neural network can thus determine important features for predicting user interactions relative to electronic messages based on the input attributes. In one or more embodiments, input attributes for a message include a transmission time or time bin of the message. Input attributes of a message can also include properties of the transmission (e.g., content of the message or information sent with the message) such as a subject line of the message or images (e.g., banner images).

As described in greater detail below, a recurrent neural network can include a plurality of layers. The term refers to analytical elements within a neural network. In particular, a layer can include interconnected nodes containing an activation function. The recurrent neural network can utilize layers to conduct analysis at different levels to determine significant features corresponding to a particular output. To illustrate, the layers can include a dense layer (or fully connected layer) with an activation function and/or a survival layer with a survival function. As used herein, the term “survival layer” refers to a layer that utilizes a survival function to generate a hazard metric (e.g., a hazard rate or hazard ratio). As used herein, the term “survival function” refers to a function that is a property of a variable (e.g., message) that maps an event or set of events onto time. In particular, a survival function captures the probability that an analyzed message survives (e.g., no interaction with the message) beyond a given specified time. In one specific example, a survival function outputs a hazard metric, which can include a hazard rate or a hazard ratio, for an electronic message for a user.

As used herein, the term “engagement metric” refers to a measurement of user interactions corresponding to an electronic message of a digital content campaign. An engagement metric can include a hazard metric that indicates an instantaneous probability of a user interaction with an electronic message for a user for a given time bin. For example, as used herein, a “hazard metric” includes the instantaneous probability that, for an electronic message sent within a time bin (and given a recipient has yet to open/interact with the message by time t), the recipient will open the electronic message instantaneously after time t. Alternatively, an engagement metric can include an interaction probability. As used herein, an “interaction probability” refers to the probability of a user interacting with an electronic message if the message is sent within a given time bin.

As used herein, the term “time bin” refers to a range of time. In particular, in one or more embodiments, the term “time bin” includes a range of time within a planning window (e.g., a range of time within a time period for executing a digital content campaign). The term “time bin” includes a range of time within which a set of electronic messages have been sent. In particular, the campaign management system can group past electronic messages for a plurality of users into a plurality of time bins based on the timestamps of the electronic messages for use in training the recurrent neural network. The campaign management system can partition messages into a plurality of time bins such that each time bin includes the same (or approximately the same) number of electronic messages. Alternatively, the campaign management system can partition messages into a plurality of time bins, each time bin corresponding to a predetermined, fixed time period (e.g., a specific time range of a specific day of the week, month, or year).

Referring now to the figures,FIG. 1illustrates an embodiment of an environment in which a campaign management system102operates. In particular, the environment100includes an administrator client device104associated with an administrator106, a user client device108associated with a user110, and server device(s)112in communication via a network114. Moreover, as shown, the administrator client device104includes an administrator application116.

Furthermore, the server device(s)112includes the campaign management system102, which includes a recurrent neural network118. As mentioned above, the campaign management system102utilizes the recurrent neural network to determine transmission times for sending new electronic messages to users based on past electronic messages for the users.

According to one or more embodiments, the campaign management system102allows the administrator106to administer a digital content campaign associated with a product, service, individual, entity, or other category. For example, the digital content campaign may be a marketing campaign to promote and distribute information associated with a product, service, or business. Administering the digital content campaign can include creating digital content and distributing the digital content in the campaign. To illustrate, the administrator106can use the administrator client device104to communicate with the campaign management system102on the server device(s)112via the network114to provide and obtain information associated with administering the campaign.

As described in more detail below, the administrator client device104can include a computing device capable of performing operations associated with managing and administering a digital content campaign. For example, the administrator client device104can include a smartphone, tablet, desktop computer, laptop computer, or other device that is able to access electronic messages via the network114and receive user input to interact with the electronic messages (e.g., any of the devices discussed below in reference toFIG. 8). The administrator client device104can include one or more applications (e.g., administrator application116) by which the administrator106can review, plan, modify, and/or execute digital content campaigns. To illustrate, the administrator106can use the administrator application116to access the campaign management system102via the network114.

In one or more embodiments, the campaign management system102facilitates the administration of campaigns by obtaining and analyzing analytics data associated with digital content campaigns. For instance, the campaign management system102can obtain information associated with past electronic messages associated with the digital content campaigns for a plurality of recipient users. Additionally, the campaign management system102can obtain information indicating user interactions with the past electronic messages for the plurality of recipient users. In at least some instances, the campaign management system102assists the administrator106in administering digital content campaigns such that the campaign management system102collects the information associated with the campaigns.

In other embodiments, the campaign management system102obtains the information associated with the campaigns from other systems or devices. For example, the administrator client device104(or a system associated with the administrator client device104) may collect the analytics data for the digital content campaigns and then provide the analytics data to the campaign management system102via the network114. Alternatively, the campaign management system102may receive the analytics data for the digital content campaigns from a third-party system that is separate from the server device(s)112and the administrator client device104.

Once the campaign management system102has obtained the analytics data associated with the digital content campaigns, the campaign management system102analyzes the analytics data using the recurrent neural network118. The campaign management system102trains the recurrent neural network118for a particular user by analyzing past messages for the user. The campaign management system102can include a plurality of recurrent neural networks, with each recurrent neural network trained for a separate user. Accordingly, the campaign management system102can train a plurality of recurrent neural networks to send electronic messages to a plurality of users in connection with one or more digital content campaigns.

In one or more embodiments, the campaign management system102groups past electronic messages for a plurality of recipient users into time bins based on the timestamps of the messages. The campaign management system102can then train the recurrent neural network118for a user by analyzing the past messages for the user according to the time bins in which the past messages for the user are grouped. Once trained, the recurrent neural network118can generate engagement metrics for scoring the time bins in connection with a new electronic message.

Indeed, the campaign management system102can use the engagement metrics of the recurrent neural network118to determine a schedule (e.g., transmission times) for sending one or more new electronic messages in a digital content campaign to the user client device108. According to one or more embodiments, the campaign management system102provides the schedule for the digital content campaign to the administrator client device104to execute the campaign. The administrator106can execute the campaign for the user using the administrator application116by sending one or more electronic messages to the user client device108according to the provided schedule. In one or more alternative embodiments, the campaign management system102automatically executes the digital content campaign in response to determining the schedule for the user by sending one or more electronic messages from the campaign to the user client device108and without requiring the administrator106to provide additional input.

In one or more embodiments, when sending an electronic message to the user110, the campaign management system102sends the electronic message to at least the user client device108. Additionally, the campaign management system102can send the electronic message to any number of client devices associated with the user. For instance, the campaign management system102can send the electronic message to one or more client devices via a user account of the user110. The user110can then access the electronic message from one or more client devices (including the user client device108) from which the user110can access the user account.

According to one or more embodiments, the user client device108includes a computing device that allows the user110to access and interact with electronic messages in a digital content campaign. For example, the user client device108can include a computing device (e.g., any of the devices discussed in reference toFIG. 8) that is able to access electronic messages via the network114and receive user input to interact with the electronic messages. The user client device108can include one or more applications by which the user110can access and interact with the electronic messages such as an email application. Via the email application, the user110can open emails and interact with content within the emails. Additionally, the one or more applications can include an application (e.g., a web browser) that allows the user client device108to access external content from electronic messages.

Although the environment100ofFIG. 1is depicted as having various components, the environment100may have any number of additional or alternative components (e.g., any number of server devices, client devices, or other components in communication with the campaign management system102). Accordingly, the campaign management system102can utilize any number of recurrent neural networks to determine transmission times for sending electronic messages to any number of users. Similarly, the campaign management system102can provide campaign management services to any number of administrators via any number of administrator client devices. Furthermore, more than one component or entity in the environment100can implement the operations of the campaign management system102described herein. Indeed, the campaign management system102can alternatively be implemented entirely (or in part) on the administrator client device104or as part of another component or system.

As mentioned previously, the campaign management system can train and utilize a recurrent neural network to determine transmission times for electronic messages in a digital content campaign.FIG. 2illustrates training and utilizing a recurrent neural network for determining a transmission time (e.g., send time) for a new electronic message for a user in accordance with one or more embodiments. As an overview,FIG. 2shows that the campaign management system102first performs an act202of obtaining past electronic messages (which include past electronic messages for the user and past electronic messages for other users). Additionally, the campaign management system102performs an act204of determining campaign parameters of the digital content campaign. The campaign management system102then performs an act206of partitioning the past electronic messages into time bins. The campaign management system102also includes a step208for utilizing a recurrent neural network and the past electronic messages for the user to determine a time bin of the plurality of time bins for sending a new electronic message for the user. Moreover, the campaign management system102then performs an act210of sending the new electronic message based on the determined time bin.

As just mentioned, in relation toFIG. 2the campaign management system102performs the act202of obtaining past electronic messages for a plurality of users. The campaign management system102can obtain the past electronic messages from a repository of electronic messages. The repository can be locally stored by the campaign management system102or as part of a third-party system. Past electronic messages for any particular user can be identified from the past electronic message for the plurality of users.

In one or more embodiments, the campaign management system102performs an act204of determining campaign parameters for a digital content campaign. As mentioned above, the campaign management system102can determine and utilize a variety of campaign parameters that cause the campaign management system102to determine how to manage and/or execute the campaign. For example, the campaign management system can determine parameters such as campaign length (e.g., a planning window for the campaign), number of messages, number of recipients, and frequency of messages for the campaign. The campaign parameters can also include a target audience or segment.

The campaign management system can identify campaign parameters in a variety of ways. For example, in one or more embodiments, the campaign management system102identifies campaign parameters from an administrator of the digital content campaign. In other embodiments, the campaign management system determines campaign parameters algorithmically. To illustrate, the campaign management system102can determine the number and frequency of messages for a campaign based on models that indicate user response fatigue in connection with receiving email messages. In another example, the campaign management system102can determine a target audience or segment based on characteristics received from the administrator or based on characteristics of content of messages in the campaign.

As part of identifying campaign parameters in the act204, the campaign management system can also determine a type of engagement metric. For instance, the campaign management system102can determine a type of engagement metric for all or part of a digital content campaign. The type of engagement metric can indicate whether the campaign management system102trains the recurrent neural network to output an interaction probability metric based on an interaction classification or a hazard metric based on survival analysis.

In general, survival analysis is an area of statistical modeling primarily interested in modeling the time to an event. Survival analysis utilizes a time-to-interact model to estimate a survival probability—a probability that an individual will survive beyond a time t (i.e., not experience an event of interest by t). Historically, the event of interest was the death of medical patients; however, the campaign management system102adapts survival analysis for an event of interacting with an electronic message by a recipient. In other words, the campaign management system102can utilize survival analysis to estimate a probability the recipient of an electronic message will not interact with the message before a time t.

Using survival analysis, as described briefly above, the campaign management system102can generate a hazard metric. Specifically, a hazard metric includes the probability that, given a recipient user has yet to experience the event (e.g., opening a message) by time t, the recipient will experience the event instantaneously after time t. In one example, a hazard metric can include a hazard rate, which refers to the instantaneous probability of a user opening a message given that the message has not been opened until time t. More specifically, the hazard rate includes a time element that indicates the instantaneous probability of the user interacting with the message within a certain time based on the message being sent within the time bin. Accordingly, the higher the hazard rate for a time bin, the higher the chances are of the user interaction (e.g., email being opened) within a certain amount of time when the message is sent during that time bin for a user.

Due to the origins of survival analysis, hazard functions generally assume that all individuals will eventually experience the event. However, this is not the case in the context of electronic messages as many recipients will delete or otherwise not view/open/interact with the received message. The campaign management system102can modify the survival analysis to account for those recipients who are expected to never open the electronic message. In one or more embodiments, the campaign management system102determines the probability that a particular recipient will interact with a received electronic message based on the input attributes corresponding to the past electronic messages. The campaign management system102can introduce this probability into the survival function as a mixture probability so the time-to-interact for a particular recipient is dependent on attributes that affect whether the particular recipient is likely to interact with the received message and/or attributes that affect the time in which the particular recipient will open the message.

In one or more embodiments, a hazard metric is a hazard ratio, which refers to the time-independent component of the hazard rate. To illustrate, a higher hazard ratio indicates a higher instantaneous risk of occurrence of the event (e.g., opening a message). For instance, the campaign management system102can obtain a hazard ratio by dividing a given hazard rate by a standard time period (as determined by the campaign management system102based on other campaign parameters past electronic messages).

In one or more embodiments, an interaction probability metric refers to the probability that a given interaction with a message will occur given that the message is sent within a time bin. To illustrate, an open probability metric indicates the probability that a user will open the message when the message is sent within a given time bin. A higher open probability metric indicates a higher probability, and a lower open probability metric indicates a lower probability.

The campaign management system can determine a type of engagement metric to optimize in selecting one or more transmission times for a user within a digital content campaign. For example, in one or more embodiments, an administrator selects between a hazard metric or an interaction probability metric. In some embodiments, the campaign management system can automatically select (or suggest) a type of engagement metric. For example, the campaign management system can select a type of engagement metric based on details regarding a digital advertising campaign (e.g., select a hazard metric for time-sensitive digital content, such as a flash sale message, or select an interaction probability metric for informational digital content) and/or based on historical analytics data (e.g., information indicating that a particular product, service, entity, or advertisement has produced higher open rates utilizing a particular type of engagement metric).

The campaign management system102also determines a planning window for the campaign. The planning window indicates a time period for which the campaign management system102will execute the campaign (or otherwise provide electronic messages according to scheduled transmission times). For instance, the planning window can indicate a specific time period with a beginning date and an ending date (e.g., December 1 to December 31). Alternatively, the planning window can indicate a length of time from a starting date yet to be determined for the campaign. The campaign management system102can determine the planning window based on a specific selection by an administrator or based on similar and/or previous campaigns.

To illustrate an example of the act204of determining campaign parameters, in one or more embodiments, the administrator provides campaign management system with a group of customers to send messages to (e.g., a segment, email list, or set of characteristics). The administrator indicates a type of engagement metric (e.g., hazard metric), and selects a total number of messages (e.g., K messages) to be sent throughout the planning window (e.g., 1 weeks' time). Further, the administrator specifies constraints on the maximum frequency (e.g., maximum number of messages to be sent per day). The campaign management system then utilizes these campaign parameters to determine transmission times. As mentioned above, these parameters can also be determined algorithmically (rather than directly from an administrator).

As shown inFIG. 2, the campaign management system102can also perform the act206of partitioning the past electronic messages into time bins. In particular, the campaign management system102partitions the past electronic messages for the plurality of users into a plurality of time bins based on timestamps corresponding to the past electronic messages. Because the past electronic messages for the plurality of users include the past electronic messages for the user, the campaign management system102also partitions the past electronic messages for the user into the time bins. In one or more embodiments, the campaign management system102partitions a portion of the past electronic messages while holding a separate portion for testing (e.g., testing accuracy of the trained recurrent neural network).

In one or more embodiments, partitioning the past electronic messages into time bins includes partitioning the past electronic messages into each time bin according to a grouping proportion. In at least some embodiments, a “grouping proportion” refers to an equal or approximately equal number of messages into each time bin. For example, the campaign management system102can partition the past electronic messages evenly into non-overlapping time bins to take into account variations of open rates of the past electronic messages due to the send times of the past electronic messages. To illustrate, the campaign management system102can dynamically adjust the time bins (e.g., modify the time range of each time bin) such that the resulting time bins have an equal number of past electronic messages. To illustrate, the campaign management system102can generate a first time bin so that the first time bin includes the first N number of past electronic messages from a first time range of a first length, and the campaign management system102can generate a second time bin so that the second time includes another N number of past electronic messages from a second time range of a second length. In the foregoing example, the past electronic messages in the second time bin can follow the past electronic messages in the first time bin chronologically. Accordingly, the time ranges for the time bins can be arbitrary (to satisfy the grouping proportion), such that the first length may be different than the second length depending on the timestamps of the past electronic messages that are in the first and second time bins.

In alternative embodiments, partitioning the past electronic messages includes partitioning the past electronic messages into time bins according to fixed, predetermined time ranges. For example, rather than partitioning the past electronic messages equally into a plurality of time bins, the campaign management system102can partition the electronic messages into time bins that each correspond to specific time ranges. To illustrate, the campaign management system102can partition past electronic messages with timestamps that fall within a first time range into a first time bin and past electronic messages with timestamps that fall within a second time range into a second time bin. In such a case, the time bins may not have equal numbers of past electronic messages, but the time ranges corresponding to the time bins are not dynamic and are set beforehand. Furthermore, the fixed time ranges corresponding to the time bins may be equal or unequal (e.g., the time bins may correspond to specific time ranges manually designated by an administrator).

Additionally, the campaign management system102can partition the past electronic messages in connection with an analysis window. In particular, in one or more embodiments, the campaign management system102can set an analysis window defining a time range for analyzing past electronic messages. The campaign management system102can then select and utilize past electronic messages falling within the analysis window.

The campaign management system102can determine an analysis window based on user input (e.g., based on input from an administrator). Moreover, the campaign management system102can determine the analysis window based on a plurality of electronic messages (e.g., a time range for electronic messages previously sent to a particular user or a plurality of users). In other embodiments, the analysis window is based on the planning window (e.g., the planning window and the analysis window are the same length). To illustrate, if the planning window is for a month-long campaign, the campaign management system102can determine that the analysis window is covers a month of past electronic messages.

In one or more embodiments, the number of time bins is based on the length of the analysis window. Specifically, the campaign management system102can determine the number of time bins based on the number of past electronic messages within the analysis window. For instance, if the analysis window covers a period of time with N number of past messages, the campaign management system102can partition the past messages into X time bins. If the analysis window covers a period of time with 2N number of past messages, the campaign management system102can partition the past messages into 2X time bins. Accordingly, the number of time bins may be directly proportional (or otherwise correlate) to the number of past electronic messages within the analysis window.

The administrator can also set a precision of the scheduling to indicate how much flexibility the campaign management system102has in scheduling the sending of new electronic messages. In particular, the precision of a campaign indicates a threshold or range of time bin sizes so that the campaign management system102analyzes the transmission times of messages at a particular level of detail. To illustrate, the campaign management system102can determine the scheduling precision to be a minimum of one-hour segments, five-minute segments, etc.

The analysis window and/or the precision for analyzing a campaign may also be constrained by additional factors. To illustrate, the campaign management system102can apply constraints or limitations to scheduling message transmission times based on server capabilities, user preferences associated with a user profile (e.g., opt-out preferences for specific types of messages, user preferences related to preferred time ranges), or other limitations. To illustrate, the campaign management system102can limit transmission times based on server limits so as to prevent the campaign management system102from overloading server(s) that send the messages to a plurality of users for a campaign.

As shown inFIG. 2, in addition to the act206, the campaign management system102also performs a step208for utilizing a recurrent neural network and the past electronic messages for the user to determine a time bin of the plurality of time bins for sending a new electronic message for the user. As shown, the step208includes the acts212,214(including214aor214b),216, and218. Additional detail regarding acts in the step208also described below with regard toFIG. 2(additional detail is also provided regarding such acts in relation toFIGS. 3 and 4).

As shown inFIG. 2, the step208can include the act212. In particular, after determining the campaign parameters (and type of engagement metric), the campaign management system102performs an act212of training the recurrent neural network using the past electronic messages of a user (and/or the other users). For example, the campaign management system102can train the recurrent neural network for the user based on the past electronic messages for the user and/or the other users (e.g., as described in more detail inFIG. 3below). To illustrate, the campaign management system102identifies electronic messages that have previously been sent to the user (and/or other users), and the campaign management system102trains the recurrent neural network to output the selected engagement metric based on the past electronic messages sent to the user (and/or other users).

In training the recurrent neural network, the campaign management system102also identifies various input attributes of the past electronic messages for the user. The input attributes can include information describing the past electronic messages, user behavior, and/or interactions with the past electronic messages by the user. For example, the campaign management system102can collect and provide input attributes to the recurrent neural network for each past electronic message. The input attributes can include information indicating responses by the user to messages sent prior to a currently analyzed message (e.g., open/click status and/or open/click times), most recent purchase behavior for the user, and transmission times of the messages (e.g., based on the time bin in which a message is partitioned). The input attributes can also include information obtained from other sources, such as postal databases, loyalty card databases, purchase databases, etc.

Using the input attributes, the campaign management system102trains the recurrent neural network to determine the engagement metric. In particular, the campaign management system102can train the recurrent neural network to identify significant features (e.g., weighted parameters) for determining the engagement metric. For example, for a hazard metric, the recurrent neural network automatically determines features by predicting a hazard metric for a prior electronic document and then comparing the predicted hazard metric with a ground truth metric. The recurrent neural network can then modify features (e.g., weighted parameters) to reduce loss between the predicted hazard metric and the ground truth metric. Alternatively, for an interaction probability metric, the recurrent neural network determines features by predicting an interaction probability metric for a prior electronic document and then comparing the predicted probability metric with a ground truth metric. The recurrent neural network can then modify features to reduce loss between the predicted probability metric and the ground truth metric.

As described in more detail below with respect toFIG. 4, the campaign management system102can utilize recurrent neural networks having a different structure based on whether the recurrent neural network is utilizing a hazard metric or an interaction probability metric. Specifically, the campaign management system102can utilize a recurrent neural network with a survival layer that utilizes a survival algorithm when generating a hazard metric. The campaign management system102can omit a survival layer when generating an interaction probability metric. Additionally, the campaign management system102can use different activation functions within the recurrent neural network when generating different engagement metrics.

After training the recurrent neural network using past electronic messages, the campaign management system102can perform the act214of utilizing the recurrent neural network to generate engagement metrics for a new electronic message for the user. Specifically, the campaign management system102determines input attributes for the new electronic message (similar to the input attributes described above).

Additionally, the campaign management system102determines a planning window for the new electronic message. As described in connection withFIG. 2, the planning window indicates a time window for executing the digital content campaign. Accordingly, the campaign management system102can determine the planning window to identify possible dates and times for transmitting the new electronic message. As mentioned, the analysis window (for grouping past electronic messages in training the recurrent neural network) can correspond to the planning window.

In connection with determining the planning window, the campaign management system102can determine time bins for the planning window. In one or more embodiments, the campaign management system102can identify time bins from the analysis window that also fall within the planning window. For example, if the analysis window includes 100 time bins of past electronic messages, the campaign management system102can divide the planning window into the same 100 time bins. Additionally, the time ranges associated with the time bins in the planning window can correspond to the time ranges associated with the time bins in the analysis window.

Furthermore, in one or more embodiments, the campaign management system102utilizes the trained recurrent neural network to analyze input attributes for the new electronic message and generate predicted engagement metrics for the new electronic message. As shown inFIG. 2, if the selected engagement metric type is a hazard metric, the campaign management system102can perform the act214aof utilizing the recurrent neural network to generate hazard metrics for the time bins for the new electronic message. In particular, the campaign management system102can use the recurrent neural network to determine hazard metrics for the new electronic message for each time bin generated at the act204. By generating a separate time-to interact metric for each time bin, the campaign management system102can model the user's time-to-interact with the electronic message if the electronic message was sent within each time bin. In this manner, the campaign management system102can score each time bin for the new electronic message, where the scores indicate the predicted time-to-interact if the new electronic message were sent to the user within the time bins.

Alternatively, if the selected engagement metric is an interaction probability metric, the campaign management system102performs the act214bof utilizing the recurrent neural network to generate interaction probability metrics for the time bins. In particular, the campaign management system102can use the recurrent neural network determine interaction probability interaction metrics for the new electronic message for each time bin generated at the act204. By generating a separate probability interaction metric for each time bin, the campaign management system102can model the user's probability of interacting with the electronic message if the electronic message was sent within each time bin. In this manner, the campaign management system102can score each time bin for the new electronic message, where the scores indicate the probability of interaction if the new electronic message were sent to the user within the time bins.

In each case, the campaign management system102can utilize the recurrent neural network to determine the engagement metrics for sending the new message within the time bins based on past user interactions with past messages. Moreover, as outlined in greater detail below (e.g., in relation toFIG. 3), the campaign management system102can utilize the recurrent neural network to sequentially analyze prior electronic messages in generating an engagement metric for a new electronic message. Thus, via the recurrent neural network, the campaign management system102can consider timing and sequential features of past electronic messages in determining a transmission time for a new electronic message.

Upon generating the engagement metrics (i.e., the hazard metrics or interaction probability metrics) for the time bins, the campaign management system102can then utilize the engagement metrics for the time bins to determine a transmission time for the new electronic message. For example, the campaign management system102can determine a time bin for sending the new electronic message in a digital content campaign by performing the act216of ranking the time bins using the engagement metrics.

Specifically, the campaign management system102can perform the act216by comparing the engagement metrics for the time bins to identify the time bins with the best engagement metrics. For example, the campaign management system102can rank the time bins based on hazard metrics and then select one or more time bins that have the highest hazard metrics (or lowest time-to-open value). Alternatively, the campaign management system102can rank the time bins based on interaction probability metrics and then select one or more time bins that have the highest interaction probability metrics.

In one or more embodiments, after ranking the time bins, the campaign management system102then includes an act218of determining a time bin for sending a new electronic message using one or more of the campaign parameters. In particular, while the ranked time bins allow the campaign management system102to determine the best individual time bins for sending electronic messages, campaign parameters can include certain constraints for sending messages in a campaign. The campaign management system102can select a time bin for each of the messages in the campaign to send to the user using the rankings of the time bins and within the constraints established by the campaign parameters.

For example, in one or more embodiments, the campaign parameters indicate a maximum number of messages to send as part of the campaign. The campaign management system102can therefore use the maximum number of messages to determine how many time bins to select. To illustrate, if the maximum number of messages is five for the campaign, the campaign management system102selects no more than five time bins after ranking the time bins. While the maximum number of messages can place an upper limit on the number of messages for a user, the campaign management system102may send fewer than the maximum number, depending on the user's preferences (e.g., a user preference in a user profile indicating a number of messages that the user wishes to receive, or time periods in which the user does not wish to receive messages) or based on one or more predicted engagement metrics not meeting a predetermined threshold. Thus, the campaign management system102can tailor the number of messages for a user to produce the best results for that user.

Additionally, the campaign parameters can indicate a frequency parameter for sending messages in the campaign. In particular, the frequency parameter indicates a number of messages that the campaign management system102can send to the user within a specific time window. For instance, the frequency parameter can limit the campaign management system102to sending no more than one message per day to a user. If the campaign management system102determines that more than one time bin in a single day is ranked within the top N number of time bins (where N corresponds to the maximum number of messages for the campaign), the campaign management system102can skip the lower ranked time bin on that day. Thus, the campaign management system102can select N−1 time bins for sending messages instead of N time bins. Alternatively, the campaign management system may select a next-highest ranked time bin (i.e., the next highest ranked time bin).

In one or more embodiments, the campaign management system102also determines whether each time bin meets a predetermined engagement threshold value. For example, the campaign management system102can set a predetermined engagement threshold value that indicates a maximum or minimum value (based on the corresponding engagement metric) for a time bin. If a time bin satisfies the predetermined threshold (e.g., the engagement metric that the recurrent neural network generates for that time bin satisfies the threshold), the time bin is eligible to be selected; otherwise, the time bin is excluded from selection.

Upon determining the time bin for sending a new electronic message, the campaign management system102can perform the act210of sending the new electronic message based on the determined time bin. In one or more embodiments, the campaign management system102selects a time that falls within the time bin for sending the electronic message. For example, the campaign management system102can select a transmission time at the beginning of a time bin, at the end of the time bin, or at some other time within a time bin. For instance, in one or more embodiments, the campaign management system102selects a transmission time based on past electronic messages within the time bin. To illustrate, the campaign management system102can average the timestamps for past electronic messages within the time bin to determine the transmission time for the electronic message and then execute the campaign by sending the new electronic message at the send time. In yet other embodiments, the campaign management system102can select a transmission time within a time bin based on available network and/or computer processing power within the time bin. For example, the campaign management system102can queue the electronic message to be sent at a time within the time bin that reduces computer processing power (e.g., at a time when there is a minimum amount of other messages being sent or at a time that avoids spikes within the time bin).

The acts202-218described in relation toFIG. 2are intended to be illustrative of an exemplary sequence of acts in accordance with the present disclosure, and are not intended to limit potential embodiments. Alternative embodiments can include additional, fewer, or different acts than those articulated inFIG. 2. For instance, although the description above references a single new electronic message, the campaign management system102can select time bins for a plurality of new messages. Indeed, the campaign management system102can utilize the approach described above to select five transmission times (e.g., five time bins) for five new messages. In particular, the campaign management system102can identify the five time bins corresponding to the five highest engagement metrics (subject to constraints) and then send one of the five new messages within each of the five time bins.

In one or more alternative embodiments, rather than determining the length of the analysis window and the length of the planning window to be the same, the campaign management system102can determine the length of the analysis window to be different than the length of the planning window. Specifically, the campaign management system102can use an analysis window that includes a lesser or greater time range and fewer or more time bins than the planning window includes. Accordingly, the campaign management system102can train a recurrent neural network for a user based on a plurality of past electronic messages for the user within a time range that is different than the time range of the planning window.

For example, the campaign management system102can train the recurrent neural network using an analysis window of one month that includes 100 time bins. The campaign management system102can then utilize the recurrent neural network for a campaign with a planning window of two weeks. The campaign management system102can analyze a subset of the 100 time bins from the analysis window (e.g., the subset of 50 time bins from the first two weeks of the analysis window that overlap the two week planning window). Based on the analysis of the subset of time bins, the campaign management system102can select a send time for one or more new messages. Accordingly, the campaign management system102can train the recurrent neural network with electronic messages from an analysis window that reflects a longer time period than a planning window.

Additionally, the acts described herein may be performed in a different order, may be repeated or performed in parallel with one another, or may be performed in parallel with different instances of the same or similar acts. For example, in one or more embodiments, rather than obtaining an engagement metric and campaign parameters from an administrator prior to training the recurrent neural network, the campaign management system102can obtain one or both after training the recurrent neural network for determining send times for new electronic messages. For instance, the campaign management system102can include a recurrent neural network for each possible engagement metric and then use the recurrent neural networks to output more than one prediction for a new message. The campaign management system102can then use the applicable output from the recurrent neural networks to determine the send times for messages based on a selected engagement metric. The models that include the recurrent neural networks can include a different structure for each of the engagement metrics, as described in more detail with respect toFIG. 4. Furthermore, as described below, the campaign management system102can use campaign parameters to determine a schedule of send times for the campaign after training the recurrent neural networks.

Additionally, in one or more embodiments, the campaign management system102trains and/or utilizes a recurrent neural network for a plurality of users, rather than for an individual user. In particular, the campaign management system102can determine a target audience (e.g., based on input from a campaign administrator selecting the target audience) and use past electronic messages for the target audience (or a subset of the target audience) to train the recurrent neural network. The campaign management system102can then utilize the recurrent neural network to output an engagement metric for sending a new electronic message to a single user or the plurality of users in the target audience within one or more time bins.

As described above, the step208includes training a recurrent neural network for a user based on past electronic messages for the user (and other users).FIG. 3illustrates a schematic diagram of training a recurrent neural network based on past electronic messages for a user (and/or other users). Specifically, training a recurrent neural network first includes inputting a plurality of past electronic messages302(e.g., emails) into the recurrent neural network304. For instance, the campaign management system102selects a plurality of past electronic messages that the campaign management system102has sent to the user in connection with one or more previous digital content campaigns. As previously mentioned, the campaign management system102partitions the past electronic messages302(together with past electronic messages for other users) into time bins.

In addition, as shown inFIG. 3, the campaign management system102provides input attributes306associated with the past electronic messages302as input to the recurrent neural network304. The input attributes306can include available information regarding the past electronic messages. For example, the input attributes306can include information regarding responses to the past electronic messages302. For example, the input attributes306can include information associated with electronic messages chronologically prior to a given electronic message such as, but not limited to, the last message opened, the last message selected, whether the last message was mass emailed (e.g., sent to seventy-five percent of users for messages in the training period), days since the last message was sent, days since the last purchase by the user, binary indicators for whether the user made a web (i.e., online) purchase since the last message or whether the user made an offline purchase, non-electronic messages sent since the last message, etc. The input attributes306can also include a send time or time bin of the given message (e.g., based on the timestamp of the given message).

Furthermore, the campaign management system102also provides a ground-truth engagement metric312as an input to the recurrent neural network. In particular, the ground-truth engagement metric312can include a ground truth hazard metric, such as the actual time between transmission time of a past electronic message and interaction by the user. Similarly, the ground-truth engagement metric312can include ground-truth interaction probability metric, such as a binary indicator of whether or not a user actually interacted with a past electronic message. The campaign management system102can compare predicted engagement metrics with the ground-truth engagement metric312to generate a trained recurrent neural network316.

Indeed, as illustrated, during training the recurrent neural network304analyzes input attributes of each of the past electronic messages302and generates a predicted engagement metrics310a-310c. In particular, the recurrent neural network304analyzes different features of the past electronic messages at different levels of abstraction utilizing different layers of the recurrent neural network304. For example, the recurrent neural network can utilize various layers (discussed in greater detail inFIG. 4) to generate weighted parameters to predict engagement metrics for each message.

For example, the recurrent neural network304generates a first predicted engagement metric310afor a first message302aof the past electronic messages302using input attributes306corresponding to the first message302a. The recurrent neural network304also generates a second predicted engagement metric310bfor a second message302bof the past electronic messages302using input attributes306corresponding to the second message302b. The recurrent neural network304continues analyzing the past electronic messages302and generating predictions for each message until generating an Nth predicted engagement metric310nfor a message302nof the past electronic messages302using input attributes306corresponding to the message302n.

As shown, upon generating a predicted engagement metric for a message, the campaign management system102can compare the predicted engagement metric to a ground-truth engagement metric. For example, the campaign management system102can compare the first predicted engagement metric310ato a ground truth engagement metric312a, the second predicted engagement metric310bto a second ground truth engagement metric312b, etc., and the Nth predicted engagement metric310nto an Nth ground truth engagement metric312n. The campaign management system102can thus compare each predicted engagement metric to the corresponding ground-truth engagement metric upon generating each predicted engagement metric.

As shown, the campaign management system102can compare predicted engagement metrics and ground truth engagement metrics utilizing a loss function314. The loss function314can include a function that describes the difference of each of the prediction/ground truth pairs. For example, the predicted engagement metric310acan include a predicted hazard metric of 10 minutes and a ground-truth hazard metric of 15 minutes. The campaign management system102can compare the predicted hazard metric and the ground-truth hazard metric utilizing the loss function314to determine a measure of error (e.g., 5 minutes).

In addition, as shown, the campaign management system102can use the loss function314to generate the trained recurrent neural network316. In particular, the campaign management system102can modify one or more weights (e.g., weights for parameters) of the recurrent neural network304to reduce the differences between the predicted engagement metrics310a-310nand the ground truth engagement metrics312a-312n. For example, in one or more embodiments, the campaign management system102utilizes a binary cross-entropy algorithm as a loss function. Moreover, the campaign management system102utilizes a stochastic gradient descent as an optimizer to reduce the measure of loss for each prediction. Similarly, in other embodiments, the campaign management system102utilizes a negative log partial likelihood function with Efron's approximation as the loss function. In another example, the campaign management system102utilizes a survival loss function (i.e., a loss function based on a hazard metric generated by a survival function). By modifying the recurrent neural network304to reduce the measure of loss, the campaign management system102creates a trained recurrent neural network316that is capable of generating predictions of engagement metrics for new electronic messages to be sent to the user.

Turning now toFIG. 4, additional detail will be provided regarding an exemplary architecture of a recurrent neural network in accordance with one or more embodiments. In addition, additional detail is provided regarding utilizing a trained neural network to generate an engagement metric for one or more new messages. For example,FIG. 4illustrates utilizing a recurrent neural network architecture trained based on past electronic messages to generate an engagement metric for one or more new messages.

As illustrated inFIG. 4, the campaign management system102processes a plurality of past electronic messages in a plurality of stages400a-400baccording to a sequential order (e.g., chronological order) of the electronic messages. Each stage utilizes a recurrent neural network with a plurality of layers that allow the campaign management system102to generate predictions for each of the past electronic messages and then compare the predictions against ground-truth values to train the recurrent neural network. Furthermore, the structure of the stages400a-400bis based on the type of engagement metric to be output by the stages400a-400b. More specifically, the stages400a-400buse survival analysis for generating hazard metrics but do not use survival analysis for generating interaction probability metrics.

In one or more embodiments, the campaign management system102uses information derived from the past electronic messages for the user (and/or other users) as an input to the recurrent neural network for analyzing the past electronic messages and training the recurrent neural network. In particular, the campaign management system uses attributes402aassociated with the past electronic messages as an input to a first stage of the process (e.g., for a first message in the sequence of past messages). For example, the campaign management system102can determine a feature vector that represents features of a plurality of past electronic messages for the user. To illustrate, the feature vector can include an elementwise average of the feature vectors of the most recent messages (e.g., prior to the first message in the sequence of past electronic messages or prior to the new electronic message) to create a single vector that describes the average number of messages opened, selected, or with which the user has otherwise interacted, etc., as previously described. Thus, the feature vector can represent a recency effect of past interactions by the user, and the campaign management system102can vary the number of most recent messages to vary the recency effect.

In each subsequent stage in the training process (e.g., stage400b), the campaign management system102can input attributes associated with the corresponding message into the recurrent neural network. As previously described, the input attributes indicate user interactions or send characteristics associated with current message and/or previous messages, as well as the time bin for the current message. For instance, the attributes402bfor a second message in the sequence of past messages can include user interactions or send/transmission characteristics associated with the first message and the time bin of the second message. Thus, the attributes402bfor the second stage400bcan reflect the user interactions with the past messages (especially the first message) sequentially prior to the second message.

Furthermore, as the campaign management system102processes each message in separate stages, the campaign management system102can determine data that the recurrent neural network determines is important for predicting the engagement metrics according to a sequential order of the messages in the campaign. In particular,FIG. 4illustrates that each stage400a-400bpasses sequence data404a-404bto the next stage for use in analyzing the next sequential message. The sequence data can include internal parameters or other significant features that the recurrent neural network determined in predicting engagement metrics for the messages. For example, after analyzing the first message of the past messages for the user in the first stage400a, the first stage400apasses sequence data404ato the second stage400bfor analyzing the second message of the past messages for the user. Similarly, the second stage400bcan pass sequence data404bthat is based on the second stage400band the first stage400ato the third stage401for predicting engagement metrics for the new electronic message. Thus, the recurrent neural network captures data corresponding to the sequential order of the messages for use in predicting user interactions with each new message.

As mentioned, the campaign management system102uses the recurrent neural network to generate predictions for each past message when training the recurrent neural network. Specifically, each stage400a-400boutputs a predicted engagement metric406a-406bfor the corresponding message. To illustrate, the first stage400aanalyzes the first message and outputs an engagement metric406afor the first message. Similarly, the second stage400banalyzes the second message and outputs an engagement metric406bfor the second message.

To generate an engagement metric, each of the plurality of stages400a-400binclude a recurrent neural network layer (“RNN Layer”)408a-408band a dense layer410a-410b. The RNN layers408a-408binclude the recurrent neural network that receives input attributes and/or other data associated with a currently analyzed electronic message and past electronic messages for informing the output of the RNN layers408a-408b. For example, as previously described, the RNN layer408aof the first stage400acan receive attributes402acorresponding to the first message and then output sequence data404ato the second stage400b. The RNN layer408bof the second stage400bcan receive attributes402bcorresponding to the second message and sequence data404afrom the first stage400aand then outputs sequence data404bto a subsequent stage (not shown).

The RNN layers408a-408bcan include a variety of nodes to process arbitrary sequences of inputs (e.g., arbitrary sequences of electronic messages). For instance, the RNN layers408a-408bmay include a series of fully connected layers or nodes for processing a sequence of electronic messages and outputting engagement metrics for the messages. The RNN layers408a-408boutput data to the dense layers410a-410bfor additional processing of the data.

In one or more embodiments, the dense layers410a-410binclude fully connected layers including activation functions at the output of the RNN layers408a-408b. For example, the campaign management system102can use a classification model that generates an interaction probability metric at each stage400a-400b. Specifically, in the classification model, the dense layers410a-410bcan include a sigmoid activation function at the output of the RNN layers408a-408b, which then produce interaction probability metrics for the corresponding past electronic messages. To illustrate, for a given stage (e.g., stage400a), the RNN layer408aand the dense layer410aoutput an interaction probability metric (e.g., a probability for classifying the message as being interacted with, or not) for the first message of the past electronic messages for the user.

Alternatively, the campaign management system102can use a survival model that generates a hazard metric at each stage400a-400b. In particular, in the survival model, the dense layers410a-410ccan include an exponential activation function at the output of the RNN layers408a-408c, and survival layers412a-412care connected to the outputs of the dense layers410a-410cto produce hazard metrics for a plurality of time bins. To illustrate, for a given stage (e.g., stage400a), the RNN layer408a, the dense layer410a, and the survival layer412aoutput a hazard metric for each of the plurality of time bins.

At each stage, the campaign management system102then uses the engagement metrics406a-406bto train the recurrent neural network to reduce error in the predictions. Specifically, the engagement metrics can compare the engagement metrics406a-406bto the corresponding ground-truth actual values for the past messages. For instance, the campaign management system102can compare the engagement metric406ato the ground-truth actual value for the first message (e.g., from user interaction data associated with the first message) to update the RNN layer408ain the first stage400a. Likewise, the campaign management system102can compare the engagement metric406bto the ground-truth actual value for the second message to update the RNN layer408bin the second stage400b.

WhileFIG. 4illustrates only two stages400a-400bfor training the recurrent neural network on past electronic messages, the campaign management system102can use more stages for training the recurrent neural network. For example, the recurrent neural network can include a stage for each past electronic message for the user within the analysis window. Accordingly, if the analysis window includes fifty past electronic messages for the user partitioned into a plurality of time bins, the recurrent neural network can include fifty stages—one for each past electronic message for the user. Furthermore, the stages are ordered to analyze the past electronic messages according to a sequential order of the past electronic messages (e.g., the most recent past electronic message corresponding to a final stage of the training stages and the oldest past electronic message corresponding to a first stage of the training stages).

As mentioned, the campaign management system102can train the model including the recurrent neural network based on past electronic messages for a user and use the trained network to analyze new electronic messages. For instance, as shown inFIG. 4the campaign management system102then utilizes the trained neural network resulting from the plurality of stages400a-400bto generate predictions for a new electronic message for the user. In particular, the campaign management system102includes a stage401that follows the sequential order of the past electronic messages (e.g., occurs chronologically after a most recent past electronic message). The stage401utilizes the trained recurrent neural network with a structure that corresponds to the type of engagement metric that the stage401outputs. Specifically, for generating an interaction probability metric, the stage401includes an RNN layer408cand a fully connected dense layer410cincluding a sigmoid activation function at the output of the RNN layer408c. For generating a hazard metric, the stage401includes an RNN layer408c, a fully connected dense layer410cincluding an exponential activation function at the output of the RNN layer408c, and a survival layer412cconnected to the outputs of the dense layer410c.

To predict an engagement metric406cfor a new electronic message, the stage401receives, as input to the RNN layer408c, attributes402ccorresponding to the new electronic message and sequence data from a final stage of the training process. In particular, the attributes402cinclude information describing a plurality of past electronic messages for the user (e.g., user interactions and transmission characteristics, as described previously). Additionally, a stage immediately prior to stage401(e.g., a final stage in the training process for a most recent past electronic message in the training dataset) passes on sequence data that the recurrent neural network has learned during the training process.

The stage401uses the attributes402cand the sequence data to generate the engagement metric406cfor the new electronic message and for a time bin. For instance, the campaign management system102varies the time bin for the new electronic message according to a plurality of time bins within a planning window to predict a plurality of engagement metrics406cfor the plurality of time bins. Thus, the campaign management system102generates a separate engagement metric406cfor each time bin to obtain a plurality of engagement metrics406cfor the plurality of time bins within the planning window.

As also mentioned previously, each stage of the model ofFIG. 4generates an engagement metric for each of the time bins. Using the engagement metrics, the campaign management system102determines a send time for an electronic message by ranking the time bins. In particular, the campaign management system102can rank the time bins to select the time bin with the best engagement metric (e.g., maximum interaction probability metric or minimum hazard metric). The campaign management system102can also determine a send time associated with the selected time bin for sending the electronic message during the determined send time. For sending a plurality of electronic messages, the campaign management system102can also use one or more campaign parameters (e.g., message frequency) for determining which time bins to select.

After sending the electronic message at a time corresponding to a time bin based on the engagement metric406c, the campaign management system102can track user interactions with the electronic message. For example, the campaign management system102can determine whether and/or when the user opens the message, selects content within the message, follows a link within the message to an external site, completes a purchase as a result of the message, etc. The campaign management system102can leverage the user interaction information with the message to update the recurrent neural network by comparing a ground-truth actual value for the message to the engagement metric406cto determine a loss and then updating one or more algorithms of the recurrent neural network accordingly.

The campaign management system102can also be described in terms of particular variables, equations, and or pseudo-code for determining transmission times utilizing one or more computer devices. Accordingly, additional detail will now be provided regarding one or more of these exemplary embodiments.

For a user c and the history of past messages sent to the user denoted as (M1, M2, . . . , Mk-1}. Each Mi, i∈{1, . . . , k−1} has a set of features, denoted by Xi, for the ith message. The feature set includes the input attributes as previously described relative toFIG. 3(e.g., user interactions and send characteristics). For instance, a user interaction with message Miis denoted by {Yi, Ti, δi}, where Yiis a binary indicator of whether the message being analyzed was opened and Tiis the time difference between send time and open time. δiis a binary indicator of whether the open event is censored (i.e., if the open event is not observed within a certain time period Ci. In a case of censoring (i.e., δi=1), Yi=0 and Ti=Ci.

Furthermore, as mentioned, the time bin for sending an electronic message is one of the features in Xi. For example, in the present embodiment, the time bin is one of 100 possible time bins within the planning window. Additionally, the time bin may be encoded using a one-hot vector of length100. The number of time bins may vary based on the planning window length and the scheduling precision, as determined by the campaign management system102.

The campaign management system102uses the recurrent neural network to determine the optimal send time kth message onwards. For each message, the campaign management system102scores the time bins using the interaction probability metric or the hazard metric for the message (i.e., when the message is sent during the scored time bin). Because the time bin is one of the input attributes for a message, varying the time bin obtains different scores for each bin. The two types of scoring models (i.e., classification and survival) optimize for different administrator-specified objectives. Both models use a recurrent neural network, which helps to model the sequential interaction data associated with a plurality of messages.

For the classification model, the campaign management system predicts whether a customer will interact with the kth message. Specifically, input to the classification model includes the feature vectors of the historical sequence of past messages sent to the user. The campaign management system restricts the length of the sequence to m messages in the past. Consequently, the input to the model is characterized as {Xk-m+i, Xk-m+2, . . . , Xk}, and the expected output for the sequence is Yk. Generalized, the output is Yc,k+1,bfor the cth user, the (k+1)th message, and the bth bin.

The campaign management system102first passes the input sequence through a recurrent neural network layer one message at a time, followed by a fully connected layer with sigmoid activation function. At each step, the campaign management system102optimizes the model to predict Y1for that message. Furthermore, the classification model can use stochastic gradient descent as an optimizer with binary cross-entropy as the loss function.

In alternative embodiments involving a survival model, the campaign management system predicts an interaction time for a message. Specifically, the survival model of the present embodiment predicts an open time for a message. In one example, the campaign management system102uses a proportional hazards framework for modeling the process of predicting the time-to-interact of a message. For each message, the campaign management system102can compute a hazard ratio, which is the time-independent component of a hazard rate and only depends on the input attributes. The hazard rate represents the instantaneous probability of the user opening the message given it has not been opened till now. The higher the hazard ratio, the higher the instantaneous risk of occurrence of opening the message. Generalized, the survival model outputs a hazard rate h(t), for (i, k+1, b).

The campaign management system102first passes the input sequence through a recurrent neural network one message at a time, followed by a fully connected layer with exponential activation function (as described in relation toFIG. 40. The campaign management system102can also use a negative log partial likelihood function with Efron's approximation as the loss function. Efron's approximation changes the likelihood function to account for data points with tied time-to-open values. The output layer predicts that hazard ratio, Ψ(Xi), for each message. The survival function has access to the actual value {Ti, δi} on which the campaign management system102calculates the loss.

Once the campaign management system102has generated the engagement metrics for a plurality of time bins for a message, the campaign management system102ranks the time bins and selects a time bin for the message. To illustrate, for a classification model, the campaign management system102sorts the time bins based on the corresponding interaction probability metrics. For a survival model, the campaign management system102sorts the time bins based on the corresponding hazard ratios.

Because campaign administrators typically plan for a plurality of messages within a campaign, the administrator may request to generate a send schedule for all of the messages in the campaign. Accordingly, the selected time bin for one or more of the messages may change depending on the total number of messages and limits on the frequency of messages to send. Thus, given the number of messages to be sent for a campaign within a planning window, the campaign management system102determines the optimal send time for each message such that the total sum of scores for the time bins is maximized (or minimized, depending on the type of scoring/metrics) subject to constraints such as total volume and frequency of messages.

For example, the campaign management system102can sort the time bins according to the scores for the corresponding engagement metric. To illustrate, the campaign management system102can select a plurality of time bins equal to the total number of messages in the campaign. The campaign management system102can also filter out time bins violating the frequency constraint. Thus, if any of the selected time bins violates the frequency constraint (e.g., by being too near another time bin), the campaign management system102can remove or skip over the selected time bin. The campaign management system102can generate a schedule for sending the plurality of messages using the selected time bins that meet the maximum message and frequency constraints.

Furthermore, as previously mentioned briefly, when scheduling a plurality of messages to be sent, the campaign management system may not have input attributes for the future messages within the planning window. To overcome this deficiency, the campaign management system102can make a simplifying assumption by constructing input attributes for each future message at the planning window (e.g., using the input attributes for past electronic messages). The campaign management system102can then assume that these input attributes remain constant throughout the planning window (i.e., as an input for each stage of the model).

To determine the benefits provided by the campaign management system102, researchers have also performed research based on one or more experimental embodiments. In particular, in one experimental embodiment, an initial dataset included user interactions by approximately 1.2 million users with emails in a campaign by an e-retail company over a period of 2 years (December 2013 to December 2015). In addition to email tracking information (e.g., user interaction information), the dataset included customer purchase dates, purchase frequency, nature of the transactions (online or offline), email open rates, email click/selection rates, postal features, etc.

In this experimental embodiment, the campaign management system102considered only data from December 2013 to April 2014 for evaluating the models. Users receiving less than two messages within the five-month period were excluded. Additionally, the evaluation kept only the users who had either received more than ten messages or had been receiving messages from the company at least once every thirty days. This allowed the campaign management system102to have sufficient data for modeling the users' interactions. Additionally, the campaign management system102excluded transactional messages (identified as messages corresponding to campaigns sent to less than 400 recipients). Finally, messages with more than ten responses were removed because the responses were assumed to possibly include automated responses.

Thus, for the final dataset of the experimental embodiment, the number of recipient users was approximately 0.8 million, the number of messages was approximately 74.5 million, the number of unique campaigns was 2600, the percentage of emails opened was approximately 20.8%, and the percentage of clicks was approximately 2.3%. The campaign management system102randomly sampled 20,000 users and constructed features for the users. A maximum sequence length for the models is sixteen (i.e., only the last m=16 messages are considered for a user if available. Furthermore, the campaign management system102reported the results after performing 5-fold cross-validation on the randomly sampled subset of users.

For the classification models, the model including a recurrent neural network is compared against two other models—namely, logistic regression and feedforward neural network. Because logistic regression does not allow for explicitly taking sequential data as input, the evaluation used a feature-wise average of the feature vectors in the input sequence for the logistic regression. The actual output collected is whether the last message was opened. The evaluation used the same input and output formulation for the feedforward neural network. Table 1 below illustrates the performance for the three models in terms of area under a receiver operating characteristic curve (AUC) and F-measure. As illustrated in Table 1, the classification model using a recurrent neural network has the highest AUC and the second highest F-measure.

For the survival models, the model including a recurrent neural network is compared against two survival analysis models—namely, a Cox Proportional Hazard model (Cox-PH) and a Weibull Accelerated Failure Time model (Weibull AFT). Similar to the baseline classification models, the input attribute vector for the Cox-PH and Weibull AFT models is a feature-wise average of the feature vectors in the input sequence. Table 2 illustrates the performance for the three models in terms of a concordance index, which is a standard evaluation metric used in survival models. Specifically, the concordance index is a generalization of AUC to censored data and is defined as the fraction of comparable pairs of data points where the actual values and predicted order of time-to-open are the same. Comparable pairs are those in which at most one data point is censored. Table 2 indicates that the recurrent neural network model achieves the highest concordance index, which may be attributed to the use of recurrent neural networks for modeling sequential data.

As described above, the campaign management system102provides improvements over conventional models. Specifically, the campaign management system102can provide highly accurate predictions of interaction probability and hazard metrics. Additionally, the campaign management system102provides the flexibility for obtaining multiple different metrics based on the selected engagement metric.

As mentioned above, the campaign management system102assist in planning and executing digital content campaigns. In particular,FIGS. 5A-5Cillustrate graphical user interfaces generated by the campaign management system102in accordance with one or more embodiments for planning and executing strategic transmission times for electronic messages in a digital content campaign.

FIG. 5Aillustrates an embodiment of a campaign management interface500by which an administrator of a digital content campaign can manage the campaign. Specifically, the campaign management interface500operates on an administrator client device (e.g., the administrator client device104). Additionally, the campaign management interface500may be part of an administrator application (e.g., the administrator application116) that allows the administrator to manage various aspects of the campaign, including transmitting electronic messages for the campaign and/or executing the campaign by sending the electronic messages to one or more users.

As illustrated, the campaign management interface500provides a plurality of options that allow the administrator to select one or more campaign parameters for the campaign. For instance, the campaign parameters can include a message number parameter502and a message frequency parameter504. To illustrate, the message number parameter502indicates a total number of messages to send in connection with the campaign.FIG. 5Aillustrates that the administrator has selected the total number of messages to be “7” for the campaign.

Additionally, the message frequency parameter504indicates the maximum number of messages sent in a given time period. For example, the message frequency parameter504can be for a day, week, or any other time period.FIG. 5Aillustrates that the administrator has selected the maximum number of messages sent per day as three. Additionally, the time period may be modifiable, allowing the administrator to set the number of messages and the time period for the message frequency parameter504. Thus, the campaign management interface500can provide a variety of customizability options for the administrator to use in executing the campaign.

Furthermore, whileFIG. 5Aillustrates the message number parameter502in connection with a slider bar and the message frequency parameter504in connection with a dropdown menu, the campaign management interface500can provide other methods of allowing the administrator to choose the campaign parameters. For instance, either or both the message number parameter502and the message frequency parameter504can be set using a slider bar, dropdown menu, input field, or other method of inputting a number value.

Additionally, the campaign management interface500may also include additional campaign parameters. For example, the campaign management interface500may include an option to set a planning window (or analysis window) of the campaign. To illustrate, the administrator may select a campaign start date and a campaign end date to establish the planning window for the campaign. As previously described, the campaign management system can use the planning window to determine an analysis window for partitioning past messages to train the recurrent neural network (or a plurality of recurrent neural networks) for a plurality of potential recipients.

Additionally, the campaign management interface500can include an engagement metric parameter that allows the administrator to select a type of engagement metric for the campaign. As described previously, the types of engagement metric can include a time-to-interact metric or an interaction probability metric. In addition to the engagement metrics, the administrator can also select the specific interaction for the engagement metric. Thus, the administrator can specify that the time-to-interact metric or interaction probability metric corresponds to an open event, select event, click event, redirect event, conversion event, or other user interaction associated with an electronic message.

In one or more embodiments, once the campaign management system102has the campaign parameters, the campaign management system102partitions past messages within the planning window and trains a recurrent neural network for each user. Then, using each recurrent neural network, the campaign management system102generates engagement metrics for the send time bins according to the selected engagement metric type. The campaign management system102ranks the time bins based on the engagement metrics and selects the top K time bins according to the message number parameter502. Additionally, the campaign management system102can verify that the top K time bins comply with the message frequency parameter504.

After determining the time bins for a user based on the campaign parameters, the campaign management system102can further group the user with other users that have the same send time schedule based on the determined time bins for the users. As illustrated inFIG. 5A, the campaign management interface500includes a cluster region506that includes a plurality of clusters (e.g., represented by the circles). Each cluster corresponds to a group of users that have the same send time schedule (e.g., common time bins). The campaign management system102can generate as many clusters as there are groups of users with the same send time schedules. Furthermore, the visual sizes of the clusters within the campaign management interface500correspond to the number of users within the clusters.

In addition, in one or more embodiments, the campaign management system102also displays (e.g., positions) the clusters to indicate a measure of overlap between the send time schedules of different clusters. For example, two clusters that partially overlap in the cluster region506can indicate a partial overlap in transmission times (e.g., time bins) within the send time schedules of each cluster. Two additional clusters that overlap to a greater degree in the cluster region506can indicate a greater degree of overlap in transmission times (e.g., a greater number of common time bins).

In one or more embodiments, selecting a cluster508within the cluster region506causes the campaign management interface500to display a send time schedule510for the cluster. The send time schedule510can include a plurality of scheduled times for sending messages in the campaign. As previously described, the campaign management system102schedules the messages at times corresponding to the determined time bins for the user(s). Accordingly, the send time schedule510can include a plurality of times corresponding to the number of time bins that the campaign management system102selected for the cluster of users.

As illustrated inFIG. 5B, selecting a new cluster512of users updates the campaign management interface500with a new send time schedule514for the new cluster512. The send time schedule514for the new cluster512can include a different distribution of send times than the send time schedule510for the first cluster508. In particular, the send times for the new cluster512are based on the recurrent neural network(s) for the users in the new cluster512identifying time bins based on the users' interactions with past electronic messages. Accordingly, each cluster has a different send time schedule with a different distribution of time bins. In one or more embodiments, some clusters can have different numbers of send times in the send time schedule if the users do not have as many time bins that meet a predetermined threshold.

Additionally, changing the campaign parameters can cause the campaign management system102to perform new calculations for the users. For example, the administrator can select new campaign parameters (e.g., new planning window, new message number parameter502of “5”, and new message frequency parameter504of “1”). Specifically, the campaign management system102can partition the past electronic messages and train the recurrent neural networks for the users to identify send time schedules for the users according to the new campaign parameters. Analyzing the past electronic messages for the users based on new campaign parameters can result in new clustering of the users according to new send time schedules.

FIG. 5Cillustrates another embodiment of the campaign management interface500with a cluster region516comprising a different arrangement of clusters than the cluster region506ofFIGS. 5A-5B. As illustrated, selecting a cluster518causes the campaign management interface500to display a send time schedule520for the cluster518with the new campaign parameters. Likewise, selecting other clusters in the cluster region516can display corresponding send time schedules that also meet the new campaign parameters. The campaign management system102can perform a new analysis and generate new clusters of users with different sent time schedules every time the administrator changes the campaign parameters, or based on the administrator changing the campaign parameters and selecting an option to run a new analysis.

Although the description above describes training a recurrent neural network for each user, the campaign management system102may alternatively train a recurrent neural network for a group of users. For example, the campaign management system102can train a recurrent neural network for a selected target audience that includes users that all have a specific characteristic. When training the recurrent neural network, the campaign management system102can combine the past electronic messages and the user interactions for the target audience. Thus, the recurrent neural network can output predicted engagement metrics for the target audience, which can provide processing improvements over training recurrent neural networks for individual users. Additionally, the campaign management system102can generate and provide send time schedules for clusters that are based on groups of target audiences (e.g., multiple target audiences that have the same send time schedule).

FIG. 6illustrates a detailed schematic diagram of an embodiment of the campaign management system102ofFIG. 1. As shown, the campaign management system102can be part of computing device(s)600. Additionally, the campaign management system102can include, but is not limited to, a message manager602, a user profile database604, a campaign manager606, and a data storage manager608. The campaign management system102can be implemented on any number of computing devices (e.g., the server device(s)112and/or the administrator client device104ofFIG. 1). For example, the campaign management system102can be implemented in a distributed system of server devices for managing digital content campaigns for sending electronic messages such as emails to a plurality of client devices of a plurality of users. Alternatively, the campaign management system102can be implemented on a single computing device such as the administrator client device104ofFIG. 1.

In one or more embodiments, each of the components of the campaign management system102are in communication with one another using any suitable communication technologies. Additionally, the components of the campaign management system102can be in communication with one or more other devices including the administrator client device104, as illustrated inFIG. 1. It will be recognized that although the components of the campaign management system102are shown to be separate inFIG. 6, any of the subcomponents may be combined into fewer components, such as into a single component, or divided into more components as may serve a particular implementation. Furthermore, although the components ofFIG. 6are described in connection with the campaign management system102, at least some of the components for performing operations in conjunction with the campaign management system102described herein may be implemented on other devices within the environment.

The components of the campaign management system102can include software, hardware, or both. For example, the components of the campaign management system102can include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices (e.g., the computing device(s)600). When executed by the one or more processors, the computer-executable instructions of the campaign management system102can cause the computing device(s)600to perform the campaign management methods described herein. Alternatively, the components of the campaign management system102can comprise hardware, such as a special purpose processing device to perform a certain function or group of functions. Additionally or alternatively, the components of the campaign management system102can include a combination of computer-executable instructions and hardware.

Furthermore, the components of the campaign management system102performing the functions described herein with respect to the campaign management system102may, for example, be implemented as part of a stand-alone application, as a module of an application, as a plug-in for applications including content management applications, as a library function or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components of the campaign management system102may be implemented as part of a stand-alone application on a personal computing device or a mobile device. Alternatively or additionally, the components of the campaign management system102may be implemented in any application that allows creation and delivery of marketing content to users, including, but not limited to, applications in ADOBE® ANALYTICS CLOUD, such as ADOBE® ANALYTICS, ADOBE® AUDIENCE MANAGER, ADOBE® CAMPAIGN, ADOBE® EXPERIENCE MANAGER, ADOBE® MEDIA OPTIMIZER, ADOBE® PRIMETIME, ADOBE® SOCIAL, and ADOBE® TARGET. “ADOBE”, “ADOBE ANALYTICS CLOUD”, “ADOBE ANALYTICS”, “ADOBE AUDIENCE MANAGER”, “ADOBE CAMPAIGN”, “ADOBE EXPERIENCE MANAGER”, “ADOBE PRIMETIME”, “ADOBE SOCIAL”, and “ADOBE TARGET” are registered trademarks of Adobe Systems Incorporated in the United States and/or other countries.

As mentioned, the campaign management system102can include a message manager602to facilitate the management of electronic messages associated with one or more digital content campaigns. Specifically, the message manager602can manage the content of electronic messages, the types of electronic messages, and the sending/tracking of electronic messages to one or more recipients. For example, the message manager602can track electronic messages that are sent to recipients as part of a digital content campaign and then track user interactions with the messages. Additionally, the message manager602can determine characteristics (e.g., based on content, send times, related messages/campaigns) of the electronic messages for use by the campaign manager606. The message manager602can communicate with the data storage manager608to store past electronic messages previously sent to recipients and new electronic messages to be sent to recipients.

The message manager602can also partition electronic messages into time bins for use by the campaign manager606. Specifically, the message manager602can receive information about a campaign (e.g., campaign parameters such as a planning window, analysis window, number of messages, frequency of messages) and then partition past electronic messages into a plurality of time bins. The message manager602can partition the messages equally across the time bins or in using other partitioning methods, such as within specific time windows.

The campaign management system102also includes a user profile database604to facilitate the management and storing of user profile information associated with a plurality of recipient users. For example, the user profile database604can include a user profile for each user in a plurality of users to which the campaign management system102has sent messages or will send messages. A user profile can include personal information about a user (e.g., user demographics), as well as information indicating the user's historical interactions with electronic messages, user interests, and user preferences. The user profile database604can update a user profile as new information associate with the corresponding user becomes available (e.g., in response to new user interactions or changed user information).

In one or more embodiments, the campaign management system102includes a campaign manager606to manage digital content campaigns for one or more entities. In particular, the campaign manager606can manage a plurality of digital content campaigns for a plurality of entities that provide marketing messages or informational messages via one or more digital content campaigns to a plurality of users. Additionally, the campaign manager606can provide access to campaign data to campaign administrators in connection with managing and executing campaigns.

The campaign manager606can also provide analysis and predictive information associated with executing campaigns. For example, the campaign manager606can include a recurrent neural network610for generating predicted engagement metrics associated with new electronic messages to be sent to a user. The campaign manager606can train the recurrent neural network610based on past electronic messages and then utilize the recurrent neural network610to generate the predicted engagement metrics for time bins, as determined by the message manager602. The campaign manager606can also include a plurality of recurrent neural networks (e.g., one for each user or for each target audience) to generate predicted engagement metrics.

The campaign manager606can include a schedule manager612that uses the predicted engagement metrics to determine transmission times for sending new electronic messages in one or more digital content campaigns. In particular, the schedule manager612can rank time bins based on the predicted engagement metrics. Furthermore, the schedule manager612can use campaign parameters to determine a plurality of transmission times based on the ranked time bins for generating a send schedule for a user or group of users. The schedule manager612can also assist a campaign administrator in executing the digital content campaign by communicating with the message manager to send messages at transmission times according to a send schedule.

The campaign management system102also includes a data storage manager608(that comprises a non-transitory computer memory) that stores and maintains data associated with digital content campaigns. For example, the data storage manager608can include a database that stores a plurality of past electronic messages for each user. Additionally, the data storage manager608can store information associated with the users, such as user profiles and user interaction information associated with a user's interactions with the past electronic messages. Furthermore, the data storage manager608can store one or more recurrent neural networks610for one or more users.

Turning now toFIG. 7, this figure illustrates a flowchart of a series of acts700of determining and applying strategic digital content transmission times using machine-learning. WhileFIG. 7illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and/or modify any of the acts shown inFIG. 7. The acts ofFIG. 7can be performed as part of a method. Alternatively, a non-transitory computer readable medium can comprise instructions, that when executed by one or more processors, cause a computing device to perform the acts ofFIG. 7. In still further embodiments, a system can perform the acts ofFIG. 7.

The series of acts700includes an act702of partitioning past electronic messages into time bins. For example, act702involves partitioning a plurality of past electronic messages into a plurality of time bins, wherein the plurality of past electronic messages comprises past electronic messages for a user. Act702can involve determining a planning window for a digital content campaign, and, based on the planning window, generating the time bins such that the time bins span the planning window and the plurality of past electronic messages are divided among the time bins according to a grouping proportion. For example, the grouping proportion can include an equal division of past electronic messages across the plurality of time bins.

The series of acts700also includes an act704of training a recurrent neural network to generate an engagement metric corresponding to a time bin. For example, act704involves analyzing the past electronic messages via a recurrent neural network according to the sequential order to train the recurrent neural network to generate an engagement metric corresponding to a time bin for a new electronic message to the user.

As part of act704, the series of acts700include an act704aof generating a first predicted engagement metric for a first electronic message. For example, act704ainvolves generating a first predicted engagement metric for a first electronic message of the past electronic messages for the user based on a first time bin corresponding to the first electronic message. For example, the first predicted engagement metric can include a first predicted hazard metric, and the first predicted hazard metric indicates an instantaneous probability of a user interaction with the first electronic message resulting from sending the first electronic message to the user within the first time bin.

As part of act704, the series of acts700also includes an act704bof comparing the first predicted engagement metric to a ground-truth engagement metric. For example, act704binvolves comparing the first predicted engagement metric for the first electronic message to a ground-truth engagement metric for the first electronic message.

As part of act704, the series of acts700can include an act of providing, as input to the recurrent neural network, input attributes associated with the first electronic message, the input attributes comprising user interactions with at least one past electronic message sent prior to the first electronic message and the first time bin corresponding to the first electronic message.

The series of acts700can also include an act of generating a second predicted engagement metric for a second electronic message of the past electronic messages for the user based on a second time bin corresponding to the second electronic message. The series of acts700can then include an act of comparing the second predicted engagement metric for the second electronic message to a ground-truth engagement metric for the second electronic message. Additionally, the series of acts700can include an act of generating sequence data corresponding to the first electronic message, and further comprising instructions that, when executed by the at least one processor, cause the system to generate the second predicted engagement metric based on the sequence data corresponding to the first electronic message.

The series of acts700can further include an act of providing for display, within a user interface of an administrator client device, an option to select an interaction probability type of engagement metric or a hazard type of engagement metric. The series of acts700can also include an act of receiving, from the administrator client device, an indication of a selection of the hazard type of engagement metric. The series of acts700can additionally include an act of training the recurrent neural network utilizing the first predicted hazard metric based on the selection of the hazard type of engagement metric.

The series of acts700also includes an act706of determining a time bin for sending a new electronic message to the user. For example, act706involves for a digital content campaign comprising a new electronic message for the user, determining a time bin of the plurality of time bins for sending the new electronic message to a client device of the user.

As part of act706, the series of acts700also includes an act706aof generating a hazard metric for the time bin. For example, act706ainvolves generating, for the time bin, a hazard metric using a recurrent neural network trained on the past electronic messages to the user, wherein the hazard metric indicates an instantaneous probability of a user interaction with the new electronic message resulting from sending the new electronic message to the user within the time bin.

As part of act706, the series of acts700includes an act706bof selecting the time bin based on the hazard metric. For example, act706binvolves selecting the time bin from the plurality of time bins based on the hazard metric for the time bin.

Additionally, act706can also involve generating, for a second time bin of the plurality of time bins, a second hazard metric for the new message to the user using the recurrent neural network trained on the past electronic messages to the user. Act706can involve selecting the time bin from the plurality of time bins by comparing the hazard metric for the first time bin and the second hazard metric for the second time bin. Act706can further involve selecting the second time bin by comparing the first time bin and the second time bin to a frequency parameter that limits a frequency of sending electronic messages to the client device of the user.

The series of acts700can also include determining, utilizing the recurrent neural network, a subset of time bins for sending a plurality of electronic messages to the client device of the user. The series of acts700can then include generating a send schedule for sending the plurality of electronic messages according to the determined subset of time bins.

Additionally, the series of acts700includes an act706of executing the digital content campaign. For example, act706involves executing the digital content campaign by sending the new electronic message to the client device of the user based on the determined time bin. Act706can involve executing the digital content campaign in response to a request to initiate the digital content campaign from an administrator client device.

Additionally, the series of acts700can include an act of executing the digital content campaign by sending the plurality of electronic messages to the client device of the user according to a send schedule for the user. The series of acts700can also include grouping the user into a cluster of users corresponding to the send schedule, and presenting, within a user interface of the administrator client device, the cluster of users with the send schedule.

FIG. 8illustrates a block diagram of exemplary computing device800that may be configured to perform one or more of the processes described above. One will appreciate that one or more computing devices such as the computing device800may implement the campaign management system102. As shown byFIG. 8, the computing device800can comprise a processor802, a memory804, a storage device806, an I/O interface808, and a communication interface810, which may be communicatively coupled by way of a communication infrastructure812. In certain embodiments, the computing device800can include fewer or more components than those shown inFIG. 8. Components of the computing device800shown inFIG. 8will now be described in additional detail.

In one or more embodiments, the processor802includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions for dynamically modifying workflows, the processor802may retrieve (or fetch) the instructions from an internal register, an internal cache, the memory804, or the storage device806and decode and execute them. The memory804may be a volatile or non-volatile memory used for storing data, metadata, and programs for execution by the processor(s). The storage device806includes storage, such as a hard disk, flash disk drive, or other digital storage device, for storing data or instructions for performing the methods described herein.

The communication interface810can include hardware, software, or both. In any event, the communication interface810can provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device800and one or more other computing devices or networks. As an example and not by way of limitation, the communication interface810may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI.

Additionally, the communication interface810may facilitate communications with various types of wired or wireless networks. The communication interface810may also facilitate communications using various communication protocols. The communication infrastructure812may also include hardware, software, or both that couples components of the computing device800to each other. For example, the communication interface810may use one or more networks and/or protocols to enable a plurality of computing devices connected by a particular infrastructure to communicate with each other to perform one or more aspects of the processes described herein. To illustrate, the digital content campaign management process can allow a plurality of devices (e.g., a client device and server devices) to exchange information using various communication networks and protocols for sharing information such as electronic messages, user interaction information, engagement metrics, or campaign management resources.