Systems and methods for mixing models to optimize media placement

Methods and systems are disclosed for using multiple reach optimization models together. These models may be used by an optimization system that helps an advertisement campaign manager decide how advertisements should be allocated between different media assets. In some embodiments, the models may be run simultaneously while their results and compared, and execution of some models may be ended based on the comparisons. In some embodiments, models may be combined together to create new models. In some embodiments, results from a first model may used as input in a second model running concurrently. In some embodiments, results from a previous execution of a model may be used to warm-start a subsequent execution of the model.

BACKGROUND

Advertisement managers make decisions about where to place advertisements based on the number of people they expect to reach with the placement. Advertisement managers often use optimization models to estimate the number of people they will reach with different combinations of placements. Different optimization models may be associated with different results and different computational complexity. It may be determined which optimization model is best suited for any situation. It may also be difficult to run several optimization models simultaneously due to the computation resources required to do so.

SUMMARY

Systems and methods are disclosed herein for mixing models to optimize media placement. In some aspects, the method may include receiving a user input of a plurality of combinations of decision variables, wherein each of the plurality of combinations is associated with a potential media placement slot. For example, an optimization system may receive user input from an advertisement campaign manager of decision inputs describing advertisement slots on channel4between 7:00 pm and 9:00 pm.

In some embodiments, the method may include retrieving a first and a second subset of the plurality of combinations of decision variables. For example, the optimization system may retrieve a first subset of advertisement slots between 7:00 pm and 8:00 pm on channel4and a second subset of advertisement slots between 8:00 pm and 9:00 pm on channel4.

In some embodiments, the method may include applying the retrieved first subset to a first linear model that represents a first relationship between the plurality of combinations and associated reach metrics to generate a first reach metric associated with the first subset. In some embodiments, the method may include applying the retrieved second subset to a second non-linear model that represents a second relationship between the plurality of combinations and associated reach metrics to generate a second reach metric associated with the second subset. For example, the optimization system may apply the first model to the first subset to calculate an impressions range of 3500-4500 and the second model to the second subset to calculate an impressions range of 2800-3000.

In some embodiments, the method may include comparing the first reach metric and the second reach metric. For example, the optimization system may compare the first range of 3500-4500 and the second range of 2800-3000. In some embodiments, the method may include determining, based on the comparison, that the first reach metric is greater than the second reach metric. For example, the optimization system may determine that the range of 3500-4000 is greater than the range of 2800-3000.

In some embodiments, the method may include storing, in a media placement database, an identifier of a media asset in a media placement slot associated with the plurality of combinations of decision variables in the first subset. For example, the optimization system may purchase advertisement slots on channel4between 7 pm and 8 pm, and store an identifier of a media asset to be used as the advertisement in the purchased media placement slots.

In some embodiments, applying a subset to a model may include executing the model. For example, the optimization system may apply the first model to the first subset by executing equations and instructions associated the first model using control circuitry. The control circuitry may use the first subset as input values for the model being executed.

In some embodiments, the method may comprise applying the first and second subset to the first and second model for a threshold amount of time. For example, the optimization system may execute both models for one minute. In some embodiments, the first and second linear models may be combined. For example, the optimization system may use the first model for a first partition of the plurality of combinations and the second model for a second partition of the plurality of combinations. In some embodiments, the method may comprise using an output from one model as an input for another model. For example, the optimization system may tweak the second model using output from the first model. In some embodiments, comparing the reach metrics may comprise comparing an upper bound of one reach metric to a lower bound of another reach metric.

In some embodiments, the method may include storing an association between the plurality of combinations input by the user and the first model. In some embodiments, the method may include determining that a second input is similar to a first user input, and modifying the first linear model based on the first reach metric to warm-start the first linear model.

In some embodiments, the first and second reach metrics may be a range, a unique range value, or an impressions value. In some embodiments, the method may comprise determining a unique reach value using an impressions value. In some embodiments, the method may comprise generating for display the plurality of combinations of decision variables in the first subset, wherein the plurality of combinations of decision variables include times, lengths of media assets, times elapsed in media assets, audience demographics, and user equipment.

DETAILED DESCRIPTION

The disclosure is directed to methods and systems for using multiple reach optimization models together. These models may be used by an optimization system that helps an advertisement campaign manager decide how advertisements should be allocated between different media assets. In some embodiments, the models may be run simultaneously while their results are compared, and execution of some models may be ended based on the comparisons. In some embodiments, models may be combined together to create new models. In some embodiments, results from a first model may used as input in a second model running concurrently. In some embodiments, results from a previous execution of a model may be used to warm-start a subsequent execution of the model.

The amount of content available to users in any given content delivery system can be substantial. Consequently, many users desire a form of media guidance through an interface that allows users to efficiently navigate content selections and easily identify content that they may desire. An application that provides such guidance is referred to herein as an interactive media guidance application or, sometimes, a media guidance application or a guidance application. An application that allows advertisement campaign managers to place advertisements based on their preferences is called an optimization system. In some embodiments, the optimization system may place advertisements in an interactive media guidance application, or in media assets accessed through an interactive media guidance application.

FIG. 1shows illustrative grid of a program listings display100arranged by time and channel that also enables access to different types of content in a single display. Display100may include grid102with: (1) a column of channel/content type identifiers104, where each channel/content type identifier (which is a cell in the column) identifies a different channel or content type available; and (2) a row of time identifiers106, where each time identifier (which is a cell in the row) identifies a time block of programming. Grid102also includes cells of program listings, such as program listing108, where each listing provides the title of the program provided on the listing's associated channel and time. With a user input device, a user can select program listings by moving highlight region110. Information relating to the program listing selected by highlight region110may be provided in program information region112. Region112may include, for example, the program title, the program description, the time the program is provided (if applicable), the channel the program is on (if applicable), the program's rating, and other desired information.

Grid102may provide media guidance data for non-linear programming including on-demand listing114, recorded content listing116, and Internet content listing118. A display combining media guidance data for content from different types of content sources is sometimes referred to as a “mixed-media” display. Various permutations of the types of media guidance data that may be displayed that are different than display100may be based on user selection or guidance application definition (e.g., a display of only recorded and broadcast listings, only on-demand and broadcast listings, etc.). As illustrated, listings114,116, and118are shown as spanning the entire time block displayed in grid102to indicate that selection of these listings may provide access to a display dedicated to on-demand listings, recorded listings, or Internet listings, respectively. In some embodiments, listings for these content types may be included directly in grid102. Additional media guidance data may be displayed in response to the user selecting one of the navigational icons120. (Pressing an arrow key on a user input device may affect the display in a similar manner as selecting navigational icons120.)

Advertisement124may provide an advertisement for content that, depending on a viewer's access rights (e.g., for subscription programming), is currently available for viewing, will be available for viewing in the future, or may never become available for viewing, and may correspond to or be unrelated to one or more of the content listings in grid102. Advertisement124may also be for products or services related or unrelated to the content displayed in grid102. Advertisement124may be selectable and provide further information about content, provide information about a product or a service, enable purchasing of content, a product, or a service, provide content relating to the advertisement, etc. Advertisement124may be targeted based on a user's profile/preferences, monitored user activity, the type of display provided, or on other suitable targeted advertisement bases.

Another display arrangement for providing media guidance is shown inFIG. 2. Video mosaic display200includes selectable options202for content information organized based on content type, genre, and/or other organization criteria. In display200, television listings option204is selected, thus providing listings206,208,210, and212as broadcast program listings. In display200the listings may provide graphical images including cover art, still images from the content, video clip previews, live video from the content, or other types of content that indicate to a user the content being described by the media guidance data in the listing. Each of the graphical listings may also be accompanied by text to provide further information about the content associated with the listing. For example, listing208may include more than one portion, including media portion214and text portion216. Media portion214and/or text portion216may be selectable to view content in full-screen or to view information related to the content displayed in media portion214(e.g., to view listings for the channel that the video is displayed on).

The listings in display200are of different sizes (i.e., listing206is larger than listings208,210, and212), but if desired, all the listings may be the same size. Listings may be of different sizes or graphically accentuated to indicate degrees of interest to the user or to emphasize certain content, as desired by the content provider or based on user preferences. Various systems and methods for graphically accentuating content listings are discussed in, for example, Yates, U.S. Patent Application Publication No. 2010/0153885, filed Nov. 12, 2009, which is hereby incorporated by reference herein in its entirety.

Users may access content and the media guidance application (and its display screens described above and below) from one or more of their user equipment devices.FIG. 3shows a generalized embodiment of illustrative user equipment device300. The optimization system may be an example of illustrative user equipment device300. More specific implementations of user equipment devices are discussed below in connection withFIG. 4. User equipment device300may receive content and data via input/output (hereinafter “I/O”) path302. I/O path302may provide content (e.g., broadcast programming, on-demand programming, Internet content, content available over a local area network (LAN) or wide area network (WAN), and/or other content) and data to control circuitry304, which includes processing circuitry306and storage308. Control circuitry304may be used to send and receive commands, requests, and other suitable data using I/O path302. I/O path302may connect control circuitry304(and specifically processing circuitry306) to one or more communications paths (described below). I/O functions may be provided by one or more of these communications paths, but are shown as a single path inFIG. 3to avoid overcomplicating the drawing.

A user may send instructions to control circuitry304using user input interface310. User input interface310may be any suitable user interface, such as a remote control, mouse, trackball, keypad, keyboard, touch screen, touchpad, stylus input, joystick, voice recognition interface, or other user input interfaces. Display312may be provided as a stand-alone device or integrated with other elements of user equipment device300. For example, display312may be a touchscreen or touch-sensitive display. In such circumstances, user input interface310may be integrated with or combined with display312. Display312may be one or more of a monitor, a television, a liquid crystal display (LCD) for a mobile device, amorphous silicon display, low temperature poly silicon display, electronic ink display, electrophoretic display, active matrix display, electro-wetting display, electrofluidic display, cathode ray tube display, light-emitting diode display, electroluminescent display, plasma display panel, high-performance addressing display, thin-film transistor display, organic light-emitting diode display, surface-conduction electron-emitter display (SED), laser television, carbon nanotubes, quantum dot display, interferometric modulator display, or any other suitable equipment for displaying visual images. In some embodiments, display312may be HDTV-capable. In some embodiments, display312may be a 3D display, and the interactive media guidance application and any suitable content may be displayed in 3D. A video card or graphics card may generate the output to the display312. The video card may offer various functions such as accelerated rendering of 3D scenes and 2D graphics, MPEG-2/MPEG-4 decoding, TV output, or the ability to connect multiple monitors. The video card may be any processing circuitry described above in relation to control circuitry304. The video card may be integrated with the control circuitry304. Speakers314may be provided as integrated with other elements of user equipment device300or may be stand-alone units. The audio component of videos and other content displayed on display312may be played through speakers314. In some embodiments, the audio may be distributed to a receiver (not shown), which processes and outputs the audio via speakers314.

The guidance application may be implemented using any suitable architecture. For example, it may be a stand-alone application wholly-implemented on user equipment device300. In such an approach, instructions of the application are stored locally (e.g., in storage308), and data for use by the application is downloaded on a periodic basis (e.g., from an out-of-band feed, from an Internet resource, or using another suitable approach). Control circuitry304may retrieve instructions of the application from storage308and process the instructions to generate any of the displays discussed herein. Based on the processed instructions, control circuitry304may determine what action to perform when input is received from input interface310. For example, movement of a cursor on a display up/down may be indicated by the processed instructions when input interface310indicates that an up/down button was selected.

User equipment device300ofFIG. 3can be implemented in system400ofFIG. 4as user television equipment402, user computer equipment404, wireless user communications device406, or any other type of user equipment suitable for accessing content, such as a non-portable gaming machine. For simplicity, these devices may be referred to herein collectively as user equipment or user equipment devices, and may be substantially similar to user equipment devices described above. User equipment devices, on which a media guidance application may be implemented, may function as a standalone device or may be part of a network of devices. Various network configurations of devices may be implemented and are discussed in more detail below.

A user equipment device utilizing at least some of the system features described above in connection withFIG. 3may not be classified solely as user television equipment402, user computer equipment404, or a wireless user communications device406. For example, user television equipment402may, like some user computer equipment404, be Internet-enabled allowing for access to Internet content, while user computer equipment404may, like some television equipment402, include a tuner allowing for access to television programming. The media guidance application may have the same layout on various different types of user equipment or may be tailored to the display capabilities of the user equipment. For example, on user computer equipment404, the guidance application may be provided as a web site accessed by a web browser. In another example, the guidance application may be scaled down for wireless user communications devices406.

In system400, there is typically more than one of each type of user equipment device but only one of each is shown inFIG. 4to avoid overcomplicating the drawing. In addition, each user may utilize more than one type of user equipment device and also more than one of each type of user equipment device.

System400includes content source416and media guidance data source418coupled to communications network414via communication paths420and422, respectively. Paths420and422may include any of the communication paths described above in connection with paths408,410, and412. Communications with the content source416and media guidance data source418may be exchanged over one or more communications paths, but are shown as a single path inFIG. 4to avoid overcomplicating the drawing. In addition, there may be more than one of each of content source416and media guidance data source418, but only one of each is shown inFIG. 4to avoid overcomplicating the drawing. (The different types of each of these sources are discussed below.) If desired, content source416and media guidance data source418may be integrated as one source device. Although communications between sources416and418with user equipment devices402,404, and406are shown as through communications network414, in some embodiments, sources416and418may communicate directly with user equipment devices402,404, and406via communication paths (not shown) such as those described above in connection with paths408,410, and412.

In a third approach, users of user equipment devices inside and outside a home can use their media guidance application to communicate directly with content source416to access content. Specifically, within a home, users of user television equipment402and user computer equipment404may access the media guidance application to navigate among and locate desirable content. Users may also access the media guidance application outside of the home using wireless user communications devices406to navigate among and locate desirable content.

As referred herein, the term “in response to” refers to initiated as a result of. For example, a first action being performed in response to another action may include interstitial steps between the first action and the second action. As referred herein, the term “directly in response to” refers to caused by. For example, a first action being performed directly in response to another action may not include interstitial steps between the first action and the second action.

As referred to herein, “advertisement manager” refers to a user of the optimization system. An advertisement manager may be a person, a group of people, or a computer program. An advertisement manager may use the optimization system to determine placement slots for advertisements.

As referred to herein, “decision variables” are variables that define different advertisement slots. Decision variables may include targeted demographics (e.g., females under 30 or retired males), times (e.g., primetime, 2:00 pm-3:00 pm), locations (e.g., New York City, Massachusetts), mediums (e.g., television program, on-demand movie), lengths of advertisements (e.g., 30 seconds, two minutes), and other such variables. An advertisement manager may combine decision variables in any combination to define advertisement slots of interest. For example, an advertisement manager may indicate, using user input interface310, that the advertisement manager is interested in advertisement slots that are 30 seconds long, and targeted at females under 30 living in New York City who watch primetime television programs. A combination of decision variables may describe any number of advertisement slots.

FIG. 5shows two exemplary models for determining reach values associated with different combinations of decision variables. Model500is a linear model, and model550is a non-linear quadratic model. Non-linear models may include other types of models as well, such as cubic models or polynomial models. The optimization system may execute models500and550, as well as other types of models, simultaneously.

Models500and550include x-axes510and560that represent different combinations of decision variables. For example, x-axis510includes points512,514,516,518, and520that each represent advertisement slots defined by a different combination of decision variables. Point512may represent advertisement slots in a television program on channel5from 6:00 pm to 6:30 pm. Point514may represent advertisement slots in a television program on channel5from 6:30 pm to 7:00 pm. Point516may represent advertisement slots in a television program on channel6from 6:00 pm to 6:30 pm. Point518may represent advertisement slots in a television program on channel6from 6:30 pm to 7:00 pm. Point520may represent advertisement slots in a television program on channel7from 6:00 pm to 7:00 pm. The optimization system may have received input from the advertisement manager using user input interface310indicating an interest in decision variables including channels watched by females under 30 between 6:00 pm and 7:00 pm. The optimization system may have created points512,514,516,518, and520by analyzing the user input, determining which channels fulfill the demographic criteria, and creating different combinations of the decision variables entered by the user based on the determined channels.

X-axis560includes points562,564,566,568, and570. Points on x-axis560may have been created in a manner similar to points on x-axis510. Points on x-axis560may represent the same combinations of decision variables as points on x-axis560. X-axes510and560may include any number of points representing any combination of decision variables. Points on x-axes510and560may represent combinations of decision variables that may or may not represent overlapping advertisement slots.

Y-axes502and552represent one or more reach metrics. As referred to herein, “reach metrics” include impressions and reach values, as well as upper and lower bounds associated with each of these values. In some embodiments, y-axes502and552represent impressions. As referred to herein, “impressions” describes the number of viewers reached by an advertisement. In some embodiments, y-axes502and552represent reach. As referred to herein, “reach” describes the number of unique viewers reached by an advertisement.

Model500may include curve524. Curve524may be created by using a linear relationship that takes as input different combinations of decision variables and produces as an output an estimated impressions value. Curve524at point512may represent the estimated impressions value if advertisements are placed in the advertisement slots represented by the combination of decision variables described by point512. Curve524at point514may represent the estimated impressions value if advertisements are placed in the advertisement slots represented by the combination of decision variables described by point512and point514. Model500may include upper and lower bounds to represent uncertainty in the model. The upper and lower bounds may be created using the same linear relationship as used to create curve524, or a different linear relationship. Curve526may represent a lower bound of impressions values and curve522may represent an upper bound of impressions values.

The optimization system may periodically calculate a reach value from impressions values created by model500. The frequency of the calculation may be determined by the computational complexity of the calculation. For example, if the optimization system takes 10 seconds to perform a reach calculation from impressions values, and model500has a total execution time of 60 seconds, then the optimization system may perform the reach calculation six times or less during the total execution time. In another example, if the optimization system takes 20 seconds to perform a reach calculation from impressions values, and model500has a total execution time of 60 seconds, then the optimization system may perform the reach calculation three times or less during the total execution time.

Model550may be similar to model500. Curves572,574, and576may be created using one or more quadratic relationships. Curve572may represent an upper bound while curve576may represent a lower bound. In some embodiments, model550may be more computationally complex than model500. The optimization system may determine that model550requires more resources such as memory, processing power, or bandwidth than model500.

The optimization system may compare the results of model500and model550to determine whether the execution of either of the models should be ended. In some embodiments, the optimization system may compare the impressions values estimated by model500to the impressions values estimated by model550. In some embodiments, the optimization system may compare the reach value periodically computed for model500to the reach value periodically computed for model550. In some embodiments, if model550estimates reach values instead of impressions values, the optimization system may compare the reach value periodically computed for model500to the reach values estimated by model550. In some embodiments, the optimization system may compare the upper and lower bounds of one model to the upper and lower bounds of the other model.

In some embodiments, the optimization system may determine that upper bound572of model550is lower than lower bound526of model500at a current execution time, and hence end execution of model550. In some embodiments, the optimization system may determine that the trend produced by curve574of model550will be lower than curve524of model500at the end of the execution period, and hence end execution of model550. In some embodiments, the optimization system may determine that curve574of model550closely resembles curve524of model500, but model550is much more computationally complex than model500, and hence end execution of model550. In some embodiments, the optimization system may reallocate the processing power used by model550to model500.

FIG. 6shows two executions of the same model to illustrate a warm-start model. As referred to herein, a “warm-start model” refers to a model that uses information from a previous execution to start the current execution. Execution600shows an initial execution of the model, while execution650shows a later execution of the model. The model may be any type of model, for example, model500or model550as discussed above. X-axes602and652represent different combinations of reach metrics as discussed above. Y-axes604and654represent reach metrics as discussed above. Curves606and656represent outputs of the models. For simplicity, upper bounds and lower bounds are not shown inFIG. 6.

Execution600may be an initial execution of a model. The optimization system may analyze curve606to determine that curve606increases sharply in the beginning, and much more slowly after the initial sharp increase. Execution650may be a later execution of the same model. Execution650may use the same or different decision variables as execution600. The optimization system may use the information from execution600to provide a warm-start and eliminate the sharp rise from the beginning of execution650. Hence, optimization system may allow execution650to provide more accurate results than execution600.

FIG. 7shows two models approaching output limits. Model700may be a linear model as described in relation to model500. Model750may be a quadratic model as described in relation to model550. X-axes702and752may represent different combinations of decision variables. Y-axes704and754may represent reach metrics. Curves706and756represent outputs of the models. For simplicity, upper bounds and lower bounds are not shown inFIG. 7. As referred to herein, “limit” refers to a value on the y-axis that a curve approaches but does not reach.

The optimization system may analyze model700to determine model700has a limit708. The optimization system may also analyze model750to determine model750has a limit758. In some embodiments, the optimization system may determine that limit758is lower than or equal to limit708and hence end execution of model750. In some embodiments, the optimization system may determine that difference between limits708and758is less than a threshold difference, and that model750is more computationally complex than model700, and hence end execution of model750.

FIG. 8shows an illustrative process for executing multiple models simultaneously. Process800may be used by the optimization system to execute models500and550simultaneously. The optimization system may use process800to execute any number of models simultaneously.

Process800begins at block802, when the optimization system receives decision variables from an advertisement manager through user input interface310. For example, the optimization system may receive decision variables describing a demographic of retired males, a time slot of 4:00 pm-5:00 pm, and advertisements 30 seconds in length.

Process800continues to block804, when the optimization system executes multiple models. The optimization system may execute any combination and number of linear and non-linear models. In some embodiments, the optimization system may select the models to execute based on the received decision variables. For example, the optimization system may always execute a cubic model if the decision variables describe advertisements 30 seconds in length. The optimization system may execute the multiple models simultaneously. In some embodiments, the optimization system may use control circuitry304to execute the multiple models.

Process800continues to block806, when the optimization system determines whether the threshold amount of time has been reached. The optimization system may execute the models for a threshold amount of time. The optimization system may periodically compare the current execution time of the models with the threshold amount of time using control circuitry304. If the threshold amount of time has been reached, process800proceeds to block808. If the threshold amount of time has not been reached, process800proceeds to block810.

Process800continues to block808, when the optimization system ends the currently executed models and presents the results to the user. In some embodiments, the user may be the advertisement manager. If only one model was being executed, the optimization system uses control circuitry304to end the model currently being executed and presents the reach metrics calculated by the model to the user. If multiple models were being executed, the optimization system uses the control circuitry304to select one of the models and then present the reach metrics calculated by the model to the user. The optimization system may select one of multiple models by determining which model determined the highest reach metric, the most accurate reach metric, or analyzed the most combinations of decision variables. In some embodiments, the optimization system may use control circuitry304to calculate the average of the reach metrics calculated by all models, and present the result to the advertisement manager.

Process800continues to block810, when the optimization system determines whether multiple models are currently being executed using control circuitry304. In some embodiments, control circuitry304may use a table to track all models currently being executed. If the optimization system ends execution of a model, control circuitry304may remove the model from the table, or change its status in the table. In some embodiments, control circuitry304stores a measure of the models' computational complexity in the table as well. The computational complexity may be measured during the current execution of the model, or may be predetermined and retrieved from storage circuitry308. If more than one model is currently being executed, process800proceeds to block812. The optimization system can then end certain models' execution if needed. If only one model is currently being executed, process800proceeds to block806.

Process800continues to block812, when the optimization system determines whether the time to periodically calculate reach has been reached. In some embodiments, control circuitry304may determine whether the current execution time is divisible by the period for calculating reach values. For example, the optimization system may use control circuitry304to determine that the current execution time of the models, from the time when the models started execution at block804to the current time, is 40 seconds. The optimization system may further determine that reach should be calculated every 8 seconds. Since 40 is divisible by 8, the optimization system may determine that the time to periodically calculate reach has been reached. If the optimization system determines the time has been reached, process800proceeds to block814. If the optimization system determines the time has not been reached, process800proceeds to block818.

Process800proceeds to block814, when the optimization system calculates range of reach. In some embodiments, the models being executed may determine impressions values. The optimization system may calculate, using control circuitry304, reach from the impressions value. The optimization system may also calculate, using control circuitry304, upper and lower bounds of reach from upper and lower bounds of impressions values, if they exist. In some embodiments, the optimization system may calculate rough estimates of reach values from the impressions values using heuristics or mathematical estimations. In some embodiments, if the models calculate reach instead of impressions values, then the optimization system may skip block814for those models. In some embodiments, calculating reach may be more computationally complex than calculating impressions values, so reach value may be calculated only periodically and not directly by the models.

Process800proceeds to block816, when the optimization system compares reach ranges and ends execution of certain models if needed. In some embodiments, the optimization system may end execution of a first model if the upper bound of the first model is less than the lower bound of second model, as described in relation toFIG. 5. Control circuitry304may update its table of models currently being executed if the optimization system ends execution of a model. Control circuitry304may also redistribute computational resources among the models after resources are freed due to the ending of a model. In some embodiments, the optimization system may not end execution of any models. Process800then proceeds to block818.

Process800proceeds to block818, when the optimization system determines whether a limit is being approached. Limits are discussed above in relation toFIG. 7. If the optimization system determines a limit for at least one of the models using control circuitry304, process800proceeds to block820. If no limits are determined, process800proceeds to block822.

Process800proceeds to block820, when the optimization system ends more computationally complex models based on the models' limits. Control circuitry304may determine the complexity of each model by using the information present in its table of models currently being executed and their associated complexity. If the optimization system determines that two models have the same limit, the optimization system may end execution of the more computationally complex model. In some embodiments, if the optimization system determines that the difference between the limits associated with two models is less than a threshold difference, the optimization system may end execution of the more computationally complex model. In some embodiments, the optimization system may compare the limit associated with a first model to a upper bound, a lower bound, or a curve of a second model, if a limit is not available for the second model. In some embodiments, the optimization system may determine a trend for a model and estimate the model's output value at the end of the execution time based on the model's trend. The optimization system may then use the estimated output at the end of the execution time as the limit of the model. Process800then proceeds to block822.

Process800proceeds to block822, when the optimization system trades reach metrics among the models currently being executed. Control circuitry304may use output values from one model as an input in another model to tweak the coefficients being used in the model and calculate more consistent results between models. Process800then proceeds to block806.

FIG. 9shows an illustrative process for beginning execution of multiple models. Process900maybe used to begin execution of models500and550. Process900may be used to begin execution of any number and combination of models.

Process900begins as block902, when the optimization system receives user input of decision variables. The optimization system may receive the input in a manner similar to that described in relation to block802.

Process900proceeds to block904, when the optimization system determines whether the current input matches a previous input. If the optimization system determines that a current input matches a previous input, process900proceeds to block906. If the optimization system determines that a current input does not match a previous input, process900proceeds to block908.

Control circuitry304may store a list of previously received inputs in storage circuitry308. In some embodiments, the optimization system may determine that the current input matches a previous input if the current input is identical to an input in the list of previously received input. In some embodiments, the optimization system may determine that a current input matches a previous input if the current input is a subset of a previous input in the list of previously received input. For example, control circuitry304may determine that a current input of decision variables describing television programs broadcast between 7:00 pm and 8:00 pm is a subset of a previous input of decision variables describing television programs broadcast between 6:00 pm and 9:00 pm. Hence, the optimization system may determine that the current input matches a previous input. In some embodiments, the optimization system may determine that the current input matches a previous input if a difference between the current input and a previous input is less than a threshold difference. For example, if a current input of decision variables describes first-run programs on channel7, and a previous input describes repeat programs on channel7, control circuitry304may determine that the difference between these inputs is less than a threshold difference. Hence, the optimization system may determine that the current input matches a previous input.

Process900proceeds to block906, when the optimization system warm-starts the previous model. The optimization system may send the previous model the output from the previous execution of the previous model, to increase the previous model's accuracy and/or precision in the current execution. Warm-start models are described in further detail in relation toFIG. 6. Process900then proceeds to block916.

Process900proceeds to block908, when the optimization system determines whether the input matches only one of the available models. In some embodiments, control circuitry304may store a list of all available models and their corresponding requirements using storage circuitry308. Control circuitry304may compare the input decision variables to the requirements of all available models to determine which available models the input matches. For example, model A may have the requirement of being applicable only to broadcast television programs, model B may have the requirement of being applicable only for advertisement campaigns costing under $1 million, and model C may have the requirement of being applicable only for advertisement campaigns targeted towards males. The optimization system may receive user input including decision variables describing a $500,000 campaign targeted to females using broadcast television. Control circuitry304may compare the user input to the requirements of the available models to determine that models A and B match the user input.

If the optimization system determines that the input matches only one of the available models, process900proceeds to block910. If the optimization system determines that the input matches more than one available model, process900proceeds to block912. In some embodiments, the optimization system may use any threshold number for this comparison. For example, the optimization system may determine whether the input matches three of the available models.

Process900proceeds to block910, when the optimization system begins execution of the matching model. Control circuitry304may execute the matching model as described in relation toFIG. 8. After the model has been executed, process900proceeds to block916.

Process900proceeds to block912, when the optimization system begins execution of all matching models determined to match the input at block908. In some embodiments, control circuitry304may execute the multiple models in separate partitions, threads, cores, or virtual machines to keep the models separate. In some embodiments, control circuitry304may execute only a threshold number of multiple models. For example, if control circuitry304determines at block908that five models match the input, and the threshold number of models that can be simultaneously executed is three, control circuitry304may select three models with the least computational complexity to execute simultaneously. The models may be executed as described in relation toFIG. 8.

Process900proceeds to block916, when the optimization system creates an association in a database. Storage circuitry308may include a database with associations between various inputs entered and corresponding models executed. This database may be used by control circuitry304to determine whether a current input matches a previous input at block904. This database may also include information about the output of each execution of a model. Control circuitry304may use this information from the database to warm-start a previous model at block906. This database may also include information about the requirements associated with each model. Control circuitry304may use this information to determine if the input matches one or more available models at block908. Storage circuitry308may store this database in the form of a list, wherein each model in the list is associated with multiple inputs and results of executions.

Control circuitry304may edit the database in storage circuitry308to include an association between the models executed and the input received at block902. Control circuitry304may further edit the database in storage circuitry308to include information about the output of each execution of the models. In some embodiments, models whose execution was ended early as described in relation to blocks816and820may not have associations created for them in the database. In some embodiments, models whose execution was ended early as described in relation to blocks816and820may have associations created for them in the database along with an indication that the models' execution was ended early.

Process900proceeds to block920, when the optimization system determines whether the database in storage circuitry308includes models with complimentary ranges. Control circuitry304may analyze each model and the inputs associated with it to determine of models have complimentary ranges. Control circuitry304may determine that models are complimentary if they are associated with mutually exclusive decision variables. For example, control circuitry304may determine that model A is executed without being ended early for advertisement campaigns costing under $1 million, while model B is executed without being ended early for advertisement campaigns costing above $1 million. Control circuitry304may further determine that the decision variable describing campaigns costing under $1 million is mutually exclusive with the decision variable describing campaigns costing above $1 million. Control circuitry304may hence determine these models are complimentary. If the optimization system determines the database has models with complimentary ranges, process900continues to block922. If the optimization system determines the database has no models with complimentary ranges, process900ends.

Process900proceeds to block922, when the optimization system combines complimentary models. For example, control circuitry304may determine that models A and B are complimentary because model A is executed without being ended early for advertisement campaigns costing under $1 million, while model B is executed without being ended early for advertisement campaigns costing above $1 million. Control circuitry304may create a new model which uses information from model A for decision variables describing campaign costs under $1 million, and information from model B for decision variables describing campaign costs above $1 million. Control circuitry304may store this new model in the database.

FIG. 10shows an illustrative process for processing user input of decision variables. User input may be received from an advertisement manager. Any number or combination of decision variables may be received using process1000.

Process1000begins at block1002, when the optimization system receives user input from an advertisement manager through a user interface. In some embodiments, the optimization system may receive user input from a user interface accessed by the advertisement manager through user input interface310. In some embodiments, the optimization system may receive user input through communications network414from a user interface accessed by the advertisement manager at a remote location. In some embodiments, the optimization system may use the user interface to provide predetermined options for the advertisement manager to select. In some embodiments, the optimization system may use the user interface to allow the advertisement manager to enter in free-form input describing decision variables.

Process1000proceeds to block1004, when the optimization system analyzes the user input to remove ambiguities. Removing ambiguities may comprise analyzing the user input and processing it to replace natural language identifiers with specific identifiers of media assets, channels, times, zip codes, and the like.

In some embodiments, the optimization system may map target demographic information to particular media assets. For example, control circuitry304may determine that the user input includes decision variables describing a target audience of toddlers. Control circuitry304may determine that television programs “Bob the Builder” and “Bubble Guppies” are targeted at toddlers, and replace the user input decision variables with decision variables describing the media assets “Bob the Builder” and “Bubble Guppies.”

In some embodiments, the optimization system may map location information to specific zip codes. For example, control circuitry304may determine that the user input includes decision variables describing a target location of New York City. Control circuitry304may identify zip codes associated with New York City, and replace the user input decision variables with decision variables identifying New York City zip codes.

In some embodiments, the optimization system may map channel attributes to specific channels. For example, control circuitry304may determine that the user input includes decision variables describing channels targeted towards women. Control circuitry304may determine that channels5and8are targeted at women, and replace the user input decision variables with decision variables describing channels5and8.

Process1000proceeds to block1006, when the optimization system modifies the input to express it in a standardized form. In some embodiments, the optimization system may include a standard template for user input with fields such as “media assets,” “times (hh:mm:ss),” “length of advertisement (seconds),” “channels,” “zip codes,” and the like. The optimization system may modify the input to express it using this standardized template.

For example, control circuitry304may receive, from user input interface310, user input including decision variables that describe a target audience of toddlers, a location of New York City, and channels targeted at women. As described above, control circuitry304may remove ambiguities from this input. Control circuitry304may then express this user input in a standardized form as follows. Control circuitry304may include default information for fields such as “times” and “length of advertisement” where no user input information exists.

FIG. 11shows an exemplary process1100by which control circuitry304can receive user input of decision variables using user input interface310. Display312may display a form to an advertisement manager, the form containing various options, menus, buttons, checkboxes, radio buttons, or text fields associated with different decision variables. The advertisement campaign manager may be able to choose any combination of decision variables using this form. In some embodiments, the advertisement campaign manager may verbally enter information into the form using speech-to-text software.

Process1100begins at block1102, when the optimization system receives selection of a “submit” button in the user interface used by the advertisement manager. Control circuitry304may receive a user selection in the form of a click using a mouse, a brief touch using a touchscreen, a verbal commend through a speech-to-text software, or any such manipulation using user input interface310. An advertisement campaign manager may select the “submit” button after completing the form and selecting the desired decision variables.

Process1100proceeds to block1104, when the optimization system retrieves data from the user interface. The user interface may be the form used by the advertisement campaign manager. Control circuitry304may proceed sequentially through each form field and store the user input associated with each form field using storage circuitry308. The data from each form field may be stored as a boolean, a number, or a text value. For example, control circuitry304may store a radio button selection of either a “Yes” or a “No” option as a boolean value. In another example, control circuitry304may store a budget text field entry of “$1000” as a number value. In another example, control circuitry304may store a desired demographic text field entry as a text value of “women under 30.” In some embodiments, the optimization system may send the stored data using communications network414to a separate server for analyzing user input.

Process1100proceeds to block1106, when the optimization system identifies a first subset of fields with predetermined response options. Control circuitry304may determine a field has predetermined response options if an advertisement campaign manager can only enter a finite number of responses for the field. For example, control circuitry304may determine that drop-down menus, radio buttons, and checkbox buttons have predetermined response options. Control circuitry304may identify the first subset by sequentially identifying the type of each field in the user interface, and determining which fields have finite response options.

Process1100proceeds to block1108, when the optimization system determines selected options for each field in the first subset of fields. Control circuitry304may retrieve the user input stored at block1104and determine which of the finite options associated with each field was selected by the advertisement campaign manager.

Process1100proceeds to block1110, when the optimization system stores selected options for each field in the first subset of fields as integer values. Control circuitry304may associate each option of the finite number of options for a field with an integer value, and store the integer value(s) corresponding to the user selected options. For example, a field for selected desired geographical range may have four options, any number of which can be selected by the user using checkbox buttons. The four options may be: “East,” “West,” “North,” and “South.” Control circuitry304may associate these three options with the integers 0, 1, 2, and 3, respectively. Control circuitry304may retrieve data indicating the user selected options “East” and “North” and store these as text values at block1104. Control circuitry304may then retrieve these text values, determine they correspond to integer values of 0 and 2, and store integer values 0 and 2 associated with the geographical range field using storage circuitry308. In some embodiments, control circuitry304may delete the text values stored at block1104once integer values have been stored at block1110.

Process1100proceeds to block1112, when the optimization system identifies a second subset of fields with open-ended responses. Control circuitry304may determine that a field has open-ended responses if users can enter any alphanumeric combination as input associated with that field. For example, fields with open-ended input may include text input, numerical values, and numerical ranges.

Process1100proceeds to block1114, when the optimization system creates a buffer for each field in the second subset, wherein the length of the buffer equals or exceeds the length of the open-ended responses in the second subset of fields. In some embodiments, control circuitry304may create a single buffer to store all open-ended user input fields. In some embodiments, control circuitry304may create separate buffers associated with each open-ended user input field.

Process1100proceeds to block1116, when the optimization system stores the open-ended responses for each field in the second subset of fields as strings in the buffer. In some embodiments, control circuitry304may store the user input as text in the buffer, and store the buffer in long-term memory using storage circuitry308. In some embodiments, control circuitry304may edit the text to standardize it before storage, as described in relation toFIG. 10.

FIG. 12shows an illustrative process1200for retrieving advertisement slot identifiers associated with a subset of the plurality of received combination of decision variables. Process1200may be used to execute blocks1604and1608. Process1200may use as an input combinations of decision variables received from an advertisement campaign manager, as described in relation toFIG. 11.

Process1200begins at block1202, when the optimization system accesses a remote database listing potential advertisement slots. In some embodiments, the remote database may be associated with media content source416or media guidance data source418. Control circuitry304may access the remote database using communications network414. In some embodiments, the potential advertisement slots may include advertisement slots during television programs, on-demand movies, interactive Internet content, or any other media asset that allows advertisement placement.

Process1200proceeds to block1204, when the optimization system retrieves a user-selected option for a first field from the user input. In some embodiments, the user input may have been processed as described in relation toFIG. 11. For example, control circuitry304may retrieve a user-selected option for a field such as a selected option of “$1000” for a budget field. Control circuitry304may retrieve the user-selected option using storage circuitry308.

Process1200proceeds to block1206, when the optimization system generates a query for the remote database based on the retrieved option. Control circuitry304may generate a query in a standardized format. For example, if control circuitry304received a user-selected budget value of “$1000,” control circuitry304may generate a query of the form: “BUDGET_VALUE<=1000.00?”. The query format may be generated based on database rules, XML formatting, or any other appropriate markup language.

Process1200proceeds to block1208, when the optimization system transmits the generated query to the database. Control circuitry304may transmit the query using communications network414.

Process1200proceeds to block1210, when the optimization system receives identifiers of potential advertisement slots that match the generated query. For example, control circuitry304may receive identifiers of advertisement slots that would fit into a budget of under $1000. The remote database may generate these identifiers by cross-referencing the received query with information associated with advertisement slots stored in the remote database. Control circuitry304may receive any number of identifiers. Control circuitry304may store the received identifiers in a temporary cache using storage circuitry308.

Process1200proceeds to block1212, when the optimization system determines whether any of the received identifiers have been received before, and remove those. Control circuitry304may perform this determination to create a set including only unique identifiers of advertisement slots, and no duplicate identifiers of advertisement slots.

Process1200proceeds to block1214, when the optimization system stores non-duplicate received identifiers in a subset of plurality of combinations of decision variables. Control circuitry304may transfer the non-duplicate received identifiers from a temporary cache to long-term storage. Control circuitry304may then delete the data in the temporary cache.

Process1200proceeds to block1216, when the optimization system retrieves a user-selected option for a next field from the user input. Block1216may be performed by control circuitry304in a manner similar to block1206. In some embodiments, process1200may be terminated if no further fields of user input are available, and all fields have been processed by control circuitry304using process1200. Process1200then returns to block1206.

FIG. 13shows an illustrative process1300for applying a subset of decision variables to a model. The decision variables may be received from an advertisement campaign manager as described in relation toFIG. 11.

Process1300begins at block1302, when the optimization system retrieves a model representing a relationship. Control circuitry304may retrieve the model from a remote source or from local memory using communications network414and/or storage circuitry308. The model may be a linear, non-linear, piecewise, or any other type of model. In some embodiments, the model may be expressed as a mathematical function. In some embodiments, the model may be expressed as a graph.

Process1300proceeds to block1304, when the optimization system retrieves a first combination of decision variables representing an advertisement slot. Control circuitry304may generate all possible permutations of the user input combinations of decision variables. Control circuitry304may then randomly select one of these permutations as the first permutation, or select the first permutation in the sequence. For example, control circuitry304may use decision variables “ad slot for New York City,” “ad slot for Boston,” and “cost between $41 and $42” to generate the following permutations: {New York City, $41}, {New York City, $42}, {Boston, $41}, and {Boston, $42}. Control circuitry304may then select {New York City, $41} as the first combination, and sequentially select the other permutations in further iterations of process1300.

Process1300proceeds to block1306, when the optimization system uses the combination as an input for the model. The model may take as an input combinations of decision variables and produce as an output reach metrics, such as a reach or impressions value. The combination of decision variables may be represented as numbers to be used as an input for the model. Control circuitry304may represent the decision variables as numbers as described in relation to block1100.

Process1300proceeds to block1308, when the optimization system replaces the input variable in the relationship associated with the model with the selected combination of decision inputs. For example, control circuitry304may replace a variable with a value corresponding to the selected combination in a mathematical relationship associated with the model. In another example, control circuitry304may determine a value corresponding to the combination on a graph associated with the model.

Process1300proceeds to block1310, when the optimization system calculates the relationship to find a reach metric. In some embodiments, control circuitry304may calculate a reach metric, as well as an upper bound and a lower bound for the reach metric to represent uncertainty or flexibility in the model. In one example, control circuitry304may replace a variable with a value corresponding to the selected combination in a mathematical relationship associated with the model, and then calculate the mathematical relationship to determine a result. In another example, control circuitry304may determine a value corresponding to the combination on a graph associated with the model, and then determine a value on the x-axis or y-axis corresponding to the value.

Process1300proceeds to block1312, when the optimization system forms a connection between a previous data point and a current data point. In some embodiments, control circuitry304may use a default value as a previous data point if none exists. Control circuitry304may determine a trend for the graph based on the connection between the previous data point and the current data point. In some embodiments, control circuitry304may use the connection as an error check. For example, control circuitry304may determine that no connection associated with a particular model should have a negative slope. If control circuitry304creates a connection with a negative slope associated with the current data point, control circuitry304may discard the current data point or recalculate the current data point.

Process1300proceeds to block1314, when the optimization system stores a data point comprising the combination as input, the reach metric as output, and the connection. For example, control circuitry304may use an input of {New York City, $41} to determine that it will result in an impressions value of 4000. Control circuitry304may also determine a connection between a previous data point and the data point has a slope of 2. Control circuitry304may store the following data structure using storage circuitry308: {{New York City, $41}, 4000, 2}.

Process1300proceeds to block1316, when the optimization system adds the next combination of decision variables to the current combination. For example, control circuitry304may perform the next iteration of process1300with the combination of decision variables made by combining current combination ({New York City, $41}) to a next combination ({New York City, $42}). Process1300then returns to block1306. If control circuitry304determines no further combinations of decision variables are left to be used as an input for the model, control circuitry304may terminate process1300.

FIG. 14shows an illustrative process1400for comparing the reach metrics output by one model to the reach metrics output by another model. It is understood that any number of models and their output values may be compared as discussed in relation to process1400.

Process1400begins at block1402, when the optimization system stores an upper bound, a lower bound, a limit, and curve values for a first model in a first group of registers. Control circuitry304may store these values as integer or floating point values. Control circuitry304may store a “Not an Integer” or “NaN” value if one of these values does not exist. The curve value may represent an output reach metric value obtained from a model. Control circuitry304may retrieve these values from storage circuitry308and store them in the registers. Control circuitry304may store these values in registers because it is quicker to compare values in registers than it is to compare values stored using storage circuitry308.

Process1400proceeds to block1404, when the optimization system stores an upper bound, a lower bound, a limit, and curve values for a second model in a second group of registers. Control circuitry304may perform as discussed in relation to block1402.

Process1400proceeds to block1406, when the optimization system compares all combinations of the first group of registers with the second group of registers, and stores the results indicating which model is associated with the higher value for each comparison in a data structure. For example, control circuitry304may determine that an upper bound of a first model is more than an upper bound of a second model and more than a lower bound of a second model. Control circuitry304may further determine that a lower bound of a first model is less than an upper bound of a second model and a lower bound of a second model. Control circuitry304may store these comparison results as {first model, first model, second model, second model} using storage circuitry308. If a “NaN” value is involved in a comparison, control circuitry304my store a “null” value as the comparison result to indicate the comparison did not occur. Control circuitry304may use these stored results to perform the determinations in blocks1408-1412.

Process1400proceeds to block1408, when the optimization system determines whether the upper bound of a first model is lower than the lower bound of a second model. If control circuitry304determines this condition is true, process1400proceeds to block1416and control circuitry304ends execution of the first model. If control circuitry304determines this condition is not true, process1400proceeds to block1410.

Process1400proceeds to block1410, when the optimization system determines whether a limit of a first model is lower than a limit of a second model. If control circuitry304determines this condition is true, process1400proceeds to block1416, and control circuitry304ends execution of the first model. If control circuitry304determines this condition is not true, process1400proceeds to block1412.

Process1400proceeds to block1412, when the optimization system determines whether the difference between the same value for both models is lower than a threshold. For example, control circuitry304may determine whether the difference between lower bounds for both models is lower than a threshold. If control circuitry304determines this condition is true, process1400proceeds to block1416, and control circuitry304ends execution of the more computationally complex model. If control circuitry304determines this condition is not true, control circuitry304ends execution of process1400.

FIG. 15shows an illustrative process1500for storing an identifier of a media asset in a media placement slot. The optimization system may use process1500for buying advertisement slots based on the output of the optimization models. The optimization system may buy advertisement slots on behalf of an advertisement campaign manager in accordance with decision variables input by the advertisement campaign manager.

Process1500begins at block1502, when the optimization system retrieves an advertisement slot identifier associated with a combination of decision variables in a first subset. Control circuitry304may use process1200for retrieving advertisement slot identifiers.

Process1500proceeds to block1504, when the optimization system transmits an identifier to a remote server with a purchase request. The remote server may be a media placement database. Control circuitry304may transmit the identifier using communications network414. The remote server may be configured to process purchasing requests and allow advertisement campaign managers to purchase advertisement slots.

Process1500proceeds to block1506, when the optimization system receives a price value associated with identifier from remote server. Control circuitry304may receive the price value using communications network414.

Process1500proceeds to block1508, when the optimization system determines that an advertisement campaign manager's account has sufficient funds. For example, if control circuitry304received a price value of $40 and an advertisement campaign manager's account has $400, control circuitry304may determine the advertisement campaign manager's account has sufficient funds. In another example, if control circuitry304received a price value of $40 and an advertisement campaign manager's account has $4, control circuitry304may determine the advertisement campaign manager's account does not have sufficient funds, and terminate process1500.

Process1500proceeds to block1510, when the optimization system transmits approval to the remote server with the advertisement campaign manager's payment details. The payment details may include credit card information, bank account information, an escrow account information, and the like.

Process1500proceeds to block1512, when the optimization system receives confirmation of purchase of an advertisement slot associated with the identifier from the remote server. The confirmation may indicate that the advertisement campaign manager is allowed to display an advertisement during the advertisement slot.

Process1500proceeds to block1514, when the optimization system transmits an identifier of a media asset provided by the advertisement campaign manager to the remote server. For example, control circuitry304may receive confirmation an advertisement slot has been purchased. Control circuitry304may then transmit to the remote server, using communications network414, an advertisement that should be provided to viewers during the advertisement slot.

Process1500proceeds to block1516, when the optimization system modifies the advertisement campaign manager's account to include an association between the media asset and the advertisement slot identifier. Control circuitry304may generate an association to confirm to the advertisement campaign manager that a certain advertisement will be displayed during an advertisement slot. In some embodiments, control circuitry304may also modify the funds associated with the advertising campaign manager's account. For example, if control circuitry304bought an advertisement slot with a price value of $40 and an advertisement campaign manager's account had $400, control circuitry304may modify the advertisement campaign manager's account to include $360.

FIG. 16shows an illustrative process1600for using multiple optimization models to optimize reach. Process1600may be performed by the optimization system in response to input received from an advertisement campaign manager. Although process1600is described in relation to two models for illustrative purposes, it is understood any number of models may be used in a similar manner.

Process1600begins at block1602, when the optimization system receives a user input of a plurality of combinations of decision variables, wherein each of the plurality of combinations is associated with a potential media placement slot. Control circuitry304may execute block1602as described in relation toFIGS. 10 and 11.

Process1600proceeds to block1604, when the optimization system retrieves a first subset of the plurality of combinations of decision variables. Process1600then proceeds to block1606, when the optimization system applies the retrieved first subset to a first linear model that represents a first relationship between the plurality of combinations and associated reach metrics to generate a first reach metric associated with the first subset. Control circuitry304may perform this process as described in relation toFIG. 13.

Process1600also proceeds to block1608, when the optimization system retrieves a second subset of the plurality of combinations of decision variables. Process1600then proceeds to block1610, when the optimization system applies the retrieved second subset to a second non-linear model that represents a second relationship between the plurality of combinations and associated reach metrics to generate a second reach metric associated with the second subset. Control circuitry304may execute these blocks as described in relation to blocks1604and1606respectively. Control circuitry304may execute blocks1604and1606in parallel with blocks1608and1610.

Process1600proceeds to block1612, when the optimization system compares the first reach metric and the second reach metric. Process1600then proceeds to block1614, when the optimization system determines, based on the comparison in block1612, that the first reach metric is greater than the second reach metric. Control circuitry304may execute blocks1612and1614as described in relation toFIG. 14. In some embodiments, control circuitry304may execute blocks1604-1614in a loop until a threshold execution time has been reached.

Process1600then proceeds to block1616, when the optimization system stores, in a media placement database, an identifier of a media asset in a media placement slot associated with the plurality of combinations of decision variables in the first subset. Control circuitry304may execute block1616as described in relation toFIG. 15. Control circuitry304may terminate process1600once block1616has been executed.

It should be noted that processes800-1600, or any step thereof could be performed on, or provided by, the optimization system on any of the devices shown inFIGS. 3-4. For example, processes800-1600may be executed by processing circuitry304(FIG. 3) as instructed by processing circuitry implemented on user equipment402,404, and/or406(FIG. 4) in order to provide information to a user. In addition, one or more steps of processes800-1600may be incorporated into, or combined with, one or more steps of any other process or embodiment.

It is contemplated that the steps or descriptions ofFIGS. 8-16may be used with any other embodiment of this disclosure. In addition, the steps and descriptions described in relation toFIGS. 8-16may be done in alternative orders or in parallel to further the purposes of this disclosure. For example, each of these steps may be performed in any order or in parallel or substantially simultaneously to reduce lag or increase the speed of the system or method.