Patent Description:
Many images and video frames include areas where text and other content may be inserted without obscuring important part or parts of the image or the video frame. Those areas are sometimes referred to as copy space. For example, an image or a video frame may focus on a ship and the people on that ship. However, that image or video frame may include areas where all is shown is a sky and/or ocean. Those areas may be used for displaying text or other media content (e.g., images). In one example, those areas may be used to display additional content items to a viewer. In another example, copy space may be used to insert links and other information into an image or a video frame.

Various systems are available today that enable a curator to mark copy space on an image. That process is usually time consuming and inefficient when hundreds of thousands of images of video frames must be marked. Therefore, it is desirable to automatically and accurately identify copy space on images and video frames.

<CIT> discloses: Some embodiments of the present disclosure provide a content integration system. The content integration system is configured to retrieve a source digital content, retrieve a target digital content, identify a region within the target digital content for placing or integrating the source digital content, and place or integrate the target digital content onto the identified region of the source digital content. The content integration system can be configured to place the source digital content into the target digital content in an aesthetically-pleasing, unobtrusive, engaging, and/or otherwise favorable manner. The content integration system can be particularly useful for advertisements, enhanced expression, entertainment, information, or communication.

<CIT> discloses: One embodiment provides a method comprising identifying a product placement opportunity for a product in a frame of a piece of content during playback of the piece of content on a display device. The method further comprises determining a location in the frame to insert product placement content for the product based on a learned statistical model representing learned placement patterns related to the product. The method further comprises modifying the product placement content based on one or more objects present in the frame, and inserting a product placement for the product in the piece of content by inserting the modified product placement content in the frame based on the location. The modified product placement content appears to occur naturally in the piece of content.

<CIT> discloses: A method and apparatus inserts virtual advertisements or other virtual contents into a sequence of frames of a video presentation by performing real-time content-based video frame processing to identify suitable locations in the video for implantation. Such locations correspond to both the temporal segments within the video presentation and the regions within an image frame that are commonly considered to be of lesser relevance to the viewers of the video presentation. This disclosure presents a method and apparatus that allows a non-intrusive means to incorporate additional virtual content into a video presentation, facilitating an additional channel of communications to enhance greater video interactivity.

<CIT> discloses systems and methods for identifying one or more portions of images or video frames that are appropriate for augmented overlay of advertisement or other visual content, and augmenting the image or video data to include such additional visual content. Identifying the portions appropriate for overlay or augmentation may include employing one or more machine learning models configured to identify objects or regions of an image or video frame that meet criteria for visual augmentation. The pose of the augmented content presented within the image or video frame may correspond to the pose of one or more real-world objects in the real world scene captured within the original image or video.

The disclosed embodiments have other advantages and features which will be more readily apparent from the detailed description, the appended claims, and the accompanying figures (or drawings). A brief introduction of the figures is below.

One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the embodiments described herein.

The figures use like reference numerals to identify like elements. A letter after a reference numeral, such as "115a," indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as "<NUM>," refers to any or all of the elements in the figures bearing that reference numeral.

The placement of text over an image is an important part of producing high-quality.

Disclosed herein are methodologies for detecting space suitable for overlaying media content onto an image (copy space). One methodology presented is a neural networks-based system, method and computer readable storage medium for detecting space suitable for overlaying media content onto an image (copy space). Another methodology presented is a heuristics-based system, method and computer readable storage medium for detecting space suitable for overlaying media content onto an image (copy space).

Disclosed herein are a neural networks-based system, method and computer readable storage medium for detecting space suitable for overlaying media content onto an image (copy space). The system receives a candidate image which may be an image or a video frame. Generally, an image or a video frame may have some space for inserting media content without covering vital portions of the image. The candidate image is then input into a neural network that has been trained with training data including a multitude of images and, for each image of the plurality of images, one or more corresponding bounding boxes. The neural network may output coordinates and one or more dimensions representing one or more bounding boxes for inserting media content into the candidate image. The one or more bounding boxes may be transmitted with a request for a media content item to be displayed in a bounding box. The request may include the one or more dimensions of the one or more bounding boxes. In response to the request, the media content item may be received, and the candidate image and the media content item overlaid on top of the candidate image within the bounding box may be displayed.

<FIG> illustrates an exemplary system for detecting space suitable for overlaying media content onto an image. The system may include a media content storage system <NUM> and a media content insertion system <NUM>. Media content insertion system <NUM> includes a communications module <NUM>, bounding box detection module <NUM>, and overlay generation module <NUM>.

Communications module <NUM> may include hardware, software, or a combination of the two to communicate and may be used by media content insertion system <NUM> to communicate with other systems (e.g., media content storage system) and devices. Communications module <NUM> may receive a candidate image for overlaying media content. Communications module <NUM> may receive the candidate image from another device or a location within the media content insertion system. The candidate image may be any appropriately formatted image. In some embodiments, the candidate image may be a frame of a video content item. Communications module <NUM> may pass the candidate image to bounding box detection module <NUM>.

Bounding box detection module <NUM> may receive the candidate image from communications module <NUM> and process the candidate image. Bounding box detection module <NUM> may input the candidate image into a neural network to receive as output one or more bounding boxes. The input operation occurs after the neural network has been trained with training data. The training data in this instance includes images and, for each image one or more corresponding bounding boxes. In some embodiments, the neural network may be a convolution neural network. The network may use as input image pixels transformed into vectors. In some embodiments, the neural network may be a multi-layer perceptron that includes an input layer, a hidden layer and an output layer. This type of network may include other layers.

As referred to herein, a bounding box refers to an area on an image for placing media content. Although a bounding box is illustrated as a rectangle, the bounding box may be any shape, including but not limited to, a square, a circle, a pentagon, and any other suitable shape. The bounding box may be of an irregular dimension and/or include multiple shapes.

For example, a bounding box may be represented by a rectangle including one or more coordinates on the candidate image and one or more dimensions extending from the one or more coordinates. In some embodiments, a bounding box may be represented by a coordinate that is counted in terms of a number of pixels horizontally and vertically from the upper left-hand corner of the candidate image and two dimensions. That coordinate may represent the upper left-hand corner of the bounding box. The two dimensions may include a vertical offset and a horizontal offset extending from the coordinate. Based on the coordinate and the offsets, a rectangular bounding box having an area based on the offsets and the location based on the coordinate may be generated.

<FIG> illustrates images with exemplary corresponding bounding boxes. Image <NUM> illustrates bounding box <NUM>. However, different size/shape bounding boxes may be added to image <NUM>. Image <NUM> is illustrated with two corresponding bounding boxes (bounding box <NUM> and bounding box <NUM>). Although bounding box <NUM> and bounding box <NUM> are illustrated as rectangular those bounding boxes may be of different shapes and sizes.

In some embodiments, media content insertion system <NUM> may receive, from the neural network, a first coordinate representing a first offset along a horizontal axis of the candidate image and a second coordinate representing a second offset along a vertical axis of the candidate image. The two coordinates may represent a particular pixel on the candidate image. The pixel may serve as a point associated with the bounding box. For example, as discussed above, the point may be a center of a circle, a corner of a rectangle, a triangle, a square, or another suitable point. Media content insertion system <NUM> may receive a first dimension extending from the first coordinate along the horizontal axis. For example, the first dimension may be a number of units (e.g., pixels) that media content insertion system <NUM> may use to calculate the next point (e.g., in a horizontal direction) associated with the bounding box. In some embodiments, the first dimension may correspond to a diameter of a circle, a radius of a circle, a side of a rectangle, a side of a square or a side of a triangle.

In some embodiments, media content insertion system <NUM> may receive a second dimension extending from the second coordinate along the vertical axis. The second dimension may be a number of units (e.g., pixels) that media content insertion system <NUM> may use to calculate the next point (e.g., in a vertical direction) associated with the bounding box. In some embodiments, the second dimension may correspond to radii or diameters of an oval, a second side of a rectangle, or a triangle. In some embodiments, the received bounding box may only have one dimension and the second dimension may not be shown.

In some embodiments, media content insertion system <NUM> may receive a probability that a corresponding bounding box is located on the candidate image in an area suitable for inserting the media content into the candidate image. For example, the neural network may return a multitude of possible bounding boxes each having a probability of the bounding box being a valid bounding box for inserting media content into the candidate image. Bounding box detection module <NUM> may determine for each bounding box whether a corresponding probability meets a threshold probability. For example, the threshold probability may be a percentage value, a value between zero and one or another suitable value. The threshold probability may be set to, for example,. <NUM> or to another suitable value. Bounding box detection module <NUM> may, for example, retrieve the threshold probability from memory. In response to determining that a probability for a particular bounding box does not meet the threshold probability, bounding box detection module <NUM> removes the particular bounding box from the request. That is, the bounding boxes received from the neural network may be filtered based on probability.

<FIG> illustrates an example of a data structure with corresponding fields that represents an output of the neural network representing a bounding box. Data structure <NUM> of <FIG> may include one or more fields. Field <NUM> may include one or more coordinates associated with the bounding box. For example, field <NUM> may include a coordinate of a corner of a rectangular bounding box, a center of a circular bounding box or another suitable point associated with the bounding box. In some embodiments field <NUM> may include a vector of points representing a bounding box. Field <NUM> may include a first dimension stored as a number of units (e.g., pixels) to offset from a point represented by the coordinates in field <NUM>. Field <NUM> may include a second dimension stored as a number of units (e.g., pixels) to offset from a point represented by the coordinates in field <NUM>. There may be other dimension stored in data structure <NUM>. Field <NUM> stores a probability that the bounding box represented by the particular data structure is a valid bounding box. In some embodiments, each field of data structure <NUM> may be a coordinate in a vector of coordinates representing an irregular bounding box.

As discussed above, in some embodiments, the bounding box(es) may be of different shapes. For example, a bounding box may be a square (e.g., represented by a coordinate and one offset), a circle (e.g., represented by a coordinate being the center of the circle and one dimension representing a radius or a diameter of the circle), an oval (e.g., represented by two radii and a coordinate at the center). As discussed above, a bounding box may be any shape and/or may have irregular dimensions. In some embodiments, a bounding box may be represented by a vector of points representing coordinates on the candidate image. The vectors and connections of those vectors may represent a bounding box.

In some embodiments, the neural network may be trained using images (sometimes referred to as training images) with known bounding boxes. The neural network may be trained using a training module (not shown) which may be part of media content insertion system <NUM> or a different system. The training module may receive a multitude of training images and corresponding vectors. Each vector may include a set of coordinates and a set of dimensions, representing a particular bounding box. In some embodiments, some training images may include more than one bounding box and, thus, may correspond to multiple vectors. The training module may input the training images and the corresponding vectors into the neural network to train the neural network.

Referring back to <FIG>, bounding box detection module <NUM> may receive, from the neural network, coordinates and one or more dimensions representing one or more bounding boxes for inserting media content into the candidate image. Bounding box detection module <NUM> may receive a data structure as illustrated in <FIG>. Bounding box detection module <NUM> passes the one or more bounding boxes to overlay generation module <NUM>.

Overlay generation module <NUM> may include hardware, software, or a combination of both. Overlay generation module <NUM> may generate a request for a media content item to be displayed in a bounding box received from bounding box detection module <NUM>. The request may include one or more dimensions of the one or more bounding boxes. Overlay generation module <NUM> may pass the request to communications module <NUM> that may transmit the request to media content storage system <NUM> (e.g., to communications module <NUM>).

Communications module <NUM> may receive the request and pass the request to media content selection module <NUM>. Communications module <NUM> may include hardware, software, or a combination of both. Media content selection module <NUM> may extract data associated with each bounding box (e.g., one or more data structures <NUM> of <FIG>) and search for media content (e.g. in a database) having dimensions that match the dimensions of the one or more bounding boxes. Media content selection module <NUM> may retrieve (e.g. from the database) one or more media content items that match the one or more bounding box dimensions and generate a response to the request that includes one or more media content items. In some embodiments, in addition to each media content item, the response may include metadata associated with each media content item. The metadata may include the dimensions of the corresponding media content item as well as other information (e.g., name, type (image, video, etc.)), orientation, and/or other suitable information. Media content selection module <NUM> may pass the response to communications module <NUM>. Communications module <NUM> may transmit the response to media content insertion system <NUM>.

In some embodiments, the response may include one or more identifiers of the one or more media content items. Each identifier may be part of a data structure that also includes dimensions for each media content item. Other information may be included in the data structure (e.g., name, type (image, video, etc.)). This may be advantageous because media content items may be large in size and may take a long time to transfer in comparison to just transferring an identifier. In some embodiments, the response may include a link to each media content item so that it can be retrieved.

Media content insertion system <NUM> may receive, using communications module <NUM>, the response to the request that includes one or more media content items. Communications module <NUM> may pass the response to overlay generation module <NUM>. Overlay generation module <NUM> may include hardware, software, or a combination of both. Overlay generation module <NUM> may extract from the response the one or more media content items.

In some embodiments, overlay generation module <NUM> may receive multiple media content items and may select one of the received media content items to be displayed based on the dimensions of the content item matching the dimensions of a bounding box that has a highest probability of being a valid bounding box. For example, in response to the request, media content insertion system <NUM> may receive multiple media content items corresponding to multiple bounding boxes. Overlay generation module <NUM> may identify from the plurality of media content items, a particular media content item corresponding to a bounding box with the highest probability. Overlay generation module <NUM> may access the metadata of each received media content item and retrieve the dimensions for each media content item. Overlay generation module <NUM> may retrieve probabilities for each of the detected bounding boxes and determine which bounding box has the highest probability.

Overlay generation module <NUM> may retrieve dimensions of the bounding box with the highest probability and compare those dimensions to the dimensions of media content items. Overlay generation module <NUM> may select the media content item that best matches the dimensions. For example, some media content items may be too large, thus, overlay generation module <NUM> may filter those out. Overlay generation module <NUM> may select the media content item with the dimensions closest to those of the selected bounding box. In some embodiments, overlay generation module <NUM> may take into account orientation of the media content item. For example, if a particular media content item should be displayed in portrait orientation and the orientation of the bounding box would force the media content item to be sideways, based on the dimensions, overlay generation module <NUM> may filter (e.g., remove from consideration) that particular media content item. When overlay generation module <NUM> selects a media content item, overlay generation module <NUM> may cause a display of the candidate image and the media content item overlaid on top of the candidate image within the bounding box.

<FIG> illustrates various media content items being overlaid over a candidate image. Image <NUM> illustrates a media content item in one bounding box. In this instance the dimensions of the bounding box are not apparent from the overlay. Image <NUM> illustrates an example of a bounding box that fits the media content item. The bounding box may be illustrated by the box around the media content item. In this case, the media content item may be text or an image. In some embodiments, the media content item may be associated with a selectable graphical user element that when selected instructs a computing device to open a link to, for example, a web page associated with the content of the bounding box.

Image <NUM> illustrates another media content item that may use the same bounding box. It should be noted that the media content items in image <NUM> and <NUM> are different. The media content item in image <NUM> has a transparent component while the media content item in image <NUM> has no transparent portions. Images <NUM>, <NUM>, and <NUM> illustrate other examples of bounding boxes on the same image.

<FIG> is a flowchart illustrating an overall process for detecting space suitable for overlaying media content onto an image using a neural networks-based approach, as performed by the neural networks-based media content insertion system <NUM>, according to one embodiment. Various embodiments can perform the steps of <FIG> in different orders than those indicated herein. Moreover, other embodiments can include different and/or additional steps than the ones described herein.

At <NUM>, media content insertion system <NUM> receives a candidate image for a media content overlay. For example, media content insertion system <NUM> may receive the candidate image through a network using a network interfaced device. Media content insertion system <NUM> may store the candidate image in memory.

At <NUM>, media content insertion system <NUM> inputs the candidate image into a neural network that has been trained with training data including a plurality of images and, for each image of the plurality of images, one or more corresponding bounding boxes. The neural network may be stored in a memory. The neural network may include an application programming interface for inputting images and corresponding metadata. Media content insertion system <NUM> may use one or more processors to perform the input and may use those same processors to process the candidate image within the neural network.

At <NUM>, media content insertion system <NUM> receives, from the neural network, coordinates and one or more dimensions representing one or more bounding boxes for inserting media content into the candidate image. Media content insertion system <NUM> may receive the coordinates and dimensions and store them in memory.

At <NUM>, media content insertion system <NUM> transmits a request for a media content item to be displayed in a bounding box of the one or more bounding boxes, the request including the one or more dimensions of the one or more bounding boxes. For example, media content insertion system <NUM> may use a network interface device to transmit the request over a network.

At <NUM>, media content insertion system <NUM> receives the media content item in response to the request. For example, media content insertion system <NUM> may receive the request using a network interface device through a network. At <NUM>, media content insertion system <NUM> causes a display of the candidate image and the media content item overlaid on top of the candidate image within the bounding box. For example, media content insertion system <NUM> may transmit the overlaid candidate image to a client device for display. In some embodiments, media content insertion system <NUM> may use a visual interface to cause the display of the overlaid candidate image.

In some embodiments, the image may be a frame of a video content item. Media content insertion system <NUM> may process those frames in a different manner. For example, media content insertion system <NUM> may determine that the candidate image is a video frame associated with a video content item. Overlay generation module <NUM> may make the determination based on an indicator in the image metadata or based on another signal received with the candidate image.

Bounding box detection module <NUM> may retrieve a set of video frames of the video content item. For example, the bounding box detection module may request the frames using an application programming interface (API). The set of video frames may include video frames that are played subsequently to the candidate image. For example, the candidate image may be a first frame in a set of consecutive frames to be played back by a video player and it may be desirable to insert a media content item (e.g., an advertisement) into at least some of those frames.

Bounding box detection module <NUM> may input each video frame of the set of video frames into the neural network. For example, bounding box detection module <NUM> may input the frames into the neural network consecutively based on order of playback. Bounding box detection module may receive, from the neural network for each video frame in the set of video frames, corresponding coordinates and corresponding dimensions representing one or more bounding boxes. That is, bounding box detection module <NUM> may process each video frame in the same manner as a candidate image to identify on or more bounding boxes in each frame.

Bounding box detection module <NUM> may identify in each video frame of the set of video frames, a bounding box matching a bounding box in each other video frame within the set. For example, bounding box detection module <NUM> may determine that the same copy space (e.g., bounding box) is available on all the frames in the set or at least a subset of frames in the set. Bounding box detection module <NUM> may then include the bounding box in the request. In some embodiments, bounding box detection module may detect one bounding box in some frames in the set and another bounding box in other frames. Based on that bounding box detection module <NUM>, may include both bounding boxes in the request. After the response to the request is received, overlay generation module <NUM> may cause a display of the set of video frames and the media content item overlaid on top of each of the plurality of subsequent video frames within the bounding box. That is, the video may be played back with one or more bounding boxes included in one or more consecutive frames.

Disclosed herein are system, method and computer readable storage medium for a heuristics-based approach for detecting space suitable for overlaying media content onto an image (i.e., copy space, insertion space). The system receives an image which may be an image or a video frame. Generally, an image or a video frame may have space for inserting media content without covering vital portions of the image. Embodiments of the system process the image to determine occupied and unoccupied spaces in the image, and subsequently select regions in the unoccupied spaces to overlay media content on the image. The image is processed using a number of image processing techniques in order to automatically propose spaces for inserting media content onto the candidate image. The proposed spaces may then be further analyzed using a heuristic rules-based approach to select insertion spaces for inserting media content. The selected insertion spaces may be defined by bounding boxes by the system. Subsequently, one or more media content items may be selected for insertion onto the selected bounding boxes in the image. The system may then cause a display of the image with the selected media content item overlaid onto the image within the selected bounding boxes.

<FIG> illustrates an exemplary image <NUM>. As depicted, one portion <NUM> of the image <NUM> may have multiple visual elements, while another portion <NUM> of the image <NUM> may be devoid of visual elements or may be identified as having visually non-essential elements (for e.g., textural background, blank space, clouds, tree canopy, beach sand, etc.). Embodiments described herein are directed to automatically identify insertion spaces, also termed bounding boxes herein, with visually non-essential elements, and where media content items may be inserted without visually impacting any visually essential elements of the image. For example, an insertion space <NUM> may be identified by an embodiment such that any media content may be overlaid on top of image <NUM> within the identified insertion space <NUM>.

<FIG> illustrate exemplary image processing that may be performed on an image in accordance with some embodiments.

<FIG> shows the greyscale image <NUM> depicted in <FIG>. In embodiments described herein, when insertion space needs to be selected for a color image, preprocessing may be performed on the color image to first obtain a corresponding greyscale image <NUM> and subsequently processed as described herein. In some embodiments, the greyscale image <NUM> may be blurred (e.g., by using a low pass filter) to remove noise in the image before further processing occurs.

<FIG> depicts the results <NUM> of performing edge detection on the image <NUM>. The identified image discontinuities in the gradient image <NUM> are organized by an edge detection filter into a set of line segments, i.e., edges. The depicted edge-detected image <NUM> is obtained using the ShenCasten edge detection filter on gradient image <NUM>. Similar results may be obtained using the Sobel filter.

<FIG> depicts the results <NUM> of performing saliency filtering on the edge-detected image <NUM>. Saliency filtering is an image segmentation process that assists in locating objects and boundaries. The depicted saliency-filtered image <NUM> is obtained using the saliency filter on edge detected image <NUM>.

<FIG> depicts the results <NUM> of further processing the greyscale image <NUM> to block out text, faces, and people using single shot text detector (ssd) networks and standard classifiers and object detection. This processing determines regions <NUM> and <NUM> which are subsequently blocked out as occupied spaces by people detection and face detection classifiers. While not depicted herein, any textual elements in the greyscale image <NUM> may also be blocked out by processing the image <NUM> using ssd networks. In embodiments herein, this processing to block out text, faces, and people may be carried out as independent processing on the greyscale image <NUM> to provide further indications of occupied and unoccupied space, in addition to the image processing described herein.

<FIG> illustrate exemplary image processing that may be performed on an image to generate a binary matrix in accordance with an embodiment.

<FIG> depicts the results <NUM> of performing local adaptive binarization on the saliency filtered image <NUM>.

<FIG> depicts a foreground/background separation image <NUM> that is generated from the original greyscale image <NUM> and the saliency filtered image <NUM> on which adaptive binarization has been performed, i.e., on image <NUM>.

<FIG> depicts the results of using a visual entropy filter on the foreground/background separation image <NUM> to generate the visual entropy filtered image <NUM>.

<FIG> depicts the generation of a binary matrix image <NUM> from the visual entropy filtered image <NUM> using downsampling. A binary matrix image <NUM> has values of <NUM>'s and <NUM>'s (representing black and white values respectively) for each pixel. Thus, unoccupied spaces in the binary matrix image or the downsampled binary matrix image may be viewed, without limitation, as consisting of spaces with pixel values of '<NUM>'.

<FIG> depicts an image <NUM> in which candidate bounding boxes have been proposed in the binary matrix image <NUM>. A "max rectangle in binary matrix" technique may be used to establish a set of rectangle candidates that satisfy predefined threshold parameters (e.g., a minimum width, a minimum height, etc.). Image <NUM> depicts two exemplary bounding box candidates <NUM> and <NUM>.

<FIG> depicts applying an example of a heuristic rule to select an optimal bounding box. An optimal bounding box candidate may be first selected using non-max suppression techniques based on Intersection over Union (IoU) approaches combined with confidence thresholds. In image <NUM>, this results in the choice of bounding box <NUM> as the optimal bounding box candidate. A subsequent heuristic rule may be applied to the optimal bounding box candidate involving a grid snapping approach in which an imaginary grid is superimposed on the image, and an optimal bounding box is chosen by snapping the optimal bounding box candidate to the superimposed grid. Thus, grid <NUM> may be imposed on the image <NUM>, and optimal bounding box <NUM> may be chosen as the insertion space for media content item by the grid snapping process, since bounding box <NUM> may be the closest bounding box from bounding box <NUM> when snapped to the grid <NUM>.

<FIG> depicts applying another example of a heuristic rule to select an optimal bounding box. Once an optimal bounding box candidate is chosen (e.g., using non-max suppression techniques based on Intersection over Union (IoU) approaches, etc.) another heuristic rule applied in some embodiments involves a pyramidal column grouping approach. In this approach, the image may be divided into multiple columns with the columns themselves grouped in a pyramid style. For example, the image may be divided into twelve columns, with four groups of three columns each, three groups of four columns, two groups of six columns each and a final group of all twelve columns. The group of columns that exhibits the best (i.e., within a predefined minimum threshold) overlap with the optimal bounding box candidate is chosen as the optimal bounding box. Thus, in <FIG>, the image <NUM> is divided into columns <NUM>. The grouping <NUM> of six columns is subsequently chosen as the closest overlapping group to the optimal candidate bounding box <NUM>. The insertion space for media content item is selected to be the space <NUM>.

<FIG> illustrates another image <NUM> with two exemplary corresponding bounding boxes (bounding box <NUM> and bounding box <NUM>). <FIG> illustrates that heuristic rules may be applied that define preferred shapes for selecting optimal bounding boxes, and that more than one optimal bounding box may be selected in accordance with some embodiments for inserting media content items. Thus, bounding box <NUM> is a rectangle, while bounding box <NUM> is an ellipse.

<FIG> is a block diagram illustrating components of a heuristics-based media content insertion system <NUM>, in accordance with some embodiments. The heuristics-based media content insertion system <NUM> includes an image receiving module <NUM>, an image processing module <NUM>, an insertion space proposal module <NUM>, a bounding box selection module <NUM>, a media content selection module <NUM>, a media content overlay module <NUM>, and a data store <NUM>. Other embodiments may include more or fewer modules than those shown in <FIG>. Functionality indicated as being performed by a particular module may be performed by other modules than those indicated herein. Furthermore, steps of any processes described herein can be performed in an order different from that illustrated herein.

The image receiving module <NUM> receives an image for overlaying media content items. The image receiving module <NUM> may receive the image over a network or may receive instructions for retrieving the image from an image store. The image may be any appropriately formatted image. In some embodiments, the image may be a frame of a video content item that is also received by the image receiving module <NUM>. In some embodiments, the image receiving module <NUM> may receive a video content item and extract image frames from the received video content item for further processing. In some embodiments, the image receiving module <NUM> may receive an identifier for a specific starting image frame (e.g., a time value associated with the specific frame) such that display of the video content from that frame onwards requires insertion of media content. In these embodiments, the specific starting image frame as well as image frames in the video content subsequent to the specific image frame may be processed by the heuristics-based media content insertion system <NUM>. In some embodiments, the image receiving module <NUM> may receive an identifier for a specific ending image frame (e.g., an ending time value, a starting time value and a time interval, etc.). In these embodiments, the image frames extracted and stored for processing by the heuristics-based media content insertion system <NUM> may involve the frames that occur from the starting image frame up to and including the ending image frame. For example, the starting image frame may be a first frame in a set of consecutive frames to be played back by a video player and it may be desirable to insert a media content item (e.g., an advertisement) into at least some of those frames. The image receiving module <NUM> may pass the one or more images to the image processing module <NUM> or may store the received image in the data store <NUM> for retrieval by other modules of the heuristics-based media content insertion system <NUM>.

The image processing module <NUM> processes the image using one or more image processing techniques. The image processing module <NUM> receives an image from the image receiving module <NUM>. In some embodiments, the image processing module <NUM> may retrieve the image from the data store <NUM>. In an embodiment, the image may be a color image. In this case, the image processing module <NUM> may first obtain a corresponding greyscale image. In some embodiments, the greyscale image <NUM> may be blurred (e.g., by using a low pass filter) to remove noise in the image before further processing occurs. The greyscale image may subsequently filtered using a gradient filter. The gradient filtering is performed to perform edge detection in the image, i.e., identify pixel locations where the image brightness changes by at least some predefined threshold. An example of a gradient filter is the Sobel gradient filter. In some embodiments, the greyscale image may be filtered by the image processing module <NUM> using an edge detection filter instead of using a gradient filter. Any suitable edge detection filter, such as the ShenCasten filter, the Sobel filter, or the Canny edge detector, etc., may be used by the image processing module <NUM>. The edge-detected image is then processed by the image processing module <NUM> using an image saliency filter that performs local image segmentation.

Some embodiments of the image processing module <NUM> may use supplemental machine learning techniques to identify text, people and faces in the greyscale or RGB image - e.g., text in the image may be located using a single-shot text detector (ssd) network that identifies locations of words while people and faces in the image may be located using object classifiers. The identified text, people, and face regions in the image may then be blocked out as occupied spaces in the image.

The image processing module <NUM> performs adaptive binarization on the saliency filtered image uses the adaptive binarized image along with the original greyscale image to perform foreground/background estimation and generate a foreground/background estimated image. Visual entropy filtering may be performed by the image processing module <NUM> to generate a binary matrix image. This binary matrix image may be subsequently downsampled. The image processing module <NUM> may retrieve predefined parameter values for performing the various image processing techniques described herein from the data store <NUM>. In some embodiments, the image processing module <NUM> may store the downsampled binary matrix image in the data store. In some embodiments, the image processing module <NUM> may send the downsampled binary matrix to the insertion space proposal module <NUM>.

The insertion space proposal module <NUM> generates one or more proposed spaces in the image for media content item overlay. The insertion space proposal module <NUM> receives the downsampled binary matrix image generated by the image processing module <NUM>. The insertion space proposal module <NUM> may retrieve the downsampled binary matrix image from the data store or it may receive the downsampled binary matrix image from the image processing module <NUM>. The module <NUM> may also retrieve predefined minimum threshold parameters (e.g., width of rectangle, height of rectangle, and area of rectangle, etc.) from the data store <NUM>.

The binary matrix image has values of <NUM>'s and <NUM>'s (representing black and white values respectively) for each pixel. Thus, unoccupied spaces in the binary matrix image or the downsampled binary matrix image may be viewed, without limitation, as consisting of spaces with pixel values of '<NUM>'. The insertion space proposal module <NUM> determines rectangular spaces that satisfy the predefined minimum threshold parameters and that fit into the unoccupied spaces in the downsampled binary matrix image received from the image processing module <NUM>. In determining the rectangular spaces, the insertion space proposal module <NUM> may use a suitable variant of a "max rectangle in binary matrix" technique. The "max rectangle in binary matrix" technique determines possible rectangles in the downsampled binary matrix that reside in the unoccupied spaces and that satisfy the predefined minimum threshold parameters. In some embodiments, all determined rectangular spaces (i.e., rectangles) may be placed in a list that is sorted by area. In some embodiments, the insertion space proposal module <NUM> may store the sorted list of proposed rectangles in the data store <NUM>. In some embodiments, the insertion space proposal module <NUM> may provide the sorted list of the rectangular spaces as the one or more proposed spaces for media content item overlay in the received image to the bounding box selection module <NUM>. In some embodiments, the rectangles in the sorted list of rectangles may be each associated with a confidence measure. An exemplary confidence measure associated with a rectangle in the sorted list may be based on the occupied space residing within the rectangle when superimposed on the binary matrix image that is generated prior to being downsampled.

The bounding box selection module <NUM> applies one or more heuristic rules to automatically select bounding boxes in the image for media content item overlay from the proposed one or more spaces. The bounding box selection module <NUM> receives a set of proposed rectangles and associated confidence measures from the insertion space proposal module <NUM>. The set of proposed rectangles may be in the form of the sorted list of rectangles generated by the insertion space proposal module <NUM>. A bounding box refers to an area on an image that is selected as an insertion space for placing media content items. In an embodiment, multiple bounding boxes may be generated based on the number of media content items that may need to be inserted in the image. The information regarding the number of media content items that may need to be inserted may be retrieved from the data store by the bounding box selection module <NUM>. In embodiments described herein, the bounding box may be any shape, including but not limited to, a rectangle, a square, a circle, a polygon, and any other suitable shape. The bounding box may be of an irregular dimension and/or combine multiple shapes. For example, a bounding box may be a square (e.g., represented by a coordinate and one offset), a circle (e.g., represented by a coordinate being the center of the circle and one dimension representing a radius or a diameter of the circle), an ellipse (e.g., represented by two radii and a coordinate at the center). In some embodiments, a bounding box may be described by a vector of points representing coordinates on the image. The vectors and connections of those vectors may represent a bounding box.

The bounding box selection module <NUM> applies heuristic rules to the sorted list of rectangles to select a target number of bounding boxes for media content item overlay from the proposed rectangles in the sorted list. The target number of bounding boxes may be based on a target number of media content items that need to be overlaid on the image. The heuristic rules applied to select the target number of bounding boxes for media content item insertion in the image may include heuristic rules that are based on core design principles. Such heuristic rules may include rules enforcing similarity between rectangles, enforcing proximity between rectangles, enforcing closure for a combination of rectangles to generate a combined irregular polygonal space, etc. The heuristic rules applied by the bounding box selection module <NUM> may have predefined priorities associated with them that define an order in which the heuristic rules may be applied. The bounding box selection module <NUM> may retrieve the heuristic rules as well as associated predefined threshold parameters for applying the heuristic rules from the data store <NUM>.

In some embodiments, the bounding box selection module <NUM> may prune the sorted list generated by the insertion space proposal module <NUM> by selecting a single rectangle from each subset of rectangles in the list that meet a predefined threshold criteria for similarity (i.e., retain only dissimilar rectangles). The predefined similarity threshold parameters may be combinations of parameters such as location, area, width, and height for the proposed rectangles (for example, a heuristic rule for similarity may be that "two proposed rectangles are similar if the centroids of the two rectangles are located on the image at less than a predefined threshold distance from each other, and their aspect ratios differ by less than a predefined threshold value," a heuristic rule for pruning may be "if a subset of rectangles are labelled as similar, replace all of them by a single rectangle that is a weighted combination of individual rectangles in the subset," etc.). In some embodiments, the bounding box selection module <NUM> may use a variant of a "non-max suppression technique" based on an iterative approach using intersection over union (IoU) ratios in association with the confidence measures for pairs of proposed rectangles to select one or more rectangles.

In some embodiments, a heuristic rule may be applied to modify the selected rectangles to a minimal extent needed to ensure that the bounding boxes are the resulting rectangles that are affixed to lie (i.e., snapped) on an grid that is superimposed on the image. In some embodiments, another heuristic rule may involve a pyramidal column grouping approach. In this approach, the image may be divided into multiple adjacent columns with the adjacent columns themselves grouped in a pyramid style. For example, the image may be divided into twelve columns, with four groupings of three adjacent columns each, three groupings of four adjacent columns, two groupings of six adjacent columns each and a final grouping of all twelve columns. A heuristic rule may be applied to determine, for each of the selected rectangles, a grouping of adjacent columns from among the groupings of columns that overlaps best with the selected rectangle. Thus, a generated bounding box may be selected for media content overlay in the image based on determining a grouping of adjacent columns from multiple groupings of columns that are superimposed on the image such that the determined grouping of columns satisfies a predefined minimum threshold of overlap with a selected rectangle.

In some embodiments, the heuristic rules may specify predefined target properties for the generated bounding boxes, such as choosing a predefined number of bounding boxes, a target area, a target shape, target aspect ratios, and target location of a bounding box. For example, a heuristic rule may specify a preference for a media content overlay that takes the form of "a single banner that is overlaid centrally on the top of the image. " In some embodiments, if there is additional information that provides estimated depth values to pixel locations in the image, these depth estimates may also be used to select bounding boxes.

In some embodiments, the bounding box selection module <NUM> may generate the bounding box as a weighted combination of the selected one or more rectangles, the weights based on confidence measure associated with each of the selected rectangles. A bounding box may be described by parameters describing a location of the bounding box in the received image and parameters describing a layout of the bounding box in the received image. For example, a bounding box may be described by a rectangle including one or more coordinates on the image and one or more dimensions extending from the one or more coordinates. The coordinates may represent a particular pixel on the image. In some embodiments, a bounding box may be represented by a coordinate that is counted in terms of a number of pixels horizontally and vertically from the upper left-hand comer of the image and two dimensions. That coordinate may represent the upper left-hand corner of the bounding box. The two dimensions may include a vertical offset and a horizontal offset extending from the coordinate. Based on the coordinate and the offsets, a rectangular bounding box having an area based on the offsets and the location based on the coordinate may be generated. Other descriptions of the bounding box are possible in embodiments described herein, e.g., an ellipse that is a closest circumscribing ellipse to the weighted combination of the selected rectangles, etc. In some embodiments, the insertion space selection module <NUM> may store the description of a bounding box in the data store <NUM>. In some embodiments, the bounding box selection module <NUM> may send the description of the bounding box to the media content overlay module <NUM>.

The media content selection module <NUM> may receive the description of one or more bounding boxes. In some embodiments, the media content selection module <NUM> may retrieve the description of the one or more bounding boxes from the data store <NUM>. The media content selection module <NUM> may extract dimensions from the descriptions associated with each bounding box and search for media content items in the data store <NUM> having dimensions that best match the extracted dimensions. The media content selection module <NUM> may select media content items from multiple media content items based on the dimensions of the content item matching the dimensions of a bounding box. Media content selection module <NUM> may retrieve the selected one or more media content items from the data store <NUM>. In some embodiments, in addition to each selected media content item, the media content selection module <NUM> may extract metadata associated with each media content item. The metadata may include the dimensions of the corresponding media content item as well as other information (e.g., name, type (image, video, etc.)), orientation, and/or other suitable information. The media content selection module <NUM> may send the selected media content item and associated metadata to the media content overlay module <NUM>. In some embodiments, the media content selection module <NUM> may alternatively send one or more identifiers of the selected media content items to the media content overlay module <NUM>. Each identifier may be part of a data structure that also includes dimensions for each media content item. Other information may be included in the data structure (e.g., name, type (image, video, etc.)). This may be advantageous because media content items may be large in size and may take a long time to transfer in comparison to just transferring an identifier.

The media content overlay module <NUM> receives descriptions of one or more bounding boxes. In some embodiments, the descriptions may be received from the bounding box selection module <NUM>. In some embodiments, the media content overlay module <NUM> may retrieve the description from the data store <NUM>. The media content overlay module <NUM> may receive the media content items to be displayed in a bounding box from the media content item selection module <NUM>. In some embodiments, the media content overlay module <NUM> may retrieve the media content items from the data store <NUM>.

As noted previously, in some embodiments, the image may be an image frame of a video content item. The media content overlay module <NUM> may store location and layout information about the determined bounding boxes for one or more of the preceding frames, and apply heuristic rules to modify any of: the dimensions, layout, or location of the determined bounding boxes received from the bounding box selection module <NUM> to ensure that there is a smooth transition of the overlaid media content in each of the video frame images being displayed subsequent to the display of the received image when displaying the video content item.

The data store <NUM> receives and stores data for use and easy access by modules of the heuristics-based media content insertion system <NUM>. The data store <NUM> may store one or more images received from the image receiving module <NUM>. The data store <NUM> may store the binary matrix as well as the downsampled binary matrix generated by the image processing module <NUM>. The data store <NUM> stores heuristic rules and threshold parameter values for use by processes that are executed by modules of the media content insertion system <NUM>, for e.g., target bounding box description parameters, number of target bounding boxes, target area, location, and shape parameters, etc. The data store <NUM> may store intermediate data for use by the various modules of the heuristics-based media content insertion system <NUM>, such as a proposed list of rectangles that may be used for selecting the actual insertion spaces. In some embodiments, the data store <NUM> may also be used to store media content items and associated metadata parameter values. The data store <NUM> is a memory, such as a read only memory (ROM), dynamic random-access memory (DRAM), static random-access memory (SRAM), or some combination thereof.

<FIG> is a flowchart illustrating an overall process for detecting space suitable for overlaying media content onto an image, as performed by the heuristics-based media content insertion system <NUM>, according to one embodiment. Various embodiments can perform the steps of <FIG> in different orders than those indicated herein. Moreover, other embodiments can include different and/or additional steps than the ones described herein.

The heuristics-based media content insertion system <NUM> receives <NUM> an image for media content item overlay. For example, heuristics-based media content insertion system <NUM> may receive <NUM> the image over a network or may receive instructions for retrieving the image from an image store. The heuristics-based media content insertion system <NUM> may receive <NUM> a video content item and may extract one or more image frames from the video content item.

The heuristics-based media content insertion system <NUM> processes <NUM> the received image using one or more image processing techniques to automatically generate one or more proposed spaces in the image for media content item overlay. The image processing techniques performed by the heuristics-based media content insertion system <NUM> include any combination of generating a greyscale images, blurring the image, performing gradient filtering or edge detection filtering, saliency filtering, using supplemental machine learning methods such as ssd networks and classifiers for text, people and face detection, foreground/background estimation, visual entropy filtering, generating a binary matrix, and downsampling, etc. The heuristics-based media content insertion system <NUM> determines rectangular spaces that satisfy predefined minimum threshold parameters in a binary matrix of the image, blocks out text and faces in the rectangular spaces in the generated binary matrix. Subsequently, the heuristics-based system <NUM> generates a sorted list of the rectangular spaces, wherein the sorting is performed based on a predefined parameter (e.g., area), and provides the rectangular spaces in the generated sorted list as one or more proposed spaces for media content item overlay in the image.

The heuristics-based media content insertion system <NUM> applies <NUM> heuristic rules to automatically select one or more bounding boxes from the one or more proposed spaces in the image for media content item overlay. The heuristic rules may be prioritized for application by the heuristics-based media content insertion system <NUM>. The heuristic rules may involve, for example, (i) generating a bounding box based on modifying one or more proposed spaces to lie on a grid that is superimposed on the image, (ii) dividing the image into a number of adjacent columns, grouping the columns in a pyramidal manner, and generating a bounding box generated by selecting a group of adjacent columns that overlaps with a proposed space by at least a predefined threshold value (e.g., at least <NUM>% of the selected rectangle overlaps with the selected group of adjacent columns), (iii) generating a bounding box that is a weighted combination of the one or more proposed spaces, (iv) generating a bounding box based on predefined target properties using the one or more proposed spaces, with predefined target properties including, e.g., number of bounding boxes, area, shape, aspect ratio, and location of a bounding box, etc., among other heuristic rules.

The heuristics-based media content insertion system <NUM> selects <NUM> media content items for overlay in the selected corresponding one or more bounding boxes in the image. In some embodiments, heuristics-based media content insertion system <NUM> may receive the description of one or more bounding boxes, and select media content items from multiple media content items based on the dimensions of the media content item matching the dimensions of a received description of a bounding box.

The heuristics-based media content insertion system <NUM> causes <NUM> a display of the image and the selected one or more media content items overlaid within the corresponding one or more bounding boxes in the image. For example, the heuristics-based media content insertion system <NUM> may transmit the overlaid candidate image to a client device for display. In some embodiments, the media content insertion system <NUM> may use a visual interface to cause the display of the overlaid candidate image.

Each of media content storage system <NUM> and neural networks-based media content insertion system <NUM> depicted in <FIG>, and the heuristics-based media content insertion system <NUM> depicted in <FIG> may include one or more components described in <FIG> is a block diagram illustrating components of an example machine able to read instructions from a machine-readable medium and execute them in a processor (or controller). Specifically, <FIG> shows a diagrammatic representation of a machine in the example form of a computer system <NUM> within which program code (e.g., software) for causing the machine to perform any one or more of the methodologies discussed herein may be executed. The program code may be comprised of instructions <NUM> executable by one or more processors <NUM>. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions <NUM> (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute instructions <NUM> to perform any one or more of the methodologies discussed herein.

The example computer system <NUM> includes a processor <NUM> (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these), a main memory <NUM>, and a static memory <NUM>, which are configured to communicate with each other via a bus <NUM>. The computer system <NUM> may further include visual display interface <NUM>. The visual interface may include a software driver that enables displaying user interfaces on a screen (or display). The visual interface may display user interfaces directly (e.g., on the screen) or indirectly on a surface, window, or the like (e.g., via a visual projection unit). For ease of discussion the visual interface may be described as a screen. The visual interface <NUM> may include or may interface with a touch enabled screen. The computer system <NUM> may also include alphanumeric input device <NUM> (e.g., a keyboard or touch screen keyboard), a cursor control device <NUM> (e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a storage unit <NUM>, a signal generation device <NUM> (e.g., a speaker), and a network interface device <NUM>, which also are configured to communicate via the bus <NUM>.

The storage unit <NUM> includes a machine-readable medium <NUM> on which is stored instructions <NUM> (e.g., software) embodying any one or more of the methodologies or functions described herein. The instructions <NUM> (e.g., software) may also reside, completely or at least partially, within the main memory <NUM> or within the processor <NUM> (e.g., within a processor's cache memory) during execution thereof by the computer system <NUM>, the main memory <NUM> and the processor <NUM> also constituting machine-readable media. The instructions <NUM> (e.g., software) may be transmitted or received over a network <NUM> via the network interface device <NUM>.

While machine-readable medium <NUM> is shown in an example embodiment to be a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store instructions (e.g., instructions <NUM>). The term "machine-readable medium" shall also be taken to include any medium that is capable of storing instructions (e.g., instructions <NUM>) for execution by the machine and that cause the machine to perform any one or more of the methodologies disclosed herein. The term "machine-readable medium" includes, but not be limited to, data repositories in the form of solid-state memories, optical media, and magnetic media.

The components of <FIG> may be used in a system or process for detecting space suitable for overlaying media content onto an image using a neural networks-based approach and a heuristics-based approach among others.

Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. Modules may constitute either software modules (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware modules. A hardware module is tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

Accordingly, the term "hardware module" should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. As used herein, "hardware-implemented module" refers to a hardware module. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where the hardware modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different hardware modules at different times. Software may accordingly configure a processor, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time.

Where multiple of such hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the hardware modules.

Similarly, the methods described herein may be at least partially processor-implemented. For example, at least some of the operations of a method may be performed by one or processors or processor-implemented hardware modules. In some example embodiments, the processor or processors may be located in a single location (e.g., within a home environment, an office environment or as a server farm), while in other embodiments the processors may be distributed across a number of locations.

The one or more processors may also operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs).

For example, some embodiments may be described using the term "connected" to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact.

Claim 1:
A computer implemented method for detecting space suitable for overlaying media content onto an image (<NUM>, <NUM>), the method comprising:
training a neural network using a plurality of images (<NUM>, <NUM>), each of the plurality of images (<NUM>, <NUM>) corresponding to one or more vectors, each vector comprising: a coordinate representing a point associated with a bounding box (<NUM>, <NUM>) in a corresponding image, and a dimension of the bounding box (<NUM>, <NUM>), wherein each bounding box (<NUM>, <NUM>) is suitable for media content overlay;
receiving by a media content insertion system (<NUM>, <NUM>) a candidate image for a media content overlay;
inputting the candidate image into the neural network;
outputting, by the neural network, one or more bounding boxes as one or more data structures (<NUM>), each of which comprises at least a coordinate representing a point associated with the bounding box (<NUM>, <NUM>), a dimension of the bounding box (<NUM>, <NUM>), and a probability that the bounding box (<NUM>, <NUM>) is located in an area suitable for the media content overlay;
for at least one of the one or more bounding boxes, determining whether the probability meets a threshold probability;
responsive to determining that the probability meets the threshold probability, determining that the bounding box (<NUM>, <NUM>) is a candidate bounding box (<NUM>) of the candidate image;
receiving, from the neural network, the data structure (<NUM>) corresponding to the candidate bounding box (<NUM>) of the candidate image;
transmitting a request for a media content item to be displayed in the candidate bounding box (<NUM>), the request comprising the dimension of the candidate bounding box (<NUM>);
receiving the media content item in response to the request; and
causing a display of the candidate image and the media content item overlaid on top of the candidate image within the candidate bounding box (<NUM>).