Patent Description:
Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a transformation to a received input to generate an output. <NPL>, describes, in accordance with its abstract, a GAN-based framework for learning from complex, high-dimensional incomplete data. The proposed framework learns a complete data generator along with a mask generator that models the missing data distribution.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that performs image extension. The invention to which this European patent relates is defined in the appended independent claims <NUM>, <NUM> and <NUM>. Preferred embodiments of the invention to which this European patent relates are set out in the appended dependent claims.

According to a first aspect there is provided a method performed by one or more data processing apparatus, the method comprising: providing an input that comprises the provided image to a generative neural network having a plurality of generative neural network parameters, wherein: the generative neural network processes the input in accordance with trained values of the plurality of generative neural network parameters to generate an extended image; the extended image has (i) more rows, more columns, or both than the provided image, and (ii) is predicted to be a realistic extension of the provided image; and the generative neural network has been trained using an adversarial loss objective function.

In this way, the additional rows and/or columns provide an extension beyond one or more of the provided image's original borders and provide a predicted realistic extension of the provided image, such as, for example preserving high-level semantic characteristics and low-level structures and textures of the provided image.

The method may comprise receiving a request for an image to be presented in a block; determining that the provided image is responsive to the request; and after generating the extended image, providing the extended image in response to the request.

The method may comprise determining that the provided image qualifies for image extension based on a size of the block differing from a size of the provided image and in response providing the input that comprises the provided image to the generative neural network.

The size of the block may specify an aspect ratio of the block.

The request may specify additional elements to be presented with the image in the block, and providing the extended image in response to the request may include overlaying the additional elements on an extended portion of the extended image.

The block may be presented alongside search results or on a third-party webpage.

The generative neural network may comprise a plurality of convolutional neural network layers.

The generative neural network may comprise a plurality of skip connections.

The generative neural network may comprise a plurality of instance normalization layers.

The input to the generative neural network may comprise: a baseline image having a same number of rows and columns as the extended image, wherein the baseline image comprises: (i) a first portion that corresponds to the provided image, and (ii) a second portion having default pixel values, and a mask image having the same number of rows and columns as the extended image, wherein the mask image identifies the first portion and the second portion of the baseline image.

The pixels in the mask image corresponding to the first portion of the baseline image may each have a first pixel value, and the pixels in the mask image corresponding to the second portion of the baseline image may each have a second pixel value that is different than the first pixel value.

The generative neural network may be jointly trained with a discriminative neural network having a plurality of discriminative neural network parameters that is configured to process a given image to generate a discriminative output characterizing a likelihood that the given image was generated using the generative neural network.

Training the generative neural network using the adversarial loss objective function may comprise processing a training input that comprises a training image using the generative neural network and in accordance with current values of the generative neural network parameters to generate a training extended image that extends the training image; generating a discriminative neural network input based on the training extended image; processing the discriminative neural network input using the discriminative neural network and in accordance with current values of the discriminative neural network parameters to generate a discriminative output characterizing a likelihood that the discriminative neural network input was generated using the generative neural network; and adjusting the current values of the generative neural network parameters based on the adversarial loss objective function, wherein the adversarial loss objective function depends on the discriminative output characterizing the likelihood that the discriminative neural network input was generated using the generative neural network.

Generating a discriminative neural network input based on the training extended image may include overwriting a portion of the training extended image corresponding to the training image with the training image.

The method may include adjusting the current values of the generative neural network parameters based on a reconstruction loss objective function that characterizes a similarity of the training extended image to a target image, wherein the training image is a cropped representation of the target image.

The discriminative neural network may be conditioned on a semantic feature representation of a target image, wherein the training image is a cropped representation of the target image.

The semantic feature representation of the target image may be determined using an intermediate output of a classification neural network by processing the target image.

The discriminative output may be based on: (i) an output of a final layer of the discriminative neural network, and (ii) a similarity measure between an intermediate output of the discriminative neural network and the semantic feature representation of the target image.

According to another aspect there is provided a system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the operations of the respective method of the first aspect.

According to another aspect there is provided one or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of the first aspect.

Optional features of one aspect may be combined with another where appropriate.

The image extension system described in this specification generates extended images (e.g., that extend input images beyond their original borders) using a generative neural network. The generative neural network is jointly trained with a "discriminative" neural network that is configured to process an input image to generate an output defining a prediction for whether the input image is (i) a "real" image, or (ii) an extended image generated by the generative neural network. To facilitate the joint training of the generative neural network and the discriminative neural network, the system conditions the discriminative neural network on semantic features corresponding to input images processed by the discriminative neural network. Conditioning the discriminative neural network in this manner can enable the system to be trained to achieve an acceptable performance level over fewer training iterations, thereby enabling the system to consume fewer computational resources (e.g., memory and computing power) during training than some conventional systems.

Some conventional in-painting systems "fill in" a portion of an image that is surrounded in all directions by original image data. In contrast, the image extension system described in this specification extends an image beyond its original borders to generate an extended image that extends the high-level semantic characteristics and low-level image structures and textures of the image. Directly applying a conventional in-painting system to perform image extension tasks can, in some cases, result in lower quality extended images than those generated by the image extension system described in this specification. For example, conventional in-painting systems may generate extended images with blurry or repetitive pixels and inconsistent semantics, whereas the extended images created by the presently described image extension system will not have the same blurry, repetitive pixel, or inconsistent semantic characteristics.

The image extension system described in this specification generates extended images that can be used for any of a variety of applications, e.g., virtual reality applications, computational photography applications, and digital component distribution applications. In digital component distribution applications, a digital component distribution system transmits digital components for presentation in blocks, e.g., alongside search results or on third-party webpages. In some cases, an image included in a digital component may have a size that prevents the image from filling a defined block, e.g., because the aspect ratio of the image and the block are different. In this situation, the image extension system described in this specification can be used for resizing the image to enable the image to fill the block, thereby enabling more efficient use of resources (in particular, more efficient use of the space available in blocks) and preventing blank spots and/or distorted image presentations in a user interface. In other words, the system described in this application enables not only the "fill in" of an image, but the extension of an image beyond its borders, which enables an image to be modified to fit multiple different sized blocks even when only a single image size is available. This reduces the number of images that need to be stored since a single image can be modified to fit in various sized blocks while retaining features of the original image. The image extension system can obviate the need to distort an image by stretching it or compressing it along certain dimensions to change the aspect ratio of the image to cause the image to fill a block. Moreover, in digital component distribution applications, some digital component providers may require that images included in digital components be presented without being distorted (e.g., by being stretched or compressed along certain dimensions).

This specification describes technology for processing an image to generate an "extended" image that realistically extends the image beyond its original borders, e.g., that extends the image beyond its original borders while preserving high-level semantic characteristics and low-level structures and textures of the image. Such processing may be used in image enhancement. For example, if an image has dimensions which do not match a block in which the image is to be placed, the image dimensions may expanded to match the block, while preserving characteristics of content within the original image. In another example, if an image is damaged, such as a portion of the image is missing, aspects of the present disclosure may be used to automatically generate the missing portion. For example, an improperly captured or a captured version of a damaged image that is incomplete can be regenerated with the techniques discussed herein.

To extend an image, the image extension system described in this specification processes the image using a generative neural network that is trained using an adversarial loss objective function to generate a corresponding extended image. The adversarial loss objective function evaluates an extended image generated using the generative neural network based on an output generated by a discriminative neural network by processing the extended image. The discriminative neural network is trained to predict whether an input image is (i) a "real" (i.e., natural) image, or (ii) an extended image generated using the generative neural network. Jointly training the generative neural network and the discriminative neural network encourages the image extension system to generate extended images that are difficult to distinguish from real images.

The image extension system can be used for any of a variety of applications, e.g., resizing images included in digital components that are to be presented in defined blocks of various dimensions to enable the images to fill the blocks without being distorted.

As used throughout this document the phrase digital component refers to a discrete unit of digital content or digital information that can include one or more of, e.g., images, video clips, audio clips, multimedia clips, text segments, or uniform resource locators (URLs). A digital component can be electronically stored in a physical memory device as a single file or in a collection of files, and digital components can take the form of video files, audio files, multimedia files, image files, or text files and include streaming video, streaming audio, social network posts, blog posts, and/or advertising information, such that an advertisement is a type of digital component. Generally, a digital component is defined by (or provided by) a single provider or source (e.g., an advertiser, publisher, or other content provider), but a digital component provided from one source could be enhanced with data from another source (e.g., weather information, real time event information, or other information obtained from another source).

These features and other features are described in more detail below.

<FIG> shows an example image extension system <NUM>. The image extension system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The image extension system <NUM> is configured to process an image <NUM> to generate an extended image <NUM> that realistically extends the image <NUM> beyond its original borders. More specifically, a two-dimensional (2D) array of pixels representing the extended image <NUM> generally has more rows, more columns, or both, than the image <NUM>. In the example depicted in <FIG>, the extended image <NUM> has one or more additional columns <NUM> on its right-hand side compared to the original image <NUM>. In a particular example, the image <NUM> may be <NUM> × <NUM> dimensional, while the extended image <NUM> may be <NUM> × <NUM> dimensional. In another particular example, the image <NUM> may be <NUM> × <NUM> dimensional, while the extended image may be <NUM> × <NUM> dimensional. The units of the dimension may be any suitable units, such as a number of pixels. An example of an extended image generated using the system <NUM> is shown in <FIG>.

In some cases, prior to generating an extended image <NUM>, the image extension system <NUM> may uniformly shrink or expand the original image <NUM> (i.e., while preserving its aspect ratio and thereby avoiding distortions). In a particular example, to generate an output image with dimensionality <NUM> × <NUM> from an input image with dimensionality <NUM> × <NUM>, the image extension system may first uniformly shrink the input image to dimensionality <NUM> × <NUM>, and then generate an extended image with dimensionality <NUM> × <NUM> from the stirutiken image of dimensionality <NUM> × <NUM>.

The image <NUM> (and the extended image <NUM>) can be represented in any appropriate format, e.g., as a 2D array of pixels that are each associated with a corresponding grayscale intensity or with a color vector representing the color of the pixel in an appropriate color space. The color space may be, e.g., a red-green-blue (RGB) color space or a CIELAB color space.

To generate the extended image <NUM>, the system <NUM> determines a generative network input <NUM> from the image <NUM> and processes the generative network input <NUM> using a generative neural network <NUM>. A few examples of generative network inputs follow.

In one example, the generative network input <NUM> includes a baseline image <NUM> and a mask image <NUM> that both have the same dimensionality (i.e., number of rows and columns) as the extended image that is to be generated by the generative neural network <NUM>. The baseline image <NUM> includes: (i) a first portion <NUM>-A that corresponds to the image <NUM>, and (ii) a second portion <NUM>-B having default pixel values (e.g., black or white pixel values). That is, the baseline image <NUM> includes the image <NUM>, and can be understood to extend the image <NUM> by one or more rows or columns of default pixel values. The mask image <NUM> identifies which portions of the baseline image correspond to: (i) the image <NUM>, and (ii) the default pixel values. For example, the pixels in the mask image <NUM> corresponding to the proper subset of the baseline image <NUM> that matches the image <NUM> may have a first value (e.g., the value <NUM>), and the remaining pixels in the mask image <NUM> may have a second value (e.g., the value <NUM>). In this example, the baseline image <NUM> and the mask image <NUM> may be spatially concatenated channel-wise to form the generative network input <NUM>.

In another example, the generative network input <NUM> may include only the baseline image <NUM>, i.e., without the mask image <NUM>.

In another example, the generative network input <NUM> may directly correspond to the image <NUM>, i.e., without including either the baseline image <NUM> or the mask image <NUM>.

The generative neural network <NUM> is configured to process the generative network input <NUM> in accordance with trained values of a set of generative neural network parameters to generate the extended image <NUM>. The generative neural network <NUM> generally has a convolutional neural network architecture, that is, a neural network architecture that includes one or more convolutional neural network layers (and optionally, other appropriate sorts of neural network layers).

The adversarial training system <NUM> is configured to determine trained values of the generative neural network parameters that enable the generative neural network <NUM> to generate realistic extended images <NUM>. The adversarial training system <NUM> jointly trains the generative neural network <NUM> along with a discriminative neural network using an adversarial loss objective function, as will be described in more detail with reference to <FIG>.

In some cases, the system <NUM> can create "panoramic" extended images by repeatedly extending an original image. For example, the system <NUM> can extend an original image to generate a first extended image, then extend a cropped portion of the first extended image to generate a second extended image, and so on. Finally, the system can concatenate the extended images to generate a panoramic image that substantially extends the original image, e.g., by a factor of <NUM> × , <NUM> ×, <NUM> ×, or any other appropriate factor.

The extended images <NUM> generated by the system <NUM> can be used in any of a variety of applications. A few examples follow.

In one example, in virtual reality applications, it is sometimes necessary to simulate an image that is captured from a different camera orientation than was actually used to capture an original image. The image extension system <NUM> can be used to simulate an image in this manner by filling in content outside of the bounds of the original image.

In another example, in panorama stitching applications (e.g., where multiple images are "stitched" together to generate a single combined image), some conventional techniques require cropping the jagged edges of stitched projections to achieve a rectangular panorama. The image extension system <NUM><NUM> can obviate the need to crop jagged edges by generating extended images that fill in the gaps between the jagged edges instead, thereby enabling more effective panorama stitching.

In another example, the image extension system <NUM> can be used to enable videos that are captured filmed at a certain aspect ratio to be displayed on a screen with a different aspect ratio without resorting to stretching or cropping. In particular, the image extension system <NUM> can be used to process each video frame in a video to generate a corresponding extended video frame, e.g., having the same aspect ratio as the screen where the video is to be displayed.

In another example, the image extension system <NUM> can be used by a digital component distribution system to extend images that are to be displayed in defined blocks alongside search results or on third-party websites to enable the images to fill the blocks without resorting to stretching or cropping. Using the image extension system <NUM> as part of a digital component distribution system is described in more detail with reference to <FIG>.

<FIG> is an example data flow <NUM> illustrating operations that can be performed by an adversarial training system <NUM> for training the generative neural network <NUM> jointly with the discriminative neural network <NUM> using an adversarial loss objective function <NUM>. Generally, the generative neural network <NUM> is trained to generate extended images that realistically extend input images, and the discriminative neural network <NUM> is trained to generate discriminative outputs characterizing the likelihood that input images were generated using the generative neural network <NUM> (i.e., as opposed to being natural images). Training of the generative neural network <NUM> will be described in detail next, followed by a description of training the discriminative neural network <NUM>.

The generative neural network <NUM> is trained over multiple training iterations, and at each training iteration, the current parameter values of the generative neural network parameters are adjusted based on a current batch (i.e., set) of training examples. Each training example includes: (i) a training generative network input, and (ii) a target extended image that should be generated by the generative neural network <NUM> by processing the corresponding training generative network input.

The target extended images <NUM> included in the training examples are generally real (i.e., natural) images. The training generative network input corresponding to a target extended image <NUM> can be represented in any of a variety of ways, as described with reference to <FIG>. In one example, the training generative network input corresponding to a target extended image <NUM> includes a baseline image <NUM> and a mask image <NUM>. The baseline image <NUM> is generated by "masking" a portion of the target extended image <NUM>, e.g., by setting the pixel values in a portion of the target extended image <NUM> to a default value (e.g., the value <NUM>). The mask image <NUM> identifies the portion of the target extended image <NUM> that has been masked in the baseline image. The portion of the target extended image that is included in the training generative network input may be referred to in this document as a "training image".

Generally, the current batch of training examples can include multiple training examples. However, for convenience, the description which follows will reference a particular training example in the current batch.

The generative neural network <NUM> processes the training generative network input included in the training example to generate a corresponding training extended image <NUM>. After generating the training extended image <NUM>, the adversarial training system <NUM> processes the training extended image <NUM> using the discriminative neural network <NUM> to generate a discriminative output characterizing the likelihood that the training extended image was generated by the generative neural network <NUM>. Thus, the discriminative output characterizes whether the training extended image <NUM> generated by the generative neural network <NUM> is a realistic image that plausibly extends the training image. As will be described in more detail below, the adversarial loss objective function <NUM> depends on the discriminative output generated by the discriminative neural network <NUM>.

In some cases, rather than directly processing the training extended image <NUM>, the adversarial training system <NUM> first overwrites the training image onto the corresponding portion of the training extended image <NUM>. That is, adversarial training system <NUM> overwrites the training image onto the portion of the training extended image <NUM> that should match the training image, without modifying the portion of the training extended image <NUM> that extends beyond the borders of the training image. Modifying the training extended image <NUM> in this manner encourages the discriminative output to characterize not only whether the training extended image is inherently a realistic image, but also whether the training extended image is a realistic extension of the original training image.

Prior to the training extended image <NUM> being provided to the discriminative neural network <NUM>, the mask image <NUM> may be spatially concatenated channel-wise to the training extended image <NUM>.

In addition to processing the training extended image <NUM> (or an input based on the training extended image <NUM>), the discriminative neural network <NUM> may be "conditioned" on a semantic feature representation of the target extended image <NUM> of the training example. That is, the discriminative output generated by the discriminative neural network <NUM> for the training extended image <NUM> may depend on the semantic feature representation of the target extended image <NUM>.

The semantic feature representation of the target extended image <NUM> refers to an ordered collection of numerical values, e.g., a vector or matrix of numerical values, which implicitly or explicitly characterizes the contents of the target extended image <NUM>. The adversarial training system <NUM> determines the semantic feature representation of the target extended image <NUM> to be an intermediate output generated by a pre-trained image processing neural network by processing the target extended image <NUM>. An intermediate output of a neural network refers to an output generated by one or more intermediate layers of the neural network, i.e., layers following the input layer but preceding the output layer.

The image processing neural network may have any appropriate neural network architecture (e.g., an InceptionV3 architecture) and may be pre-trained to perform any of a variety of image processing tasks, e.g., classification tasks or regression tasks. In a particular example, the image processing neural network may be pre-trained to perform a classification task by generating an output characterizing respective likelihoods that an input image depicts an object from each of a predetermined number of object classes (e.g., person, vehicle, bicycle, and so on). In another particular example, the image processing neural network may be pre-trained to perform a regression task by generating an output characterizing locations of bounding boxes that are predicted to enclose objects depicted in the input image. The image processing neural network being "pre-trained" refers to the network having been previously trained to perform the image processing task.

The adversarial training system <NUM> uses a normalization engine <NUM> to normalize the semantic representation of the target extended image prior to using it to condition the discriminative neural network <NUM>. For example, the normalization engine <NUM> may determine the normalized semantic representation Cnorm as: <MAT> where C is the un-normalized semantic representation, <MAT> is the expected value of semantic representations of images (e.g., from a set of training images), and | • |<NUM> refers to an L<NUM> norm. Moreover, the adversarial training system <NUM> may process the normalized semantic representation using one or more neural network layers <NUM> (e.g., fully-connected layers) that are jointly trained with the generative and discriminative neural networks prior to using the semantic representation to condition the discriminative neural network.

The discriminative neural network <NUM> is conditioned on the semantic representation of the target extended image in any of a variety of ways. In one example, the adversarial training system <NUM> may provide the semantic representation of the target extended image as an additional input to the discriminative neural network, e.g., to an input layer or an intermediate layer of the discriminative neural network. In another example, the adversarial training system <NUM> may determine the discriminative output based at least in part on a similarity measure between (i) the semantic representation of the target extended image and (ii) an intermediate output of the discriminative neural network, e.g., the discriminative output D may be given by: <MAT> where DO is the scalar output of a final neural network layer of the discriminative neural network, DN is an intermediate output of the discriminative neural network (e.g., generated by intermediate layer <NUM>), S is the semantic representation of the target extended image, and <•,•> refers to the dot product operation.

Conditioning the discriminative neural network <NUM> on the semantic representation may stabilize the joint training of the generative neural network <NUM> and the discriminative neural network <NUM>, and also improve the quality of extended images generated by the trained generative neural network <NUM>. In particular, conditioning the discriminative neural network <NUM> on the semantic representation may enable the discriminative neural network <NUM> to generate a discriminative output that depends on the entire semantic content of the target extended image, including from the masked portion of the target extended image. The semantic representation may provide effective conditioning information (i.e., that enables the discriminative neural network to generate more accurate discriminative outputs) as a result of characterizing the "global" context of the target extended image. In contrast, conditioning the discriminative network directly on the pixel values of the target extended image may be less effective, e.g., because the pixel values are inherently "local" and potentially noisy. Conditioning the discriminative neural network on the semantic representation may also enable the discriminative neural network to be trained more effectively (e.g., over fewer training iterations) by reducing the burden on the discriminative neural network to directly account for semantic content in the output of its final layer. For example, referring to equation (<NUM>), the discriminative output may take the semantic content into account by an inner product between an intermediate output of the discriminative neural network and the semantic representation.

After generating the discriminative output, the adversarial training system <NUM> can adjust the current values of the generative neural network parameters using gradients of an objective function <IMG> that includes the adversarial loss objective function <IMG> <NUM> and a reconstruction objective function <IMG> <NUM>, e.g.,: <MAT> <MAT> <MAT> where λ is a scalar hyper-parameter, x is the target extended image, x̂ is the training extended image generated by the generative neural network, | • |<NUM> is an L<NUM> norm, and D(x̂) is the discriminative output generated by the discriminator neural network by processing the training extended image (or an input based on the training extended image).

The objective functions characterized by equations (<NUM>) - (<NUM>) are provided as illustrative examples of objective functions, but other objective functions are possible. Generally, the reconstruction objective function <IMG> can characterize the similarity between the target extended image x and the training extended image x̂ in any of a variety of ways, and the adversarial loss objective function <IMG> can depend on the discriminative output D(x̂) in any of a variety of ways.

The adversarial training system <NUM> jointly trains the discriminative neural network <NUM> and the generative neural network <NUM>, e.g., by alternating between training the discriminative neural network <NUM> and the generative neural network <NUM> for predetermined numbers of training iterations.

At each training iteration during training of the discriminative neural network, the current parameter values of the discriminative neural network are adjusted based on a current batch of training examples. As before, the current batch of training examples can include multiple training examples, but for convenience, the description which follows will reference a particular training example.

To train the discriminative neural network <NUM>, the adversarial training system <NUM> processes the training generative network input of the training example using the generative neural network <NUM> to generate a corresponding training extended image. The adversarial training system <NUM> processes both the training extended image (or an input based on the training extended image) and the corresponding target extended image using the discriminative neural network to generate corresponding discriminative outputs. The adversarial training system <NUM> then adjusts the current parameter values of the discriminative neural network <NUM> based on a discriminative objective function that depends on the discriminative outputs generated for the training extended image and the target extended image. For example, the discriminative objective function <IMG> may be given by: <MAT> where ReLU(•) is a rectified linear unit function, D(x) is the discriminative output generated by the discriminative neural network for the target extended image, and D(x̂) is the discriminative output generated by the discriminative neural network for the training extended image generated by the generative neural network. The discriminative objective function characterized by equation (<NUM>) is provided for illustrative purposes only, and other discriminative objective functions are possible.

The discriminative neural network <NUM> can have any appropriate neural network architecture. In one example, the discriminative neural network <NUM> has six strided convolutional layers with leaky ReLU activation functions followed by a fully-connected layer.

During training of the generative neural network and the discriminative neural network, the adversarial training system <NUM> can determine gradients of the objective functions using, e.g., backpropagation techniques. The adversarial training system <NUM> can use the gradients of the objective functions to adjust the current parameter values of the generative neural network and the discriminative neural network using any appropriate gradient descent optimization procedure, e.g., Adam or RMSprop.

The adversarial training system <NUM> can continue jointly training the generative neural network <NUM> and the discriminative neural network <NUM> until a training termination criterion is satisfied, e.g., until a predetermined number of training iterations have been performed for each neural network, or until a performance criterion is satisfied.

<FIG> is an illustration of an example of an extended image <NUM> that is generated by the image extension system <NUM> described with reference to <FIG> by processing the original image <NUM>. It can be appreciated that the extended image <NUM> is a realistic extension of the original image <NUM> that preserves high-level semantic characteristics and low-level structures and textures of the original image <NUM>.

<FIG> is a block diagram of an example environment <NUM> in which a digital component distribution system <NUM> uses the image extension system <NUM> while transmitting digital components from a digital component database <NUM> for presentation with electronic documents. As will described in more detail below, the digital component distribution system can transmit digital components in response to requests for digital components to be presented with electronic document at user devices, e.g., in blocks alongside search results or on third party websites.

After identifying a digital component to be transmitted in response to a request, the distribution system may identify that the image included in the digital component qualifies for image extension. In one example, the distribution system may identify that the image qualifies for image extension based on the size (e.g., aspect ratio) of the image differing (e.g., by at least a threshold amount) from the size of the block where the image will be presented. In this example, the digital component request can include data characterizing the size of the block where the digital component will be presented. In another example, the distribution system may identify that the image qualifies for image extension based on a provider of the digital component having specified that other visual elements of the digital component (e.g., text, logos, and interactive elements) should not be overlaid on the image of the digital component.

In response to identifying that the image included in the digital component qualifies for extension, the distribution system can process the image using the image extension system <NUM> to generate an extended image. In one example, the distribution system may generate an extended image having an aspect ratio that matches the aspect ratio of the block where the extended image will be displayed. In another example, the distribution system may generate an extended image, and then overlay the additional visual elements of the digital component (e.g., text, logos, and interactive elements) only on the extended portion of the image (i.e., without modifying the original image).

After extending the image included in the digital component, the distribution system may transmit the digital component with the extended image for presentation in a block at a user device that generated the component request.

The example environment <NUM> and the operations performed by the distribution system are described in more detail next.

The example environment <NUM> includes a network <NUM>, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network <NUM> connects electronic document servers <NUM>, client devices <NUM>, digital component servers <NUM>, and the digital component distribution system <NUM> (also referred to as the "distribution system" <NUM>). The example environment <NUM> may include many different electronic document servers <NUM>, client devices <NUM>, and digital component servers <NUM>.

A client device <NUM> is an electronic device that is capable of requesting and receiving resources over the network <NUM>. Example client devices <NUM> include personal computers, mobile communication devices (e.g., mobile phones), and other devices that can send and receive data over the network <NUM>. A client device <NUM> typically includes a user application, such as a web browser, to facilitate the sending and receiving of data over the network <NUM>, but native applications executed by the client device <NUM> can also facilitate the sending and receiving of data over the network <NUM>.

An electronic document is data that presents a set of content at a client device <NUM>. Examples of electronic documents include webpages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e.g., "apps"), such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents can be provided to client devices <NUM> by electronic document servers <NUM> ("Electronic Doc Servers"). For example, the electronic document servers <NUM> can include servers that host publisher websites. In this example, the client device <NUM> can initiate a request for a given publisher webpage, and the electronic server <NUM> that hosts the given publisher webpage can respond to the request by sending machine executable instructions that initiate presentation of the given webpage at the client device <NUM>.

In another example, the electronic document servers <NUM> can include app servers from which client devices <NUM> can download apps. In this example, the client device <NUM> can download files required to install an app at the client device <NUM>, and then execute the downloaded app locally.

Electronic documents can include a variety of content. For example, an electronic document can include static content (e.g., text or other specified content) that is within the electronic document itself and/or does not change over time. Electronic documents can also include dynamic content that may change over time or on a per-request basis. For example, a publisher of a given electronic document can maintain a data source that is used to populate portions of the electronic document. In this example, the given electronic document can include one or more tags or scripts that cause the client device <NUM> to request content from the data source when the given electronic document is processed (e.g., rendered or executed) by a client device <NUM>. The client device <NUM> integrates the content obtained from the data source into the given electronic document to create a composite electronic document including the content obtained from the data source.

In some situations, a given electronic document can include one or more digital component tags or digital component scripts that reference the digital component distribution system <NUM>. In these situations, the digital component tags or digital component scripts are executed by the client device <NUM> when the given electronic document is processed by the client device <NUM>. Execution of the digital component tags or digital component scripts configures the client device <NUM> to generate a request for one or more digital components <NUM> (referred to as a "component request"), which is transmitted over the network <NUM> to the digital component distribution system <NUM>. For example, a digital component tag or digital component script can enable the client device <NUM> to generate a packetized data request including a header and payload data. The component request <NUM> can include event data specifying features such as a name (or network location) of a server from which the digital component is being requested, a name (or network location) of the requesting device (e.g., the client device <NUM>), and/or information that the digital component distribution system <NUM> can use to select one or more digital components provided in response to the request. The component request <NUM> is transmitted, by the client device <NUM>, over the network <NUM> (e.g., a telecommunications network) to a server of the digital component distribution system <NUM>.

The component request <NUM> can include event data specifying other event features, such as the electronic document being requested and characteristics of locations of the electronic document at which digital component can be presented. For example, event data specifying a reference (e.g., URL) to an electronic document (e.g., webpage) in which the digital component will be presented, available locations of the electronic documents that are available to present digital components, sizes of the available locations, and/or media types that are eligible for presentation in the locations can be provided to the digital component distribution system <NUM>. Similarly, event data specifying keywords associated with the electronic document ("document keywords") or entities (e.g., people, places, or things) that are referenced by the electronic document can also be included in the component request <NUM> (e.g., as payload data) and provided to the digital component distribution system <NUM> to facilitate identification of digital components that are eligible for presentation with the electronic document. The event data can also include a search query that was submitted from the client device <NUM> to obtain a search results page, and/or data specifying search results and/or textual, audible, or other visual content that is included in the search results.

Component requests <NUM> can also include event data related to other information, such as information that a user of the client device has provided, geographic information indicating a state or region from which the component request was submitted, or other information that provides context for the environment in which the digital component will be displayed (e.g., a time of day of the component request, a day of the week of the component request, a type of device at which the digital component will be displayed, such as a mobile device or tablet device). Component requests <NUM> can be transmitted, for example, over a packetized network, and the component requests <NUM> themselves can be formatted as packetized data having a header and payload data. The header can specify a destination of the packet and the payload data can include any of the information discussed above.

The component distribution system <NUM> chooses digital components that will be presented with the given electronic document in response to receiving the component request <NUM> and/or using information included in the component request <NUM>. In some implementations, a digital component is selected (using the techniques described herein) in less than a second to avoid errors that could be caused by delayed selection of the digital component. For example, delays in providing digital components in response to a component request <NUM> can result in page load errors at the client device <NUM> or cause portions of the electronic document to remain unpopulated even after other portions of the electronic document are presented at the client device <NUM>. Also, as the delay in providing the digital component to the client device <NUM> increases, it is more likely that the electronic document will no longer be presented at the client device <NUM> when the digital component is delivered to the client device <NUM>, thereby negatively impacting a user's experience with the electronic document. Further, delays in providing the digital component can result in a failed delivery of the digital component, for example, if the electronic document is no longer presented at the client device <NUM> when the digital component is provided.

In some implementations, the digital component distribution system <NUM> is implemented in a distributed computing system that includes, for example, a server and a set of multiple computing devices <NUM> that are interconnected and identify and distribute digital components in response to requests <NUM>. The set of multiple computing devices <NUM> operate together to identify a set of digital components that are eligible to be presented in the electronic document from a corpus of millions of available digital components (DC1-x). The millions of available digital components can be indexed, for example, in a digital component database <NUM>. Each digital component index entry can reference the corresponding digital component and/or include distribution parameters (DP1-DPx) that contribute to (e.g., condition or limit) the distribution-transmission of the corresponding digital component. For example, the distribution parameters can contribute to the transmission of a digital component by requiring that a component request include at least one criterion that matches (e.g., either exactly or with some pre-specified level of similarity) one of the distribution parameters of the digital component.

In some implementations, the distribution parameters for a particular digital component can include distribution keywords that must be matched (e.g., by electronic documents, document keywords, or terms specified in the component request <NUM>) in order for the digital component to be eligible for presentation. In other words, the distribution parameters are used to trigger distribution (e.g., transmission) of the digital components over the network <NUM>. The distribution parameters can also require that the component request <NUM> include information specifying a particular geographic region (e.g., country or state) and/or information specifying that the component request <NUM> originated at a particular type of client device (e.g., mobile device or tablet device) in order for the digital component to be eligible for presentation.

The distribution parameters can also specify an eligibility value (e.g., ranking score, bid, or some other specified value) that is used for evaluating the eligibility of the digital component for distribution/transmission (e.g., among other available digital components), for example, by the component evaluation process. In some situations, the eligibility value can specify a maximum amount of compensation that a provider of the digital component is willing to submit in response to the transmission of the digital component (e.g., for each instance of specific events attributed to the presentation of the digital component, such as user interaction with the digital component).

The identification of the eligible digital component can be segmented into multiple tasks 417a-417c that are then assigned among computing devices within the set of multiple computing devices <NUM>. For example, different computing devices in the set <NUM> can each analyze a different portion of the digital component database <NUM> to identify various digital components having distribution parameters that match information included in the component request <NUM>. In some implementations, each given computing device in the set <NUM> can analyze a different data dimension (or set of dimensions) and pass (e.g., transmit) results (Res <NUM>-Res <NUM>) 418a-418c of the analysis back to the digital component distribution system <NUM>. For example, the results 418a-418c provided by each of the computing devices in the set <NUM> may identify a subset of digital components that are eligible for distribution in response to the component request and/or a subset of the digital components that have certain distribution parameters. The identification of the subset of digital components can include, for example, comparing the event data to the distribution parameters, and identifying the subset of digital components having distribution parameters that match at least some features of the event data.

The digital component distribution system <NUM> aggregates the results 418a-418c received from the set of multiple computing devices <NUM> and uses information associated with the aggregated results to: (i) select one or more digital components that will be provided in response to the request <NUM>, and (ii) determine transmission requirements for the one or more digital components. For example, the digital component distribution system <NUM> can select a set of winning digital components (one or more digital components) based on the outcome of one or more component evaluation processes. In turn, the digital component distribution system <NUM> can generate and transmit, over the network <NUM>, reply data <NUM> (e.g., digital data representing a reply) that enables the client device <NUM> to integrate the set of winning digital components into the given electronic document, such that the set of winning digital components and the content of the electronic document are presented together at a display of the client device <NUM>.

In some implementations, the client device <NUM> executes instructions included in the reply data <NUM>, which configures and enables the client device <NUM> to obtain the set of winning digital components from one or more digital component servers. For example, the instructions in the reply data <NUM> can include a network location (e.g., a Uniform Resource Locator (URL)) and a script that causes the client device <NUM> to transmit a server request (SR) <NUM> to the digital component server <NUM> to obtain a given winning digital component from the digital component server <NUM>. In response to the request, the digital component server <NUM> will identify the given winning digital component specified in the server request <NUM> (e.g., within a database storing multiple digital components) and transmit, to the client device <NUM>, digital component data (DC Data) <NUM> that presents the given winning digital component in the electronic document at the client device <NUM>.

To facilitate searching of electronic documents, the environment <NUM> can include a search system <NUM> that identifies the electronic documents by crawling and indexing the electronic documents (e.g., indexed based on the crawled content of the electronic documents). Data about the electronic documents can be indexed based on the electronic document with which the data are associated. The indexed and, optionally, cached copies of the electronic documents are stored in a search index <NUM> (e.g., hardware memory device(s)). Data that are associated with an electronic document is data that represents content included in the electronic document and/or metadata for the electronic document.

Client devices <NUM> can submit search queries to the search system <NUM> over the network <NUM>. In response, the search system <NUM> accesses the search index <NUM> to identify electronic documents that are relevant to the search query. The search system <NUM> identifies the electronic documents in the form of search results and returns the search results to the client device <NUM> in a search results page. A search result is data generated by the search system <NUM> that identifies an electronic document that is responsive (e.g., relevant) to a particular search query, and includes an active link (e.g., hypertext link) that causes a client device to request data from a specified network location (e.g., URL) in response to user interaction with the search result. An example search result can include a web page title, a snippet of text or a portion of an image extracted from the web page, and the URL of the web page. Another example search result can include a title of a downloadable application, a snippet of text describing the downloadable application, an image depicting a user interface of the downloadable application, and/or a URL to a location from which the application can be downloaded to the client device <NUM>. In some situations, the search system <NUM> can be part of, or interact with, an application store (or an online portal) from which applications can be downloaded for install at a client device <NUM> in order to present information about downloadable applications that are relevant to a submitted search query. Like other electronic documents, search results pages can include one or more slots in which digital components (e.g., advertisements, video clips, audio clips, images, or other digital components) can be presented.

To select a digital component to be transmitted in response to a component request, the distribution system <NUM> may identify a set of digital components that are eligible to be transmitted in response to the component request. The distribution system <NUM> may then select one or more of the eligible digital components to be transmitted through, e.g., an auction procedure. In some implementations, the distribution system <NUM> performs an auction procedure by ranking the eligible digital components in accordance with their respective eligibility values, and selecting one or more highest-ranked digital components to be transmitted in response to the component request.

For example, the distribution system <NUM> may identify digital components A, B, and C as eligible to be transmitted in response to a component request. In this example, digital component a has an eligibility value of $<NUM>, digital component B has an eligibility value of $<NUM>, and digital component C has an eligibility value of $<NUM>, where the eligibility values of the digital components represent bids associated with the digital components. The distribution system <NUM> may rank (e.g., in descending order) the digital components in accordance with their respective eligibility values as: C, A, B. Finally, the distribution system <NUM> may select the highest ranked digital component C for transmission in response to the component request.

After selecting a digital component to be transmitted in response to a digital component request, the distribution system <NUM> determines a transmission requirement for the selected digital component. A transmission requirement specifies an action to be performed by the provider of a digital component in response to a transmission of the digital component. For example, the transmission requirement may specify that the provider of the digital component submit an amount of compensation in response to the transmission of the digital component. In some cases, the amount of compensation specifies an amount to be submitted for each instance of specific events attributed to the presentation of the digital component (e.g., user interactions with the digital component).

The distribution system <NUM> may determine the transmission requirement of the selected digital component based on the eligibility value of the selected digital component and/or the eligibility values of the other digital components that were determined as eligible to be transmitted in response to the component request. For example, the distribution system <NUM> may identify digital components A, B, and C as eligible for transmission in response to a digital component request, where A, B, and C have respective eligibility values of $<NUM>, $<NUM>, and $<NUM>. The distribution system <NUM> may select digital component C for transmission (since it has the highest eligibility value), and may determine the transmission requirement for digital component C to be the next highest eligibility value from amongst the eligibility values of the eligible digital components. In this example, next highest eligibility value is $<NUM> (i.e., the eligibility value of digital component A), and therefore the distribution system <NUM> may determine the transmission requirement of digital component C to be $<NUM>.

<FIG> shows a digital component that is being displayed on a screen of a user device, e.g., a smartphone. The digital component includes an image that fills the screen of the device and additional visual elements including a logo <NUM>, a segment of text <NUM>, and an interactive element <NUM> that overlay the image. In this example, the image includes a first portion <NUM> that was provided to the digital component distribution system, and a second portion <NUM> that was generated using the image extension system <NUM>. After receiving a request for a digital component to be provided to the user device, the digital component distribution system generates an extended image that fills the screen of the device, and overlays the additional elements <NUM>, <NUM>, and <NUM> on the extended part <NUM> of the image. In this example, the "block" where the digital component is displayed corresponds to the entire screen of the user device, but the block could occupy less than all of the screen.

<FIG> shows a digital component that is being displayed in a "banner" block, e.g., along with search results or on a third party website. Similar to <FIG>, the digital component includes an image that fills the block and additional elements <NUM>. In this example, the image includes a first portion <NUM> that was provided to the digital component distribution system, and a second portion <NUM> that was generated using the image extension system <NUM>. After receiving a request for a digital component to be presented in a banner block on a user device, the digital component distribution system generates an extended image that fills the block, and overlays the additional elements <NUM> on the extended part <NUM> of the image.

<FIG> is an illustration of an example user interface <NUM> that can be presented to an entity (e.g., an advertiser, publisher, or other content provider) that provides a digital component for transmission by a digital component distribution system (e.g., as described with reference to <FIG>). The user interface <NUM> presents an image <NUM> included in the provided digital component and multiple extended images <NUM>-A-C that extend the image <NUM>. Generally, each of the extended images <NUM>-A-C are different as a result of being generated using different image extension procedures. For example, one of the extended images may have been generated using the image extension system described with reference to <FIG>, while another of the extended images may have been generated using an image extension procedure based on partial differential equations (PDEs), which will be described in more detail below.

The user interface <NUM> prompts the provider of the digital component to select a preferred one of the extended images (e.g., that is most visually appealing to the digital component provider). The digital component provider can select a particular extended image, e.g., by clicking on particular extended image using a mouse <NUM> (or in any of a variety of other ways). If the distribution system later determines that the image included in the provided digital component qualifies for extension, the distribution generates the extension of the image using the image extension procedure corresponding to the particular extended image selected by the user.

Images can be extended in a variety of ways, e.g., using the image extension system described with reference to <FIG>, or using an image extension procedure based on PDEs. In a PDE extension method, the pixel values in the extended portion of an image are identified as an (approximate or exact) solution to a PDE subject boundary conditions specified by the pixel values on a portion of the boundary of the original image. The PDE may be, e.g., a diffusion PDE, or any other appropriate PDE. The solution to the PDE subject to the boundary conditions may be obtained using any appropriate numerical technique, e.g., a finite element technique.

<FIG> is a flow diagram of an example process <NUM> for providing an extended image in response to a request. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a distribution system, e.g., the distribution system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system determines that a provided image qualifies for image extension (<NUM>). In one example, the system may receive a request for a digital component to be presented in a block, e.g., alongside search results or on a third party webpage, and thereafter determine that a digital component that includes the provided image is responsive to the request. In this example, the system may determine that the provided image qualifies for extension based on the size of the image differing from the size of the block, e.g., if the image and the block have different aspect ratios.

The system provides an input the includes the image to a generative neural network that processes the input in accordance with trained values of the generative neural network parameters to generate an extended image (<NUM>). The extended image has (i) more rows, more columns, or both than the provided image, and (ii) is predicted to be a realistic extension of the provided image.

The input processed by the generative neural network may include a baseline image and a mask image, both having the same number of rows and columns as the extended image. The baseline image includes: (i) a first portion that corresponds to the provided image, and (ii) a second portion having default pixel values. The mask image identifies the first portion and the second portion of the baseline image. In one example, the pixels in the mask image corresponding to the first portion of the baseline image each have a first pixel value, and the pixels in the mask image corresponding to the second portion of the baseline image each have a different second pixel value.

The generative neural network includes multiple convolutional neural network layers and can have any appropriate neural network architecture. For example, the generative neural network may include one or more skip connections, one or more instance normalization layers, or both.

The generative neural network has been jointly trained with a discriminative neural network using an adversarial loss objective function. The discriminative neural network is configured to process a given image to generate a discriminative output characterizing a likelihood that the given image was generated using the generative neural network.

The system trains the generative neural network using the adversarial loss objective function over multiple training iterations. At each training iteration, the system processes a training input that includes a training image using the generative neural network to generate a training extended image that extends the training image, and then generates a discriminative neural network input based on the training extended image. In one example, the system generates the discriminative neural network input by using the training image to overwrite the portion of the training extended image corresponding to the training image. The system processes the discriminative neural network input using the discriminative neural network to generate a discriminative output characterizing a likelihood that the discriminative neural network input was generated using the generative neural network. Thereafter, the system adjusts the current values of the generative neural network parameters based on the adversarial loss objective function, where the adversarial loss objective function depends on the discriminative output generated by the discriminative neural network.

The training image is a cropped representation of a target image, and the discriminative neural network is conditioned on a semantic feature representation of the target image. The system may determine the semantic representation of the target image to be an intermediate output generated by a classification neural network by processing the target image. Conditioning the discriminative neural network on the semantic feature representation of the target image may include, e.g., determining the discriminative output based on a similarity measure between an intermediate output of the discriminative neural network and the semantic feature representation of the target image. The discriminative output may be additionally determined based on the output of the final layer of the discriminative neural network.

In addition to training the generative neural network using the adversarial loss objective function, the system may additionally train the generative neural network using a reconstruction loss objective function. More specifically, the system may adjust the current values of the generative neural network parameters based on a reconstruction loss objective function that characterizes a similarity of the training extended image to the target image.

The system provides the extended image in response to a request (<NUM>). In some cases, the request specifies additional elements to be presented with the image in the block, and the system overlays the additional elements on the extended portion of the extended image. After the system provides the extended image in response to the request, the extended may be, e.g., presented alongside search results or on a third-party webpage.

<FIG> is a block diagram of an example computer system <NUM> that can be used to perform operations described above. The system <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, and an input/output device <NUM>. Each of the components <NUM>, <NUM>, <NUM>, and <NUM> can be interconnected, for example, using a system bus <NUM>. The processor <NUM> is capable of processing instructions for execution within the system <NUM>. In one implementation, the processor <NUM> is a single-threaded processor. In another implementation, the processor <NUM> is a multi-threaded processor. The processor <NUM> is capable of processing instructions stored in the memory <NUM> or on the storage device <NUM>.

In one implementation, the input/output device <NUM> can include one or more network interface devices, e.g., an Ethernet card, a serial communication device, e.g., and RS-<NUM> port, and/or a wireless interface device, e.g., and <NUM> card. In another implementation, the input/output device can include driver devices configured to receive input data and send output data to other input/output devices, e.g., keyboard, printer and display devices <NUM>.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks, and CD-ROM and DVD-ROM disks.

Claim 1:
A method performed by one or more data processing apparatus, the method comprising:
providing an input that comprises a provided image to a generative neural network having a plurality of generative neural network parameters, wherein:
the generative neural network processes the input in accordance with trained values of the plurality of generative neural network parameters to generate an extended image;
the extended image has (i) more rows, more columns, or both than the provided image, and (ii) is predicted to be an extension of the provided image;
the generative neural network is jointly trained with a discriminative neural network having a plurality of discriminative neural network parameters that is configured to process a given image to generate a discriminative output characterizing a likelihood that the given image was generated using the generative neural network;
the generative neural network has been trained using an adversarial loss objective function that depends on the discriminative output;
training the generative neural network using the adversarial loss objective function comprises processing a training input that comprises a training image using the generative neural network; and
the discriminative neural network is conditioned on a semantic feature representation of a target image, the training image being a cropped representation of the target image,
wherein the discriminative neural network is conditioned on the semantic feature representation of the target image by either:
providing the semantic feature representation of the target image as an additional input to the discriminative neural network; or
determining the discriminative output based at least in part on a similarity measure between the semantic representation of a target extended image and an intermediate output of the discriminative neural network, and
wherein the semantic representation of the target extended image is normalized by a normalization engine prior to using it to condition the discriminative neural network.