Patent Description:
It is with respect to these and other general considerations that the aspects disclosed herein have been made. Although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving the specific problems identified in the background or elsewhere in this disclosure.

<CIT> discloses a method for automatically generating an edited video based on product images.

The present invention provides a computer-implemented method and system for automatically generating a video in an e-commerce system as defined in the appended claims.

According to the present disclosure, the above and other issues are resolved by automatically generating a video associated with an item in an item listing based on a machine-learning model. The present disclosure relates to automatically generating video data of goods (e.g., an item or multiple items for sale) in an e-commerce shopping marketplace using one or more of machine-learning models. An image receiver receives images associated with an item of an item listing and/or multiple item listings. An image extractor generates visual descriptors associated with each image through computer vision analysis and extracts a unique set of images by removing redundant/duplicate images. An image sorter sorts the images using a machine-learning model and generates a sequence of images, which corresponds to a sequence of video segments of a video. A text placer automatically identifies a region in a video frame and/or an image using another machine-learning model and inserts text data into the video frame and/or the image. A video data optimizer optimizes the video based on a pattern of previous adjustments and refinements to other video data done manually by users. The disclosed technology stores and publishes the automatically generated video data for viewing by viewers (e.g., buyers) in an online shopping marketplace.

The disclosed technology includes a first machine-learning model, referred to as a visual description model, to predict and generate a visual descriptor for an image. Additionally or alternatively, the disclosed technology includes a second machine-learning model, referred to as a visual transition model, to predict a sequence of images associated with an item in video data based on a category of the item and/or attributes of the seller. Additionally or alternatively, the disclosed technology includes a third machine-learning model, referred to as a region model, to predict regions within video frames for placing text data. The third machine-learning model further predicts text attributes of a text for insertion into the video data. Additionally or alternatively, a fourth machine-learning model, referred to as a video data model, automatically optimizes video data based on exemplary adjustments that are previously made manually by video editors.

The present disclosure relates to systems and methods for automatically generating video data according to at least the examples provided in the sections below. In particular, the present disclosure relates to a computer-implemented method for automatically generating a video for an item listing in an e-commerce system. The method comprises receiving a set of images, wherein the set of images comprises a plurality of images received as part of the item listing; generating a set of visual descriptors for the set of images, wherein generating the set of visual descriptors comprises computing a visual descriptor, using a first machine-learning model, for each image in the set of images; comparing visual descriptors from the set of visual descriptors to identify one or more redundant images in the set of images; generating, based on the compared visual descriptors, a unique set of images by removing the one or more redundant images from the set of images; determining, based on the unique set of images, an ordered set of images, wherein the ordered set of images is ordered based at least upon a category of the item listing; automatically determining, using additional data from the item listing, text associated with one or more images in the ordered set of images, wherein the text associated with the one or more images is automatically added to the one or more images; automatically generating, based on the ordered set of images, the video, wherein the video comprises the ordered set of images; and providing the video as part of the item listing.

The system comprises a processor; and a memory storing computer-executable instructions that when executed by the processor cause the system to receiving a set of images, wherein the set of images comprises a plurality of images received as part of the item listing; generating a set of visual descriptors for the set of images, wherein generating the set of visual descriptors comprises computing a visual descriptor, using a first machine-learning model, for each image in the set of images; comparing visual descriptors from the set of visual descriptors to identify one or more redundant images in the set of images; generating, based on the compared visual descriptors, a unique set of images by removing the one or more redundant images from the set of images; determining, based on the unique set of images, an ordered set of images, wherein the ordered set of images is ordered based at least upon a category of the item listing; automatically determining, using additional data from the item listing, text associated with one or more images in the ordered set of images, wherein the text associated with the one or more images is automatically added to the one or more images; automatically generating, based on the ordered set of images, the video, wherein the video comprises the ordered set of images; and providing the video as part of the item listing.

The method comprises receiving a set of images, wherein each of images in the set of images describes in part an item for transaction in an online shopping marketplace, wherein the set of images include a first image and a second image, and wherein the first image includes a first region depicting the item; determining, based on a category of the item, a sequence of the set of images using a machine-learning model, wherein the sequence includes the first image preceding the second image; automatically generating the video, wherein the video includes a first video segment with the first image proceeding a second video segment with the second image; determining a second region in the first video segment, wherein the second region is without an overlap with the first region; inserting text data into the second region; and publishing the video in the online shopping marketplace for viewing.

This Summary is provided to introduce a selection of concepts in a simplified form, which is further described below in the Detailed Description. Additional aspects, features, and/or advantages of examples will be set forth in part in the following description and, in part, will be apparent from the description, or may be learned by practice of the disclosure.

Various aspects of the disclosure are described more fully below with reference to the accompanying drawings, which from a part hereof, and which show specific example aspects. However, different aspects of the disclosure may be implemented in many different ways and should not be construed as limited to the aspects set forth herein; rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the aspects to those skilled in the art. Aspects may be practiced as methods, systems, or devices. Accordingly, aspects may take the form of a hardware implementation, an entirely software implementation or an implementation combining software and hardware aspects. The following detailed description is, therefore, not to be taken in a limiting sense.

Online shopping systems, including e-commerce shopping marketplace systems, present the information associated with the item to buyers. The information associated with the item may include image data, video data, and textual descriptions of the item. For example, the item may be a pair of shoes for sale. In aspects, a webpage associated with the pair of shoes may include a name and textual description of the pair of shoes, one or more images of the pair of shoes, and one or more videos about the pair of shoes. The online shopping systems provide a set of tools for the sellers to upload the information associated with the pair of shoes.

In some cases, online shopping systems rely upon sellers to upload the information associated with the item for sale. Alternatively or additionally, some online shopping systems provide stock images for users to associate with their listing. Furthermore, an online shopping system can provide a set of tools for the sellers to upload the information to a server (e.g., a cloud) over the Internet. While preparing image data associated with an item may be relatively easy for the sellers by taking photos of the item, preparing video data often causes burdens upon the sellers because of a level of skills needed to create a video. Some traditional online shopping systems provide tools for automatically generating a video based on a set of image and textual data uploaded to the server by the sellers. Some automated tools concatenate the image data in a predetermined time interval of video data, in a sequence of the image data uploaded to the server without automatic sorting. As a result, the automatically generated video in the traditional systems may appear disorganized and counter-intuitive to viewers (e.g., buyers). Not all images may be appropriate for including in the video. For example, some sellers may upload multiple image data of the same appearance of the item but at distinct pixel resolutions of image data. Some systems leave steps of deciding a selection of image data and a sequence of the image data to the sellers as manual operations.

Some tools automatically insert a text (e.g., "<NUM>% Off Plus Free Shipping") at random or predefined positions within video frames of the video data. Such tools may insert the text in a predefined font, size, orientation, and color. As a result, the text may block an image of the item by overlaid on the item. The text may be illegible in the video because its color that blends into the image data. The font and size may appear intuitively awkward to the buyers in comparison to the item appearing in the video. Some systems leave steps of inserting texts into the video data to the sellers as manual operations. A burden imposed upon the seller to edit video data by mastering a video editing tool may be excessive.

Accordingly, there is a trade-off in an online marketplace between generating a video that appear logical and persuasive to buyers and costs of doing so. The present application solves this trade-off issue by providing a system that automatically sorts image data using a first machine-learning model with context analysis, generates video data, and inserts text data into video data using a second machine-learning model with weighted image analysis. The present application enables balancing competing demands of effectiveness and efficiency of the marketplace by use of the automatic video generator.

As discussed in more detail below, the present disclosure relates to machine-learning, models for automatically generating a video that describes an item in an online shopping marketplace. In particular, the models include predicting a sequence of image data in video frames, predicting a region for inserting text data into the video frames, and optimizing video data.

<FIG> illustrates an overview of an example system <NUM> for automatically generating a video. System <NUM> represents a system for using machine learning models (e.g., artificial intelligence) to sort image data associated with an item for generating video frames for video data, insert text data into video frames of the video data, and optimizing the video data. System <NUM> includes a client device <NUM>, an application server <NUM>, an online shopping server <NUM>, and a network <NUM>. The client device <NUM> communicates with the application server <NUM>, which includes one or more sets of instructions to execute as applications on the client device <NUM>. The application server <NUM> includes an online shopping app <NUM> (e.g., a buyer application) and a storefront maintenance app <NUM> (e.g., a seller application). The one or more sets of instructions in the application server <NUM> may provide an interactive user interface (not shown) through an interactive interface <NUM>. In alternate embodiments, a single application may perform the buying and selling functionality of the online shopping app <NUM> and the storefront maintenance app <NUM>.

The online shopping server <NUM> includes a storefront server <NUM>, an item database <NUM>, and a video generator <NUM>. The network <NUM> provides network connectivity among the client device <NUM>, the application server <NUM>, and the online shopping server <NUM>. Additionally or alternatively, the video generator <NUM> may be outside the online shopping server <NUM> across the network <NUM>, thereby the network <NUM> providing network connectivity among the client device <NUM>, the application server <NUM>, the online shopping server <NUM>, and the video generator <NUM>.

The client device <NUM> connects with the application server <NUM> via the network <NUM> to execute applications that include user interactions through the interactive interface <NUM>. The application server <NUM> interacts with the client device <NUM> and the online shopping server <NUM> via the network <NUM> to perform online shopping as a seller or a buyer of items.

The client device <NUM> is a general computer device providing user-input capabilities e.g., via the interactive interface <NUM> for online shopping over the network <NUM>. In some aspects, the client device <NUM> optionally receives user input from a seller of items. The seller uploads information about an item for sales transactions in an online shopping marketplace. The information about the item includes image data of the item, a brief description of the item, price information, quantity information, and the like. The interactive interface <NUM> may render a graphical user interface associated with a web browser, for example. In aspects, the client device <NUM> may communicate over the network <NUM> with the application server <NUM>.

The application server <NUM> is a server that enables a seller (who may post items for sale) and a buyer (who purchases the items) to interactively access and use the system <NUM> on the client device <NUM>. The application server <NUM> may comprise applications including the online shopping app <NUM> and the storefront maintenance app <NUM>. The online shopping app <NUM> may provide a rendering of items for a purchase by the user as the buyer. The storefront maintenance app <NUM> may provide one or more tools for the seller to upload information associated with items (i.e., item information) for posting items for sale in the e-commerce shopping marketplace. In aspects, the item information include a name and textual descriptions of the item, image data associated with the item, a price and a quantity of the item available for sales. In aspects, the storefront maintenance app <NUM> may connect with a storefront server <NUM> of the online shopping server <NUM> to post information about an item for sale on an online shopping site (not shown). In some aspects, the storefront maintenance app <NUM> may further connect with a video generator <NUM> of the online shopping server <NUM> to upload image data associated with the item. The video generator <NUM> may automatically generate video data that describes the item for use by the storefront server <NUM>. In some other aspects, the storefront maintenance app <NUM> may provide a tool for uploading video data that describe the item when the seller prepares the video data on its own. The storefront maintenance app <NUM> may receive a confirmation from the online shopping server <NUM> when the online shopping server <NUM> successfully receives the information about the item.

In aspects, the information about the item may include a name, a brief description of the item, a quantity, a price, and one or more image data that depict the item. Additionally or alternatively, the information about the item includes category information of the item. For example, the item may be a pair of shoes. The one or more image data may include photos of the pair of shoes in different views with background scenes for use. The one or more image data may further include a close-up of product information label with a product code, a serial number of the pair of shoes, and a list of features of the pair of shoes.

The online shopping server <NUM> represents the applications/systems used for automatically generating video data associated with items for sale. The online shopping server <NUM> further provides a storefront to sell the item on the online shopping site (not shown). The video generator <NUM> automatically generates video data for a given set of image data associated with an item for sale. The video generator <NUM> uses one or more artificial intelligence models, machine-learning models, or the like, to improve accuracy in the automatic operations. The video generator <NUM> includes an image receiver <NUM>, an image extractor <NUM>, an image sorter <NUM>, a visual transition model <NUM>, a text placer <NUM>, a region model <NUM>, a video data optimizer <NUM>, a video data model <NUM>, and a video data storage <NUM>.

The image receiver <NUM> receives image data associated with an item from the storefront maintenance app <NUM> used by a seller through the interactive interface <NUM> on the client device <NUM>. Among other things, the image data about the item includes photos of a product logo, a brief description, and image data of the item. The image receiver <NUM> may further receive information associated with the item. The information associated with the item may include but not limited to a category of the item. For example, the image receiver <NUM> may receive from the storefront maintenance app <NUM> of the Application server <NUM> a set of image data associated with a pair of shoes and a category of the item as shoes. In some aspects, the received image data may include images that are received as part of the item listing.

The image extractor <NUM> extracts one or more image data from the received image data by use of image analysis. The image extractor <NUM> analyzes the received image data and identifies those that are identical or very similar. In aspects, the image extractor <NUM> selects a set of image data by removing redundant image data. In some aspects, the image extractor <NUM> select a unique set of image data such that a number of image data matches with a predetermined number of image data for generating video data with a predetermined time duration. Extraction criteria may include by image data that are redundant and/or a level of clarity of the image data. In aspects, the image extractor <NUM> outputs the unique set of image data.

For example, a seller may upload image data associated with a pair of shoes for sale. The image data may include thirty photos of the pair of shoes from various views and in distinct pixel resolutions. Some of the photos may look identical or very similar to one another. The image data may further include an image of a list of features of the pair of shoes (e.g., a name, color, size available, materials used, and the like). The predetermined time duration of a video may be for sixty seconds, which may translate into including ten pieces of image data as the predefined number of image data for extraction. Accordingly, the image extractor <NUM> may extract a unique set of image data by selecting a number of unique photos and image data that is sufficient for a number of image data needed for the video.

The image sorter <NUM> sorts a set of image data extracted from the received image data for automatically generating video data. In particular, the image sorter <NUM> automatically determines a sequence of the image data using a visual transition model <NUM>. In aspects, the visual transition model <NUM>, when trained, predicts a sequence (e.g., an ordering) of image data based on a given category of an item for sale. In aspects, the visual transition model <NUM> may further specify time duration of respective video segments with respective contexts of image data (e.g., an overview photo, close-ups, item features, and the like) under the given category of an item. In aspects, conditions for a sequence (e.g., order criteria) may include a category of an item for sale.

In aspects, a video comprises video data. The video data includes a plurality of video segments in varying time durations for each video segment. A time duration of each video frame may be constant throughout the video data. Image data appear as a part of content of a video frame. A sequence of image data may correspond to a sequence of video segments that include the respective image data.

For example, the image sorter <NUM> may sort a set of image data associated with a pair of shoes under a category of item "shoes. " The visual transition model <NUM> may be trained to predict a sequence of image data for generating a sequence of video frames of a video. The video describes a pair of shoes. The sequence of image data may start with a photo of the pair of shoes in a birds eye view, followed by close-ups of views from various angles, followed by descriptions of the pair of shoes, and ending by the bird's eye view of the item. The image sorter <NUM> sorts the extracted image data of the pair of shoes into the sequence associated with the product category of shoes.

In aspects, the image sorter <NUM> generates video data uses the sequence of image data in the sequence of video frames of the video data. The image sorter <NUM> may generate the video data with the predetermined time durations by generating respective video frames according to time durations as specified by the visual transition model <NUM>. In aspects, the visual transition model <NUM> may be trained based on video data stored in the video data storage <NUM>. The video data stored in the video data storage <NUM> may correspond to other items under a variety of categories.

The text placer <NUM> places texts into video frames of the video data. In particular, the text placer <NUM> determines one or more video frames and regions within the one or more video frames and inserts a given text in the regions. In aspects, the text placer <NUM> receives text data and determines a set of video frames and regions within the set of video frames for inserting the text data. The text placer <NUM> may use the visual transition model <NUM> to determine the set of video frames. The text placer <NUM> may further use the region model <NUM> to determine the regions within the set of video frames for inserting the text data. In aspects, the visual transition model <NUM> may predict a set of video frames for inserting a particular context associated with the text data. For example, when the text data represent a context of a promotional aspects of sales transactions (e.g., "<NUM>% Off PLUS Free Shipping"), the visual transition model <NUM> may predict inserting the text data in a set of video frame with the birds eye view of the item toward the end of the video.

In aspects, the region model <NUM> may predict a region for inserting text data into a video frame by preventing the text data from blocking an image of an item. The region model <NUM> further predicts textual attributes (e.g., a font, size, color, and the like) for inserting the text data. For example, use of the predicted textual attributes may prevent the text data from becoming illegible because by predicting a color that is distinct from the background color of the region. The text placer <NUM> outputs video data that includes with text data in video frames.

For example, in a given text data "<NUM>% Off PLUS Free Shipping" for insertion into video data of the pair of shoes, the text placer <NUM> selects a set of video frames with the birds eye view of the pair of shoes toward the end of the video data. The text placer <NUM> selects the set of video frames as predicted by the visual transition model <NUM>. The text placer <NUM> then determines a region in the respect video frames for inserting the text data. The region model <NUM> may predict a region that does not overlap with an image of the pair of shoes in the set of video frames and specify a font, color, and size for rendering the text data.

The video data optimizer <NUM> optimizes the video data with texts. In aspects, the video data optimizer <NUM> uses a video data model <NUM> to predict ways for optimizing the video data. The video data model <NUM> may be a model that is trained based on video data stored in the video data storage <NUM>, which includes examples of adjustments and customizations performed by the sellers on automatically generated video data. In some other aspects, the video data model <NUM> may be trained based on review data on respective video data by viewers (including the buyers and the sellers) of the video data on the online shopping site. For example, adjustments to the video data may include changes in a brightness, a video aspect ratio, a resolution, a file size, and the like. The video data optimizer <NUM> optimizes the video data and stores the video data in the video data storage <NUM>.

As will be appreciated, the various methods, devices, applications, features, etc., described with respect to <FIG> are not intended to limit the system <NUM> to being performed by the particular applications and features described. Accordingly, additional controller configurations may be used to practice the methods and systems herein and/or features and applications described may be excluded without departing from the methods and systems disclosed herein.

<FIG> illustrates an example of a system for extracting and sorting image data in according to the aspects of the present disclosure. For example, an image extractor may include the image extractor <NUM> as shown in <FIG>. An image sorter may include the image sorter <NUM> as shown in <FIG>. A system <NUM> includes the image extractor <NUM> and the image sorter <NUM>. The image extractor <NUM> receives a set of image data <NUM> as input and generates a set of select image data (i.e., extracted image data) by selecting one or more image data from the set of image data <NUM> by removing redundant image data. In aspects, the image sorter <NUM> uses a visual transition model <NUM> for sorting the extracted image data into a sequence (i.e., an ordered set) of video frames based on the item category data <NUM>. The image sorter <NUM> generates an ordered set of video frames as video data based on the sorted sequence of the select image data. For example, the set of image data <NUM> includes a set of image data of a pair of shoes as an item for sale. The item category data <NUM> may include "shoes" as a category of the item.

The image extractor <NUM> may include a visual descriptor generator <NUM>, a descriptor distance determiner <NUM>, and an image selector <NUM>. The visual descriptor generator <NUM> identifies and/or generates visual descriptors <NUM> associated with respective image data in the set of image data <NUM>. The visual descriptors <NUM> may indicate visual characteristics of image data. In aspects, the visual descriptors <NUM> may include but not limited to a type of visual content, shape, appearance, color, viewpoint, and the like. In aspects, the visual descriptor generator <NUM> may use a visual description model <NUM> to generate a visual descriptor associated with each image data. In some aspects, the visual description model <NUM> represents a machine-learning model that is used to predict a visual descriptor for a given image based on pattern matching of images. The visual description model <NUM> may be trained based on examples of images and true examples of image descriptors. In aspects, the visual descriptor generator <NUM> may employ a technique that relates to a computer vision analysis identifying a visual descriptor for an image.

The descriptor distance determiner <NUM> determines distances between one or more pairs of image data in the extracted image data. In aspects, the descriptor distance determiner <NUM> determines a distance between each pair of a combination of pairs of the extracted image data. The distance may be based on similarities in the visual descriptors <NUM> of the extracted image data. For example, the descriptor distance determiner <NUM> may determine two photos of the pair of shoes in a birds eye view as short in distance because both photos depict the pair of shoes in the same viewpoint. In aspects, the descriptor distance determiner <NUM> determines a set of image data that include image data that are similar in visual descriptors.

The image selector <NUM> generates a set of select image data by removing image redundant data by comparing the determined distances with a predefined threshold distance. In some aspects, the image selector <NUM> may remove image data that are below a predetermined level of visual quality. For example, a level of visual quality may indicate a severity level of noises in image data.

The image sorter <NUM> sorts the image data using a combination of the set of select image data, item category data <NUM>, and the visual transition model <NUM>. The image sorter <NUM> includes a sequence determiner <NUM> and an image sequencer <NUM>. The sequence determiner <NUM> determines a sequence of the extracted image data based on a combination of at least the following three aspects. A first is the set of image data with similar distances in visual descriptors. A second is the received item category data <NUM>. A third is a sequence of video frames based on the image data as predicted by the visual transition model <NUM>. For example, the sequence determiner <NUM> determines a sequence of video frames for describing the pair of shoes based at least on a combination of photos and images with distinct visual descriptors about the pair of shoes, the item category of shoes, and a predicted sequence of visual transitions about a shoes video.

The visual transition model <NUM> predicts a sequence of image data for generating video frames based on a given attribute of selling an item in the online shopping site. The attribute of selling an item may include but not limited to a category of an item for sale, a type of viewers (e.g., a level of expertise of the buyers using the item), and the like. For example, the visual transition model <NUM> predicts a sequence of video frames for a video that describes a pair of shoes. The sequence may correspond to a video with sixty seconds of duration. The sequence may include a first scene with an overview of the pair of shoes, a second scene with close-ups (e.g., in a sequence of a right side view, a left side view, a top view, a view of the sole, and the like), a third scene with the overview of the pair of shoes. The sequence may also include information associated with inserting text data into the respective video frames. For example, the information may include marking the third scene as appropriate for inserting text data associated with sales promotion of the item. In aspects, the video transition model predicts a sequence of video frames with visual descriptors associated with the respective video frames. Use of the visual transition model <NUM> enables the sequence determiner <NUM> to determine a sequence of video frames with visual transitions that are smooth and contextually meaningful to the viewers.

The image sequencer <NUM> generates video data with a sequence of video frames with image data as specified by the visual transition model <NUM>. In aspects, the visual transition model <NUM> specifies time durations of respective video segments of the video data. For example, the image sequencer <NUM> generates a sequence of video frames <NUM> as video data for the pair of shoes based on the extracted image data of the pair of shoes and the predicted sequence of video frames by the visual transition model <NUM>.

<FIG> illustrates an example of a system with a text placer in accordance with the aspects of the present disclosure. For example, the text placer may include the text placer <NUM> as shown in <FIG>. A system <NUM> includes a text placer <NUM>. The text placer <NUM> may receive a set of video frames <NUM> and text data <NUM> as input, identifies video frames of a video segment and a region within the video frames, and inserts the text data <NUM> into the region within in the video frames of the video segment. The text placer <NUM> may use a background prediction model <NUM> for determining the region for inserting the text data <NUM>. In aspects, the disclosed technology may automatically determine the text data <NUM> for insertion into video frames <NUM> based on information associated with item listing. For example, the information associated with the item listing may include a name of an item, a catch copy associated with the item, a description of sales promotion, and the like. In aspects, the text placer <NUM> may obtain the text data <NUM> from the item database <NUM>.

The text placer <NUM> includes a video frame extractor <NUM>, a placement determiner <NUM>, and a text inserter <NUM>. The video frame extractor <NUM> extracts respective video frames from the set of video frames <NUM> for analyzing content of the respective video frames. In aspects, the video frame extractor <NUM> generates feature maps <NUM> (e.g., a feature map <NUM> as shown in <FIG>) corresponding to with the respective video frames. A feature map includes locations of feature points of a video frame. The feature points may indicate locations within a video frame that depict features (e.g., an edge) of an image of an item. The feature points may be on the edges, the corners or at any location indicative of a feature. For example, the disclosed technology may use SIFT (the scale-invariant feature transform) feature points and calculate and/or determine a map based on them. Additionally or alternatively, the disclosed technology may use a model for determining optimal text attributes without detecting feature points.

Maybe that single model will generate a map which defines where the optimal text box would be. The text placer <NUM> determines a location for inserting the text data <NUM> by avoiding an overlay of the text data on the feature points in the map.

The placement determiner <NUM> determines a region and textual attributes for inserting the text data <NUM> in the video frames. In aspects, the placement determiner <NUM> determines a region for inserting the text data <NUM> based on the feature maps <NUM> and the background prediction model <NUM>. In aspects, the background prediction model <NUM> predicts a region and color for inserting text data in a video frame based on the feature maps <NUM> of the respective video frames. The region may be without overlapping or including feature points in the feature maps <NUM>. The placement determiner <NUM> further determines a font and size of the text data for fitting the text data into the region. For example, the placement determiner <NUM> determines a region that does not overlap with an image of the pair of shoes in the video frames and further determines a font and a font size that enables fitting the text data "<NUM>% PLUS Free Shipping" inside the region. In aspects, the disclosed technology may train the background prediction model <NUM> based on exemplar data using the video data stored in the video data storage (e.g., the video data storage <NUM> as shown in <FIG>).

The text inserter <NUM> inserts the text data <NUM> into a region within the video frame using a font, size, and color as specified by the placement determiner <NUM>. In aspects, the text inserter <NUM> generates a set of video frames with texts <NUM>. For example, the text inserter <NUM> may insert a text string "<NUM>% Off PLUS Free Shipping" in a set of frame data corresponding to a video segment toward the end of the video data. The text inserter <NUM> may insert the sales promotion information in a region that does not overlap with an image of the pair of shoes in the video frames.

<FIG> illustrates an example of image data in accordance with the aspects of the present disclosure. The example 400A includes unsorted image data <NUM> and sorted image data <NUM>. The unsorted image data <NUM> includes image data associated with a pair of shoes as an item for sale, in a sequence of the respective image data uploaded by the seller. The image data may be in a sequence of a sole <NUM>, an overview with a scenery <NUM>, an upper view <NUM>, an item description data <NUM>, and an overview in a vertical orientation <NUM>.

In aspects, the image sorter (e.g., the image sorter <NUM> as shown in <FIG>) may sort the unsorted image data <NUM> into the sorted image data <NUM> based on an item category <NUM> of shoes and a visual transition model <NUM> (e.g., the visual transition model <NUM> as shown in <FIG>). The visual transition model <NUM> may predict a sequence of video frames for generating a video based on an item category <NUM>. In the example 400A, the visual transition model <NUM> specifies a sequence of frame data with particular visual descriptors of each of the image data in the unsorted image data <NUM> for generating a video on the pair of shoes. In aspects, the sorted image data <NUM> reflects the predicted sequence of video frames. For example, the predicted sequence may be the upper view <NUM>, the overview in a vertical orientation <NUM>, the sole <NUM>, the item description data <NUM>, and then the overview with a scenery <NUM>. In aspects, the sorted image data <NUM> includes time <NUM>, which describes time durations of respective video segments as visual transitions take place at predetermined times.

<FIG> illustrates an example of inserting text data into a video frame in accordance with aspects of the present disclosure. An example 400B includes a video frame <NUM> before text insertion, a feature map <NUM> associated with the video frame <NUM>, and a video frame <NUM> with text insertion. In aspects, a text placer (e.g., the text placer <NUM> as shown in <FIG>) extract the video frame <NUM> from a set of video frames, generates a feature map <NUM>, determines a region for inserting text data, and inserts the text data into content of the video frame.

In aspects, the feature map <NUM> includes a set of points. Each point indicates a feature of content of the video frame. The feature may represent an edge of an image within the content based on changes in color and/or an image texture of the content. For example, a feature point <NUM> corresponds to a tip of a shoe as shown in the video frame <NUM>. The text <NUM> represents text data ("<NUM>% Off Plus FREE-SHIPPING") for inserting into the video frame <NUM>. The background prediction model <NUM> may predict a region <NUM> and textual attributes for the text insertion based on a combination of the content of the video frame <NUM> (e.g., an image of the shoe with a scenery) the feature map <NUM>, and the text <NUM>.

Additionally or alternatively, the disclosed technology may include training a model to estimate, in a direct manner, the positions of an optimal text box based on annotated or collected data for training. For example, training data may be images with annotations that indicate corresponding optimal text positions. The model may use predetermined features or texture maps as input. The model may receive an image as input for determining the positions.

In aspects, the background prediction model <NUM> identifies the region <NUM> as an optimal location and a size to insert the text data. For example, the region <NUM> is without overlapping with the image of the shoe and the background scenery. The background prediction model <NUM> may further predict textual attributes for inserting the text <NUM> based on an image texture with curves in the content of the video frame, a number of characters in the text <NUM>. The background prediction model <NUM> may further specify letter cases used for letters of the text <NUM>. Accordingly, the text placer inserts the text <NUM> and generates the video frame <NUM> using a font with curves that matches with the image texture, size that fits into the region <NUM>, and letter cases that optimizes expression of the text as a promotion announcement of the item.

<FIG> is an example of a method for automatically generating a video in accordance with aspects of the present disclosure. A general order of the operations for the method <NUM> is shown in <FIG>. Generally, the method <NUM> begins with start operation <NUM> and ends with end operation <NUM>. The method <NUM> may include more or fewer steps or may arrange the order of the steps differently than those shown in <FIG>. The method <NUM> can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. Further, the method <NUM> can be performed by gates or circuits associated with a processor, an ASIC, an FPGA, a SOC or other hardware device. Hereinafter, the method <NUM> shall be explained with reference to the systems, components, devices, modules, software, data structures, data characteristic representations, signaling diagrams, methods, etc., described in conjunction with <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>.

Following start operation <NUM>, the method <NUM> begins with receive operation <NUM>, which receives a set of images. The set of image may include a plurality of images uploaded from a storefront maintenance app (e.g., the storefront maintenance app <NUM> of the Application server <NUM> as shown in <FIG>).

A generate visual descriptors operation <NUM> generates visual descriptors associated with the respective image data. Visual descriptors represent visual characteristics of image data, including but not limited to color, appearance, texture, and the like.

A generate select image sets operation <NUM> generates a set of select image data that are without redundant image data. In aspects, the generate select image sets operation <NUM> may select image data by removing those image data that are duplicates and/or very similar based on visual descriptors. In aspects, the similarity is based on a distance between visual descriptors of video frames.

A generate video data operation <NUM> generates video data based on a set of select image data, sorted based on the visual transition model. In aspects, the visual transition model predicts a sequence of video frames based on a given item category data. For example, the visual transition model may specify a particular sequence of video segments (i.e., a sequence of image data) associated with a shoe as an item category.

An insert operation <NUM> inserts text data into the video data. In particular, the insert operation <NUM> may include extracting video frames from the video data, generating a feature map associated with a video frame, and determining a region in the video frame using a background prediction model (e.g., the background prediction model <NUM> as shown in <FIG>) for inserting text data associated with an item. Additionally or alternatively, the disclosed technology may generate the video data after inserting the text data into image data. In aspects, the insert operation <NUM> may precede the generate video data operation <NUM>.

An optimize operation <NUM> automatically optimizes the video data with text inserts using a video data model. For example, the video data model (e.g., the video data model <NUM> as shown in <FIG>) may predict adjustment to be made on the video data based on past occurrences of a user manually adjusting various parameters associated with the video data. In aspects, the various parameters may include brightness and contrast of content of video frames, color, a pixel resolution, and the like.

A store operation <NUM> stores the optimized video data in a video data storage (e.g., the video data storage <NUM> as shown in <FIG>). In aspects, the storefront server (e.g., the storefront server <NUM> as shown in <FIG>) may post the video data on item listing pages of the online shopping site for viewing and downloading by viewers and buyers. The method <NUM> ends with an end operation <NUM>.

<FIG> illustrates a simplified block diagram of the device with which aspects of the present disclosure may be practiced in accordance with aspects of the present disclosure. One or more of the present embodiments may be implemented in an operating environment <NUM>. This is only one example of a suitable operating environment and is not intended to suggest any limitation as to the scope of use or functionality. Other well-known computing systems, environments, and/or configurations that may be suitable for use include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics such as smartphones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

In its most basic configuration, the operating environment <NUM> typically includes at least one processing unit <NUM> and memory <NUM>. Depending on the exact configuration and type of computing device, memory <NUM> (instructions to perform generating a video associated with an item in an item listing as described herein) may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.), or some combination of the two. This most basic configuration is illustrated in <FIG> by dashed line <NUM>. Further, the operating environment <NUM> may also include storage devices (removable, <NUM>, and/or non-removable, <NUM>) including, but not limited to, magnetic or optical disks or tape. Similarly, the operating environment <NUM> may also have input device(s) <NUM> such as keyboard, mouse, pen, voice input, on-board sensors, etc. and/or output device(s) <NUM> such as a display, speakers, printer, motors, etc. Also included in the environment may be one or more communication connections, <NUM>, such as LAN, WAN, a near-field communications network, point to point, etc..

Operating environment <NUM> typically includes at least some form of computer readable media. Computer readable media can be any available media that can be accessed by at least one processing unit <NUM> or other devices comprising the operating environment. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other tangible, non-transitory medium which can be used to store the desired information. Computer storage media does not include communication media.

Communication media embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

The operating environment <NUM> may be a single computer operating in a networked environment using logical connections to one or more remote computers. The remote computer may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above as well as others not so mentioned. The logical connections may include any method supported by available communications media. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure as claimed in any way. The aspects, examples, and details provided in this application are considered sufficient to convey possession and enable others to make and use the best mode of claimed disclosure. The claimed disclosure should not be construed as being limited to any aspect, for example, or detail provided in this application. Regardless of whether shown and described in combination or separately, the various features (both structural and methodological) are intended to be selectively included or omitted to produce an embodiment with a particular set of features. Having been provided with the description and illustration of the present application, one skilled in the art may envision variations, modifications, and alternate aspects falling within the scope of the broader aspects of the general inventive concept embodied in this application that do not depart from the broader scope of the claimed disclosure.

The present disclosure relates to systems and methods for automatically generating video data according to at least the examples provided in the sections below. In particular, the present disclosure relates to a computer-implemented method for automatically generating a video for an item listing in an e-commerce system. The method comprises receiving a set of images, wherein the set of images comprises a plurality of images received as part of the item listing; generating a set of visual descriptors for the set of images, wherein generating the set of visual descriptors comprises computing a visual descriptor, using a first machine-learning model, for each image in the set of images; comparing visual descriptors from the set of visual descriptors to identify one or more redundant images in the set of images; generating, based on the compared visual descriptors, a unique set of images by removing the one or more redundant images from the set of images; determining, based on the unique set of images, an ordered set of images, wherein the ordered set of images is ordered based at least upon a category of the item listing; automatically determining, using additional data from the item listing, text associated with one or more images in the ordered set of images, wherein the text associated with the one or more images is automatically added to the one or more images; automatically generating, based on the ordered set of images, the video, wherein the video comprises the ordered set of images; and providing the video as part of the item listing. The visual descriptor includes at least one of: a type of visual content, shape, appearance, color, or viewpoint. The category of the item listing includes a type of items for transaction in an online shopping marketplace. The method further comprises determining the ordered set of images using a second machine-learning model, wherein the second machine-learning model is trained on visual transitions for predicting a sequence of video segments of the video based on order criteria, and wherein the order criteria includes the category of the item listing. The method further comprises extracting a video frame from the video data, wherein the video frame includes the image; generating a feature map for the video frame, wherein the feature map includes a plurality of feature points indicating locations of at least an edge of an item depicted in the image; determining, based on the feature map, a region in the video frame using a third machine-learning model; determining a textual attribute based on one or more attributes of the region using the third machine-learning model, wherein the textual attribute include at least color or size of the text, and wherein the one or more attributes of the region include one or more colors appearing in the region; and inserting the text into the region using the textual attribute. The method further comprises receiving the text associated with the one or more images in the ordered set of image data; automatically determining, based at least on background color of the one or more images in the ordered set of images, a color of the text associated with the one or more images; and automatically determining, based at least on a texture of a background of one or more images of the ordered set of images, a position of one or more characters in the text. The method further comprises automatically generating, the video using a fourth machine-learning model, wherein the fourth machine-learning model is a trained model for predicting one or more adjustments to be made upon the video, and wherein the fourth machine-learning model is trained based on one or more adjustments made upon videos manually by users. The color of the text is distinct from the one or more colors appearing in the region.

Another aspect of the technology relates to a system for automatically generating a video describing an item in an e-commerce marketplace. The system comprises a processor; and a memory storing computer-executable instructions that when executed by the processor cause the system to receiving a set of images, wherein the set of images comprises a plurality of images received as part of the item listing; generating a set of visual descriptors for the set of images, wherein generating the set of visual descriptors comprises computing a visual descriptor, using a first machine-learning model, for each image in the set of images; comparing visual descriptors from the set of visual descriptors to identify one or more redundant images in the set of images; generating, based on the compared visual descriptors, a unique set of images by removing the one or more redundant images from the set of images; determining, based on the unique set of images, an ordered set of images, wherein the ordered set of images is ordered based at least upon a category of the item listing; automatically determining, using additional data from the item listing, text associated with one or more images in the ordered set of images, wherein the text associated with the one or more images is automatically added to the one or more images; automatically generating, based on the ordered set of images, the video, wherein the video comprises the ordered set of images; and providing the video as part of the item listing. The visual descriptor includes at least one of: a type of visual content, shape, appearance, color, or viewpoint. The category of the item listing includes a type of items for transaction in an online shopping marketplace. The computer-executable instructions when executed further cause the system to determine the ordered set of images using a second machine-learning model, wherein the second machine-learning model is trained on visual transitions for predicting a sequence of video segments of the video based on order criteria, and wherein the order criteria includes the category of the item listing. The computer-executable instructions when executed further cause the system to extract a video frame from the video data, wherein the video frame includes the image; generate a feature map for the video frame, wherein the feature map includes a plurality of feature points indicating locations of at least an edge of an item depicted in the image; determine, based on the feature map, a region in the video frame using a third machine-learning model; determine a textual attribute based on one or more attributes of the region using the third machine-learning model, wherein the textual attribute include at least color or size of the text, and wherein the one or more attributes of the region include one or more colors appearing in the region; and insert the text into the region using the textual attribute. The computer-executable instructions when executed further cause the system to receive the text associated with the one or more images in the ordered set of image data; automatically determine based at least on background color of the one or more images in the ordered set of images, a color of the text associated with the one or more images; and automatically determine, based at least on a texture of a background of one or more images of the ordered set of images, a position of one or more characters in the text. The computer-executable instructions when executed further cause the system to automatically generate, the video using a fourth machine-learning model, wherein the fourth machine-learning model is a trained model for predicting one or more adjustments to be made upon the video, and wherein the fourth machine-learning model is trained based on one or more adjustments made upon videos manually by users. The color of the text is distinct from the one or more colors appearing in the region.

In still further aspects, the technology relates to a computer-implemented method for automatically generating a video in an e-commerce system. The method comprises receiving a set of images, wherein each of images in the set of images describes in part an item for transaction in an online shopping marketplace, wherein the set of images include a first image and a second image, and wherein the first image includes a first region depicting the item; determining, based on a category of the item, a sequence of the set of images using a machine-learning model, wherein the sequence includes the first image preceding the second image; automatically generating the video, wherein the video includes a first video segment with the first image proceeding a second video segment with the second image; determining a second region in the first video segment, wherein the second region is without an overlap with the first region; inserting text data into the second region; and publishing the video in the online shopping marketplace for viewing. The text data corresponds to a message associated with promoting the item in the online-shopping marketplace. The machine-learning model is trained for predicting one or more visual transitions in the video that describes the item based on the category of the item. The determining the second region in the first video segment uses another machine-learning model for predicting the second region and textual attributes of the text data for insertion based on a combination of a feature of an image in the first video segment and the text data.

Claim 1:
A computer-implemented method for automatically generating a video (<NUM>) for an item listing in an e-commerce system, the method comprising:
receiving (<NUM>) a set of images, wherein the set of images (<NUM>) comprises a plurality of images received as part of the item listing;
generating (<NUM>) a set of visual descriptors for the set of images, wherein generating the set of visual descriptors comprises computing a visual descriptor, using a first machine-learning model, for each image in the set of images;
comparing visual descriptors from the set of visual descriptors to identify one or more redundant images in the set of images;
generating (<NUM>), based on the compared visual descriptors, a unique set of images by removing the one or more redundant images from the set of images;
determining (<NUM>), based on the unique set of images, an ordered set of images using a second machine-learning model, wherein the ordered set of images is ordered based at least upon a category (<NUM>) of the item listing, wherein the second machine-learning model is trained on visual transitions for predicting a sequence of video segments of the video based on order criteria, and wherein the order criteria includes the category of the item listing;
automatically determining (<NUM>), using additional data from the item listing, text (<NUM>) associated with one or more images in the ordered set of images, wherein the text associated with the one or more images is automatically added to the one or more images;
automatically generating (<NUM>), based on the ordered set of images, the video (<NUM>), wherein the video (<NUM>) comprises the ordered set of images; and
providing the video (<NUM>) as part of the item listing.