Determining image handle locations

Systems and techniques are described for determining image handle locations. An image is provided to a neural network as input, and the neural network translates the input image to an output image that includes clusters of pixels against a background that have intensities greater than an intensity of the background and that indicate candidate handle locations. Intensities of clusters of pixels in an output image are compared to a threshold intensity level to determine a set of the clusters of pixels satisfying an intensity constraint. The threshold intensity level can be user-selectable, so that a user can control a density of handles. A handle location for each cluster of the set of clusters is determined from a centroid of each cluster. Handle locations include a coordinate for the handle location and an attribute classifying a degree of freedom for a handle at the handle location.

BACKGROUND

Images are often used to create animation sequences, such as by deforming an image to generate another image in an animation sequence. For images represented by vector graphics, such as curves, splines (e.g., piecewise polynomials), and the like, deforming the image involves editing basis points of curves of the vector graphics. These editing techniques are extremely time consuming, especially for images consisting of a large number of curves, and require a high level of expertise on behalf of the graphic designer.

Consequently, editing techniques (e.g., animation workflows) have been recently developed that do not deform an image by direct manipulation of a curve representation of the image, but instead deform an image based on handles affixed to an image. For instance, a user may select and drag a handle on an image to deform the image. As an example,FIG. 1illustrates example images100in accordance with one or more aspects of the disclosure. Images100includes image102and image104that both include an object, e.g., artwork. In the example inFIG. 1, image104has been generated by deforming the artwork of image102. For instance, image102depicts a person (e.g., the artwork of image102) having a plurality of handles inserted on the person, including handle106-1, handle106-2, handle106-3, handle106-4, handle106-5, and handle106-6(collectively handles106).

Image104is generated by selecting handle106-1on image102and dragging handle106-1towards the right (e.g., with a mouse). This dragging motion is indicated by arrow108. Since handle106-1is located on the person's head in image102, the person in image104is deformed according to the movement of handle106-1indicated by arrow108. In this case, the person in image104leans to one side with a tilted head based on the movement of handle106-1indicated by arrow108. To further illustrate the deformation caused by moving handle106-1in image102to generate image104, image102includes indicator110and image104includes indicator112. In image110, indicator110is horizontal, while in image104, based on the movement of handle106-1illustrated by arrow108, indicator112in image104is moved from horizontal and depicts an angle of tilt of the person's head in image104with respect to horizontal.

The locations of handles on an image can have significant impact on the quality of images generated by deforming the image according to the handles on the image. For instance, an animation sequence generated from poorly-placed handles on an image usually looks unrealistic. Unfortunately, placing handles at locations on an image to generate a realistic animation sequence from the image usually requires experience levels beyond all but highly-trained experts. As a result, users often repeat steps in an animation process, such as by trying different handle locations, until an acceptable animation sequence is obtained. This process is both frustrating for the user and time-consuming for the user, and often produces poor animation results despite significant user effort.

SUMMARY

Techniques and systems are described to determine handle locations on an image. An output image is generated from a representation of an input image, such as a rasterized version of an input image or a mesh of an object in an input image (e.g., a mesh of primitives, such as a triangle mesh). The output image includes clusters of pixels against a background, such as a uniform black background. The clusters of pixels have intensities greater than an intensity of the background and indicate candidate handle locations. The output image is generated with a neural network that is trained with input images having expert-specified handle locations and output images generated by convolving Gaussian filters with white patches at the expert-specified handle locations on a black background. In one example, the neural network is a generative adversarial network. A set of the clusters of pixels having respective intensity levels above a threshold intensity level is determined for the output image, and a respective handle location for each cluster of the set of clusters is found from the centroid of each cluster. The threshold intensity level can be user-selectable, and used to control the number of handle locations generated for an image. Handle locations can include a coordinate for the respective handle location and an attribute classifying a degree of freedom for a handle at the respective handle location, such as rigid, flexible, and anchor. Furthermore, a user-selection may indicate a desired density of handle locations to be determined for an object in an image, such as coarse, medium, and fine, and a neural network may be selected from a plurality of available neural networks based on the desired density, such as neural networks that have been trained with different densities of handle locations.

DETAILED DESCRIPTION

Overview

Creating animation sequences often relies on deforming an object in an image such as artwork of an image. To deform an object in an image, some designers manually edit basis points of vector graphics constructs (e.g., curves) representing artwork. Additionally or alternatively, designers may manually assign control points such as handle locations to artwork (e.g., an object in an image), and deform the artwork by translating or rotating a handle at an assigned handle location. However, these methods are slow and often require multiple iterations of editing tasks by the user with no guarantee that a realistic animation sequence is produced. For instance, a user may place a handle at a location on artwork of an image and deform the artwork according to the handle, revert the deformation because the user is not satisfied, move the handle, and again deform the artwork. The user may repeat these steps multiple times to generate an animation sequence. Hence, these methods are both frustrating and time-consuming for the user, and often produce poor animation results despite significant user effort.

Accordingly, this disclosure describes systems and techniques for determining image handle locations from a representation of an input image, such as a rasterized version of an input image, a mesh of primitives for an input image (e.g., a triangle mesh of artwork of an image), and the like. A representation of an input image is translated to an output image. In one example, a rasterized image in greyscale is provided to a neural network as an input image, and the neural network produces an output image based on the rasterized image. Additionally or alternatively, a mesh for an image can be provided to a neural network as input, and the neural network can produce an output image based on the mesh.

An output image generated by translating an input image includes clusters of pixels against a background, such as a uniform background (e.g., black). The clusters of pixels have intensities greater than an intensity of the background and indicate candidate handle locations. For instance, a handle location may be determined within a cluster of pixels. Intensities of clusters of pixels in an output image are compared to a threshold intensity level to determine a set of the clusters of pixels satisfying an intensity constraint. In one example, an intensity constraint includes determining a set of the clusters of pixels having respective intensity levels above the threshold intensity level. For instance, a respective intensity level for each cluster of pixels in an output image may be determined from a respective average intensity for each cluster of pixels, and compared to a threshold intensity level to determine whether a cluster of pixels is included in a set of the clusters of pixels satisfying an intensity constraint.

A handle location for each cluster of the set of clusters is determined. In one example, a handle location for each cluster of the set of clusters is found from a centroid (e.g., a center of mass) of each cluster. Handle locations include a coordinate for a respective handle location, such as an x-y coordinate that identifies a location on an object in an image (e.g., a rasterized image). In one example, handle locations include an attribute classifying a degree of freedom for a handle at a respective handle location. For instance, a handle location for a person's shoulder may include an attribute indicating the shoulder is more rigid, and less flexible, than an attribute of a handle location for a person's hand. A handle location may include an attribute identifying a handle as an anchor. A handle identified as an anchor may remain at a fixed position so that the artwork can be deformed by rotating the artwork about the position, rather than moving the anchor from the position. For instance, a handle location for a person's foot may be designated as an anchor so that the person's foot remains fixed while the artwork of the person is deformed.

Furthermore, a user interface is generated that exposes user-selectable options to control handle locations determined for images. In one example, a user interface exposes a control (e.g., a slider control in the user interface) to adjust a threshold intensity level used to determine a set of clusters in an output image satisfying an intensity constraint. By varying the threshold intensity level, the number of clusters in the set of clusters that have respective intensities satisfying the intensity constraint (e.g., greater than the threshold intensity level) is varied. Accordingly, a user may control the number of handles and handle locations determined for an object in an image by setting a threshold intensity.

Additionally or alternatively, a user interface can include options for a desired density of handles and handle locations, such as menu options for coarse, medium, and fine densities of handles. A user-selection may indicate a desired density of handle locations to be determined for an object in an image, and a neural network may be selected based on the user-selection from a plurality of available neural networks, such as neural networks that have been trained with different densities of handle locations (e.g., training images containing different densities of handle locations).

A neural network that translates a representation of an input image to generate an output image including clusters of pixels indicating candidate handle locations can be any suitable neural network. In one example, a neural network is trained with input images having expert-specified handle locations (e.g., locations specified by trained experts in placing handles on an object of an image, such as trained graphic designers) and output images generated by convolving Gaussian filters with white patches at the expert-specified handle locations on a black background. Different training sets representing different densities of handle locations can be used to train different neural networks that are user-selectable via a user interface, such as based on a user-selection indicating a desired density of handle locations. A neural network can be trained with a loss function including a pixel loss term, an adversarial loss term, and a perceptual loss term. In one example, a neural network that generates an output image including clusters of pixels from a representation of an input image is a conditional generative adversarial network that includes a generator trained to produce output images that cannot be distinguished from “real” images by an adversarially trained discriminator that is trained to detect the generator's “fake” images.

Hence, handle locations for an image are determined quickly and reliably based on a neural network that is trained with handle locations that have been determined for images by experts. Accordingly, even novice users can generate handle locations for artwork of an image at a desired density of handles that can be used to deform the artwork and produce a realistic animation sequence, without user frustration and without wasted efforts caused by repeating steps of the animation process until a desired result is achieved.

Furthermore, the inventors have determined that using skeletal extraction techniques, in which a surface flow degenerates to a skeleton approximating a medial axis, may be used to determine image handle locations, such as by placing handles on the medial axis (e.g., at joints of the medial axis).

In the following discussion an example digital medium environment is described that may employ the techniques described herein. Example implementation details and procedures are then described which may be performed in the example digital medium environment as well as other environments. Consequently, performance of the example procedures is not limited to the example environment and the example environment is not limited to performance of the example procedures.

Example Digital Medium Environment

FIG. 2is an illustration of a digital medium environment200in an example implementation that is operable to employ techniques described herein. As used herein, the term “digital medium environment” refers to the various computing devices and resources that can be utilized to implement the techniques described herein. The illustrated digital medium environment200includes a user202having at least one computing device. In the example inFIG. 2, user202is illustrated as having three computing devices, computing devices204-1,204-2, and204-3(collectively204). For instance, computing device204-1depicts a desktop computer, computing device204-2depicts a tablet or smart phone, and computing device204-3depicts a pair of eye glasses (e.g., smart goggles). Computing devices204are example computing devices, and any suitable computing device is contemplated, such as a mobile phone, tablet, laptop computer, desktop computer, gaming device, goggles, glasses, camera, digital assistant, echo device, image editor, non-linear editor, digital audio workstation, copier, scanner, and the like. Furthermore, discussion of one of computing devices204is not limited to that computing device, but generally applies to each of the computing devices204. Moreover, computing devices204may range from full resource devices with substantial memory and processor resources (e.g., personal computers, game consoles) to a low-resource device with limited memory or processing resources (e.g., mobile devices).

In one example, computing devices204include a wearable device that is designed to be worn by, attached to, carried by, or otherwise transported by user202. Examples of wearable devices include glasses, a smart band or watch, and a pod device such as clip-on fitness device, media player, or tracker. Other examples of a wearable device include but are not limited to a badge, a key fob, an access card, and a ring, an article of clothing, a glove, and a bracelet.

Various types of input devices and input instrumentalities can be used to provide input to computing devices204. For example, computing devices204can recognize input as being a mouse input, stylus input, touch input, input provided through a natural user interface, and the like. Thus, computing devices204can recognize multiple types of gestures including touch gestures and gestures provided through a natural user interface. In one example, computing devices204include speech recognition, identification, and synthesis functionalities, microphones, and speakers that allow computing devices204to communicate with user202in a conversation, e.g., a user conversation.

Furthermore, computing devices204may be representative of one or a plurality of different devices, such as one or more devices connected to a network that perform operations “over the cloud” as further described in relation toFIG. 12. In one example, computing devices204are communicatively coupled to each other, such as with a low power wireless communication standard (e.g., a Bluetooth® protocol). For instance, computing device204-1can communicate wirelessly with computing device204-2and computing device204-3. Hence, an asset (e.g., image, video, text, drawing, document, file, and the like) generated, processed (e.g., edited), or stored on one device (e.g., computing device204-1) can be communicated to, and displayed and processed on another device (e.g., computing device204-3).

In the example illustrated inFIG. 2, computing device204-1obtains image206. Image206is an example of an asset, and can be obtained in any suitable way, such as from another computing device, from file storage on computing device204-1, and the like. In one example, image206includes a rasterized image. Additionally or alternatively, image206can be represented by curves, such as n-th order polynomial splines, (e.g., n=1, 2, 3, or 4), Bezier segments, combinations thereof, and the like. In one example, image206is represented by a mesh of primitives (e.g., a triangle mesh of artwork of an image), as described in U.S. patent application Ser. No. 15/861,908 entitled Generating A Triangle Mesh For An Image Represented By Curves to Batra et al., filed Jan. 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

Image206in the example illustrated inFIG. 2includes an object, e.g., artwork depicting an alligator. User202provides image206to image handle system208, which generates image210. Image210denotes various handles212at locations on the artwork depicting the alligator. For instance, image handle system208generates handle locations for image206, and inserts handles212on the artwork of the alligator at the handle locations to produce image210. Handles212can be denoted by any suitable indicator. In the example inFIG. 2, handles212are denoted with black circles enclosed by white rings. In one example, handles212are represented by designators that illustrate an attribute of a respective handle, such as circles for flexible handles, squares for rigid handles, and triangles for handles that are anchor points. Accordingly, image210can be deformed, such as by moving one or more of handles212, to generate an animation sequence (e.g., depicting the alligator walking).

Computing devices204are also coupled to network214. Network214communicatively couples computing devices204with server216(for clarity, only computing device204-1is illustrated inFIG. 2as coupled to network214, though computing devices204-2and204-3can also be coupled to server216via network214). Network214may include a variety of networks, such as the Internet, an intranet, local area network (LAN), wide area network (WAN), personal area network (PAN), cellular networks, terrestrial networks, satellite networks, combinations of networks, and the like, and as such may be wired, wireless, or a combination thereof.

Server216may include one or more servers or service providers that provide services and/or resources to computing devices204. Generally, resources provided by server216may be licensed, purchased, or may be made freely available, (e.g., without authentication, license, or account-based access). The resources can include any suitable combination of services and content, such as made available over network214by one or more providers. Some examples of services include, but are not limited to, an on-line shopping service, a photo editing service, a web development and management service, a collaboration service, a social networking service, a messaging service, an advertisement service, a graphics design service, an animation service, an image storage service (including storage of photos, documents, records, files, and the like), a graphics editing service, an asset distribution service, and so forth. Content may include various combinations of assets, including videos, ads, audio, multi-media streams, animations, images, web documents, web pages, applications, device applications, text documents, drawings, presentations, stock photographs, user profiles, user preferences, user data (e.g., images stored in an image gallery), maps, computer code, and the like. Assets may be made available to image handle system208, image handle support system218, or combinations thereof, and stored at assets220of server216. Hence, image206can include any suitable asset stored at assets220of server216.

Furthermore, server216includes image handle support system218configurable to receive signals from computing devices204, process the received signals, and send the processed signals to computing devices204to support determining image handle locations. For instance, computing device204-1may obtain any suitable representation of an image, such as a rasterized image, vector-graphics curve representation, triangle mesh and the like, and communicate any suitable data (e.g., a rasterized version of image206, user-selections, such as indicating a desired density of handles, a threshold intensity level, and the like) to server216. Server216, using image handle support system218, may calculate handle locations and attributes of handles from the data received from computing device204-1. Server216may then provide handle locations and attributes of handles back to computing device204-1, which can display designators for the handles based on the attributes on an image, such as image210, at locations corresponding to the handle locations. Accordingly, image handle support system218of server216can include a copy of image handle system208, including image handle application242(discussed below in more detail).

Computing devices204include image handle system208to determine image handle locations and attributes of the handles. For clarity, computing device204-3is illustrated inFIG. 2as including image handle system208, though computing device204-1and computing device204-2also include copies of image handle system208(not shown).

Image handle system208includes a display222. Display222can expose any suitable data used by or associated with image handle system208. In one example, display222displays a user interface for exposing assets, images (e.g., rasterized images, images represented by vector graphics, output images generated by a neural network, and the like), triangle meshes, handles (e.g., designators of handles that distinguish between attributes of the handles that describe a degree of freedom of the handle, such as rigid, flexible, and the like), animation sequences, user-selectable control options, such as a mechanism to select a threshold intensity level, e.g., a slider control, menu options for desired densities of handles and handle locations, such as coarse, medium, and fine, combinations thereof, and the like. Display222can expose a user interface configurable to edit an image, such as by deforming a mesh.

Display222can include any suitable type of display, such as a touchscreen, liquid crystal display, plasma display, head-mounted display, projector and screen, and the like. A touchscreen of display222can include any suitable type of touchscreen, such as a capacitive touchscreen, a resistive touchscreen, a surface acoustic wave touchscreen, an infrared touchscreen, an optical imaging touchscreen, an acoustic pulse recognition touchscreen, combinations thereof, and the like.

Image handle system208also includes processors224. Hence, image handle system208may be implemented at least partially by executing instructions stored on storage226on processors224. For instance, processors224may execute portions of image handle application242.

Storage226can be any suitable type of storage accessible by or contained in image handle system208. Storage226stores and provides access to and from memory included in storage226for any suitable type of data. For instance, storage226includes image data228, such as a rasterized image (e.g., a bitmap, pixel data, or combinations thereof), curves of an image, graphics of the image generated according to the curves (e.g., adding color), metadata of an image, such as data governing usage rights of the image, a source location of the image, date an image was generated, etc., a thumbnail version of an image, a copy of an image, a mesh representation of an image, an output image from a neural network including clusters of pixels against a background to indicate candidate handle locations, an identification number of an image, such as a number for an image in an animation sequence, a number to locate an image in a database of images, and the like.

Storage226also includes neural network data230, such as training data (e.g., pairs of input images and output images, alpha masks, etc.), neural networks (e.g., a plurality of neural networks that have been trained with training sets corresponding to different densities of handles and handle locations), indicators of a loss of a neural network (e.g., a loss measurement of a neural network over a training set, a loss measurement for an output image generated by a neural network, and the like), weighting parameters of a loss function (e.g., respective weights of a pixel loss term, an adversarial loss term, and a perceptual loss term), encoder and decoder parameters (e.g., filter sizes and numbers of filters), normalization parameters, activation functions, indicators of skip connections, and the like.

Storage226also includes cluster data232, such as data regarding clusters of pixels in an output image generated by a neural network, including numbers of clusters, locations of clusters (e.g., regions, quadrants and the like of an output image that include clusters), respective intensities of clusters (e.g., measurements of an average intensity of a cluster), threshold intensity levels, sets of clusters of pixels satisfying an intensity constraint (e.g., having respective intensity levels greater than a threshold intensity level), user preferences of threshold intensity levels, combinations thereof, and the like

Storage226also includes handle data234, such as data regarding handles of an object (e.g., artwork) of an image, including a number of handles, handle locations (e.g., coordinates on an image or an object that locate the handle on the object, such as Cartesian coordinates or polar coordinates, a vertice number on a mesh on which a handle is located, combinations thereof, and the like), attributes of handles (e.g., attributes describing degrees of freedom of handles at handle locations, such as rigid to indicate partial, limited movement, flexible to indicate full movement, and anchor to indicate no movement, a rotation attribute to indicate that artwork may be deformed by rotating the artwork about a handle designated with a rotation attribute, and the like), an indicator of whether a handle was manually placed (e.g., by a user) or automatically placed (e.g., by image handle system208), combinations thereof, and the like.

Storage226also includes user interface data236, including data associated with user interfaces, such as user preferences (e.g., font size and style, locations and sizes of panels presented in a user interface, indicators of neural networks used by, or preferred by users, and the like), data of users operating a user interface (e.g., user histories of edits including user-selections of threshold intensity levels, densities of handles, and the like, user interface configurations (e.g., different layouts, language options, etc.), controls and adjusters (e.g., sliders, lists of user gestures to control a user interface, etc.), options for handle indicators, such as circles, rings, squares, triangles, colors, and the like used to indicate a handle, attribute of a handle, or combinations thereof, user interface version numbers, lists of operating systems supported by various user interfaces, thumbnail images of images to display in a user interface, toolbar icons, speech libraries for voice-actuated commands and gestures, and the like.

Furthermore, image handle system208includes transceiver module238. Transceiver module238is representative of functionality configured to transmit and receive data using any suitable type and number of communication protocols. For instance, data within image handle system208may be transmitted to server216with transceiver module238. Furthermore, data can be received from server216with transceiver module238. Transceiver module238can also transmit and receive data between computing devices204. In one example, transceiver module238includes a low power wireless communication standard (e.g., a Bluetooth® protocol) for communicating data between computing devices204.

Image handle system208also includes image gallery module240. Image gallery module240is representative of functionality configured to obtain and manage images of image handle system208, such as images that can have handles assigned to objects of the image, images in an animation sequence, thumbnail representations displayed in a user interface (e.g., thumbnail images of images in an animation sequence exposed in a user interface), images including a mesh, a rasterized image, images represented by vector graphics, and the like. Hence, image gallery module240may use transceiver module238to obtain any suitable data from any suitable source, including obtaining digital images from a user's directory of files on computing devices204or server216, obtaining images from a user's photo gallery (e.g., an online photo sharing service, images stored in a user's image editing application, such as Photoshop®), images a user has posted in a social media post, blog, online comment, and the like, images a user has attached to an email, text, or other communication sent from or received by computing devices204, images provided by a search service, such as an online search for digital images related to a search query, images obtained from a database of stock images, images provided by user202, images captured by a computing device, such as with a camera integrated into one of computing devices204, and the like. Images obtained by image gallery module240are stored in image data228of storage226and made available to modules of image handle application242.

Image handle system208also includes image handle application242. Image handle application242includes image module244, user interface module246, neural network module248, cluster module250, and handle module252. These modules work in conjunction with each other to determine handle locations for artwork of an image and attributes of handles at the handle locations. Handles placed at the handle locations on artwork of an image can be used to deform the artwork and create an animation sequence.

Furthermore, though the description of image handle system208and image handle application242describes determining handle locations for a representation of an image (e.g., a rasterized image or mesh of artwork of an image) of an object, such as artwork, image handle system208and image handle application242can be used to determine handle locations for any suitable asset, such as a document, web page, map, slide, presentation, and the like.

Image module244is representative of functionality configured to obtain a representation of an image of an object, such as a rasterized image, a mesh of artwork of an image, vector graphics of artwork of an image, combinations thereof, and the like. Image module244can obtain any suitable representation of an image in any suitable way. In one example, image module244obtains an image from a database of images, such as a gallery maintained by image gallery module240or a database maintained by server216in assets220. Additionally or alternatively, image module244can obtain an image from storage226that has been reconstructed from a deformed image or deformed mesh. In one example, image module244obtains a mesh of artwork of an image, such as a mesh generated as part of an animation sequence.

An image obtained by image module244can be any suitable type of image, such as a stand-alone image (e.g., an image not associated with other images), an image in a sequence of images (e.g., an animation sequence, a video, a page in a chapter of a book, a slide in a slideshow, and the like), or combinations thereof. In one example, an image obtained by image module244is extracted from an asset that contains other types of media than images, such as a web page containing images and text.

Furthermore, image module244can obtain an image represented by curves, such as a spline including piecewise segments of Bezier curves, polynomials of any suitable order (e.g., quadratic, cubic, quartic, etc.), cubic splines, lines, primitive shapes such as squares, rectangles, triangles, circles, ellipses, polygons, combinations thereof, and the like. In one example, image module244converts an image represented by curves to a rasterized image in greyscale by rasterizing the image represented by curves, and image handle system208determines handle locations for the rasterized image generated by image module244.

A rasterized image generated by image module244can be any suitable type of rasterized image, such as a bit map, pixel values, dot matrix data structure, combinations thereof, and the like. In one example, a rasterized image includes a grayscale image with a transparency parameter (e.g., alpha channel) to represent transparency of pixels in the image with a percentage of the transparency parameter. Furthermore, a rasterized image generated by image module244can include any suitable number of raster elements (e.g., pixels) whose values are represented by any suitable type of data, such as a number of bits, values in a coordinate system (e.g., a color coordinate system), combinations thereof, and the like. Moreover, image module244can rasterize an image in any suitable way, such as based on user-specified parameters (e.g., a user-designated resolution in terms of numbers of pixels), based on analyzing an image (e.g., for spectral content) and determining a resolution based on results of the analyzing (e.g., using a higher number of pixels for images with higher spectral content than images with lower spectral content), according to a default resolution, and the like.

A representation of an image obtained by image module244, along with any suitable information, such as a source location of an image, a file format of an image, an indication whether the image is related to other images, such as a sequence number in an animation sequence, image metadata (e.g., information regarding a mesh of an image, curves representing an image, etc.), a rasterized version of an image, and the like, used by or calculated by image module244are stored in image data228of storage226and made available to modules of image handle application242. In one example, image module244provides an image to neural network module248and user interface module246.

User interface module246is representative of functionality configured to generate, manage, and cause display on any suitable user interface, such as a user interface including a digital image and indicators of handles for the digital image. A user interface of user interface module246can expose any suitable data, such as an input image (e.g., a rasterized image), a mesh of artwork of an image, an animation sequence of images, a deformed image (e.g., an image formed by moving a handle on an image to deform the image), an output image generated by a neural network, such as an output image including clusters of pixels against a background (e.g., clusters of bright pixels indicating candidate handle locations against a uniform black background), training images, alpha masks, combinations thereof, and the like.

A user interface of user interface module246can expose any suitable control options, such as options for selecting images, including lists of images and thumbnail representations of images, options for selecting a threshold intensity level, options for selecting a desired density of handles and handle locations for artwork of an image, options for selecting a neural network from a plurality of neural networks (e.g., a list of neural networks with a description of the training sets used to train the neural networks), options to move a handle (e.g., a button to enable a handle at a selected location to be relocated to another location without deforming artwork of the image), combinations thereof, and the like.

A user interface of user interface module246can receive user-selections of any suitable control option exposed in the user interface. In one example, a user interface of user interface module246receives a user-selection indicating a density of handles for an object in an image, such as a selection of course, medium, or fine densities. Additionally or alternatively, a user interface of user interface module246can receive a user-selection of a threshold intensity level for comparison against an intensity level of a cluster of pixels. For instance, a user may adjustment a slider control exposed in a user interface of user interface module246to select a threshold intensity level.

In one example, a user interface of user interface module246exposes thumbnail representations of images, such as images obtained by image module244. A user can select a thumbnail representation of an image and cause the selected image to be processed by image handle system208, so that handle locations and attributes of handles at the handle locations for the selected image are exposed in a user interface of user interface module246.

A user interface generated by user interface module246, along with any suitable information, such as configurations settings of the user interface, user gestures, thumbnail images, user preferences, such as preferred locations of digital images exposed in a user interface, and the like, used by or calculated by user interface module246are stored in user interface data236of storage226and made available to modules of image handle application242. In one example, a user interface generated by user interface module246is displayed by display222, and user interface module communicates a selected threshold intensity level to cluster module250and a selected density of handles to neural network module248.

Neural network module248is representative of functionality configured to generate an output image by applying a representation of an image to a neural network. An output image generated by neural network module248includes clusters of pixels, such as clusters of pixels indicating candidate handle locations against a uniform background. For instance, clusters of pixels can have intensities greater than an intensity of the background.

Neural network module248can include any suitable type and number of neural networks. A neural network of neural network module248can be trained in any suitable way. In one example, a neural network of neural network module248is trained with input images having user-specified handle locations used to determine ground-truth images. An input image may be a rasterized image (e.g., a bit map in greyscale), a mesh of an object in an image (e.g., a triangle mesh), combinations thereof, and the like. A neural network of neural network module248can be trained with output images (e.g., ground-truth images) generated by convolving Gaussian filters with white patches at user-specified handle locations on a black background. The user-specified handle locations correspond to expert-specified handle locations, such as locations determined by trained experts in generating animation sequences, as opposed to novice users. In one example, a neural network of neural network module248is trained with a loss function including a pixel loss term, an adversarial loss term, and a perceptual loss term, such as by adjusting parameters of the neural network to minimize the loss function.

Additionally or alternatively, neural network module248includes a plurality of neural networks, such as pre-trained neural networks that have been trained with different training sets (e.g., different input images, different output images, or both different input images and different output images). In one example, neural network module248includes a plurality of neural networks that have been trained with different densities of handles and handle locations, such as coarse, medium, and fine. For instance, a fine training set (e.g., input image and output image pairs) may be generated at a fine resolution with input images having user-specified handle locations and output images generated by convolving Gaussian filters with white patches at the user-specified handle locations on a black background. Coarse and medium training sets can be generated by decimating (e.g., removing) handle locations from images of a fine training set, such as by removing one third of handle locations from images of a fine training set to generate a medium training set and removing one half of handle locations from images of a fine training set to generate a coarse training set. Fine, medium, and coarse training sets can be used to train neural networks according to different densities of handles. Accordingly, a user may select one of a plurality of densities of handle locations, one of a plurality of neural networks, or combinations thereof, to generate a desired density of handle locations for object or artwork of an image.

Neural network module248can include any suitable number of neural networks corresponding to any suitable density of handle locations. Coarse, medium, and fine densities described above are examples of different densities. Additionally or alternatively, neural network module248can include N neural networks for some integer N, corresponding to N different densities of handle locations, and a user may select a value of N (e.g., a number from one to ten), such as via a user interface of user interface module246, to select one of N neural networks of neural network module248.

In one example, a neural network of neural network module248generates attributes for each cluster of pixels in an output image generated by neural network module248. Hence, a neural network of neural network module248generates attributes for a handle of a candidate handle location indicated by clusters of pixels in an output image generated by neural network module248. Neural network module248can generate any suitable attribute for a handle at a handle location determined by image handle system208. In one example, attributes of handles generated by neural network module248describe degrees of freedom of a respective handle, such as rigid, flexible, and anchor. For instance, a handle location for a person's shoulder may include an attribute indicating the shoulder is more rigid, and less flexible, than an attribute of a handle location for a person's hand. Additionally or alternatively, a handle location may include an attribute identifying a handle as an anchor. A handle identified as an anchor may remain at a fixed position when the artwork is deformed.

A neural network of neural network module248can generate attributes of handles in any suitable way. In one example, neural network module248includes a plurality of neural networks that have been trained with different data sets that include different attributes of handles, such as expert-specified attributes of handles. For instance, expert-users (e.g. trained graphic designers) may specify a degree of freedom of a handle in a training set, such as rigid, flexible, or anchor, or a numerical value (e.g., one to ten) indicating a degree of freedom. A numerical value of one may represent a handle location corresponding to an anchor handle, and a numerical value of ten may represent a handle location corresponding to a fully-flexible handle, with numerical values between one and ten denoting handle locations corresponding to handles with degrees of freedom proportional to respective numerical values between one and ten.

A neural network of neural network module248can be any suitable neural network. In one example, a neural network of neural network module248includes a conditional generative adversarial network having a generator with skip connections (described below in more detail with regards toFIG. 7andFIG. 8). A generative adversarial network includes a generator trained to produce output images that cannot be distinguished from “real” images by an adversarially trained discriminator that is trained to detect the generator's “fake” images. Skip connections concatenate activations from a layer of an encoder of the generator to a layer of a decoder of the generator so that all information of a generator does not need to be processed by all layers of the generator, saving time and processing resources when operating a neural network of neural network module248.

A neural network of neural network module248can be trained according to any suitable objective function. In one example, a neural network of neural network module248is trained with a loss function including a pixel loss term, an adversarial loss term, and a perceptual loss term. For instance, a loss function used to train a neural network of neural network module248can be expressed as
=λ1·Lpix+λ2·Ladv+λ3·Lper
where Lpixdenotes a pixel loss term, Ladvdenotes an adversarial loss term, Lperdenotes a perceptual loss term, and λi, i=1, 2, 3 are real-valued weights, such as numbers between zero and one.

In one example, a pixel loss term is determined from a distance between an output image generated by a neural network of neural network module248(e.g., by a generator of a generative adversarial network) and a ground-truth image (e.g., an output image of a pair of training images used to train a neural network), such as
Lpix=E{∥ŷi−G(xi)∥}
where E{·} denotes statistical expectation, ∥·∥ denotes any suitable norm, such as l1, l2, and the like, ŷ denotes a ground-truth image, and G(xi) denotes an output image produced by generator G(·) for input image xi.

An adversarial loss term can be determined from a generator G(·) and discriminator D(·) of a generative adversarial network of neural network module248. Generator G(·) and discriminator D(·) are jointly trained so that the discriminator tries to distinguish between images generated by the generator and ground-truth images, while the generator tries to fool the discriminator into thinking its generated output image is real. In one example, an adversarial loss term is expressed as
Ladv=ΣilogD(G(xi)).

A perceptual loss term can be any suitable measure of perceptual loss (e.g., loss based on user perception). In one example, a perceptual loss term is determined by extracting features from images and determining a difference in a feature space of the features. For instance, features can be extracted from an output image generated by a neural network of neural network module248and from a ground-truth image using any suitable feature extractor, such as a pre-trained neural network (e.g., a visual geometry group (VGG) convolutional neural network). A perceptual loss term can be determined from the difference between features of the output image and the ground-truth image. Accordingly, a perceptual loss term can be determined from
Lpix=E{∥[ŷi]−[G(xi)]∥}
where[·] denotes features extracted from an image.

A neural network of neural network module248, along with any suitable information, such as training data (e.g., pairs of images including an input image and an output (or ground-truth) image), a database of neural networks (e.g., a structured database organizing pre-trained neural networks in a hierarchy including an input-image level (e.g., different input-image levels in the hierarchy may correspond to different types of input images, such as a rasterized image and a triangle mesh), a handle-density level (e.g., different handle-density levels in the hierarchy may correspond to different densities of handles, such as coarse, medium, and fine), and the like), data used to generate training images (e.g., training images may be perturbed by translation, rotation, and scaling to expand a training set to include additional images, and data regarding the translation, rotation, and scaling, such as a maximum amount of translation, rotation, or scaling may be included the data used to generate training images), and the like, used by or calculated by neural network module248are stored in neural network data230of storage226and made available to modules of image handle application242. In one example, neural network module248communicates an output image generated by neural network module248to cluster module250.

Cluster module250is representative of functionality configured to determine a set of clusters of pixels in an output image generated by neural network module248. In one example, cluster module250determines a set of clusters of pixels having a respective intensity level above a threshold intensity level, such as a user-specified threshold intensity level. A user-specified threshold intensity level can control a number of clusters in a set of clusters of pixels, and hence control a number of handle locations that are determined for an object (e.g., artwork) in an image.

Cluster module250can determine a set of clusters of pixels in an output image in any suitable way. In one example, cluster module250determines a set of clusters of pixels in an output image by identifying first clusters of pixels satisfying a cluster constraint, such as clusters of pixels including a minimum number of pixels grouped together and having at least a minimum intensity level (e.g., an average value of intensity for a cluster of pixels is equal to or greater than a minimum intensity level). Cluster module250can then determine a set of clusters of pixels (e.g., a subset of all clusters) by applying an intensity constraint to the first clusters of pixels. For instance, cluster module250includes only those clusters having an intensity level above a threshold intensity level (e.g., a user-specified intensity level) in a set of clusters of pixels determined by cluster module250.

Cluster module250can determine any suitable representation of a set of clusters of pixels, such as a list including locations of pixels in clusters of a set of clusters, a mask of an image that identifies clusters of a set of clusters, a bit-map that identifies clusters of a set of clusters, combinations thereof, and the like.

A set of clusters of pixels of an output image determined by cluster module250, along with any suitable information, such as a representation of a set of clusters of pixels (e.g., a list of clusters, a mask of an image depicting clusters, etc.), intensity levels of clusters (e.g., an average intensity level of a cluster of pixels generated by cluster module250and used to compare to a threshold intensity level), a threshold intensity level used to generate a set of clusters of pixels, a number of clusters in a set of clusters of pixels, a number of clusters not satisfying a threshold intensity constraint and not included in a set of clusters of pixels determined by cluster module250, a size of a cluster in a set of clusters of pixels (e.g., a surface area in number of pixels), locations of clusters of pixels, and the like, used by or calculated by cluster module250are stored in cluster data232of storage226and made available to modules of image handle application242. In one example, cluster module250provides a set of clusters of pixels (e.g., a representation of clusters of pixels, such as a list or mask) to handle module252.

Handle module252is representative of functionality configured to determine a respective handle location for each cluster of a set of clusters from cluster module250. In one example, handle module252determines a respective handle location of an object in an image for each cluster of a set of clusters from a respective centroid of each cluster. A centroid can include any suitable centroid calculation, such as a center-of-mass, barycentric coordinates, a weighted center-of-mass (e.g., with weights assigned to pixels, such as in proportion to an intensity of a pixel), a centroid calculated for some, but not all pixels of a cluster (e.g., pixels with intensities below a specified intensity or beyond a threshold distance from a location in a cluster, such as a center of a circle encompassing a specified percentage of the pixels of a cluster, may be omitted from the centroid calculation), combinations thereof, and the like.

In one example, a handle location determined by handle module252includes a coordinate for the handle location, such as a Cartesian coordinate, polar coordinate, and the like. A coordinate of a handle location locates the handle location on an image (e.g., an object or artwork in an image). For instance, handle module252may generate a table of handle locations, including an entry for each handle location populated with a coordinate locating the handle location.

Additionally or alternatively, a handle location determined by handle module252can include an attribute classifying a degree of freedom for the handle location (e.g., a degree of freedom such as rigid, flexible, or anchor for a handle at the handle location). Handle module252may generate a table of handle locations, including an entry for each handle location populated with an attribute classifying a degree of freedom for a handle of the handle location.

In one example, handle locations determined by handle module252are exposed in a user interface of user interface module246. For instance, a handle can be displayed with a designator at a coordinate on an image for each handle location determined by handle module252. A designator of a handle can be any suitable designator. In the example inFIG. 2, handles212are denoted with black circles surrounded by a white ring. In one example, a designator of a handle at a handle location indicates an attribute of the handle at the handle location. For instance, different designators can be used to visually distinguish between handles of different attribute types, such as different color designators, different shapes, numerical values (e.g., a value of a degree of freedom of a handle from one to ten), combinations thereof, and the like.

In one example, emoticons are used to indicate handles at handles locations on an object of an image, and different emoticons represent different attributes of handles. For instance, an emoticon conveying fast movement, such as a road runner, sports car, jet airplane, and the like can be used to indicate a handle with full movement (e.g., flexible or having a value of ten on a scale of one to ten). An emoticon conveying slow movement, such as a turtle or snail can be used to indicate a handle with limited movement (e.g., rigid with some, but not full movement, such as having a value of three on a scale of one to ten), and an emoticon conveying no movement, such as a stop sign, can be used to indicate an anchor, such as having a value of zero on a scale of one to ten.

An example of computer code that can be used by cluster module250, handle module252, or both cluster module250and handle module252to determine a set of clusters of pixels in an output image and a respective handle location for each cluster of a set of clusters is described below in Table 1.

In the example computer code of Table 1, a threshold intensity level is denoted by the variable “level”. For each cluster of pixels having an intensity greater than a value of “level”, handle locations are determined with Cartesian coordinates.

Handle locations determined by handle module252, along with any suitable information, such as coordinates of handle locations, attributes of handles, a centroid algorithm used to determine a handle location (e.g., center-of-mass, weighted center-of-mass, and the like), thresholds used to include or exclude pixels of a cluster in a centroid calculation, an indication of a density of handle locations, such as coarse, medium, or fine, and the like, used by or calculated by handle module252are stored in handle data234of storage226and made available to modules of image handle application242. In one example, handle module252provides handle locations including coordinates and attributes of handles to user interface module246, which generates a user interface that exposes the handles at the handle locations with designators indicating the attributes.

Having considered an example digital medium environment, consider now a discussion of example images in accordance with one or more aspects of the disclosure.

Example Images

FIG. 3illustrates example images300in accordance with one or more aspects of the disclosure. Images300include two pairs of images, and each pair of images includes an input image, such as an input image obtained by image module244inFIG. 2, and an output image generated by neural network module248. A first pair of images includes input image302and output image304, and a second pair of images includes input image306and output image308.

Input image302includes an object, artwork depicting a person (e.g., a woman). Input image306includes an object, artwork depicting a bird. Input image302and input image306are examples of rasterized images in greyscale. Additionally or alternatively, (not shown) an input image can include a mesh (e.g., a triangle mesh) of artwork of an input image.

Output image304is an example of an output image generated by a neural network of neural network module248when input image302is provided as input to the neural network. Output image304includes clusters of pixels having a bright intensity against a black, uniform background. Thus, output image304includes clusters of pixels having intensities greater than an intensity of the background of output image304. Clusters of pixels in output image304indicate candidate handle locations. For instance, a handle location for the artwork of input image302may be determined for each of the clusters of pixels in output image304. Accordingly, clusters of pixels in output image304are in the shape of the person in input image302.

Output image308is an example of an output image generated by a neural network of neural network module248when input image306is provided as input to the neural network. Output image308includes clusters of pixels having a bright intensity against a black, uniform background. Thus, output image308includes clusters of pixels having intensities greater than an intensity of the background of output image308. Clusters of pixels in output image308indicate candidate handle locations. For instance, a handle location for the artwork of input image306may be determined for each of the clusters of pixels in output image308. Accordingly, clusters of pixels in output image308are in the shape of the bird in input image306.

FIG. 4illustrates example images400in accordance with one or more aspects of the disclosure. Images400include image402and image404. Image402and image404are examples of an output image generated by a neural network of neural network module248and processed by cluster module250and handle module252.

Image402includes a plurality of clusters of pixels having different intensities that are each greater than an intensity of the background (e.g., a black, uniform background). For instance, image402includes cluster406-1, cluster406-2, cluster406-3, cluster406-4, cluster406-5, and cluster406-6(collectively clusters406). Clusters406are denoted inFIG. 4by a respective dotted circle enclosing the respective cluster. The background of image402is a uniform black background.

In the example inFIG. 4, cluster406-1, cluster406-2, and cluster406-3have intensities greater than intensities of cluster406-4, cluster406-5, and cluster406-6. For instance, pixels of cluster406-1, cluster406-2, and cluster406-3are brighter than pixels of cluster406-4, cluster406-5, and cluster406-6.

Cluster module250evaluates respective intensities of clusters406against an intensity constraint, such as by comparing respective intensities of clusters406to a user-specified threshold intensity level. In the example inFIG. 4, cluster module250identifies cluster406-1, cluster406-2, and cluster406-3as belonging to a set of clusters of image402satisfying an intensity constraint, such as having respective intensities greater than a threshold intensity level, while cluster406-4, cluster406-5, and cluster406-6do not satisfy the intensity constraint, and therefore are not included in the set of clusters determined by cluster module250.

For each cluster in the set of clusters in image402identified as satisfying an intensity constraint, e.g., cluster406-1, cluster406-2, and cluster406-3, handle module252determines a respective handle location. Hence, image404identifies handle location408-1, handle location408-2, and handle location408-3(collectively handle locations408) that correspond to cluster406-1, cluster406-2, and cluster406-3, respectively. Handle locations408are designated with cross hairs at coordinates on image404corresponding to respective centroids of cluster406-1, cluster406-2, and cluster406-3. By contrast, since cluster406-4, cluster406-5, and cluster406-6do not satisfy the intensity constraint of cluster module250, handle module252does not determine handle locations for cluster406-4, cluster406-5, and cluster406-6, and image404does not include cross hairs designating handle locations for cluster406-4, cluster406-5, and cluster406-6.

FIG. 5illustrates example images500in accordance with one or more aspects of the disclosure. Images500are examples of training images that can be used to train a neural network of neural network module248. Images500includes four sets of images, each set including an input image, an alpha mask, and an output image. Input images include input image502-1, input image502-2, input image502-3, and input image502-4(collectively input images502). Alpha masks include alpha mask504-1, alpha mask504-2, alpha mask504-3, and alpha mask504-4(collectively alpha masks504). Output images include output image506-1, output image506-2, output image506-3, and output image506-4(collectively output images506).

An input image of input images502and a corresponding output image of output images506form a pair of training images, such as input image502-1and output image506-1. Output images506include ground-truth images and can be generated in any suitable way. In one example, an output image (e.g., one of output images506) is formed by receiving user-specified handle locations for an input image, (e.g., expert-specified handle locations for one of input images502) and convolving Gaussian filters with white patches at the user-specified handle locations on a black background. Hence, one of input images502and a corresponding one of output images506form a pair of training images suitable to train a neural network to generate an output image like output image304or output image308inFIG. 3.

To expand the number of images to train a neural network, images500includes images that have been formed from another image of images500. In the example inFIG. 5, input image502-2, input image502-3, and input image502-4have been generated by altering input image502-1, such as by scaling, translating, rotating, or combinations thereof, the artwork of input image502-1. Hence, the artworks in input image502-2, input image502-3, and input image502-4include a fish that has been perturbed by scaling, translating, rotating, and the like, the fish included in the artwork of input image502-1. For instance, relative to the fish of input image502-1, the fish of input image502-2has been moved down (towards the bottom of input image502-2), the fish of input image502-2has been moved up (towards the top of input image502-2), and the fish of input image502-2has been enlarged. In one example, scaling, translating, rotating, or combinations thereof is done with a neural network, such as a neural network module248.

A same scaling, translating, and rotating applied to an input image is also applied to its corresponding output image to expand the training data set. For instance, clusters of pixels of output image506-2have been moved by a same amount as the artwork of input image502-2, clusters of pixels of output image506-3have been moved by a same amount as the artwork of input image502-3, and clusters of pixels of output image506-4have been scaled by a same amount as the artwork of input image502-4.

Images500also includes alpha masks504that have been created for respective input images502. For instance, alpha mask504-1is generated for input image502-1, alpha mask504-2is generated for input image502-2, alpha mask504-3is generated for input image502-3, and alpha mask504-4is generated for input image502-4. An alpha mask is provided as an additional input to a neural network to help the neural network distinguish the artwork of an image from a background of an image. Accordingly, a neural network of neural network module248can generate handle locations for artwork of an input image rather than a background of the input image.

Having considered example images, consider now a discussion of example systems usable to determine image handle locations in accordance with one or more aspects of the disclosure.

Example Image Handle Systems

FIG. 6illustrates an example system600usable to determine image handle locations in accordance with one or more aspects of the disclosure. In this implementation, system600includes the modules of image handle application242as described inFIG. 2, e.g., image module244, user interface module246, neural network module248, cluster module250, and handle module252. System600is one example of image handle system208that can be constructed using the modules of image handle application242. For instance, signals can be redefined, and modules can be modified, combined, divided, added, or removed to form a modified system, without altering the functionality of system600. Accordingly, such modified systems are considered to be within the scope of the disclosure.

Furthermore, for simplicity system600is limited to the modules of image handle application242and a description of some of their interconnects. System600can, however, include any suitable signals and communications between modules omitted for simplicity. Such signals may include system clocks, counters, image indicators, sequence indicators, reset signals, cluster indicators, and the like. In one example, system600can operate in real time (e.g., with no perceptible delay to a user). Accordingly, signals can be calculated by the modules of system600and communicated between the modules of system600without significant delay, so that a handle locations can be generated and exposed in a user interface without perceptible delay to a user.

Moreover, system600can be implemented on any suitable device or devices. In one example, system600is implemented on one computing device (e.g., one of computing devices204inFIG. 2). In another example, system600is implemented on more than one computing device. For instance, parts of system600can be implemented by a first computing device, such as computing device204-1inFIG. 2, and other parts of system600can be implemented by an additional computing device or devices, such as computing device204-2. In one example, a server implements parts of system600, such as server216inFIG. 2. A server can be remote, e.g., because it is not collocated with the first computing device. A server may be configured to receive signals of system600from a computing device (e.g., one or more of computing devices204), process the received signals, such as with image handle support system218, and transmit results of the processing back to the computing device. Hence, image handle support system218of server216inFIG. 2may include system600.

Additionally or alternatively, parts of system600can be implemented by an additional computing device that is collocated with a first computing device. For instance, one of computing devices204may be a first computing device, and another of computing devices204may be an additional, collocated computing device. The first computing device and the additional computing device may be operated by one or more users. Hence, system600provides for multiple users within an environment to share data included in system600. For instance, an image can be obtained and output image generated for the input image by a first computing device operated by a first user, and the output image sent to another computing device operated by a second user. The second user can then adjust a threshold intensity level to control a number of clusters of pixels generated in a set of clusters of pixels in the output image that satisfy an intensity constraint, and send the set of clusters back to the first user and the first computing device. The first user on the first computing device can then use system600to determine handle locations from centroids of clusters in the set of clusters received from the second user on the second computing device. A user interface exposing handles at the handle locations and attributes of the handles can be displayed on the first computing device and shared with the second computing device, so both users can review the input image with handles overlaid on it.

Image module244obtains a representation of an image, such as a rasterized version of an image, vector graphics of an image, a mesh of artwork of an image, combinations thereof, and the like. Image module244can obtain an image in any suitable way. In one example, a user loads an image into system600, such as an image including artwork, and creates an animation sequence based on the image obtained by image module244.

Additionally or alternatively, image module244sends a request for an image. For instance, image module244may send a request for an image to a server, such as server216inFIG. 2, to obtain an image from a database of images, such as a database of animation sequences, artwork, graphics, and the like. Responsive to sending a request for an image, image module244can receive any suitable representation of an image.

In one example, a user enters a search term via a user interface of user interface module246, and image module244constructs a search query based on the user-entered search term to obtain an image, such as by searching databases, the Internet, a computing device (e.g., one of computing devices204inFIG. 2), combinations thereof, and the like.

Additionally or alternatively, image module244can obtain an image by extracting the image from an asset (e.g., a web page, document, and the like), such as by removing the image from the asset. For instance, image module244may extract an image from an asset that contains an image and text by removing the image from the asset and discarding the text.

Image module244provides an image, such as a rasterized image, to neural network module248and user interface module246.

User interface module246receives an image from image module244. For instance, user interface module246may expose a user interface on a display of a computing device, including displaying an image obtained by image module244. User interface module246also receives and a user input. User interface module246may receive any suitable user input. In the example inFIG. 6, user interface module246receives a user input (e.g., a user-selection of an option exposed in a user interface) indicating a desired density of handle locations to be determined for an object in an image obtained by image module244, such as by selecting one of coarse, medium, and fine options exposed in a user interface. User interface module246provides an indication of a handle density, such as coarse, medium, or fine, corresponding to a received user input to neural network module248.

User interface module246also receives a user input (e.g., a user-selection of a slider control exposed in a user interface) specifying a threshold intensity level. The threshold intensity level controls a number of handle locations determined for an image obtained by image module244by controlling a number of clusters in a set of clusters determined by cluster module250. Accordingly, user interface module246provides an indication of a threshold intensity level corresponding to a received user input to cluster module250.

Neural network module248receives a representation of an image from image module244, such as a rasterized image. Neural network module248also receives an indication of a handle density, such as coarse, medium, or fine, from user interface module246. Based on the indication of handle density received from user interface module246, neural network module248, selects a neural network from a plurality of available neural networks. Additionally or alternatively, neural network module248can select a neural network from a plurality of available neural networks based on a type of image received from image module244. For instance, neural network module248may search a structured database of neural networks that maintains a hierarchy of neural networks, with levels of the hierarchy corresponding to different types of input images, such as a level for rasterized images, a level for triangle meshes, etc. Hence, neural network module248may access a particular level of a structured database based on a type of image received from image module244.

Neural network module248applies an image received from image module244as an input image to a neural network selected by neural network module248to produce an output image. Output image304and output image308inFIG. 3are examples of output images generated by neural network module248. An output image generated by neural network module248includes clusters of pixels against a background, e.g., a uniform background. The clusters of pixels in an output image generated by neural network module248have intensities greater than an intensity of the background of the output image, and indicate candidate handle locations. For instance, a handle location may be determined for each cluster of pixels satisfying an intensity constraint. Neural network module248provides an output image generated by neural network module248to cluster module250.

Cluster module250receives an output image from neural network module248and an indication of a threshold intensity level (e.g., corresponding to a received user input) from user interface module246. Cluster module250determines a set of clusters of pixels in an output image received from neural network module248based on a threshold intensity level received from user interface module246. Cluster module250can determine a set of clusters of pixels in any suitable way.

In one example, cluster module250determines a set of clusters of pixels that satisfy an intensity constraint. For instance, cluster module250can determine a respective intensity level for each cluster of pixels in an output image, such as by calculating an average intensity level for each cluster of pixels, a representative intensity level for each pixel (e.g., by selecting an intensity level of one pixel within each cluster of pixels), and the like. Cluster module250can then compare a respective intensity level determined for each cluster of the clusters of pixels to a threshold intensity level received from user interface module246, and determine a set of clusters of pixels based on the comparison. For instance, each cluster of pixels having a respective intensity level greater than a threshold intensity level can be included in a set of clusters of pixels determined by cluster module250. Clusters of pixels not satisfying the intensity constraint (e.g., having respective intensity levels not greater than a threshold intensity level) are not included in the set of clusters of pixels determined by cluster module250. A set of clusters of pixels determined by cluster module250to satisfy an intensity constraint based on a threshold intensity level is provided to handle module252.

Handle module252receives a set of clusters of pixels from cluster module250and determines a respective handle location for each cluster of the set of clusters. In one example, handle module252evaluates a centroid of each cluster to determine coordinates of a respective handle location. A centroid can include a center-of-mass calculation to determine a coordinate (e.g., Cartesian coordinate, polar coordinate, and the like) for a handle location on an object or artwork of an image.

Additionally or alternatively, handle module252determines, for each handle location, an attribute classifying a degree of freedom for a handle at the respective handle location, such as rigid, flexible, anchor, a numerical indicator indicating a relative degree of freedom (e.g., a number from one to ten), combinations thereof, and the like. In one example, attributes are determined by neural network module248and included in metadata of an output image provided to cluster module250. Cluster module250extracts the metadata and tags each cluster with an attribute of a handle for each cluster.

Handle module252provides handle locations, including coordinates for handle locations and attributes of handles at the handle locations, to user interface module246.

User interface module246receives handle locations (e.g., coordinates for handle locations and attributes of handles at the handle locations), and generates a user interface based on the handle locations. In one example, user interface module246overlays indicators of handles at the handle locations on an input image received from image module244. For instance, image602is an example of a part of a user interface generated by user interface module246. Image602includes artwork depicting a tiger. Superimposed on image602are a plurality of handles604at handle locations determined by system600.

Handles604have been placed on the artwork of image602by system600at handle locations that respect the structure and symmetry of the artwork, such as on the tiger's feet, ears, tail, cheeks, and torso. Accordingly, handles604can be used to deform the tiger and create a realistic animation sequence (e.g., so that the tiger walks in a realistic fashion).

FIG. 7illustrates an example system700in accordance with one or more aspects of the disclosure. System700is an example of a neural network of neural network module248inFIG. 2. System700includes a conditional generative adversarial network in which a generator and a discriminator are jointly trained so that the discriminator tries to distinguish between images generated by the generator and ground-truth images, while the generator tries to fool the discriminator into thinking its generated output image is real.

System700includes input image702which is provided to generator704as input. In the example inFIG. 7, input image702is a rasterized image in greyscale. However, system700can be trained with any suitable representation of an image, such as a rasterized image, a triangle mesh of an object in an image, vector graphics of artwork of an image, and the like.

Generator704generates output image706from input image702. Generator704can be any suitable generator that translates an input image of a first type of image (e.g., a rasterized image in greyscale, a triangle mesh of an object in an image, vector graphics of artwork of an image, and the like) to an output image of a second type of image (e.g., an image including clusters of pixels against a uniform background to designate candidate handle locations, such as output image706). In one example, generator704includes an encoder-decoder network where the input is passed through multiple layers of an encoder followed by multiple layers of a decoder (discussed below in more detail with regards toFIG. 8). Each layer performs a plurality of convolutions with multiple filters, and can be denoted according to Ck, where k denotes the number of filters in a layer. In one example, an encoder of generator704includes seven layers denoted by C64-C128-C256-C512-C512-C512-C512, and a corresponding decoder of generator704includes the seven layers denoted by C512-C512-C512-C512-C256-C128-C64.

Output image706generated by generator704and input image702are provided to discriminator708. Discriminator708includes a neural network trained to distinguish between images generated by generator704and ground-truth images (e.g., images in a training set of images where handle locations have been determined by trained experts, such as output images506inFIG. 5). Discriminator708can include any suitable discriminator to distinguish between “real” images (e.g., ground-truth images) and “fake” images (e.g., images generated by generator704). In one example, discriminator708includes a neural network including four layers denoted by C64-C128-C256-C512.

In the example inFIG. 7, discriminator708receives output image706generated by generator704and input image702in the top half ofFIG. 7, and ground-truth image710and input image702in the bottom half ofFIG. 7. Accordingly, discriminator708determines that output image706is not a real image (e.g., not a ground-truth image), and generates an output indicating a “fake” image in the top half ofFIG. 7. Furthermore, discriminator708determines that ground-truth image710is not a fake image (e.g., not an image generated by generator704), and generates an output indicating a “real” image in the bottom half ofFIG. 7.

System700can be trained with a loss function including a pixel loss term, an adversarial loss term, and a perceptual loss term, as described above. In one example, generator704and discriminator708are trained by alternating between updating weights (e.g., filter coefficients) of generator704and updating weights of discriminator708. For instance, weights of generator704may be updated on a first update cycle, and used to compute output images on a second update cycle for which weights of discriminator708are updated, and the process repeated so that weights of generator704and discriminator708are updated on alternate update cycles.

An update cycle may include processing of any suitable number of images. In one example, a training set of images includes P total image pairs of input images and ground-truth images. For instance, P may be 1000 image pairs in a training set. For efficiency, weights of generator704, discriminator708, or generator704and discriminator708may be updated on a batch basis, such as for every Q image pairs of the training set processed with P>Q. For instance, with P=1000, an appropriate value of Q may be 10.

By jointly training a generator and a discriminator so that the discriminator attempts to distinguish between images generated by the generator and ground-truth images, and the generator attempts to trick the discriminator into thinking its generated output image is real, system700is able to reliably generate output images to determine image handle locations. In one example, system700is pre-trained, so that at run-time (e.g., when system700is used in a user computing device, such as one of computing devices204inFIG. 2), discriminator708can be disabled while pre-trained generator704can generate output images for user-supplied input images. In this case, weights of generator704and discriminator708are not adjusted based on user data. Additionally or alternatively, discriminator708may not be disabled at run-time, so that generator704and discriminator708can learn in an on-line fashion. In this case, weights of generator704and discriminator708can be adjusted based on user data (e.g., user-supplied input images to system700and output images generated by system700).

FIG. 8illustrates example systems800in accordance with one or more aspects of the disclosure. Systems800includes network802and network804. Network802and network804are examples of neural networks that can be included in generator704inFIG. 7. Network802is an example of an encoder-decoder, and network804is an example of an encoder-decoder with skip connections.

In network802, an input image is passed through a series of layers of encoder806that progressively downsample, until bottleneck layer808is reached. In decoder810of network802, the process of encoder806is reversed, and layers of decoder810progressively upsample data. In one example, layers of encoder806each downsample by a factor of two, and layers of decoder810each upsample by a factor of two.

Each layer of encoder806performs convolutions, and each layer of decoder810performs deconvolutions using spatial filters. In one example, convolutions in encoder806are performed with 4×4 spatial filters with stride2. Stride refers to an amount a filter is shifted for each calculation of a convolution or deconvolution. Additionally or alternatively, deconvolutions in decoder810can be performed with 3×3 spatial filters and stride1.

The architecture of network802requires that all information flow through all layers of network802. To speed processing time and conserve processing resources, network804includes skip connections812. Skip connections concatenate all channels at layer ρ in encoder806to those at a mirrored layer−ρ of decoder810, whereis the total number of layers. Accordingly, network804efficiently processes data when information can be passed from one layer of encoder806to a corresponding layer of decoder810, thus bypassing bottleneck layer808.

The systems described herein constitute an improvement over systems that require adjustment of basis points of vector graphics to deform artwork of an image, or require manual placement of handles and handle locations on the artwork to deform the artwork, such as when generating an animation sequence. By using a neural network trained to translate input images (e.g., rasterized greyscale images) to output images that include clusters of pixels having intensities greater than a uniform background and that indicate candidate handle locations, handle locations are reliably and quickly determined by the systems described herein. Hence, a user's time needed to deform an artwork (e.g., to generate an animation sequence), and the associated user-frustration in doing so, are significantly reduced compared to systems that require adjustment of basis points of vector graphics or manual placement of handle locations on the artwork. Furthermore, by receiving user selections for desired densities of handles and threshold intensity levels, a user can control the number of handles and handle locations determined for artwork of an image by the systems described herein, and therefore does not need to waste time adding or removing handles, as may be required by other systems that require manual placement of handles on artwork of an image.

Having considered example systems, consider now a discussion of example procedures for determining image handle locations in accordance with one or more aspects of the disclosure.

Example Procedures

FIG. 9illustrates an example procedure900for determining image handle locations in accordance with one or more aspects of the disclosure. Aspects of the procedure may be implemented in hardware, firmware, or software, or a combination thereof. The procedure is shown as a set of blocks that specify operations performed by one or more devices and are not necessarily limited to the orders shown for performing the operations by the respective blocks. In at least some aspects, the procedure may be performed in a digital medium environment by a suitably configured computing device, such as one or more of computing devices204or server216ofFIG. 2that makes use of an image handle system, such as system600, system700, system800, or image handle system208. An image handle system implementing procedure900may be an independent application that has been installed on the computing device, a service hosted by a service provider that is accessible by the computing device, a plug-in module to the computing device, or combinations thereof.

A representation of an image of an object is obtained (block902). In one example, image module244obtains a representation of an image of an object. A representation of the image can include a rasterized version of the image. Additionally or alternatively, the representation of the image includes a triangle mesh of the image. The image can be any suitable image, and the object can be any suitable object. In one example, the object includes artwork, such as a cartoon or drawing, that can be deformed to generate an animation sequence.

An output image is generated by applying the representation of the image to a neural network, the output image including clusters of pixels against a uniform background (block904). In one example, neural network module248generates an output image by applying the representation of the image to a neural network, the output image including clusters of pixels against a uniform background.

In one example, a user-selection indicating a desired density of handle locations to be determined for the object is received, and the neural network is selected from a plurality of available neural networks based on the desired density of handle locations indicated by the user-selection.

Additionally or alternatively, the neural network can be a conditional generative adversarial network having a generator with skip connections that concatenate activations from one layer of an encoder of the neural network to a corresponding layer of a decoder of the neural network. In one example, the neural network is trained with a loss function including a pixel loss term, an adversarial loss term, and a perceptual loss term.

A set of the clusters of pixels having a respective intensity level above a threshold intensity level is determined (block906). In one example, cluster module250determines a set of the clusters of pixels having a respective intensity level above a threshold intensity level. Additionally or alternatively, the threshold intensity level can be a user-specified threshold intensity level and controls a number of handle locations that are determined for the object.

A respective handle location of the object for each cluster of the set of clusters is determined from a respective centroid of said each cluster (block908). In one example, handle module252determines a respective handle location of the object for each cluster of the set of clusters is determined from a respective centroid of said each cluster. The respective handle location of the object can include a coordinate for the respective handle location and an attribute classifying a degree of freedom for the respective handle location. In one example, the attribute includes one of rigid, flexible, and anchor. Additionally or alternatively, the attribute includes a numerical indicator indicating a relative degree of freedom.

FIG. 10illustrates an example procedure1000for determining image handle locations in accordance with one or more aspects of the disclosure. Aspects of the procedure may be implemented in hardware, firmware, or software, or a combination thereof. The procedure is shown as a set of blocks that specify operations performed by one or more devices and are not necessarily limited to the orders shown for performing the operations by the respective blocks. In at least some aspects, the procedure may be performed in a digital medium environment by a suitably configured computing device, such as one or more of computing devices204or server216ofFIG. 2that makes use of an image handle system, such as system600, system700, system800, or image handle system208. An image handle system implementing procedure1000may be an independent application that has been installed on the computing device, a service hosted by a service provider that is accessible by the computing device, a plug-in module to the computing device, or combinations thereof.

An image of an object is obtained (block1002). In one example, image module244obtains an image of an object. The image can be any suitable image, and the object can be any suitable object. In one example, the object includes artwork, such as a cartoon or drawing, that can be deformed to generate an animation sequence.

A user-selection indicating a density of handles for the object is received (block1004). In one example, user interface module receives a user-selection indicating a density of handles for the object. Additionally or alternatively, the density of handles for the object indicated by the user-selection can be one of coarse, medium, and fine.

Based on the user-selection, a neural network is determined from a plurality of neural networks corresponding to different densities of handles (block1006). In one example, neural network module248determines, based on the user-selection, a neural network from a plurality of neural networks corresponding to different densities of handles.

An output image is generated by applying the image to the neural network, the output image including clusters of pixels indicating candidate handle locations (block1008). In one example, neural network module248generates an output image by applying the image to the neural network, the output image including clusters of pixels indicating candidate handle locations.

In one example, the neural network is trained with input images having user-specified handle locations and output images generated by convolving Gaussian filters with white patches at the user-specified handle locations on a black background. Additionally or alternatively, the neural network can be further trained by providing to the neural network, for each of the input images, a respective alpha mask distinguishing a respective object of said each of the input images from a respective background of said each of the input images.

A set of the clusters of pixels having a respective intensity level above a threshold intensity level is determined (block1010). In one example, cluster module250determines a set of the clusters of pixels having a respective intensity level above a threshold intensity level.

A respective handle location of the object is determined for each cluster of the set of clusters (block1012). In one example, handle module252determines a respective handle location of the object for each cluster of the set of clusters. The respective handle location of the object for each cluster of the set of clusters can be determined from a center-of-mass of said each cluster.

In one example, a respective handle location of the object includes a coordinate for the respective handle location and an attribute indicating a degree of freedom of a respective handle at the respective handle location, further comprising displaying each said respective handle at the coordinate on the image for each said respective handle location with a respective designator indicating the attribute. Additionally or alternatively, the neural network can generate the clusters of pixels for handles having a same degree of freedom on a same channel of the neural network. For instance, clusters of pixels for anchors may be generated on one channel of a neural network, and clusters of pixels for flexible handles may be generated on another channel of the neural network.

Additionally or alternatively, a triangle mesh for the object can be generated, with each said respective handle location of the object being at a respective vertice of the triangle mesh. The image can be deformed based on the triangle mesh by translating or rotating at least one handle at a vertice of the triangle mesh.

FIG. 11illustrates an example procedure1100for determining image handle locations in accordance with one or more aspects of the disclosure. Aspects of the procedure may be implemented in hardware, firmware, or software, or a combination thereof. The procedure is shown as a set of blocks that specify operations performed by one or more devices and are not necessarily limited to the orders shown for performing the operations by the respective blocks. In at least some aspects, the procedure may be performed in a digital medium environment by a suitably configured computing device, such as one or more of computing devices204or server216ofFIG. 2that makes use of an image handle system, such as system600, system700, system800, or image handle system208. An image handle system implementing procedure1100may be an independent application that has been installed on the computing device, a service hosted by a service provider that is accessible by the computing device, a plug-in module to the computing device, or combinations thereof.

An image of an object is obtained (block1102). In one example, image module244obtains an image of an object. The image can be any suitable image, and the object can be any suitable object. In one example, the object includes artwork, such as a cartoon or drawing, that can be deformed to generate an animation sequence.

An output image is generated by applying the image to a neural network, the output image including a background and clusters of pixels having intensities greater than an intensity of the background (block1104). In one example, neural network module248generates an output image by applying the image to a neural network, the output image including a background and clusters of pixels having intensities greater than an intensity of the background.

A respective intensity level of each cluster of the clusters of pixels is compared to a threshold intensity level (block1106). In one example, cluster module250compares a respective intensity level of each cluster of the clusters of pixels to a threshold intensity level.

In one example, the threshold intensity level is determined based on a number of the clusters of pixels in the output image. Additionally or alternatively, the threshold intensity level can be determined based on the intensities of the clusters of pixels in the output image. For instance, cluster module250can determine a threshold intensity level so that a predetermined amount of clusters of pixels have intensities below the threshold intensity level, such as by determining a threshold intensity level so that at least a number of clusters of pixels (e.g., at least one cluster), at least a percentage of the clusters of pixels (e.g., 10%), and the like, do not satisfy an intensity constraint because they have respective intensity levels below the threshold intensity level.

A set of the clusters of pixels is determined based on the comparing (block1108). In one example, cluster module250determines a set of the clusters of pixels based on the comparing.

A respective handle location of the object is determined for each cluster of the set of clusters based on a centroid of said each cluster of the set of clusters (block1110). In one example, handle module252determines a respective handle location of the object for each cluster of the set of clusters based on a centroid of said each cluster of the set of clusters.

The procedures described herein constitute an improvement over procedures that require adjustment of basis points of vector graphics to deform artwork of an image, or require manual placement of handles and handle locations on the artwork to deform the artwork, such as when generating an animation sequence. By using a neural network trained to translate input images (e.g., rasterized greyscale images) to output images that include clusters of pixels having intensities greater than a uniform background and that indicate candidate handle locations, handle locations are reliably and quickly determined by the systems described herein. Hence, a user's time needed to deform an artwork (e.g., to generate an animation sequence), and the associated user-frustration in doing so, are significantly reduced compared to procedures that require adjustment of basis points of vector graphics or manual placement of handle locations on the artwork. Furthermore, by receiving user selections for desired densities of handles and threshold intensity levels, a user can control the number of handles and handle locations determined for artwork of an image by the systems described herein, and therefore does not need to waste time adding or removing handles, as may be required by other procedures that require manual placement of handles on artwork of an image.

Having considered example procedures in accordance with one or more implementations, consider now example systems and devices that can be utilized to practice the inventive principles described herein.

Example Systems and Devices

FIG. 12illustrates an example system generally at1200that includes an example computing device1202that is representative of one or more computing systems and devices that may implement the various techniques described herein. This is illustrated through inclusion of image handle system208, system600, system700, system800, image handle application242, and image handle support system218, which operate as described above. Computing device1202may be, for example, a user computing device (e.g., one of computing devices204), or a server device of a service provider, (e.g., server216). Furthermore, computing device1202may include an on-chip system, multiple computing devices, combinations thereof, or any other suitable computing device or computing system. Accordingly,FIG. 12illustrates computing device1202as one or more of a tablet, a laptop computer, a smart phone, smart eye glasses, and a camera, though these examples are illustrative and in no way are meant to limit the type or number of devices included in computing device1202.

Computer-readable storage media1206is illustrated as including memory/storage1212. Storage226inFIG. 2is an example of memory/storage included in memory/storage1212. Memory/storage component1212may include volatile media (such as random access memory (RAM)), nonvolatile media (such as read only memory (ROM), Flash memory, optical disks, magnetic disks, and so forth), or combinations thereof. Memory/storage component1212may include fixed media (e.g., RAM, ROM, a fixed hard drive, and so on) as well as removable media (e.g., Flash memory, a removable hard drive, an optical disc, and so forth). Computer-readable media1206may be configured in a variety of other ways as further described below.

Computing device1202also includes applications1214. Applications1214are representative of any suitable applications capable of running on computing device1202, and may include a web browser which is operable to access various kinds of web-based resources (e.g., assets, media clips, images, content, configuration files, services, user profiles, and the like). Applications1214include image handle application242, as previously described. Furthermore, applications1214includes any applications supporting image handle system208, system600, system700, system800, and image handle support system218.

Combinations of the foregoing may also be employed to implement various techniques described herein. Accordingly, software, hardware, or executable modules may be implemented as one or more instructions, logic embodied on some form of computer-readable storage media or by one or more hardware elements1210, or combinations thereof. Computing device1202may be configured to implement particular instructions and functions corresponding to the software and hardware modules. Accordingly, implementation of a module that is executable by computing device1202as software may be achieved at least partially in hardware, e.g., through use of computer-readable storage media and hardware elements1210of processing system1204. The instructions and functions may be executable/operable by one or more articles of manufacture (for example, one or more computing devices1202or processing systems1204) to implement techniques, modules, and examples described herein.

Cloud1216includes and is representative of a platform1218for resources1220. Platform1218abstracts underlying functionality of hardware (e.g., servers) and software resources of cloud1216. Resources1220may include applications, data, or applications and data that can be utilized while computer processing is executed on servers that are remote from computing device1202. Resources1220can also include services provided over the Internet, through a subscriber network, such as a cellular or Wi-Fi network, or combinations thereof. Resources1220can include asset store1222, which stores assets, such as images, photographs (e.g., user images in a gallery, a database of stock photographs, and the like), document templates, user profile data, user image libraries, photographs posted in a shared photo service, metadata of assets, and the like, and may be accessed by computing device1202.

Platform1218may abstract resources and functions to connect computing device1202with other computing devices. Platform1218may also serve to abstract scaling of resources to provide a corresponding level of scale to encountered demand for resources1220that are implemented via platform1218. Accordingly, in an interconnected device embodiment, implementation of functionality described herein may be distributed throughout system1200. For example, the functionality may be implemented in part on computing device1202as well as via platform1218that abstracts the functionality of cloud1216.

CONCLUSION

In one or more implementations, a digital medium environment includes at least one computing device. Systems and techniques are described herein for determining image handle locations. An image, such as a rasterized image in greyscale, is provided to a neural network as input, and the neural network translates the input image to an output image that includes clusters of pixels against a background that have intensities greater than an intensity of the background and that indicate candidate handle locations. Intensities of clusters of pixels in an output image are compared to a threshold intensity level to determine a set of the clusters of pixels satisfying an intensity constraint. The threshold intensity level can be user-selectable, so that a user can control a density of handles. A handle location for each cluster of the set of clusters is determined from a centroid (e.g., a center of mass) of each cluster. Handle locations include a coordinate for a respective handle location, such as an x-y coordinate that identifies a location on an object in an image and an attribute classifying a degree of freedom for a handle at a handle location.

Although the invention has been described in language specific to structural features and methodological acts, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed invention.