Abstract:
An image editing system for use with a web-based video editing system is provided. The image editing system comprises: a detection system configured to receive image data; detect regions of the image representing faces; store the data representing the coordinates of the faces as points of interest; and receive user input adding additional points of interest, modifying the points of interest, or deleting the pre-selected points of interest; and a cropping subsystem configured to determine the portion of the image containing the maximum number of points of interest that will fit within the crop area; and crop the image.

Description:
[0001]    This application claims priority to and the benefit of U.S. Provisional Application No. 60/898,201, filed on Jan. 29, 2007 and U.S. Provisional Application No. 60/913,204, filed on Apr. 20, 2007, which is incorporated by reference as if set forth in full herein. This application is also related to the co-pending U.S. Provisional Application No. 60/897,552, filed on Jan. 26, 2007, titled “Video Downloading and Scrubbing System and Method” (Atty. docket no. 58745), U.S. Provisional Patent Application No. 60/897,558, filed on Jan. 26, 2007, titled “Real Time Online Video Editing System and Method” (Atty. docket no. 58746), U.S. Provisional Patent Application No. 60/897,559, filed on Jan. 26, 2007, titled “Payment System and Method for Web-Based Video Editing System” (Atty. docket no. 58891), U.S. Provisional Patent Application No. 60/897,544, filed on Jan. 26, 2007, titled “System and Method for Editing Web-Based Video” (Atty. docket no. 58981), and U.S. Provisional Patent Application No. 60/915,427, filed on May 1, 2007, titled “System and Method for Flow Control in Web-Based Movie Editing System” (Atty. docket no. 59323), the entire contents of each of which are expressly incorporated herein by reference. 
     
    
     FIELD OF THE INVENTION 
       [0002]    The present invention relates to an image editing system, and more particularly, to a system for detecting and editing points of interest on images and manipulating those images using the points of interest for specific applications such as cropping, animation, and navigation. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0003]      FIG. 1  is a block diagram of a web-based video editing system according to a first embodiment of the present invention. 
           [0004]      FIG. 1   a  is a block diagram of a web-based video editing system according to a second embodiment of the present invention. 
           [0005]      FIG. 2  is a block diagram of one embodiment of an image editing system of the web-based video editing system of  FIG. 1 . 
           [0006]      FIG. 3  is a flowchart showing a method of operation for the detection subsystem. In some embodiments, the subsystem receives data pertaining to a particular image in the system. 
           [0007]      FIG. 3   a  shows a sample image containing points of interest. In this example, the image contains two figures with recognizable human faces. 
           [0008]      FIG. 4  shows a flowchart for a method of operation to automatically crop an image using data for predetermined points of interest in the image. 
           [0009]      FIG. 5   a  is a visual representation of the method by which image data in a landscape orientation is processed for photo-cropping. 
           [0010]      FIG. 5   b  is a visual representation of a method by which image data in a portrait orientation is processed for photo-cropping. 
           [0011]      FIG. 5   c  is a visual representation of a method by which image data is processed for photo-cropping. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0012]      FIG. 1  is a block diagram of a web-based video editing system according to a first embodiment of the present invention. The editing system includes one or more communication devices  110  each having a graphical user interface  115 , a server  120  having a connection manager  130  and a image editing system  140  operating on the server, and a network  150  over which the one or more communication devices and the server communicate. The communication devices include, but are not limited to, a personal computer, a mobile telephone, a PDA, or any other communication device configured to operate as a client computer to the server. The network to which the server and devices are coupled may be a wireless or a wireline network and may range in size from a local area network to a wide area network to the Internet. A dedicated open socket connection exists between the connection manager and the client computers. 
         [0013]    In some embodiments of the system, one or more client computers are configured to transmit information to and receive information from the server. In some embodiments, each of the client computers is configured to send a query for information and the server is configured to respond to the query by sending the requested information to the client computer. In some embodiments, one or more of the client computers is configured to transmit commands to the server and the server is configured to perform functions in response to the command. 
         [0014]    In some embodiments, each of the client computers is configured with an application for displaying multimedia on the graphical user interface of the client computer. The application may be Adobe Flash® or any other application capable of displaying multimedia. 
         [0015]    The connection manager is configured to determine the condition of the server and perform asynchronous messaging to one or more of the client computers over the dedicated open socket connection. In some embodiments, the content of the messages is indicative of the state of the server. 
         [0016]    The server is configured to receive requests from one or more of the client computers and perform functions in response to the received requests. The server performs any number of functions typically performed in the server of a web-based video editing system. The server also provides an image editing system for the web-based video editing system. 
         [0017]      FIG. 1   a  is a block diagram of some embodiments of the web-based video editing system of  FIG. 1 . In these embodiments, the system does not include a connection manager  130   a  for communication between the client computer  110   a  and the server  120   a . Otherwise, client computer  110   a  is an instance of client computer  110 , graphical user interface  115   a  is an instance of graphical user interface  115 , server  120   a  is an instance of server  120 , image editing system  140   a  is an instance of image editing system  140 , and internet  150   a  is an instance of internet  150 . All the components in  FIG. 1   a  are thus identical to those in  FIG. 1 , and they are all configured to operate as described in  FIG. 1 . 
         [0018]    In some embodiments, still referring to  FIGS. 1 and 1   a , the combination of data representing specified portions of an image is sometimes referred to as a point of interest. These points of interest may take the shape of a square, rectangle, or other quadrilateral, a circle, an ellipse, or any other closed two-dimensional shape. A user operates the client computer  110  to manage and manipulate points of interest on a computer image. The user can add, reposition, and resize points of interest through the graphical user interface  115 . In some embodiments, the server  120  automatically detects points of interest by using algorithms such as face detection and feature recognition. The server-generated points of interest are also presented to the user in a graphical fashion, allowing the user to adjust their size and position and remove false positives or undesired interest points in an identical fashion to the points added by the user. Once these points of interest have been saved, they are stored in a database on the server along with the image and used appropriately as the image is displayed in different contexts. 
         [0019]    In some embodiments, still referring to  FIGS. 1 and 1   a , when the image needs to be displayed with a different aspect-ratio, the points of interest are used to crop the image to best fit the new aspect ratio. 
         [0020]    In some embodiments, users can choose to present the image in an animated fashion, and the client computer will use the points of interest to intelligently pan around the image and focus on parts of the image. 
         [0021]    In some embodiments, the points of interest become ‘hot spots’ suitable for web-based navigation. 
         [0022]      FIG. 2  is a block diagram of one embodiment of an image-editing system  140  of the web-based video editing system of  FIG. 1 . The image editing system includes a detection subsystem  210 . In some embodiments, the detection subsystem includes a processor  212 , memory  214 , and computer code product including instruction stored in the memory and adapted to cause the processor, and thereby the detection subsystem, to receive and process user point of interest detection and selection requests. The memory also stores information indicative of the user selection requests. The memory may be any type of read-write memory, including, but not limited to, random access memory. 
         [0023]    In some embodiments, the user input received by the detection subsystem includes the identity of the user, the image to be cropped, the size to which the image is to be cropped, and the points of interest edited by the user and/or generated by the server and stored on the server in an earlier stage. Each point of interest consists of a rectangular region of the image that encompasses the point of interest, although in some embodiments this point could be a circle (position and radius) or any other closed two-dimensional shape. 
         [0024]    The data is be stored in the editing system server (not shown), the detection subsystem memory  214 , or at a remote location connected to the network of  FIG. 1 . The data is provided by the web-based video editing system or is data generated by the user. The data may include uncropped images, cropped images, meta data for cropping the images, and points of interest. 
         [0025]    In some embodiments, the system includes an animation subsystem configured to examine the points of interest and present an intelligent fly through of the image. In some embodiments, the system includes a navigation subsystem that allows a user to annotate the points of interest with text boxes and hyperlinks. 
         [0026]      FIG. 3  is a flowchart showing a method of operation for the detection subsystem. In some embodiments, the subsystem receives  310  data pertaining to a particular image in the system. This data is a pointer to a file on the server, a pointer to a file stored in a remote location connected to the server by the network in  FIG. 1 , the actual data for the image, or any other data that would allow the subsystem access to the image data. The subsystem then processes  320  the image using a facial recognition detection algorithm to determine all of the faces in the image and creates points of interest for each of the detected faces. Depending on the embodiment, the subsystem either transmits (not shown) the data to the image editing system, or the subsystem transmits  330  the data to the client computer. If the subsystem transmits  330  the data to the client computer, the client computer displays the image alone with markers indicating each of the automatically selected points of interest. The system then receives  340  input from the user indicating any changes to the selected points of interest that he desires. For example, the user may remove some of the pre-selected points of interest, or the user may add additional points of interest. The additional points of interest may include any particular features of the image in addition to the automatically detected faces that the user wishes to have included in the cropped image. Once the subsystem receives  340  this input, it transmits  350  the data for the points of interest to the image editing system. 
         [0027]      FIG. 3   a  shows a sample image containing points of interest. In this example, the image  300  contains two figures  310  with recognizable human faces. When the detection subsystem processes this image, it automatically applies points of interest  320  to these faces. In some embodiments where the user may add or delete points of interest, the image also contains points of interest  330  added by the user so that a pet  340  and a painting  350  are also marked as being important in the image and thus represent visuals within in the image that the user desires to include in the final cropped image. 
         [0028]      FIG. 4  shows a flowchart for a method of operation to automatically crop an image using data for predetermined points of interest in the image. In some embodiments, the image editing system receives  410  data pertaining to points of interest either automatically generated or generated by the user and data pertaining to the image to be cropped. The system farther receives  420  data pertaining to the size and orientation to which the image is to be cropped. Based on this data, the system determines  430  whether to crop the image along its height or along its width. The system then analyzes every valid configuration of the crop rectangle by beginning  440  at one side of the image. The system examines  450  the locations of the points of interest in relation to the position of the crop rectangle, and assigns  460  a score to the current crop rectangle configuration. In assigning the score, the system tries to maximize the area of points of interest displayed inside the crop rectangle, to avoid partially cropping a point of interest and cutting it off. In some embodiments, the system also adjusts the score to display the points of interest in accordance with basic photographic rules, such as the rule of thirds, centering, framing, balance, rotation such that all faces in the image are vertical, use of the golden-ratio, etc. When a point of interest is entirely within the crop rectangle, its contribution to the score is proportional to its area. Points of interest entirely outside the crop rectangle do not contribute to the score, and points of interest only partially inside the crop rectangle subtract from the overall score. In some embodiments applying photographic rules, such as the rule of thirds, the photographic rules contribute small amounts to the score. In some embodiments, the photographic rules contribute just enough to the score to allow the system to choose between crop configurations that would otherwise be tied. Once the system has assigned  460  a score to the current configuration, it determines  470  whether it has reached the other end of the image. If the system has not reached the other end, it moves  480  the crop area by one pixel, analyzes  450  the current position, and assigns  460  it a score. If instead, the system determines  470  that it has reached the end of the image, it selects  490  and stores the crop configuration with the highest score. The system may then crop the image accordingly. 
         [0029]    In some embodiments, the system also allows the user to preview the cropped image overlaid on the original image and adjust the cropped region. 
         [0030]      FIG. 5   a  is a visual representation of the method by which image data in a landscape orientation is processed for photo-cropping. The image  510   a  contains three points of interest  520   a . When the system processes this image for cropping, it begins with the cropping area  530   a  at the left side of the image and walks it over to the right side pixel by pixel. As can be seen in the figure, when the cropping area is in position  530   a , two points of interest are captured; however, when it is in position  540   a , only one point of interest is captured. Thus, when the system crops this image, it will crop it at position  530   a.    
         [0031]      FIG. 5   b  is a visual representation of a method by which image data in a portrait orientation is processed for photo-cropping. The image  510   b  contains five points of interest  520   b ,  530   b ,  540   b ,  550   b , and  560   b . When the image is processed for cropping, the cropping area  590   b  moves from top to bottom in search of the area that contains the most points of interest. In this case, position  570   b  contains three complete points of interest and therefore has the highest concentration. The next highest concentration is found in position  580   b , which only has two complete points of interest. Thus, when the system crops this image, it will crop it at the coordinates corresponding to the cropping area in position  570   b.    
         [0032]      FIG. 5   c  is a visual representation of a method by which image data is processed for photo-cropping. The image  510   c  in  FIG. 5   c  is processed much in the same manner as discussed in reference to  FIGS. 5   a  and  5   b , above. In this image, while there are two positions  550   c  and  560   c  for the cropping area where two points of interest would be captured, in position  560   c , the point of interest  530   c  is only partially contained within the cropping area. In position  550   c , both the points of interest are completely contained in the cropping area. Position  550   c  would thus have a higher score than position  560   c , and the system would therefore crop this image at position  550   c.    
         [0033]    In some embodiments, the points of interest are used as hot-spots for navigation. Users can associate hyperlinks or text with a hot-spot through a graphical user interface. When a viewer watches the image on a client machine, positioning the mouse cursor over a hot-spot can display a text popup, and clicking o the hotspot can perform a navigation action, such as opening a new web-page. 
         [0034]    In some embodiments, the points of interest are used as reference points for automatic animation of the image. Different animation styles exist and the user can choose the desired style. For example, one animation style consists of focusing on a point of interest and gradually zooming into it. Another style consists of a slow pan between all of the points of interest in a picture. There are dozens of such styles, all which use the points of interest to generate more intelligent animation. 
         [0035]    In some embodiments, the user may have the system assign hot-spots and/or perform animation on the cropped image, or the user may use the system solely to detect points of interest and then use an uncropped image for hot-spots and/or animation.