Abstract:
A three-dimensional movie browser or editor is disclosed. In one aspect, video frames are quads in space. A movie plays at full quality by moving the quads in space at a speed of the movie and having a camera follow the quads. A focus frame in the movie is put forward by a cover-flow/dock like expansion around it. Video frames are loaded at a resolution and with an interval that depends on their position regarding a focus point or the optical flow between the frames.

Description:
TECHNICAL FIELD 
       [0001]    The subject matter of this patent application generally relates to browsers or editors. 
       BACKGROUND 
       [0002]    A hallmark of modern movie browsers or editors is that they display video frames of a movie in a two-dimensional display environment. The video frames are tiled in the two-dimensional display environment on a display screen. This environment may require multiple screens, or display small video frames on a single screen, making the result overwhelming. If multiple screens are used, users may find it difficult to browse through the multiple screens. If a single screen is used, the small display size of the video frames may make it difficult for users to browse or edit them. In addition, the two-dimensional display environment may make it difficult for the user to gain a holistic view of the video frames. 
         [0003]    For example, users may find it difficult to navigate through multiple screens, or through small video frames on a single screen, to determine a focus point. Selecting video frames can also be difficult as the user may have to navigate through multiple screens, or through small video frames on a single screen, to select the video frames. The two-dimensional environment may also make it difficult for users to judge temporal distance between video frames or compare video frames. The two-dimensional environment may also use a large amount of resources as the video frames are displayed at a single resolution. 
       SUMMARY 
       [0004]    A three-dimensional movie browser or editor is disclosed. In one aspect, video frames are quads in space. A movie plays at full quality by moving the quads in space at a speed of the movie and having a camera follow the quads. A focus frame in the movie is put forward by a cover-flow/dock like expansion around it. Video frames are loaded at a resolution and with an interval that depends on their position regarding a focus point or the optical flow between the frames. 
         [0005]    Other implementations of a three-dimensional movie browser or editor are disclosed, including implementations directed to methods and computer-readable mediums. 
     
    
     
       DESCRIPTION OF DRAWINGS 
         [0006]      FIG. 1  is a screen shot showing a three-dimensional movie browser or editor with video frames displayed concurrently along a timeline. 
           [0007]      FIG. 2  is a screen shot showing the three-dimensional content browser or editor with video frames displayed concurrently along a timeline of  FIG. 1  with video frames selected for image editing. 
           [0008]      FIG. 3  is a screen shot showing the three-dimensional content browser or editor with video frames displayed concurrently along a timeline of  FIG. 1  with image analysis tools displayed along the timeline proximate to the video frames. 
           [0009]      FIG. 4  is a screen shot showing the video frames displayed concurrently along a timeline with image analysis tools displayed along the timeline proximate to the video frames of  FIG. 3  at a zoomed-in view. 
           [0010]      FIG. 5  is a screen shot showing a zoomed-in view of a timeline at a location of a focus frame. 
           [0011]      FIG. 6  is a screen shot showing a timeline where video frames are displayed at an interval depending on the optical flow between the video frames. 
           [0012]      FIG. 7  is a screen shot showing a zoomed-in view of a timeline at a location of a focus frame with image analysis tools displayed along the timeline proximate to video frames. 
           [0013]      FIG. 8  is a screen shot showing a zoomed-in view of a timeline at a location of a focus frame with video frames flipped or rotated. 
           [0014]      FIG. 9  is a screen shot showing the zoomed-in view of a timeline at a location of a focus frame with video frames flipped or rotated of  FIG. 8  with image analysis tools displayed along the timeline proximate to the video frames. 
           [0015]      FIG. 10  is a screen shot showing an even further zoomed-in view of a timeline at a location of a focus frame with video frames flipped or rotated. 
           [0016]      FIG. 11  is a screen shot showing the even further zoomed-in view of a timeline at a location of a focus frame with video frames flipped or rotated of  FIG. 10  with image analysis tools displayed along the timeline proximate to the video frames. 
           [0017]      FIG. 12A  illustrates an example of a three-dimensional movie browser or editor with objects in a video frame identified for tracking. 
           [0018]      FIG. 12B  illustrates an example of video frames displayed concurrently along a timeline with linear paths displayed along the timeline displaying the movement of each identified object. 
           [0019]      FIG. 13  shows a method to display video frames in a three-dimensional movie browser or editor. 
           [0020]      FIG. 14  shows a method to track objects in video frames displayed in a three-dimensional movie browser or editor. 
           [0021]      FIG. 15  is a block diagram of an example software stack. 
           [0022]      FIG. 16  is a block diagram of an example system architecture. 
       
    
    
     DETAILED DESCRIPTION 
     Three-Dimensional Movie Browser or Editor Overview 
       [0023]      FIG. 1  is a screen shot showing a three-dimensional movie browser or editor  100  (also referred to herein as a “browser”) with video frames  155  displayed concurrently along a timeline  160 . In particular, a focus frame  150 , as described in further detail below with reference to  FIG. 5 , can be determined by a user as a focus point. The browser  100  is used to display and edit video frames, as described in further detail below with reference to  FIGS. 2 and 3 . The browser  100  can include a file browser  110 . A user can obtain movie data by using the file browser  110  to locate the movie data. Video frames  155  corresponding to the movie data are then displayed concurrently, and in temporal order, along the timeline  160  in a three-dimensional display environment, as described in further detail below with reference to  FIG. 13 . 
         [0024]    The three-dimensional display environment can be implemented in many ways. In some implementations, a reflective surface is used to reinforce the three-dimensional characteristic of the environment. Reflections of video frames, image analysis tools, etc. appear in the reflective surface. In some implementations, the three-dimensional display environment is composed of blank space. For example, the environment can have a background of a pure color, e.g., black. In other implementations, the three-dimensional display environment can be displayed using a grid. For example, the background can include a visual grid that displays an x-, y-, and z-axis. 
         [0025]    The timeline  160  is not constrained to the form of a straight line. In some implementations, the timeline is displayed in the form of a geometric shape. For example, the timeline can be displayed in the form of a rectangle, triangle, circle, or an oval. The timeline can also be displayed in the form of a linear path other than a straight line. For example, the timeline can be displayed in a parabola, sine wave, cosine wave, triangular wave, or square wave. 
       Video Frame Navigation and Selection 
       [0026]    The user interacts with the browser  100  by manipulating navigational elements  120 , navigational controls  130 , and selection controls  140 . The navigational elements  120  can allow the user to switch among different camera angles to view the three-dimensional display environment. For example, the user can view the environment from 360 degrees along any axis, e.g.,  FIGS. 5 ,  8 , and  10 . In some implementations, the navigational controls allow the user to zoom in on a particular area of the three-dimensional display environment, e.g., FIGS.  5  and  7 - 11 . For example, the user can select pre-determined zoom views or the user can manually zoom in to a particular area of the three-dimensional display environment. In yet other implementations, the user uses the navigational elements to flip or rotate the video frames  155  to view them from a different angle, e.g.,  FIGS. 8 and 10 . 
         [0027]    Navigation controls  130  are used to control the playback speed of the movie data or video frames. Examples of such navigation controls  130  include: play, rewind, fast forward, pause, next frame, and previous frame. In some implementations, the navigational controls are used to navigate among different sets of movie data that are located by using the file browser  110 . Examples of such navigation controls  130  include: next movie and previous movie. 
         [0028]    Selection controls  140  are used to select video frames  155  from the movie data. For example, selection controls  140  can include: start, end, reset, and export. The start and end controls are used to select one or more video frames  155 , and the export control is used to export these video frames as a new set of movie data. 
         [0029]    In some implementations, the navigation controls  130  and selection controls  140  include controls relating to different user interactions. For example, touch motions (e.g., sliding a finger over the screen of an iPhone) can be used to navigate among or select video frames. As another example, pinching along the screen of an iPhone can be used to zoom in and out of views. As yet another example, one or more taps on the screen of an iPhone can be used to select video frames. As another example, the scroll wheel of an iPod can be used to navigate among and select video frames. Additional controls relating to other user interactions are possible. 
         [0030]    In some implementations, the navigation controls  130  and selection controls  140  are used to display or edit one or more movies in the browser. For example, a user can use the file browser  110  to display a first movie and a second movie. The user can then navigate among the video frames (e.g., using the navigational controls  130 ), select video frames (e.g., using selection controls  140 ), and export video frames, for example, from the first and second movies (e.g., using the selection controls  140 ), to create a third movie that can be browsed or edited in the browser. 
       Image Editing 
       [0031]      FIG. 2  is a screen shot showing the three-dimensional content browser or editor with video frames displayed concurrently along a timeline of  FIG. 1  with video frames selected  210  for image editing. The user selects one or more video frames to edit using different image processing tools  200 . 
         [0032]    In some implementations, the user selects individual video frames. For example, the use can perform a mouse-click on an individual video frame to select the individual video frame. In other implementations, the user selects two points along the timeline  210 . By selecting the two points along the timeline, the video frames along the portion of the timeline between the two points are selected. In yet other implementations, the user uses the selection controls  140  to select a start and end time on the timeline. By selecting the two times on the timeline, the video frames along the portion of the timeline between the two times are selected. 
         [0033]    After video frames are selected, image processing tools  200  are used to edit the selected video frames. Examples of image processing controls include: geometric transformations such as enlargement, reduction and rotation; color corrections such as brightness and contrast adjustments, quantization, or conversion to a different color space; registration or alignments of images; combinations of images; interpolation and demosaicing of images; image segmentation; image editing and digital retouching; noise removal; and image restoration such as deconvolution. 
       Image Analysis 
       [0034]      FIG. 3  is a screen shot showing the three-dimensional content browser or editor with video frames displayed concurrently along a timeline of  FIG. 1  with image analysis tools  300  displayed along the timeline proximate to the video frames.  FIG. 4  shows another view of the screenshot in  FIG. 3 .  FIG. 4  is a screen shot showing the video frames displayed concurrently along a timeline with image analysis tools  400  displayed along the timeline proximate to the video frames of  FIG. 3  at a zoomed-in view. 
         [0035]    Image analysis tools (e.g., histogram  300  and histogram  400 ) allow a user to analyze information from the video frames. In addition, image analysis tools can be used to monitor the information from the video frames as they are edited. Examples of image analysis tools include: image histograms, image projections, image apertures, image profiles, image scattergrams, motion graphs (e.g., graphs that show the optical flow or amount of motion between frames), and image transforms (e.g., Hough Transforms and Fast Fourier Transforms). 
         [0036]    In some implementations, the image analysis tools can be displayed by using controls for the image processing tools  200 . For example, a color correction tool can have a check-box to display image histograms. When the check-box is enabled, image histograms (e.g., histogram  300  and histogram  400 ) are displayed along the timeline proximate to the video frames. In some implementations, the image analysis tools can have independent display controls. 
         [0037]    Image analysis tools can be displayed in various locations in the three-dimensional display environment. In some implementations, an image analysis tool is displayed above the timeline proximate to the video frame it analyzes. In some implementations, the image analysis tool is displayed in front of the timeline proximate to the video frame it analyzes. Alternatively, the image analysis tool can also be displayed behind the video frame it analyzes. The image analysis tool can also remain hidden from the user&#39;s view until the user selects a video frame or determines a focus frame. Upon selection of a video frame or determination of a focus frame, the image analysis tool is displayed proximate to the video frame it analyzes. 
       Focus Frame 
       [0038]      FIG. 5  is a screen shot showing a zoomed-in view of a timeline at a location of a focus frame  500 .  FIG. 7  shows another view of the screenshot in  FIG. 5 .  FIG. 7  is a screen shot showing a zoomed-in view of a timeline at a location of a focus frame with image analysis tools  700  displayed along the timeline proximate to video frames. 
         [0039]    Returning to  FIG. 5 , the user determines a focus frame  500  as a focus point. For example, the user can hover a cursor over a particular video frame, and that video frame becomes a focus frame. In some implementations, the user can hover a cursor over a time along the timeline proximate to a video frame, or use navigational controls to pause at a certain time corresponding to a video frame. This video frame becomes a focus frame. In other implementations, the user can perform a mouse click upon a video frame to determine the focus frame. In yet other implementations, users can determine more than one focus frame. Furthermore, the three-dimensional movie browser or editor can also determine a video frame to be a default focus frame. 
       Display Intervals 
       [0040]    After a focus frame is determined, the video frames are displayed at an interval on the timeline. 
         [0041]    The interval  510  can depend on the relative position of the video frames to the focus frame  500 . In some implementations, the interval decreases as the temporal distance between the video frames and the focus frame increases. For example, a first video frame located immediately in time before the focus frame is displayed at a first interval relative to the focus frame. A second video frame located immediately in time before the first video frame is displayed at a second interval relative to the first video frame. The second interval is smaller than the first interval. A third video frame located immediately in time before the second video frame is displayed at a third interval relative to the second video frame. The third interval is smaller than the second interval. The intervals for the frames located in time after the focus frame are displayed in a similar manner, based on the temporal distance between the video frames and the focus frame. The size of the interval decreases as the temporal distance between a video frame and the focus frame increases. In other implementations, the first interval is larger than the second interval, but the remaining intervals are equal to the second interval. 
         [0042]    Alternatively, the interval can depend on the optical flow, or amount of motion, between the frames. Displaying the video frames at an interval depending on the optical flow between the video frames allows a user to more easily view the most significant information of the video frames (e.g., a new movie shot or a change in a movie shot) while still viewing temporal information (e.g., how long a movie shot lasts and where the movie shot is located in the movie data). 
         [0043]      FIG. 6  is a screen shot showing a timeline where video frames are displayed at an interval depending on the optical flow between the video frames. Interval  600  is an interval between a first set of consecutive video frames with a first amount of motion between the first set of consecutive video frames. Interval  610  is an interval between a second set of consecutive video frames with a second amount of motion between the second set of consecutive video frames. Interval  600  is smaller than interval  610  because the amount of motion between the first set of consecutive video frames is less than the amount of motion between the second set of consecutive video frames. Interval  620  is an interval between a third set of consecutive video frames with a third amount of motion between the third set of consecutive video frames. Interval  620  is smaller than interval  610  because the amount of motion between the third set of consecutive video frames is less than the amount of motion between the second set of consecutive video frames. The size of the interval increases as the amount of motion between the frames increases. 
         [0044]    In some implementations, the interval can be created along an axis other than the axis of the timeline. For example, the interval can be created along an axis perpendicular to the timeline, so that the interval appears to create vertical spreading among the focus frame and the other video frames. 
       Display Resolutions 
       [0045]    The three-dimensional movie browser or editor can display the video frames at different resolutions. All of the video frames can be initially displayed at a first resolution. Returning to  FIG. 5 , after a focus frame is determined, the focus frame  500  is displayed at a second resolution. This second resolution is a greater resolution than the first resolution. 
         [0046]    In some implementations, the second resolution is a full temporal resolution. For example, the quad that contains the focus frame can be used to play all of the video frames corresponding to the movie data. In another example, the quad that contains the focus frame can be used to play the subset of the video frames that includes the focus frame and the video frames that temporally follow the focus frame. In yet another example, the quad that contains the focus frame can be used to play the subset of video frames that includes the video frames that temporally precede the focus frame and end at the focus frame. In some implementations, the second resolution is a full image resolution. 
       Other Views 
       [0047]      FIG. 8  is a screen shot showing a zoomed-in view of a timeline at a location of a focus frame  800  with video frames flipped or rotated. The user can use the navigational elements  120  to flip or rotate the video frames. In addition, the user can also use the navigational elements  120  to zoom in on the timeline. The video frames are displayed at an interval  810  along the timeline depending on the relative position of the video frames to the focus frame. In some implementations, the video frames are flipped or rotated so that the video frames are displayed horizontally, as shown in  FIG. 8 . In other implementations, the video frames are flipped or rotated so that the video frames are displayed vertically. 
         [0048]    Image analysis tools and image processing tools can still be displayed in zoomed-in views.  FIG. 9  is a screen shot showing the zoomed-in view of a timeline at a location of a focus frame with video frames flipped or rotated of  FIG. 8  with image analysis tools  900  displayed along the timeline proximate to the video frames. 
         [0049]    In some implementations, the three-dimensional movie browser or editor can display even further zoomed-in views.  FIG. 10  is a screen shot showing an even further zoomed-in view of a timeline at a location of a focus frame  1000  with video frames flipped or rotated. The video frames are displayed at an interval  1010  along the timeline depending on the relative position of the video frames to the focus frame.  FIG. 11  is a screen shot showing the even further zoomed-in view of a timeline at a location of a focus frame with video frames flipped or rotated of  FIG. 10  with image analysis tools  1100  displayed along the timeline proximate to the video frames. 
       Tracking Objects 
       [0050]      FIG. 12A  illustrates an example of a three-dimensional movie browser or editor with objects  1220  and  1230  in a video frame  1210  identified for tracking. A user uses tracking controls  1200  to identify an object or objects (e.g., objects  1220  and  1230 ) for tracking. For example, the user can first determine a focus frame by performing a mouse click on a video frame. Then, the user can use the tracking controls  1200  and mouse clicks to identify objects  1220  and  1230  to be tracked. 
         [0051]      FIG. 12B  illustrates an example of video frames displayed concurrently along a timeline with linear paths  1240  and  1250  displayed along the timeline displaying the movement of each identified object. For each video frame along the timeline, the browser plots points indicating the location of an identified object in the video frames. The browser then connects the points and displays a linear path along the timeline to show the movement of the object in the video frames. For example, the linear path  1240  displays the movement of the object  1230  in the video frames. The linear path  1250  displays the movement of the object  1220  in the video frames. 
         [0052]    In some implementations, the points indicating the location of an identified object in the video frames are displayed in the video frames. The browser can display a linear path that connects these points through the video frames. 
         [0053]    In some implementations, the linear path has disconnects. For example, if the identified object does not appear in certain video frames, the displayed linear path can have disconnects to indicate the absence of the identified object from the video frames. 
       Example Method to Display Video Frames 
       [0054]      FIG. 13  shows a method to display video frames in a three-dimensional movie browser or editor. Implementations of this method are described in further detail above with respect to  FIGS. 1-12 . The method includes obtaining movie data ( 1300 ). Obtaining movie data can include using a file browser  110  to locate movie data. The movie data can be located in memory, storage devices, computer-readable mediums, etc. 
         [0055]    After obtaining movie data, the browser displays video frames at a first resolution in a three-dimensional display environment ( 1310 ). The user can determine a focus frame ( 1320 ). After determining a focus frame, the browser displays the video frames at an interval depending on the relative position of the video frames to the focus frame ( 1330 ). The browser also displays the focus frame at a second resolution ( 1340 ). 
       Example Method to Track Objects 
       [0056]      FIG. 14  shows a method to track objects in video frames displayed in a three-dimensional movie browser or editor. Implementations of this method are described in further detail above with respect to  FIGS. 12A-12B . The method includes obtaining movie data ( 1400 ). Obtaining movie data can include using a file browser  110  to locate movie data. The movie data can be located in memory, storage devices, computer-readable mediums, etc. 
         [0057]    After obtaining movie data, the browser displays video frames in a three-dimensional display environment ( 1410 ). The user can identify one or more objects in a video frame to track ( 1420 ). After determining one or more objects to track, the browser displays a location of the one or more objects in each of the video frames along the timeline ( 1430 ). The browser also displays a linear path along the timeline showing the movement of each identified object ( 1440 ). 
       Software Stack 
       [0058]      FIG. 15  is a block diagram of an example software stack  1500 . In some implementations, the software stack includes an operating system OS kernel  1510  (e.g., a UNIX kernel), a library system  1520 , application frameworks  1530  and an applications layer  1540 . 
         [0059]    The OS kernel  1510  manages the resources and allows other programs to run and use these resources. Some examples of resources include processor, memory, and I/O. For example, the kernel  1510  can determine which running processes should be allocated to a processor, processors or processor cores, allocates memory to the processes and allocates requests from applications and remote services to perform I/O operations. In some implementations, the kernel  1510  provides methods for synchronization and inter-process communications with other devices. 
         [0060]    The library system  1520  provides various services applications running in the application layer. Such services can include audio services, video services, database services, image processing services, graphics services, etc. For example, the browser can make use of such services as QuickTime®, Core Image®, Core Video®, and OpenGL®. 
         [0061]    The application framework  1530  provides an object-oriented application environment including classes and Application Programming Interfaces APIs (e.g., Quartz Composer®) that can be used by developers to build applications using well-known programming languages (e.g., Objective-C, Java). 
         [0062]    The applications layer  1540  is where various applications exist in the software stack  1500 . Developers can use the APIs and environment provided by the application framework  1530  to build applications, such as the three-dimensional movie browser or editor. 
       System Architecture 
       [0063]      FIG. 16  is a block diagram of an example system architecture  1600 . The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor  1602  coupled by a bus system  1610  to receive data and instructions from, and to transmit data and instructions to, a data storage system  1612 , at least one input device  1608 , at least one output device  1604 , and at least one network interface  1606 . A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language (e.g., Objective-C, Java), including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. 
         [0064]    Suitable processors for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one of multiple processors or cores, of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, ASICs application-specific integrated circuits. 
         [0065]    To provide for interaction with a user, the features can be implemented on a computer having a display device such as a CRT cathode ray tube or LCD liquid crystal display monitor for displaying information to the user and a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer. 
         [0066]    The features can be implemented in a computer system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system can be connected by any form or medium of digital data communication such as a communication network via a network interface. Examples of communication networks include, e.g., a LAN, a WAN, and the computers and networks forming the Internet. 
         [0067]    The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. 
         [0068]    In an implementation, the system  1600  includes instructions defining an operating system  1614  stored in a data storage system  1612 . Example operating systems can include the MAC OS® X series operating system, the WINDOWS® based operating system, or other operating systems. Upon execution of the operating system instructions, access to various system objects is enabled. Example system objects include data files, the three-dimensional movie browser or editor  1622 , other applications  1624 , functions, windows, etc. The network communication module  1616  allows the data storage system  1608  to communicate with the system  1600 . 
         [0069]    To facilitate an intuitive user experience, the system  1600  includes a graphical user interface that provides the user access to the various system objects and conveys information about the system  1600  to the user in an intuitive manner. Upon execution, the UI engine  1618  and the graphics engine  1620  generate a graphical user interface at an output device  1604 . 
         [0070]    While the three-dimensional movie browser or editor is described herein with respect to a computer system, it should be apparent that the disclosed implementations can be incorporated in, or integrated with, any electronics, media players, game devices, mobile phones, email devices, personal digital assistants (PDAs), embedded devices, televisions, etc. For example, a browser can be used on any user interface including but not limited to hardware devices (e.g., desktops, iPods®, and iPhones®), browser or application windows (e.g., QuickTime®, iTunes®, Final Cut®, and iMovie®), and menu systems (e.g., DVD menus and DVD scene searching tools). 
         [0071]    A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. As yet another example, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.