Abstract:
A method and apparatus for dynamically maintaining a horizontal framing of a video. The system permits the user to freely rotate the device while filming, while visualizing the final output in an overly on the device viewfinder or screen during shooting. The resulting recording is subsequently corrected to maintain a single orientation with a stable horizon. The system and method is operative display an overly over a captured representation of the captured video wherein the overlay indicates a modified image with respect to said orientation

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims priority from U.S. Provisional Application No. 62/003,281 filed May 27, 2014 having attorney docket number PU140089. 
     
    
     BACKGROUND OF THE INVENTION 
       [0002]    Portable electronic devices are becoming more ubiquitous. These devices, such as mobile phones, music players, cameras, tablets and the like often contain a combination of devices, thus rendering carrying multiple objects redundant. For example, current touch screen mobile phones, such as the Apple iPhone or Samsung Galaxy android phone contain video and still cameras, global positioning navigation system, internet browser, text and telephone, video and music player, and more. These devices are often enabled an multiple networks, such as wifi, wired, and cellular, such as 3G, to transmit and received data. 
         [0003]    The quality of secondary features in portable electronics has been constantly improving. For example, early “camera phones” consisted of low resolution sensors with fixed focus lenses and no flash. Today, many mobile phones include full high definition video capabilities, editing and filtering tools, as well as high definition displays. With this improved capabilities, many users are using these devices as their primary photography devices. Hence, there is a demand for even more improved performance and professional grade embedded photography tools. 
         [0004]    For example, many videos on mobile devices are recorded in a manner where the user may inadvertently rotate the mobile device, thereby tilting the video horizon the vertical orientation of the video for the viewer. In an extreme case, a user may start filming with the camera in a vertical orientation and change to a horizontal orientation. This would result in a video which starts out oriented properly, but ends up rotated 90 degrees when being displayed to a viewer. To correct this problem post processing is required, which is an undesirable option for a user wishing to directly share the video via a social network. 
         [0005]    Additionally, recording a video with the mobile device in a vertical position often results in a video which is taller than it is wide. This end result is not optimal for consumption on most displays, such as television screens, which are typically wider than they are tall. In many cases users shoot video without specific attention to horizontal orientation, especially when filming a social activity, live event or other subject matter where the user is engaged in the experience that takes their focus off the device they are recording with. Further, most mobile phones are designed to be used in a vertical orientation. Thus, a user may start using the device in its intended orientation, only to realize later that video should be filmed in a horizontal orientation. 
         [0006]    Thus, it is desirable to overcome these problems with current video cameras embedded in mobile electronic devices. 
       SUMMARY OF THE INVENTION 
       [0007]    A method and apparatus for dynamically maintaining a horizontal framing of a video. The system permits the user to freely rotate the device while filming, while visualizing the final output in an overlay on the device viewfinder or screen during shooting. The resulting recording is subsequently corrected to maintain a single orientation with a stable horizon. The system and method is operative display an overlay over a captured representation of the captured video wherein the overlay indicates a modified image with respect to said orientation. 
         [0008]    In one embodiment, the present invention also involves a method of processing a video stream. The method comprises the steps of initializing a video capture mode, receiving data representing a video stream, displaying a representation of said video stream, receiving a data representing an aspect ratio, receiving a data representing rotational position, interpolating an orientation based on the received data representing rotational position, and overlaying a graphic representative of said aspect ratio and said orientation over said representation of said video stream. 
         [0009]    In another embodiment, an apparatus is provided. The apparatus comprises an image sensor, an inertial sensor, a processor, and a memory. The image sensor is for capturing image data having a first orientation. The inertial sensor is for determining rotational values. The processor is for determining a second orientation based on said rotational values received from inertial sensor, interpolating an intermediate orientation based on the first and second orientations, and reorienting said image data from said first orientation to said second orientation through said intermediate orientation. The memory is coupled to the processor and is for storing image data and reorientated image data. 
         [0010]    Another embodiment involves a method of saving image data comprising the steps of receiving data representing a first image having a first orientation, receiving data representing a second orientation indicating a device vertical orientation with respect to gravity, reorienting said first image such that said second orientation becomes a vertical orientation of said first image to generate a reoriented image, wherein the framing of the image is maintained, and saving said reoriented image. 
         [0011]    Another embodiment involves a method of processing a video stream comprising the steps of initializing a video capture mode, receiving a first data representing a video stream, displaying a representation of said video stream, receiving a second data representing an aspect ratio, receiving a third data representing a rotational position, and overlaying a graphic representative of said aspect ratio and said rotational position over said representation of said video stream wherein the framing of video is maintained. 
         [0012]    In another aspect, the present disclosure also involves an apparatus comprising an image sensor for capturing an image data having a first orientation, an inertial sensor for determine a rotational value, a processor for determining a second orientation in response to said rotational value and for reorienting said image data in response to said second orientation while maintaining the framing of the image data to generate a reoriented image having the same framing, and storing said reoriented image. 
     
    
     
       DETAILED DESCRIPTION OF THE DRAWINGS 
         [0013]    These and other aspects, features and advantages of the present disclosure will be described or become apparent from the following detailed description of the preferred embodiments, which is to be read in connection with the accompanying drawings. In the drawings, wherein like reference numerals denote similar elements throughout the views: 
           [0014]      FIG. 1  shows a block diagram of an exemplary embodiment of mobile electronic device; 
           [0015]      FIG. 2  shows an exemplary mobile device display having an active display according to the present invention; 
           [0016]      FIG. 3  shows an exemplary process for image stabilization and reframing in accordance with the present disclosure; 
           [0017]      FIG. 4  shows an exemplary mobile device display having a capture initialization  400  according to the present invention; and 
           [0018]      FIG. 5  shows an exemplary process for initiating an image or video capture  500  in accordance with the present disclosure. 
           [0019]      FIG. 6  shows and exemplary process for tracking rotation during video capture in accordance with the present disclosure; 
           [0020]      FIG. 7  shows and exemplary process for tracking rotation during image capture in accordance with the present disclosure; and 
           [0021]      FIG. 8  shows and exemplary process for maintaining framing of a shot with rotations during video capture accordance with the present disclosure. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
       [0022]    The examples set out herein illustrate preferred embodiments of the invention, and such examples are not to be construed as limiting the scope of the invention in any manner. 
         [0023]    Referring to  FIG. 1 , a block diagram of an exemplary embodiment of mobile electronic device is shown. While the depicted mobile electronic device is a mobile phone  100 , the invention may equally be implemented on any number of devices, such as music players, cameras, tablets, global positioning navigation systems etc. A mobile phone typically includes the ability to send and receive phone calls and text messages, interface with the Internet either through the cellular network or a local wireless network, take pictures and videos, play back audio and video content, and run applications such as word processing, programs, or video games. Many mobile phones include GPS and also include a touch screen panel as part of the user interface. 
         [0024]    The mobile phone includes a main processor  150  that is coupled to each of the other major components. The main processor  150  may be a single processor or more than one processor as known by one skilled in the art. The main processor  150 , or processors, routes the information between the various components, such as the network interfaces  110 ,  120 , camera  140 , inertial sensor  170 , touch screen  180 , and other input/output I/O interfaces  190 . The main processor  150  also processes audio and video content for play back either directly on the device or on an external device through the audio/video interface. The main processor  150  is operative to control the various sub devices, such as the camera  140 , inertial sensor  170  touch screen  180 , and the USB interface  130 . The main processor  150  is further operative to execute subroutines in the mobile phone used to manipulate data similar to a computer. For example, the main processor may be used to manipulate image files after a photo has been taken by the camera function  140 . These manipulations may include cropping, compression, color and brightness adjustment, and the like. 
         [0025]    The cell network interface  110  is controlled by the main processor  150  and is used to receive and transmit information over a cellular wireless network. This information may be encoded in various formats, such as time division multiple access (TDMA), code division multiple access (CDMA) or Orthogonal frequency-division multiplexing (OFDM). Information is transmitted and received from the device trough the cell network interface  110 . The interface may consist of multiple antennas encoders, demodulators and the like used to encode and decode information into the appropriate formats for transmission. The cell network interface  110  may be used to facilitate voice or text transmissions, or transmit and receive information from the internet. The information may include video, audio, and/or images. 
         [0026]    The wireless network interface  120 , or wifi network interface, is used to transmit and receive information over a wifi network. This information can be encoded in various formats according to different wifi standards, such as 802.11g, 802.11b, 802.11ac and the like. The interface may consist of multiple antennas encoders, demodulators and the like used to encode and decode information into the appropriate formats for transmission and decode information for demodulation. The wifi network interface  120  may be used to facilitate voice or text transmissions, or transmit and receive information from the internet. This information may include video, audio, and/or images. 
         [0027]    The universal serial bus (USB) interface  130  is used to transmit and receive information over a wired link, typically to a computer or other USB enabled device. The USB interface  120  can be used to transmit and receive information, connect to the internet, transmit and receive voice and text calls, etc. Additionally, the wired link may be used to connect the USB enabled device to another network using the mobile devices cell network interface  110  or the wifi network interface  120 . The USB interface  130  can be used by the main processor  150  to send and receive configuration information to a computer. 
         [0028]    A memory  160 , or storage device, may be coupled to the main processor  150 . The memory  160  may be used for storing specific information related to operation of the mobile device and needed by the main processor  150 . The memory  160  may be used for storing audio, video, photos, or other data stored and retrieved by a user. 
         [0029]    The inertial sensor  170  may be a gyroscope, accelerometer, axis orientation sensor, light sensor or the like, which is used to determine a horizontal and/or vertical indication of the position of the mobile device. 
         [0030]    The input output (I/O) interface  190 , includes buttons, a speaker/microphone for use with phone calls, audio recording and playback, or voice activation control. The mobile device may include a touch screen  180  coupled to the main processor  150  through a touch screen controller. The touch screen  180  may be either a single touch or multi touch screen using one or more of a capacitive and resistive touch sensor. The smartphone may also include additional user controls such as but not limited to an on/off button, an activation button, volume controls, ringer controls, and a multi-button keypad or keyboard 
         [0031]    Turning now to  FIG. 2 , an exemplary mobile device display having an active display  200  according to the present invention is shown. The exemplary mobile device application is operative for allowing a user to record in any framing and freely rotate their device while shooting, visualizing the final output in an overlay on the device&#39;s viewfinder during shooting and ultimately correcting for their orientation in the final output. 
         [0032]    According to the exemplary embodiment, when a user begins shooting the user&#39;s current orientation is taken into account and the vector of gravity based on the device&#39;s sensors is used to register a horizon. For each possible orientation, such as portrait  210 , where the device&#39;s screen and related optical sensor is taller than wide, or landscape  250 , where the device&#39;s screen and related optical sensor is wider than tall, an optimal target aspect ratio is chosen. An inset rectangle  225  is inscribed within the overall sensor that is best-fit to the maximum boundaries of the sensor given the desired optimal aspect ratio for the given (current) orientation. The boundaries of the sensor are slightly padded in order to provide ‘breathing room’ for correction. The inset rectangle  225  is transformed to compensate for rotation  220 ,  230 ,  240  by essentially rotating in the inverse of the device&#39;s own rotation, which is sampled from the device&#39;s integrated gyroscope. The transformed inner rectangle  225  is inscribed optimally inside the maximum available bounds of the overall sensor minus the padding. Depending on the device&#39;s current most orientation, the dimensions of the transformed inner rectangle  225  are adjusted to interpolate between the two optimal aspect ratios, relative to the amount of rotation. 
         [0033]    For example, if the optimal aspect ratio selected for portrait orientation was square (1:1) and the optimal aspect ratio selected for landscape orientation was wide (16:9), the inscribed rectangle would interpolate optimally between 1:1 and 16:9 as it is rotated from one orientation to another. The inscribed rectangle is sampled and then transformed to fit an optimal output dimension. For example, if the optimal output dimension is 4:3 and the sampled rectangle is 1:1, the sampled rectangle would either be aspect filled (fully filling the 1:1 area optically, cropping data as necessary) or aspect fit (fully fitting inside the 1:1 area optically, blacking out any unused area with ‘letter boxing’ or ‘pillar boxing’. In the end the result is a fixed aspect asset where the content framing adjusts based on the dynamically provided aspect ratio during correction. So for example a  16 : 9  video comprised of 1:1 to 16:9 content would oscillate between being optically filled  260  (during 16:9 portions) and fit with pillar boxing  250  (during 1:1 portions). 
         [0034]    Additional refinements whereby the total aggregate of all movement is considered and weighed into the selection of optimal output aspect ratio are in place. For example, if a user records a video that is ‘mostly landscape’ with a minority of portrait content, the output format will be a landscape aspect ratio (pillar boxing the portrait segments). If a user records a video that is mostly portrait the opposite applies (the video will be portrait and fill the output optically, cropping any landscape content that falls outside the bounds of the output rectangle). 
         [0035]    Referring now to  FIG. 3 , an exemplary process for image stabilization and reframing  300  in accordance with the present disclosure is shown. The system is initialized in response to the capture mode of the camera being initiated  310 . The initialization may be initiated according to a hardware or software button, or in response to another control signal generated in response to a user action. Once the capture mode of the device is initiated, the mobile device sensor  320  is chosen in response to user selections. User selections may be made through a setting on the touch screen device, through a menu system, or in response to how the button is actuated. For example, a button that is pushed once may select a photo sensor, while a button that is held down continuously may indicate a video sensor. Additionally, holding a button for a predetermined time, such as 3 seconds, may indicate that a video has been selected and video recording on the mobile device will continue until the button is actuated a second time. 
         [0036]    Once the appropriate capture sensor is selected, the system then requests a measurement from an inertial sensor  330 . The inertial sensor may be a gyroscope, accelerometer, axis orientation sensor, light sensor or the like, which is used to determine a horizontal and/or vertical indication of the position of the mobile device. The measurement sensor may send periodic measurements to the controlling processor thereby continuously indicating the vertical and/or horizontal orientation of the mobile device. Thus, as the device is rotated, the controlling processor can continuously update the display and save the video or image in a way which has a continuous consistent horizon. 
         [0037]    After the inertial sensor has returned an indication of the vertical and/or horizontal orientation of the mobile device, the mobile device depicts an inset rectangle on the display indicating the captured orientation of the video or image  340 . As the mobile device is rotated, the system processor continuously synchronizes inset rectangle with the rotational measurement received from the inertial sensor  350 . The representation of the orientation may be checked, further be refined, or improved by analyzing the image using techniques such as line recognition, facial recognition, or the like. 
         [0038]    The user may optionally indicate a preferred final video or image ration, such as 1:1, 9:16, 16:9, or any other ratio selected by the user. The system may also store user selections for different ratios according to orientation of the mobile device. For example, the user may indicate a 1:1 ratio for video recorded in the vertical orientation, but a 16:9 ratio for video recorded in the horizontal orientation. In this instance, the system may continuously or incrementally rescale video  360  as the mobile device is rotated. Thus a video may start out with a 1:1 orientation, but could gradually be rescaled to end in a 16:9 orientation in response to a user rotating from a vertical to horizontal orientation while filming. Optionally, a user may indicate that the beginning or ending orientation determines the final ratio of the video. 
         [0039]    In certain other embodiments, the framing of the image inside the output box is maintained regardless of the aspect ratio. Thus, no matter how the device is oriented, scene is zoomed in or out to keep consistent framing. In some such cases, the object of interest in the video would stay the same size regardless of orientation and aspect ratio. In other cases, where maximum resolution is desired, the zooming in and out may change the size of the object of interest. Alternately, the image may be cropped to maintain the size regardless of orientation and aspect ratio. 
         [0040]    Turning now to  FIG. 4 , an exemplary mobile device display having a capture initialization  400  is shown. An exemplary mobile device is show depicting a touch tone display for capturing images or video. According to an aspect of the present invention, the capture mode of the exemplary device may be initiated in response to a number of actions. Any of hardware buttons  410  of the mobile device may be depressed to initiate the capture sequence. Alternatively, a software button  420  may be activated through the touch screen to initiate the capture sequence. The software button  420  may be overlaid on the image  430  displayed on the touch screen. The image  430  acts as a viewfinder indicating the current image being captured by the image sensor. An inscribed rectangle  440 , as described previously, may also be overlaid on the image to indicate an aspect ratio of the image or video to be captured. 
         [0041]    Referring now to  FIG. 5 , an exemplary process for initiating an image or video capture  500  in accordance with the present disclosure is shown. Once the imaging software has been initiated, the system waits for an indication to initiate image capture. Once the image capture indication has been received by the main processor  510 , the device begins to save the data sent from the image sensor  520 . In addition, the system initiates a timer. The system then continues to capture data from the image sensor as video data. In response to a second indication from the capture indication, indicating that capture has been ceased  530 , the system stops saving data from the image sensor and stops the timer. 
         [0042]    The system then compares the timer value to a predetermined time threshold  540 . The predetermined time threshold may be a default value determined by the software provider, such as 1 second for example, or it may be a configurable setting determined by a user. If the timer value is less than the predetermined threshold  540 , the system determines that a still image was desired and saves  560  the first frame of the video capture as a still image in a still image format, such as jpeg or the like. The system may optionally chose another frame as the still image. If the timer value is greater than the predetermined threshold  540 , the system determines that a video capture was desired. The system then saves  550  the capture data as a video file in a video file format, such as mpeg or the like. The system may then return to the initialization mode, waiting for the capture mode to be initiated again. If the mobile device is equipped with different sensors for still image capture and video capture, the system may optionally save a still image from the still image sensor and start saving capture data from the video image sensor. When the timer value is compared to the predetermined time threshold, the desired data is saved, while the unwanted data is not saved. For example, if the timer value exceeds the threshold time value, the video data is saved and the image data is discarded. 
         [0043]    One potential issue that may be encountered is that the measurements from the inertial sensor and the times stamps of the video or still image may not be ideally synchronized. For example, the period for updated positional or rotational measurements and/or data may be too infrequent for the capture of video or images resulting in an inaccurate orientation for the video or image. This can result in choppy looking video. To address this positional or rotational data can be interpolated to determine an intermediate orientation. An example of this can be seen in  FIGS. 6 and 7 . 
         [0044]      FIG. 6  depicts a flow diagram  600  of an exemplary methodology for processing a video stream wherein rotation occurs during capture. At the most basic level, the methodology involves seven steps. The first step is initiating capture of a video stream (step  610 ). The next step is receiving data representing a video stream (step  620 ). A representation of the video stream can then be displayed (step  630 ). The fourth step is receiving data representing an aspect ratio (step  640 ). The fifth step is receiving data representing rotational position (step  650 ). The orientation can then be interpolated using the data representing rotational position (step  660 ) The last step of the basic method is overlaying a graphic representative of said aspect ratio and said orientation over said representation of said video stream (step  670 ). 
         [0045]      FIG. 7  depicts a flow diagram  700  of an exemplary methodology for processing a still image wherein rotation occurs during capture. At the most basic level, the methodology involves seven steps. The first step is initializing a capture mode (step  710 ). The next step is receiving data representing an image (step  720 ). During capture data representing rotational position is received (step  730 ). The capture is then deactivated (step  740 ). The orientation can then be interpolated using the data representing rotational position (step  750 ). Once orientation is interpolated, the image can be rotated accordingly (step  760 ) The last step of the basic method is saving the rotated image (step  770 ). 
         [0046]    In some embodiments positional or rotational measurements can be obtained at the beginning and end of capture and the intermediate orientation can be interpolated. Such interpolation can further include the time stamp information of the video to more accurately orientate the video in accordance with time. Other techniques such as running average velocity can also be used to refine the interpolation. Other estimation techniques can also be used. In other embodiments, the frequency of rotational measurements may be increased during video capture to provide more accurate positional information in time with the video. 
         [0047]    In certain other embodiments, the framing of the image inside the output box is maintained regardless of the aspect ratio. Thus, no matter how the device is oriented, scene is zoomed in or out to keep consistent framing. An example of this can be seen in  FIG. 8 . 
         [0048]      FIG. 8  depicts a flow diagram  800  of an exemplary methodology for processing a video stream wherein rotation occurs during capture. At the most basic level, the methodology involves six steps. The first step is initiating capture of a video stream (step  610 ). The next step is receiving data representing a video stream (step  820 ). A representation of the video stream can then be displayed (step  830 ). The fourth step is receiving data representing an aspect ratio (step  840 ). The fifth step is receiving data representing rotational position (step  850 ). The last step of the basic method is overlaying a graphic representative of said aspect ratio and said orientation over said representation of said video stream wherein the framing of the original video is maintained in the overly (step  860 ). 
         [0049]    In some embodiments, the object of interest or subject in the video would stay the same size regardless of orientation and aspect ratio. In other embodiments, where maximum resolution is desired, the zooming in and out may change the size of the object of interest. Alternately, the image may be cropped to maintain the size regardless of orientation and aspect ratio. 
         [0050]    It should be understood that the elements shown and discussed above, may be implemented in various forms of hardware, software or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory and input/output interfaces. The present description illustrates the principles of the present disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its scope. All examples and conditional language recited herein are intended for informational purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure. Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herewith represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like represent various processes which may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.