Method to stabilize video stream using on-device positional sensors

A video device is provided. The video device comprises a sensor to determine an amount of movement of the video device, a memory component to store video-related information, and a processor to receive movement information from the sensor and to interrupt communication of video-related information to the memory component when the movement information indicates that acceleration of the video device exceeds a threshold.

BACKGROUND

Digital video cameras typically include an optical sensor capable of converting an optical image into an electronic signal. Digital information associated with the electronic signal may then be stored in a memory component. An image derived from the electronic signal may be displayed on a viewfinder on the video camera for viewing by the photographer. A moving image might be captured as a series of frames that can be stored and then later replayed in sequence to recreate the moving image.

DETAILED DESCRIPTION

Implementations of the present disclosure can stabilize the video stream recorded by video cameras and other devices capable of capturing moving images. As used herein, the term “video device” can refer to any apparatus capable of capturing moving images, regardless of whether the apparatus is a stand-alone video camera or a multi-function device with videography capabilities, such as a smart phone, a tablet computer, or a similar apparatus.

In an implementation, a motion sensing component in a video device is capable of measuring the amount of movement the video device is undergoing. When the amount of movement is below a low threshold, it can be assumed that the video device is experiencing only a minimal amount of motion. For example, any movement the video device is undergoing might be caused only by instability in the photographer's hands, in such cases, the video device might continue to record images in its usual manner. When the amount of movement is above a high threshold, it can be assumed that the photographer is deliberately moving the video device in order to follow a moving subject. In such cases, the video device might also continue to record images in its usual manner. When the amount of movement is above the low threshold and below the high threshold, it can be assumed that the video device has undergone a sudden and substantial movement, such as a movement that might be caused when the photographer is jostled or the video device is dropped. In such cases, the video device might temporarily stop sending electronic signals to its memory component, thus preventing the recording of images during the period of jostling, in this way, the playback of a video recording will not include images that entered the video device's lens during the period of jostling, and erratic scenes will be eliminated. In some such cases, the video device might also stop sending live images to its viewfinder.

In an alternative implementation, rather than a low movement threshold and a high movement threshold being defined, an acceleration threshold might be defined. When the movement of a video device exceeds the acceleration threshold, the video device might temporarily stop sending electronic signals to its memory component.

FIG. 1is a simplified block diagram of components that might be present in a video device100capable of functioning in this manner. The video device100includes a lens110, an optical sensor120, and a processor130that can work in the manner well known to those of skill in the ah to capture video images. More specifically, the lens110might send an optical image to the optical sensor120, which might convert the optical image to an electronic signal that can then be sent to the processor130. The processor130might perform additional processing on the electronic signal and can then send a digital representation of the electronic signal to a memory component140for storage. The memory component140might he a removable memory card or some other type of storage medium. The processor130can also send the electronic signal to a viewfinder150for display of the images that are generated by the optical sensor120and the processor130. That is, the processor130might create two data streams, a first data stream160going to the memory140and a second data stream170going to the viewfinder150.

The video device100can also include a positional sensor180capable of measuring the amount of movement the video device100is undergoing. The positional sensor180might be a gyroscope, an accelerometer, a magnetometer, or any other component that can measure changes in the position of the video device100. The positional sensor180might provide its motion information to the processor130.

The video device100might also include other components not shown, such as control buttons or other well-known input mechanisms. Also, one or more additional memory components may be present and may store instructions executable by the processor130in carrying out the implementations described herein. In addition, it should be understood that the components illustrated inFIG. 1are not necessarily discrete components as shown, but could be any combination of hardware, firmware, software, and mechanical components capable of capturing video images. For example, the functions of one of the components inFIG. 1might be shared by one or more of the other components, or the functions of multiple components could be combined into a single component.

In an implementation, the positional sensor180can measure the amount of movement of the video device100over a period of time and can provide the motion measurements to the processor130. In one set of implementations, movements of the video device100are classified as either low movements, sustained movements, or jolts. In another set of implementations, the acceleration of the video device100is determined from the motion measurements, and an acceleration that exceeds a threshold is classified as a jolt.

In the first set of implementations, any movement of the video device100that might occur when no deliberate movement of the video device100is occurring and when no accidental movement of the video device100is occurring could be considered to be less than a low movement threshold and could be classified as a low movement. An accidental movement might be defined as a large-scale movement that is not intended by the photographer, such as movement that might occur when the photographer is jostled or drops the video device100. An example of a low movement might be the motion caused by minor trembling in the photographer's hands. Any noise in the positional sensor180that could appear to be actual motion might also be classified as a low movement. The detection of low movement could result in the video device100operating in its standard manner. That is, the memory140and the viewfinder150could continue to be updated with images generated by the optical sensor120and the processor130.

If the positional sensor180measures sustained, large-scale movement of the video device100, the video device100can be considered to exceed a high movement threshold. An example of sustained movement might be the motion that occurs when the photographer moves the video device100to follow a moving subject. In such sustained movement cases, the video device100could operate in its standard manner by continuing to update the memory140and the viewfinder150with images generated by the optical sensor120and the processor130.

However, if the positional sensor180measures movement of the video device100that is larger than the low movement threshold but smaller than the high movement threshold, the movement of the video device100could be classified as a “jolt”. That is, a jolt may be defined as a brief, sudden movement that is more extensive than a movement that might be expected when a photographer is attempting to keep the video device100still and is more sudden than a movement that might be expected when a photographer is deliberately moving the video device100to follow a moving object. The low movement threshold and the high movement threshold could be defined in such a way that a movement is classified as a jolt when the size of a movement and the period of time over which that movement occurs are characteristic of the video device100being bumped or otherwise moving in an extensive and unintended manner.

In the second set of implementations, a jolt could also be defined in terms of acceleration. That is, small unintentional movements, such as those caused by instability in the photographer's hands, would typically have a small rate of change of velocity over time and would thus have a small acceleration. When a photographer moves the video device100to follow a moving object, the photographer typically attempts to do so in a smooth manner, without excessive jerking of the video device100. Such intentional movements would also be likely to have a small rate of change of velocity over time and thus would also have a small acceleration. Either of these types of movement could be defined to not be jolts and would not result in any change in the traditional operation of the video device100. On the other hand, large unintentional movements, such as those that might occur when the photographer is jostled or drops the video device100, would be likely to have a large rate of change of velocity over time and thus would have a large acceleration and could be considered jolts. A large acceleration of the video device100might also occur when the photographer deliberately moves the video device100very quickly, such as when the photographer suddenly redirects the video device100from one subject to another. Such a deliberate fast movement of the video device100could also be considered a jolt. Therefore, a jolt could be defined as a movement that exceeds a specified level of acceleration.

In either of these sets of implementations, when a jolt is detected, the processor130can temporarily stop sending data to the memory140. That is, the data stream160from the processor130to the memory140can be temporarily interrupted upon detection of a jolt, and the memory140might not be updated with live images during the interruption. As used herein, the term “live image” can refer to the most recent image generated by the optical sensor120and the processor130.

The duration of the temporary interruption of the data stream160to the memory140might be determined in various ways. In one implementation, no data is sent to the memory140for as long as the jolt is being detected. That is, data flow on the data stream160to the memory140does not resume until the positional sensor180detects that the movement of the video device100is below the low movement threshold or above the high movement threshold. Alternatively, data flow on the data stream160to the memory140does not resume until the positional sensor180detects that the acceleration of the video device100is below the acceleration threshold. In another implementation, the interruption of the data stream160might continue for a predetermined period of time. In another implementation, the photographer might take a manual action to cause the resumption of data flow on the data stream160to the memory140. For example, when a jolt occurs, a notification may appear on the viewfinder150indicating that recording has been temporarily interrupted and that a manual action is needed for recording to resume. When the photographer wishes to resume recording, the photographer might press a button, tap an area in the viewfinder150, or take some other action on an input mechanism to cause data flow on the data stream160to resume.

Interrupting the data stream160in this manner can cause a played back video recording to appear to jump from the last image recorded before a jolt to the first image recorded after the jolt. Throughout the duration of the jolt, the memory140could be managed in at least two different manners. In one implementation, no updates are made to the memory140during the period of the jolt. In this case, the played back recording would appear to jump without a pause from the last image recorded before the jolt to the first image recorded after the jolt. In another implementation, the last image recorded before the jolt could be copied to the memory140at every frame at which a live image would have been stored if the jolt had not occurred. For example, if a jolt lasts for ten frames, a copy of the last image recorded before the jolt could be copied to the memory140in each of those ten frames. When the recording is played back, the recording would appear to pause on that image for the duration of ten frames and would then jump to the first image recorded after the jolt.

In an implementation, when a jolt is detected, in addition to the processor130temporarily interrupting the data stream160to the memory140, the display of live images on the viewfinder150might temporarily be interrupted. In one implementation, the viewfinder150might be blank throughout the duration of the jolt. In another implementation, the last image displayed on the viewfinder150before the jolt occurred might continue to be displayed on the viewfinder150as a temporary placeholder until recording resumes. The appearance of such a still image in the viewfinder150could also serve as an indicator that recording has been temporarily interrupted. These implementations could prevent a live image in the viewfinder150from moving erratically during the jolt and possibly disorienting the photographer.

However, these implementations could also present a difficulty to the photographer in properly framing an image when the photographer wishes to resume recording, since the viewfinder150would display either a blank screen or the last recorded image rather than the scene toward which the video device100is currently pointed. The photographer might not be able to determine if the scene is properly framed until after recording has resumed and live images reappear on the viewfinder150.

In an implementation, such a difficulty could be overcome in the case where the photographer is provided with the option of manually resuming recording after a jolt, as described above. In this implementation, two images could appear in the viewfinder150after a jolt has occurred and before recording has resumed. One of the images could be a live image of the scene that is currently entering the lens110, and the other image could be a still image of the last scene recorded before the jolt occurred. The photographer could view the live image and ensure that the live image is framed as desired before resuming the recording. When recording resumes, the still image might disappear, and only the live image might remain in the viewfinder150.

This implementation could also provide a smooth transition between the last image recorded before the jolt and the first image recorded after the jolt when the video recording is played back. When the photographer is ready to resume recording, the photographer could compare the two images in the viewfinder150and try to match the live image to the still image. If the photographer frames the live image to approximately match the still image and then manually resumes recording, the played back recording could show only a minimal amount of jumping between the last image recorded before the jolt and the first image recorded after the jolt. In this way, the jolt could, in effect, be edited out of the recording.

In another implementation, throughout the duration of the jolt, the viewfinder150might display either a blank screen or the last image recorded before the jolt began. When the jolt ends, the viewfinder150might resume displaying live images. However, recording of images to the memory140might not resume until the photographer provides an input into the video device100, such as pushing a button or tapping on the viewfinder150.

In the implementations described above, images are not recorded when a jolt occurs. In another implementation, when a jolt occurs, images are stored either in the memory140or in some other temporary storage location. The photographer can decide at a later time whether to retain the images that were recorded during the jolt or to delete those images.

As mentioned above, the video device100might be a stand-alone video camera or might be a multi-function device with videography capabilities, such as a smart phone, a tablet computer, or a similar device.FIGS. 2aand2bdepict front and rear views, respectively, of a smart phone200that might have components and capabilities similar to those of the video device100. A display screen on the front of the smart phone200can act as the viewfinder150when the smart phone200is being used as a video camera. The lens110might be located on the rear of the smart phone200. Other components, such as those shown inFIG. 1, may be present internally in the smart phone200and are not shown inFIGS. 2aand2b.

FIG. 3illustrates a method for stabilizing a video stream in the implementations where a low movement threshold and a high movement threshold are used. At box310, a positional sensor on a video device takes a movement measurement. At box320, it is determined whether the movement is above a low movement threshold. If the movement is above the low movement threshold then, at box330, it is determined whether the movement is below a high movement threshold. If movement is below the high movement threshold then, at box340, video-related data is not sent to a memory component on the video device. One of skill in the art will recognize that the decisions at boxes320and330do not necessarily need to occur in the order shown. If it is determined at box320that the movement is not above the low movement threshold, or if it is determined at box330that the movement is not below the high movement threshold then, at box350, video-related data is sent to the memory component in the traditional manner. After the determination is made whether or not to send data to the memory component, another movement measurement might be made at box310, and the procedure can be repeated.

As mentioned above, the implementations described herein may be implemented by a smart phone or some other type of user equipment (UE). One exemplary device is described below with regard toFIG. 4and may be equivalent to the smart phone illustrated inFIG. 2. UE3200is typically a two-way wireless communication device having voice and data communication capabilities. UE3200generally has the capability to communicate with other computer systems on the Internet. Depending on the exact functionality provided, the UE may be referred to as a data messaging device, a two-way pager, a wireless e-mail device, a cellular telephone with data messaging capabilities, a wireless Internet appliance, a wireless device, a mobile device, or a data communication device, as examples.

Where UE3200is enabled for two-way communication, it may incorporate a communication subsystem3211, including a receiver3212and a transmitter3214, as well as associated components such as one or more antenna elements3216and3218, local oscillators (LOs)3213, and a processing module such as a digital signal processor (DSP)3220. As will be apparent to those skilled in the field of communications, the particular design of the communication subsystem3211will be dependent upon the communication network in which the device is intended to operate.

Network access requirements will also vary depending upon the type of network3219. In some networks network access is associated with a subscriber or user of UE3200. A UE may require a removable user identity module (RUIM) or a subscriber identity module (SIM) card in order to operate on a network. The SIM/RUIM interface3244is normally similar to a card-slot into which a SIM/RUIM card can be inserted and ejected. The SIM/RUIM card can have memory and hold many key configurations3251, and other information3253such as identification, and subscriber related information.

When required network registration or activation procedures have been completed, UE3200may send and receive communication signals over the network3219. As illustrated inFIG. 4, network3219can consist of multiple base stations communicating with the UE.

Signals received by antenna3216through communication network3219are input to receiver3212, which may perform such common receiver functions as signal amplification, frequency down conversion, filtering, channel selection and the like. Analog to digital (A/D) conversion of a received signal allows more complex communication functions such as demodulation and decoding to be performed in the DSP3220. In a similar manner, signals to be transmitted are processed, including modulation and encoding for example, by DSP3220and input to transmitter3214for digital to analog (D/A) conversion, frequency up conversion, filtering, amplification and transmission over the communication network3219via antenna3213, DSP3220not only processes communication signals, but also provides for receiver and transmitter control. For example, the gains applied to communication signals in receiver3212and transmitter3214may be adaptively controlled through automatic gain control algorithms implemented in DSP3220.

UE3200generally includes a processor3236which controls the overall operation of the device. The processor3238ofFIG. 4may be equivalent to the processor130ofFIG. 1. Communication functions, including data and voice communications, are performed through communication subsystem3211. Processor3238also interacts with further device subsystems such as the display3222, flash memory3224, random access memory (RAM)3226, auxiliary input/output (I/O) subsystems3228, serial port3230, one or more keyboards or keypads3232, speaker3234, microphone3236, other communication subsystem3240such as a short-range communications subsystem and any other device subsystems generally designated as3242. Serial port3230could include a USB port or other port known to those in the art.

Some of the subsystems shown inFIG. 4perform communication-related functions, whereas other subsystems may provide “resident” or on-device functions. Notably, some subsystems, such as keyboard3232and display3222, for example, may be used for both communication-related functions, such as entering a text message for transmission over a communication network, and device-resident functions such as a calculator or task list.

Operating system software used by the processor3238may be stored in a persistent store such as flash memory3224, which may instead be a read-only memory (ROM) or similar storage element (not shown). Those skilled in the art will appreciate that the operating system, specific device applications, or parts thereof, may be temporarily loaded into a volatile memory such as RAM3226. Received communication signals may also be stored in RAM3226.

As shown, flash memory3224can be segregated into different areas for both computer programs3258and program data storage3250,3252,3254and3256. These different storage types indicate that each program can allocate a portion of flash memory3224for their own data storage requirements. Processor3233, in addition to its operating system functions, may enable execution of software applications on the UE. A predetermined set of applications that control basic operations, including at least data and voice communication applications for example, will normally be installed on UE3200during manufacturing. Other applications could be installed subsequently or dynamically.

Applications and software may be stored on any computer readable storage medium. The computer readable storage medium may be a tangible or in transitory/non-transitory medium such as optical (e.g., CD, DVD, etc.), magnetic (e.g., tape) or other memory known in the art.

One software application may be a personal information manager (PIM) application having the ability to organize and manage data items relating to the user of the UE such as, but not limited to, e-mail, calendar events, voice malls, appointments, and task items. Naturally, one or more memory stores may be available on the UE to facilitate storage of PIM data items. Such PIM application may have the ability to send and receive data items, via the wireless network3219. Further applications may also be loaded onto the UE3200through the network3219, an auxiliary I/O subsystem3228, serial port3230, short-range communications subsystem3240or any other suitable subsystem3242, and installed by a user in the RAM3226or a non-volatile store (not shown) for execution by the processor3238. Such flexibility in application installation increases the functionality of the device and may provide enhanced on-device functions, communication-related functions, or both. For example, secure communication applications may enable electronic commerce functions and other such financial transactions to be performed using the UE3200.

In a date communication mode, a received signal such as a text message or web page download will be processed by the communication subsystem3211and input to the processor3238, which may further process the received signal for output to the display3222, or alternatively to an auxiliary I/O device3228.

A user of UE3200may also compose data items such as email messages for example, using the keyboard3232, which may be a complete alphanumeric keyboard or telephone-type keypad, among others, in conjunction with the display3222and possibly an auxiliary I/O device3223. Such composed items may then be transmitted over a communication network through the communication subsystem3211.

For voice communications, overall operation of UE3200is similar, except that received signals may typically be output to a speaker3234and signals for transmission may be generated by a microphone3236. Alternative voice or audio I/O subsystems, such as a voice message recording subsystem, may also be implemented on UE3200. Although voice or audio signal output is preferably accomplished primarily through the speaker3234, display3222may also be used to provide an indication of the identity of a calling party, the duration of a voice call, or other voice call related information for example.

Serial port3230inFIG. 4may normally be implemented in a personal digital assistant (PDA)-type UE for which synchronization with a users desktop computer (not shown) may be desirable, but is an optional device component. Such a port3230may enable a user to set preferences through an external device or software application and may extend the capabilities of UE3200by providing for information or software downloads to UE3200other than through a wireless communication network. The alternate download path may for example be used to load an encryption key onto the device through a direct and thus reliable and trusted connection to thereby enable secure device communication. As will be appreciated by those skilled in the art, serial port3230can further be used to connect the UE to a computer to act as a modem.

Other communications subsystems3240, such as a short-range communications subsystem, is a further optional component which may provide for communication between UE3200and different systems or devices, which need not necessarily be similar devices. For example, the subsystem3240may include an infrared device and associated circuits and components or a Bluetooth™ communication module to provide for communication with similarly enabled systems and devices. Subsystem3240may further include non-cellular communications such as WiFi or WiMAX.

In an implementation, a method for stabilizing a video stream being captured by a video device is provided. The method comprises, when movement of the video device is detected by a sensor on the video device to be greater than a low threshold and less than a high threshold, temporarily interrupting communication of video-related information to a memory component on the video device.

In another implementation, a video device is provided. The video device comprises a sensor to determine an amount of movement of the video device, a memory component to store video-related information, and a processor coupled to the sensor and the memory component. The processor is configured to receive movement information from the sensor and to interrupt communication of video-related information to the memory component when the movement information indicates that acceleration of the video device exceeds a threshold.

In another implementation, a method for managing a video recording on a video device is provided. The method comprises interrupting communication of video-related information to a memory component on the video device when movement information provided by a sensor on the video device indicates that acceleration of the video device exceeds a threshold.