Blurred image detection for text recognition

Techniques are described for identifying blurred images and recognizing text. One or more images of text may be captured. A change of movement associated with each image of the one or more images may be calculated. The change of movement associated with an image of the one or more images represents a change in an amount of acceleration of the device used to capture the image while the image was being captured. A steady image may be selected from the one or more images to use for text recognition. The steady image can be selected using the variances of acceleration associated with each image of the one or more images.

BACKGROUND

In various situations, it is useful to convert text to machine-encoded text. Oftentimes, to perform optical character recognition, a page containing printed text is electronically captured using a flatbed scanner, mounted camera, or some other equivalent device. The electronically captured image of the printed page is processed to create machine-encoded text that is an accurate representation of the information contained in the printed text. However, in some circumstances, the printed text is not scanned using a fixed position device. Rather, a handheld device, such as a camera, is used to capture an image of the text that is to be converted to machine-encoded text. The use of a handheld device to capture images can result in blurred pictures. This blurring is often due to unintentional movement of a user's hand, arm, and/or body while the user is attempting to hold the handheld device still at the time an image is being captured. As the amount of time to capture an image increases (e.g., the longer the camera's shutter is open), the amount of blurring in the image can be expected to increase. While some techniques for reducing blur in captured images exist, these techniques tend to require significant computing power and can be difficult to execute efficiently on a handheld device due to factors such as the device's processing capabilities and its power source.

SUMMARY

In some embodiments, a system for recognizing text is presented. The system may include an image capture device configured to capture one or more images. Each image of the one or more images may comprise text. A movement measurement device may be configured to measure movement of the image capture device. A processor may be configured to determine, using the movement measurement device, a change of movement value associated with each of the one or more images. Each change of movement value represents a change in an amount of movement of the image capture device while a corresponding one of the one or more images was being captured. The processor may also be configured to select a particular image from the one or more images to use for text recognition. The particular image may be selected using the change of movement value associated with each image of the one or more images.

The processor may be further configured to perform text recognition on at least some of the text of the particular image. The processor may be further configured to determine whether the particular image is desirable to use for text recognition. The processor may be further configured to select the particular image from the one or more images to use for text recognition based on the change of movement value associated with the particular image being less than a threshold change of movement value. The processor may be further configured to select the particular image from the one or more images to use for text recognition based on the change of movement value associated with the particular image having the smallest magnitude as compared with the change of movement values associated with other images of one or more images and the one or more images comprise at least two images. The processor, the image capture device, and the movement measurement device may be part of a cellular phone. The image capture device may be selected from a group consisting of: a still camera; and a video camera. The movement measurement device may be selected from a group consisting of: an accelerometer; and a gyroscope. The movement measurement device may be an acceleration measurement device; and the change of movement values associated with each of the one or more images may be variances of acceleration values.

In some embodiments, a method for capturing sharper images is presented. The method may include capturing, by a computerized device, one or more images. The method may also include determining, by the computerized device, a change of movement value of the computerized device associated with each image of the one or more images. Each change of movement value may represent a change in an amount of movement of the computerized device while a corresponding one of the one or more images was being captured. The method may also include selecting, by the computerized device, a particular image from the one or more images. The particular image may be selected using the change of movement value associated with each image of the one or more images.

The method may further include determining whether the particular image is desirable to use for text recognition, wherein the one or more images comprises text. Selecting the particular image from the one or more images may be based on the change of movement value associated with the particular image being less than a threshold change of movement value. The computerized device may include an accelerometer. The computerized device may include a cellular phone. The one or more images include at least two images. Each change of movement value may be a variance of acceleration value. The computerized device may comprise an image capture device used to capture the one or more images, the image capture device selected from a group consisting of: a still camera; and a video camera.

In some embodiments, an apparatus for capturing a steady image is presented. The apparatus may include means for capturing one or more images. The apparatus may include means for detecting movement of the means for capturing the one or more images. The apparatus may include means for determining a change of movement value associated with each image of the one or more images. Each change of movement value may represent a change in an amount of movement of the means for capturing the one or more images while a corresponding one of the one or more images was being captured. The apparatus may include means for selecting a particular image from the one or more images. The particular image may be selected using the change of movement value associated with each image of the one or more images.

The apparatus may include means for performing text recognition on text of the particular image. The particular image may be selected from the one or more images to use for text recognition based on the change of movement value associated with the particular image being less than a threshold change of movement value. While the image capture device is capturing each image of the one or more images, a person may be holding the means for capturing the one or more images and the person is attempting to hold the means for capturing the one or more images stationary. The particular image may be selected from the one or more images to use for text recognition based on the change of movement value associated with the particular image having the smallest magnitude as compared with the change of movement values associated with other images of the one or more images. The one or more images may include at least two images.

In some embodiments, a computer program product residing on a non-transitory processor-readable medium and comprising processor-readable instructions is presented. The instruction may be configured to cause a processor to induce a set of one or more images to be captured. The instruction may be configured to cause the processor to determine a change of movement value associated with each image of the one or more images. Each change of movement value may represent a change in an amount of movement of an image capture device while a corresponding one of the one or more images was being captured by the image capture device. The instruction may also be configured to cause the processor to select a particular image from the one or more images. The particular image may be selected using the change of movement value associated with each image of the one or more images.

The instruction may be further configured to cause the processor to comprising processor-readable instructions configured to cause the processor to perform text recognition on text of the particular image. The instruction may be further configured to cause the processor to use acceleration measurements to determine each change of movement value associated with each image of the one or more images. The instruction may be further configured to cause the processor to select the particular image from the one or more images to use for text recognition based on the change of movement value associated with the particular image being less than a threshold change of movement value. The instruction may be further configured to cause the processor to select the particular image from the one or more images based on the change of movement value associated with the particular image having the smallest magnitude as compared with the change of movement values associated with other images of the one or more images, wherein the one or more images comprises at least two images.

DETAILED DESCRIPTION

Rather than attempting to reduce or eliminate the blurring in an image captured by a handheld image capture device, images may be captured until a sharp image is captured while the image capture device is held sufficiently motionless. By using changes in movement, such as variances in acceleration, it can be determined with a good probability whether an image will be sufficiently sharp for text present in the image to be converted to machine-encoded text. In order to obtain a sufficiently sharp image, one or more images may be captured until the change in movement of the image capture device at the time the image is captured is below a threshold value. In some configurations, multiple images are captured and variance of acceleration values are associated with each image. The image associated with the smallest variance in acceleration (or, more generally, the smallest change in movement value) may be considered as likely having the least amount of blurring. This steady image can be used for text recognition or for some other purpose, such as imaging a barcode, taking a sharp picture, or recognizing an object.

Using a sharp image for text recognition or some other form of image processing can be more efficient than processing a blurry image to make it sharper because less computing power may be used to capture multiple frames and select a sharp frame than process either a de-blurring algorithm that creates a clean image from a blurred image or perform an image-based motion detection process that detects motion by analyzing pixel movement between images taken in succession. Using either a de-blurring algorithm or an image-based motion detection process can be computationally demanding, and thus may take too long and/or consume too much power if executed on a handheld device that has a relatively slow processor and/or is powered by a battery.

In various categories of handheld devices, such as cellular phones (including smart phones), still cameras, webcams, and video cameras, movement measuring devices are incorporated to measure motion. These devices, such as accelerometers, are likely to be fairly inaccurate in determining the acceleration and/or velocity of the handheld device. While the acceleration and/or movement measured by a movement measuring device may not be suitably accurate, these same movement measuring devices may provide acceptably accurate changes in movement (e.g., variances in acceleration). While a device with no change in movement or variance in acceleration may be accelerating at a constant rate, in practice if the device is handheld and the variance in acceleration is roughly zero, the device is likely stationary and can be considered to be roughly motionless.

Therefore, rather than attempting to remove some or all of the blurring in a single image captured by a handheld device, one or more images may be captured by the handheld device until an image is captured when the change in movement of the handheld device is at or below a threshold value. This image is processed to convert text in the image to machine-encoded text. While the configurations described herein focus on text recognition, similar techniques can be used to capture information besides text, such as barcodes, pictures, symbols, or other visually-displayed information.

FIG. 1illustrates an embodiment100of an image capture device120being used to capture text150that is to be converted to machine-encoded text. Image capture device120can be a still camera configured to capture only still images. Image capture device120can also be a video camera that is configured to capture frames (e.g., images) in rapid succession to be played back as video. Image capture device120can also be capable of functioning as both a still camera and a video camera. In some configurations, image capture device120is a standalone device, such as a dedicated still camera or video camera. In other configurations, image capture device120is incorporated as part of a computerized device, such as cellular phone110. Cellular phone110and image capture device120are handheld by a user. A user's hand130is illustrated as holding cellular phone110. Typically, despite the user's efforts to hold cellular phone110motionless while an image is being captured, natural movement of the user's hand, arm, and/or body, results in some motion of cellular phone110. This natural movement can result in movement in any three dimensional direction, as illustrated by imaginary dotted arrows170.

In embodiment100, the user is attempting to capture an image of text150(which reads “Text Text”). Text150is present on surface140. Surface140can represent any surface displaying text that a user may wish to capture. For example, surface140may be a billboard, sign, advertisement, digital display, television screen, book, magazine, newspaper, or part of a vehicle. Of course, many other surfaces displaying text are also possible. In order to capture text150, the user may hold (with hand130) cellular phone110and aim image capture device120at text150. The field of view of image capture device120is illustrated by dotted lines160-1and160-2. In some instances, such as if surface140is a book, cellular phone110is held at a small distance, such as one foot, from surface140. In other instances, such as if surface140is a roadside billboard, cellular phone110is held at a greater distance, such as fifty feet, from surface140. In conjunction with the movement of the user, factors such as distance, amount being used, ambient lighting conditions (which can affect how long image capture device120requires to capture an image), and quality of image capture device120can influence the amount of blurring in images captured of text150.

Cellular phone110(or image capture device120) may contain other components not illustrated. For example, an accelerometer may be incorporated as part of cellular phone110. When one or more images of text150are captured by image capture device120, an accelerometer or other movement measurement device may measure the acceleration (or, more generally, the movement) of the image capture device120and/or cellular phone110at the time each of the one or more images is captured. Rather than directly using an accelerometer's measured amount of acceleration (or some other unit of movement measured by a movement measurement device), which can be inaccurate, the variance in acceleration (or other change in movement value) is computed using the measured amounts of acceleration (or measured amounts of movement) and is used to determine whether a captured image is likely to be sharp enough for text recognition. As such, blurred images are detected and not used for text recognition.

FIG. 2illustrates a sample piece of text captured by an image capture device with a low variance of acceleration. As can be seen, the text of Korean characters and Arabic numerals appears sharp in captured image200. However, inFIG. 3, this is not the case.FIG. 3illustrates the same piece of text ofFIG. 2captured by an image capture device with a higher variance of acceleration. Here, the text of Korean characters and Arabic numerals appear blurred in captured image300.

FIG. 4illustrates a block diagram of an embodiment400of a computerized device410capable of blurred image detection. Computerized device410may be a cellular phone, still camera, video camera, webcam, or some other handheld device capable of capturing an image. Computerized device410may include: processor420, image capture device430(which may represent image capture device120ofFIG. 1), movement measurement device440, non-transitory computer-readable storage medium450, and user input device460.

When a user wishes to capture text and for it to be converted into machine-encoded text, the user holds and points image capture device430of computerized device410at the text to be converted. Once the text is present in the field of view of image capture device430, the user provides an input to computerized device410via user input device460. User input device460is a button (e.g., a physical button or a soft button, which may be a location on a touch screen designated to function as a button), switch, or voice-activated device. In some configurations, user input device460is some other form of trigger that a user can use to indicate that text is to be captured and converted to machine-encoded text.

Processor420receives input from user input device460. Processor460can be a general purpose processor or a specialized processor. After receiving a signal from user input device460, processor460sends a signal to image capture device430to instruct it to capture one or more images. In some configurations, user input device460directly triggers image capture device430. Upon receiving a signal from processor420, image capture device captures a predefined number of images or may capture images until instructed otherwise by processor430. As discussed in relation to image capture device120, image capture device can be a still camera, video camera, or both.

For each image captured by image capture device430, a measurement by movement measurement device440is taken. As such, one or more measurements of movement, such as acceleration measurements, are associated with each image captured by image capture device430. These measurement(s) of movement are transmitted from movement measurement device440to processor420. At least partially based on the received movement measurements from movement measurement device440, processor420calculates the change in movement (such as the variance in acceleration) of the image capture device430(which is the change of movement of the movement measurement device440and computerized device410assuming they are packaged together). Once an image associated with an acceptably low change in movement has been captured, processor430may signal to image capture device430to cease capturing images or may not instruct image capture device430to capture additional images. Similarly, processor430may signal to movement measurement device440to cease additional measurements.

Some or all of the images captured by image capture device430are, at least temporarily, stored in non-transitory computer-readable storage medium450. Similarly, the associated movement, change in movement, acceleration and/or variance in acceleration calculations are stored using non-transitory computer-readable storage medium450. In some configurations, non-transitory computer-readable storage medium450is random access memory, flash memory, or a hard drive. Non-transitory computer-readable storage medium450also stores software executable by processor420to perform tasks such as determining the variance in acceleration, determining which images are likely to be blurry, determining which images are likely sharp enough for text recognition, and performing text recognition.

While computerized device410contains components to capture images, measure changes in movement, store data, and perform text recognition, other components are also present. For example, if computerized device410is a cellular phone, components to perform wireless communication functions, such as web browsing, data transfer, phone calls, and text messaging are present.

FIG. 5illustrates an embodiment of a method for blurred image detection. At stage510, an image containing text that is to be converted to machine encoded text is captured by an image capture device, such as image capture device120or image capture device430. While the image is being captured, a user holding the image capture device (or holding the device that contains the image capture device) is likely attempting to hold the image capture device stationary (with respect to the text being captured) to reduce blurring. While the user is attempting to hold the image capture device stationary, some amount of movement is expected due to natural motion of the user's hand, arm, and body.

At stage520, one or more movement measurements are taken at the same time, or roughly the same time, as when the image was captured at stage510. The movement measurement(s) are made by a movement measurement device, such as an accelerometer or a gyroscope. The movement measurements may be measurements of acceleration or velocity.

At stage530, a change in movement associated with the image captured at stage510is calculated using the movement measurement(s) taken at stage520. In some embodiments, these change in movement calculations are variances of acceleration. To determine the variance of acceleration, equation 1 is used.

In equation 1, M(t) indicates the variance of acceleration of the acceleration measurement device, which is coupled with the image capture device. Therefore, the greater the value of M(t), the more the image capture device is determined to be moving. The acceleration at time t is defined according to equation 2:
a(t)εR2Eq. 2

Using equation 1, the variance of acceleration is calculated in current time frame (t−N+1:t) where equation 3 (which is a portion of equation 1) represents the average acceleration at time (t−N+1:t).

Determining the motion of the image capture device using variances in acceleration, rather than acceleration measurements from an acceleration measurement device, may allow for the magnitude of motion of the image capture device to be determined more accurately.

At stage540, the image may or may not be selected for text recognition. In some configurations, the image is selected for text recognition if the value of M(t) is less than a predefined threshold value. If M(t) is not less than the threshold value, method500proceeds to stage560. At stage560, recognition of text (or any other symbol or object) present in the image captured at stage510is not attempted. In some configurations, the text (or other symbol or object), although not being converted to a machine-encoded format, is tracked. Method500then returns to stage510to capture another image and repeat method500. Since the user is expected to be attempting to hold the image capture device stationary, it can be expected that the same, or mostly the same, text will be imaged each time stage510is executed until stage550has been performed. Method500may repeat through these stages until an image is determined to have an M(t) value below the predefined threshold value. In some configurations, if an M(t) value below the predefined threshold value is not obtained within a predefined period of time, method500aborts without text recognition (or recognition of any other symbol or object) being performed.

If M(t) is less than the threshold value, method500proceeds to stage550. At stage550, text recognition is performed using the image that has been determined to have been captured while the image capture device was held steady. The machine-encoded text is then output to some other device or piece of software. If necessary, following the text being converted to machine-encoded text, the method returns to stage510to capture additional images for other text or in case the text conversion fails. In some configurations, method500ends after stage550. If method500ceases, the image capture device may be instructed to cease capturing images.

While method500discusses a steady image being captured for text recognition, a steady image may be captured for some other reason. For example, a steady image of many other non-textual entities may be desired. For example, a steady image of a person, object, barcode, art, or location are only a few possible examples.

In method500, if the user is holding the image capture device sufficiently still, the first image captured of the text might have a low enough associated M(t) that a second image does not need to be captured. In some configurations, multiple images are always captured. For example,FIG. 6illustrates an embodiment of method600for blurred image detection for text recognition that results in at least two images being captured whenever the method is executed. At stage610, two or more images containing the text the user wishes to convert to machine-encoded text are captured and at least temporarily stored. These images are captured in short succession, such as a tenth of a second, a half of a second, or a second apart.

At stage620, movement measurements are taken at the same times, or roughly the same times, as when the images were captured at stage610. Therefore, for each image captured at stage610, one or more associated movement measurements are made at the time the image was captured. The one or more movement measurements are made by a movement measurement device, such as an accelerometer or a gyroscope. The movement measurements may be acceleration measurements.

At stage630, the change in movement associated with each of the multiple images is calculated. The change in movement, may be a variance in acceleration, which can be calculated as previously described in relation to equations 1 through 3, is performed at stage630for each captured image.

At stage640, using the values of M(t) calculated for each image captured at stage610, an image to use for text recognition is selected. The image captured at stage610associated with the smallest magnitude value of M(t) is selected. In some configurations, the selected image must also be associated with a lower value of M(t) than a threshold value. If none of the values of M(t) are lower than the threshold value, method600can return to stage610to capture additional images and repeat the method.

At stage650, text recognition is performed on the selected steady image. While stage650discusses text recognition, a selected steady image may be captured for some other reason. For example, a steady image of many other non-textual entities may be desired. For example, a steady image of a person, object, barcode, art, or location are only a few possible examples.

The threshold value of M(t) used to determine whether an image is blurry or not may be adjusted such that greater or fewer images are identified as blurry.FIG. 7illustrates a graph of the likelihood of a false alarm versus the likelihood of detecting blurred images. On the y-axis, the percentage of images determined to be blurred, that is, determined to have an M(t) value above a threshold, is graphed against the false alarm rate on the x-axis, that is, the percentage of images that are identified as blurry but are actually sufficiently sharp for successful text recognition. Therefore, as the threshold value is lowered and more images are determined to be blurred, the number of images that are sufficiently sharp for text recognition but are identified as blurry based upon the variance in acceleration of the image capture device also increases.

A computer system as illustrated inFIG. 8may incorporate as part of the previously described computerized devices.FIG. 8provides a schematic illustration of one embodiment of a computer system800that can perform the methods provided by various other embodiments, as described herein, and/or can function as the host computer system, a remote kiosk/terminal, a point-of-sale device, a mobile device, and/or a computer system. It should be noted thatFIG. 8is meant only to provide a generalized illustration of various components, any or all of which may be utilized as appropriate.FIG. 8, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner.

The computer system800is shown comprising hardware elements that can be electrically coupled via a bus805(or may otherwise be in communication, as appropriate). The hardware elements may include one or more processors810, including without limitation one or more general-purpose processors and/or one or more special-purpose processors (such as digital signal processing chips, graphics acceleration processors, and/or the like); one or more input devices815, which can include without limitation a mouse, a keyboard and/or the like; and one or more output devices820, which can include without limitation a display device, a printer and/or the like.

The computer system800might also include a communications subsystem830, which can include without limitation a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device and/or chipset (such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, cellular communication facilities, etc.), and/or the like. The communications subsystem830may permit data to be exchanged with a network (such as the network described below, to name one example), other computer systems, and/or any other devices described herein. In many embodiments, the computer system800will further comprise a working memory835, which can include a RAM or ROM device, as described above.

As mentioned above, in one aspect, some embodiments may employ a computer system (such as the computer system800) to perform methods in accordance with various embodiments of the invention. According to a set of embodiments, some or all of the procedures of such methods are performed by the computer system800in response to processor810executing one or more sequences of one or more instructions (which might be incorporated into the operating system840and/or other code, such as an application program845) contained in the working memory835. Such instructions may be read into the working memory835from another computer-readable medium, such as one or more of the storage device(s)825. Merely by way of example, execution of the sequences of instructions contained in the working memory835might cause the processor(s)810to perform one or more procedures of the methods described herein.

The terms “machine-readable medium” and “computer-readable medium,” as used herein, refer to any medium that participates in providing data that causes a machine to operate in a specific fashion. In an embodiment implemented using the computer system800, various computer-readable media might be involved in providing instructions/code to processor(s)810for execution and/or might be used to store and/or carry such instructions/code (e.g., as signals). In many implementations, a computer-readable medium is a physical and/or tangible storage medium. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical and/or magnetic disks, such as the storage device(s)825. Volatile media include, without limitation, dynamic memory, such as the working memory835. Transmission media include, without limitation, coaxial cables, copper wire and fiber optics, including the wires that comprise the bus805, as well as the various components of the communication subsystem830(and/or the media by which the communications subsystem830provides communication with other devices). Hence, transmission media can also take the form of waves (including without limitation radio, acoustic and/or light waves, such as those generated during radio-wave and infrared data communications).

The communications subsystem830(and/or components thereof) generally will receive the signals, and the bus805then might carry the signals (and/or the data, instructions, etc. carried by the signals) to the working memory835, from which the processor(s)805retrieves and executes the instructions. The instructions received by the working memory835may optionally be stored on a storage device825either before or after execution by the processor(s)810.