Enhancing video using super-resolution

A method and apparatus for processing images. A portion of a selected image in which a moving object is present is identified. The selected image is one of a sequence of images. Pixels in a region of interest are identified in the selected image. First values are identified for a first portion of the pixels using the images and first transformations. The first portion of the pixels corresponds to the background in the selected image. A first transformation is configured to align features of the background between one image in the images and the selected image. Second values are identified for a second portion of the pixels using the images and second transformations. The second portion of the pixels corresponds to the moving object in the selected image. A second transformation is configured to align features of the moving object between one image in the images and the selected image.

BACKGROUND INFORMATION

The present disclosure relates generally to image processing and, in particular, to enhancing images using super-resolution. Still more particularly, the present disclosure relates to a method and apparatus for enhancing video containing moving objects using super-resolution.

Currently, many different algorithms are available for performing super-resolution. Super-resolution is a process in which images of the same scene are used to reconstruct an image with a higher resolution than the resolution of the images of the scene. As used herein, the term “resolution” is used to mean “spatial resolution”. Spatial resolution refers to the smallest possible feature that can be detected in an image. For example, smaller features can be detected in images having higher resolutions as compared to images having lower resolutions.

With super-resolution, multiple images of low resolution (LR) for a scene are used to reconstruct an image of high resolution (HR). Further, with super-resolution, the low resolution images used to construct the high resolution image cannot all be identical. Variation needs to be present between these low resolution images. For example, the low resolution images may be taken at different times, under different conditions, or both.

Typically, the low resolution images used to perform super-resolution are obtained from a sequence of images in the form of video. A sequence of images is a plurality of images ordered with respect to time. The different types of variation that may be present in these images include, for example, translational motion, rotational motion, different viewing angles, other types of motion, or any of these different types of variations in any combination. Translational motion between two images occurs when a current image of a scene is translated with respect to a previous image of the same scene parallel to the image plane. Rotational motion occurs when the current image is rotated with respect to the previous image. In some cases, variation is created when the current image is generated from a position closer to or further away from the scene as compared to the previous image.

With super-resolution, the low resolution images are first registered with respect to a selected reference coordinate system. Registration of these low resolution images includes aligning these images with respect to a reference coordinate system. For example, features in a first image may be aligned with the same features in a second image. However, when moving objects are present in the scene with respect to a background in the scene, registration of the low resolution images of the scene may not account for the moving objects. As a result, the high resolution image constructed from these low resolution images may be less accurate than desired.

Therefore, it would be advantageous to have a method and apparatus that takes into account at least some of the issues discussed above, as well as possibly other issues.

SUMMARY

In one advantageous embodiment, a method for processing images is provided. A portion of a selected image in which a moving object is present is identified. The selected image is one of a sequence of images. The moving object moves with respect to a background in the selected image. A plurality of pixels in a region of interest is identified in the selected image. First values are identified for a first portion of the plurality of pixels using the sequence of images and first transformations. The first portion of the plurality of pixels corresponds to the background in the selected image. A first transformation in the first transformations is configured to align features of the background in one image in the sequence of images to the features of the background in the selected image. Second values are identified for a second portion of the plurality of pixels using the sequence of images and second transformations. The second portion of the plurality of pixels corresponds to the moving object in the selected image. A second transformation in the second transformations is configured to align features of the moving object in the one image in the sequence of images to the features of the moving object in the selected image.

In another advantageous embodiment, a method for enhancing an image is provided. A portion of a selected image in which a moving object is present is identified. The selected image is one of a sequence of images. The moving object moves with respect to a background in the selected image. A plurality of pixels in a region of interest is identified in the selected image. First values are identified for a first portion of the plurality of pixels using the sequence of images and first transformations. The first portion of the plurality of pixels corresponds to the background in the selected image. A first transformation in the first transformations is configured to align features of the background in one image in the sequence of images to the features of the background in the selected image. Second values are identified for a second portion of the plurality of pixels using the selected image. The second portion of the plurality of pixels corresponds to the moving object in the selected image.

In yet another advantageous embodiment, an apparatus comprises an object tracking module and an enhancement module. The object tracking module is configured to identify a portion of a selected image in which a moving object is present. The selected image is one of a sequence of images. The moving object moves with respect to a background in the selected image. The enhancement module is configured to identify a plurality of pixels in a region of interest in the selected image. The enhancement module is further configured to identify first values for a first portion of the plurality of pixels using the sequence of images and first transformations. The first portion of the plurality of pixels corresponds to the background in the selected image. A first transformation in the first transformations is configured to align features of the background in one image in the sequence of images to the features of the background in the selected image. The enhancement module is further configured to identify second values for a second portion of the plurality of pixels using the sequence of images and second transformations. The second portion of the plurality of pixels corresponds to the moving object in the selected image. A second transformation in the second transformations is configured to align the features of the moving object in the one image in the sequence of images to the features of the moving object in the selected image.

DETAILED DESCRIPTION

The different advantageous embodiments recognize and take into account several different considerations. For example, the different advantageous embodiments recognize and take into account that currently, many different algorithms for performing super-resolution are available. However, the different advantageous embodiments recognize and take into account that some of these currently-available algorithms may be incapable of handling video of a scene containing objects that move relative to a background of the scene. As one illustrative example, some of these currently-available algorithms for super-resolution may be unable to take into account the movement of vehicles traveling along a highway in a scene.

The different advantageous embodiments recognize and take into account that some currently-available methods for performing super-resolution have limits on the amount of motion in the images. The different advantageous embodiments recognize and take into account that it may be desirable to have a method that can enhance video using super-resolution without placing limits on the amount of motion in the images.

Further, the different advantageous embodiments recognize and take into account that other currently-available methods for enhancing video containing moving objects may use optical flows identified in the video. Optical flow in video is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between the video camera system and the scene. The different advantageous embodiments recognize and take into account that the currently-available methods for enhancing video using optical flows may produce results with more errors than desired. Additionally, these methods may require more time, processing resources, or both, than desired.

Thus, the different advantageous embodiments provide a method and apparatus for enhancing video using super-resolution. In particular, the different advantageous embodiments provide a method and apparatus that enhances video of a scene in which objects move relative to a background of the scene using super-resolution techniques.

In one advantageous embodiment, a method for processing images is provided. A portion of a selected image in which a moving object is present is identified. The selected image is one of a sequence of images. The moving object moves with respect to a background in the selected image. A plurality of pixels in a region of interest is identified in the selected image. First values are identified for a first portion of the plurality of pixels using the sequence of images and first transformations. The first portion of the plurality of pixels corresponds to the background in the selected image.

A first transformation in the first transformations is configured to align features of the background in one image in the sequence of images to the features of the background in the selected image. Second values are identified for a second portion of the plurality of pixels using the sequence of images and second transformations. The second portion of the plurality of pixels corresponds to the moving object in the selected image. A second transformation in the second transformations is configured to align features of the moving object in the one image in the sequence of images to the features of the moving object in the selected image.

With reference now toFIG. 1, an illustration of an image processing environment is depicted in accordance with an advantageous embodiment. In this illustrative example, image processing environment100includes imaging system102, imaging system103, and computer system104. Computer system104is located at control station105in this example. Computer system104is configured to process images generated by imaging system102and imaging system103.

As depicted, imaging system102is attached to unmanned aerial vehicle (UAV)106. Imaging system102is configured to generate images of scene108as unmanned aerial vehicle106flies over scene108. In this illustrative example, moving objects110are present in scene108. Moving objects110in scene108include vehicles112and people114.

The images that are generated by imaging system102form a video of scene108. A video is a sequence of images ordered with respect to time. As unmanned aerial vehicle106flies over scene108, the images generated may capture different portions of scene108over time, capture scene108from different viewing angles with respect to scene108, or both. As a result, variations are present between the images generated by imaging system102.

Additionally, imaging system103is attached to building111. Imaging system103is also configured to generate images that form a video of scene108. Further, imaging system103may move as the video of scene108is being generated. For example, imaging system103may pan scene108. In this manner, imaging system103may generate images that capture different portions of scene108over time, capture scene108from different viewing angles with respect to scene108, or both. As a result, variations are present between the images generated by imaging system103.

Imaging system102and imaging system103send the images generated for scene108to computer system104for processing. In particular, imaging system102and imaging system103send these images to computer system104using wireless communications link116and wireless communications link118, respectively. In this illustrative example, imaging system102and imaging system103may be configured to send the images generated for scene108to computer system104in substantially real-time. In other words, these imaging systems may send images to computer system104for processing as these images are being generated.

Computer system104is configured to process the images generated by imaging system102and the images generated by imaging system103to construct images of scene108that have a higher resolution as compared to the images generated by imaging system102and imaging system103. In particular, computer system104may use super-resolution to enhance the video generated by imaging system102and the video generated by imaging system103.

Further, computer system104takes into account the presence of moving objects110within scene108when enhancing the video generated by imaging system102and the video generated by imaging system103. In particular, computer system104enhances the portion of the videos corresponding to the background in the video independently of the portion of the videos corresponding to moving objects110captured in the videos using super-resolution.

With reference now toFIG. 2, an illustration of a block diagram of an image processing environment is depicted in accordance with an advantageous embodiment. Image processing environment100inFIG. 1is an example of one implementation for image processing environment200. In these illustrative examples, imaging processing environment200includes imaging system202and image processing system204. Image processing system204is configured to process images206generated by imaging system202.

As depicted, imaging system202is configured to generate images206of scene208. Scene208may be any physical area of interest, such as, for example, without limitation, an area in a city, a neighborhood block, an area in a forest, an area over a desert, a region of airspace, a region of space, a portion of a highway, an area inside a manufacturing facility, or some other suitable area of interest.

As depicted, each image in images206of scene208includes background209and foreground210. Background209is the part of an image that lies outside the one or more objects of interest captured by the image. Background209in an image may include, for example, without limitation, land, trees, bushes, desert sand, water, air, space, clouds, roads, mountains, valleys, a river, an ocean, buildings, manmade structures, or other types of objects, either alone or in combination, depending on scene208.

The one or more objects of interest in scene208captured by an image form foreground210of the image. For example, foreground210of an image in images206may capture number of moving objects211in scene208. As used herein, a “number of” items means one or more items. In this manner, a number of moving objects means one or more moving objects.

Number of moving objects211may include, for example, without limitation, at least one of a vehicle, an aircraft, a person, an animal, a mobile platform, or some other suitable type of object that is capable of moving. As used herein, the phrase “at least one of”, when used with a list of items, means different combinations of one or more of the listed items may be used and only one of each item in the list may be needed.

For example, “at least one of item A, item B, and item C” may include, for example, without limitation, item A or item A and item B. This example also may include item A, item B, and item C, or item B and item C. In other examples, “at least one of” may be, for example, without limitation, two of item A, one of item B, and ten of item C; four of item B and seven of item C; and other suitable combinations.

In these illustrative examples, imaging system202is configured to generate images206that form video212. When the camera system generates images206in the form of video212, imaging system202may be referred to as a camera system or a video camera system. Further, images206in video212may take the form of, for example, without limitation, visible light images, infrared images, radar images, and/or other suitable types of images, depending on the type of imaging system.

In these illustrative examples, imaging system202may be associated with platform214. Platform214may be selected from one of, for example, without limitation, a mobile platform, a stationary platform, a land-based structure, an aquatic-based structure, a space-based structure, an aircraft, a surface ship, a tank, a personnel carrier, a train, a spacecraft, a space station, a satellite, a submarine, an automobile, a power plant, a bridge, a dam, a manufacturing facility, a building, and some other suitable type of platform.

The association between imaging system202and platform214is a physical association in these depicted examples. Imaging system102attached to unmanned aerial vehicle106and imaging system103attached to building111inFIG. 1are examples of implementations for imaging system202associated with platform214.

In these illustrative examples, a first component, such as imaging system202, may be considered to be associated with a second component, such as platform214, by being secured to the second component, bonded to the second component, mounted to the second component, welded to the second component, fastened to the second component, connected to the second component in some other suitable manner, or in a combination of suitable manners. The first component also may be connected to the second component using a third component. The first component may also be considered to be associated with the second component by being formed as part of or as an extension of the second component.

Imaging system202sends video212to image processing system204for processing. Image processing system204may be implemented using hardware, software, or a combination of both. In these illustrative examples, image processing system204is implemented in computer system216. Computer system216may comprise a number of computers. When more than one computer is present in computer system216, these computers may be in communication with each other.

In one illustrative example, registration module218receives images206in video212as these images are generated. In other words, registration module218may receive images206in substantially real-time. Of course, in other illustrative examples, registration module218may receive images206in video212after all of images206have been generated.

Registration module218stores images206in image buffer224. In particular, image buffer224has length, N. In other words, image buffer224comprises N elements. Image buffer224can store up to N images at any given point in time in these N elements. Image buffer224takes the form of a circular buffer in these illustrative examples. A circular buffer is a buffer of fixed length that overwrites the oldest data in the buffer when the buffer is full.

In these illustrative examples, the one or more images stored in image buffer224form sequence of images226in image buffer224. Sequence of images226may include one to N consecutive images. In particular, sequence of images226includes current image228and set of previous images230. As used herein, a “set of” items means zero or more items. For example, a set of previous images is zero, one, or more previous images. In this manner, a set of previous images may be a null or empty set.

Current image228is stored in the first element in image buffer224, while set of previous images230are stored in the elements after the first element in image buffer224. In some illustrative examples, sequence of images226may be referred to as “joint frames”, and image buffer224may be referred to as a “joint frame buffer.”

Further, registration module218identifies first transformations232for registering images206with respect to background209in each of images206. Registering images comprises aligning the images with respect to common features, a common coordinate system, some other common frame of reference, or multiple frames of reference. First transformation234is an example of one of first transformations232. First transformation234is configured to align features of background209in one image in images206with features of background209in another image in images206.

In these illustrative examples, first transformations232may take the form of transformations selected from a group consisting of affine transformations, homographic transformations, some other suitable type of transformation, or a combination of transformations. An affine transformation may comprise one or more of a linear transformation, a rotation, a scaling, a translation, and a shear transformation.

Registration module218is configured to store first transformations232generated by registration module218in background transformation buffer236. In these illustrative examples, background transformation buffer236has length N and also may take the form of a circular buffer. Registration module218may store first transformations232to background transformation buffer236as first transformations232are generated.

Object tracking module220is configured to detect and track number of moving objects211in scene208in one or more of images206. Object tracking module220may use any number of currently-available methods for detecting and tracking number of moving objects211in images206. For example, object tracking module220may use moving object detection algorithms, object segmentation algorithms, a mean-shift tracking algorithm, the Lucas-Kanade method, other suitable types of algorithms, either alone or in combination, to detect and track number of moving objects211.

In particular, object tracking module220generates object masks238using images206. An object mask may also be referred to as an object map. Object mask240is an example of one of object masks238. Object mask240is generated for an image in images206that has been processed and added to image buffer224. In these illustrative examples, object mask240is a binary image. A binary image is an image in which each pixel has a value of either logic “1” or logic “0”.

For example, object mask240may have set of areas241that represents moving objects. In particular, each area in set of areas241represents a moving object that has been detected in the image for which object mask240was generated. Each of the pixels within set of areas241has a value of logic “1”, and each of the pixels outside set of areas241has a value of logic “0”. In this manner, pixels having a value of logic “1” in object mask240represent foreground210in the corresponding image. Further, pixels having a value of logic “0” in object mask240represent background209in the corresponding image.

Object tracking module220stores object masks238generated by object tracking module220in object mask buffer246. Object mask buffer246has length N and also may take the form of a circular buffer. Object tracking module220stores object masks238as object masks238are generated.

Additionally, object tracking module220also may be configured to generate sets of tracks242for moving objects detected in images206. Sets of tracks242may be generated using object masks238, images206, or both. In these illustrative examples, each set of tracks in sets of tracks242corresponds to an image in images206. Further, each track in a set of tracks is for a moving object detected in the corresponding image. In this manner, each set of tracks in sets of tracks242may correspond to set of areas241. In other words, a track in one of sets of tracks242may be generated for an object represented in a corresponding one of set of areas241in a corresponding image.

Track244is an example of a track in sets of tracks242generated for an image. Object tracking module220generates track244when moving object245is detected in an image in images206. In these illustrative examples, track244includes a track identifier that is unique to moving object245. In other words, tracks generated for different images may have the same track identifier when these tracks are generated for the same moving object in the different images.

Further, track244may include a location at which moving object245is depicted in the corresponding image. This location may be selected as, for example, a center of moving object245in the corresponding image. In these illustrative examples, this location may be defined using, for example, an x-y coordinate system for the image, some other suitable type of coordinate system, or a combination of coordinate systems for the image.

Track244also may include a size of moving object245in the corresponding image. The size of the object may be defined as minima and maxima for both the x and y coordinates for moving object245. In other illustrative examples, the location, size, or both location and size of moving object245included in track244may be defined with respect to the x-y coordinate system for the object mask in object masks238generated for the corresponding image.

In some illustrative examples, object tracking module220may replace the values of pixels in an area in set of areas241in object mask240with a track identifier for the track corresponding to the area. For example, an area in set of areas241in object mask240may represent moving object245. Object tracking module220may replace the values of pixels in this area with the track identifier for track244for moving object245. This type of object mask may be referred to as an augmented object mask.

In this manner, each pixel in an object mask may have a value selected from one of a first value and a second value. When the object mask is a regular object mask, the first value may be a logic “0” and the second value may be a logic “1”. When the object mask is an augmented object mask, the first value may be a logic “0” and the second value may be a track identifier.

Registration module218uses images206, object masks238, and sets of tracks242to generate second transformations250for each one of number of moving objects211detected in sequence of images226. Similar to first transformations232, second transformations250may take the form of, for example, without limitation, transformations selected from the group consisting of affine transformations, homographic transformations, some other suitable type of transformation, or a combination of transformations.

As one illustrative example, second transformations250may be generated for moving object245. Second transformation251is an example of one of second transformations250. Second transformation251may be configured to align features of moving object245in one image in images206with features of moving object245in another image in images206.

In these illustrative examples, registration module218stores second transformations250for moving object245in object transformation buffer252. Object transformation buffer252has length N and also may take the form of a circular buffer. A different object transformation buffer may be generated for each moving object detected in images206. Registration module218stores second transformations250in object transformation buffer252as second transformations250are generated.

In these illustrative examples, enhancement module222uses sequence of images226stored in image buffer224, first transformations232stored in background transformation buffer236, and second transformations250stored in object transformation buffer252to enhance video212generated by imaging system202. In particular, enhancement module222uses sequence of images226, first transformations232, and second transformations250to enhance region of interest254for selected image256in sequence of images226.

For example, enhancement module222selects region of interest254in selected image256for enhancement. Selected image256may be, for example, current image228in sequence of images226stored in image buffer224. Of course, in other illustrative examples, selected image256may be any one of sequence of images226. In these illustrative examples, region of interest254may include an area in selected image256in which moving object245is detected. Enhancing region of interest254comprises increasing the resolution of region of interest254as compared to the other portions of selected image256.

In these depicted examples, enhancement module222uses currently-available methods for performing super-resolution to enhance a first portion of region of interest254corresponding to background209in selected image256independently of a second portion of region of interest254corresponding to moving object245. Super-resolution of these two portions of region of interest254is then combined to construct enhanced image258. Enhanced image258of region of interest254has a higher resolution as compared to region of interest254in selected image256.

In one illustrative example, enhancement module222divides region of interest254into plurality of pixels260. Enhancement module222may divide region of interest254into plurality of pixels260using scaling factor262. Scaling factor262determines how many pixels in plurality of pixels260make up a single pixel in selected image256. For example, selected image256may have 100 pixels arranged in a 10 by 10 grid. If scaling factor262is selected as three, each pixel in selected image256is divided into nine pixels arranged in a three by three grid.

Enhancement module222identifies which pixels in plurality of pixels260form first portion265of plurality of pixels260corresponding to background209in selected image256and which pixels form second portion267of plurality of pixels260corresponding to moving object245in selected image256. In particular, enhancement module222identifies first portion265and second portion267using the corresponding object mask in objects masks238generated for selected image256.

In these illustrative examples, object mask240may correspond to selected image256. Further, set of areas241in object mask240may be one area that represents moving object245that appears in selected image256. A pixel in plurality of pixels260having a corresponding pixel in object mask240with a value of logic “0” is included in first portion265. A pixel in plurality of pixels260having a corresponding pixel in object mask240with a value of logic “1” is included in second portion267.

Enhancement module222identifies first values264for first portion265of plurality of pixels260corresponding to background209in selected image256using sequence of images226and first transformations232. Further, enhancement module222identifies second values266for second portion267of plurality of pixels260corresponding to moving object245in selected image256using sequence of images226and second transformations250.

In these illustrative examples, enhancement module222uses first values264and second values266to construct enhanced image258of region of interest254. When a new image in images206is received, this new image may be added to image buffer224as current image228. Further, the images stored in image buffer224are shifted such that the oldest image stored in image buffer224is pushed out of image buffer224and a new sequence of images is formed.

Selected image256for this new sequence of images may be the new image added as current image228. Enhancement module222is configured to generate a new enhanced image using the new sequence of images. The new enhanced image may be generated for region of interest254or a new region of interest. In some cases, a new scaling factor may also be selected.

The illustration of image processing environment200inFIG. 2is not meant to imply physical or architectural limitations to the manner in which an advantageous embodiment may be implemented. Other components in addition to or in place of the ones illustrated may be used. Some components may be unnecessary. Also, the blocks are presented to illustrate some functional components. One or more of these blocks may be combined, divided, or combined and divided into different blocks when implemented in an advantageous embodiment.

For example, in some illustrative examples, sets of tracks242may be stored in an additional buffer or some other suitable type of data structure. In other illustrative examples, a separate module may be configured to store images206in image buffer224instead of registration module218.

With reference now toFIG. 3, an illustration of an image buffer and a background transformation buffer is depicted in accordance with an advantageous embodiment. In this illustrative example, image buffer300is an example of one implementation for image buffer224inFIG. 2. Further, background transformation buffer302is an example of one implementation for background transformation buffer236inFIG. 2. Image buffer300and background transformation buffer302are circular buffers in this depicted example.

Registration module218inFIG. 2stores images, such as images206inFIG. 2, in image buffer300. The images stored in image buffer300form sequence of images226inFIG. 2. Further, registration module218inFIG. 2stores transformations, such as first transformations232inFIG. 2, in background transformation buffer302.

As depicted, image buffer300comprises elements304,306,308,310, and312. In image buffer300, element304stores the oldest image in image buffer300, while element312stores the most recent image. Images314,316,318,320, and322have been stored in elements304,306,308,310, and312, respectively. Images314,316,318,320, and322are examples of the images in sequence of images226inFIG. 2.

In this illustrative example, image314, stored in element304, is the oldest image stored in image buffer300. Image322, stored in element312, is the most recent image stored in image buffer300and is the current image being processed.

Background transformation buffer302comprises elements324,326,328,330, and332. In background transformation buffer302, element324is configured to store the oldest transformation, while element332is configured to store the most recent transformation.

As depicted, transformations334,336,338,340, and342are stored in elements324,326,328,330, and332, respectively. Transformations334,336,338,340, and342are examples of first transformations232inFIG. 2. These transformations are identified by registration module218inFIG. 2with respect to the most recent image added to image buffer300, which is image322.

In this depicted example, transformations334,336,338,340, and342correspond to images314,316,318,320, and322, respectively. In particular, transformations334,336,338,340, and342are configured to align features of the backgrounds of images314,316,318,320, and322, respectively, with features of the background in image322.

As one illustrative example, transformation334is configured to align features of the background in image314with features of the background in image322when transformation334is applied to image314. As another example, transformation340is configured to align features of the background in image320with features of the background in image322when transformation340is applied to image320. Further, transformation342is an identity transformation for image322in this illustrative example. In other words, transformation342does not change image322.

When a new image, such as image344, is added to image buffer300, the images previously stored in the elements of image buffer300are shifted by one element such that the oldest image stored in image buffer300is pushed out of image buffer300. In particular, when image344is added to image buffer300, image316shifts from being stored in element306to being stored in element304. In this manner, image314is overwritten in element304when image344is added to image buffer300.

Further, when image344is added to image buffer300, registration module218inFIG. 2adds a new identity transformation corresponding to image344to background transformation buffer302. This new identify transformation is stored in element332as transformation350. Thereafter, the previously identified transformations stored in background transformation buffer302are shifted by one element and updated to form new transformations. When the previously identified transformations are shifted, transformation334is removed from background transformation buffer302.

In particular, registration module218inFIG. 2identifies a new transformation that is configured to align features of the background in image322to features of the background in image344. This new transformation is combined with transformation342to form transformation352. Transformation352is stored in element330.

Further, the new transformation is also combined with transformation340to form transformation354. Transformation354is then stored in element328. The new transformation is combined with transformation338to form transformation356. Transformation356is then stored in element326. Additionally, the new transformation is combined with transformation336to form transformation358. Transformation358is stored in element324.

In this manner, image buffer300and background transformation buffer302are updated each time a new image is added to image buffer300.

With reference now toFIG. 4, an illustration of an image buffer, an object mask buffer, and an object transformation buffer is depicted in accordance with an advantageous embodiment. In this illustrative example, image buffer300fromFIG. 3is depicted. Further, object mask buffer400and object transformation buffer402are also depicted. Object mask buffer400is an example of one implementation for object mask buffer246inFIG. 2. Object transformation buffer402is an example of one implementation for object transformation buffer252inFIG. 2.

In this illustrative example, a moving object, such as moving object245inFIG. 2, is detected in each of images314,316,318,320, and322. As a result, each of object masks414,416,418,420, and422contains at least one area having pixels with a value of logic “1”.

For example, object transformation436is configured to align an area in image316in which a moving object has been detected with an area in image322in which the same moving object has been detected. Similarly, object transformation438is configured to align an area in image318in which the moving object has been detected with an area in image322in which the same moving object has been detected.

Registration module218inFIG. 2identifies object transformations434,436,438,440, and442using both the images stored in image buffer300and the object masks stored in object mask buffer400. For example, when identifying object transformation438, registration module218takes into account only the pixels in image318that have corresponding pixels in object mask418with a value of logic “1” and the pixels in image322that have corresponding pixels in object mask422with a value of logic “1”.

When image344is added to image buffer300, object tracking module220inFIG. 2generates object mask443for image344and shifts the object masks stored in object mask buffer400by one element such that object mask414is overwritten. Additionally, when object mask414has been added to object mask buffer400, registration module218inFIG. 2updates the object transformations stored in object transformation buffer402. In particular, registration module218updates object transformation buffer402with object transformations444,446,448,450, and452stored in elements424,426,428,430, and432, respectively.

In this illustrative example, object transformations444,446,448,450, and452are configured to align features of the moving object detected in images316,318,320,322, and344, respectively, with features of the moving object detected in image344. In this manner, object transformation452may be an identity transformation.

Further, in this illustrative example, the object transformations stored in object transformation buffer402may be individually recomputed each time a new image is added to image buffer300. In this manner, a loss of a detection of a moving object in one of the images stored in image buffer300may only cause one element in object transformation buffer402to be undetermined.

With reference now toFIG. 5, an illustration of an image generated by an imaging system is depicted in accordance with an advantageous embodiment. In this illustrative example, image500is an example of one implementation for one of images206in video212inFIG. 2. Image500is part of video generated by an imaging system, such as imaging system202inFIG. 2. As depicted, moving object502is captured in image500. Moving object502is a vehicle in this illustrative example.

Turning now toFIG. 6, an illustration of an enhanced image is depicted in accordance with an advantageous embodiment. In this illustrative example, enhanced image600is an example of one implementation for enhanced image258inFIG. 2. Enhanced image600is generated using image500inFIG. 5and previous images in the video generated by the imaging system.

As depicted, enhanced image600has a higher resolution as compared to image500inFIG. 5. Further, the portion of enhanced image600corresponding to moving object502from image500is being tracked using track602.

With reference now toFIG. 7, an illustration of pixels for a selected image is depicted in accordance with an advantageous embodiment. Selected image700is a low resolution image. In particular, in this depicted example, selected image700is an example of one implementation for selected image256inFIG. 2. In particular, selected image700is current image228in sequence of images226stored in image buffer224inFIG. 2.

In this illustrative example, selected image700is divided into grid702of squares703. Each pixel for selected image700is a center of a square in grid702. For example, center704of square706in grid702is pixel708. Pixel708for square706may correspond to a background or a moving object in selected image700.

As depicted, enhancement module222inFIG. 2may select region of interest710in selected image700for enhancement. Enhancement module222divides the portion of squares703in region of interest710into smaller squares using a scaling factor to form grid712of smaller squares713.

In this illustrative example, a scaling factor of 4 is used. With this scaling factor, each square in grid702in region of interest710is divided into smaller squares to form a section of four squares by four squares in region of interest710. These different sections form grid712for region of interest710. As one illustrative example, square714in grid702is divided to form a section of four squares by four squares.

Enhancement module222inFIG. 2uses grid712for region of interest710to form an enhanced image, such as, for example, enhanced image258inFIG. 2. In particular, grid712is used to form an enhanced image in which each pixel for the enhanced image is a center of a square in grid712. In these illustrative examples, each of the pixels in the enhanced image may be referred to as a super-resolution (SR) pixel. In some cases, each pixel in the enhanced image may be referred to as a high resolution (HR) pixel.

For example, the center of square716in grid712is pixel718. Pixel718may be referred to as a super-resolution pixel. Enhancement module222estimates the final value for pixel718using set of previous images230stored in image buffer224inFIG. 2.

In particular, enhancement module222determines whether pixel720for square714in grid702corresponding to pixel718in square716of grid712is for a background in selected image700or a moving object in selected image700. Enhancement module222uses an object mask generated for selected image700to determine whether the value for pixel720for square714indicates that pixel720is for the background or that pixel720is for the moving object.

If pixel720is for the background in selected image700, enhancement module222uses first transformations232stored in background transformation buffer236inFIG. 2to identify a location in each of set of previous images230in sequence of images226stored in image buffer224that corresponds to pixel720. In other words, when pixel718is for background in selected image700, enhancement module222identifies locations for pixel718in other images in sequence of images226using first transformations232.

If pixel720is for the moving object in selected image700, enhancement module222uses second transformations250stored in object transformation buffer252inFIG. 2to identify a location in each of set of previous images230in sequence of images226stored in image buffer224that corresponds to pixel720. In other words, when pixel718is for the moving object in selected image700, enhancement module222identifies locations for pixel718in other images in sequence of images226using second transformations250.

Further, enhancement module222uses the object mask generated for each of set of previous images230to determine whether to use the pixel at the location identified in each of set of previous images230in estimating the final value for pixel718.

Similarly, when pixel720is for the moving object, enhancement module222determines whether the object mask corresponding to each previous image in set of previous images230indicates that the pixel at the location identified in the previous image is also for the moving object. If the pixel in the previous image is for the moving object, the pixel is included in the estimation of the final value for pixel718. Otherwise, the pixel is not included.

Enhancement module222determines the value of the pixels included for estimation of the final value for pixel718. The values of these pixels may be determined using an interpolation algorithm, such as, for example, without limitation, a nearest neighbor algorithm, a linear interpolation algorithm, a cubic interpolation algorithm, or some other suitable type of algorithm. Enhancement module222may estimate the final value for pixel718using weighting, averaging, outlier elimination based on distribution, other techniques, or combinations of techniques.

With reference now toFIG. 8, an illustration of a flowchart of a process for processing images is depicted in accordance with an advantageous embodiment. The process illustrated inFIG. 8may be implemented using image processing system204inFIG. 2. In particular, this process may be implemented using registration module218, object tracking module220, and enhancement module222inFIG. 2.

The process begins by identifying first transformations for a sequence of images (operation800). The sequence of images may be, for example, sequence of images226inFIG. 2. Each of the first transformations is configured to align features of the background in one image in the sequence of images to the features of the background in the selected image.

The process then identifies second transformations for the sequence of images (operation802). Each of the second transformations is configured to align features of the moving object in the one image in the sequence of images to the features of the moving object in the selected image.

Next, the process identifies a portion of a selected image in which a moving object is present as a region of interest (operation804). The selected image is one of the sequence of images. In these illustrative examples, the selected image may be the most recent image in the sequence of images. Further, the moving object moves with respect to a background in the selected image.

Thereafter, the process identifies a plurality of pixels in the region of interest in the selected image (operation806). The process identifies first values for a first portion of the plurality of pixels using the sequence of images and first transformations (operation808). The first portion of the plurality of pixels corresponds to the background in the selected image. Next, the process identifies second values for a second portion of the plurality of pixels using the sequence of images and second transformations (operation810). The second portion of the plurality of pixels corresponds to the moving object in the selected image.

The process then generates an enhanced image using the first values for the first portion of the plurality of pixels in the region of interest and the second portion of the plurality of pixels in the region of interest (operation812), with the process terminating thereafter.

With reference now toFIG. 9, an illustration of a flowchart of a process for enhancing video is depicted in accordance with an advantageous embodiment. The process illustrated inFIG. 9may be implemented using image processing system204inFIG. 2. In particular, this process may be implemented using registration module218, object tracking module220, and enhancement module222inFIG. 2.

The process begins by adding a current image to an image buffer for processing (operation900). The image buffer stores a sequence of images. The current image may be one of images206in the form of video212inFIG. 2. In particular, the current image may be current image228in sequence of images226inFIG. 2.

The process then identifies a first transformation for aligning features of the background in a previous image to features of the background in the current image (operation902). The previous image may be the image added to the image buffer prior to the current image. Next, the process updates transformations in a background transformation buffer using the first transformation (operation904).

Thereafter, the process generates an object mask using the current image (operation906). The object mask indicates whether one or more moving objects are detected in the current image. The process then updates a set of tracks generated for moving objects detected in the sequence of images using the current image and the object mask (operation908). In operation908, updating the set of tracks may include adding a new track for a newly detected moving object.

Next, the process determines whether any moving objects are detected in the current image (operation910). If any moving objects are detected in the current image, the process identifies a second transformation for each of the moving objects (operation912).

Each transformation is configured to align features of the corresponding moving object in the previous image in the sequence of images to features of the corresponding object in the current image. Thereafter, the process updates object transformations stored in a number of object transformation buffers using each second transformation generated for each of the moving objects (operation914).

The process then selects a region of interest in the current image (operation916). The process divides the region of interest into a plurality of pixels using a scaling factor (operation918). The process then identifies first values for a first portion of the plurality of pixels corresponding to the background in the current image using the sequence of images and the transformations stored in the background transformation buffer (operation920).

The process also identifies second values for a second portion of the plurality of pixels corresponding to one or more moving objects in the current image using the sequence of images and the object transformations stored in the object transformation buffer (operation922).

The process then generates an enhanced image using the first values for the first portion of the plurality of pixels and the second values for the second portion of the plurality of pixels (operation924), with the process terminating thereafter. In these illustrative examples, the enhanced image is for the region of interest and has a higher resolution as compared to the region of interest in the current image.

With reference again to operation910, if one or more moving objects are not detected in the current image, the process proceeds to operation916as described above. In these illustrative examples, the process described inFIG. 9may be repeated for each new image added to the image buffer.

With reference now toFIGS. 10A and 10B, illustrations of a flowchart of a process for selecting pixels in a sequence of images for estimating the value of each pixel in an enhanced image of a region of interest are depicted in accordance with an advantageous embodiment. The process illustrated inFIGS. 10A and 10Bmay be implemented using image processing system204inFIG. 2.

The process begins by forming a grid for generating an enhanced image of a region of interest in a selected image in a sequence of images (operation1000). In operation1000, the sequence of images may be sequence of images226inFIG. 2, and the selected image may be selected image256inFIG. 2. In particular, the selected image may be current image228in sequence of images226inFIG. 2.

Further, in operation1000, the grid includes pixels arranged in a plurality of rows and a plurality of columns. The grid is formed using a scaling factor. Grid712inFIG. 7is an example of one implementation for the grid formed in operation1000. In these illustrative examples, the pixels in the grid may be referred to as super-resolution pixels.

The process then selects a row in the grid (operation1002). Next, the process selects a column for the selected row in the grid (operation1004). Then, the process selects a particular image in the sequence of images (operation1006). The particular image may be the current image in the sequence of images or an image in the set of previous images in the sequence of images.

The process determines whether the super-resolution pixel at the selected row and selected column in the grid is for background (operation1008). In operation1008, the process identifies the pixel in the selected image that corresponds to the super-resolution pixel at the selected row and selected column in the grid. The super-resolution pixel is for background when the corresponding pixel in the selected image is for background. Similarly, the super-resolution pixel is for a moving object when the corresponding pixel in the selected image is for the moving object.

In operation1008, the process uses an augmented object mask corresponding to the selected image to determine whether the corresponding pixel in the selected image is for background or for a moving object. The augmented object mask has pixels that correspond to the pixels in the selected image.

A pixel in the augmented object mask indicates that the corresponding pixel in the selected image is for background when the pixel in the augmented object mask has a value of logic “0”. A pixel in the augmented object mask indicates that the corresponding pixel in the selected image is not for background and is for a moving object when the value for the pixel in the augmented object mask is a track identifier. The track identifier identifies the track generated for the moving object.

When the pixel in the augmented object mask corresponding to the pixel in the selected image that corresponds to the super-resolution pixel has a value of logic “0”, the super-resolution pixel is for background. When the pixel in the augmented object mask corresponding to the pixel in the selected image that corresponds to the super-resolution pixel has a value that is a track identifier for a track, the super-resolution pixel is for a moving object being tracked by that track.

With reference again to operation1008, if the super-resolution pixel is for background, the process identifies a location of a corresponding pixel in the particular image using a first transformation (operation1010). In operation1010, the first transformation may be, for example, one of first transformations232inFIG. 2.

The process then determines whether the corresponding pixel in the particular image is for background (operation1012). Operation1012may be performed using an augmented object mask corresponding to the particular image. If the corresponding pixel in the particular image is for background, the process adds the corresponding pixel for inclusion in estimating the value of the super-resolution pixel at the selected row and selected column in the grid (operation1014).

The process then determines whether any additional unprocessed images are present in the sequence of images for the super-resolution pixel at the selected row and selected column in the grid (operation1016). If additional unprocessed images are present, the process returns to operation1006.

When additional unprocessed images are not present, the selection of pixels from the images in the sequence of images for inclusion in estimating the value of the super-resolution pixel at the selected row and selected column in the grid is complete. The values for the pixels selected may then be used to estimate the value of the super-resolution pixel.

With reference again to operation1016, if additional unprocessed images are not present in the sequence of images, the process determines whether unprocessed columns are present in the selected row in the grid (operation1018). If unprocessed columns are present, the process returns to operation1004as described above. Otherwise, the process determines whether any additional unprocessed rows are present in the grid (operation1020). If unprocessed rows are present, the process returns to operation1002as described above. Otherwise, the process terminates.

With reference again to operation1012, if the corresponding pixel in the particular image is not for background, the process then proceeds to operation1016as described above. Further, with reference again to operation1008, if the super-resolution pixel at the selected row and selected column in the grid is not for background, then the super-resolution pixel is for a moving object having a particular track identifier identified in the augmented object mask corresponding to the selected image.

In operation1008, if the super-resolution pixel is not for background, the process identifies a location of a corresponding pixel in the particular image using a second transformation (operation1022). The second transformation may be, for example, one of second transformations250inFIG. 2.

The process then determines whether the corresponding pixel in the particular image is for a moving object (operation1024). If the corresponding pixel in the particular image is for a moving object, the process determines whether a track identifier for the corresponding pixel in the particular image is the same as the track identifier for the super-resolution pixel (operation1026).

In operation1026, the track identifier for the corresponding pixel in the particular image is the value of the pixel in the augmented object mask for the particular image that corresponds to the corresponding pixel in the particular image. In operation1026, if the track identifiers are the same, the process adds the corresponding pixel in the particular image for inclusion in estimating the value of the super-resolution pixel at the selected row and selected column in the grid (operation1028).

The process then proceeds to operation1016as described above. With reference again to operation1026, if the track identifiers are not the same, the process proceeds directly to operation1016as described above. Further, with reference again to operation1024, if the corresponding pixel in the particular image is not for a moving object, the process also proceeds directly to operation1016as described above.

The flowcharts and block diagrams in the different depicted embodiments illustrate the architecture, functionality, and operation of some possible implementations of apparatuses and methods in an advantageous embodiment. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, function, and/or a portion of an operation or step. For example, one or more of the blocks may be implemented as program code, in hardware, or a combination of the program code and hardware. When implemented in hardware, the hardware may, for example, take the form of integrated circuits that are manufactured or configured to perform one or more operations in the flowcharts or block diagrams.

For example, in some illustrative examples, the process illustrated inFIGS. 10A and 10Bmay be modified such that super-resolution is performed only for the background and not for moving objects. In this case, when the super-resolution pixel at the selected row and selected column in the grid is for a moving object, the value for the super-resolution pixel becomes the value of the corresponding pixel in the selected image.

In other words, the process illustrated inFIGS. 10A and 10Bmay be modified such that, with reference to operation1008, if the super-resolution pixel at the selected row and selected column in the grid is not for background, the process includes the corresponding pixel in the selected image for inclusion in estimating the value of the super-resolution pixel but excludes the corresponding pixels in the other images in the sequence of images. This operation may be performed in place of operations1022,1024,1026, and1028inFIGS. 10A and 10B.

These modifications to the process illustrated inFIGS. 10A and 10Bmay be performed in response to a number of different types of situations. For example, when a moving object is not detected in more than a selected number of images in the sequence of images, these modifications may be made to the process illustrated inFIGS. 10A and 10B. The moving object may not be detected in images in the sequence of images in response to the moving object being blocked by some other object. Further, these modifications may be made when the information generated by object tracking module220inFIG. 2is considered unreliable or unavailable.

Turning now toFIG. 11, an illustration of a data processing system is depicted in accordance with an advantageous embodiment. In this illustrative example, data processing system1100may be used to implement one or more computers in computer system216inFIG. 2. Data processing system1100includes communications fabric1102, which provides communications between processor unit1104, memory1106, persistent storage1108, communications unit1110, input/output (I/O) unit1112, and display1114.

Memory1106and persistent storage1108are examples of storage devices1116. A storage device is any piece of hardware that is capable of storing information, such as, for example, without limitation, data, program code in functional form, other suitable information, or combinations of information, either on a temporary basis, a permanent basis, or both. Storage devices1116may also be referred to as computer readable storage devices in these examples. Memory1106, in these examples, may be, for example, a random access memory or any other suitable volatile or non-volatile storage device. Persistent storage1108may take various forms, depending on the particular implementation.

For example, persistent storage1108may contain one or more components or devices. For example, persistent storage1108may be a hard drive, a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination of the above. The media used by persistent storage1108also may be removable. For example, a removable hard drive may be used for persistent storage1108.

Communications unit1110, in these examples, provides for communications with other data processing systems or devices. In these examples, communications unit1110is a network interface card. Communications unit1110may provide communications through the use of either or both physical and wireless communications links.

Input/output unit1112allows for input and output of data with other devices that may be connected to data processing system1100. For example, input/output unit1112may provide a connection for user input through a keyboard, a mouse, some other suitable input device, or a combination of devices. Further, input/output unit1112may send output to a printer. Display1114provides a mechanism to display information to a user.

Instructions for the operating system, applications, programs, either alone or in any combination, may be located in storage devices1116, which are in communication with processor unit1104through communications fabric1102. In these illustrative examples, the instructions are in a functional form on persistent storage1108. These instructions may be loaded into memory1106for execution by processor unit1104. The processes of the different embodiments may be performed by processor unit1104using computer-implemented instructions, which may be located in a memory, such as memory1106.

These instructions are referred to as program code, computer usable program code, or computer readable program code that may be read and executed by a processor in processor unit1104. The program code in the different embodiments may be embodied on different physical or computer readable storage media, such as memory1106or persistent storage1108.

Program code1118is located in a functional form on computer readable media1120that is selectively removable and may be loaded onto or transferred to data processing system1100for execution by processor unit1104. Program code1118and computer readable media1120form computer program product1122in these examples. In one example, computer readable media1120may be computer readable storage media1124or computer readable signal media1126. Computer readable storage media1124may include, for example, an optical or magnetic disk that is inserted or placed into a drive or other device that is part of persistent storage1108for transfer onto a storage device, such as a hard drive, that is part of persistent storage1108.

Computer readable storage media1124also may take the form of a persistent storage, such as a hard drive, a thumb drive, or a flash memory, that is connected to data processing system1100. In some instances, computer readable storage media1124may not be removable from data processing system1100. In these examples, computer readable storage media1124is a physical or tangible storage device used to store program code1118rather than a medium that propagates or transmits program code1118. Computer readable storage media1124is also referred to as a computer readable tangible storage device or a computer readable physical storage device. In other words, computer readable storage media1124is a media that can be touched by a person.

Alternatively, program code1118may be transferred to data processing system1100using computer readable signal media1126. Computer readable signal media1126may be, for example, a propagated data signal containing program code1118. For example, computer readable signal media1126may be an electromagnetic signal, an optical signal, any other suitable type of signal, or a combination of signals. These signals may be transmitted over communications links, such as wireless communications links, optical fiber cable, coaxial cable, a wire, any other suitable type of communications link, or combinations of links. In other words, the communications link may be physical or wireless in the illustrative examples.

In some advantageous embodiments, program code1118may be downloaded over a network to persistent storage1108from another device or data processing system through computer readable signal media1126for use within data processing system1100. For instance, program code stored in a computer readable storage medium in a server data processing system may be downloaded over a network from the server to data processing system1100. The data processing system providing program code1118may be a server computer, a client computer, or some other device capable of storing and transmitting program code1118.

In still another illustrative example, processor unit1104may be implemented using a combination of processors found in computers and hardware units. Processor unit1104may have a number of hardware units and a number of processors that are configured to run program code1118. With this depicted example, some of the processes may be implemented in the number of hardware units, while other processes may be implemented in the number of processors.

Additionally, a communications unit may include a number of devices that transmit data, receive data, or transmit and receive data. A communications unit may be, for example, a modem or a network adapter, two network adapters, or some combination thereof. Further, a memory may be, for example, memory1106, or a cache, such as found in an interface and memory controller hub that may be present in communications fabric1102.

Thus, the different advantageous embodiments provide a method and apparatus for enhancing video using super-resolution. In particular, the different advantageous embodiments provide a method and apparatus that enhances video of a scene in which objects move relative to a background of the scene using super-resolution techniques.

In one advantageous embodiment, a method for processing images is provided. A portion of a selected image in which a moving object is present is identified. The selected image is one of a sequence of images. The moving object moves with respect to a background in the selected image. A plurality of pixels in a region of interest is identified in the selected image. First values are identified for a first portion of the plurality of pixels using the sequence of images and first transformations. The first portion of the plurality of pixels corresponds to the background in the selected image. A first transformation in the first transformations is configured to align features of the background in one image in the sequence of images to the features of the background in the selected image. Second values are identified for a second portion of the plurality of pixels using the sequence of images and second transformations. The second portion of the plurality of pixels corresponds to the moving object in the selected image. A second transformation in the second transformations is configured to align features of the moving object in the one image in the sequence of images to the features of the moving object in the selected image.