Ambient correction in rolling image capture system

Embodiments are disclosed that relate to the correction of an image acquired in a rolling image capture system for ambient light. One embodiment provides a device comprising a screen, a rolling image capture system, a local light source, and a controller. The controller is configured to operate the local light source while acquiring first and second frames of image data to integrate each field of image sensor pixels for a duration of local+ambient light and for a duration of ambient light such that a sum tlocal+ambient+tambient for the first frame is different than a sum tlocal+ambient+tambient for the second frame for each field of pixels. The controller is further configured to determine an ambient light value, and to adjust one or more pixels based upon the ambient light value.

BACKGROUND

Touch-sensitive devices may detect touch via several different mechanisms, including but not limited to optical, resistive, and capacitive mechanisms. Some optical touch-sensitive devices detect touch by capturing an image of a backside of a touch screen via an image sensor, and then processing the image to detect objects located on the screen. Such devices may include a light source within the device to illuminate the backside of the display screen such that objects on the screen reflect the incident light toward the image sensor, thereby allowing the object to be detected.

One difficulty that may be encountered with optical touch screen devices involves differentiating between external (ambient) light and light reflected from the light source within the device. Ambient light of sufficient brightness may be mistaken for an object touching the device, and therefore may degrade the performance of the device. Further, the use of a rolling image capture system may introduce additional difficulties with correcting an image for ambient light.

SUMMARY

Accordingly, various embodiments are disclosed that relate to the correction of an image acquired in a rolling image capture system for ambient light. For example, one disclosed embodiment provides an optical touch-sensitive device comprising a screen, a rolling image capture system configured to acquire an image of the screen, a local light source configured to illuminate the screen with local light, and a controller in electrical communication with the rolling image capture system and the local light source. The controller is configured to operate the local light source while acquiring first and second frames of image data to integrate each field of image sensor pixels for a duration of local+ambient light and for a duration of ambient light such that a sum tlocal+ambient+tambientfor the first frame is different than a sum tlocal+ambient+tambientfor the second frame for each field of pixels. The controller is further configured to determine an ambient light value for a pixel in the image data by one or more of (a) comparing a value of the pixel in the first frame with a value of the pixel in the second frame and (b) comparing the value of the pixel in the first frame with a value of another pixel in the first frame, and to adjust one or more pixels for ambient light based upon the ambient light value.

DETAILED DESCRIPTION

As described above, ambient light sources, such as room lighting, sunlight, etc., may harm the performance of a vision-based touch detection system. The term “ambient light” is used herein to describe light originating from a source other from a local light source that is part of the vision-based touch system, as described in more detail below. Even where a bandpass filter is used in front of an image sensor to prevent unwanted wavelengths of light from reaching the image sensor, ambient light within the wavelength range transmitted by the bandpass filter may still reach the image sensor. As one specific example, a vision system configured to detect touch via locally emitted infrared light reflected from objects in contact with to a display screen may be affected by infrared light emitted by incandescent room lighting, sunlight, and the like. Ambient light of a sufficient intensity may cause a vision-based touch detection system to mistakenly identify ambient light as a touch input, and/or may result in a reduction in image contrast that makes touch detection more difficult.

Various techniques may be used to cancel or otherwise correct for ambient light in an image captured by an image sensor in a vision-based touch detection system. For example, a local light source may be strobed such that alternate frames are exposed to “ambient” and “ambient+local” light. This allows the ambient light intensity to be determined by subtracting the “ambient” frame from the “ambient+local” frame to correct for ambient. However, because the local light is turned on every other frame, this effectively cuts the frame rate of a device in half, which may increase the difficulty of tracking movement of a touch input.

Another potential technique is to utilize a separate sensor (possibly with an optical filter) configured to integrate ambient light. However, the use of an additional sensor may be expensive, and may be prone to errors due to the different positioning of the sensors in the device. Yet another potential technique may be to utilize an extremely bright local light source in combination with a band-pass filter to boost the intensity of reflected light relative to ambient light. However, this approach may be susceptible to failure where the ambient light exceeds some percentage of local light.

The use of a rolling image capture system to detect touch inputs in a vision-based touch sensing system may introduce additional difficulties in correcting for ambient light. Rolling image capture systems, such as a rolling shutter camera, a rolling sensor-in-panel arrangement (where image sensor pixels are integrated into a display panel, thereby allowing the omission of a separate camera), or the like, capture images by integrating an image sensor progressively across an area of the image sensor. For example, some rolling image capture systems may be configured to progressively integrate an image sensor from a top row of the image sensor to a bottom row of the image sensor. Thus, different pixels of the image sensor begin and end light integration at different times.

Additional challenges in ambient correct may arise in a rolling image capture system due to the different times at which different rows or columns of pixels of a rolling image capture system integrate light. For example, simply turning a single backlight on and off at a 50% time cycle to capture alternating images with and without local lighting may result in the rows of the image sensor integrating local light for different durations, thereby causing difficulties with ambient correction. Further, some pixels may be exposed to equal amounts of local light each frame, thereby preventing ambient correction for those pixels.

Accordingly,FIG. 1shows a flow diagram of a method100for correcting an image for ambient light in a device comprising a rolling shutter image capture system. Method100comprises, at102, acquiring a first frame of image data with a rolling image capture system, wherein the rolling image capture system comprises one or more fields of pixels. As indicated at104, this may comprise operating a local light source in such a manner as to integrate each field of pixels for a first duration, or total sum, of local+ambient light (tlocal+ambient) and ambient light (tambient). The total sum of these two durations of image sensor pixel integration may be referred to herein as tlocal+ambient+tambient.

The term a “field of pixels” as used herein represents a group of pixels in which each pixel in a field is integrated for equal durations of local+ambient light (tlocal+ambient) and ambient light (tambient) in a frame of image data. Therefore, in embodiments in which the rolling image capture system comprises a single field of pixels, all pixels of the first frame of image data may be exposed for equal durations of tlocal+ambientand tambient. Likewise, in embodiments comprising two or more fields of pixels, the first and second fields of pixels in the first frame of image data may be integrated for different total sums of (tlocal+ambient+tambient). For example, in one specific embodiment, two fields of pixels in a single frame of image data are integrated such that the fields are exposed to different durations of tambientbut similar durations of tlocal+ambient. The two fields may be interlaced fields (for example, odd/even rows or odd/even columns of pixels), or may have any other suitable spatial relationship. Furthermore, in some embodiments, three or more fields of pixels may be exposed for different total sums of (tlocal+ambientand tambient). Examples of methods to expose different fields of pixels to different durations of tlocal+ambient+tambientin a single image frame are described in more detail below. In other embodiments, the rolling image capture system may comprise a single field of pixels.

Next, method100comprises, at106, acquiring a second frame of image data with the rolling image capture system. As indicated at108, this may comprise operating a local light source to integrate each field of pixels for a second total sum of tlocal+ambient+tambientthat is different than the first total sum of tlocal+ambient+tambientfor that field of pixels.

Next, method100comprises determining, at110, an ambient light value for one or more pixels of image data to allow those pixels to be adjusted for ambient light. The ambient light value for a pixel of image sensor data may be determined by comparing the value of the pixel in the first frame of image data and a value of the same pixel in the second frame of image data, as indicated at112, by comparing the value of the pixel in the first frame of image data to the value of another pixel in the first frame of image data, as indicated at114, or from a combination of these processes, depending upon the method used to acquire the first and second frames of image data. Further, as indicated at116and118, in some embodiments, it may be determined whether any objects imaged in the frames of image data are in motion between the first and second frame of image data to assist in selecting an ambient value determination, as explained in more detail below.

Method100next comprises, at120, adjusting one or more pixels of image data to correct for the ambient light based upon the ambient light value determined. In some embodiments, the image data may be adjusted if it is first determined, at122, if the ambient light measure is over a threshold value. Compared to other methods of correcting for ambient light, method100allows a correction for ambient light to be made to image data without the use of an additional image sensor or other additional parts, and also without any loss of frame rate.

Prior to discussing the correction for ambient light in more detail, an embodiment of a suitable use environment is described.FIG. 2shows an embodiment of an interactive display device200comprising a rolling image capture system, shown schematically at202, such as a rolling shutter camera, a rolling sensor-in-panel arrangement (where image sensor pixels are integrated into a display panel, thereby allowing the omission of a separate camera), or the like. Rolling image capture system202captures images by exposing an image sensor progressively across an area of the image sensor, such as from a top horizontal row to a bottom horizontal row of the image sensor, from a left column to a right column, etc. Thus, different pixels of the image sensor begin and end light integration at different times. It will be understood that the term “row” as used herein represents any linear array of sensor pixels, whether arranged vertically, horizontally, diagonally, etc.

Interactive display device200further comprises a projection display system having an image source204comprising a lamp and an image-producing element, such as the depicted liquid crystal display (LCD) or other suitable image producing element, and a display screen206onto which images are projected. While shown in the context of a projection display system, it will be understood that other embodiments may utilize a liquid crystal display panel to present images to a user, or any other suitable image-producing element, rather than a rear projection system.

The depicted display screen206includes a transparent portion208, such as sheet of glass, and a diffuser layer210disposed on top of the transparent portion208. Diffuser layer210helps to avoid the imaging of objects that are not in contact with or positioned within a few millimeters of display screen206, and therefore helps to ensure that objects that are not touching or in close proximity to display screen206are not detected. In some embodiments, an additional transparent layer (not shown) may be disposed over diffuser screen layer210to provide a smooth look and feel to the display surface. Further, in other embodiments, such as some that utilize a LCD panel rather than a projection image source to display images on display screen206, diffuser layer210may be omitted.

Continuing withFIG. 2, interactive display device200further includes an electronic controller212comprising memory214and a processor216. Controller212may further (or alternatively) include a field programmable gate array (FPGA)218, and/or any other suitable electronic components, including application-specific integrated circuits (ASICs) (not shown), digital signal processors (DSPs) (not shown), etc. configured to conduct one or more ambient light correction calculations, as described below. While shown as part of controller212, it will be appreciated that FPGA218and/or other electronic components may also be provided as one or more separate devices in electrical communication with controller212. It will also be understood that memory214may comprise instructions stored thereon that are executable by the processor216to control the various parts of interactive display device200to effect the methods and processes described herein. Likewise, the FPGA222also may be configured to perform one or more of the correction methods described in detail below.

To assist in detecting objects and/or touches placed on display screen206, display device200may further include a local light source configured to illuminate display screen206with infrared or visible light. Light from the local light source may be reflected by objects placed on display screen206and then detected by rolling image capture system202. In the embodiment ofFIG. 2, the local light source comprises a rolling local light source220configured to provide local lighting in a spatially rolling pattern synchronized with rolling image capture system202. The depicted rolling local light source220comprises an arbitrary number of individually controllable light sources, illustrated as light set1222, light set2224, and light set n226, where each light set may comprise one or more light sources, such as a plurality of infrared LEDs. The use of infrared local light as opposed to visible local light may help to avoid washing out the appearance of images projected on display screen206. Further, an infrared bandpass filter (not shown) may be utilized to pass light of the frequency emitted by the local light source but prevent light at frequencies outside of the bandpass frequencies from rolling image capture system202.

Each light set222,224,226may have any suitable configuration. For example, in some embodiments, each light set222,224,226may comprise a plurality of relatively thin bands of LEDs, where each band is configured to illuminate a sub-set of rows of pixels of the image sensor. Such bands may be configured to illuminate any desired number of rows or columns of a display screen, and may even comprise a separate set of backlights for each row or column of the display screen. In other embodiments, the rolling local light source220may comprise a relatively lesser number of relatively wider bands of LEDs. In yet other embodiments, the rolling local light source220may include a mechanically scannable light source configured to scan a band of light across the display screen in synchronization with the rolling image capture system. It will be understood that these examples of rolling local light sources are described for the purpose of example, and are not intended to be limiting in any manner. For example, other light sources than LEDs may be used for each band of light. It further will be understood that the rolling local light source220may further comprise any suitable optics for focusing a band of light on a desired subset of rows or columns pixels of a display screen.

FIG. 2also depicts an object230placed on display screen206. Light from the rolling local light source220reflected by object230may be detected by rolling image capture system202, thereby allowing the object230to be detected on the screen. Object230represents any object that may be in contact with display screen206, including but not limited to fingers, brushes, optically readable tags, etc.

FIG. 3shows a timing diagram300depicting an example of a method of collecting image data to allow for the correction of ambient light that utilizes a rolling image capture system that comprises two fields of pixels. Generally, the embodiment depicted in timing diagram300allows a rolling image capture system to acquire a first frame of image data by progressively reading a first field of pixels while illuminating the screen with the rolling local light source and then progressively reading the second field of pixels while not illuminating the screen with the rolling local light source, and acquire the second frame of image data by progressively reading the second field of pixels while illuminating the screen with the rolling local light source and then progressively reading the first field of pixels while not illuminating the screen with the rolling local light source. As will be described below, this allows an image to be corrected for ambient light while preserving the overall frame rate of the device. Timing diagram300is shown in the context of a rolling local light source comprising four sets or bands of local light, but it will be understood that any other suitable number of sets of bands of local light may be used.

First referring to image data frame n ofFIG. 3, odd rows1-1079are read out and reset during a first portion of the frame n readout cycle, and then even rows2-1080are read out and reset during a second portion of the frame n readout cycle. Referring next to image data frame n+1, even rows2-1080are read out before odd rows1-1079. This pattern may repeated throughout operation of the rolling image capture system. Along with this readout and reset pattern, the local light sets are cycled such that each set is turned on for approximately one half of an image frame, and then turned off for approximately one half of each frame. Referring again to frame n ofFIG. 3, light set1turns off at the time row1of the image sensor is read and then turns off after all odd rows have been read, approximately halfway through the image n readout cycle. Then, light set1turns on for the remainder of the image n readout cycle. Light sets2,3and4toggle on and off in a similar manner, except that the toggling of these light sets are synchronized with the readout and reset of rows271,541and811, respectively.

Next referring to image data frame n+1 ofFIG. 3, the light sets are toggled in the same manner as for image data frame n, but the even rows are read out and reset before the odd rows. In this manner, the even and odd rows of each frame of image data image are exposed to different durations of ambient light, but similar durations of local light. For example, at the time row1of frame n+1 is read out and reset, it has been exposed to ambient light for 1.5 integration cycles and to local light for 0.5 integration cycles. In contrast, at the time row2of frame n+1 is read out, it has been exposed to ambient light for 0.5 integration cycles and local light for 0.5 integration cycles. In this manner, a single image contains pixels that have been exposed to different durations of ambient light but similar amounts of local light. Further, each field of pixels is also exposed to different total durations of ambient light in frame n compared to frame n+1. The depicted integration and local lighting pattern ofFIG. 3thus allows ambient correct to be performed with intraframe data (“spatial correction”) and/or interframe data (“temporal correction”).

FIGS. 4-10illustrate various spatial and temporal correction methods that may be used to correct image data acquired via the method shown inFIG. 3for ambient light. In order to illustrate various ambient correction methods, a representative group of intensity data from two image frames, labeled frames n−1 and n, are described with reference toFIG. 4. Specifically,FIG. 4illustrates how the readout from the rolling image capture system for the two image frames, which show a stationary scene, may appear when integrated and read according to the process shown inFIG. 3. First, a simple stationary scene with no ambient light is shown at402, and a 3×3 matrix of pixels from scene402is shown at404. For the purpose of simplicity, the images inFIG. 4have three intensity levels, wherein the lightest pixels signify the most integrated light and the darkest pixels signify the least integrated light.

In frame n−1, the odd rows have a greater duration of ambient exposure than the even rows. The addition of this ambient pattern to the 3×3 scene yields the intensity data shown at406. Likewise, in frame n, the even rows have a greater duration of ambient exposure than the odd rows. The addition of this ambient pattern to the 3×3 scene yields the intensity data shown at408. Referring next toFIG. 5, the ambient light can be calculated for the odd rows by subtracting frame n from frame n−1 (as shown at502), and for the even rows by subtracting frame n−1 from frame n (as shown at504). Combining the ambient determined for the odd rows with the ambient determined for the even rows yields an overall ambient506for the 3×3 matrix.

FIGS. 6-8show examples of various methods that may be used to correct an image frame for ambient light with the image data shown inFIG. 4. These figures are shown in the context of determining ambient for a single pixel at a time. This may allow different ambient calculation methods to be used for different pixels depending upon pixel-specific factors. It will be understood that the illustrated methods may be applied to each pixel in a frame of image data to correct the overall frame of image data for ambient light.

First referring toFIG. 6, an ambient light value at a pixel (for example, the center pixel of the 3×3 matrix shown inFIGS. 4-5) may be calculated as described above forFIG. 5by simply subtracting frame n−1 from frame n. Likewise, ambient values for pixels in the top and bottom rows of the 3×3 matrix may be determined simply by subtracting frame n from frame n−1. This method utilizes information from temporally adjacent frames but does not utilize information from spatially adjacent pixels. Therefore, the method illustrated inFIG. 6may be referred to herein as a “temporal-local” correction. However, due to the sensor readout pattern shown inFIG. 3, after subtraction of ambient, the intensity at that pixel is the same as in an adjacent frame. Thus, the temporal-local correction may effectively halve the frame rate of the device. For this reason, this correction may be used for stationary objects.

FIG. 7shows another example of a method for correcting an image frame for ambient light. As opposed to that shown inFIG. 6, the method shown inFIG. 7takes into account both temporal information (i.e. temporally adjacent image frames) and spatial information (i.e. spatially adjacent pixels) when calculating the ambient for a pixel. Therefore, the method shown inFIG. 7may be referred to as a “temporal-spatial” correction. While shown in the context of a 3×3 matrix, it will be appreciated that the concepts shown inFIG. 7, as well asFIG. 8, may be applied to a matrix of any size of pixels and any shape/pattern around the pixel of interest, including but not limited to 5×5 and 7×7 matrices, as well as other shapes (such as a cross-shaped matrix formed by omitting each corner pixel from a 5×5 matrix).

The temporal-spatial correction shown inFIG. 7utilizes a weighted average intensity of the pixels in the sample matrix to determine an ambient value, wherein the center pixel is weighted more strongly (¼) than the side pixels (⅛ each), which are in turn weighted more strongly than the corner pixels. To perform the correction, the intensities of the pixels are multiplied by the shown weighting factors, the two frames are added, and then the value at each pixel in the matrix after the addition of the two frames is summed to yield the ambient intensity at the center pixel. Because spatial data is taken into account in addition to temporal data, the temporal-spatial correction allows a frame rate to be maintained.

FIG. 8shows another example of a method for correcting a frame of image data for ambient light. As opposed to the methods shown inFIGS. 6 and 7, the method ofFIG. 8utilizes spatial information, and not temporal information, in making the ambient correction. In other words, the correction is made entirely from a weighted average of intra-frame data, utilizing no inter-frame data. As depicted, this calculation may lead to slightly high values of ambient light, but can avoid calculation problems due to motion that may occur in methods that utilize temporal information.

As mentioned above, in some embodiments it may be determined whether the ambient light exceeds a predetermined threshold level before performing any of the above ambient correction methods. Where ambient light is of sufficiently low intensity or is absent, the touch-sensitive device may be able to detect objects without any problems caused by ambient. Therefore, before performing any of the above-described corrections (or any others), it may be determined whether there is any potentially problematic ambient by comparing the sum of the intensities in the first field in a frame to the sum of the intensities in the second field in the frame. Because the intensities in the two fields differ by the amount of ambient light integrated, if the sums are relatively close together, it can be determined that the ambient light levels are sufficiently low not to interfere with device operation, and adjustment for ambient may be omitted, as shown inFIG. 9.

In some embodiments, it may be determined whether any movement of any objects on the display screen206has occurred, and then an ambient adjustment method may be selected depending upon whether any movement of any objects is detected.FIGS. 10A-Dillustrate an embodiment of a such method for correcting for ambient light. Referring first toFIG. 10A, a 5×5 region of pixels in a current frame of image data (frame n) and a single pixel in two prior frames of image data (frames n−1, n−2) are considered for ambient correction. However, it will be appreciated that a 3×3 region of pixels, or any other suitable region of pixels, in a current frame may be considered in the ambient correction. First referring toFIG. 10A, a center of a current frame is compared to a pixel from frame n−2, which was read in the same field order. If the difference between these pixels exceeds a threshold amount, this indicates that motion may have occurred, and a “motion flag” for that pixel is set. The value of the motion flag is compared to motion flags for nearby pixels (for example, via a Boolean “OR” operation), and if the result is zero (i.e. frame n−2 and frame n look the same in a local region), then a temporal ambient correction is performed by determining difference between a current center pixel in frame n and the same pixel in frame n−1, as indicated inFIG. 10C.

On the other hand, if the OR operation with adjacent motion flags result in a value of 1, this indicates that there may have been some nearby motion in this frame. In this case, prior frames may be ignored for the ambient correction, and a spatial correction utilizing adjacent pixels in frame n is performed. Any suitable weighting factor scheme may be used to perform this spatial correction.FIG. 10Dshows one non-limiting example of a suitable weighting factor scheme for a 5×5 pixel spatial correction.

The determination of whether to utilize a 5×5 or a 3×3 pixel region for ambient correction may depend upon factors such as the resolution and stability of the image sensor. For example, a 3×3 region may yield a slightly noisier result, while a 5×5 region may blur the result slightly. Other region sizes may be used, including but not limited to a 1×3 region (which may be noisier than a 3×3 region). It will be understood that these specific examples are presented for the purpose of example, and are not intended to be limiting in any manner.

FIG. 11shows a flow diagram depicting a method1100for performing an ambient light correction that takes into account the various factors described above. The method ofFIG. 11may be performed on a pixel-by-pixel basis, or in any other suitable manner. Method1100first comprises, at1102, acquiring one or more image data frames, and then, at1104, determining for an image data frame whether the global ambient is below a threshold value. This can be determined, for example, by subtracting the sum of the intensities of all pixels in a first field from the sum of all intensities of pixels in a second field, and determining if the result of the calculation is below a threshold value.

If the global ambient is below a threshold value, then method1100ends without performing any correction. On the other hand, if the global ambient is not below a threshold value, then method1100comprises, at1106, determining whether any motion is perceived in the intensity data. This may be performed, for example, by subtracting the intensity value for the pixel in the current frame (frame n) from the intensity value for the same pixel in frame n−2 (as the same pixel in n−1 has a different ambient exposure time). If the difference between these intensity values is sufficiently small, then it can be determined that the intensity data contains no motion information. In this case, a temporal local correction that utilizes no spatial information may be selected, as indicated at1108. On the other hand, if the differences between the intensity values is sufficiently large, it can be assumed that the pixel contains motion data (as long as the frame rate has been corrected for any periodically fluctuating ambient light), and either a spatial or a temporal-spatial correction may be selected, as indicated at1110.

The decision whether to utilize a spatial or temporal-spatial correction may be made in any suitable manner. Generally, a spatial correction may be used where all spatial variation in a frame can be corrected with other information in the frame. One example of a method for making this determination is as follows. First, if any pixels in row (i−1) of the sample matrix differ significantly from the pixels in the same column in row (i+1), there is spatial information that may be corrected via a temporal-spatial correction. Likewise, if any of the pixels in row (i) of the sample matrix minus the mean for row (i) differs significantly from the corresponding pixels in row (i−1) minus the mean for the pixels in row (i−1) then there is spatial information that may be corrected via a temporal-spatial correction. In other cases where there is perceived motion but these conditions are not met, a spatial correction may be used. Alternatively, either a spatial or temporal-spatial may be used exclusive of the other where motion information is contained in a frame.

In addition to the above-described methods of correcting for comparing pixel values to correct for ambient light, various other image processing techniques may be performed in an ambient correction process. For example, image processing may be performed to compensate for light leakage between different light sets, as the light sets may or may not be discretely separated.

The correction calculations and calculation selection routine described above may be performed in any suitable manner. For example, in one embodiment, an FPGA (as shown at122inFIG. 1) may be programmed to perform a plurality of different correction calculations simultaneously for each frame. Then, the best ambient value for each pixel in a frame may be selected based upon the specific temporal and local characteristics of that pixel. Alternatively, the best ambient calculation for a pixel may be determined before performing the correction, such that one correction is performed for each pixel. It will be appreciated that these specific examples of how to perform an ambient correction from the intensity data integrated and collected are described for the purpose of illustration, and are not intended to be limiting in any manner.

As mentioned above, in some use environments, it may be acceptable to reduce the frame rate when correcting a frame of image data for ambient light. Therefore, rolling local light source220may be used to illuminate the display screen with local light every other image frame to allow temporally adjacent image frames to be compared to correct for ambient light. In such an embodiment, rolling image capture system202may be considered to have a single field of pixels that is exposed to equal durations of ambient light (tambient) but durations of local+ambient light (tlocal+ambient) in adjacent frames of image data.FIG. 12shows a timing diagram1200that depicts an embodiment of a method for correcting for ambient light in a rolling image capture system via such a method. As with the timing diagram ofFIG. 3, timing diagram1200depicts illumination cycles for four light sets, and also shows how lighting for 1080 rows of pixels may be divided among the four light sets (i.e. bands of backlight). However, it will be understood that a rolling local light source may comprise any other suitable number of sets of light sets, and may divide the local lighting of the rows or columns of pixels of a display screen in any other suitable manner.

First referring to frame n ofFIG. 12, it can be seen that light set1turns off when the readout and reset of row1occurs, and remains off until the readout and reset of row1occurs for frame n+1, at which time the backlight turns on. In this manner, row1integrates ambient light for the duration of the frame n readout, and then integrates ambient+local light for the duration of the frame n+1 readout. In this manner, frame n comprises local light+ambient light integrated during the readout cycle before frame n, and frame n+1 comprises ambient light, but no local light, integrated during the readout of frame n. Light sets2,3and4are turned on and off in a similar manner, except that the turning off and on of these light sets are synchronized with the readout of rows271,541and811, respectively.

In this manner, each row of pixels of the image sensor is exposed to an entire frame of ambient+local light, followed by an entire frame of ambient light without local light. Thus, frames n and n+1 may be compared or otherwise mathematically manipulated to correct for ambient light. While the embodiment ofFIG. 12shows alternating frames with local light and with no local light, it will be understood that any other suitable pattern and/or ratio of frames with local light and frames with no local light may be used.

Other embodiments of interactive display devices may comprise a global local light source, rather than a rolling local light source, used in conjunction with a rolling image capture device.FIG. 13shows an embodiment of such an interactive display device1300, where the global local light source is indicated at1302and the rolling image capture device is illustrated at1304.

As described above, simply turning a single global local light source on and off at a 50% time cycle to capture alternating images with and without local lighting may result in the rolling image capture system integrating local light for different durations, thereby causing difficulties with ambient correction. Further, some pixels may be exposed to equal amounts of local light each frame, thereby preventing ambient correction for those pixels.

Therefore,FIG. 14shows a timing diagram1400that addresses the problem of exposure to local light for uneven durations by shortening the local illumination to a brief, bright flash of local light, thereby essentially turning the rolling image capture system into a global shutter sensor in terms of local light. InFIG. 14, the local illumination pattern is shown at1402, and the image sensor readout pattern is shown at1404, both as a function of time. As depicted, for the acquisition of frame n, the local light source emits a short flash of light1406after the completion of a previous readout cycle1408and before starting a next readout cycle1410. In this manner, all pixels of the image sensor may be exposed approximately equally to the local light during the acquisition of image n. On the other hand, the local light source does not emit a similar flash for the acquisition of image n+1. Because the ambient exposure of images n and n+1 occurs for an equal time, images n and n+1 may be used to correct for ambient light by subtraction of the images or other mathematical operation. It will be understood that the integration cycles and/or illumination cycles may be synchronized with any locally oscillating ambient light sources (e.g. incandescent lights with a 60 Hz line frequency) to help to ensure consistent performance.

It will be noted that the ambient correction methods illustrated inFIGS. 12 and 14may cause a 0.5× decrease in frame rate, as every two images acquired are combined into a single image for object detection. Therefore, in applications where tracking of relatively fast motion is desired, the timing method illustrated inFIG. 3may be used in conjunction with a spatial ambient correction method to preserve frame rate. Further, while disclosed herein in the context of an interactive display device, it will be appreciated that the disclosed embodiments may also be used in any other suitable optical touch-sensitive device, as well as in any other touch-sensitive device in which a background signal correction may be performed to improve device performance.

It will further be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated may be performed in the sequence illustrated, in other sequences, in parallel, or in some cases omitted. Likewise, the order of any of the above-described processes is not necessarily required to achieve the features and/or results of the embodiments described herein, but is provided for ease of illustration and description.