Non-uniform correction illumination pattern

An example system in accordance with aspects of the present disclosure includes a controller and a camera communicatively coupled to the controller to capture an image of a work surface. A projector is also provided and is coupled to the controller to project an illumination pattern onto the work surface during image capture by the camera.

BACKGROUND

When using a camera to take a photograph, there must be sufficient lighting. Some cameras have built-in flashes while other cameras use external lighting sources. Insufficient lighting can render the resulting photograph less than desirable.

DETAILED DESCRIPTION

A mixed reality system is one in which a projector is used to project an image on a work surface while one or more cameras may be present to monitor and detect objects placed on or around the work surface by a user. In one mode of usage, a user can view an image (e.g., a document, a photograph, a video, etc.) projected onto the work surface by the projector and then desire to take a digital photograph of an object placed on the work surface. The object could be, for example, a photograph or a three dimensional (3D) object. References to taking a digital photograph, capturing an image, and acquiring an image are synonymous. When using one of the system's built-in cameras, the projector's mode of operation changes from displaying an image (document, photograph, video, etc.) to illuminating the work surface so as to use the projector as a lighting source for the camera. As soon as the image has been acquired by the camera, the projector reverts back to displaying the original image. The projector thus functions as a “flash” for the camera during image capture.

If the illumination source (the projector) is configured to project a completely uniform illumination pattern, the light pattern received by the camera after the uniform illumination pattern reflects off of the work surface and object(s) located thereon may be non-uniform due to various irregularities such as project lens non-uniformity and geometric non-uniformities resulting from the angle and distance traveled by the light rays to reach the work surface. The resulting image of the object as captured by the camera may look different depending on where on the work surface the user places the object. This problem is addressed as described below by calibrating the system to compute a non-uniform correction illumination pattern. In general, a uniform illumination pattern is solid white. That is, each output illumination pixel of the uniform illumination pattern is the same as all other pixels in the image. In some implementations, the output illumination pixels are each set to an individually predetermined output intensity.

During the calibration process, the system described below captures an image of a blank work surface using a uniform illumination pattern. A blank work surface is a work surface with no objects (e.g., photographs, documents, etc.) placed thereon and no image projected onto it by the projector. The resulting captured image then may be inverted to produce a non-uniform correction illumination pattern. The non-uniform correction illumination pattern subsequently may be used as the projected illumination pattern during image acquisition. The non-uniform correction illumination pattern is computed in such a way that the irregularities noted above cause the light reflected off the work surface into the camera's lens to generally be uniform.

The illustrative system described below includes a projector that projects an image onto a surface by way of a reflecting mirror. Other suitable implementations include a direct projector, that is, a projector that projects light and an image directly onto the viewing surface rather than by way of a reflecting mirror. Further still, the camera described may be a component of a communication device such as a smart phone. Moreover, in some implementations, the system may be part an all-in-one computer that comprises, among other things, a camera, a projector, and a display. The following description pertains to one implementation but other implementations are possible as well.

EXAMPLE EMBODIMENTS

FIGS. 1A and 1Bare perspective, exterior views illustrating one example of a projection capture system10and an interactive workspace12associated with system10.FIG. 2is a perspective view illustrating one example of a projection capture system10with exterior housing13removed. Referring toFIGS. 1A,1B, and2, projection capture system10includes a digital camera14and a projector16. Camera14is usable, for example to capture an image of an object20in workspace12and projector16may be used to project an object image22into workspace12. Camera14may be a color camera. In some examples, camera14is usable to capture an image of the projected object image22. The lower part of housing13includes a transparent window21over projector16(and infrared camera30).

In the example shown inFIG. 1A, a two dimensional object20(e.g., a hardcopy of a photograph) placed onto a work surface24in workspace12has been photographed by camera14(FIG. 2). Object20is shown removed to the side of workspace12, and object image22is projected onto the work surface24. The object image22itself can be photographed by camera14(FIG. 2) and/or otherwise manipulated by a user and re-projected into workspace12. In the example shown inFIG. 1B, a three dimensional object20(a cube) placed onto work surface24has been photographed by camera14(FIG. 2) and then removed to the side of workspace12. An object image22is projected into workspace12where the object image can be photographed by camera12and/or otherwise manipulated by a user and re-projected into workspace12.

In one example implementation for system10, projector16is configured to project object image22into the same position in workspace24as the position of object20when its image was captured by camera14. Thus, a one-to-one scale digital duplicate object image22of an object20can be projected over the original allowing a digital duplicate in its place to be manipulated, moved, and otherwise altered as desired by a local user or by multiple remote users collaborating in the same projected workspace12. The projected image can also be shifted away from the original, allowing a user to work with the original and the duplicate together in the same workspace12.

InFIG. 1A, work surface24is part of a desktop or other underlying support structure23. InFIG. 1B, work surface24is on a portable mat25that may include touch sensitive areas. InFIG. 1A, for example, a user control panel27is projected on to work surface24while inFIG. 1Bcontrol panel27may be embedded in a touch sensitive area of mat25. Similarly, an A4, letter or other standard size document placement area29may be projected onto work surface24inFIG. 1Aor printed on a mat25inFIG. 1B. Other configurations for work surface24are possible as well. For example, it may be desirable in some applications for system10to use an otherwise blank mat25to control the color, texture, or other characteristics of work surface24, and thus control panel27and document placement area29may be projected onto the blank mat25inFIG. 1Bjust as they are projected on to the desktop23inFIG. 1A.

In the examples shown inFIGS. 1-2, system10includes an infrared digital stylus28and an infrared camera30for detecting stylus28in workspace12. The stylus28may be battery-operated and rest, when not in use in stylus charging dock54(FIGS. 2-4). The stylus28may include an infrared light, a touch sensitive nib switch to turn on and off the infrared light automatically based on touch, and a manual on/off switch to manually turn the infrared light on and off. The infrared light may be positioned, for example, at or near the tip of the stylus. Infrared light from the stylus reflects off of mirror38and is received and detected by infrared camera30(FIGS. 2-4). Infrared camera30is used to track movement of the stylus28, thereby permitting the user to move the stylus about on the work surface while system10tracks its movement. The stylus can thus be used as a pointing device (similar to a mouse), a writing instrument, a drawing instrument, etc.

Although any suitable user input device may be used, a digital stylus (stylus28) has the advantage of allowing input in three dimensions, including along work surface24and without a sensing pad or other special surface. Thus, system10can be used on a greater variety of work surfaces24. Also, the usually horizontal orientation of work surface24makes it useful for many common tasks. The ability to use traditional writing instruments on work surface24is advantageous over vertical or mobile computing interfaces. Projecting an interactive display on to a working desktop mixes computing tasks with the standard objects that may exist on a real desktop. Thus physical objects can coexist with projected objects. As such, the comfort of using real writing instruments as well as their digital counterparts (like stylus28) is an effective use model. A three-dimensional pad-free digital stylus enables annotation on top of or next to physical objects without having a sensing pad get in the way of using traditional instruments on work surface24.

Referring toFIGS. 1-4, projector16is positioned near base36outside projector display area34(FIG. 4) and focused on mirror38so that light from projector16is reflected off mirror38and onto workspace12. Projector16and mirror38define a three dimensional display space53in workspace12within which projector16can effectively display images. Projector display space53overlaps camera capture space51(FIGS. 3-4) and is bounded in the X and Y dimensions by display area34on work surface24.

Projector16may include any suitable light projector. In one example, the projector may be a liquid crystal on silicon (LCOS) projector or a digital light processing projector which is advantageously compact and power efficient. Projector16may also employ a shift lens to allow for complete optical keystone correction in the projected image. The use of mirror38increases the length of the projector's effective light path, thereby mimicking an overhead placement of projector16, while still allowing a commercially reasonable height for an integrated, standalone device.

As explained previously, the projector16may serve as the light source for camera14during image capturing. Camera capture area32(FIG. 3) and projector display area34(FIG. 4) substantially overlap on work surface24. Thus, a substantial operating efficiency can be gained using projector16both for projecting images and for camera lighting.

Since projector16acts as the light source for camera12for image capture, the projector light should be bright enough to swamp out any ambient light that might cause defects from specular glare. In some examples, a projector light of 200 lumens or greater may be sufficiently bright to swamp out ambient light for the typical desktop application for system10. For still image capture and if the projector is based on light emitting diode (LED) technology, the projector's red, green, and blue LED's can be turned on simultaneously for the camera flash to increase light brightness in workspace12, helping swamp out ambient light and allowing faster shutter speeds and/or smaller apertures to reduce noise in the image.

As explained above, due to various irregularities involved with the projector16relative to the work surface24, a uniform illumination pattern projected by the projector will result in a non-uniform light pattern as received by the camera14after being reflected off of work surface24thereby potentially causing undesirable image capture quality by camera14.

In some implementations, the projection capture system may be integrated in or attached to an all-in-one computer, a display, or a tablet device. For example, the projection capture system may be positioned atop a vertical support post that also supports an all-in-one computer (i.e., a display that also houses the computer's system board) or that supports a display. In such implementations, the projection capture system projects directly onto a work surface and/or or touchmat rather than reflecting off of a mirror.

FIG. 5shows an example of a light pattern105as received by camera14resulting from a uniform illumination pattern projected initially by projector16. The light pattern105is what is received by the camera14, and is not the illumination pattern projected by the projector16. The projector16, in the example ofFIG. 5, projected an illumination pattern that was substantially uniform (i.e., every pixel has a substantially similar illumination level). The received pattern105has been normalized to an illumination level between 0 and 1 as indicated by the vertical axis. The example light pattern105has a maximum value of approximately 1 as indicated by reference numeral110. The maximum value at point110represents the point in the captured image that has the highest signal reception and may be the closest point to the projector16. Away from point110, the illumination level decreases in both the X and Y directions as shown.

Referring toFIG. 6and in accordance with the disclosed principles, the system10performs a calibration process by which a non-uniform correction illumination pattern130is computed. The non-uniform correction illumination pattern130, such as that shown in the example ofFIG. 6, is the pattern that is projected by projector16(not the pattern actually received by camera14). The correction illumination pattern is computed in such a way that upon its reflection off of work surface24, the light pattern received into the camera14is substantially uniform. Because the camera receives a substantially uniform illumination pattern, image capture is improved relative to image capturing that would have resulted with the light pattern ofFIG. 5.

The Hardware:

FIG. 7shows an example in which system10includes a controller18coupled to the camera14and projector16. A storage device15is also provided for storing the non-uniform correction illumination pattern17for subsequent use during image capture by camera14. During image capture, the controller18retrieves the non-uniform correction illumination pattern17and causes the projector16to project the non-uniform correction illumination pattern17while camera14acquires the image.

FIG. 8provides additional detail of another example of the projection capture system10. The projection capture system10includes the camera14, projector16, and controller18as noted above. The system10also includes the mirror38and a user input device26. The user input device26may include the digital stylus28. The controller18may include a processor42, memory44, and an input/output device46. The memory44is any suitable type of non-transitory computer-readable storage device such as volatile storage (e.g., random access memory), non-volatile storage (e.g., hard disk, optical disc, etc.), or combinations of both volatile and non-volatile storage devices. Memory44may include code that is executable by processor42to implement some or all of the functionality described herein. Thus, any function described herein as attributed to system10is implemented or controlled by controller18and, more specifically in the example ofFIG. 8is implemented by processor42executing software module(s) stored on memory44. In other implementations of projection capture system10, the controller18with processor42may be implemented as a state machine in an application specific integrated circuit (ASIC).

The input/output device46may receive information from or send information to an external device. Such information, for example, may be information to be displayed on work surface24, or acquired images to be transmitted to an external device.

Projection capture system10inFIG. 8may also include an object recognition device (ORD)104for distinguishing between real and virtual objects in the workspace. In the example shown, object recognition device104includes an infrared camera106, an infrared light108, and a depth sensor109. In some examples, a combination of infrared camera106, camera14, and depth sensor109may be used to detect the absence of physical objects on the work surface24. The depth sensor109indicates when a 3D object is on the work surface. The infrared camera indicates when a thin object (e.g., photos, paper, etc.) are on the work surface. The camera14(which may be a color camera) indicates when the work surface contains color content when the projector projects white. The object recognition device104may be used to detect the presence or absent of a real object on work surface24. This ability is useful in the calibration process described below as the calibration process should be performed without any real objects on the work surface.

The infrared light108may be used with camera106to illuminate the workspace to improve object recognition. Also, while it may be possible to use the same infrared camera for both object recognition (camera106) and for sensing an IR stylus (camera30inFIGS. 2-4), it is expected that the camera frame rate for object recognition may not need to be as high as the frame rate for sensing stylus position but may require higher resolution. Consequently, it may be desirable for some implementations to use separate infrared cameras for object recognition and stylus sensing.

The calibration process for computing the non-uniform correction illumination pattern will now be described. The calibration process may be performed whenever no physical objects are placed on the work surface24, and no content is being projected by projector16onto the work surface, that is, the work surface is blank. A user of system10may manually trigger the performance of the calibration process using the digital stylus28by selecting a “calibration start” button displayed on the work surface or by other suitable means. Alternatively or additionally, the object recognition device104may be used to constantly or periodically detect when the work surface24is blanks and when the work surface is determined to be blank, the object recognition device104may trigger the controller18(e.g., send a signal to the controller) to perform a calibration process. The calibration process thus may be performed multiple times to compute the non-uniform correction illumination pattern. The non-uniform correction illumination pattern may change depending on various factors such as ambient lighting conditions.

The illustrative calibration process ofFIG. 9may be performed by controller18. The operations shown may be performed in the order shown or in a different order and two or more of the operations may be performed concurrently rather than serially. Upon determining work surface24to be blank (210) as described above using the object recognition device104, the calibration process includes controller18causing the camera14to acquire (214) an image of the blank work surface using a uniform illumination pattern. During the image acquisition of the blank work surface, whatever image the projector16was otherwise projecting is temporarily suspended and the projector16is caused to project the uniform illumination pattern. Due to the irregularities involved in system10relative to the work surface24, the resulting acquired image may have a non-uniform pattern such as that shown in the example ofFIG. 5.

At216, the acquired image is normalized (e.g., placed on a scale of 0 to 1). At218, the normalized image is inverted to compute the non-uniform correction illumination pattern. In one example, the inversion of the normalized acquired image is computed by subtracting each pixel of the normalized acquired image from 1 (assuming the normalization results in each pixel being in the range of 0 to 1). Once the non-uniform correction illumination pattern is computed, the pattern is saved to memory44for subsequent use in acquiring an image by camera14.

FIG. 10shows an example of a method for using camera14to acquire an image using the non-uniform correction illumination pattern. The method may be initiated manually by a user, for example, by a user selecting an “image capture” button displayed on work surface24by projector16.

At202, the method includes the controller18retrieving the non-uniform correction illumination pattern from memory44. At204, the method further includes projecting the non-uniform correction illumination pattern. During operation204, whatever image was already being projected by projector16, that image's projection is temporarily is suspended in favor of the projection of the non-uniform correction illumination pattern. The image is then captured by camera14at206. Once the image has been captured, the controller may again cause the projector to revert back to projecting whatever image was being projected before image capture occurred.