Patent Publication Number: US-8982261-B2

Title: Imaging with interleaved detection accumulations

Description:
BACKGROUND 
     In a virtual-object collaborative environment, remote collaborators can interact with and modify a shared virtual object. The shared virtual object can be in the form of visible images, instances of which are presented locally to respective collaborators. Interactions with the virtual object can be effected using human input devices. For example, the virtual object can be a virtual document page; a collaborator can annotate the virtual document page using an IR pen (a stylus with a tip that emits infra-red light). The annotations can then be presented to remote collaborators. In other words, the images of the document pages at the different locations can be reconciled to the effect that they all represent the same object. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The following figures represent examples and not the invention itself. 
         FIG. 1  is a schematic diagram of an imaging system in accordance with an example. 
         FIG. 2  is a flow chart of a process in accordance with an example. 
         FIG. 3  is a schematic diagram of a collaborative environment in accordance with an example. 
         FIG. 4  is a schematic diagram of an imaging system of the collaborative environment of  FIG. 3 . 
         FIG. 5  is a flow chart of a collaborative process in accordance with an example. 
     
    
    
     DETAILED DESCRIPTION 
     PCT Patent Application number PCT/US11/58896 filed 2011 Nov. 2 and entitled “Projection Capture System, Programming And Method” discloses a virtual-object collaborative environment (e.g., a mixed reality system) with respective imaging systems provided for each collaborator. Each imaging system includes a projector for projecting a visible image representing a virtual object, e.g., a document page. Each imaging system includes a visible-light camera to track changes in the virtual object, e.g., due to actions by the local collaborator using an IR pen. To this end, an IR camera is used to track the position and motion of the IR pen. 
     Such a system must address the challenge of color registration. First of all, the images taken by the IR camera must be aligned (registered) with the visible light image. Furthermore, if the visible-light camera uses separate sensor elements for color components (e.g., red, green, and blue) of the visible light, the resulting monochrome images must be registered to avoid undesirable artifacts in the composite image. Calibration and post-processing procedures may be available to address registration problems, but they can incur undesirable penalties in time and processing power. For example, the post-processing can increase the latency required to update a projected visible image based on a captured digital image. 
     Examples disclosed herein address registration challenges in part by using full-range sensor elements to capture all colors, albeit at different times. As explained below with respect to imaging system  100  ( FIG. 1 ), detections output by each sensor element are routed to at least three different color-dedicated accumulators as a function of time so that each accumulator accumulates detections for a respective color. As explained further below with reference to collaborative environment  300  ( FIGS. 3 and 4 ), the inclusion of an accumulator dedicated to IR detections addresses the problem of registration of the IR light to the visible light and provides for interleaving of the accumulations to minimize time-based registration problems even in the case of scenes with moving elements, e.g., a moving IR pen. 
     An imaging system  100 , shown in  FIG. 1 , includes a sensor array  102  of sensor elements  104 . Each sensor element  104  is to convert incident light  106  to detections  108 . Depending on the type of technology employed, detections  108  may take the form of electric charges or some other form. 
     In addition to sensor elements  104 , imaging system  100  includes detection accumulators  110 . Associated with each sensor element  104  is a set of n detection accumulators, where n is an integer greater than or equal to three (n≧3) so that, for example, at least a first detection accumulator  110 R can be dedicated to accumulating detections of red light, at least a second detection accumulator  110 G can be dedicated to accumulating detections of green light, and at least a third detection accumulator  110 B can be dedicated to detections of blue light. Alternatively, each of three or more detection accumulators can be dedicated to a respective dimension of a color space other than red-green-blue (RGB). Imaging system  100  further includes a switch set  112  of switches  114 . Each switch  114  is to route detections from a sensor element  104  to detection accumulators  110  as a function of the color of the incident light being converted to the detections. 
     An imaging process  200  that can be implemented using imaging system  100  or another imaging system is flow charted in  FIG. 2 . At  201 , incident light (e.g., incident a sensor element  104 ) is converted to detections. At  202 , the detections are routed (e.g., by switch set  114 ) to “color-dedicated” detection accumulators  100  as a function of the color of the incident light being converted to the detections. 
     The use of plural color-dedicated accumulators advantageously decouples the process of inputting detection to the accumulators from the process of outputting accumulations from accumulators. If only one accumulator is used for plural colors, the accumulator must be read out and reset whenever the color to be detected is changed. When plural dedicated-color accumulators are used, one color can be read while another is accumulated. Also, when plural accumulators are used (per sensor element), accumulations can be “interleaved”, i.e., a first color can be accumulated, then another color can be accumulated, and then the first color can be accumulated again, all without an intervening readout or reset. 
     For example, to achieve a frame duration of 30 milliseconds (ms) using a single accumulator, the following pattern could be used: 10 ms of red, readout and reset, 10 ms of green, readout and reset, 10 ms of blue, readout and reset, and repeat. If there are moving elements in the scene or object being imaged, there will be an average of about 15 ms of opportunity for color misregistration due to motion in the scene being imaged. The examples described herein reduce this opportunity for color misregistration due to scene motion. 
     Plural color-dedicated accumulators would permit an interleaving pattern such as 5 ms red, 5 ms green, 5 ms blue, 5 ms red, 5 ms green, 5 ms blue, readout and reset, and repeat. In this case, the average time available for misregistration is reduced by half to about 7.5 ms. The time available for misregistration due to scene movement can be reduced to less than 2 ms by cycling through 1 ms phases for each color. In that case, the average misregistration between colors is limited by the amount of scene movement that occurs with 2 ms. Thus, the interleavings provided for by using plural dedicated color accumulators can readily reduce color misregistration by an order of magnitude or more. Further advantages are attainable when infra-red capabilities are integrated into the camera, as in the following example. 
     A collaborative environment  300  is illustrated schematically in  FIG. 3  including imaging systems  310  and  320  coupled by a network  330 . While two imaging systems are represented in  FIG. 3 , other examples include other numbers of imaging systems. Imaging system  310  includes an RGB projector  312 , an RGB-IR camera  314 , and an IR pen  316 . Imaging system  320  is essentially similar to imaging system  310 ; imaging system  320  includes an RGB projector  322 , an RGB-IR camera  324 , and an IR pen  326 . 
     Imaging system  310  can be used to generate a machine-readable digital image  332  of a physical object  334  (which can be, for example, a three-dimensional object or a page of a document). To this end, RGB projector  312  can be used to illuminate physical object  334 , and RGB-IR camera  314  can capture light reflected by physical object  334  to produce digital image  332 . Digital image  332  can then be input to RGB projector  312  so that RGB projector  312  projects (e.g., generates) a human-perceptible visible image  336 . 
     Visible image  336  can serve as a virtual object that can be manipulated/modified by a user, e.g., using IR pen  316  as well as other human-input devices. IR pen  316  is a stylus with a tip that emits IR light; the tip may also emit visible light to let a user known when the IR light is active. A local collaborator (or other user) can use IR pen  316 , for example, to control the position of a cursor in visible image  336  and to manipulate or annotate visible image  336  by gesturing or “writing” with IR pen  316 . RGB-IR camera  314  detects the position of IR pen  316  so that the position of IR pen  316  can be tracked. Commands can be implemented and cursor position adjusted in digital image  332 ; as it is updated, digital image  332  can be input to RGB projector  312  to update visible image  336 . 
     In addition to its use locally with respect to imaging system  310 , digital image  332  can be communicated over network  330  to imaging system  320 . There, digital image  332  can be input to RGB projector  322  to generate visible image  338 . Like visible image  336 , visible image  338  can be manipulated by a remote collaborator (or other user). e.g., using IR pen  326 . RGB-IR camera  324  can track the position of IR pen  326  and update digital image  332 . In practice, each imaging system  310 ,  320  maintains an instance of digital image  332 ; programs on the imaging systems ensure that both instances are updated so that visible images  336  and  338  are reconciled (i.e., synchronized) in near real time. Thus, local and remote collaborators can work together to dialog about and edit the virtual object represented by visible images  336  and  338 . 
     Collaborative environment  300  and its imaging systems  310  and  320  have some applications and some elements in common with their counterparts disclosed in PCT Patent Application number PCT/US11/58896 entitled “Projection Capture System, Programming And Method”. However, imaging systems disclosed in that application used separate cameras for visible colors and for IR. RGB-IR cameras  314  and  324  combine visible color capture and IR capture for simplicity, economy, and better registration of color and IR image components. 
     RGB-IR camera  314 , as shown in  FIG. 4 , includes a channel array  400  of detection channels  402 . Each detection channel  402  includes a respective sensor element  404 , a respective switch  406 , and a respective accumulator set  408 . Accordingly, channel array  400  includes a two-dimensional sensor array  410  of sensor elements  404 , a two-dimensional switch array  412  of switches  406 , and a two-dimensional accumulator-set array  414  of accumulator sets  408 . Sensor array may be formed on the back side of a backside-illuminated CMOS sensor, while switch array  412  and accumulator-set array  414  may be formed on the front side of the backside-illuminated CMOS sensor. Each accumulator set  408  includes plural color-dedicated accumulators  416 , in this case, a red-dedicated accumulator  416 R, a green-dedicated accumulator  416 G, a blue-dedicated accumulator  416 B, and an IR-dedicated accumulator  416 J. Accumulators  416  can be implemented as integrating capacitors. 
     Sensor elements  404  are “full-range” in that each sensor element  404  can detect incident red, green, blue, and infra-red light. In response to detection of incident light, e.g., in the form of photons, each sensor element  404  outputs “detections”, in this case, in the form of electrical charges. All accumulators  416  are essentially similar; the differing labels reflect the fact that, in use, each accumulator is dedicated to a respective color (e.g., red. green, blue, and infra-red). 
     Each switch  406  has outputs coupled to inputs of respective accumulators  416  of a respective accumulator set  408 . Accumulator outputs are coupled to readout circuitry  418 , which includes an analog-to-digital converter (ADC) for converting analog accumulator values to digital values. In other examples, ADCs are located between accumulators and readout circuitry. 
     The output of readout circuitry  418  is received by an image-data handler  420 , e.g., a computer, which can use the received digital data to construct and update digital image  332 . Digital image  332  is communicated from image-data handler  420  to RGB projector  312  to update visible image  336 . In addition, image-data handler  420  communicates via network  330  with its counterparts in other imaging systems, e.g., imaging system  320 , to reconcile (i.e., resolve differences between, synchronize, equalize) instances of digital image  332 . 
     A timing controller  422  controls and synchronizes (i.e., coordinates the timings for) switches  406 , readout circuitry  418 , and red, green, and blue emitters  424 R,  424 G,  424 B of RGB projector  312 . Each switch  406  is synchronized with projector  312  so that, at any given time, detections are routed to the accumulator corresponding to the color being emitted by projector  312 . During gaps in emissions by projector  312 , IR detections are routed to IR accumulators  416 J. Thus detections are routed to accumulators as a function of the color of the incident light from which the detections resulted. Timing controller  422  can include circuitry within RGB projector  312 , within RGB-IR camera  314 , and/or external to both RGB projector  312  and RGB-IR camera. 
     An imaging process  500 , flow charted in  FIG. 5 , can be implemented using collaborative environment  300  or another environment. At  501 , timings of an imaging system are controlled so that emitting by a projector, switching within a camera, and readout from the camera are synchronized (i.e., their timings are coordinated). In the context of collaborative environment  300 , timing controller  422  is configured to synchronize timings used for operation of projector  312 , camera switches  408 , and readout circuitry  418 . For example, while timing controller  422  is causing RGB projector  312  is emitting red, switch  406  is directing (presumably red) detections to red accumulator  416 R. For another example, while timing controller  422  is causing RGB projector  312  to not emit any color, timing controller  422  causes switch  406  to direct (presumably IR) detections to IR accumulator  416 J. Also, timing controller  422  causes readout circuitry  418  to read out respectively from accumulators  416 R,  416 G,  416 B, and  416 J, only while they are not receiving detections. 
     For example, RGB projector  312  can emit colors sequentially to yield emitted visible light  428 , resulting in visible light  430  to be incident to camera  314  due to reflections of emitted visible light  428 . Due to control of RGB projector  312  by timing controller  412 , incident visible light  430 , like emitted visible light  428 , can consist of cyclical phases, e.g., a red phase  430 R, a green phase  430 G, a blue phase  430 B, and a gap phase  430 X. Switches  406  route detections during red phases  430 R to red accumulator  416 R, detections during green phases  430 G to green accumulator  416 G, detections during blue phases  430 B to blue accumulator  416 B, and detections during gap phases  430 X to IR accumulator  416 J. The phases can have different durations and frequencies. For example, a pattern RGBXRGBXRGBX, etc., allows the position of IR PEN  316  (which may be moving) to be sampled more frequently than visible image  336  (which may be a stationary document). 
     At  502 , colors are emitted sequentially, i.e., one at a time. For example, the sequence can be RGBRGBRGB . . . in which a sequence RGB is repeated; alternatively, a sequence may not have a repeating pattern. In a case such as imaging system  310  (in which RGB-IR camera  314  detects more colors than RGB projector  312  emits), the pattern can include gaps, e.g., RGBXRGBX or RXGXBXRXGXBX, e.g., to allow a camera to detect light from a different source (such as infra-red from an IR pen). The number m of different colors represented is greater than or equal to three so that a full color image can be obtained by integrating over the different colors. The light emitted over each color phase of each cycle can be uniform (e.g., to illuminate a physical object) or image-bearing (e.g., when projecting an image). 
     At  503 , a collaborator or other entity interacts with a scene using a human interface device such as IR pen  316 . The scene can include a physical object, e.g., illuminated by RGB projector  312 , and/or a visible image, e.g., projected by projector  312 . For example, a collaborator may use IR pen  316  to point to a portion of visible image  336 ; collaborative environment  300  can then reconcile visible image  338  so that collaborators using imaging systems  310  and  320 , respectively, can focus on the same area of the common virtual object. Also, IR pen  316  can be used to annotate a document image (or even a physical object) or issue commands (e.g., “rotate”, “zoom”, etc.). 
     At  504 , incident light (visible and IR) is detected by sensor array  410 , resulting in detections. At  505 , switches are operated to direct detections to n≧m color-dedicated accumulations. Switches  406  can be controlled by timing controller  422  in synchronization with RGB projector  312  so that the detections are routed to the accumulator  416 R,  416 G,  416 B, or  416 J corresponding to the phase of the incident light causing the detections. For example, while RGB projector  312  is emitting red light, switches  406  are set so that detections are routed to “red” accumulator  416 R. For another example, during gaps between color emissions by RGB projector  312 , switches  406  are set so that detections are routed to IR accumulator  416 J. Note that the color phases can vary in number and/or duration by color. For example, there can be two green cycles or a double-length green cycle to take advantage of the fact that green is perceived as most closely related to intensity (brightness), to which human eyes are more sensitive, than hue. 
     At  506 , accumulations are interleaved. Herein, accumulations are “interleaved” when an accumulation within one accumulator coupled to a switch to receive detections from a sensor includes detections acquired both before and after detections accumulated within another accumulator coupled to the switch to receive detections from the sensor. “Interleaving” can include single-color interleaving, e.g., RGBG, in which only one accumulator (in this case, green accumulator ( 416 G) is interleaved, and all-color interleaving, e.g., RGBRGB in which all accumulations are interleaved. Interleaving is made possible by the presence of dedicated-color accumulators (as opposed to using a single accumulator that must be emptied before it can be used to store detections associated with the next color). As explained below, interleaving makes possible dramatic reductions in problems with color misregistration due to moving elements in a scene. 
     At  507 , accumulations are read out; once its contents have been read, an accumulator is reset (e.g., to zero). For embodiments that do not employ interleaving, accumulators can be read out and reset each color cycle, e.g., RGB, or RGBJ. For embodiments that do employ interleaving, readouts/resets can occur after plural color cycles, e.g., RGBRGB or RGBJRGBJ (where J can correspond to switch settings in which projector  312  is not emitting a color and detections routed are to IR accumulator  416 J. 
     In some examples, the number of color-phase cycles (e.g., RGBX instances) can be large, e.g., tens or hundreds between readouts so that the accumulations in a set of accumulators are highly overlapped in time so as to minimize misregistration of colors in a digital image (without requiring color registration post processing, which can be time consuming and, thus, delay collaborative image updating). In the course of  507 , readout circuitry  418  can convert analog values stored in the accumulators to digital values for use by image-data handler  420 . 
     At  508 , a digital image  332  is created/updated based on the data read out from the accumulators. At  509  the image data is interpreted, e.g., by image data handler  420  to track IR pen position and motion. Note that, if the IR pen is to be tracked against a stationary object or projected image, the pen position can be sampled more frequently than the object or projected image. For example, projector  312  can emit repetitions of RXGXBX and detections can be routed with repetitions of the pattern RJGJBJ. 
     At  510 , instances of a digital image at different imaging systems are reconciled. For example, copies of digital image  332  ( FIG. 3 ) stored by respective imaging systems  310  and  320  can be reconciled. At  511 , the reconciled digital images can be input to respective projectors to generate/update visible images. Since digital images are reconciled, the visible images generated from them are also reconciled. For example, visible images  336  and  338  ( FIG. 3 ) are reconciled so that collaborators using respective imaging systems  310  and  320  can operate on the same virtual objects. 
     In other examples, other color sets are used. For example, ultra-violet light may be detected instead of or in addition to infra-red. Detection color phases may be longer and/or more frequent than others. In some examples, there is more than one accumulator per color per sensor. For example, plural IR accumulators can be used to detect phase and thus depth of an IR pen. 
     In the illustrated examples, m&lt;n, i.e., the number m of emitters in the projector involved in time-sequential emissions is less than or equal to the number n of accumulators per accumulator set, i.e., per sensor channel. For example, three accumulators can be used with a projector that emits three colors (RGB), or four accumulators can be used with a projector that emits four colors (RGB-IR), or four accumulators can be used with a projector that emits three colors (RGB), with the fourth accumulator used to detect IR from a source other than the projector. 
     In alternative examples, m&gt;n. For example, a projector can have m&gt;n emitters, but limit the number used to n. Thus, a projector may have emitters for R, G, B, IR, and UV, but use only one of IR and UV at a time. For another example, a projector can have six emitters: red, green, blue, magenta, cyan, and yellow, and three accumulators can be used during even cycles for red, green, and blue and during odd cycles for magenta, cyan, and yellow. In the illustrated examples, each accumulator is dedicated to accumulating detections only for a single color (R, G, B, or IR). However, in some examples, e.g., where m&gt;n, the color accumulator by an accumulator may be changed, e.g., for different readout cycles. 
     Herein, a “system” is a set of interacting non-transitory tangible elements, wherein the elements can be, by way of example and not of limitation, mechanical components, electrical elements, atoms, physical encodings of instructions, and process segments. Herein “device” refers to a hardware or hardware+software system. Herein, “process” refers to a sequence of actions resulting in or involving a physical transformation. An “imaging process” is a process for creating visible and/or digital images. 
     An “imaging system” is a system that creates visible and/or digital images. Herein, “image” refers to a (uniform or non-uniform) spatial distribution of light or a digital representation of such a distribution of light. A “visible image” is an image that a human can perceive; a “digital image” is a non-transitory tangible encoding of data in digital format that represents a visible image (and may include other data). 
     Herein, a “virtual object” is a digitally-defined object that represents a human-manipulable object and that can be manipulated by a human as if it were that manipulable object. For example, a projected image of a document can represent a hardcopy document and can be manipulated, e.g., annotated using an IR pen, (more or less) as if it were the hardcopy document. 
     Herein “light” is electromagnetic radiation. “Light” encompasses “visible light”, which consists of light within a wavelength range perceptible to the human eye, and “invisible light”, which is light outside the wavelength range perceptible to the human eye and encompasses “infra-red light” and “ultra-violet light”. A “sensor” is a hardware device for converting incident light (i.e., light that reaches the sensor) into detections. A “sensor array” is a sensor constituted by an array, typically two-dimensional) of “sensor elements”, each of which is a sensor in its own right. 
     Herein, a “detection” is a tangible entity produced in response to incident light. A detection can be, for example, an electrical charge or an set of electrical charges. Alternatively, a detection can be a voltage level or a light intensity level (where the sensor effectively generates amplified light in response to incident light), or a detection can take another form. 
     Herein, a “detection accumulator” or just “accumulator” is a device that accumulates or counts detections. For example, an accumulator can be an integrating capacitor that increases its charge level as detections in the form of electrical charges are received. In other examples, an accumulator can take another form such as a counter that counts light pulses generated in response to incident light. 
     Herein, “switch” refers to a device with an input, plural outputs, and a control port for receiving a signal that selects one of the outputs to be connected to the input. In the present case, a switch input is connected to an output of a sensor element for receiving a detection therefrom; each switch output is coupled to a respective color-dedicated accumulator to, when coupled to the switch input, direct the detection to the respective accumulator; and each control port is coupled to the timing controller so that the switch settings can be synchronized with projector emitters. Herein, “partitioned” means “allocated”, in the sense that each detection is allocated to a respective accumulator according to the switch setting at the time the detection is made. Herein, “set” requires at least two elements. 
     Herein, a “readout system” and “readout subsystem” refer to systems for reading out values from other devices, such as accumulators, e.g., to determine the number or amount of detections accumulated by an accumulator. “Reset” herein refers to initializing a device, e.g., setting an accumulator so that the amount of detections it represents is zero. 
     Herein, “computer” refers to a hardware machine for processing physically encoded data in accordance with physically encoded instructions. A “server” is a computer that performs services for other computers. Depending on context, reference to a computer or server may or may not include software installed on the computer. Herein, “storage medium” and “storage media” refer to a system including non-transitory tangible material in or on which information is or can be encoded with information including data and instructions. “Computer-readable” refers to storage media in which information is encoded in computer-readable form. 
     In this specification, related art is discussed for expository purposes. Related art labeled “prior art”, if any, is admitted prior art. Related art not labeled “prior art” is not admitted prior art. In the claims, “said” introduces elements for which there is explicit verbatim antecedent basis; “the” introduces elements for which the antecedent basis may be implicit. The illustrated and other described embodiments, as well as modifications thereto and variations thereupon are within the scope of the following claims.