Patent ID: 12212899

DETAILED DESCRIPTION

Projector stacking is a projection-based display method for projecting and stacking images side by side, or overlaid on a same area, onto an image projection surface to display a combined image. The images are projected and aligned to overlap at least partially (e.g., at the image edges) to display a larger combined image and/or an image with higher brightness and/or resolution. Projector stacking includes aligning and edge blending the images to display a smooth or uninterrupted (e.g., seamless) combined image. Aligning the individual projected images, which can be distorted in the projection process, corrects the geometry of the images to produce the combined image without distortion.

Without geometric correction, the projected images can be skewed on the image projection surface according to the respective projection angles from the projectors to the image projection surface. The images can be geometrically corrected by aligning the relative positions of the projectors, and accordingly the respective projection angles of the projected images, with respect to the image projection surface. For example, the projectors can be aligned during a calibration process. Because the positions of the projectors are sensitive to shifts, such as because of movements, stresses, shocks, or vibrations in the projectors, the calibration process may be repeated over time to maintain the smooth or seamless nature of combined images.

The geometric correction of images can also be performed by image processing methods that are faster and more accurate than manually or mechanically aligning the projectors. Geometric correction based on image processing methods includes capturing initial projected images by one or more cameras, determining distortion information in the images, and processing to-be-projected images according to this information to mitigate the distortions in the displayed images. The image processing methods require the processing of multiple or different patterns of images to determine the image distortions, which can be computationally costly (e.g., in term of processing speed and time). Such methods can also depend on the relative positions of the cameras with respect to the projectors, which requires calibration.

This description includes examples suitable for the geometric correction of images with less complexity and computation cost in comparison to other image processing based geometric correction methods for projector stacking. The complexity and computation cost is reduced by processing fewer images and fewer image points in the initial captured images by respective cameras. The image points are obtained from overlap regions between fields of views of the cameras and processed to generate relative homography transforms between respective camera frames of reference, also referred to herein as pair-wise homography transforms. The pair-wise homography transforms represent the mapping of the image points between frames of reference of respective pairs of cameras. The mapping relationship between the frames of reference of the respective cameras is useful to map the images to a common camera frame of reference, which allows the geometric correction without calibrating the cameras. The images in the common camera frame of reference are then processed to determine corrected image geometries, also referred to herein as corrected quadrilaterals. The corrected quadrilaterals can then be processed for projecting new images after performing geometric correction for projector stacking. The new images can be processed, such as by an image warping engine, based on the corrected quadrilaterals to warp the new images prior to projection and accordingly compensate for the distortions in the display system. Projecting the warped new images produces aligned images with corrected geometries for displaying a new combined image according to projector stacking without distortion.

In some examples, the images processed based on the pair-wise homography transforms are two-dimensional images projected on a planar surface. This allows the processing of two images to obtain each pair-wise homography transform. A pair-wise homography transform for a pair of images can also be generated based on four image points from an overlap region between two fields of views of a respective pair of cameras. The four image points in the overlap region can be captured within the two fields of views. In other examples, a pair-wise homography transform can be generated with more than four image points.

In some examples, a pair of images is initially projected simultaneously in a 2×1 array to partially overlap on the image projection surface and display a combined image. A pair of cameras is configured (e.g., by adjusting the orientations of the cameras with respect to the image projection surface) with respective fields of views that are also partially overlapping to capture the pair of projected images. The four image points in the overlap region are captured by both cameras and processed to generate a pair-wise homography transform between the respective frames of reference of the two cameras. The captured images are processed to produce a pair of corrected quadrilaterals, which are processed in turn based on the pair-wise homography transform to perform the geometric correction for other to-be-projected pairs of images according to a 2×1 projector configuration. This method can be extended, such as by determining pair-wise homography transforms for multiple pairs of images, to display systems in a 2×2 projector configuration, or other projector configurations that project more images. In some examples, the images can partially overlap to form a larger combined image or can fully overlap to display an image with higher brightness and/or resolution.

FIG.1is a diagram of a display system100, in accordance with various examples. The display system100includes a display device101which is a projection-based display that projects images or video for viewing. As shown inFIG.1, the display device101includes projectors105(e.g.,105A and105B) which are configured to project, simultaneously, respective images onto an image projection surface110in a certain projector configuration. For example, the display device101includes the projectors105stacked in a rack or an arm assembly to project the images in a 2×1 or 2×2 projector configuration. The image projection surface110can be a wall or a wall mounted screen. In other examples, the image projection surface110may be a screen of a heads up display (HUD), a projection surface in a vehicle such as a windshield, an outdoor environment such as a road, an AR or VR combiner, a three-dimensional (3D) display screen, or other display surfaces for projection-based display systems.

A projector105is configured to project a respective modulated light112onto the image projection surface110to display the respective image114. For example, as shown inFIG.1, the display device101includes two projectors105A and105B that project, according to a 2×1 projector configuration, two respective images114A and114B that are partially overlapping in a 2×1 array on the image projection surface110. The partially overlapping images114A and114B are aligned on the image projection surface110to display a combined image. In other examples, the display device101includes four projectors105that project, according to a 2×2 projector configuration, four respective images114which partially overlap in a 2×2 array on the image projection surface110. In further examples, the display device101includes other numbers of projectors105in a certain projector configuration to project respective images114that fully or partially overlap on the image projection surface110. For example, the display device101can include three projectors105that project, according to a 3×1 projector configuration, three respective images114which partially overlap in a 3×1 array on the image projection surface110. In general, the display device101can include L projectors105that project, according to a M×N projector configuration, L respective images114which partially overlap in a M×N array on the image projection surface110, where M and N are integers and L=M×N.

Each modulated light112(e.g.,112A and112B) can be modulated in each respective projector105(e.g.,105A and105B) by each respective light modulator115(e.g.,115A and115B) to project respective images114(e.g.,114A and114B), such as video frames, onto the image projection surface110. A light modulator115can be a microelectromechanical system (MEMS) based SLM, such as a DMD, or a liquid crystal-based SLM, such as an LCD or a liquid crystal on silicon (LCoS) device. Each light modulator115(e.g.,115A and115B) modulates the intensity of a light from one or more respective light sources120(e.g.,120A and120B) based on optical elements that are controlled to manipulate the light and accordingly form the pixels of a respective displayed image114(e.g.,114A and114B). In some examples, the light modulator115is a DMD, where the optical elements are adjustable tilting micromirrors that are tilted by applying voltages to the micromirrors through respective electrodes. The micromirrors are tilted to project dark pixels or bright pixels with color shades. In other examples, the light modulator115is an LCD or an LCoS device, where the optical elements are liquid crystals that are controlled by voltage to modulate the intensity of light across the image pixels. The intensity of light is modulated by applying voltage to the liquid crystals, which reorients the crystals, also referred to herein as switching the crystals, and accordingly controls the amount of light projected per pixel. The optical elements can be a transmissive array of liquid crystal cells such as in an LCD, or a reflective array of liquid crystal cells such as in an LCoS device. The cells of liquid crystals can be controlled by voltages, through respective electrodes, to modulate light.

In other examples, the light modulator115can be a phase light modulator (PLM) or a ferroelectric liquid crystal on silicon (FLCoS) device. A PLM can be a MEMS device including micromirrors that have adjustable heights with respect to the PLM surface. The heights of the micromirrors can be adjusted by applying voltages. The micromirrors may be controlled with different voltages to form a diffraction surface on the PLM. For example, each micromirror can be coupled to respective electrodes for applying a voltage and controlling the micromirror independently from the other micromirrors of the PLM. The diffraction surface is a phase altering reflective surface to light incident from one or more light sources120onto the surface of the light modulator115. The phase altering reflective surface represents a hologram for projecting illumination patterns of light that form an image on an image projection surface. The hologram is formed as a diffraction surface by adjusting the heights of the micromirrors of the PLM. The hologram is formed based on an image that is to be displayed by projecting the light on the image projection surface110. An FLCoS device includes ferroelectric liquid crystals (FLCs) that have a faster voltage response than other liquid crystal devices (e.g., LCDs and LCoS devices) and accordingly can project images at a higher rate. Other examples of the light modulator115include micro-light emitting diodes (micro-LEDs) and micro-organic light emitting diodes (micro-OLEDs).

The modulated light112can be formed as a combination of color modes (e.g., blue, green, and red) from an incident light125, which is generated by one or more light sources120. For example, three color modes can provide three basic color components for displaying an image in full color. The color modes in the incident light125can be transmitted concurrently or by time multiplexing the light sources120. The incident light125with the different color modes is modulated by the light modulator115in the projector105to produce the modulated light112for displaying images114or video on the image projection surface110.

In other examples, the display device101can include a single projector105including multiple light modulators115that each forms a respective modulated light112. Each light modulator115(e.g.,115A and115B) of the display device101can be optically coupled to respective light sources120(e.g.,120A and120B) and to respective controllers130(e.g.,130A and130B) or the same controller130. In some examples, the light modulators115can share a single light source120. The projector105includes one or more controllers130configured to control the light modulator115and the light sources120to display the images114or video. For example, each controller130(e.g.,130A and130B) can include a respective first controller132(e.g.,132A and132B) for controlling the respective light sources120(e.g.,120A and120B) to transmit a respective incident light125(e.g.,125A and125B) concurrently or consecutively by time multiplexing. The controllers130(e.g.,130A and130B) can also include a respective second controller134(e.g.,134A and134B) for controlling a respective light modulator115(e.g.,115A and115B) to modulate the respective incident light125(e.g.,125A and125B) from the respective light sources120(e.g.,120A and120B).

The first controller132and the second controller134can be different controllers. The first controller132can be a digital controller configured to switch the light sources120on and off. In other examples, the first controller132can be an analog controller that changes the level of light intensity of the incident light125from the light sources120. The analog controller can also transmit pulse width modulation (PWM) signals to the light modulator115to synchronize the adjustment of the optical elements in the light modulator115with the transmission of the incident light125from the light sources120. The second controller134may be an analog or a digital controller that switches the optical elements of the light modulator115. For example, the second controller134is an analog controller or a digital controller that switches the angles of micromirrors of an SLM or the heights of micromirrors of a PLM. In some examples, the second controller134is a digital controller coupled to a static random access memory (SRAM) (not shown) including an array of memory cells each configured to store voltage values, such as in bits, to adjust respective micromirrors of an SLM or a PLM. The micromirrors can be adjusted according to the bit values in the corresponding SRAM cells, such as based on PWM signals from the first controller132. In other examples, the light modulator115is an LCD, an LCoS device, or a FLCoS device and the optical elements are liquid crystals that are controlled by the second controller134to modulate the incident light125across the image pixels.

The display device101also includes a processor140configured to process images and produce processed images for projection. The processed images can be projected by the light modulators115simultaneously according to a certain projector stacking configuration. For example, the images include image data that represent a sequence of frames of images, such as video frames at a certain display rate. The processed images can be digital images useful to provide control signals from the controllers130to the light modulators115and the light sources120. The second controllers134in the projectors105can receive from the processor140the image data in the form of a sequence of frames and produce display image data based on the received image data. The display image data are transmitted from each second controllers134(e.g.,134A and134B) to the respective light modulator115(e.g.,115A and115B) on a respective interface150(e.g.,150A and150B). The second controllers134is configured to provide control signals based on the display image data to the light modulators115, which then modulate the incident light125according to the control signals to display video or images114. The light modulators115projects the modulated light112on the image projection surface110to display the images114for viewing by a human eye160, also referred to herein as the human visual system (HVS) pupil.

The display device101also includes an image warping engine170configured to warp images based on corrected quadrilaterals, which can be determined from initially projected images114. The warped images are projected simultaneously by respective light modulators115to produce, on the image projection surface110, aligned images without distortion and accordingly a combined image with a corrected geometry. The image warping engine170can be coupled to the processor140that processes the warped images for projection. In other examples, the image warping engine170can be part of the processor140. For example, the image warping engine170and the processor140can be coupled to or integrated in one or more electronic chips.

The display system100also includes cameras180(e.g.,180A and180B) configured to capture the overlapping images114(e.g.,114A and114B), respectively. In some examples, the projectors105(e.g.,105A and105B) can be packaged separately, and each camera180(e.g.,180A and180B) can be packaged with or coupled to a respective projector105. The camera180can be positioned arbitrarily in the display system100independent of the positions of the projectors105or the display device101. A camera180can also be arbitrarily positioned with respect to another camera180. The cameras180(e.g.,180A and180B) have respective fields of views185(e.g.,185A and185B) that are aligned, by orienting the cameras180accordingly, with the respective images114. Because the images114overlap, the fields of views185of the cameras180are also aligned to overlap. Accordingly, a camera180can fully capture a respective image114and a portion of one or more other images114. For example, a pair of cameras180capture, within an overlap region188between the respective fields of views185, an overlap region189of a pair of images114on the image projection surface110. The cameras180are coupled to a processor190configured to generate, based on the captured images114, one or more pair-wise homography transforms and determine, based on the transforms, corrected quadrilaterals for the respective images114. The processor190can be coupled to the display device101to provide the corrected quadrilaterals to the image warping engine170. In other examples, the image warping engine170and the processor190can be combined in a single apparatus that is coupled to or located in the display device101. For example, one or more electronic chips including the image warping engine170and the processor190can be coupled to or integrated with the processor140. The display device101can further include one or more input/output devices (not shown), such as an audio input/output device, a key input device, a display, and the like.

FIG.2is a diagram of a display system200, in accordance with various examples. The display system200can be an example of the display system100or a display system configured similar to the display system100according to a 2×2 projector configuration for projecting simultaneously four overlapping images in a 2×2 array on the image projection surface110. The display system200includes the display device201for projecting images or video with projector stacking. The display device201in the display system200can be an example of the display device101in the display system100or configured similar to the projector the display system100. The display device201includes projectors205configured to project, simultaneously, respective images208in a 2×2 projector configuration. A projector205is configured to project a respective modulated light212to display a respective image208. The projectors205in the display system200can be an example of the projectors105in the display system100or configured similar to the projectors105. As shown inFIG.2, the display device201includes four projectors205(e.g.,205A to205D) that project four respective images208(e.g.,208A to208D) that partially overlap in a 2×2 array on an image projection surface110(not shown inFIG.2). The partially overlapping images208are aligned to display a larger combined image. InFIG.2, the images208are shown at a plane parallel to the image projection surface110.

The display system200also includes four cameras280(e.g.,280A to280D) configured to capture the images208A to208D, respectively. The cameras280in the display system200can be an example of the cameras180in the display system100or configured similar to the cameras180. The cameras280can be positioned arbitrarily in the display system200independent of the positions of the projectors205or the display device201. A camera280can also be arbitrarily positioned with respect to other cameras280. The cameras280A to280D have respective fields of views285A to285D that are aligned with the images208A to208D, respectively. Each camera280can fully capture a respective image208and a portion of one or more other images208.

For example, the camera280A captures, within the field of view285A, the image208A projected by the modulated light212A from the projector205A. The camera280A can also capture, within the field of view285A, a bottom portion of the image208D and left portions of the image208B and the image208C. The camera280B captures, within the field of view285B, the image208B projected by the modulated light212B from the projector205B. The camera280B can also capture, within the field of view285B, a bottom portion of the image208C and right portions of the image208A and the image208D. The camera280C captures, within the field of view285C, the image208C projected by the modulated light212C from the projector205C, and can also capture a top portion of the image208B and left portions of the image208A and the image208D. The camera280D can capture, within the field of view285D, the image208D projected by the modulated light212D from the projector205D, a top portion of the image208A, and right portions of the image208C and the image208D.

As shown inFIG.2, a first overlap region288A between the fields of views285A and285B of the cameras280A and280B, respectively, includes image points289A and289B along the right edge of the image208A and image points290A and290B along the left edge of the image208B. A second overlap region288B between the fields of views285A and285D of the cameras280A and280D, respectively, includes image points289B and289C along the top edge of the image208A and image points291A and291B along the bottom edge of the image208D. A third overlap region288C between the fields of views285B and285C of the cameras280B and280C, respectively, includes image points290B and290C along the top edge of the image208B and image points292A and292B along the bottom edge of the image208C. A fourth overlap region288D between the fields of views285C and285D of the cameras280C and280D, respectively, includes image points291B and291C along the right edge of the image208D and image points292B and292C the left edge of the image208C.

In some examples, the image points in the overlap regions captured by a pair of cameras, such as a pair of cameras280or180, are useful to determine a mapping relationship of image points between the frame references of the respective cameras. The frame of reference of a camera represents mathematically a two-dimensional plane (e.g., Cartesian plane) for positioning the image points of an image in the camera plane. For example, the image points289A,289B,290A, and290B in the overlap region288A can be processed by the processor290to establish a mapping relationship between the images208A and208B in the frames of reference of cameras280A and280B, respectively. The mapping relationship can be represented mathematically by a pair-wise homography transform, which is a transfer function that maps the positions of the image points of an image between respective frames of reference of respective cameras. Similarly, the image points289B,289C,291B, and291A in the overlap region288B can be processed to establish a pair-wise homography transform between the frames of reference of cameras280A and280D. The image points290B,290C,292B, and292A in the overlap region288C can be processed to establish a pair-wise homography transform between the frames of reference of cameras280B and280C. The image points291B,291C,292B, and292C in the overlap region288D can also be processed to establish a pair-wise homography transform between the frames of reference of cameras280D and280C.

FIG.3is a diagram of frames of reference300A and300B of a pair of respective cameras310A and310B, in accordance with various examples. The frames of reference300A and300B represent two-dimensional planes (e.g., Cartesian planes) where the coordinates of image points are set for respective images305A and305B captured by the cameras310and310B, respectively. The frame of references300A and300B are maintained by the cameras310A and310B, respectively, such as by a processor and a memory coupled to the cameras310A and310B. The cameras310A and310B can be a pair of cameras in the display system200or the display system100. For example, the cameras310A and310B can be the cameras180A and180B, the cameras280A and280B, or the cameras280D and280C, respectively. The pair of cameras310A and310B are configured to capture, respectively, the images305A and305B within the respective fields of view315A and315B. The fields of view315A and315B partially overlap in an overlap region320. The overlap region320includes image points (represented by dots in the images305A and305B) captured by both cameras310A and310B, also referred to herein as share image points. As shown inFIG.3, the image points of the images305A and305B can represent skewed images in the frames of reference300A and300B, respectively. The images305A and305B can be distorted (e.g., skewed) because of the orientation of the cameras310A and310B with respect to an image projection surface (not shown inFIG.3). The images305A and305B can also be distorted by the orientation of the respective projectors (not shown inFIG.3) that project the images305A and305B onto the image projection surface. In some examples, the images305A and305B are partially overlapping on the image projection surface plane (not shown), and the shared image points can be located in an overlap region between the images305A and305B on the image projection surface plane.

The shared image points captured by both cameras310A and310B can be processed to generate a pair-wise homography transform between the frames of reference300A and300B. In some examples, the pair-wise homography transform can be generated from four image points in the overlap region320that are captured by both cameras310A and310B. The four image points include a top right image point332and a bottom right image point334of the image305A and also include a top left image point336and a bottom left image point338of the image305B.

In examples, a display system, such as the display system100or200, can include multiple pairs of light modulators or projectors (e.g., light modulators115or projectors205) that project respective pairs of overlapping images according to a projector stacking configuration. Accordingly, a pair-wise homography transform can be generated for each pair of images, such as the pair of images305A and305B, based on four shared image points in the respective overlap region, such as the image points332,334,336, and338in the overlap region320. Each pair-wise homography transform is also generated independently from image points outside the overlap region.

For example, the display system200projects four images208in a 2×2 projector configuration which partially overlap on an image projection surface. Each image208is then captured by a respective camera280within a respective field of view285. The four cameras280also capture, in the respective overlap regions288A to288D, at least four shared image points between the images208. Based on the shared image points in the respective overlap regions288A to288D, four pair-wise homography transforms can be generated for four pairs of images208. In each of the overlap regions288, four image points are sufficient to calculate a respective pair-wise homography transform for each pair of cameras280. Since no more than four image points are needed to determine each pair-wise homography transform for geometric correction, the field of view of each camera280can be limited to capture the respective image with the edges of other images, which simplifies the setup and calibration of the display system.

FIG.4is a diagram of pair-wise homography transforms400for a 2×2 array of images405(405A to405B), in accordance with various examples. The four images405A to405B (labeled 1 to 4) partially overlap on a plane of an image projection surface, as shown inFIG.4. A pair-wise homography transform400can be generated for each pair of images405based on four shared image points which can be obtained from a group of shared image points in the overlap region between respective fields of views of the cameras. For example, a first pair-wise homography transform (H 1-2) can be generated based on four shared image points from a group of share image points between a first pair of images405A and405B in an overlap region captured by a first camera and a second camera. Similarly, a second pair-wise homography transform (H 2-3) can be generated based on shared respective image points between a second pair of images405B and405C captured by the second camera and a third camera. The pair-wise homography transform400also include a third pair-wise homography transform (H 4-3) for a third pair of images405D and405C captured by a fourth camera and the third camera, and a fourth pair-wise homography transform (H 1-4) for a fourth pair of images405A and405D captured by the first camera and the fourth camera.

The pair-wise homography transforms400are mathematical transformations between two camera frames of reference, which are two-dimensional planes. Such transformations can be represented by 3×3 transformation matrices in a homogenous coordinate space, such as Cartesian space. For example, a pair-wise homography matrix is generated by solving the following equation (1):

[xaya1]=H*[xbyb1]=[h⁢1⁢1h⁢1⁢2h⁢1⁢3h⁢2⁢1h⁢2⁢2h⁢2⁢3h⁢3⁢1h⁢3⁢21]*[xbyb1],(1)
where xa, yaare coordinates of an image point in a first frame of reference, and xb, ybare coordinates of an image point in a second frame of reference. The parameters of the pair-wise homography transform, H, can be calculated by solving equation (1) with coordinates of four shared image points in the overlap region between respective fields of views of the cameras. The parameters of the pair-wise homography transform are also calculated without and independently from image points outside the overlap region.

The four pair-wise homography transforms H 1-2, H 2-3, H 4-3, and H 1-4, are useful to map the images405A to405D to a common camera frame of reference. The common camera frame of reference refers herein to the frame of reference set in one of the cameras that capture the respective images. For example, the images405B to405D can be mapped to a first frame of reference (labeled 1 inFIG.4) of the first camera as a common frame of reference. The image405A is captured by the first camera in the first frame of reference without applying a homography transform. The image405B is mapped from a second frame of reference (labeled 2) of the second camera to the first frame reference of the first camera by applying H 1-2 to the image points of the image405B. The image405D is mapped from a fourth frame of reference (labeled 4) of the fourth camera to the first frame reference by applying H 1-4 to the image points of the image405D. The image405C is mapped from a third frame of reference (labeled 3) of the third camera to the first frame reference by applying a fifth pair-wise homography transform between the first frame of reference and the fourth frame of reference. The fifth pair-wise homography transform (H 1-3) can be calculated based on H 1-4 and H 4-3, such as according to the following mathematical equation (2):
H1-3=H1-4×H4-3.  (2)

The fifth pair-wise homography transform (H 1-3) can also be calculated based on H 1-2 and H 2-3 according to H 1-3=H 1-2×H 2-3. Mapping the images405A to405D to a common camera frame of reference allows determining corrected quadrilaterals for the images405A to405D, respectively, which are useful for warping to-be-projected images and accordingly correcting the geometry in the displayed images or combined image.

The pair-wise homography transforms, each calculated based on four image points according to equation (1), reduce the computation complexity and cost for geometric correction in comparison to other geometric correction methods for projector stacking. For example, a single image can be captured for each projector to perform the geometric correction based on the pair-wise homography transforms. The images can then be mapped based on the pair-wise homography transforms to a common camera frame of reference to calculate the corrected quadrilaterals without calibrating the cameras. In comparison, other geometric correction methods may require multiple images with a larger number of image points to calibrate the cameras, which increases the computation complexity and cost, such as in terms of storage space and processing time.

FIGS.5A to5Fare diagrams of a process of determining corrected quadrilaterals505for a 2×2 array of images501, in accordance with various examples.FIG.5Ashows the images501(501A to501D) mapped, based on pair-wise homography transforms, to a common camera frame of reference510.FIG.5Bshows the respective corrected quadrilaterals505(505A to505D) in the common camera frame of reference510. The corrected quadrilaterals505inFIG.5Bcan be determined from the images501inFIG.5Aby a series of geometric calculations shown inFIGS.5C to5F.

The geometric calculations include first determining the intersections of image sides for each pair of images501, as shown inFIG.5C. The determined intersection points for each pair of images501form a respective interior frame for the pair. As shown inFIG.5C, for the pair of images501A and501B, the intersection of the right side of the image501A with the image501B is determined at the bottom right corner516of the image501A and at the intersection517of the right side of the image501A and the top side of the image501B. The intersection of the left side of the image501B with the image501A is determined at the intersection518of the left side of the image501B and the top side of the image501A and at the bottom left corner519of the image501B. Connecting the points at the bottom right corner516, the intersection517, the intersection518, and the bottom left corner519determines a first interior frame520for the pair of images501A and501B.

In some examples, the four points at the respective corners516to519of the first interior frame520are obtained based on a center of gravity (COG) calculation. The COG calculation is useful for any orientations of projectors of the images with respect to the image projection surface. The COG point represents the balance point or the average position of all the points in the images501A to501D. The steps of COG calculation include extrapolating the sides of the images501A to501D to obtain eight intersection points, and calculating a COG point for the eight intersection points. For each of the eight intersection points, a respective Euclidean distance can be calculated with the COG point. From the eight intersection points, four intersection points with the shortest respective Euclidian distances can then be selected as the four corners516to519that determine the first interior frame520. Similarly, inFIG.5C, a second interior frame521is determined for the pair of images501C and501D, a third interior frame522is determined for the pair of images501B and501C, and a fourth interior frame523is determined for the pair of images501A and501B.

FIG.5Dshows a next step of the geometric calculations where an overall exterior frame525for the images501can be determined based on the interior frames520to523. The exterior sides of the interior frames520to523are extrapolated to intersect with the exterior sides of the images501(shown by the arrows inFIG.5D). The intersection points form the overall exterior frame525. In other examples, to determine the overall exterior box, four points can be selected based on a COG calculation from a larger group of intersection points. The steps of the COG calculation include extrapolating the exterior sides of the determined interior frames to obtain the group of intersection points. A COG point is then calculated for the group of intersection points, and accordingly, a respective Euclidean distance is obtained for each of the intersection points. Four intersection points with the shortest respective Euclidian distances can then be selected from the group of intersection points to determine an overall exterior frame.

FIG.5Eshows a next step of the geometric calculations where four intersection points of the interior frames520to523can then be determined (shown by the ‘X’ markers inFIG.5E). The lines connecting the intersection points are then extrapolated to intersect with the sides of the overall exterior frame525(shown by the arrows inFIG.5E).

FIG.5Fshows a next step of the geometric calculations where the extrapolated lines connect the intersection points of the interior frames520to523to the sides of the overall exterior frame525to create eight corrected frames530to538. For example, the corrected frame530inFIG.5Frepresents the overlap between the interior frames520,521,522, and523inFIG.5E. The corrected frame531inFIG.5Fis determined by extrapolating the top and bottom sides of the corrected frame530to intersect with the left side of the overall exterior frame525. Similarly, the corrected frames532to538can be determined based on the corrected frame530and the sides of the exterior frame525.

The corrected quadrilaterals505A to505D can then be determined based on the corrected frames530to538within the overall exterior frame525. For example, the corrected frames530,531,532, and533are combined to determine the corrected quadrilateral505A. Similarly, the corrected frames530,533,534, and535determine the corrected quadrilateral505B, the corrected frames530,535,536, and537determine the corrected quadrilateral505C, and the corrected frames530,531,537, and538determine the corrected quadrilateral505D.

FIG.6is a diagram of a geometric correction for a 2×1 projector configuration, in accordance with various examples. Before geometric correction, two overlapping images605A and605B are projected, according to the 2×1 projector configuration, on an image projection surface610to display a larger combined image615. For example, the projectors105A and105B in the display system100can be configured according to the 2×1 projector configuration to project the images605A and605B, respectively, in a 2×1 array on the image projection surface110. The projection of the images605A and605B causes distortion in the displayed images605A and605B and accordingly in the combined image615. The distortion in the images605A and605B changes the geometry of the images605A and605B on the image projection surface610in comparison to the initial geometry before projection, at the respective projector planes. For example, the images605A and605B are rectangle shaped images in the frames of reference of the respective light modulators, as processed by the display device101. Because of the respective projection angles of the images605A and605B with respect to the image projections surface610, the geometries of the images605A and605B are distorted on the plane of the image projection surface610. The distorted images605A and605B can have quadrilateral non-rectangular geometries, as shown inFIG.6. Because the projection angles of the images605A and605B can be different, the distorted geometries of the images605A and605B can also be different. Accordingly, the combined image615formed by the overlapping images605A and605B is also non-rectangular.

To perform geometric correction, the pair-wise homography transform is determined for the images605A and605B between two respective camera frames of reference. The images605A and605B are then mapped based on the pair-wise homography transform to a common camera frame of reference, such as the frame of reference of one of the two cameras, and processed to determine the respective corrected quadrilaterals. The images605A and605B can then be warped, by an image warping engine, based on the respective quadrilaterals to obtain the respective warped images620A and620B, which are then projected again on the image projection surface610. As shown inFIG.6, after the geometric correction, projecting the warped images620A and620B based on the respective quadrilaterals compensates for the distortion and corrects the geometries of the images605A and605B on the image projection surface610. Accordingly, a corrected and rectangle shaped combined image625is displayed without distortion.

FIG.7is a diagram of a geometric correction for a 2×2 projector configuration, in accordance with various examples. Before geometric correction, four overlapping images705A to705D are projected, according to the 2×2 projector configuration, on an image projection surface710to display a larger combined image715. For example, the projectors205A to205D in the display system200can be configured according to the 2×2 projector configuration to project the images705A to705D, respectively, in a 2×2 array on the image projection surface710. The projection of the images705A to705D causes distortion in the displayed images705A to705D and accordingly in the combined image715. The distortion in the images705A to705D changes the geometry of the images705A to705D on the image projection surface710in comparison to the initial geometry before projection, at the respective projector planes. For example, the images705A to705D are rectangle shaped images in the frames of reference of the respective light modulators, as processed by the display device201. The geometries are distorted by the projection process on the plane of the image projection surface710, which distorts the combined image715.

To perform geometric correction, pair-wise homography transforms are determined for respective pairs of the images705A to705D, such as according to equations (1) and (2). The images705A to705D are then mapped based on the pair-wise homography transforms to a common camera frame of reference of one of the four cameras, and processed to determine the respective corrected quadrilaterals, such as according to the process500. The images705A to705D can then be warped, by an image warping engine, based on the respective quadrilaterals to obtain the respective warped images720A and720B, which are then projected again on the image projection surface610. As shown inFIG.6, after the geometric correction, projecting the warped images720A to720D based on the respective quadrilaterals compensates for the distortion and corrects the geometries of the images705A to705D on the image projection surface710. Accordingly, a corrected and rectangle shaped combined image725is displayed without distortion.

FIG.8is a flow diagram of a method800of a geometric correction for a projector configuration, in accordance with various examples. For example, the method800can be performed by the display system100or the display system200for a 2×1 or 2×2 projector configuration. The method800is performed to determine the corrected quadrilaterals based on the pair-wise homography transforms for geometry correction in projection stacking.

At step810of the method800, a processor generates for a first camera and a second camera, a pair-wise homography transform for a pair of partially overlapping images from the first camera and the second camera. The pair-wise homography transform is based on shared image points in an overlap region of the images. For example, the processor190of the display system100or the processor290of the display system200generates a pair-wise homography transform between the respective frames of reference of two cameras180or280. A pair-wise homography transform can be generated based on four image points in the overlap region between the respective fields of views of the cameras, such as according to equation (1). In some examples, a pair-wise homography transform is generated for each pair of images, according to equations (1) and (2), in an m×n (e.g., 2×1, 1×2, 2×2, etc.) projector configuration, where m and n are integers.

At step820, the processor maps, based on the pair-wise homography transform, the pair of partially overlapping images to a common frame of reference, such as the frame of reference of the first camera. For example, for a 2×2 array of partially overlapping images, the processor generates the pair-wise homography transforms400and maps the image points of three of the images from respective camera frames of reference to the first frame of reference of the first camera.

At step830, the processor determines a pair of corrected quadrilaterals for the pair of partially overlapping images in the common frame of reference of the first camera. For example, for a 2×2 array of partially overlapping images, the four images in the common camera frame of reference based on pair-wise homography transforms, can be processed according to the process500to determine the respective corrected quadrilaterals.

At step840, the processor geometrically projects, based on the pair-wise homography transform, the pair of corrected quadrilaterals from the common frame of reference to a pair of camera frames of reference for the first camera and the second camera, respectively. After determining each corrected quadrilateral in the common camera frame of reference, the corrected quadrilateral is geometrically projected back to a frame of reference of a respective camera. Each corrected quadrilateral of an image can be geometrically projected back to a respective camera frame of reference by processing the image points of the corrected quadrilateral with an inverse transform. The inverse transform is the inverse of the pair-wise homography transform which maps the image to the common camera frame of reference. For example, the second corrected quadrilateral505B can be geometrically projected back to a respective camera frame of reference based on the inverse of the pair-wise homography transform which maps the image501B to the common camera frame of reference510. Similarly, the corrected quadrilaterals505C and505D are geometrically projected back to respective camera frames of reference based on inverse transforms to the pair-wise homography transforms which map the images501C and501D, respectively, to the common camera frame of reference510.

At step850, the processor geometrically projects the pair of corrected quadrilaterals from the pair of camera frames of reference to a pair of projector frames of reference based on a pair of respective image-to-camera homography transforms. A projector frame of reference represents mathematically a two-dimensional plane (e.g., Cartesian plane) for positioning the image points of an image at the respective projector or light modulator plane. The image-to-camera homography transform represents the mapping of the image points between frames of reference of respective light modulators (or projectors) and cameras. For example, an image-to-camera homography transform between a light modulator (or projector) and a camera can be obtained according to equation (1) based on respective image points between the light modulator (or projector) and the camera planes.

FIG.9is a flow diagram of a method900of a geometric correction for a projector configuration, in accordance with various examples. The method900is performed to warp and project images with corrected geometries based on respective corrected quadrilaterals. The corrected quadrilaterals can be determined according to the method800which can be performed prior to the method900. For example, the method900can be performed by the display system100or the display system200after performing the method800to obtain the corrected quadrilaterals. According to the method800, the corrected quadrilaterals can be determined based on the pair-wise homography transforms. The method900is then performed based on the corrected quadrilateral for projection stacking, such as according to a 2×1 or 2×2 projector configuration.

At step910of the method900, an image warping engine warps a pair of new images respectively based on a pair of corrected quadrilaterals in a pair of image frames of reference. The new images are images processed for projection to display a to-be-projected combined image with geometric correction based on the corrected quadrilaterals. At step920, a display device projects the warped pair of images on an image projection surface. For example, the image warping engine170of the display device101warps to-be-projected images in a 2×1 or 2×1 projector configuration based on respective corrected quadrilaterals determined by the obtained from the processor190after performing the method800. The to-be-projected images can be digital images processed by the processor140and warped by the image warping engine170. The processor140sends the warped images to the controllers130of the projectors105, which control accordingly the respective light modulators115to modulate and project the modulated light112on the image projection surface110. The images are displayed accordingly with correct geometry and without distortion because of projection. Warping the to-be-projected images based on the respective corrected quadrilaterals mitigates distortion and corrects geometric alignment in the displayed to-be-projected combined image.

FIG.10is a flow diagram of a method1000of a geometric correction for a projector configuration, in accordance with various examples. For example, the method1000can be performed by the display system100or the display system200for a 2×1 or 2×2 projector configuration. The method1000is performed to determine the corrected quadrilaterals for respective images in a common camera frame of reference. For example, the corrected quadrilaterals can be determined in the step830of the method800, after mapping the images to a common camera frame of reference.

At step1010of the method1000, an interior frame of intersection is determined between a pair of partially overlapping images. In projector configurations with more than one pair of partially overlapping images, such as a 2×2 projector configuration, the interior frame is determined for each pair of partially overlapping images. For example, the processor190of the display system100or the processor290of the display system200determines the intersections of sides of each pair of images501of the process500in a common camera frame of reference. The determined intersection points between the sides are connected to obtain the interior frames520to523for the images501A to501D.

At step1020, an overall exterior frame is determined for the pair of partially overlapping images based on extrapolating sides of the interior frame to sides of the partially overlapping images. For example, the processor190or the processor290determines, according to the process500, the overall exterior frame525based on the interior frames520to523by extrapolating the exterior sides of the interior frames520to523to intersect with the exterior sides of the images501. The four points located by the intersection of the different sides are connected to obtain the overall exterior frame525.

At step1030, corrected frames are determined for the pair of partially overlapping images based on extrapolating sides between the interior frame and the overall exterior frame. For example, the processor190or the processor290determines, in the process500, the corrected frames530to538by locating the four common intersection points between the interior frames520to523. The four points are then connected by four lines. The lines are extrapolated to intersect with the sides of the overall exterior frame525and define the corrected frames530to538.

At step1040, the pair of corrected quadrilaterals are determined based on the corrected frames. For example, the processor190or the processor290determines the corrected quadrilaterals505A to505D based on the corrected frames530to538and the overall exterior frame525in the process500. Each four adjacent corrected frames at each corner of the overall exterior frame525are combined to form four overlapping quadrants which represents four corrected quadrilaterals for the four partially overlapping images in the common camera frame of reference.

In this description, the term “couple” may cover connections, communications, or signal paths that enable a functional relationship consistent with this description. For example, if device A generates a signal to control device B to perform an action: (a) in a first example, device A is coupled to device B by direct connection; or (b) in a second example, device A is coupled to device B through intervening component C if intervening component C does not alter the functional relationship between device A and device B, such that device B is controlled by device A via the control signal generated by device A.

A device that is “configured to” perform a task or function may be configured (e.g., programmed and/or hardwired) at a time of manufacturing by a manufacturer to perform the function and/or may be configurable (or reconfigurable) by a user after manufacturing to perform the function and/or other additional or alternative functions. The configuring may be through firmware and/or software programming of the device, through a construction and/or layout of hardware components and interconnections of the device, or a combination thereof.

A device that is described herein as including certain components may instead be coupled to those components to form the described device. For example, a structure described as including one or more elements (such as one or more processors and/or controllers) may instead include one or more elements within a single physical device (e.g., a display device) and may be coupled to at least some of the elements to form the described structure either at a time of manufacture or after a time of manufacture, for example, by an end-user and/or a third-party.

While certain components may be described herein as being of a particular process technology, these components may be exchanged for components of other process technologies. Device components described herein are reconfigurable to include the replaced components to provide functionality at least partially similar to functionality available prior to the component replacement.

In this description, unless otherwise stated, “about,” “approximately” or “substantially” preceding a parameter means being within +/−10 percent of that parameter. Modifications are possible in the described examples, and other examples are possible within the scope of the claims.