Abstract:
Multiple electronic displays are arranged to define an ensemble of the user&#39;s choosing. The processor generates and displays a uniquely encoded emblem encoding a unique identifier for that display. A camera captures at least one image of the ensemble so that the encoded emblem for each display is captured. The captured images are then processed to extract the following attributes: a) unique identifier of the display; b) position of the encoded emblem relative to the reference coordinate system; and c) pointing direction of the encoded emblem relative to the reference coordinate system. The processor collectively processes the attributes for each of the emblem images to compute for each display a transformation matrix that selects a portion of the image data for display on that monitor and performs pointing direction correction so that the information presented on that display appears spatially consistent with the information presented on the other displays.

Description:
FIELD 
     The present disclosure relates generally to computer graphics and video display technology. More particularly, the disclosures relates to the display of three-dimensional images using multiple display screens simultaneously. 
     BACKGROUND 
     This section provides background information related to the present disclosure which is not necessarily prior art. 
     Current three-dimensional display technology allows the user to have a three-dimensional viewing experience by separately displaying image data for the left eye and right eye, respectively. Conventionally, the user wears special glasses that are designed to selectively filter the left and right information so that each eye sees only the visual image for that eye&#39;s vantage point. There are a variety of different filtering techniques. Some use different colored lenses for each eye; some use different polarizations for each eye; and some multiplex the image in synchronism with image-blanking lenses in the glasses so that each eye receives only the visual information intended for it. 
     In most situations, this 3D experience is provided using a single display screen. In such case, the signal processing needed to supply the left and right images is fairly straightforward. The visual image data for each eye are separately processed by the appropriate filter or multiplex process and the processed data are then fed to the display device. 
     There are applications, however, where it would be desirable to offer a viewing experience using multiple displays or monitors. Examples of such applications might include panoramic theaters, trade show display kiosks, lecture halls and the like. 
     Physically setting up and interconnecting multiple displays to work in concert has traditionally required a painstaking setup process that is very technical, needs special tools and is very time-consuming. This setup process is further complicated when 3D viewing is desired because the precise position of each display must be accurately taken into account in order to properly render the left and right eye images. 
     SUMMARY 
     This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features. 
     The present disclosure eliminates the tedious setup process by providing a system and computer-implemented apparatus that allows the location of each monitor within the ensemble to be automatically discovered, with its pointing direction or attitude and physical location relative to other monitors being automatically calculated. In one embodiment, the monitors are equipped with cameras and are able to automatically discover themselves. 
     In accordance with one aspect, the method of presenting information across multiple electronic displays involves first arranging multiple electronic displays to define an ensemble. The arrangement can be at the user&#39;s discretion, as the apparatus and method learns the arrangement, generating and displaying on each display a uniquely encoded emblem that expresses or encodes a unique identifier for that display. At least one digital image of the ensemble is captured so that the encoded emblem for each display is captured. A reference coordinate system is defined corresponding to the space occupied by the ensemble of electronic displays. 
     Next, each of the captured encoded emblem images is processed by the computer system to extract the following attributes:
         a) the unique identifier of the display;   b) the position of the encoded emblem relative to the reference coordinate system; and   c) the pointing direction of the encoded emblem relative to the reference coordinate system.       

     The information to be presented as image data to be displayed across the ensemble of displays is then supplied or input to the processor. The processor collectively processes the attributes for each of the encoded emblem images to compute for each display a transformation matrix that selects a portion of the image data for display on that monitor and that performs pointing direction correction so that the information presented on that display appears spatially consistent with the information presented on the other displays. 
     In this way, the computer system automatically learns or discovers the positions and orientations of each of the displays, and then uses this information to generate transformation matrices that are uniquely configured for each display, given its discovered location, pointing direction and other information about the display (such as the display size and resolution). 
     Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. 
    
    
     
       DRAWINGS 
       The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure. 
         FIGS. 1   a - 1   d  depict four exemplary use cases where the disclosed method can be used; 
         FIG. 2  is a block diagram overview of the method; 
         FIG. 3  is an overview flowchart and system block diagram of the system and method; 
         FIG. 4  is a more detailed flowchart showing the method; 
         FIG. 5  is a further detailed flowchart of the method; 
         FIG. 6  is a data flow diagram illustrating assignment of the detected codes and computed position parameters; 
         FIG. 7  is a flowchart illustrating one embodiment for calculating the monitor position and orientation; 
         FIG. 8  is a flowchart depicting how the monitor position and orientation data are used in combination with a head-tracking algorithm; and 
         FIG. 9  is a flowchart depicting the 3D rendering matrix and its computation. 
     
    
    
     Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings. 
     DETAILED DESCRIPTION 
     Example embodiments will now be described more fully with reference to the accompanying drawings. 
     Referring to  FIGS. 1   a - 1   d , four different use cases of the disclosed method are illustrated. In  FIG. 1   a , three display devices  10  are arranged to generally define a panoramic ensemble onto which computer graphic images are displayed. In the illustrated embodiment of  FIG. 1   a , note that the depiction of a globe at  12  occupies only a single display device, whereas the depiction of the dragon  14  stretches across all three display devices. The methods and system described herein are used to determine the proper transformation matrices required to render these visual images properly, given the position and orientations of the respective displays or monitors. As will be seen, the disclosed technique does not require the displays to be carefully positioned in any known locations. On the contrary, the displays can be simply placed in convenient locations. The disclosed system and method is then used to ascertain the display positions and orientations automatically and then compute the needed transformation matrices. 
       FIG. 1   b  shows another ensemble of displays  10 . In this depiction, two of the displays are positioned at approximately right angles to one another and are then both positioned above a third display  10   a  that is positioned on the flat, that is, in a plane orthogonal to the other two monitors. In the embodiment illustrated in  FIG. 1   b , the image being displayed is a computer graphic image representing a section of DNA molecule, made to appear as if it is floating in the space in front of and above the displays. 
       FIG. 1   c  shows yet another possible display ensemble configuration, where the displays are arranged in a series of panels, where the left-most panels  101  form an angle  18  with the right-most panels  10   r . The displayed car image  20  is rendered properly, without distortion, notwithstanding the fact that the respective left and right panels are positioned at an angle to one another. This is because the system and method has been used to compute the transformation matrices needed to render the car  20  without distortion by compensation for the position and orientation of the respective displays. 
     While the first three example use cases have featured uses that might be appropriate in large public spaces, the techniques described here are not limited to such applications. Thus,  FIG. 1   d  shows another embodiment where two displays  10  are positioned at right angles to one another and serve as the display viewed by user  22  in a teleconferencing system. The image can be generated, as illustrated, to make it appear as if certain components are floating in front of other components. In the depicted illustration in  FIG. 1   d , a tablet  24  floats in front of the image  26  of the other communicating party, to simulate, for example, the parties&#39; shared calendar. 
     Referring now to  FIG. 2 , an overview of the method will be described. As depicted at  30 , the ensemble of displays  10  is first set up by simply placing a plurality of monitors in any desired locations. Next, the computer-implemented system  32  (the details of which are described below) ascertains the resolution (e.g., pixels per centimeter) of each display and then generates for each display a QR code emblem  34  of known fixed size on each of the displays, as depicted at  36 . Each QR code emblem encodes a unique display ID assigned to that monitor. 
     Next, a camera  38  captures one or more images of the display ensemble so that each of the QR code emblems is photographically captured in at least one image. If desired, the camera  38  can be activated automatically by system  32 . The images obtained by the camera  38  are then sent to system  32  for further processing, as will be described. The camera  38  can be a still camera, taking one or more individual image frames, or it can be a video camera, capturing multiple frames in a time-synchronized sequence. Although the camera  38  has been illustrated in  FIG. 2  as a handheld camera, the techniques described here can also be performed using cameras that are in fixed locations, including cameras that are incorporated into the displays themselves. 
     Next, as illustrated at  40 , data for generating a displayed image is supplied from a data source  42  and then processed by the system  32  to allocate which portions of the image are to be shown on each of the displays, and further manipulating the data according to calculated transformation matrices so that the displayed image (e.g., DNA molecule  16 ) is rendered properly. 
     Referring to  FIG. 3 , the computer-implemented system  32  is shown in more detail. The system includes at least one processor  50  with associated memory  52 . The processor computes the transformation matrices needed to properly display the image on plural displays  10  using a rendering algorithm  54 . The processor communicates with the camera  38  (or with multiple cameras). 
     The basic process according to the disclosed method will now be discussed in connection with  FIG. 4 . The method begins by placement of the displays as at step  60 . The displays can be positioned anywhere the user wishes. Next (step  62 ), the processor ascertains the resolution of each monitor and then generates QR code emblems of a predetermined fixed size on each of the monitors. Each QR code corresponds to a unique ID assigned by the system  32  to each monitor. The resolution of each monitor is obtained by consulting lookup tables or by direct interrogation of the monitor hardware, and this information is used to calculate how many pixels are required by the monitor to produce a displayed image of the QR code emblem of a predetermined size. For example, the QR code emblem may be generated to have dimensions 10 centimeters by 10 centimeters on each of the displays. Displays having higher pixel-per-centimeter density will require a greater number of pixels in order to achieve the fixed size emblem. 
     Next (step  64 ), the camera or cameras are used to capture one or more images of the collective ensemble of displays, making sure that the QR code emblem for each display is fully visible in at least one of the images. Although the camera will likely be able to “see” the actual display devices themselves, the preferred embodiment bases its calculations on the QR code emblems, rather than the physical monitors, because the size and shape of these emblems are more reliably controlled by the system. 
     Next (step  66 ), the system determines the position and orientation of each display by analyzing the position and orientation of the QR code emblems as captured in the images at step  64 . One embodiment for accomplishing this will be described below in connection with  FIG. 7 . In addition, at step  66  the user position is also determined, if needed, to generate 3D content. If 3D content is not being rendered (if the desired ensemble displays only a 2D image), then user position may not be necessary. 
     Finally (step  68 ), the content from the image data source  42  ( FIG. 2 ) is now rendered and shown on the multiple displays within the ensemble. 
       FIG. 5  shows the method in somewhat greater detail.  FIG. 5  has been labeled, where applicable, to show the steps depicted in  FIG. 4 .  FIG. 5  expands upon some of these steps, giving greater detail. Thus, after the display placement step  60 , the system generates a distinctive QR code (step  61 ) for each display and these codes are then used (step  62 ) to display a different QR code on each display. 
     Next, in one embodiment, the system prompts the user to take a photo (step  63   a ) or, alternatively, automatically controls the camera to take the photo or photos (step  63   b ). The images are then captured in an image buffer as at  63   c.    
     The determination of display position and orientation (step  66 ) involves first finding the QR code emblem in each photo (step  67   a ) and then recognizing the QR code to identify the display (step  67   b ) and then computing or recognizing the 3D position and 3D orientation of the QR code emblem (step  67   c ). From these data, each display&#39;s size, 3D position and 3D orientation are computed at  67   d.    
     Then at step  67   e , the position of the user is determined. This can be done, for example, by using the cameras mounted in the respective displays and then implementing a head-tracking algorithm (discussed in connection with  FIG. 8 ). 
     Once the data have been gathered in the aforementioned steps, the system computes the rendering matrix or transformation matrix for each display, taking into account the size, position and orientation of each display as determined by the preceding steps. This is depicted at step  68   a . Then, in step  68   b , the objects are rendered in 3D by delivering the image data (processed by the transformation matrices) to the displays. 
     If desired, the process can be iteratively performed multiple times (as depicted at step  69 ) until the user is satisfied with the view. 
     To compute each display&#39;s size, 3D position and 3D orientation, the system draws a distinct two-dimensional QR code on each display. The system maintains a mapping of all pairs of these QR codes, along with the associated display ID assigned to each display. This has been illustrated diagrammatically in  FIG. 6  at  70 . This mapping of pairs may be stored, for example, in the memory  52  ( FIG. 3 ) of the system. Later, when the images of the display ensemble are processed, each QR code located in the image (or images) is graphically identified and then used in a lookup process  72 , which interrogates the data store of QR code-display ID mapping pairs  70  to identify the display ID of each QR code, as indicated at  74 . 
     Next, the position, orientation and size attributes (Pn, On, Sn) are computed as explained in connection with  FIG. 7 , and these computed data are then assigned to the electronic display device in association with its display ID. The assignment step is depicted in  FIG. 6  at  76 . 
     The preceding steps are performed for each of the detected QR codes, so that the display ID, position, orientation and size attributes for each display are obtained. These data are stored as at  78  as a 4×4 matrix that is used to construct the transformation matrices, which is in turn used to manipulate the source image data. The transformation matrix essentially maps the overall scene to be displayed onto the respective displays, assigning selective regions of the initial image to the display that lies in the region to which it is mapped. Thus, for example, using the image illustrated in  FIG. 1   a  as an example, the globe  12  lies in the left-most region of the image and is thus mapped to the left-most display. 
     The transformation matrix takes into account not only position of the display (so as to place regions of the source image in the proper location) but also the resolution, size and orientation of each display. Size is taken into account by mapping onto the overall scene, the portion of that scene covered by each display. Orientation is taken into account to perform shearing and/or rotation transformation operations on the image date to, in effect, compensate for differences in angular orientation or attitude of one display relative to the others. Thus, if one display is angled relative to the others (as the right-most and left-most displays in  FIG. 1   a  are angled relative to the center display), the angular distortions that would otherwise occur are removed by applying the appropriate shear and/or rotation processes on the data being routed to those angularly oriented displays. 
     Referring now to  FIG. 7 , one embodiment for ascertaining the display ID, position, orientation and size attributes will be described. Referring to  FIG. 7  at step  100 , the system generates and displays a QR code emblem of known fixed size with unique ID for each display, as diagrammatically shown at  102 . Next, using a still camera or video camera, images of the ensemble are captured so that all of the QR code emblems are captured in at least one of the images (step  104 ). Then, for each QR code emblem in the camera image or images, the following steps are performed beginning at  106 . The order of the steps performed is not essential. Thus, the depiction shown in  FIG. 7  is merely intended as one example. 
     At step  108 , the QR code is graphically identified by suitable means, such as by employing a color filter, a contrast filter or other image processing techniques. The code is then read or interrogated and the corresponding display ID is thus ascertained and stored at  110 . As discussed in connection with  FIG. 6 , the display ID can be ascertained by performing a lookup which interrogates the stored mapping of pairs  70  ( FIG. 6 ). 
     As illustrated at  112 , the center of the QR code emblem is located and the location is assigned its (x,y) position in the image space and stored at  114 . While there are several ways of determining the center of the QR code emblem, the embodiment illustrated in  FIG. 7  places a bounding box  116  around the emblem and then subdivides the bounding box&#39;s horizontal and vertical sides to find the center where the subdivisions cross. 
     To find the orientation of the QR code emblem, the four corners of the QR code emblem are graphically identified and their respective angles are measured and used to compute the pointing vector that is orthogonal to the plane of the emblem (step  118 ). For example, if the four corners are all 90-degree angles, it is known that the emblem is facing directly towards the camera and thus the pointing vector would appear as a single point corresponding to the arrowhead of the vector. However, if the emblem is oriented at an angle, as illustrated in  FIG. 7 , the yaw and pitch of the emblem plane will correspond to a vector (r) pointing in a direction other than at the viewer&#39;s vantage point. Thus the vector is computed in relation to the reference frame and stored as at  120 . 
     While the (x,y) position of the QR emblem has been assigned based on the center of the emblem (as at  114 ), the z axis dimension is separately computed either by comparing the known size of the QR code emblem with the captured image (step  122 ), by using multiple images and performing triangulation (step  124 ), or by using both techniques together. Ascertaining z dimension information from the size of the QR code emblem (step  122 ) takes into account the fact that the QR code image appears larger when it is closer to the camera. Thus, the relative size of the image can be used to determine the z axis information. The z axis information is thus captured and stored at  126  and then combined with the (x,y) position data to determine the (x,y,z) position of the monitor which is then stored at  128 . 
     The size of the display can be determined by several different techniques. In one embodiment, the display size is known by the system, by utilizing lookup table data or by interrogation of the display attributes using the system calls of the computer operating system to which the display is attached. Alternatively, the display size can be ascertained by obtaining display size information in advance and then communicating it to the system  32  either by user input or by incorporating the information into the QR codes themselves. Finally, display size can also be determined graphically by displaying a graphical image that highlights the boundaries of the display and then measuring those boundaries. Other techniques are also possible. 
     Recognition of User Position 
     User position can be determined in 3D applications by employing at least one head-tracking camera designed to follow a known indicia on the 3D glasses of the user. In one embodiment, a camera is positioned in the display ensemble, such as a camera disposed in one or more of the display devices. The camera is trained in the direction of the user and can be configured to respond to a selective wavelength or group of wavelengths of light reflected from the 3D glasses of the user. For example, the monitor may be provided with an infrared illumination source that illuminates the user&#39;s eyeglasses, reflecting from a mirrored spot or other structured configuration on the eyeglass frames. The camera then detects the reflected infrared light, identifies the structured spot or other predefined structured configuration and uses this to ascertain the location of the glasses. The position of the user&#39;s left and right eyes can then be calculated based on the detected structured reflected light. 
     In one preferred embodiment, plural such head-tracking systems are employed. For example, several (or all) of the monitors in the ensemble can be provided with head-tracking cameras and the data from each of these are processed in parallel, as shown in  FIG. 8 , by employing a head-tracking algorithm  200  on each reflected light data stream. Each head-tracking algorithm  200  generates a 4×4 matrix based on the camera data [M_c_i]. Then, a 4×4 matrix calculated by the process illustrated in  FIG. 6  [M_d_i] is fed to the coordinate transformation algorithm  210  to generate a transformed 4×4 matrix [M_u_i]. These transformed data are then fed to process  220  where the data from multiple data streams are combined. The end result is a matrix [M_u] representing the 3D position and orientation of the user. The combining algorithm combines the results of the respective data streams using a weighted average. The weighting factors are based on each tracking system&#39;s reported tracking confidence according to the following equation:
 
M=Σw i Mu i  
 
     w i  ∝ confidence reported by tracking system 
     Computing 3D Rendering Matrix for Each Display 
     As depicted in  FIG. 9 , the 3D rendering matrix for each display is computed by using the user position matrix [M_u] and the display matrix [M_d_i]. These data are used to calculate the eye position using algorithm  300  to operate on the user position matrix data and using algorithm  302  to calculate the projection matrix using the calculated eye position and also using the display matrix data [M_d_i]. The calculated projection matrix [M_p_i] is then fed to the 3D rendering algorithm  304  which then outputs the data to the display for viewing. 
     The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.