Patent ID: 12190464

DETAILED DESCRIPTION

The following description provides non-limiting exemplary implementations of the methods of the present invention. These methods may be used for identifying three-dimensional positions in space in order to align physical and virtual spaces or objects with one another. Many of the cases described below illustrate these concepts using architectural or floorplan drawings. In those cases, other users or objects may be aligned using portions of the building, etc. as reference points. However, the methods of the present invention may be used much more broadly, including any time XR content and physical environments, users, objects, etc. are to be aligned with one another, and should not be understood as being limited to alignment using buildings, floorplans, etc.

Now, with initial reference toFIG.1, in a hypothetical physical environment100having a first physical surface104and a second physical surface106is intended to be aligned by a user102with a computer-generated XR environment or model200having a first virtual surface202, a second virtual surface204, and a virtual intersecting plane206. The methods disclosed below provide a means for making this alignment in a much more accurate fashion compared to conventional alignment methods. In particular, in aligning physical environment100with model200, the presently described methods use a minimal number of points defined in a physical space to provide a unique position and orientation without reliance on uncertain user input, statistical algorithms (e.g., SLAM) or other spatial positioning, visual odometry or computer vision, or large arrays of sensors. These points act as robust fiducials (i.e., a fixed basis or point of comparison) to provide mathematics that is both necessary and sufficient for deriving alignment with closed-form, analytical solutions that minimize uncertainty. Certain methods described herein require two points on each of two surfaces, such that there exists two intersecting planes, each of which contains one of the sets of two points. These four points in space can be chosen by any method, including current tagging/anchoring methodologies (e.g., QR codes, visual tracking, controller position). However, preferably, extremely simple methods (e.g., time-of-flight, depth sensing, positional sensing) can be used to identify the points, thereby minimizing the sensors and algorithms necessary for tagging, positioning, or identifying the location of these points.

Now, in certain embodiments, the first physical surface104and the second physical surface106intersect with one another along a first elongate intersection108. In the illustrated case, the first physical surface104and the second physical surface106are each physical walls that intersect with one another at an intersection (e.g., corner108). Accordingly, in this example, the first physical surface104is defined as being positioned to the left of the corner106(as seen inFIG.1) and the second physical surface106is defined as being positioned to the right of the corner.

In certain cases, however, the physical surfaces of the physical environment100may not clearly, physically intersect one another at an intersection108as in the case discussed above. This might be the case, for example, as depicted inFIG.2, with a pair of non-intersecting angled wall segments, including first wall segment110and second wall segment112that provide an opening114between them for a door, window, etc. In such cases, each of the wall segments110,112can be projected to provide projected surfaces (i.e., planes), which may be a projected surface, that are each co-planar with the inner surface of one the wall segments and that intersect with one another. In particular, a first projected surface116is provided and is co-planar with the inner surface of first wall segment and a second projected surface118is provided and is co-planar with the inner surface of second wall segment. While first wall segment108and second wall segment110do not intersect, the first projected surface114and second projected surface116intersect at projected intersection120, which is not located on or within either wall segment108,110.

In other cases, intersecting walls may still not provide a clear “intersection” between them. For example, as shown inFIG.3, a continuous wall providing a first wall segment122that is joined continuously with a second wall segment124by a curved intersection126does not provide a clear intersection between them. As before, in such cases, each of the wall segments122,124can be projected to provide projected surfaces that are each co-planar with the inner surface of one the wall segments and that intersect with one another. In particular, first wall segment122is projected to provide first projected surface128that is co-planar with the inner surface of first wall segment and second wall segment124is projected to provide second projected surface130that is co-planar with the inner surface of second wall segment. While first wall segment122and second wall segment124do not clearly intersect, the first projected surface128and second projected surface130intersect at projected intersection132, which is located within the continuous wall.

In the cases illustrated above, the projected surfaces are co-planar with the inner surface of the relevant wall segments because the relevant intersection point of the selected “corner” is an inside corner. In other cases, the projected surfaces may be placed in other positions, including at other depths, with respect to the wall or wall segments. For example, if the selected “corner” is an outside corner, the projected surfaces would be co-planar with an outer surface of the relevant wall segments.

Next, returning toFIG.1and with reference toFIG.4, while not necessary in every embodiment of the present invention, in certain embodiments, the method further includes providing a physical environment100having selected intersecting plane134. This selected intersecting plane134is used in aligning the physical environment100and the model200at a specific and selected vertical position relative to one another. In this embodiment, an arbitrary Cartesian coordinate system is provided, which coordinate system includes an X-axis, Y-axis, and Z-axis, where the Y-axis is aligned to the “vertical” direction. Additionally, movement in the direction of the “Y” arrow is designated as “up” (i.e., opposite the direction of gravity), and movement in the direction opposite the “Y” arrow is designated as “down.” As such, each of the axes (i.e., x-axis, y-axis, and z-axis) are orthogonal to one another.

In certain cases, including in the embodiment illustrated inFIG.1, the selected intersecting plane134may correspond to another intersecting plane136, which may be a ground or floor surface. However, in other cases, the selected intersecting plane134may correspond to a ceiling or to another arbitrarily-selected plane that may be placed anywhere in the physical environment100. The selected intersecting plane134may be a physical plane (i.e., one that exists in physical space) or may be non-physical (i.e., a hypothetical or imaginary plane). In each case where a selected intersecting plane134is provided, the first physical surface104and the selected intersecting plane intersect with one another along a second elongate intersection138and the second physical surface106and the selected intersecting plane intersect with one another along a third elongate intersection140. As such, the first elongate intersection108is oriented in a first orientation (i.e., parallel with the Y-axis in this illustrated embodiment), the second elongate intersection138is oriented in a second orientation (i.e., parallel with the X-axis in this illustrated embodiment), and the third elongate intersection140is oriented in a third orientation (i.e., parallel with the Z-axis in this illustrated embodiment). Similarly, the virtual intersecting plane206of model200intersects with the first virtual surface202along a fourth elongate intersection208and intersects with the second virtual surface204along a fifth elongate intersection210.

Using a computer-based XR generation system, the model200is assigned a position and orientation such that the first virtual surface202is co-planar with the first physical surface104and the second virtual surface204is co-planar with the second physical surface106. As such, after aligning the virtual surfaces202,204with the physical surfaces104,106, respectively, in this manner, an intersection point212of model200, where the fourth elongate intersection208intersects with the fifth elongate intersection210, is disposed along the first elongate intersection108. As noted above, inFIG.4, selected intersecting plane134may be placed, such as by the tagging procedure discussed earlier, such that it corresponds to a ground or floor surface of physical environment100and virtual intersecting plane206is co-planar with the selected intersecting plane. As such, the intersection point212is not only disposed along the first elongate intersection108of physical environment100but is also co-planar with the floor136and selected intersecting plane134. Accordingly, this method aligns the model200with the physical environment100in the XY, YZ, and XZ planes.

However, in other embodiments, including the embodiment shown inFIG.5, the selected intersecting plane134is not co-planar with the floor136. Instead, the selected intersecting plane134is located at a first height H1(measured along the Y-axis) and the floor136is located at a second height H2(measured along the Y-axis) that is positioned vertically below height H1. Accordingly, in this embodiment, the selected intersecting plane134is located vertically above the floor136. Thus, by following the same alignment process above, the intersection point212is still disposed on the first elongate intersection108of physical environment100but is not co-planar with the floor136at height H2; instead, the intersection point is located above the floor at height H1. Nevertheless, this method simultaneously aligns the translational and rotational positions of the model200with the translational and rotational positions of the physical environment100. Thus, this method permits physical environment100to be aligned with model200at any selected vertical position, including a user-selected vertical position. Likewise, this method permits the model200to be aligned with the physical environment at any selected vertical position, including a user-selected vertical position.

These methods may be used in mapping or in providing a system of record for multiple coordinate systems. In this mapping process, a position and a rotation are needed to completely align or coordinate the different coordinate systems. A common position may be identified using the methods discussed above. So, a next step is to provide an orientation or rotational position. The users could seek to align coordinate systems by identifying common structures to align the rotational position of one coordinate system with the rotational position of another coordinate system. However, user-based methods or manual methods can be a source of error. It is preferable to use a method of orienting coordinate systems that does not introduce user error. Therefore, in certain preferred embodiments and as illustrated inFIG.6, a further step of the method is bisecting an angle formed between the first virtual surface202and the second virtual surface204to define an alignment angle ⊖. Then, using the XR generation system to define an alignment vector E that is co-planar with the virtual intersecting plane206and that extends away from the intersection point212at the alignment angle ⊖. Also, once the intersection point212has been identified, the physical environment100and model200are preferably aligned with one another for all users102and for all XR systems that they may be using, despite the different perspectives of those users and despite the different coordinate systems, origins, etc. used by those XR systems. Advantageously, this alignment would permit all users102to identify, navigate to, interact with a consistent point of interest X within both the physical environment100and model200based on the single intersection point212alone. While not required, the alignment vector E would assist in more quickly and reliably aligning the physical environment100and model200for the users102.

The method discussed above relies on the use of planes or projected surfaces (e.g., surfaces116and118, shown inFIG.2, or surfaces128and130, shown inFIG.3) to identify a specific location in the physical environment100that is then aligned with an intersection point212this is formed at an intersection of planar surfaces in an XR model, such as surface202and204(and sometimes206) shown inFIG.1. To identify a plane in space, a minimum of three points or locations must be identified. Therefore, in the XR model200, at least three points must be identified for each of the surfaces202,204,206. As previously noted, these points in space can be chosen by any method, including current tagging/anchoring methodologies (e.g., QR codes, visual tracking, controller position).

For example,FIG.7, shows a simplified example of an XR generation system300that may be used according to the methods described herein to align XR content with a physical environment and for enabling users102to interact with that that XR content and with each other while located within the physical environment. As shown, the system300may include one or more computer systems302used for generating XR content. The computer system302may be configured to work cooperatively with a positional sensor304, including accompanying software, using time of flight (ToF) or other methods to identify points in the physical environment that are then used to identify the surfaces discussed above. Positional sensor304may include any device that can provide, within a degree of certainty, a three-dimensional location and, preferably, orientation of a point in space. This can be carried out using image sensing, time of flight, or other positional sensing methods. For example, electromagnetic methods, Wi-Fi, GPS, acoustic, and other methods may be used to detect a3D position of a point in space. Additionally, system300may include one or more output devices306, which are illustrated in this case as XR goggles but that may include other similar output devices (e.g., screens, headsets with displays, etc.) to display XR content generated by the computer systems302. Additionally, system302may include one or more input devices308, which are illustrated in this case as game controllers but that may include other similar input devices (e.g., microphones, keyboard, computer mouse, XR handset, or other XR peripherals, etc.) to receive any form of input from the users102.

In other embodiments, a different method used to align XR content with a physical environment may rely on the use of lines instead of surfaces. To define a line, only a pair of points must be identified. Therefore, as explained below, this alternative method merely requires two points to be identified for each line defined and does not require three points to define a surface as in the prior methods.

Now, with reference toFIGS.8and9, physical environment100is similar to that fromFIG.1. As before, an arbitrary Cartesian coordinate system may be defined, which system includes an X-axis, Y-axis, and Z-axis that define an XY plane, a YZ plane, and an XZ plane. The physical environment100includes a first physical surface104that is parallel with the XY plane and a second physical surface106that is parallel with the YZ plane and that intersects the first physical surface along a first elongate intersection108. As before, in certain cases, the intersecting surfaces may not intersect or may not provide a clear “intersection” between them. In those cases, projected surfaces may be used in place of the actual surfaces to define the first and second physical surfaces104,106in order to provide the elongate intersection108. Then, a selected intersecting plane134is defined as intersecting with the first physical plane104along a second elongate intersection138and as intersecting the second physical plane106along a third elongate intersection140. In some cases, including the illustrated embodiment, the selected intersecting plane134is parallel with the XZ plane. However, in other cases, the selected intersecting plane134is not parallel with the XZ plane. As detailed below, in this embodiment, a first set of the points defines a first line, and a second set of points defines a second line and those two lines are used to define a unique intersection point that is then aligned with the first elongate intersection108in order to align the physical environment100with the XR content.

Using an XR generation system, such as system300(FIG.7), points A1and A2may be defined as being co-planar with the first physical surface104and points B1and B2may be defined as being co-planar with the second physical surface106. These points may be recorded using a positional sensing device, such as the depth-sensing device304shown inFIG.7. In certain embodiments, the XR generation system may be configured to automatically define one of the pair of points. For example, in certain cases, if a user records point A1or point A2, the other point may be automatically defined by the XR system. Similarly, if a user records point B1or point B2, the other point may be automatically defined by the XR system. During this auto-recording step, the XR generation system may be configured to detect the relevant physical surface and to automatically record the second point at a pre-defined distance away from the first point on the selected physical surface. In other embodiments, the XR system is configured to auto-sample the environment and to automatically detect all points without requiring user involvement or with minimal user involvement, e.g., a confirmation step where the user confirms that eth selected points should be used in carrying out the alignment process.

As depicted, in this embodiment, the first physical surface104and the second physical surface106each form a plane. Using the XR generation system, a line is passed through points A1and A2to define line A that is co-planar with first physical surface104and a second virtual line is passed through points B1and B2to define line B that is co-planar with the second physical surface106. Next, line A and line B are each projected onto the selected intersecting plane134to provide line segment AP and line segment BP, respectively. Then, an intersection point142is defined where line AP and line BP intersect with one another. As may be seen by comparingFIGS.8and9, in certain embodiments, it may be necessary to extend line AP, line BP, or both to cause them to intersect with one another. The intersection point142of the two line segments AP and BP now defines a unique position in space that is analytically determined. Once the intersection point between lines AP and BP has been identified, that unique intersection point142is aligned co-linearly with the first elongate intersection108. Then, at that point, the intersection point142may be used to align XR content provided by the XR generation system with the physical environment100.

As before, a further step of the method is bisecting an angle formed between line segment AP and line segment BP to define an alignment angle Ω. The XR generation system may then be used to define an alignment vector F that is co-planar with selected intersecting plane134and extends away from the intersection point142at the alignment angle Ω. The alignment vector F provides a consistent direction for all users and for all the XR systems that they may each be using, despite the different perspectives of those users and despite the different coordinate systems, origins, etc. used by those XR systems. Similarly, intersection point P provides a consistent position for all users. Advantageously, this alignment vector F and intersection point P, in combination, provide a unique orientation and position that uses only the physical walls as fiducials and permits the users to align all coordinate systems to a common point of interest Y.

Using the methods discussed above, since the points can be tagged using positional sensors, rather than computer vision or position systems such as SLAM, the total uncertainty is reduced to only the intrinsics (systematic and statistical) of the positional sensors (or other surface detection). Further, even human error is eliminated using these methods because the location of the points on the walls are immaterial and simple mathematical tests can determine if the points are, in fact, on either side of intersecting planes. If no intersection is calculated from the projected line segments, the user can be alerted that they have selected points inappropriately and may then be forced to do it again. Next, even the order of tagging points does not matter because depth-sensing/surface detection can successfully determine which two points belong in each set of points based on inferred surface normal. As discussed above, the second point of each set of points can be automatically determined if basic assumptions on distance can be made. For example, if one assumes that all walls have at least a minimum length (e.g., 10 cm or 1 mm), then the XR system can automatically select a second point that is within some presumed distance from the first point selected. In those cases, therefore, the user would only need to select 2 points rather than selecting 4 points.

The present disclosure describes various methods that can be used to locate and record various points in space. With reference toFIGS.10and11, a physical environment146is shown that has a first wall148that intersects with a second wall150along an intersection152. Using the methods described above, an intersection point154may be defined in a virtual model as being aligned (i.e., co-linear) with and positioned at a height H3relative to intersection152. By locating intersection point154relative to the physical environment146, other points of interest in the virtual model may also be located in that same environment. For example, points of interest Y1and Y2, which may each represent a virtual avatar's position or user position, where each of the points of interest may have a unique coordinate system. Advantageously, the methods described allow for these points of interest Y1, Y2and their respective coordinate systems to be aligned and correctly represented, positioned and moved with respect to the other points of interest and coordinate systems.

In the illustrated embodiment, images (e.g., QR codes)156, which are placed on each of the walls148,150, may be imaged by an imaging device158such as a camera on a headset or other connected device. Capturing these images156provides a rough position of the points on the walls (e.g., A1, A2, B1, and B2). The precise position and orientation of the QR code (or other image to be detected) is not critical in selecting, detecting, or placing points, as described above, in implementing the methods. However, it is important to ensure that related points on each wall (e.g., A1and A2or B1and B2) lie on the same plane. Again, while their relative position on that plane is not particularly critical, ensuring that each point has the same “depth” is important.

For example, as depicted inFIG.11, if imaging one of the QR codes identifies a point160that is recorded as located on the outside surface of a wall148(i.e., at depth D1), it is important that other points are not recorded as located within the wall (like point162at depth D2) or located outside of the wall (like point164at depth D3). This is a dominant and, perhaps, the most critical factor in accurately and successfully carrying out the methods described above. As such, in certain embodiments of the method, a depth verification step can be used to verify the depth of each related point used in this method and to ensure that each lies on the same plane when using image detection.

To conduct this check, an imaging device158is first used to detect and record the rough position of each the points/images. Using an XR system, a ray162is cast forward from the detected point A1X to a location in space. In this case, “forward” may be defined as normal to QR code/image and is based on the orientation of the QR code/image. Again, the precise orientation of the image is immaterial to this method and the “forward” direction does not need to be normal to the physical surface (e.g., wall148) on which the image is placed. This projected point A1X is preferably projected outwards by some reasonable distance (e.g., 1 m) that is greater than some multiple of the image sensor's uncertainty in detecting the location of the wall surface. Similar points A2X, B1X, and B2X are shown for each of the other images156. In each case, from the projected point, a ray164is cast backwards towards the wall surface using a positional sensor166as a re-projected point A1. Similar points A2, B1, and B2may also be provided. In each case, the depth where this re-projected point is located with respect to the wall148is detected by sensor166and is recorded as the actual location (i.e., depth) of the point. The points A1, A2, B1, and B2may be used according to the methods described above to locate and define an intersection point168that is col-linear with intersection152, including by using intersecting lines or by using intersecting planes. Likewise, an alignment vector170may be defined using the methods described above. Using the intersection point168and, optionally, the alignment vector170, users can define the position and orientation of various points of interest (e.g., Y1and Y2), including points of interest that are each associated with a unique coordinate system that is different from the coordinate system of other points of interest, the environment, and the model.

Although this description contains many specifics, these should not be construed as limiting the scope of the invention but as merely providing illustrations of some of the presently preferred embodiments thereof, as well as the best mode contemplated by the inventor of carrying out the invention. The invention, as described herein, is susceptible to various modifications and adaptations as would be appreciated by those having ordinary skill in the art to which the invention relates.