System for virtual display and method of use

A preferred system and method for projecting a business information model at a construction site includes a network, a system administrator connected to the network, a database connected to the system administrator, a set of registration markers positioned in the construction site, and a set of user devices connected to the network. The system includes a hard hat, a set of headsets mounted to the hard hat, a set of display units movably connected to the set of headsets, a set of cameras connected to the set of headsets, and a wearable computer connected to the set of headsets and to the network. The cameras capture an image of the set of registration markers. A position of the user device is determined from the image and an orientation is determined from motion sensors. A BIM is downloaded and projected to a removable visor based on the position and orientation.

FIELD OF THE INVENTION

The present invention relates to systems and methods managing distribution and use of information at a construction site. In particular, the present invention relates to systems and methods for displaying and using a business information model and other construction information at a construction site. The present invention also relates to virtual display systems, such as heads up displays.

BACKGROUND OF THE INVENTION

A major goal of any contractor in the building industry is timely job completion. Hence, efficiency is a paramount concern.

Scheduling of construction projects often requires numerous subtasks. The subtasks are often interdependent. These subtasks must necessarily be completed in order to maximize efficiency. For example, electrical conduit and foundation pads must be in place before installation of electrical equipment can begin. If an error is made in any interdependent subtask, it must be corrected before other tasks can proceed. Hence, correcting errors made in interdependent subtasks is expensive and time consuming because it often delays project completion.

Furthermore, heavy equipment such as cranes and elevators are scheduled to be on site at specific times when they are needed. If errors in subtasks are made, then the equipment must be either stored or rescheduled quickly, leading to increased construction costs and delay in project completion.

In a similar way, delivery of certain engineering, mechanical and scheduling information is critical to timely project completion. For example, engineering change orders, site drawings, schematics, photographs, tool type and location, physical equipment specifications and diagrams and repair manuals and parts lists for heavy equipment all are required to be easily available at a construction site for maximum efficiency. Other critical construction information includes queuing times and scheduling times for skilled personnel, tools and equipment. Any delay in receiving such critical information can effect timely project completion.

In order to be useful, construction information is generally accessed in the field at a construction site by paper drawings or in some cases, on a laptop computer. However, neither paper drawings nor laptop computers display the information to scale. Viewing information in this manner is often difficult to do and can lead to dangerous and costly mistakes.

Modern construction projects have attempted to remedy many of the inefficiencies caused by lack of timely information delivery and errors in interdependent subtasks by employing a consolidated building information model (BIM). The BIM is a set of computer graphics files that, when viewed on a CAD system, provide the current displays of wire frame models of structures in the completed construction project. The CAD display is layered in a manner that allows all separate views and accurate representations of all structures, physical equipment, wiring and plumbing. While the BIM has helped coordination of tasks and schedules, it is still not completely satisfactory because it is not easily accessible in the field. Further, the BIM does not address schedules or query times.

The prior art has attempted solutions to solve some of these problems with limited success. For example, U.S. Publication No. 2014/0184643 to Friend discloses a system and method of dynamically coordinating machines and personnel about a physical worksite using augmented content on an operator display device. To receive the augmented content, which is generated by an off-board management system, the operator display device is associated with a transmitter/receiver attached to a machine, such as an excavator or bulldozer. A position of the machine or personnel is determined by a GPS system or a laser scanning system. The operator display device includes a visor or goggles with transparent lenses, a scaled-down controller that includes a processor or other electronics to communicate with a personnel transmitter/receiver carried by a person and a controller that processes information signals received from the off-board management system and project them on the lenses of the operator display device. The augmented content is projected in the person's field of view as an overlay superimposed on the surrounding environment to show restricted area for personnel, routes of travel for machinery, and areas designated for excavation. However, the operator display device of Friend cannot determine its position without a construction site. Further, it does not display or interact with a BIM model.

U.S. Publication No. 2014/0210856 to Finn et al. discloses a system and method that integrates augmented reality technology with land surveying. A 3D digital model of internal elements of a building is generated using a 3D laser scanner upon installation of the internal elements, such as electrical and plumbing before wall panels are installed. The 3D digital model is associated with a set of markers that are placed on a finished wall in the building. The markers are used to project the generated 3D model on a mobile device, such as a smartphone, in view of a user. However, the system in Finn requires the 3D model to be generated once the internal systems are already installed, sometimes incorrectly, just prior to installing wall paneling. Therefore, the system in Finn cannot be used to prevent incorrect installation of building elements leading to costly construction overruns.

U.S. Publication No. 2014/0268064 to Kahle et al. discloses a system and method for projecting an image on a surface in a building under construction. The system includes a projector mounted on a moveable support for supporting a worker at a work position in the building. The projector projects the image on a surface above the moveable support in response to an image signal defining the image to be projected. The projected image indicates the location of connectors, anchors, and holes to be affixed to, or cut through, the surface and features behind the surface. A positioning system for determining the two dimensional position of the projector includes a laser measuring system that projects a rotating beam of laser light that sweeps across the moveable support to determine the distance and heading of the moveable support. However, the system in Kahle is prone to error because the laser measuring system is easily misaligned in the construction environment, thereby providing an incorrect position to the projector. Further, the system must be attached to the moveable support and cannot be transported easily between construction sites.

Therefore, there is a need in the art for a portable augmented reality system that provides access to virtual information accurately, in real time, at a construction site to prevent mistakes, thereby increasing the usability of the information and improving safety, time use and cost efficiency.

SUMMARY

A system and method for projecting information including, as an example, segments of a business information model at a construction site includes a network, a system administrator connected to the network, a database connected to the system administrator, a set of registration markers positioned in the construction site, and a set of user devices connected to the network. Each user device includes a hard hat, a set of headsets mounted to the hard hat, a set of display units movably connected to the set of headsets, a set of registration cameras connected to the set of headsets and directed towards the set of registration markers, and a wearable computer connected to the set of headsets and to the network.

The wearable computer is programmed with a set of instructions to carry out the method which includes the steps of receiving the business information model, receiving a position image of the set of registration markers, receiving a set of motion data, determining a position of the user device and an orientation of the user device based on the position image and the set of motion data, rendering the business information model based on the position, the orientation, and the position image as a rendered business information model, and displaying the rendered business information model as a stereoscopic image to the user.

The described embodiments herein disclose significantly more than an abstract idea including technical advancements in the fields of construction management and data processing, and a transformation of data which is directly related to real world objects and situations. The disclosed embodiments enable a computer and integrated optics and dedicated electrical components to operate more efficiently and improve the optical display of the BIM and other information and construction management technology in general.

DETAILED DESCRIPTION

It will be appreciated by those skilled in the art that aspects of the present disclosure may be illustrated and described in any of a number of patentable classes or contexts including any new and useful process or machine or any new and useful improvement.

Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, C#, .NET, Objective C, Ruby, Python SQL, or other modern and commercially available programming languages.

Referring toFIG. 1, system100includes network101, system administrator102connected to network101, and a set of user devices104each of which is connected to network101. System administrator102is further connected to BIM database103for storage of relevant data. For example, receiving data may include a business information model, engineering change orders, textual data, equipment manuals and operation instructions, images, photos, text messages, videos, emails, graphics, documents, 2-dimensional and 3-dimensional drawings, and sketches.

In a preferred embodiment, each of user devices104communicates with system administrator102to access BIM database103to project a BIM as will be further described below.

It will be appreciated by those skilled in the art that any type of three-dimensional rendering may be employed in the disclosed embodiment and that a BIM is just one example of such a three-dimensional rendering.

Wearable computer201includes processor207and memory209connected to processor207, and network interface208connected to processor207. Augmented reality application210, BIM211, and a set of videos, images, and data212are stored in memory209. In one embodiment, control input device206is connected to wearable computer201. In preferred embodiment control input device206, is a remote control having a navigation pad and a selection button. Any type of control input device known in the art may be employed.

Headset202is further connected to display unit213and a set of cameras214. Headset202includes processor215, a set of sensors216connected to processor215, and memory217connected to processor215.

Referring toFIG. 3A, a preferred implementation of user device200is described. Hard hat302is worn by user301. Hard hat302has pocket305integrally formed in it. In a preferred embodiment, the hard hat includes a “pocket and clip” arrangement used to secure the headset as shown and described in U.S. Pat. No. 8,191,292 to Cummings, et al., which is incorporated herein by reference. Various types of head covers or helmets may be employed to support the headset. Headset303detachably mounts to hard hat302by flexible mounts304attached to case350and pocket305. Headset303is further connected to wearable computer313via connection314. Wearable computer313is preferably a portable computing device, such as a laptop or tablet computer, worn as a backpack by user301. Connection314provides a data and power connection from wearable computer313to headset303. Headset303includes processor310, memory311connected to processor310, and sensors312connected to processor310. Mounting arm306is slidably inserted into headset303to allow forward and backward movement. In a preferred embodiment, the mounting arm is biased by a mechanical coil spring which enables it to retract into case350. Display arm307is pivotably connected to mounting arm306for pivotal movement about axis324. Display unit308is attached to display arm307. Display unit308includes projector325, camera326, and display light guide309. Camera326has field of view328. In a preferred embodiment, field of view328is 90°. In other embodiments, other suitable field of view ranges may be employed. Display arm307is further connected to headset303with data and power connection327.

User301wears communication device315. Communication device315includes earpiece speaker316and microphone317. Communication device315is preferably connected to wearable computer313via a wireless connection such as a Bluetooth connection. In other embodiments, other wireless or wired connections are employed. Communication device315enables voice activation and voice control of an augmented reality application stored in the wearable computer313by user301.

In one embodiment, camera matrix318is detachably connected to headset303. Camera matrix318includes halo319and halo321, each of which is detachably connected to headset303. A set of base cameras320is connected to halo319and in communication with headset303. A set of angled cameras322is connected to halo321and in communication with headset303.

Referring toFIG. 3Bin another embodiment, camera matrix318is attached inside surface329of hard hat302. In this embodiment, halos319and321are attached to inside surface329of hard hat302with a suitable adhesive or fastener. Hole334is integrally formed in hard hat302adjacent to headset303for connection to camera matrix318. In a preferred embodiment, connector335is a USB 3.0 connector, connected to a processor of headset303and positioned in hole334to connect to camera matrix318. Other suitable data connections may be employed. Set of base cameras320is connected to halo319, each of which is positioned in a hole of set of holes330. Set of holes330is integrally formed in hard hat302. In one embodiment, side cameras333of set of base cameras320are attached to headset303outside of hard hat302. In another embodiment, side cameras333are eliminated. Set of angled cameras322is connected to halo321, each of which is positioned in a hole of set of holes331. Set of holes331is integrally formed in hard hat302.

In another preferred embodiment, the cameras are each mounted securely to the inside surface of the hard hat and are positioned to view the outside world through the holes.

In a preferred embodiment, a BIM is downloaded from a system administrator server into a memory resident in a wearable computer313. The BIM is transmitted from wearable computer313through headset303and projector325for viewing adjacent eye323of user301to augment the vision of user301, as will be further described below. The user can select different layers of the BIM to view via voice control. For example, the BIM includes an electrical layer, which shows the location of electrical conduit, connection points, and equipment. As the user moves, headset303and wearable computer313tracks the location of user301and the position and orientation of the user's head using camera326and/or camera matrix318.

In one embodiment, a set of data is downloaded, selected, and displayed to user301. In one embodiment, the position and orientation of the user's head is not tracked in a display mode. Rather, the data is displayed without regard to the position of the user or hard hat. Any type of data content may be selected, formatted, scaled and displayed, including images, photos, text messages, videos, emails, graphics, documents, drawings, and sketches.

In a preferred embodiment, processor310is a 2.8 GHz octa-core Snapdragon 810 processor available from QUALCOMM® Technologies, Inc. Other suitable processors known in the art may be employed.

In a preferred embodiment, sensors312is a 9-axis motion tracking system-in-package sensor, model no. MP11-9150 available from InverSense®, Inc. In this embodiment, the 9-axis sensor combines a 3-axis gyroscope, a 3-axis accelerometer, an on-board digital motion processor, and a 3-axis digital compass. In other embodiments, other suitable sensors and/or suitable combinations of sensors may be employed.

In a preferred embodiment, memory311is a 2 GB LPDDR3 RAM. Other suitable memory known in the art may be employed.

In a preferred embodiment, each of base cameras320and angled cameras322is a 16 megapixel smartphone camera capable of recording video at 30 fps that includes a CMOS image sensor, part no. 5K3M2 available from Samsung Semiconductor. Other suitable cameras and/or image sensors known in the art may be employed.

Referring toFIG. 4A, user device400includes display unit402which includes camera404on the temporal side of eye406of user401and display light guide405. Camera404has field of view413. Display light guide405is positioned in the field of view and adjacent to eye406. Display unit402is movably connected to headset403, which is detachably mounted to hard hat407. Display unit409is movably connected to headset408, which is detachably mounted to hard hat407. Display unit409includes camera410on the temporal side of eye412of user401. Camera410has field of view414. Display light guide411is in the field of view and adjacent to eye412. Display units402and409and headsets403and408are the same as previously described. Display units402and409provide a stereoscopic augmented view to user401.

In a preferred embodiment, each of cameras404and410is a 16 megapixel smartphone camera capable of recording video at 30 fps that includes a CMOS image sensor, part no. 5K3M2 available from Samsmung Semiconductor. Other suitable cameras and/or image sensors known in the art may be employed.

Referring toFIGS. 4B, 4C, and 4Din one embodiment, hard hat407includes pocket450integrally formed in hard hat407and adjacent to pocket449. Glasses451includes display arm452adjustably engaged with connector453and display arm469adjustably engaged with connector470. Mounting arm454is adjustably engaged with connector453. Mounting arm471is adjustably engaged with connector470. Data connection468connects display unit409to headset408detachably mounted to pocket449, as previously described. Display arm452includes a set of ridges462integrally formed on it. Mounting arm454includes flexible mounts455to detachably mount glasses451to hard hat407. Mounting arm454further includes a set of ridges463integrally formed on it. Connector453has mount portion456and display portion457. Mount portion456includes channel458integrally formed in it. Channel458has ridge459integrally formed on it. Mounting arm454slidingly engages with channel458. Set of ridges463engages with ridge459to enable adjustable positional movement along directions464and465. Display portion457includes channel460integrally formed in it. Channel460includes ridge461integrally formed on it. Display arm452slidingly engages with channel460. Set of ridges462engages with ridge461to enable adjustable positional movement along directions466and467.

Likewise, display arm469includes a set of ridges472integrally formed on it. Mounting arm471includes flexible mounts476to detachably mount glasses451to a pocket in hard hat407. Mounting arm471further includes a set of ridges477integrally formed on it. Connector470has mount portion478and display portion479. Mount portion478includes channel475integrally formed in it. Channel475has ridge480integrally formed on it. Mounting arm471slidingly engages with channel475. Set of ridges477engages with ridge480to enable adjustable positional movement along directions464and465. Display portion479includes channel473integrally formed in it. Channel473includes ridge474integrally formed on it. Display arm469slidingly engages with channel473. Set of ridges472engages with ridge474to enable adjustable positional movement along directions466and467. Glasses451includes display light guides405and411and display units402and409, as previously described. Display unit402is connected to headset403with a data connection.

In a preferred embodiment, channel458is generally perpendicular to channel460and vice versa. Other arrangements may be employed.

In a preferred embodiment, channel475is generally perpendicular to channel473and vice versa. Other arrangements may be employed.

In a preferred embodiment, each of display anus452and469, connectors453and470, and mounting arms454and471is made of an injection molded plastic. Other suitable materials known in the art may be employed.

In one embodiment, mount portion456and display portion457are separate pieces attached to each other with a suitable adhesive or epoxy. In another embodiment, mount portion456and display portion457are integrally formed portions of a single piece adjacent to each other. Other attachment means known in the art may be employed.

In one embodiment, mount portion478and display portion479are separate pieces attached to each other with a suitable adhesive or epoxy. In another embodiment, mount portion478and display portion479are integrally formed portions of a single piece adjacent to each other. Other attachment means known in the art may be employed.

Referring toFIG. 5, camera matrix318will be further described. Camera matrix318includes halo501and halo502connected to halo501. Each of cameras503,504,505, and506is connected to halo501. Camera503has field of view507. Camera504has field of view508. Camera505has field of view509. Camera506has field of view510. Each of cameras511,512,513, and514is connected to halo502. Camera511has field of view515. Camera512has field of view516. Camera513has field of view517. Camera514has field of view518.

In a preferred embodiment, each of cameras503,504,505, and506is positioned approximately 90° with respect to each other around halo501. Other angular intervals may be employed.

In a preferred embodiment, each of cameras511,512,513, and514is positioned approximately 90° with respect to each other around halo502. Other angular intervals may be employed.

In a preferred embodiment, each of field of views507,508,509, and510is approximately 90°. Other field of view ranges may be employed.

In a preferred embodiment, each of field of views515,516,517, and518is approximately 90°. Other field of view ranges may be employed.

In a preferred embodiment, camera matrix318provides a 360° view of the surroundings of a user. In other embodiments, other numbers of cameras, angular positions, and field of view ranges may be employed to provide a 360° view.

Referring toFIG. 6, each of display units402and409will be further described as display unit600. Display unit600will be further described with respect to a right eye of a user. It will be appreciated by those skilled in the art that the arrangement of display unit600is simply reversed for implementation on a left eye. Display unit600includes light guide601, projector602attached to light guide601, and camera603connected to and adjacent to light guide601and projector602. Camera603is connected to headset615and includes lens604. Projector602includes light source605. Light source605is connected to headset615. Collimating lens606is positioned adjacent to light source605. Light guide601includes input surface607and output surface608, each of which is attached to the interior of light guide601. Each of input surface607and output surface608is positioned at angles ω and γ, respectively from front surface613to provide total internal reflection (“TIR”) for light guide601, thereby projecting an image in field of view610of user eye609.

In a preferred embodiment, angles ω and γ, are 30° and 45°, respectively. Any angles may be employed to provide TIR for light guide601.

In use, light source605displays an image received from headset615. The image is represented by rays611and612. Rays611and612are transmitted through collimating lens606and reflected off of input surface607for TIR. Rays611and612are further reflected off of front surface613and rear surface614and output surface608in field of view610of user eye609.

In a preferred embodiment, light source605is an organic light emitting diode (“OLED”) display such as the WUXGA OLED-XL Microdisplay, part no. EMA-100801-01, available from eMagin Corporation. In another embodiment, light source605is a light emitting diode (“LED”) display. Other suitable light sources and displays known in the art may be employed.

In a preferred embodiment, light guide601is made of acrylic. In another embodiment, light guide601is made of poly (methyl methacrylate) (“PMMA”). Other suitable materials known in the art may be employed.

In a preferred embodiment, input surface607is a flat mirror and output surface608is a partially-reflective mirror, such as a half-silvered mirror. In other embodiments, other combinations for input surface607and output surface608may employed and are summarized in Table 1 below.

Referring toFIG. 7Ain use, lens705of camera704is automatically focused on real object706at a distance d from camera704and sent to headset710as image707. Headset710and wearable computer711determine distance d and the position of display unit701with respect to real object706. Wearable computer711generates virtual image708based on distance d. Projector703projects virtual image708into light guide701, as previously described. Virtual image708is displayed as virtual object709to appear at distance d′ in view of user eye702. Virtual object709is magnified to coincide with the size and position of real object706to create a perceived depth of focus d. In one embodiment, d′ is less than d. In another embodiment, d′ is equal to d. In one embodiment, d′ is a fixed distance from camera704for all real objects.

Referring toFIG. 7B, point of view712is the view a user sees while wearing a headset and display unit. Point of view712includes floor713and adjoining wall714. Registration marker715is attached to floor713. Registration marker716is attached to wall714. Real object718is being lowered into position. According to the BIM, the correct location for real object718is outlined by virtual object717. In this way, a user easily determines if real object718is properly positioned and can quickly make adjustments to ensure real object718is properly positioned.

In one embodiment, a set of data719is displayed. The set of data719includes image720and text721. Any type of data including images, photos, text messages, videos, emails, graphics, documents, drawings, schematics, diagrams, and hand-drawn sketches may be employed. For example, image720is an installation diagram of real object718and text721is a set of installation instructions for real object718.

Each of the positions and sizes of image720and text721is optionally changed by the user.

In one embodiment, set of data719is displayed simultaneously with virtual object717. In another embodiment, set of data719is displayed without virtual object717in a display mode, as will be further described below.

Referring toFIG. 8, construction site800includes floor801and adjoining walls802and803. Registration system804includes registration markers805,806, and807positioned at precise locations on floor801, wall802and wall803, respectively and serve as a set of reference points for user device808worn by user809.

Each of the positions of registration markers805,806, and807is associated with a position in a BIM. Survey location810is precisely positioned at a known location at construction site800and saved in the BIM. Reference marker811is a master reference point based on the location of the survey location810. Each of registration markers805,806, and807is positioned from reference marker811to ensure proper location of floor801and walls802and803. At least one of registration markers805,806,807, and811will be in view of a camera of user device808worn by user809and at any given time. The camera captures an image of at least one of registration markers805,806,807, and811. A wearable computer of user device808decodes the captured image to determine a real location of at least one of registration markers805,806,807, and811. The wearable computer determines a corresponding virtual location in the BIM.

For example, user809is standing in construction site800wearing user device808and looking down at location812where object813is to be installed. Registration marker805is in view of user device808. The projected BIM shows the correct installation position814in view of user809as if the user were standing inside the BIM. As user809tilts his or her head up to look at wall802the movement of the user's head is detected by user device808and registration marker806is in view of user device808. Based on the position of registration marker806, the BIM is moved and rotated in real time to align with the user's field of vision and provide an in-person view of the BIM to user809. Crane815lowers object813towards location812. Based on the projected BIM, object813should be installed at installation position814. User809uses the projected BIM to properly lower the object813and precisely install object813at proper installation position814, thereby saving time and money in the form of overrun construction costs.

If a mistake is found, user809captures still images using the camera for upload to the system administrator or records or streams video back to the system administrator. In this way, the party responsible for the mistake can be easily and quickly identified.

In a preferred embodiment, each of codes904and906is a two-dimensional bar code. In this embodiment, each of codes904and906includes a set of marker information, including a set of dimensions of shapes903and905, and a set of x, y, z coordinates position at which registration markers901and902are placed, and a description of each shape and location. Any type of code may be employed.

Shapes903and905enable detection of codes904and906, respectively, at an offset angle. For example, shape903is an equilateral triangle and shape905is a rectangle. If a camera capturing an image of shapes903and905is positioned at an offset angle, shapes903and905will appear as a scalene triangle and a parallelogram, respectively, in a skewed image.

Referring toFIG. 10, data flow1000for augmented reality application1001for a user device will be described. BIM1002is input into augmented reality application1001. Application commands1003provide input control for the processes of augmented reality application1001. Images1004are received and sent by augmented reality application1001. For example, a set of cameras captures a set of registration images. The set of marker images is used to determine the position of the user. In another example, images1004are still or video images captured by a set of cameras adjacent to the eyes of the user and saved to memory for later upload or streamed to a server. Point of view image1005is captured by the set of headset cameras adjacent to the eyes of a user. Set of data1007is input into augmented reality application1001.

In a preferred embodiment, the position of the user is determined from the set of code images1004by augmented reality application1001. Augmented reality application1001orients BIM1002according to the determined position of the user. Commands1003determine which layers of BIM1002are displayed. Augmented reality application1001overlays the selected layers of BIM1002at the determined position to generate stereoscopic image overlay1006for display.

In one embodiment, commands1003determine a subset of set of data1007to display and the size and position of the subset of the set of data. Augmented reality application1001overlays the selected subset of data1007according to the selected size and position of the set of data1007for display.

Referring toFIG. 11, commands menu1100includes standby/run toggle1101, BIM layer selection1102, reload BIM1103, save overlaid image1104, and calibrate1105. Standby/run toggle1101toggles the augmented reality application to a standby mode or a run mode. BIM layer selection1102enables the user to select any layer of the BIM to view. For example, the layers include, but are not limited to, structural, electrical, plumbing, data, and HVAC. Reload BIM button1103downloads the BIM into memory. Save overlaid image1104captures a “screen capture” of the point of view and the overlaid BIM from the perspective of the user. Calibrate1105executes a calibration process, as will be further described below. Position and orientation toggle1106toggles the position and orientation functions on and off to selectively run in a display mode. Select data1107enables the user to select which data to display and the size and the position of the selected data. Selection of1101,1102,1103,1104,1105,1106, and1107is accomplished via voice controls.

Referring toFIG. 12, state machine method1200for an augmented reality application will now be described. State machine method1200begins at step1201in a power off mode. Once the system is enabled in step1202by initiating power, state machine method1200proceeds to a standby mode at step1203. Once a “run” command is received, state machine method1200proceeds to step1204. At step1204, a position and an orientation function of the augmented reality application is toggled on or off. If toggled off then the augmented reality application runs in a display mode at step1205and optionally displays a set of data selectable by the user. The augmented reality application runs in the display mode until the user toggles the position and the orientation function on at step1204. If turned on, then state machine method1200proceeds to step1206.

At step1206, state machine method1200turns on a set of cameras and begins to search for a registration marker in a loss of “marker lock” mode. At step1207, a position and orientation of a user device is determined from the registration marker, as will be further described below. If the position and orientation of the user device cannot be determined, then state machine method1200returns to step1206to search for a registration marker. If the position and orientation of the user device is determined, then state machine method1200proceeds to step1208. At step1208, the augmented reality application runs in a “marker lock” mode, that is the position and orientation of the user device can repeatedly be determined within a predetermined time. In this step, a runtime loop for the augmented reality application is initiated and a BIM is displayed, as will be further described below. In a preferred embodiment, the predetermined time is 30 seconds. Other times may be employed.

In one embodiment, the set of data is displayed when the augmented reality application runs in the “marker lock” mode.

At step1209, a consistency is determined. In this step, if the position and orientation of the user device can be repeatedly determined within the predetermined time, then state machine method1200returns to step1208. In this step, if the BIM is properly displayed, i.e., is rotated and aligned with the user point of view, then stated machine method1200returns to step1208. If the position and orientation of the user device cannot be repeatedly determined within the predetermined time or the BIM is not properly displayed, i.e., is not rotated and aligned with the user point of view, then state machine method1200proceeds to step1210. At step1210, a message is displayed to the user indicating a position and orientation consistency problem and state machine method1200begins a calibration process at step1211, as will be further described below.

Referring toFIG. 13, method1300for registering a registration marker for a BIM will be described. The registration marker includes a shape and a code, as previously described. At step1301, a position of the registration marker is calibrated. In this step, a surveyor or a user positions the registration marker in a desired location. For example, the registration marker is placed in the middle of a wall or a column or a stud. Any desired location may be employed. Measurements are taken to ensure the registration marker is placed in the desired location. At step1302, a set of location coordinates of the placed registration marker is stored in the code and in the BIM. At step1303, a set of dimensions for the shape of the registration marker is stored in the code and in the BIM. At step1304, a description of the registration marker is stored in the code and the BIM. Method1300is repeated for each registration marker.

Referring toFIG. 14A, method1400for calibrating a position of a user device will be described. At step1401, a camera of the user device is pointed at a registration marker so that the registration marker is within a field of view of the camera.

Referring toFIGS. 14B and 14C, an offset position of user device1409with respect to registration marker1411shown in a top view and a side view, respectively, will now be described. User device1409has camera1410. Camera1410has camera axis1414. Registration marker1411is in view1413of camera1410. Registration marker1411has marker axis1412. Camera1410and user device1409is positioned offset with respect to registration marker1411. Position angle α is the angle between marker axis1412and camera axis1414in the x-z plane of coordinates1415. Position angle β is the angle between marker axis1412and camera axis1414in the y-z plane of coordinates1415. In one embodiment, registration marker1411is rotated about the z-axis of coordinates1415.

Because of the offset position of user device1409and camera1410as defined by position angles α and β, the image of registration marker1411is skewed.

Returning toFIG. 14A, at step1402, an image of the registration marker is captured by the camera.

Referring toFIG. 14D, skewed image1416includes skewed registration marker1442. Skewed registration marker1442includes skewed shape1417and skewed code1418. Skewed registration mark1442is in the x-y plane defined by x-axis1419and y-axis1420. Z-axis1421traverses perpendicularly through skewed registration mark1442. As can be seen inFIG. 14D, skewed registration mark1442appears as a parallelogram. In this example, skewed registration mark1442is rotated approximately 30° about each of x-axis1419, y-axis1420, and z-axis1421.

Returning toFIG. 14A, at step1403, a set of edges in the image of the registration marker is located. In this step, Gaussian smoothing is first applied to the image to reduce noise in the image. In a preferred embodiment, Canny edge detection is then employed to locate the set of edges. In other embodiments, other edge detection means may be employed. In one embodiment, edge thinning is applied to the set of edges to remove any unwanted points. In a preferred embodiment, the set of edges is a boundary of the shape of the registration marker.

At step1404, the image is deskewed in order to determine a set of position angles with respect to the registration marker, as will be further described below.

At step1405the code is read to determine the set of dimensions of the shape of the registration marker, including an actual height and an actual width. At step1406, a distance from the camera to the registration marker is determined.

At step1407, an absolute position of the user is calculated based on the position angles and the distance from the registration marker.

Referring toFIG. 14E, step1404will be further described as method1422for deskewing an image. Method1422begins at step1423. At step1424, a set of reference lines for the set of edges of a registration marker is determined. In a preferred embodiment, the set of references lines is determined by the Hough transform. Other suitable methods known in the art may be employed.

At step1425, a pair angle is calculated between each pair of intersecting reference lines to generate a set of pair angles. At step1426, a skew angle is calculated from set of pair angles by averaging the set of pair angles. At step1427, the image is rotated about an axis by the skew angle. The skew angle is the position angle with respect to each axis, as previously described. At step1428, whether or not the image has been deskewed for all axes is determined. If not, method1422advances to the next axis at step1429and returns to step1424. If so, method1422ends at step1430.

In a preferred embodiment, each of heights1433and1435and widths1434and1436is measured by counting the number of pixels for deskewed registration marker1444and deskewed image1443.

Referring toFIG. 14G, step1406will now be further described. Camera1437has field of view1438spanning an angle θ, which varies depending on the type of camera employed. Registration marker1439is in plane1440. Plane1440is distance1441from camera1437. Height1442of registration marker1439is retrieved from a code contained in registration marker1439. Distance1441is calculated by:

d=hx⁢⁢tan⁢⁢θ,Eq.⁢1
where d is distance1441, h is height1442, θ is angle θ of field of view1438, and x is a height percentage of the height of the deskewed registration marker in the deskewed image to the height of the deskewed image. For example, if the height of the deskewed registration marker is 60% of the height of the deskewed image, then x=0.6.

Referring toFIG. 15, runtime process1500for an augmented reality application will now be described. Runtime process1500starts at step1501. At step1502, a BIM is retrieved. In this step, the BIM is downloaded from a system administration server and saved into memory of a user device. At step1503, an image is captured from a set of cameras. At step1504, a position and an orientation of the user device is determined, as will be further described below. At step1505, a stereoscopic overlay of the BIM is rendered according to the position and the orientation of the user device, as will be further described below. At step1506, the rendered stereoscopic overlay is output to a display unit of the user device for display to the user. In a preferred embodiment, the rendered stereoscopic overlay is rendered at least 24 fps.

In one embodiment, a set of data is retrieved at step1507. In this step, the set of data is downloaded from the system administrator and saved into the memory of the user device. In one embodiment, the position and the orientation function is deactivated. In another embodiment, the position and the orientation function remain activated. At step1508, a subset of the set of data is selected for display including the size and the position of the selected set of data. At step1509, the selected subset of data is displayed on the display unit.

At step1510, a determination is made as to whether an end command has been received. If not, runtime process returns to step1503. If so, runtime process1500ends at step1511.

Referring toFIG. 16, step1504will be further described as method1600for determining a position and an orientation of a user device. Method1600begins at step1601. At step1602, a set of registration markers is identified and decoded to determine the position of the user device. In a preferred embodiment, method1400is employed. At step1603, a set of motion detection data is received from a set of sensors in the user device to determine movement of the user device. At step1604, the set of motion detection data and the position of the user device are combined to determine an x, y, z position of the user device in reality and in the BIM and a roll, pitch, and yaw or detection of the user device in reality and the BIM. In this step, the user device determines which camera captured the image of the registration marker, i.e., a temporal camera or a camera of the camera matrix. If the camera of the camera matrix captures the images, then a difference angle is calculated between an axis of the camera of the camera matrix and an axis of the temporal camera. The orientation is calculated from the set of position angles and the difference angles. The set of motion detection data received is the roll, pitch, and yaw orientation movement of the head of the user. Method1600ends at step1605.

Referring toFIG. 17, step1505will be further described as method1700for rendering a stereoscopic overlay according to the position and the orientation of the user device for a user device. Method1700begins at step1701. At step1702, a BIM is rotated and magnified based on the position and the orientation of the user device. At step1703, the BIM is “clipped” based on a set of barriers in the BIM, i.e., the nearest set of walls. For example, if the user is standing the middle of a room, the BIM is “clipped” to only show the room of the BIM in which the user is standing. Otherwise, the entire BIM of the entire building would be shown to the user. At step1704, a layer selection of the BIM is determined from the command menu. At step1705, the selected layers of the “clipped” BIM is rendered as a stereoscopic image, i.e., the BIM image is rendered as a pair of BIM images, a left BIM image for a left display unit and a right BIM image for a right display unit of the user device. Method1700ends at step1706.

In a preferred embodiment, the left BIM image and the right BIM image are shifted with respect to each other, in a range of approximately 2.5 to 3 inches to compensate for the average distance between the pupils of human eyes.

Referring toFIG. 18, method1800for updating a BIM will now be described. Method1800begins at step1801. At step1802, a virtual location of a virtual object in the BIM is determined by viewing the virtual location on a display unit of a user device. At step1803, an actual location of a real object associated with the virtual object is determined. At step1804, a tolerance for the real object location is determined by any measuring means. In a preferred embodiment, the tolerance is determined by a set of building codes. At step1805, the actual location is compared to the virtual location to determine whether the actual location is within the tolerance. If so, then method1800ends at step1809. If the actual location is not within the tolerance, then method1800proceeds to step1806. At step1806, an image is captured of the actual location and the virtual location as seen through the display by the user. At step1807, the captured image is uploaded to a system administrator server. At step1808, the captured image is saved in the BIM as a “mistakes” layer. The “mistakes” layer is then a selectable layer in the BIM once a user reloads the BIM to the user device from the system administrator server. Method1800ends at step1809.

Referring toFIG. 19in another embodiment, method1900for updating a BIM will now be described. Method1900begins at step1901. At step1902, a streaming session between a user device and a system administrator server is initiated and a video is captured and streamed in real time to the system administrator server. The video includes the point of view of the user captured by a camera with the overlaid BIM. At step1903, a virtual location of a virtual object in the BIM is determined by viewing the virtual location on the display. At step1904, an actual location of a real object associated with the virtual object is determined. At step1905, a tolerance for the real object location is determined by any measuring means. In a preferred embodiment, the tolerance is determined by a set of building codes. At step1906, the actual location is compared to the virtual location to determine whether the actual location is within the tolerance. If so, then method1900ends at step1908. If the actual location is not within the tolerance, then method1900proceeds to step1907. At step1907, the video is saved in the BIM in a “mistakes” layer as a selectable element, such as an icon or link. The “mistakes” layer is then a selectable layer in the BIM once a user reloads the BIM to a wearable computer from the system administrator server. The user selects the selectable element to stream and view the video. Method1900ends at step1908.

It will be appreciated by those skilled in the art that the described embodiments disclose significantly more than an abstract idea including technical advancements in the field of data processing and a transformation of data which is directly related to real world objects and situations in that the disclosed embodiments enable a computer to operate more efficiently and make improvements to construction management technology. Specifically, the disclosed embodiments eliminate the remanufacture of construction components and rescheduling of equipment. Further, the disclosed embodiments eliminate the reliance and use of external positioning systems, such as GPS or laser-based systems.

It will be appreciated by those skilled in the art that changes could be made to the embodiments described above without departing from the broad inventive concept. It is understood, therefore, that this disclosure is not limited to the particular embodiments herein, but it is intended to cover modifications within the spirit and scope of the present disclosure as defined by the appended claims.