Modifiable simulation of physical object behavior

A computer device is provided that includes an input device, a sensor device, a display device, and a processor. The processor is configured to detect a physical object in a physical environment based on sensor data received via the sensor device, measure one or more physical parameters of the physical object based on the sensor data, determine a physical behavior of the physical object based on the measured one or more physical parameters, present a graphical representation of the physical behavior of the physical object via the display device, generate a simulation of the physical behavior of the physical object based on the measured one or more physical parameters, receive a user input to modify the one or more physical parameters for the simulation via the input device, and present the simulation with the modified one or more physical parameters via the display device.

BACKGROUND

Current education systems utilize textbooks and two dimensional visuals on screens to convey information about the world. However, these systems are inherently separate from the real-world and are constrained to the static scenarios of the textbooks. As a result, it may be difficult for students learning using those textbook scenarios to apply that knowledge to the real-world.

SUMMARY

A computer device is provided that may include an input device, a sensor device, a display device, and a processor. The processor may be configured to detect a physical object in a physical environment based on sensor data received via the sensor device, measure one or more physical parameters of the physical object based on the sensor data, determine a physical behavior of the physical object based on the measured one or more physical parameters, present a graphical representation of the physical behavior of the physical object via the display device, generate a simulation of the physical behavior of the physical object based on the measured one or more physical parameters, receive a user input to modify the one or more physical parameters for the simulation via the input device, and present the simulation with the modified one or more physical parameters via the display device.

DETAILED DESCRIPTION

In order to address the challenges discussed above, a computer device10is provided, as shown in the example embodiment ofFIG. 1. The computer device10may include a display12, volatile memory14, non-volatile memory16, a processor18, input devices20, and sensor devices22. The input devices20may include one or more input devices, such as, for example, a keyboard, a mouse, a trackpad, a touchscreen, a microphone, a camera, and/or some other input device20. In addition to the display12, the computer device10may further include one or more other output devices, such as a speaker, a haptic feedback unit, or another type of output device. The sensor devices22may include one or more sensor devices, such as, for example, a camera22A such as an RGB camera, a microphone22B or another type of sound sensor, a depth sensor22C such as a depth camera, and other types of sensor devices such as an inertial motion unit, a global positioning system (GPS) unit, etc.

The computer device10may take the form of a head mounted display (HMD) device, a mobile computer device such as a smartphone, a laptop computer device, a tablet computer device, or another suitable type of computer device.FIG. 2illustrates two example forms of the computer device10. In one example illustrated inFIG. 2, the computer device10takes the form of a mobile computer device10A, which, for example, may be a smartphone or tablet computer device. The mobile computer device10A may include a capacitive touch screen12A, which includes both the display12and a capacitive touch sensor input device of the one or more inputs devices20. The mobile computer device10A may include other types of input devices20, such as a microphone input device20A. As illustrated, the mobile computer device10A may also include a camera22A sensor device. It should be appreciated that the mobile computer device10A may further include each computer component of computer device10described herein.

In another example illustrated inFIG. 2, the computer device10takes the form of an HMD device10B, which may be worn by a user according to an example of the present disclosure. The HMD device10B includes a near-eye display device12B. The near-eye display device12B may take the form of an at least partially see-through display that is supported in front of a viewer's eye or eyes in an augmented reality HMD device configuration. In another example, the near-eye display device12B may take the form of a non-see-through display in a virtual reality HMD device configuration.

In the example ofFIG. 2, the HMD device10B includes a frame24that wraps around the head of a user to position the near-eye display device12B close to the user's eyes. The frame24supports additional components of the HMD device10B, such as, for example, the volatile memory14, the non-volatile memory16, the processor18, input devices20, sensor devices22, and other computer components of the computer device10described herein. The processor18includes logic and associated computer memory configured to provide image signals to the near-eye display device12B, to receive sensor data from sensor devices22, and to enact various control processes described herein.

Any suitable display technology and configuration may be used to display images via the near-eye display device12B. For example, in a non-augmented reality configuration, the near-eye display device12B may be a non-see-through Light-Emitting Diode (LED) display, a Liquid Crystal Display (LCD), or any other suitable type of non-see-through display. In an augmented reality configuration, the near-eye display device12B may be configured to enable a wearer of the HMD device10to view a physical, real-world object in the physical environment through one or more partially transparent pixels displaying virtual object representations. For example, the near-eye display device12B may include image-producing elements such as, for example, a see-through Organic Light-Emitting Diode (OLED) display.

As another example, the HMD device10B may include a light modulator on an edge of the near-eye display device12B. In this example, the near-eye display device12B may serve as a light guide for delivering light from the light modulator to the eyes of a wearer. In other examples, the near-eye display device12B may utilize a liquid crystal on silicon (LCOS) display.

The sensor devices22may include various sensors and related systems to provide information to the processor18. Such sensors may include, but are not limited to, an inertial measurement unit (IMU)22D, one or more outward facing cameras22E, and one or more inward facing cameras22F. The one or more inward facing cameras22F may be configured to acquire image data in the form of gaze tracking data from a wearer's eyes.

The processor18may execute instructions to determine gaze directions of each of a wearer's eyes in any suitable manner based on the information received from the inward facing cameras22F. For example, one or more light sources, such as infrared light sources, may be configured to cause a glint of light to reflect from the cornea of each eye of a wearer. The one or more inward facing cameras22F may be configured to capture an image of the wearer's eyes. Images of the glints and of the pupils as determined from image data gathered from the image sensors may be used to determine an optical axis of each eye. Using this information, the processor18may execute instructions to determine a direction in which the wearer is gazing.

In other implementations, a different type of gaze sensor may be employed in the HMD device10B to measure one or more gaze parameters of the user's eyes. Examples of gaze parameters measured by one or more gaze sensors may include an eye gaze direction or gaze vector, head orientation, eye gaze velocity, eye gaze acceleration, change in angle of eye gaze direction, and/or any other suitable tracking information.

The one or more outward facing cameras22E may be configured to capture and/or measure physical environment attributes of the physical environment in which the HMD device10B is located. In one example, outward facing cameras22E may include a visible-light camera configured to collect a visible-light image of a physical space. Further, the one or more outward facing cameras22E may include a depth sensor22C such as a depth camera configured to collect a depth image of a physical space. More particularly, in one example the depth camera is an infrared time-of-flight depth camera. In another example, the depth camera is an infrared structured light depth camera.

Data from the outward facing camera22E may be used by the processor18to generate and/or update a three-dimensional (3D) model of the physical space. Data from the outward facing camera22E may be used by the processor18to identify surfaces of the physical space and/or measure one or more surface parameters of the physical space. The processor18may execute instructions to generate/update virtual scenes displayed on near-eye display device12B and identify surfaces of the physical space in any suitable manner. In one example, depth maps derived from depth data provided by the depth camera of camera22E may be used to accurately position and determined occlusion for virtual text displayed on the near-eye display device12B. In virtual reality configurations of the HMD device10B, image data captured by the outward facing cameras22E may be passed through and displayed on the near-eye display12B, with additional visual content superimposed on the passed through image data by the processor18.

In augmented reality configurations of HMD device10B, the position and/or orientation of the HMD device10B relative to the physical environment may be assessed so that augmented-reality images may be accurately displayed in desired real-world locations with desired orientations. As noted above, the processor18may execute instructions to generate a 3D model of the physical environment including surface reconstruction information and simultaneous localization and mapping (SLAM) that may be used to identify surfaces in the physical space to localize the HMD device10B and holograms and/or images displayed on the near-eye display12B.

In both augmented reality and non-augmented reality configurations of HMD device10B, the IMU22D of HMD device10B may be configured to provide position and/or orientation data of the HMD device10B to the processor18. In one implementation, the IMU22D may be configured as a three-axis or three-degree of freedom (3DOF) position sensor system. This example position sensor system may, for example, include three gyroscopes to indicate or measure a change in orientation of the HMD device10B within 3D space about three orthogonal axes (e.g., roll, pitch, and yaw). The orientation derived from the sensor signals of the IMU may be used to display, via the near-eye display device12B, one or more holographic images with a realistic and stable position and orientation.

In another example, the IMU22D may be configured as a six-axis or six-degree of freedom (6DOF) position sensor system. Such a configuration may include three accelerometers and three gyroscopes to indicate or measure a change in location of the HMD device10B along three orthogonal spatial axes (e.g., x, y, and z) and a change in device orientation about three orthogonal rotation axes (e.g., yaw, pitch, and roll). In some implementations, position and orientation data from the outward facing camera22E and the IMU22D may be used in conjunction to determine a position and orientation (or 6DOF pose) of the HMD device10B.

In some examples, a 6DOF position sensor system may be used to display holographic representations in a world-locked manner. A world-locked holographic representation appears to be fixed relative to one or more real world objects viewable through the HMD device10B, thereby enabling a wearer of the HMD device10B to move around a real world physical environment while perceiving a world-locked hologram as remaining stationary in a fixed location and orientation relative to the one or more real world objects in the physical environment.

Turning back toFIG. 1, the sensor devices22of the computer device10are configured to capture a stream of sensor data26that may be processed by a data analysis module28executed by the processor18. The data analysis module28may be configured to process the sensor data26using trained model data30that may be retrieved from a server system32and/or memory of the computer device10. For example, the computer device10may be configured to communicate with the server system32via a network, such as a wide area network, or another suitable type of network. The trained model data30may include one or more different types of trained models such as a physical model30A, a natural object model30B, etc. As a few other non-limiting examples, the trained model data30may include a chemistry model, a dynamic physics model, a static physics model, a geology model, a meteorology model, etc. Each of the trained models30may be downloaded separately by a user of the computer device10to selectively choose a learning focus for the computer device10.

The trained model data30may include an object recognition component. As one example of such an object recognition component, the trained object model may include a convolutional neural network trained on an image data set in which images have been semantically tagged by users with words (typically nouns) that represent the objects in the image. One example dataset that may be used for the object model is IMAGENET. As a specific example, the object recognition component may be trained to recognize physical three-dimensional models using two-dimensional image classification techniques. For example, using a database of three-dimensional models, the trained model data30may include a plurality of two-dimensional training images of each of the three-dimensional models at various angles, lighting conditions, realistic background, different colors, different materials, etc. Images taken by the camera sensors of the computer device may then be compared to these two-dimensional training images of the trained model data30to recognize the physical object.

Using the trained model data30, the data analysis module28executed by the processor18is configured to detect a physical object34in a physical environment36based on sensor data26received via the sensor devices22. For example, a camera22A sensor device and a depth sensor device22C may capture images of the physical environment36. The images of the physical environment36may be sent to the data analysis module28in a stream of sensor data26. The data analysis module28may be configured to process the captured images of the physical environment36to perform surface reconstruction, edge detection, centroid detection, object recognition, and other machine vision processing methods to detect one or more physical objects34. The types of physical objects34detected by the data analysis module28may include structures, movable objects, natural objects, and other types of objects. As a few non-limiting examples, structure objects may include buildings, bridges, and other structures mounted immovably to the physical environment36. The movable objects may include man-made objects that are movable, such as a rock, a ball, a car, etc. Natural objects may include animals, birds, plants, people, rocks, mountains, clouds, etc. It should be appreciated that the examples of structures, movable objects, and natural objects described above are merely exemplary, and that the data analysis module28may be configured to detect other types of objects based on the trained module data30.

After detecting the physical object34, the data analysis module28may be further configured to identify the physical object34by processing the sensor data26using trained model data30. The trained model data30may also include semantic classification data38for the types of physical objects34included in the trained model data30. Based on the sensor data26, the data analysis module28may the retrieve semantic classification data38associated with the physical object34, and tag the identified physical object34with the semantic classification data38. For example, using trained model data30of a natural object model30B, the data analysis module28may be configured to detect a flying object in the images captured by the sensor devices22, and further identify that the flying object is an eagle based on features such as beak shape, wing shape, size, etc., used to train the natural object model30B. Thus, the physical object34may be tagged with semantic classification data38of an eagle. It should be appreciated that identifiable physical objects34are not limited to animals, but may further include building and bridge classifications such as a specific historical building or bridge, a specific architectural design, etc. As another example, identifiable physical objects34may further include geology classifications, such as a type or composition of rocks and minerals.

As illustrated inFIG. 1, the data analysis module28is further configured to measure one or more physical parameters40of the physical object34based on the sensor data26. For example, the data analysis module28may be configured to measure a velocity, position, heading, mass, and volume of the physical object34. It should be appreciated that the physical parameters40being measured may be based on the type of physical object34identified by the data analysis module28. That is, the velocity, position, heading, mass, volume, drag, trajectory, and other physical parameters that affect movement through the physical environment may be measured for physical objects that are identified as movable objects. As another example, physical objects that are identified as structure objects may have measured physical parameters40that include physical parameters such as, for example, mass, volume, shear force, friction, and load on the structure object. As another example, the one or more physical parameters40may include parameters of the physical environment36that affect the physical object34, such as, for example, gravitational force, wind speed, humidity, elevation, etc.

These physical parameters40may be measured based on sensor data26received from a plurality of different sensor devices22. For example, velocity, position, heading, and volume parameters may be calculated based on a series of images captured by camera sensor devices22A and depth sensor devices22C. Other physical parameters40may have known values based on a location of the computer device10detected via a GPS sensor device, such as, for example, a gravitational force, elevation, location, etc. Values for physical parameters of the physical environment36that are not static may be gathered by the sensor devices22and/or retrieved from sensor data stored on the server system32, such as, for example, weather data including wind speed, humidity, etc.

Other physical parameters40such as mass, load, friction, drag, etc., may be estimated by the data analysis module28based on known values for physical objects34that have been identified as described above. For example, the data analysis module28may be configured to calculate the load placed on a bridge by detecting each car on the bridge via image analysis of images captured by the camera sensor devices22A, identifying the cars and retrieving semantic classification data for the identified cars such as a specific type of car, or a broad classification of vehicles such as truck, sedan, SUV, train, etc. The data analysis module28may estimate the weight of the identified vehicle physical objects based on a known average weight of vehicles for that semantic classification of vehicle. By estimating the total weight of each vehicle on a bridge in this manner, the data analysis module28may estimate a total load being placed on the bridge by the vehicles.

It should be appreciated that the examples of physical parameters40and processes for measuring those physical parameters based on sensor data26described above are merely exemplary, and that other types of physical parameters40may be measured based on other types of sensor data26not specifically described above.

After measuring one or more physical parameters40for the detected physical object34, the data analysis module28may be configured to determine a physical behavior42of the physical object34based on the measured one or more physical parameters40. Example types of physical behaviors42may include a path of travel of a movable object. That is, based on measured physical parameters40of a movable physical object such as an initial velocity, trajectory, gravitational force, wind speed, drag, etc., the data analysis module28may determine a path of travel for the movable object that predicts how the movable object will move through the physical environment36. The data analysis module28may be configured to determine a mathematic expression that best fits the physical behavior42of the detected physical object34using symbolic regression techniques. For example, the data analysis module28may search a space of mathematical expressions defined in the trained model data30to find the mathematical expression that best fits the measured one or more physical parameters40to describe the physical behavior42of the physical object34.

As another example, the physical behaviors42may include deformation and/or shear of the physical object34that may be determined based on a material composition of the physical object34and an estimated load physical parameter being placed on the physical object34. As another example, the physical behaviors42may include an oscillation of a pendulum physical object that may be determined based on a length and an amplitude physical parameter measured for the pendulum physical object. It should be appreciated that the example physical behaviors42described above are merely exemplary, and that other types of physical behaviors42may be determined based on any suitable type of measurable physical parameters40. In one example, the physical behaviors42may be determined and modeled by the processor18using a physics engine that is configured to simulate rigid body mechanics, fluid dynamics, etc. As a specific example, the processor18may use the one or more measured physical parameters40as input to a HAVOK physics engine that may model the physical behaviors42of the physical object34, and output a result of the physics simulation to the data analysis module28.

As illustrated inFIG. 1, a simulation module44executed by the processor18is configured to generate a graphical representation46of the determined physical behavior42of the physical object34. In one example, the graphical representation46may include mathematical functions that describe the physical behavior42as well as the physical parameters40that affect those mathematical functions. The graphical representation46may be generated for a visual format suitable for the type of display12of the computer device. For a mobile computer device10A that includes a two-dimensional display, the graphical representation46may be rendered to a two-dimensional viewport. In one example, the graphical representation46may be rendered to be superimposed on images of the physical environment36captured by the camera sensor devices22A. In this example, the processor18may be configured to present the graphical representation46of the physical behavior42of the physical object34superimposed on the physical environment36via the display12of the computer device10. In one example, the graphical representation46is rendered in a graphical user interface 48 layer that is rendered on top of images of the physical environment36captured by the camera sensor devices22A. In another example, the graphical representation46may be generated as a virtual object having a location in the physical environment36and rendered to the two dimensional viewport of the display12based on its virtual depth and location in the physical environment36.

Similarly, in a virtual reality HMD device10B example that includes a non-see-through near-eye display device12B, the graphical representation46may similarly be rendered to be superimposed on images of the physical environment36captured by the outward facing cameras22E. Further, the graphical representation46may be generated as a virtual object having a location in the 3D mapping of the physical environment36and rendered from the user's current perspective determined based on the sensor devices22including the user's detected gaze direction, pose, location, and position relative to surfaces identified in the physical environment36.

In an augmented reality HMD device10B example that includes an at least partially see-through display12B, the graphical representation46may be generated as a virtual three-dimensional hologram having a location in the 3D mapping of the physical environment36and rendered from the user's current perspective determined based on the sensor devices22including the user's detected gaze direction, pose, location, and position relative to surfaces identified in the physical environment36. The graphical representation46is rendered as a three-dimensional hologram that is projected onto the user's eye, such that the graphical representation appears to be positioned at the world-locked location and depth in the physical environment while the user is viewing the physical environment36through the at-least partially see-through display12B.

The simulation module44executed by the processor18is further configured to generate a simulation50of the physical behavior42of the physical object34based on the measured one or more physical parameters40and the sensor data26. For example, based on the images captured by the camera sensor device22A, the simulation module44may generate a virtual object or hologram with the appearance of the physical object34. The simulation50may render the virtual object or hologram of the physical object34to follow the determined physical behavior42, such as, for example, a virtual ball following a determined path of travel. The simulation50may simulate all of the physical parameters40that were measured for the physical object34and the surrounding physical environment36to accurately simulate the physical behavior42of the physical object34in real-time. For example, the simulation50may simulate the path of travel of a ball that has been thrown based on the measured velocity, trajectory, gravitational forces, wind speed, drag, and other physical parameters40measured for the physical object34and physical environment36. The simulation50may be presented to the user via the display12superimposed on the physical environment36.

After the simulation50has been generated, the user may enter user input to the input devices20to modify the one or more physical parameters40for the simulation50. For example, the user may enter input via the GUI48displayed to the user to change one or more of the physical parameters40measured for the physical object34and the physical environment36. As a specific example, the user may modify a velocity of a thrown object, and/or a gravitational force of the physical environment36to learn how those physical parameters40affect the path of travel physical behavior42of the thrown ball physical object34.

After receiving the use input to modify the one or more physical parameters52, the simulation module44determines a modified physical behavior42based on the modifications to the one or more physical parameters52. For example, the simulation module44may determine an updated path of travel for a thrown ball physical object34based on receiving a user input to modify a gravitational force physical parameter40. After modifying the simulation50, the processor18may be configured to present the simulation50with the modified one or more physical parameters40via the display device12. In the augmented or virtual reality HMD device10B example, the simulation50with the modified one or more physical parameters40may be presented via the near-eye display device12B superimposed on the physical environment36.

FIG. 3illustrates an example graphical representation of a physical behavior of a thrown physical object and an example simulation of the physical behavior. In this example, a user54is wearing an augmented reality HMD device10B and watching a baseball game occurring the physical environment36. The user's HMD device10B may have the physics trained model30A downloaded, and thus the HMD device10B may be processing the sensor data26received from the sensor devices22including the outward facing cameras22E for physical objects34having physical behaviors42that are identifiable in the physics trained model30A. In the illustrated example, the data analysis module28executed by the processor18of the HMD device10B processes image data from the outward facing cameras22E and detects a movable physical object34A, which is a thrown baseball in this example, that is currently moving through the physical environment36.

As discussed above, the data analysis module28may further measure one or more physical parameters40of the movable physical objected34A, such as velocity, trajectory, position, gravitational force, etc. Based on the measured one or more physical parameters40, the data analysis module28may determine a physical behavior42of the movable physical object34A, which, in this specific example, is a predicted path of travel for the movable physical object34A. It should be appreciated that the one or more physical parameters40and the physical behavior of the physical object34may be calculated in real-time. Thus, as the movable physical object34A is still traveling, the HMD device12B displays a graphical representation46of the physical behavior42to the user54. In the illustrated example, the graphical representation46is a virtual object that shows the quadratic question for the movable physical object's path of travel including the one or more physical parameters40that are variables in the quadratic equation.

The simulation module44is configured to generate a simulation50, which, in the illustrated example, includes a virtual movable object56that is rendered to have the appearance of the detected movable physical object34A that is a baseball. The virtual movable object56is rendered to travel along the predicted path of travel physical behavior42that was determined for the detected movable physical object34A. As shown, the simulation50may be presented to the user in real-time as the movable physical object34A is still traveling.

As discussed above, the user54may enter user input to modify one or more measured physical parameters for the simulation50. In one example, the user54may enter the user input via a gesture input detected via the outward facing cameras22E. However, it should be appreciated that the user input may be entered through any suitable input modality, such as, for example, user input to a handheld input device, a voice input to a microphone sensor device22B, etc. The user54may enter user input to modify any of the one or more physical parameters that were measured and used for the simulation50of the physical behavior42of the movable physical object34A.

FIG. 4illustrates an example where the user54has entered a user input to modify a spin physical parameter of the movable physical object34A. The data analysis module28may be configured to calculate a lift force that would be applied to the movable physical object34A from the magnus effect based on the modified spin physical parameter, and calculate a modified physical behavior of a path of travel that the movable physical object34A would travel along if the modified spin physical parameter has been applied in the real-world. The simulation module44may modify the simulation50based on the modified physical behavior42and the modified one or more physical parameters40, and render a new modified simulation50A. In the illustrated example, the modified simulation50A includes a rendering of virtual movable object56traveling along a modified path of travel that accounts for an additional lift force due to the modified spin parameter entered by the user54. As illustrated, the modified simulation50A may be rendered as being location at specified positions and depths in the physical environment36.

FIG. 5illustrates a mobile computer device10A example where the computer device10takes the form of a smart phone or tablet computer device. In this example, images of the physical environment36are captured by the camera22A, and processed by the data analysis module28as described above. The simulation module44generates a simulation50of the path of travel physical behavior42of the movable physical object34A. The simulation50is presented via the touch display12A superimposed on the captured images of the physical environment36. Additionally, a graphical representation46of the quadratic equation describing the path of travel of the movable physical object34A is also presented via the touch display12A. In this example, the user may enter input to the touch display12A to change a time physical parameter40. The simulation module44may modify the simulation50A to display a virtual object56that represents the movable physical object34A at different points in time selectable by the user by modifying the time physical parameter. The graphical representation46may present the values of each measured physical parameter40in the quadratic equation that describes the path of travel physical behavior at each point in time T0-T3.

FIG. 6illustrates an example modified simulation50A where the user has entered a user input to modify a drag physical parameter40. For example, the user may have increased a wind speed of the physical environment36against the movable physical object34A. The data analysis module28determines a modified physical behavior42for the physical object34, which, in this example, is a modified path of travel that would cause the movable physical object34A to decelerate more and travel a shorter distance with a lower arc than the movable physical object34A did in the real world when it was captured by the sensor devices of the computer device10. The modified simulation50A and the graphical representation of the modified path of travel are presented to the user via the touch display12A of the mobile computer device10A.

FIG. 7illustrates an example modified simulation50A where the user has entered a user input to modify a gravitational force physical parameter40. For example, the user may have changed the gravitational force physical parameter40of the physical environment36to a gravitational force of the moon. The data analysis module28determines a modified physical behavior42for the physical object34, which, in this example, is a modified path of travel that would cause the movable physical object34A to have a higher arc and travel further than the movable physical object34A did in the real world when it was captured by the sensor devices of the computer device10. The modified simulation50A and the graphical representation of the modified path of travel are presented to the user via the touch display12A of the mobile computer device10A. It should be appreciated that the example modifications to physical parameters40illustrated inFIGS. 4-7are merely exemplary, and that other suitable physical parameters40may also be modified by the user, and a corresponding modified simulation50A may be generated and displayed to the user.

FIG. 8illustrates an example where the physical object34is a structure physical object that is a bridge. The data analysis module28may be configured to process the images captured by the outward facing cameras22E of the HMD device10B and detect the physical object34B. Based on bridge and structure trained model data, the data analysis module28may detect the bridge physical object34B in the captured images. Further, the data analysis module28may identify the bridge physical object34B as the Golden Gate Bridge based on the trained model data, and retrieve associated semantic classification data for the Golden Gate Bridge. The HMD device10B may be configured to present the semantic classification data38via the display device12. In the illustrated example, the Golden Gate Bridge semantic classification data38is presented to the user superimposed on the physical environment36.

The data analysis module28may be configured to measure one or more physical parameters40of the bridge physical object34B. In one example, the measured physical parameters40include a load placed on the bridge by cars. As discussed previously, the load may be estimated based on identifying car physical objects in the images captured by the camera sensor devices22A, and retrieving estimated weights for the identified car physical objects. The data analysis module28may calculate a physical behavior42of the bridge physical object34A based on measured one or more physical parameters40. In the illustrated example, the physical behavior42may include equations for the tension on each suspension cable of the bridge physical object34B as the load on the bridge changes from traveling cars.

The user54may enter input to modify the one or more physical parameters40for a simulation50, such as, for example, changing a weight of a car physical object on the bridge to modify the load being placed on the bridge physical object34B. The effect of the modified load physical parameter40on the equations for tension of the suspension cables may be presented to the user54via the display12in a modified simulation.

It should be appreciated that other types of physical behaviors and physical parameters may be measured and modified in the example illustrated inFIG. 8. For example, the one or more physical parameters may include wind speeds or forces applied to the bridge physical object34B by an earthquake, and the physical behavior42may be a vibration or oscillation of the bridge physical object34B while being subjected to those forces.

Turning toFIG. 9, in one example, the computer device10may be configured to share the simulation50with other computer devices. In the illustrated example, the user54is wearing the HMD device10B, and has already captured images of a movable physical object34A as described with reference toFIG. 3. The user54's HMD device10B has generated a simulation50based on the measured physical parameters40and the determined physical behavior42of the movable physical object34A. As described previously, the user54may modify the simulation50by entering user input to modify the one or more physical parameters40. Further, the user54may enter an input to the GUI48to share the simulation50with another user.

In this example, the user54's HMD device10B is configured to send the simulation50to one or more other computer devices to cause the one or more computer devices to display the simulation50from a perspective of the one or more other computer devices. In the illustrated example, the HMD device10B sends data for the simulation50to the mobile computer device10A of a different user. The simulation50may include surface reconstruction and localization data, such that the mobile computer device10A may appropriately world-lock the virtual object56of the simulation50to the positions specified by the HMD device10B. Additionally, the renderings of the simulation50may be superimposed on images captured by the camera22A of the mobile computer device such that the virtual object56of the simulation50is rendered from the perspective of the mobile computer device10A. It should be appreciated that while the illustrated example shows the HMD device10B sharing the simulation50with a single mobile computer device10A, that the HMD device10B may share the simulation with a plurality of other computer devices taking other forms, such as, for example, other HMD devices. Further, in some examples, the mobile computer device10A may generate and send the simulation50to the HMD device10B, or another computer device10.

FIG. 10shows a flowchart of a computer-implemented method100. The method100may be implemented by the computer device10ofFIG. 1. At102, the method100may include detecting a physical object in a physical environment based on sensor data received via a sensor device of the computer device. For example, the computer device implementing the method100may include a camera sensor device as illustrated inFIG. 1. Images captured images by the camera sensor devices may be processed with surface reconstruction, edge detection, centroid detection, and other machine vision processing methods to detect one or more physical objects. The types of physical objects detected at step102may include structures, movable objects, natural objects, and other types of objects. As a few non-limiting examples, structure objects may include buildings, bridges, and other structures mounted immovably to the physical environment36. The movable objects may include man-made objects that are movable, such as a rock, a ball, a car, etc. Natural objects may include animals, birds, plants, people, rocks, mountains, clouds, etc.

At104, the method100may include measuring one or more physical parameters of the physical object based on the sensor data. The physical parameters may be measured based on sensor data received from a plurality of different sensor devices of the computer device implementing the method100, such as, for example, the computer device10ofFIG. 1. For example, velocity, position, heading, and volume parameters may be calculated based on a series of images captured by camera sensor devices22A and depth sensor devices22C. Other methods and processes for measuring physical parameters based on sensor data received via sensor devices are described above with reference toFIG. 1.

At106, the method100may include determining a physical behavior of the physical object based on the measured one or more physical parameters. An example physical behavior42may include a path of travel of a movable object that may be determined based on measured physical parameters40such as an initial velocity, trajectory, gravitational force, wind speed, drag, etc. Other types of physical behaviors42described above may include deformation, vibration, oscillation, and shear. It should be appreciated that other types of physical behaviors may be determined based on other types of physical parameters not specifically described herein.

At108, the method100may include presenting a graphical representation of the physical behavior of the physical object via a display device of the computer device. In one example, the graphical representation may be rendered in a graphical user interface layer that is rendered on top of images of the physical environment captured by the camera sensor devices of the computer device.FIG. 3illustrates an example where the graphical representation is generated as a virtual object having a location in the physical environment and rendered as a hologram that is superimposed on the physical environment. An augmented reality HMD device includes an at least partially see-through display12B, and the graphical representation is rendered from the user's current perspective determined based on the sensor devices of the HMD device including the user's detected gaze direction, pose, location, and position relative to surfaces identified in the physical environment. Other rendering methods may be utilized for other types of displays, such as non-see-through displays.

At110, the method100may include processing the sensor data using trained models to identify the physical object. In the example illustrated inFIG. 1, the computer device may utilize trained model data30that may be retrieved from a server system32and/or memory of the computer device10. The trained model data30may include one or more different types of trained models such as a physical model30A, a natural object model30B, etc. As a few other non-limiting examples, the trained model data30may include a chemistry model, a dynamic physics model, a static physics model, a geology model, a meteorology model, etc. Each of the trained models30may be downloaded separately by a user of the computer device10to selectively choose a learning focus for the computer device10.

At112, the method100may include retrieving semantic classification data associated with the physical object. In one example illustrated described with reference toFIG. 1, the computer device may be configured to detect a flying object in the images captured by the sensor devices22based on natural object trained model data, and further identify that the flying object is an eagle based on features such as beak shape, wing shape, size, etc., used to train the natural object model30B. After identification, the physical object34may be tagged with semantic classification data38of an eagle. It should be appreciated that identifiable physical objects34are not limited to animals, but may further include building and bridge classifications such as a specific historical building or bridge, a specific architectural design, etc. As another example, identifiable physical objects34may further include geology classifications, such as a type or composition of rocks and minerals.

At114, the method100may include presenting the semantic classification data via the display device. Similarly to step108, the semantic classification data may be rendered in a GUI layer on top of captured images of the physical environment, or as a virtual object having a virtual position in a 3D mapping of the physical environment and rendered based on the user's perspective. In another example, the semantic classification data may be output to the user via other output methods, such as, for example, via a speaker output device.

At116, the method100may include generating a simulation of the physical behavior of the physical object based on the measured one or more physical parameters. In the example described with reference toFIGS. 3 and 4, the computer device may generate a virtual object or hologram with the appearance of the physical object detected at step102. Rendering the simulation may include rendering the virtual object or hologram of the physical object to follow the determined physical behavior42, such as, for example, a virtual ball following a determined path of travel. The simulation may simulate all of the physical parameters that were measured for the physical object at step104in real-time.

At118, the method100may include receiving a user input to modify the one or more physical parameters for the simulation via an input device of the computer device. The user input may be received via any suitable input modality. Mobile computer device10A examples of the computer device may be configured to receive the user input via touch input to a touch screen. HMD device10B examples of the computer device may be configured to receive the user input via gestures detected based on forward facing cameras and depth sensors. The user may modify any of the physical parameters40measured at step104. The computer device is configured to modify the simulation to account for the modified physical parameters40. At120, the method100may include presenting the simulation with the modified one or more physical parameters via the display device. The simulation may be presented via the same rendering methods and display methods described above at steps108and114.

At122, the method100may include sending the simulation to one or more other computer devices to cause the one or more computer devices to display the simulation from a perspective of the one or more other computer devices.FIG. 9illustrates an example simulation sharing between an HMD device10B and a mobile computer device10A. The simulation shared by the HMD device10B may include surface reconstruction and localization data, such that the mobile computer device10A may appropriately world-lock the virtual object56of the simulation50to the positions specified by the HMD device10B. Additionally, the renderings of the simulation50may be superimposed on images captured by the camera22A of the mobile computer device such that the virtual object56of the simulation50is rendered from the perspective of the mobile computer device10A.

FIG. 11schematically shows a non-limiting embodiment of a computing system200that can enact one or more of the methods and processes described above. Computing system200is shown in simplified form. Computing system200may, for example, embody the computer device10ofFIG. 1, or may instead embody some other computing system. Computing system200may take the form of one or more personal computers, server computers, tablet computers, home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone), and/or other computing devices, and wearable computing devices such as smart wristwatches and head mounted augmented/virtual reality devices.

Computing system200includes a logic processor202, volatile memory204, and a non-volatile storage device206. Computing system200may optionally include a display subsystem208, input subsystem210, communication subsystem212, and/or other components not shown inFIG. 11.

Volatile memory204may include physical devices that include random access memory. Volatile memory204is typically utilized by logic processor202to temporarily store information during processing of software instructions. It will be appreciated that volatile memory204typically does not continue to store instructions when power is cut to the volatile memory204.

Non-volatile storage device206includes one or more physical devices configured to hold instructions executable by the logic processors to implement the methods and processes described herein. When such methods and processes are implemented, the state of non-volatile storage device206may be transformed—e.g., to hold different data.

Aspects of logic processor202, volatile memory204, and non-volatile storage device206may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.

The term “program” may be used to describe an aspect of computing system200implemented to perform a particular function. In some cases, a program may be instantiated via logic processor202executing instructions held by non-volatile storage device206, using portions of volatile memory204. It will be understood that different programs may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same program may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The term “program” encompasses individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

When included, display subsystem208may be used to present a visual representation of data held by non-volatile storage device206. As the herein described methods and processes change the data held by the non-volatile storage device206, and thus transform the state of the non-volatile storage device206, the state of display subsystem208may likewise be transformed to visually represent changes in the underlying data. Display subsystem208may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic processor202, volatile memory204, and/or non-volatile storage device206in a shared enclosure, or such display devices may be peripheral display devices.

When included, communication subsystem212may be configured to communicatively couple computing system200with one or more other computing devices. Communication subsystem212may include wired and/or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem212may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network. In some embodiments, the communication subsystem212may allow computing system200to send and/or receive messages to and/or from other devices via a network such as the Internet.

The following paragraphs provide additional support for the claims of the subject application. One aspect provides a computer device comprising an input device, a sensor device, a display device, and a processor. The processor is configured to detect a physical object in a physical environment based on sensor data received via the sensor device, measure one or more physical parameters of the physical object based on the sensor data, determine a physical behavior of the physical object based on the measured one or more physical parameters, present a graphical representation of the physical behavior of the physical object via the display device, generate a simulation of the physical behavior of the physical object based on the measured one or more physical parameters, receive a user input to modify the one or more physical parameters for the simulation via the input device, and present the simulation with the modified one or more physical parameters via the display device. In this aspect, additionally or alternatively, the physical object may be selected from the group consisting of a structure, a movable object, and a natural object. In this aspect, additionally or alternatively, the one or more physical parameters may be selected from the group consisting of velocity, position, heading, mass, volume, gravitational force, wind speed, drag, shear force, friction, and load. In this aspect, additionally or alternatively, the physical behavior of the physical object may be selected from the group consisting of a path of travel, deformation, vibration, oscillation, and shear. In this aspect, additionally or alternatively, the processor may be further configured to present the graphical representation of the physical behavior of the physical object superimposed on the physical environment. In this aspect, additionally or alternatively, the computer device may be a head mounted display device, and wherein the display may be a near-eye display device. In this aspect, additionally or alternatively, the near-eye display device may be at least partially see-through, and the processor may be further configured to present the simulation with the modified one or more physical parameters via the near-eye display device superimposed on the physical environment. In this aspect, additionally or alternatively, the processor may be further configured to process the sensor data using trained models to identify the physical object, retrieve semantic classification data associated with the physical object, and present the semantic classification data via the display device. In this aspect, additionally or alternatively, the processor may be further configured to send the simulation to one or more other computer devices to cause the one or more computer devices to display the simulation from a perspective of the one or more other computer devices.

Another aspect provides a method comprising, at a computer device including a processor, detecting a physical object in a physical environment based on sensor data received via a sensor device of the computer device, measuring one or more physical parameters of the physical object based on the sensor data, determining a physical behavior of the physical object based on the measured one or more physical parameters, presenting a graphical representation of the physical behavior of the physical object via a display device of the computer device, generating a simulation of the physical behavior of the physical object based on the measured one or more physical parameters, receiving a user input to modify the one or more physical parameters for the simulation via an input device of the computer device, and presenting the simulation with the modified one or more physical parameters via the display device. In this aspect, additionally or alternatively, the physical object may be selected from the group consisting of a structure, a movable object, and a natural object. In this aspect, additionally or alternatively, the one or more physical parameters may be selected from the group consisting of velocity, position, heading, mass, volume, gravitational force, wind speed, drag, shear force, friction, and load. In this aspect, additionally or alternatively, the physical behavior of the physical object may be selected from the group consisting of a path of travel, deformation, vibration, oscillation, and shear. In this aspect, additionally or alternatively, the method may further comprise presenting the graphical representation of the physical behavior of the physical object superimposed on the physical environment. In this aspect, additionally or alternatively, the computer device may be a head mounted display device, and wherein the display device may be a near-eye display device. In this aspect, additionally or alternatively, the near-eye display device may be at least partially see-through, and the method may further comprise presenting the simulation with the modified one or more physical parameters via the near-eye display device superimposed on the physical environment. In this aspect, additionally or alternatively, the method may further comprise processing the sensor data using trained models to identify the physical object, retrieving semantic classification data associated with the physical object, and presenting the semantic classification data via the display device. In this aspect, additionally or alternatively, the method may further comprise sending the simulation to one or more other computer devices to cause the one or more computer devices to display the simulation from a perspective of the one or more other computer devices.

Another aspect provides a head mounted display device comprising an input device, a sensor device, a near-eye display device, and a processor. The processor is configured to detect a physical object in a physical environment based on sensor data received via the sensor device, measure one or more physical parameters of the physical object based on the sensor data, determine a physical behavior of the physical object based on the measured one or more physical parameters, generate a simulation of the physical behavior of the physical object based on the measured one or more physical parameters, receive a user input to modify the one or more physical parameters for the simulation via the input device, and present the simulation with the modified one or more physical parameters via the near-eye display device. In this aspect, additionally or alternatively, the simulation displayed via the near-eye display may be superimposed on the physical environment.