Patent Description:
The current implementation and use of positioning and tracking technologies in <NUM>, UWB, RTLS, LTE and RFID, for example based on the OMLOX standard (https://omlox. com/home) results in numerous use cases around position detection. Cost-intensive tools, pallets with orders, guided or driverless transport systems, warehouse inventories are tagged and can be located with such positioning and tracking technologies.

At present, the located tags (trackables) are displayed graphically on a two-dimensional (2D) system (e.g. a two-dimensional map) or on very basic 3D CAD models. The 2D maps are sufficient as a rough orientation in the terrain.

<CIT> relates to systems and method for enabling two-way interactive operations of real-time 3D virtual replicas and real objects.

<CIT> relates to systems and methods that enable augmented reality, virtual reality, and/or other content to be associated with precise geo-spatial locations in a physical environment.

However, in some prior art systems there is a need to provide a more exact mapping of the position. In addition, some prior art systems have problems with providing sufficient information on the position in indoor environments.

The above-mentioned objects are achieved with the features of the independent claims. Dependent claims define preferred embodiments of the invention.

In particular, the present invention provides a method for generating a three-dimensional, 3D, interaction model for a virtual reality device or an augmented reality device as set out in claim <NUM>.

Various embodiments may preferably implement the following features.

The 3D model of the environment is preferably a 3D model of an indoor environment.

The aforementioned method may preferably be executed on a single device. Alternatively, the method may be executed on several (wireless or wired) connected devices.

Obtaining a three-dimensional, 3D, model of an environment preferably comprises: receiving the 3D model of the environment from an external source or generating the 3D model of the environment based on received raw data.

The 3D model of the environment is preferably based on 3D modelling, CAD data, or a 3D scan.

The 3D scan is preferably based on a light detection and ranging, LIDAR, scan. However, other known 3D scanning methods may be used as well. For such a LIDAR scan suitable cameras/devices may be used, such as the Matterport Pro <NUM> or Iphone <NUM> pro or Ipad Pro.

The use of a 3D scan is preferred, because 3D modelling and the use of CAD data may be more time-consuming and more complex than a 3D scan. That is, the 3D scan may be easier and faster than 3D modelling and the use of CAD data.

The location data are realtime location data of the position-tracked object.

The at least one 3D metamodel represents a model that is derived from historical location data of the at least one position-tracked object.

More particular, a 3D metamodel may prefereably include meta information such as position data and its characteristics and any information that may be derived for the position data and its characteristics. The 3D metamodel may be a heatmap, rewind to the past, etc..

Generating a 3D interaction model for a virtual reality device or an augmented reality device based on the 3D model of the environment and the at least one 3D model of the position-tracked object and the at least one 3D metamodel and the received location data preferably comprises: generating the 3D interaction model based on the current location data of the at least one 3D model of the position-tracked object.

The location data is preferably obtained by an indoor localization technique, wherein the localization technique is based on Ultra Wide Band, UWB, or Bluetooth Low Energie, BLE, and wherein the method preferably receives the location data over an application programming interface, API.

Although an indoor localization technique is mentioned at this point, it is clear that any known localization technique may be used that is able to provide the respective location data. In particular, if the environment is an outdoor environment, the localization technicque may preferably be an outdoor localization technique instead of an indoor localization technique. However, the may be independent of the environment being an indoor or an outdoor environment as long as the respective localization technique is able to provide the respective location data.

For receiving the location data over an API, the method may preferably receive (acquire) the location data from a service such as the "Omlox Hub" (https://omlox. com/home) or the like.

The location data is preferably repeatedly received in predetermined time intervals or continuously and the positioning of the 3D model of the position-tracked object within the 3D model of the environment is updated based on the latest location data.

The received location data is preferably stored in a database (or any API / cloud service) in association with a respective timestamp to create historical location data.

Generating a 3D interaction model for a virtual reality device or an augmented reality device based on the 3D model of the environment and the at least one 3D model of the position-tracked object and the at least one 3D metamodel and the received location data comprises: generating the 3D interaction model based on the historical location data of the at least one 3D model of the position-tracked object.

Generating the 3D interaction model based on the historical location data of the at least one 3D model of the position-tracked object comprises: displaying the 3D model of the position-tracked object in the 3D model of the environment at multiple times.

"At multiple times" may refer to times in the past, but may include also the current time.

The method may preferably further comprise: providing the 3D interaction model to a virtual reality device or an augmented reality device.

The virtual reality device or the augmented reality device may preferebaly be a mobile device, such as a mobile phone, a tablet, a laptop, a smart watch, smart glasses or other wearables etc. However, the virtual reality device or the augmented reality device may also be a stationary device such as a TV or a desktop computer.

The method may preferably further comprise: displaying the 3D interaction model on a virtual reality device or an augmented reality device based on the received location data.

The present disclosure also relates to a computing device comprising a processor, wherein the processor is configured to execute the method as described above.

In summary, the embodiments of the present disclosure consist of a unique combination of existing technologies and integrates current trends around gamification in virtual reality (VR) and augmented reality (AR) to display live location data to the end user in a more realistic, immersive environment.

In particular, the proposed 3D visualization provides an environement, e.g. an entire factory, where live location data is visualized by real 3D models. The real 3D (interaction) model can be generated by current methods of 3D scanning. By employing 3D navigation, the virtual factory can be explored from all sides or flown through. Objects to be searched for can thus be found faster and more intuitively. At the same time, the 3D representation of the factory provides a higher degree of immersion for a user. The immersion level could further be enhanced by outputting the 3D interaction model to a VR headset.

In other words, the embodiments of the present disclosure provide a 3D interaction model to display live location and metadata to the end user in a realistic, immersive and virtual environment. The embodiments employ, inter alia, new camera technologies such as Lidar to capture a 3D image of a factory. This serves as a basis to build a photorealistic virtual factory (a 3D model of the environemnt).

In addition, trackables (e.g., forklifts, AGVs, or even people) are mapped as a graphical 3D model and displayed in the 3D factory. The real position data of the trackables can be animated in the virtual factory. This makes it possible to see in the virtual environment what is happening live in the factory (telepresence). By switching to a VR view (e.g. using VR glasses), it is possible to experience this virtual world. The AR mode is intended to display specific information and models in the real environment. Thus, geofencedata can be drawn live in the real environment.

According to embodiments of the present disclosure, mapping of analytical data e.g. heatmap function for visualization of storage times or typical routes of vehicles may be provided. In addition, AR solutions allow displaying metadata of trackables within the factory. Thus, providing navigations to a specific object the user is looking for.

Via WebGL, the 3D graphics are directly executable in most browsers and can be mapped via a client/server architecture. Thus, no additional installation on the client side may be necessary.

The embodiments of the present disclosure provide a more exact mapping, e.g. for indoor navigation, which is closer to reality. In addition, the COVID-<NUM> pandemic has also created the need to present location solutions to customers in a purely virtual way and to create a positive customer experience, which is provided by the aforementioned embodiments.

Moreover, the indoor localization and visualization of a position in an indoor environment can be enhanced by the foregoing embodiments, because different building levels can be represented accuarately. Additional information, such as geofences (virtual areas) or heat maps can also be defined. The present embodiments further provide the possibility for virtual customer events that provide the necessary customer experience.

The exemplary embodiments disclosed herein are directed to providing features that will become readily apparent by reference to the following description when taken in conjunction with the accompany drawings. In accordance with various embodiments, exemplary systems, methods, devices and computer program products are disclosed herein. It is understood, however, that these embodiments are presented by way of example and not limitation, and it will be apparent to those of ordinary skill in the art who read the present disclosure that various modifications to the disclosed embodiments can be made while remaining within the scope of the present disclosure.

Fig. 1a schematically shows a method according to an embodiment of the present disclosure. In particular, the method receives (obtaines) input data. In particular, the method receives a 3D scan of the environment, e.g. a fabrication hall, storage facility or the like. The 3D scan may be raw data for creating a 3D model <NUM> of the environment. Alternatively, the respective input may be the 3D model <NUM> of the environment itself.

The method further receives (obtaines) a 3D model <NUM> of an object <NUM>. The object <NUM> may be, e.g., a fork truck, AGVs, individuals, goods, tools, airplanes or the like. The object <NUM> may be reffered to as a position-tracked object <NUM> (or "trackables"), because the object <NUM> is an object <NUM>, which position can be tracked.

The position data of said object <NUM> is received through an API, for example. For receiving the location data over an API, the method may receive (acquire) the location data from a service such as the "Omlox Hub" (https://omlox. com/home) or the like. This way, the respective object <NUM> can be tracked in realtime and the respective information is provided to the method of the present disclosure.

Alternatively or in addition, the method may further receive meta data connected to the 3D model <NUM> of the position-tracked object <NUM>. The meta data may allow the method to create specific information useful for a user, such as order number, part number, customer information, production or order state. This information may be used to be displayed in an augmented or virtual reality device <NUM> or on other devices like tablets, Smartphones, Desktops to provide a user with useful or required information.

The meta data and/or the position data may also be used to create a 3D metamodel related to the 3D model <NUM> of the position-tracked object <NUM>. A 3D metamodel is a model that is derived from historical location data of the at least one position-tracked object <NUM>. More particular, a 3D metamodel may prefereably include meta information such as position data and its characteristics and any information that may be derived for the position data and its characteristics. The 3D metamodel may be a heatmap, rewind to the past, etc..

The method combines the various inputs outlined above and generates a 3D interaction model for a virtual reality device or an augmented reality device (collectively designated with reference numeral <NUM>) based on the 3D model <NUM> of the position-tracked object <NUM>, the 3D model <NUM> of the environment and realtime location data of the position-tracked object <NUM>.

The method then provides the 3D interaction model to a suitable device <NUM> for virtual reality or augmented reality applications. For example, the user may take a virtual tour through a workshop or a storage facility without the need to be actually present in person (virtual reality application). In another example, the user may use his/her mobile device to identify objects <NUM> (such as tools) within a workshop by using an augmented reality application. In this case, the tool may be the position-tracked object <NUM> and the method can provide the user with an indication on the location of the object on his/her mobile device. Further examples are outlined in the following with reference to <FIG> and <FIG>.

<FIG> shows an exemple for how the method according to the present disclosure can be used for an augmented reality application. In particular, <FIG> shows the user device (augmented reality device <NUM>). The position tracked-object <NUM> in this case may be a certain pallet with goods, which is shown in <FIG> to be carried by a fork truck (the fork truck may itself be a position-tracked object <NUM>). Here the user may be provided with the respective customer ID (here XY) and the order number (here <NUM>) when the mobile device <NUM> is pointed towards the position-tracked object <NUM>.

Another example for how the method according to the present disclosure can be used for an augmented reality application is shown in <FIG>. Here the 3D model <NUM> of the position-tracked object <NUM> (here a fork truck) is shown on the mobile device <NUM> of the user. In this case, the 3D model <NUM> may be indicated at the current position or at a past position. The user may also be provided with further information (not shown) associated with the 3D model <NUM>, e.g. the ID of the fork truck or the like.

Another example for how the method according to the present disclosure can be used for an augmented reality application is shown in <FIG>. Here the 3D model <NUM> of the position-tracked object <NUM> (here a fork truck) is shown on the mobile device <NUM> of the user at different times. That is, historical position data is used (with or without the current position data) to indicate the positon of the 3D model <NUM> at different times. This may be used, for example, to recreate the events leading to an accident.

It is understood that the foregoing examples are not limited to display only the specific information/models described. It is also possible to display more than one model and/or information on a respective user device <NUM> at a time.

In addition, the position data and/or meta data may be used to create 3D metamodels such as heat maps. Heat maps according to the present disclosure refer to a visualization of the frequency (occurance) of an object <NUM> being at a certain position. That is, a heat map may indicate a color coded map (within the 3D model of the environment in a virtual reality application or on the user device <NUM> in an augmented reality application) on where the position-tracked object <NUM> was during a certain period of time. This may indicate the moving path of the object and/or in which areas the object <NUM> was located the most.

<FIG> is a flow chart of a method according to an embodiment of the present disclosure. In particular, <FIG> shows a method for generating a three-dimensional, 3D, interaction model for a virtual reality device or an augmented reality device based on a 3D model and realtime location data of a position-tracked object.

In an embodiment, the method according to <FIG> comprises the following steps:.

In an embodiment, the generated three-dimensional model is suitable and intended to be displayed in space on a terminal device by means of an augmented reality framework.

In an embodiment the 3D model of the environment is a 3D model of an indoor environment.

In an embodiment, obtaining a three-dimensional, 3D, model of an environment comprises: receiving the 3D model of the environment from an external source or generating the 3D model of the environment based on received raw data.

In an embodiment, the 3D model of the environment is based on 3D modelling, CAD data, or a 3D scan.

In an embodiment, the 3D scan is based on a light detection and ranging, LIDAR, scan.

In an embodiment, generating a 3D interaction model for a virtual reality device or an augmented reality device based on the 3D model of the environment and the at least one 3D model of the position-tracked object and the at least one 3D metamodel and the received location data comprises: generating the 3D interaction model based on the current location data of the at least one 3D model of the position-tracked object.

In an embodiment, the location data is obtained by an indoor localization technique, wheren the localization technique is based on Ultra Wide Band, UWB, or Bluetooth Low Energie, BLE, and wherein the method preferably receives the location data over an application programming interface, API.

In an embodiment, the location data is repeatedly received in predetermined time intervals or continuously and the positioning of the 3D model of the position-tracked object within the 3D model of the environment is updated based on the latest location data.

In an embodiment, the received location data is stored in a database in association with a respective timestamp to create historical location data.

In an embodiment, the method further comprises: providing the 3D interaction model to a virtual reality device or an augmented reality device.

In an embodiment, the method further comprises: displaying the 3D interaction model on a virtual reality device or an augmented reality device based on the received location data.

Thus, the scope of the present invention should not be limited by any of the above-described exemplary embodiments, but solely by the appended claims.

A skilled person would further appreciate that any of the various illustrative logical blocks, units, processors, means, circuits, methods and functions described in connection with the aspects disclosed herein can be implemented by electronic hardware (e.g., a digital implementation, an analog implementation, or a combination of the two), firmware, various forms of program or design code incorporating instructions (which can be referred to herein, for convenience, as "software" or a "software unit"), or any combination of these techniques.

To clearly illustrate this interchangeability of hardware, firmware and software, various illustrative components, blocks, units, circuits, and steps have been described above generally in terms of their functionality. In accordance with various embodiments, a processor, device, component, circuit, structure, machine, unit, etc. can be configured to perform one or more of the functions described herein. The term "configured to" or "configured for" as used herein with respect to a specified operation or function refers to a processor, device, component, circuit, structure, machine, unit, etc. that is physically constructed, programmed and/or arranged to perform the specified operation or function.

Furthermore, a skilled person would understand that various illustrative logical blocks, units, devices, components and circuits described herein can be implemented within or performed by an integrated circuit (IC) that can include a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, or any combination thereof. The logical blocks, units, and circuits can further include antennas and/or transceivers to communicate with various components within the network or within the device.

In this document, the term "unit" as used herein, refers to software, firmware, hardware, and any combination of these elements for performing the associated functions described herein. Additionally, for purpose of discussion, the various units are described as discrete units; however, as would be apparent to one of ordinary skill in the art, two or more units may be combined to form a single unit that performs the associated functions according embodiments of the present disclosure.

Claim 1:
A method for generating a three-dimensional, 3D, interaction model for a virtual reality device (<NUM>) or an augmented reality device (<NUM>) based on a 3D model (<NUM>) and realtime location data of a position-tracked object (<NUM>), the method comprising:
obtaining (S41) a three-dimensional, 3D, model (<NUM>) of an environment;
obtaining (S42) at least one 3D model (<NUM>) representing the position-tracked object (<NUM>) and obtaining at least one 3D metamodel related to the 3D model (<NUM>) of the position-tracked object (<NUM>), wherein the at least one 3D metamodel represents a model that is derived from historical location data of the position-tracked object (<NUM>);
receiving (S43) location data of the position-tracked object (<NUM>) associated with the 3D model (<NUM>) of the position-tracked object (<NUM>), wherein the location data are realtime location data of the position-tracked object (<NUM>); and
generating (S44) a 3D interaction model for a virtual reality device (<NUM>) or an augmented reality device (<NUM>) based on the 3D model (<NUM>) of the environment and the at least one 3D model (<NUM>) of the position-tracked object (<NUM>) and the at least one 3D metamodel and the received location data by displaying the at least one 3D model (<NUM>) of the position-tracked object (<NUM>) in the 3D model (<NUM>) of the environment at multiple positions simultaneously, the multiple positions being positions of the at least one 3D model (<NUM>) of the position-tracked object (<NUM>) at multiples times, wherein the historical location data are used for the multiple times.