Patent Description:
Technology developments for providing digital reality application content to users have enabled experiences that were not possible in the past. Particularly, digital realities, such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), change a user's perception on what they are seeing, hearing, and feeling, and how much of the real world comes into these experiences, providing the user with a sensation of a physical presence in places in the real world or an imagined world.

Augmented reality (AR) is a live view of a real-world environment whose elements are superimposed by computer-generated objects such as video, texts or 3D computer models. AR technology can be divided into <NUM> categories, specifically, marker-based and markerless AR. Marker-based AR may be grouped into image-based and location-based AR. Image-based AR needs specific labels to register the position of 3D objects on the real world image. In contrast, location-based AR uses position data such as data from GPS to identify the location. Applications using location-based AR range from presenting information based on the user's position in the real world to location-aware social networks with custom proximity notification. For realistic interaction applications simulation, assembly training or educational experiments, image-based AR is preferred.

Current AR technology suffers several drawbacks. For example, regardless of the AR applications being configured as marker-based or marker-less based, current AR applications are usually viewed by users individually depending on the shapes for which the markers are configured and/or location where the AR applications are attached. AR applications are typically not persistent and do not take into account a mapping of all geometries of the 3D world and the virtual elements and applications interactions with these elements. Experiences are also usually individualized depending on the viewing position and orientation of a user without enabling shared experiences with other users. Furthermore, users may need to download applications that are later run directly on a user's client device, making it time-and-resource-consuming for the client devices. Finally, tracking of the viewing position and orientation of client devices is decisive on how the media streams are transferred to users. However, current tracking techniques may result inaccurate and may lead to media streams being uncoordinated with a user's movements, causing problems such as vertigo, discomfort, and resulting sometimes in low interaction times and demand for such applications.

What is desired is a system and method that enables persistent and accurate location-and-geometry based persistent positioning of applications and personal or shared interactions in the real world that takes into account the relationship with the real world.

<CIT> describes a system for rendering a 3D virtual environment. The system comprises a central processing device, a plurality of user devices in data communication with the central processing device, a plurality of application servers in data communication with the central processing device, and software executing on the central processor. The software creates and renders a 3D virtual environment, receives user data from each of the plurality of user devices, renders the user data received from each of the user devices in the 3D virtual environment, receives application data from each of the application servers, renders the application data received from each of the application servers in the 3D virtual environment, and outputs the rendered 3D virtual environment to each of the user devices. The 3D virtual environment serves as a direct user interface with the Internet by allowing users to visually navigate the world wide web.

<CIT> describes a head mounted display (HMD) device, which has a display panel, a depth or distance sensor to measure distances between the HMD and a real object. The HMD device sets a close transition boundary distance (CTBD) between the HMD and a close transition boundary (CTB). A far transition boundary distance (FTBD) is set between the HMD and a far transition boundary (FTB). The CTBD is less than the FTBD. As a real object that has associated near and far virtual content moves nearer to the HMD device and crosses the CTB, the virtual content transitions to near virtual content for viewing on the HMD. As the real object moves away from the HMD and crosses the FTB, the virtual content transitions to the far virtual content for viewing on the EMD.

<CIT> describes systems and methods for AR content delivery in pre-captured environments. In one embodiment, an AR client gathers information about the physical environment with the help of sensors embedded within an AR headset. The client reconstructs a 3D model of the physical environment from the gathered information and sends it to an AR content server. Based on the 3D reconstruction information obtained at the AR content server, the server modifies the content. AR content server compares the AR content against the 3D reconstruction data and does a visibility analysis of the AR content. Based on the visibility analysis, the AR elements occluded by real objects are removed to reduce the data size of the AR content. Finally, the AR content server streams the modified content to the AR client and the AR client renders the content and sends it to the headset.

<CIT> relates to the displaying of information from search results and other sets of information as AR images. For example, one disclosed embodiment provides a method of presenting information via a computing device comprising a camera and a display. The method includes displaying a representation of each of one or more items of information of a set of electronically accessible items of information. The method further comprises receiving a user input requesting display of a selected item of the set of electronically accessible items of information, obtaining an image of a physical scene, and displaying the image of the physical scene and the selected item together on the display as an AR image.

<CIT> describes the insertion of video images of human beings, animals or other living beings or life forms, and any clothing or objects that they bring with them, into a virtual environment. Others participating in the environment can see that person as they currently look, in real-time, and from any viewpoint.

<CIT> describes the deployment and targeting of context-aware Virtual objects and behavior modeling of virtual objects based on physical principles. An indication that a content segment being consumed in the target environment has virtual content associated with it is detected. Further, the virtual content that is contextually relevant for consumption in target environment is presented. In addition, contextual information for the target environment can be captured.

According to the present invention, there is provided a system as set out in independent claim <NUM>, and a method as set out in independent claim <NUM>.

Drawbacks disclosed in the background are addressed through embodiments of the current disclosure through systems and methods for attaching digital reality applications and interactions to static objects. The systems and methods comprise merging real and virtual realities into merged realities that involve interaction between virtual replicas of real objects, digital reality applications, and users. The systems and methods enable shared or individual experiences between users and the applications. Additionally, the systems and methods provide functionality (e.g., real-time movement and eye-tracking capabilities) that enable media streams to be provided in accordance the position and orientation of the viewers. The static objects where digital reality applications are attached can be reached by physically going to the location of those objects or by going in virtual reality.

A system for attaching digital reality applications and interactions to static objects, according to embodiments of the current disclosure, comprises a server configured to store and process input data, the server comprising a memory and a processor; and a plurality of client devices connected to the server via a network and configured to enable interaction with one or more digital reality applications. In an embodiment, one or more servers contain a persistent virtual world system storing virtual replicas of static objects of the real world, the virtual replicas comprising settings such as the location and space, physics settings, and a 3D data structure, amongst other possible settings, based on the static objects. The persistent virtual world system is configured to virtually attach digital reality applications on the virtual replicas. In an embodiment, the system enables application developers to modify the persistent virtual world system by providing their own digital reality applications and instructing the persistent virtual world system to attach their digital reality applications on the virtual replicas.

When users employing client devices approach a location, object, or entity to which a digital reality application is virtually attached, the digital reality application detects the physical position and orientation of the client device. For example, the digital reality application may receive position and orientation data from the client devices or from other computing devices that track the position and orientation of the client devices. The digital reality application may, for example, compare the received position data with a trigger zone, triggering the server to transmit digital reality application content to the client devices that have entered the trigger zone. In an embodiment, the entry of the client devices into a trigger zone is detected by receiving a notification of the positions of the user devices and comparing the positions to a known boundary of the trigger zone.

The digital reality application content may be presented in merged reality, virtual reality, or real reality. The digital reality application content or interactions with the digital reality applications may be personal to an individual device or user, or shared with other client devices. The client devices may be one or more mobile devices, personal computers, game consoles, media centers, smart contact lenses, and head-mounted displays, which are configured to enable interactions between users and the digital reality application content via user interfaces.

The location and space are configured to allow setting up the location, relative three-dimensional position and orientation, and scaling of virtual replicas. Configuration of physics enables the specification of physical properties of the virtual replicas, such as rigid body dynamics, soft body physics, fluid dynamics, occlusion, and collision detection, amongst others. The 3D data structure refers to a data organization and storage format of 3D objects that enables efficient access and modification, including, for example, octrees, quadtrees, BSP trees, sparse voxel octrees, 3D arrays, and k-d trees.

The digital reality application is virtually attached to a place in the real world. Thus, a user not wearing or employing a client device may approach the place where the digital reality application was virtually attached without triggering any reaction in the system (i.e., not enabling the user to view the digital reality application). However, when the user activates and employs a client device, the digital reality application detects the viewing position and orientation of the client device and retrieves corresponding media streams aligned with the position and orientation during user interaction. In some embodiments, the digital reality application content included in the media streams includes 3D image data, 3D geometries, 3D entities, 3D sensory data, 3D dynamic objects, video data, audio data, textual data, time data, positional data, orientational data, haptic data, or lighting data, or a combination thereof.

According to an embodiment, the digital reality applications may utilize one or more of a plurality of techniques to broadcast a signal to the client devices, alerting the client devices that a digital reality application is available in proximity to the location of the client devices. In one embodiment, if the user has previously subscribed to an application in the server, the location of the client device may then at all times be available at the persistent virtual world system stored in the server. Therefore, when a user approaches a digital reality application, the digital reality application may already prepare to broadcast the signals to the client device. In another embodiment, if the user has previously subscribed to the server or to one or more specific digital reality applications from the server, the digital reality application may constantly search for registered devices prior to broadcasting the signal. In one embodiment, as a user with a client device enters a location where the digital reality application is active, the digital reality application may detect a signal from the device, indicating the device can receive a digital reality application media stream prior to broadcasting the signal.

In some embodiments, when the user employs the digital reality device, the user may select to view the elements in merged reality or virtual reality. In merged reality, the user may view the real world elements plus the one or more applications virtually attached thereon along with one or more other virtual elements. Thus, merged reality comprises physical, real-world environment elements augmented by computer-generated input such as sound, video, graphics, and GPS or other tracking data. Augmentation techniques are typically performed in real-time and in semantic context with environmental elements, such as overlaying supplemental information or virtual objects in the real world. The merged reality content allows information about the surrounding real world of the user or virtual objects overlay in the real world to become interactive and digitally manipulable. In virtual reality, the user may view the same static objects in a virtualized way, replacing the real world with a simulated one. For example, the user may, from his home location, be able to select a remote location and explore that location in order to find the one or more digital reality applications and interact with them.

In accordance with the claimed invention, the client device downloads, after pre-determined levels of interaction, the application or parts of the application and performs client-side computations and rendering.

In other example implementations not covered by the claims, the user may not need to download the digital reality application in order for the applications to become visible or accessible to the user. Instead, the server may directly retrieve application graphical representations from the server and, if levels of interaction increase, perform remote computing and rendering for the client device, which may only need to perform lightweight computations in order to display the digital reality content.

According to an embodiment, the digital reality applications may be developed via one or more third-party 3D engines, local 3D engines, or combinations thereof. Likewise, the virtual replicas may be developed via a plurality of developers from one or more sources. As all the information is included in the server, the user is able to view all applications and virtual replicas, independent of the development software used and the source of the developers. Therefore, a user looking in one direction including a plurality of applications developed via different sources may view the plurality of applications and may be able to interact with them. Likewise, a user may be able to view some or all of the interactions of other users with respective applications, depending on the level of privacy and depth of the interactions.

Virtualizing the real world before attaching the digital reality applications to corresponding static objects allows application developers to utilize the location including position and orientation of the virtual replicas when attaching the applications. In addition, effects such as collisions and occlusion between static objects and digital elements may be enabled in a more realistic way, as each static object is stored as a virtual replica in the server. For example, digital reality applications may be occluded by real-world objects because the virtual replicas corresponding to the real-world objects may prevent a user from viewing the digital reality applications, enabling a realistic effect.

According to an embodiment, the virtual replicas are created via readily-available computer-assisted drawing (CAD) or computer-assisted engineering (CAE) models of the static objects. For example, machine owners may provide an administrator of the persistent virtual world system or may input by themselves the already-existing digital CAD or CAE models of their machines. Similarly, building owners may provide building information models (BIM) with building details to be stored in the persistent virtual world system, which may include information that may not be visible or easily obtainable via sensing mechanism. In other embodiments, the modeling tools enable a car or drone-based image-scanning pipeline to be input through a variety of photo, video, depth simultaneous location and mapping (SLAM) scanning in order to model the virtual replicas. In other embodiments, radar-imaging, such as synthetic-aperture radars, real-aperture radars, AVTIS radars, Light Detection and Ranging (LIDAR), inverse aperture radars, monopulse radars, and other types of imaging techniques may be used to map and model static objects before integrating them into the persistent virtual world system. Utilizing these more technical solutions may be performed especially in cases where the original models of the structures are not available, or in cases where there is missing information or there is a need to add additional information to the virtual world entities which is not provided by the CAD models.

In an embodiment, in order to reduce hardware and network demands, contribute to the reduction of network latency, and improve the general digital reality experience, the system may connect through a network including millimeter-wave (mmW) or combinations of mmW and sub <NUM> communication systems, such as through <NUM>th generation wireless systems communication (<NUM>). In other embodiments, the system may connect through wireless local area networking (Wi-Fi), which may provide data at <NUM>. Provided communication systems may allow for about <NUM> to about <NUM> millisecond end-to-end (E2E) latency and <NUM>-<NUM> Gbps downlink speeds to end points in the field, complying with parameters necessary for executing the typically highly-interactive applications. This results in high-quality, low latency, real-time digital application content streaming. In other embodiments, the system may communicatively connect through 4th generation wireless systems communication (<NUM>), may be supported by <NUM> communication systems, or may include other wired or wireless communication systems.

In other embodiments, global navigation satellite systems (GNSS), which refers collectively to multiple satellite-based navigation systems like GPS, BDS, Glonass, QZSS, Galileo, and IRNSS, may be used for enabling positioning of devices. Employing signals from a sufficient number of satellites and techniques such as triangulation and trilateration, GNSS can calculate the position, velocity, altitude, and time of devices. In an embodiment, the external positioning system is augmented by assisted GNSS (AGNSS) through the architecture of existing cellular communications network, wherein the existing architecture comprises <NUM>. In other embodiments, the AGNSS tracking system is further supported by a <NUM> cellular communications network. In indoor embodiments, the GNSS is further augmented via radio wireless local area networks such as Wi-Fi, preferably, but not limited to, providing data at <NUM>. In alternative embodiments, the GNSS is augmented via other techniques known in the art, such as via differential GPS (DGPS), satellite-based augmentation systems (SBASs), real-time kinematic (RTK) systems. In some embodiments, tracking of devices is implemented by a combination of AGNSS and inertial sensors in the devices.

According to an embodiment, the sensing mechanisms mounted on the connected devices include a combination of inertial tracking sensing mechanisms and transceivers. The inertial tracking sensing mechanisms can make use of devices such as accelerometers and gyroscopes, which may be integrated in an inertial measuring unit (IMU). Accelerometers measure linear acceleration, which can be integrated to find the velocity and then integrated again to find the position relative to an initial point. Gyroscopes measure angular velocity, which can be integrated as well to determine angular position relatively to the initial point. Additional accelerometers and gyroscopes separate from the IMU may also be incorporated. The transceivers may be implemented to send and receive radio communication signals to and from antennas. In an embodiment, the transceivers are mmW transceivers. In embodiments where mmW antennas are employed, the mmW transceivers are configured to receive mmW signals from the antennas and to send the data back to the antennas. The inertial sensors, and positional tracking provided by mmW transceivers and the accurate tracking, low-latency and high QOS functionalities provided by mmW-based antennas may enable sub-centimeter or sub-millimeter positional and orientational tracking, which may increase accuracy when tracking the real-time position and orientation of the connected elements. In some embodiments, tracking may be implemented by employing several techniques known in the art, such as time of arrival (TOA), angle of arrival (AOA), or other tracking techniques known in the art (e.g., visual imaging, radar technology, etc.). In alternative embodiments, the sensing mechanisms and transceivers may be coupled together in a single tracking module device.

By way of example, the digital reality application is virtually attached to a static shoe placed in a specific place in a park. Without the client device, the user may only view a shoe, but when the user employs the client device, the user may view a digital reality application that may be represented by the shoe glowing or emitting a sound. The user may be alerted by the digital reality application about the presence of the digital reality application and the user may thereafter touch a virtual, graphical representation of the same, which may trigger other forms of interaction. For example, after the user touches the graphical representation of the digital reality application, the graphical representation may increase in size, may enable the user to rotate the shoe, view the shoe characteristics in the form of text, view videos about the shoe, change some characteristics of the shoe, such as color, size, or style, and may purchase the selected shoe. The shoe may remain in the specific place in the park or elsewhere until decided by the shoe provider, and may thereafter move the shoe to another place, for which the shoe location and space settings may need to be updated in the server. In other examples, the shoe may be viewed completely in virtual reality, such as showing the shoe in a different setting, such as in a desert, mountain, or another city.

According to an embodiment, a method for attaching digital reality applications and interactions to static object is provided. The method may be implemented in a system, such as systems of the current disclosure. The method includes providing a persistent virtual world system storing virtual replicas of static objects of the real world, the virtual replicas comprising a static object location and space, physics settings, and a 3D data structure; and virtually attaching digital reality applications to the virtual replicas, such that as a client device approaches a location, object, or entity to which a digital reality application is virtually attached, the digital reality application detects the physical position and orientation of the client device, triggering the server computer system to transmit digital reality application content to the client device.

In an embodiment, the method starts by creating, by a developer via a replica editor stored in a server, virtual replicas of static objects. Subsequently, the method continues by adding virtual replica real-world properties, including location and space settings, physics settings, and 3D data structure. Then, the method continues by attaching, by a developer via a replica editor stored in the server, digital reality applications to the virtual replicas.

In an embodiment, a user approaches and looks at a digital reality application. In some embodiments, the user may approach the digital reality application in a merged reality. In other embodiments, the user may approach the digital reality application in virtual reality. Approaching the digital reality application triggers the digital reality application to detect and track the client device position and orientation, and to send the client position and orientation to the server. The server then retrieves and tracks the viewing position and orientation from client device to thereafter send application media streams to the client device adjusted and aligned to the viewing position and orientation of the user. Finally, the user interacts with the digital reality application via the client device.

The above summary does not include an exhaustive list of all aspects of the present disclosure. Other features and advantages will be apparent from the accompanying drawings and from the detailed description that follows below.

Specific features, aspects and advantages of the present disclosure will be better understood with regard to the following description and accompanying drawings, where:.

In the following description, reference is made to drawings which show by way of illustration various embodiments. Also, various embodiments will be described below by referring to several examples.

<FIG> depict schematic representations of a system <NUM> for attaching applications and interactions to static objects, detailing one or more users viewing and interacting with one or more applications virtually attached to elements of the real world, according to an embodiment.

As viewed in <FIG>, the system <NUM> comprises a server <NUM> configured to store and process input data, and a plurality of client devices <NUM> employed by users <NUM>, the client devices <NUM> connected to the server <NUM> via a network <NUM>. The client devices <NUM> are configured to enable user interaction with one or more digital reality applications <NUM>. The server <NUM> contains a database with structured data containing a persistent virtual world system storing virtual replicas of static objects <NUM> of the real world. The system <NUM> enables application developers to attach digital reality applications <NUM> on the virtual replicas. When users <NUM> employing client devices <NUM> approach one or more digital reality applications <NUM> virtually attached to the static objects <NUM>, the digital reality applications <NUM> detect the physical position and orientation of the client device, triggering the server <NUM> to share digital reality application <NUM> content with the client devices <NUM>. Although the system <NUM> is described as including a single server <NUM> in examples disclosed herein, it should be understood that functions described herein as being performed by a single server (e.g., server <NUM>) may instead be performed by server system comprising multiple server computers, or vice versa.

In an embodiment, the digital reality applications <NUM> detect the position and orientation of the client devices <NUM> by receiving position and orientation data transmitted from the client devices <NUM> or from other computing devices that track the position and orientation of the client devices. In such an embodiment, the digital reality applications <NUM> may compare the received position data with a trigger zone, triggering the server <NUM> to share digital reality application <NUM> content with the client devices <NUM> that have entered the trigger zone. The client devices <NUM> may be one or more mobile devices, personal computers, game consoles, media centers, and head-mounted displays, which are configured to enable interactions with users <NUM> and the digital reality application <NUM> content via user interfaces.

When the digital reality application <NUM> is virtually attached to an element of the real world such as a static object <NUM>, the digital reality application <NUM> may, for example, be positioned or repositioned within a space in tandem with the object to which it is virtually attached, or be removed from or reintroduced into a space as the object to which it is attached is removed from or reintroduced into that space. Alternatively, the digital reality application <NUM> may be unattached or detached from an object to which it was virtually attached. If the digital reality application <NUM> is unattached, the application may be positioned in a space independent of the position of any object. If the digital reality application <NUM> is detached from an object to which it was virtually attached, the application may, for example, be removed from a space in which that object is still present, or remain in a fixed location that is independent of the location of that object, or move independently of that object.

A trigger zone can be calculated as a distance between a user and the virtual position where an application has been configured in the real world. For example, the trigger zone could be set for, e.g., from <NUM> to <NUM> meters around a location, object, or entity associated with an application. Alternatively the trigger zone can be set according to the field of view of the user, e.g., when the location, object, or entity associated with an application is in the field of view of the user, then the application may be triggered for that specific user. A field of view trigger zone may be independent of distance from the user. Alternatively, a trigger zone can also be an area defined by the field of hearing of a user (e.g., when a user cannot see a location, object, or entity associated with an application because it is hidden behind another object, but he could in theory hear any sounds from it). This type of trigger zone may be limited by distance but may also take into account any potential sound absorption by nearby materials or objects.

In the current disclosure, the term "static" is used to characterize objects that have a fixed position and orientation in the real world, and thus a corresponding fixed position and orientation input in the respective virtual replica in the persistent virtual world system. For example, the term "static" may be used to characterize objects in the real world that under typical conditions stay in the same location, such as the fountain, shoe and statue illustrated as static objects <NUM> in <FIG>. In some cases, a static object may correspond to a real-world object that is portable by nature but is expected to remain in place for an extended time. For example, the shoe <NUM> may be considered a static object when it is fixed in place or otherwise expected to remain in place (e.g., as a museum exhibit or as a promotional display in a retail store). In other cases, static objects correspond to real-world objects that by nature are not portable (such as the fountain and the statue). Such objects may remain in place except in rare situations, such as during remodeling of an area comprising the static objects. In such cases, the position and orientation of the displaced static objects may have to be updated manually or based on data obtained from cameras or other sources (e.g., optical data from cameras, radar data, and the like). Other examples of static objects may comprise natural formations, such as rocks, valleys, rivers, volcanoes, mountains, lakes, and trees.

In the current disclosure, the term "dynamic" is used to characterize objects that have a variable position and orientation in the real world, and thus a corresponding variable position and orientation in the virtual replica of the persistent virtual world system. For example, the term "dynamic" may be used to characterize objects in the real world that may normally be displaced from one area to another one, such as a car. Other example dynamic objects may be any type of moving vehicle, such as bicycles, drones, planes, and boats. In other examples, dynamic objects may also comprise living beings, such as humans and animals.

In the current disclosure, the term "virtual replica" refers to accurate and persistent virtual representations of real-world elements. In an embodiment, a virtual replica comprises data and models that provide self-computing capabilities and autonomous behavior. The data and models of the virtual replicas may be input through a plurality of software platforms, software engines, and sensors connected to real-world elements. Data are the attributes of the virtual replicas and the models are the graphical, mathematical and logic representations of any aspect of the corresponding real-world element that may be used to replicate the reality in the persistent virtual world system, such as 3D models, dynamic models, geometric models, and machine learning models.

In the current disclosure, the term "persistent" is used to characterize a state of a system that can continue to exist without a continuously executing process or network connection. For example, the term "persistent" may be used to characterize the virtual world system where the virtual world system and all of the virtual replicas, purely virtual objects and digital reality applications therein comprised continue to exist after the processes used for creating the virtual replicas, purely virtual objects and digital reality applications cease, and independent of users being connected to the virtual world system. Thus, the virtual world system is saved in a non-volatile storage location (e.g., in a server). In this way, virtual replicas, purely virtual objects and digital reality applications may interact and collaborate with each other when being configured for accomplishing specific goals even if users are not connected to the server.

"Self-computing capabilities", also referred to as "self-managing capabilities" refers herein to the ability of a virtual replica of the persistent virtual world system to apply artificial intelligence algorithms in order to autonomously manage computer resources (e.g., distributed computing resources). In an embodiment, virtual replicas with self-computing capabilities are able to autonomously manage computing resources to adapt to changes in the environment of corresponding real-world elements or in the real-world elements themselves. Thus, each virtual replica may act autonomously depending on the conditions in the real world reflected in the persistent virtual world system, by allocating required resources, autonomously sending and executing commands and generating events as required by each circumstance. Achieving this type of behavior may require training the virtual replicas with artificial intelligence algorithms during the modeling of the virtual replicas. Thus, the role of a virtual replica editor may be limited to defining general policies and rules that guide the self-management process. For example, in the case of a car accident, the virtual replicas of autonomous vehicles close to the accident may decide to lower their speed or come to a stop in order to worsen traffic conditions, and notify the relevant authorities, before the passengers in the vehicle can even know that there was an accident.

The system <NUM> of the current disclosure may be implemented in a cloud to edge infrastructure that may display distributed computing capabilities employing public or private clouds, fog servers, and edge devices and systems, such as enterprise systems, mobile platforms, and user devices, all of which may connect through a network. Using a cloud to edge computing network, access to computing power, computer infrastructure (e.g., through so-called infrastructure as a service, or IaaS), applications, and business processes can be delivered as a service to users via client devices on demand. This way, resources including physical servers and network equipment enable a shared storage and computing that may be dynamically allocated depending on factors such as the distance of the user to the resources and the network and computational demand from the users.

A pair of virtual-real twin, or twin-pair, comprises a real-world element and its corresponding virtual replica, or virtual twin, and can be considered as a Cyber-physical system or CPS. The CPS is an integration of computation with physical processes whose behavior is defined by both cyber and physical parts of the system. Therefore, a virtual replica is the cyber part of the CPS, while the physical part is the real world element. The virtual replica may then be considered as an extension of the real twin that allows connecting the physical part with artificial intelligence and simulations to improve the object's capabilities and performance.

The client device <NUM> downloads, after pre-determined levels of interaction, the digital reality application <NUM> or parts of the digital reality application <NUM> and performs client computations and rendering. In other example implementations not covered by the claims, users <NUM> may not need to download the digital reality application <NUM> in order for the applications to become visible or accessible to the user <NUM> and receive respective media streams <NUM> from the digital reality applications <NUM>. Instead, the server <NUM> may directly retrieve application graphical representations to the user <NUM> from the server <NUM> and, if levels of interaction increase, perform remote computing and rendering for the client device <NUM>, which may only need to perform lightweight computations in order to display the digital reality content.

In some embodiments, the digital reality application <NUM> content included in the media streams <NUM> includes at least one of the following: 3D image data, 3D geometries, 3D entities, 3D sensory data, 3D dynamic objects, video data, audio data, textual data, time data, positional data, orientational data, and lighting data.

In some embodiments, a virtual replica includes one or more of 3D world and building data, such as SLAM or derivate-mapping based data; 3D geometry data; 3D point cloud data; or geographic information system data representing real-world structural properties that may serve to model a 3D structure for digital reality applications.

In some embodiments, each of the virtual replicas of the persistent virtual world system may be geolocated using a reference coordinate system suitable for use with current geolocation technologies. For example, the virtual replicas may use a World Geodetic System standard such as WGS84, which is the current reference coordinate system used by GPS.

According to an embodiment, the digital reality applications <NUM> may be developed via one or more third-party 3D engines, local 3D engines, or combinations thereof. Likewise, the virtual replicas where the digital reality applications <NUM> are virtually attached may be developed via a plurality of developers from one or more sources. As all the information is included in the server <NUM>, the user <NUM> is able to view all applications and virtual replicas, independent of the development software used and the source of the developers. Therefore, a user <NUM> looking in one direction including a plurality of applications developed via different sources may view the plurality of applications and may be able to interact with them. Likewise, a user <NUM> may be able to view some or all of the interactions of other users <NUM> with respective applications, depending on the level of privacy desired or access entitlements and depth of the interactions.

In an embodiment, in order to reduce hardware and network demands, contribute to the reduction of network latency, and improve the general digital reality experience, the system <NUM> may connect through a network <NUM> including millimeter-wave (mmW) or combinations of mmW and sub <NUM> communication systems, such as through <NUM>th generation wireless systems communication (<NUM>). In other embodiments, a wireless local area networking (Wi-Fi) provides data at <NUM>. Provided communication systems may allow for low (e.g., about <NUM> to about <NUM> millisecond end-to-end (E2E) latency and high (e.g., <NUM>-<NUM> Gbps) downlink speeds to end points in the field, complying with parameters necessary for executing the typically highly-interactive digital reality applications <NUM>. This results in high-quality, low latency, real-time digital application content streaming. In other embodiments, the system <NUM> may communicatively connect through 4th generation wireless systems communication (<NUM>), may be supported by <NUM> communication systems, or may include other wired or wireless communication systems.

As viewed in <FIG>, a plurality of users <NUM> may each view a different static object <NUM>, each static object <NUM> having one or more digital reality applications <NUM> virtually attached. In the example of <FIG>, a fountain, a shoe, and a statue, are static objects <NUM> to which digital reality applications <NUM> may be virtually attached thereon, where each of the static objects <NUM> is beforehand modeled and stored in the server <NUM>. Each user <NUM>, when the field of view <NUM> comprises one static object <NUM>, may view the respective static object <NUM> and digital reality application <NUM> corresponding to the viewing position and orientation of each user <NUM>.

In another embodiment, as viewed in <FIG>, when the field of view <NUM> of a user <NUM> comprises a plurality of digital reality application <NUM>, a single user <NUM> may be able to view the respective plurality of static objects <NUM> and digital reality applications <NUM> corresponding to the viewing position and orientation of the user <NUM>. Thus, the user <NUM> may view the fountain, shoe, and statue and the digital reality applications <NUM> virtually attached respectively to each.

<FIG> depict schematic representations of a system <NUM> for attaching applications and interactions to static objects, detailing the configuration of a virtual replica and attachment of an application to the virtual replica, according to an embodiment. Some elements of <FIG> may be similar to elements of <FIG>, and thus similar or identical reference numerals may be used to identify those elements.

System <NUM> comprises a server <NUM> including a processor <NUM> and a memory <NUM>. The processor <NUM> is configured to execute instructions on data stored in the memory <NUM>. As viewed in <FIG>, the memory <NUM> contains a database with structured data containing a persistent virtual world system storing virtual replicas <NUM> of static objects selected among real-world elements, the virtual replicas <NUM> comprising the location and space <NUM>, physics settings <NUM>, and 3D data structure <NUM>, amongst others, based on the corresponding static objects <NUM>.

Virtualizing the real world before attaching the digital reality applications <NUM> to corresponding static objects allows application developers to utilize the location including position and orientation of the virtual replicas. In addition, effects such as collisions and occlusion between static objects and digital elements may be enabled in a more realistic way, as each real-world element is stored as a virtual replica in the server <NUM>. For example, digital reality applications <NUM> may be occluded by real-world objects because the virtual replicas corresponding to the real-world objects may prevent a user <NUM> from viewing the digital reality applications <NUM>, enabling a realistic effect. Likewise, as the real world is virtualized, collision effects between an application or a virtual element and a real-world object are enabled, creating a realistic effect.

The location and space <NUM> are configured to allow setting up the location, relative three-dimensional position and orientation, and scaling of virtual replicas <NUM>. Configuration of physics <NUM> enables the specification of physical properties of the virtual replicas <NUM>, such as rigid body dynamics, soft body physics, fluid dynamics, occlusion, and collision detection, amongst others. The 3D data structure <NUM> refers to a data organization and storage format of 3D objects that enables efficient access and modification, including, for example, octrees, quadtrees, BSP trees, sparse voxel octrees, 3D arrays, and k-d trees.

The system <NUM>, as viewed in <FIG>, enables application developers to attach digital reality applications <NUM>, such as digital reality applications 110a-c on respective virtual replicas 206a-c.

According to an embodiment, the virtual replicas <NUM> are created via readily-available CAD models of the static objects. For example, machine owners may provide an administrator of the persistent virtual world system or may input by themselves the already-existing digital CAD or CAE models of their machines. Similarly, building owners may provide building information models (BIM) with building details to be stored in the persistent virtual world system, which may include information that may not be visible or easily obtainable via sensing mechanism. In other embodiments, the modeling tools enable a car or drone-based image-scanning pipeline to be input through a variety of photo, video, depth simultaneous location and mapping (SLAM) scanning in order to model the virtual replicas <NUM>. In other embodiments, radar-imaging, such as synthetic-aperture radars, real-aperture radars, Light Detection and Ranging (LIDAR), inverse aperture radars, monopulse radars, and other types of imaging techniques may be used to map and model static objects before integrating them into the persistent virtual world system. Utilizing these more technical solutions may be performed especially in cases where the original models of the structures are not available, or in cases where there is missing information or there is a need to add additional information to the virtual world entities which is not provided by the CAD or CAE models.

<FIG> depict schematic representations of a system <NUM> for attaching applications and interactions to static objects, detailing the connection between a real world, virtual world, and merged world, according to an embodiment. Some elements of <FIG>3B may be similar to elements of <FIG>, and thus similar or identical reference numerals may be used to identify those elements.

The digital reality applications <NUM> are virtually attached to one or more static objects <NUM> in the real world <NUM> via the virtual replicas <NUM> stored in the virtual world <NUM>. Thus, as viewed in <FIG>, a user <NUM> not wearing or employing a client device <NUM> may approach the real-world location where the digital reality application <NUM> is virtually attached without triggering any reaction in the system <NUM> and without the user <NUM> being able to view the digital reality application <NUM>. At this moment, a virtual world <NUM> developed based on the real world <NUM> may include only information of the static objects <NUM> and digital reality applications <NUM> thereon attached.

On the other hand, as viewed in <FIG>, when the user <NUM> activates and employs a client device <NUM>, the digital reality application <NUM> detects the viewing position and orientation of the client device <NUM> and retrieves corresponding media streams <NUM> from the server <NUM>, which are adjusted in order to be aligned with the user viewing position and orientation during user interaction. Detecting a client device <NUM> activates a merged world <NUM>, whereby the movement of the user <NUM> in the real world <NUM> is registered in real-time in the virtual world <NUM>. The tracking <NUM> may, for example, include tracking the rotation <NUM> and translation <NUM> of a user's movements, which may be necessary in order to adjust the viewing position and orientation of the media streams <NUM> delivered to the client device <NUM>.

According to an embodiment, the digital reality applications <NUM> may utilize one or more of a plurality of techniques to broadcast a signal to the client devices <NUM>, alerting the client devices <NUM> that a digital reality application <NUM> is available in proximity to the location of the client devices <NUM>. In one embodiment, if the user <NUM> has previously subscribed to an application in the server <NUM>, the location of the client device <NUM> may then at all times be available at the persistent virtual world system stored in the server <NUM>. Therefore, when a user <NUM> approaches a digital reality application <NUM>, the digital reality application <NUM> may already prepare to broadcast the signals to the client device <NUM>. In another embodiment, if the user <NUM> has previously subscribed to the server <NUM> or to one or more specific digital reality applications <NUM> from the server <NUM>, the digital reality application <NUM> may constantly search for registered devices prior to broadcasting the signal. In one embodiment, as a user <NUM> with a client device <NUM> enters a location where the digital reality application <NUM> is active, the digital reality application <NUM> may detect a signal from the client device <NUM>, indicating the device can receive a digital reality application <NUM> media stream prior to broadcasting the signal.

As way of example, the digital reality application <NUM> is virtually attached to a static shoe placed in a specific place in a park. Without the client device <NUM>, the user <NUM> may only view a shoe, but when the user employs the client device <NUM>, the user <NUM> may view a digital reality application <NUM> that may be represented by the shoe glowing or emitting a sound. The user <NUM> may be alerted by the digital reality application <NUM> about the presence of the digital reality application <NUM> and the user <NUM> may thereafter touch a virtual, graphical representation of the same, which may trigger other forms of interaction. For example, after the user <NUM> touches the graphical representation of the digital reality application <NUM>, the graphical representation may increase in size, may enable the user <NUM> to rotate the shoe, view the shoe characteristics in the form of text, view videos about the shoe, change some characteristics of the shoe, such as color, size, or style, and may purchase the selected shoe. The shoe may remain in the specific place in the park or elsewhere until decided by the shoe provider, and may thereafter move the shoe to another place, for which the shoe location and space settings may need to be updated in the server <NUM>. In other examples, the shoe may be viewed completely in virtual reality, such as showing the shoe in a different setting, such as in a desert, mountain, or another city.

<FIG> depicts a schematic representation of a system <NUM> for attaching applications and interactions to static objects, detailing operational components of a client device, according to an embodiment.

A client device <NUM> may include operational components such as an input/output (I/O) module <NUM>; a power source <NUM>; a memory <NUM>; sensors <NUM> and transceivers <NUM> forming a tracking module <NUM>; and a network interface <NUM>, all operatively connected to a processor <NUM>.

The I/O module <NUM> is implemented as computing hardware and software configured to interact with users and provide user input data to one or more other system components. For example, I/O module <NUM> may be configured to interact with users, generate user input data based on the interaction, and provide the user input data to the processor <NUM> before being transferred to other processing systems via a network, such as to a server. In another example, I/O modules <NUM> is implemented as an external computing pointing device (e.g., a touch screen, mouse, 3D control, joystick, gamepad, and the like) and/or text entry device (e.g., a keyboard, dictation tool, and the like) configured to interact with client devices <NUM>. In yet other embodiments, I/O module <NUM> may provide additional, fewer, or different functionality to that described above.

The power source <NUM> is implemented as computing hardware and software configured to provide power to the client devices <NUM>. In one embodiment, the power source <NUM> may be a battery. The power source <NUM> may be built into the devices or removable from the devices, and may be rechargeable or non-rechargeable. In one embodiment, the devices may be repowered by replacing one power source <NUM> with another power source <NUM>. In another embodiment, the power source <NUM> may be recharged by a cable attached to a charging source, such as a universal serial bus ("USB") FireWire, Ethernet, Thunderbolt, or headphone cable, attached to a personal computer. In yet another embodiment, the power source <NUM> may be recharged by inductive charging, wherein an electromagnetic field is used to transfer energy from an inductive charger to the power source <NUM> when the two are brought in close proximity, but need not be plugged into one another via a cable. In another embodiment, a docking station may be used to facilitate charging.

The memory <NUM> may be implemented as computing hardware and software adapted to store application program instructions. The memory <NUM> may be of any suitable type capable of storing information accessible by the processor <NUM>, including a computer-readable medium, or other medium that stores data that may be read with the aid of an electronic device, such as a hard-drive, memory card, flash drive, ROM, RAM, DVD or other optical disks, as well as other write-capable and read-only memories. The memory <NUM> may include temporary storage in addition to persistent storage.

The sensors <NUM> may be implemented as computing hardware and software adapted to obtain data from the real world. In some embodiments, one or more of the sensors <NUM> may determine/track the position and orientation of the client devices <NUM> and send that information to the server to determine the position and orientation of the client device. For example, the sensors <NUM> may include one or more cameras, such as one or more depth cameras. The sensors may also include one or more Inertia Measuring Units (IMUs), accelerometers, and gyroscopes. The IMU is configured to measure and report the velocity, acceleration, angular momentum, speed of translation, speed of rotation, and other telemetry metadata of client devices <NUM> by using a combination of accelerometers and gyroscopes. Accelerometers within the IMU and/or configured separate from the IMU may be configured to measure the acceleration of the interaction device, including the acceleration due to the Earth's gravitational field. In one embodiment, accelerometers include a tri-axial accelerometer that is capable of measuring acceleration in three orthogonal directions.

The transceivers <NUM> may be implemented as computing hardware and software configured to enable devices to receive wireless radio waves from antennas and to send the data back to the antennas. In some embodiments, mmW transceivers may be employed, which may be configured to receive mmW wave signals from antennas and to send the data back to antennas when interacting with immersive content. The transceiver <NUM> may be a two-way communication transceiver <NUM>.

In an embodiment, the tracking module <NUM> may be implemented by combining the capabilities of the IMU, accelerometers, and gyroscopes with the positional tracking provided by the transceivers <NUM> and the accurate tracking, low-latency and high QOS functionalities provided by mmW-based antennas may enable sub-centimeter or sub-millimeter positional and orientational tracking, which may increase accuracy when tracking the real-time position and orientation of client devices <NUM>. In alternative embodiments, the sensing mechanisms and transceivers <NUM> may be coupled together in a single tracking module device.

The network interface <NUM> may be implemented as computing software and hardware to communicatively connect to a network, receive computer readable program instructions from the network sent by the server or by client devices <NUM>, and forward the computer readable program instructions for storage in the memory <NUM> for execution by the processor <NUM>.

The processor <NUM> may be implemented as computing hardware and software configured to receive and process data from sensing mechanisms and digital reality application data and instructions. For example, the processor <NUM> may be configured to provide imaging requests, receive imaging data, process imaging data into environment or other data, process user input data and/or imaging data to generate user interaction data, perform edge-based (on-device) machine learning training and inference, provide server requests, receive server responses, and/or provide user interaction data, environment data, and content object data to one or more other system components. For example, the processor <NUM> may receive user input data from I/O module <NUM> and may respectively implement application programs stored in the memory <NUM>. In other examples, the processor <NUM> may receive data from sensing mechanisms captured from the real world, or may receive an accurate position and orientation of client devices <NUM> through the tracking module <NUM>, and may prepare some of the data before sending the data to a server for further processing. In other examples, the processor <NUM> may perform edge-based rendering of media streams received from the server while executing the digital reality applications. In other examples, the processor <NUM> may receive media streams rendered by the server, and may perform lightweight operations on the media streams in order to output the media streams.

<FIG> depicts a block diagram of a method <NUM> for attaching applications and interactions to static objects, according to an embodiment. Method <NUM> may be implemented by a system, such as systems described with reference to <FIG>.

According to an embodiment, a method <NUM> for attaching digital reality applications and interactions to static objects starts in blocks <NUM> and <NUM> by creating, by a developer via a replica editor stored in a server, virtual replicas of static objects selected amongst real-world elements. Subsequently, method <NUM> continues in block <NUM> by adding virtual replica real-world properties, including location and space settings, physics settings, and 3D data structure. Then, in block <NUM>, the method <NUM> continues by attaching, by a developer via a replica editor stored in the server, digital reality applications to the virtual replicas.

In block <NUM>, the method <NUM> continues by a user approaching and looking at a digital reality application. In some embodiments, the user may approach the digital reality application in a merged reality. In other embodiments, the user may approach the digital reality application in virtual reality. Approaching the digital reality application triggers the digital reality application to detect and track the client device position and orientation, and to send the client position and orientation to the server, as viewed in block <NUM>. In block <NUM>, the server retrieves and tracks the viewing position and orientation from client device, and then, in block <NUM>, sends application media streams to the client device adjusted and aligned to the viewing position and orientation of the user. Finally, in blocks <NUM> and <NUM>, the user interacts with the digital reality application via the client device.

Claim 1:
A system comprising:
a client device (<NUM>) and one or more servers (<NUM>) comprising a memory (<NUM>) and a processor (<NUM>);
wherein the one or more servers (<NUM>) contain a persistent virtual world system storing virtual replicas (<NUM>) of static objects (<NUM>) of the real world (<NUM>), the virtual replicas comprising a static object location and space (<NUM>), physics settings (<NUM>) specifying physical properties of the virtual replicas, and a 3D data structure (<NUM>), wherein the persistent virtual world system is configured to virtually attach digital reality applications (<NUM>) on the virtual replicas, such that as the client device (<NUM>) approaches an object to which a digital reality application is virtually attached, the digital reality application detects the physical position and orientation of the client device, triggering the one or more servers to transmit digital reality application content to be displayed by the client device to the client device, wherein the transmitted digital reality application content is aligned with the detected physical position and orientation of the client device; and
wherein the system is configured such that the client device (<NUM>) downloads the digital reality application (<NUM>) or parts of the digital reality application and performs client-side computations and rendering after pre-determined levels of interaction with the digital reality application.