Patent Description:
Augmented reality (AR) is a live view of a physical, real-world environment whose elements are augmented or supplemented by computer-generated sensory input such as sound, video, graphics or Global Positioning System (GPS) data to enhance a viewer's perception of reality. Virtual reality (VR) is a computer technology that simulates a user's physical presence and environment and replaces the real world with a simulated one. AR and VR technologies make the information about the surrounding real world of the user interactive and digitally manipulable.

A method and apparatus for generating a virtual environment for controlling electronic devices is described in <CIT> in which user interactions with virtual representations of electronic devices in a virtual environment cause controlling of one or more electronic devices in the real world.

<CIT> describes a system and method for assigning a virtual user interface to a physical object in which a virtual user interface for a physical object is created and is associated with identifiers of the physical object and tracking data related to the physical object.

<CIT> describes glasses which can be paired with electronic devices. The glasses comprise: an image obtaining unit which obtains an image of an electronic device; an identification unit for extracting identification information of the electronic device from the image; a pairing target selecting unit for selecting from stored identification information an electronic device corresponding to the extracted identification information; and a wireless communications unit for pairing the glasses with the selected electronic device. <CIT> is cited under Art. <NUM>(<NUM>) EPC and describes methods and systems for virtualizing electronic devices for utilization in virtual environments. An electronic device is identified and a corresponding virtual object obtained for rendering within an immersive virtual environment. The virtual object can include an interactive portion bound to a corresponding interactive portion of the identified electronic device, such that virtual interactions therewith facilitate real interactions with the identified electronic device. <CIT> Al describes a method for presenting text information on a head-mounted display, comprising: rendering a view of a virtual environment to the head-mounted display; tracking an orientation of the head-mounted display; tracking a gaze of a user of the head-mounted display; processing the gaze of the user and the orientation of the head-mounted display, to identify a gaze target in the virtual environment towards which the gaze of the user is directed; receiving text information for rendering on the head-mounted display; presenting the text information in the virtual environment in a vicinity of the gaze target. <CIT> Al describes an electro-mechanical device for providing an input to a computer program and said computer program providing a tactile output through said electro-mechanical device to a user. An electro-mechanical virtual sword game apparatus is provided that receives positional information from sensors on the sword apparatus and the sword apparatus contains a propulsion gyrostat that under the control of a computer process may be topple to provide a torque on the housing of the sword apparatus that may be used to simulate the impact of sword blows. <CIT> Al describes calculating a moving direction of an input device based on motion information obtained from predetermined detection means for detecting attitude or motion of the input device operated by a user. Then, the object in a virtual three-dimensional space is caused to move to a position which is obtained by hypothetically moving a position of the object based on the direction in which the input device has been moved and then by correcting the position of the object hypothetically moved, only in a direction perpendicular or substantially perpendicular to the direction in which the input device has been moved.

Accordingly there is provided a method, a device , and a non-transitory computer readable medium as detailed in the claims that follow. Advantageous features are provided in the dependent claims.

The present disclosure is directed to methods, apparatuses and computer-readable media for virtualizing one or more devices via an augmented reality (AR) or virtual reality (VR) virtualized or partially virtualized device (referred to as "an AR/VR device"). The techniques described in this disclosure can provide virtualisation of a real-word device (referred to as a virtual or virtualized device) using, in part, a model that describes the device's material composition and how it can interact with in an AR/VR platform. The model defines a mapping from user input detected by the AR/VR platform to the input expected by the core functionality of the virtual device, for example, based on computer vision or other techniques.

In particular, the present disclosure introduces the concept of a "devicelet" as a successor to or further development of the ubiquitous smartphone "app. " The devicelet can provide a user a virtualized device that includes a virtual representation of a real-world, physical device. The physical device being virtualized can include any physical objects suitable for representation, such as computers, smartphones, tablets, TVs, and projectors, among others. In some implementations, the device being virtualized can include or be associated with a dummy device such as, a keyboard, mouse, touch pad, track pad, and touch screen. Using the dummy device, users can be provided a physical or tactile connection to the virtualized device.

The devicelet includes core functionality that is accessed by a user via the AR/VR device. For example, the user can interact with the virtualized device via the AR/VR device (e.g., a wearable AR headset or a non-wearable VR device). In some implementations, user actions can be detected and interpreted by the hardware of the AR/VR device. The devicelet can convert user actions made or indicated with respect to the virtual representation of the virtualized device into user actions corresponding to the underlying core functionality of the devicelet.

In some implementations, a devicelet can include three components: (i) core functionality; (ii) a virtual model for AR/VR representation; (iii) an emulation component. Core functionality includes the code that allows a devicelet to perform its main functionality. The core functionality may, for example, relate to a whiteboard, a watch, an advertising board, a picture frame, a musical instrument, a notebook, a newspaper, a book or other visualized devices.

In the example of a virtualized electronic whiteboard, this core functionality may encompass processing user touch input as a series of vectors which are then visualized for the user as virtual pen marks, erasing pen marks with a particular gesture or input, and also the saving or restoring of whiteboard sessions for the user. In the example of a virtualized smart phone, the core functionality would be the phone's Operating System (OS). The code for a devicelet's core functionality could be taken directly from the devicelet's equivalent real-world counterpart (where available), having been re-compiled for the architecture of the hosting AR/VR device, or could be written from scratch to exist solely as a virtual device.

The virtual model may include, for example, a virtual screen, frame, or window with optional additional visual embellishments creating the virtual facia of the device. The virtual model can include the visual or other sensual representation of the virtual device.

The emulation component or layer can map user interactions with respect to the virtual representation to inputs expected by the core functionality. For example, the emulation layer can map the interactions by means of gestures, virtualized menus, keypads, keyboards, or control panels to the core functionality.

In some implementations, a devicelet may be implemented, for example, by taking an existing core functionality (e.g., a smartphone app) and applying a wrapping process to embed the emulation component and the associated virtual model.

In some implementations, a devicelet can be realized in an AR/VR platform using a placeholder. In this example, a placeholder is the physical "canvas" on which the AR/VR platform can realize or represent the virtual model of the devicelet. Moreover, the placeholder may provide a physical surface, context and tactile feedback for interaction with the end user, and may also provide input and sensor feedback to the AR/VR platform. The placeholder can be dumb (i.e., containing no electronics or sensors) or partially intelligent, achieving some limited functionality, typically for interfacing with the AR/VR platform.

In some implementations, a devicelet can be secured using security technologies, such as app wrapping or an extension of a software development kit (SDK) with mobile app security, to provide mobile security in AR/VR platforms.

In some implementations, the devicelet can be platform-agnostic. The devicelet can be implemented on various AR/VR devices. The devicelet can be transferrable among different operating systems. In some implementations, a single AR/VR device may be able to support multiple devicelets that provide visualizations of multiple devices simultaneously. In some implementations, interactions between two or more virtualized devices can be implemented.

The devicelet can be implemented in various forms, such as, a hosted (thick client) devicelet, a remote (thin client) devicelet, and a hybrid devicelet. For example, the hosted (thick client) devicelet includes an architecture to host a virtual device on a physical AR/VR device that leverages computer vision to simulate virtualized components. The remote (thin client) devicelet has an architecture that includes a client virtual device residing on a physical AR/VR device and a remote server that performs substantial computations. A hybrid devicelet can have a combination of the hosted and remote architectures.

<FIG> is a schematic representation of an example hosted devicelet architecture <NUM>. The hosted devicelet architecture <NUM> includes an AR headset <NUM> and a hosted devicelet <NUM>. Although the examples described in this disclosure uses an AR headset as the example AR/VR device or platform, the AR/VR device can include other types of fully or partially visualized devices, such as handheld AR/VR devices and wearable or non-wearable AR/VR devices.

A hosted devicelet <NUM> is a thick client that executes the majority of operations relating to the devicelet locally, on the AR headset device <NUM>. This type of devicelet is suitable, but not required, for applications that require an instant response to user interactions (for example, by taking advantage of the low latency design), have a dependency on a number of emulated hardware features (such as a gyroscope or compass), or have multiple complex user interactions/gestures.

The AR headset device <NUM> can include, for example, headset hardware <NUM>, headset core functionality <NUM>, and headset device emulation layer <NUM>. The headset hardware <NUM> includes physical hardware components, such as, one or more of a screen <NUM>, a camera <NUM>, a microphone, a speaker, a connectivity or communication (e.g., GPS, Bluetooth, WiFi, or cellular) interface, storage modules (e.g., a random access memory (RAM) and file/data store) and a processor (e.g., a central processing unit (CPU)). The headset core functionality <NUM> includes, for example, providing AR or VR representations by an AR display module <NUM>, processing and analyzing images, videos, or other media by a computer vision module <NUM>, and managing hardware included or otherwise associated with the AR headset device <NUM> by a hardware management module <NUM>. The hardware management module <NUM> can perform a number of tasks for the hosting AR/VR device <NUM> as well as the hosted devicelet <NUM>. For example, the hardware management module <NUM> facilitates access to the physical resources of the hosting hardware by its own underlying Operating System (OS) and also manages the access of devicelets (e.g., via the device connectivity emulation <NUM>) and device resource emulation <NUM>. In addition, the hardware management module <NUM> will pass data (e.g., I/O data) between physical placeholder dummy devices and their paired devicelets and also provide the data channel, for example, for a remotely hosted or hybrid devicelet, between the hosting client AR/VR device and the server-side component.

The headset device emulation layer <NUM> includes a software layer that enables devicelets to interpret the world around their virtual projections and how users interact with those virtualized projections. The headset device emulation layer <NUM> can additionally maintain devicelets' access to the physical resources found on the hosting hardware. The headset device emulation layer <NUM> includes, for example, device display emulation module <NUM>, device user interaction emulation module <NUM>, device location and environment emulation module <NUM>, device connectivity emulation module <NUM>, and device resource emulation module <NUM>.

For instance, the device display emulation module <NUM> can illustrate the device and its virtual display (e.g., the AR display <NUM>). For example, the device display emulation module <NUM> can interpret the look and feel of the virtualized device and pass it to the hosting hardware to be drawn to screen.

The user interaction emulation module <NUM> can identify, process, and respond to inputs to device based on user actions (e.g., changing the location, orientation or other features of the virtualized device based on the user inputs). For example, the user interaction emulation <NUM> can interpret the user's touches, gestures and other physical interactions with the virtualized device that are received, for example, from the hosting hardware's computer vision module <NUM> and pass the interaction data to the devicelet via the input/output (I/O) application program interface (API) <NUM>.

The device location and environment emulation module <NUM> can simulate the location and environment of the device. The device location and environment emulation module <NUM> can interpret the resultant movement of the virtualized device that are received, for example, from the hosting hardware's computer vision module <NUM>, and passes the motion, orientation and acceleration data to the devicelet via the movement API <NUM>.

The device connectivity emulation module <NUM> can provide connectivity capabilities that include connectivity capabilities physically-enabled by the device being virtualized and may also include virtual connectivity capabilities, for example, enabled by the AR headset device <NUM>. For example, the device connectivity emulation module <NUM> manages access of the connectivity technologies of the hosting hardware by devicelets via the connectivity API <NUM>.

The device resource emulation module <NUM> can provide and simulate resources available or accessible by the device. For example, the device resource emulation module <NUM> manages access of the storage resources of the hosting hardware by devicelets via the resource API <NUM>.

The devicelet <NUM> can include, for example, hardware emulation APIs <NUM>, device hardware glue logic <NUM>, virtual model of device <NUM>, and device core functionality <NUM>. The hardware emulation APIs <NUM> includes, for example, I/O API <NUM>, movement API <NUM>, connectivity API <NUM>, and resource API <NUM>. These API s can provide input to and from the emulation and hardware logic.

In some implementations, the I/O API <NUM> can connect virtualized components presented in the 3D model to the device core and activate their various characteristics. For example, if a virtual LED is instructed to turn on, the I/O API <NUM> can connect the request with the correct virtualized LED on the virtualized model and make it appear illuminated. As another example, if a touch is made on a devicelet's virtualized touch-sensitive display, the I/O API <NUM> can receive the interaction from the emulation layer and then convert the interactions to location / pressure data for the device core. If a virtual display needs to refresh its current screen, the I/O API <NUM> can receive new display data from the device core, and then convert the new display data into a format that the emulation layer can utilize, which is then passed to the emulation layer for displaying on the virtualized device's surface.

In some implementations, if a virtual device contains a virtualized electromagnet, rumble motor or exterior motors, then those interactions with the user may need further interpretation within the emulation layer <NUM>. For example, other virtualized objects could be attracted to a visualized electromagnet. Virtual rumble motors may need some on-screen visual indication to the user when enabled. Virtual motors on the exterior of the virtualized device may cause the virtual device to move location without interaction by the user.

The movement API <NUM> can receive motion, orientation and acceleration data from the emulation layer <NUM> and presents the data to the device core <NUM>, for example, in terms of the various forces the virtualized motion technologies can understand. The movement API <NUM> can send data to the device core <NUM> that relates to the movement and forces acting upon the virtualized device. For example, in the case of a compass, the data would be the new direction that the virtual device is facing. In the case of an accelerometer, the data can include changes in velocity of the virtual device. In some implementations, the data can also include data associated with altimeters.

The connectivity API <NUM> can serve as a bridge between the specific connectivity technologies (e.g., Bluetooth, near filed communications (NFC), GPS, WiFi, cellular communications, and cellular communications) the device core <NUM> is expecting to have access to and the actual physical components that the hosing hardware has available.

Similar to the connectivity API <NUM>, the resource API <NUM> can form a bridge between the format of the storage of the device core <NUM> is expecting and the actual format the hosting hardware utilizes.

The device hardware glue logic <NUM> can include logic that handles interactions associated with hardware such as, touch surfaces, physical buttons, LEDs, screens, vibration generators, proximity sensors, motion detectors, compass, gyroscope, microphone, speaker, connectivity or communication (e.g., GPS, Bluetooth, WiFi, or cellular) interfaces, storage modules (e.g., a RAM and file/data store) and processors (e.g., CPUs).

The device core functionality <NUM> can include the main logic to perform specialized functionality at the device core <NUM>. For example, if the devicelet provides a virtualized smartphone or tablet, the device core <NUM> can be running and managing applications of the smartphone or tablet. The device core <NUM> can be ported directly from existing real-world devices (e.g., the smartphone or tablet) or created especially for use in a virtual environment.

Some devices need particular hardware elements emulated for them. An example of this would be a virtual device which includes a gyroscope, whereby the gyroscope component does not actually exist. In this example, the AR headset <NUM> would instead compute the movements and interactions of the virtual device and emulate the output of a gyroscope that would have been present within the virtual device based on the orientation of the virtual device in the AR space. In this example, the main component of this logic can be carried out within the headset device emulation layer <NUM>. The emulated output can be subsequently passed to a target devicelet <NUM> via a defined movement API <NUM>. Then the devicelet <NUM> can bridge this data via the device hardware glue logic <NUM> to the device core <NUM>. In some implementations, dummy devices may have some functionality available to provide feedback, such as a gyroscope.

In a similar manner, to provide virtualized devices that have access to their own connectivity hardware such as Bluetooth, Wi-Fi, Cellular and GPS that do not physically exist, the AR headset <NUM> can make its own connectivity hardware available via the headset device emulation layer <NUM> and a connectivity API <NUM>. For example, where the devicelet <NUM> is a virtualized mobile telephone, the devicelet <NUM> would use the AR headset's connectivity hardware as its own.

In some implementations, the AR headset <NUM> requires a three-dimensional (3D) model or wireframe (e.g., virtual model of device <NUM>) to virtualize a device and display it for the user. This 3D model <NUM> can outline surfaces for particular user interactions that can, then, be monitored for gestures made by the user. The 3D model <NUM> can include information about, for example, how the devicelet <NUM> tracks relatively to a real-world object, placeholder device or set of markers, details of materials used for each surface, and how the surfaces respond to touch, physical items such as buttons, switches LEDs and surface sensors. In some implementations, details about virtual menus available to the devicelet (for example, in order to change the look and feel of the facia of the device and other visual embellishments) can be contained within a separate settings XML file.

<FIG> is a schematic representation of an example client-side architecture <NUM> of a remote devicelet. A remote devicelet includes a thin client <NUM> (e.g., a client-side devicelet <NUM>) that offloads the majority of its work to a remote server (e.g., a server <NUM> in <FIG>). The client-side devicelet <NUM> can redirect user interactions and local headset hardware connectivity to the remote server which hosts the devicelet's core functionality. In some implementations, this remote type of devicelet suits more traditionally virtualized devices, such as Remote Desktop Protocol (RDP) personal computers (PCs) and Macs (e.g., remotely accessible machines over technologies such as Microsoft Remote Desktop Connection, Apple's Screen Share, Team Viewer or 'Go To My PC'). RDP devices can be hosted in a server rack and accessed by another client machine over software based on the RDP protocol. This remote type of devicelet can be used for virtualized devices that require minimal or basic user interactions, such as mouse movement or keyboard entry and in return update a well-defined screen or region.

The example client-side architecture <NUM> of the remote devicelet includes an AR headset <NUM> and a client-side remote devicelet <NUM>. The AR headset <NUM> can be similar to, or different from, the AR headset device <NUM> in the hosted devicelet architecture <NUM> as shown in <FIG>. As illustrated in <FIG>, the AR headset <NUM> can include, for example, headset hardware <NUM>, headset core functionality <NUM>, and devicelet stream management <NUM>. The headset hardware <NUM> includes, for example, one or more of a screen <NUM>, a camera <NUM>, a microphone, a speaker, and a connectivity or communication (e.g., GPS, Bluetooth, WiFi, or cellular) interface. The headset core functionality <NUM> includes, for example, AR display <NUM>, computer vision <NUM>, and hardware management <NUM>, and client-side connectivity management <NUM>. The devicelet stream management <NUM> includes, for example, client-side connectivity management <NUM> that can provide remote connectivity <NUM> to a remote server which hosts the devicelet's core functionality.

The client-side remote devicelet <NUM> can include, for example, a remote devicelet data interchange API <NUM> and remote server and devicelet connectivity details <NUM>. The connectivity details <NUM> can include, for example, the location of the server (IP Address / URL) and the services that need to be consumed. The client-side remote devicelet is light in regards to the core computational burden put on the AR headset <NUM>. In a basic form, the client-side remote devicelet <NUM> redirects user interactions and related hardware responses to a server component (e.g., a server <NUM> in <FIG>) and relays the response to the screen and other local hardware of the AR headset <NUM>. The client-side connectivity management <NUM> redirects requests, made by one or more local devicelets (e.g., the devicelet <NUM>), to their server side counterparts and returns any relevant responses via the data interchange API <NUM>. The remote server can perform, for example, data processing similar to that of a hosted devicelet (e.g., the devicelet <NUM> in <FIG>), and return the streamed visual response to the client headset's AR display (e.g., the AR display <NUM>).

<FIG> is a schematic representation of an example server-side architecture <NUM> of a remote devicelet. The example server-side architecture <NUM> includes a server <NUM> and a server-side remote devicelet <NUM>. The server <NUM> can include, for example, server hardware <NUM>, devicelet stream management <NUM>, server core functionality <NUM>, and server device emulation layer <NUM>. The server hardware <NUM> can include one or more of, for example, a connectivity or communication (e.g., Ethernet, GPS, Bluetooth, WiFi, or cellular) interface <NUM>, storage modules (e.g., a RAM <NUM> and file/data store <NUM>) and a processor (e.g., a CPU <NUM>). The devicelet stream management <NUM> includes, for example, devicelet connectivity management <NUM>. The devicelet connectivity management <NUM> can provide server-side remote connectivity <NUM> to the client-side remote devicelet (e.g., the client-side remote devicelet <NUM> in <FIG>). For example, the devicelet connectivity management <NUM> receives requests made by client-side devicelet <NUM> and redirects those requests to the appropriate emulation component. The server device emulation layer <NUM> makes sense of the data provided, communicates with the target server-side devicelet component via an appropriate API and returns corresponding responses. For example, as part of its response to requests, the server-side devicelet <NUM> can effectively send an audio and video stream response to the client-side devicelet <NUM> via the connectivity management <NUM>, so that the computational load is light on the client-side devicelet <NUM>.

The server core functionality <NUM> includes, for example, hardware management <NUM>. The server device emulation layer <NUM> includes, for example, device display emulation module <NUM>, device user interaction emulation module <NUM>, device location and environment emulation module <NUM>, device connectivity emulation module <NUM>, and device resource emulation module <NUM>. The server device emulation layer <NUM> and its components can be implemented in a similar manner as their thick-client counterparts as described with respect to <FIG>.

The server-side remote devicelet <NUM> can include, for example, hardware emulation APIs <NUM>, device hardware glue logic <NUM>, virtual model of device <NUM>, and device core functionality <NUM>. The hardware emulation APIs <NUM> includes, for example, I/O API <NUM>, movement API <NUM>, connectivity API <NUM>, and resource API <NUM>. The device hardware glue logic <NUM> includes logic handling interactions associated with hardware such as touch surfaces, physical buttons, LEDs, screens, vibration generators, proximity sensors, motion detectors, compass, gyroscope, microphone, speaker, connectivity or communication (e.g., GPS, Bluetooth, WiFi, or cellular) interfaces, storage modules (e.g., RAM and file/data store) and processors (e.g., CPUs).

The device core functionality <NUM> includes the main logic to perform specialised functionalities at the device core <NUM>. The device core <NUM> can reside on the server-side remote devicelet <NUM>.

In some implementations, a devicelet can have a hybrid architecture where varying degrees of functionality are carried out locally (such as the IO and display emulation functions that need minimal latency), while other functionalities are performed remotely by the server. One advantage of such systems may be to use currently existing hardware capable of some, but not all, functionality in a devicelet implementation. By using existing capabilities for devices, costs can be reduced with a potentially faster response time by executing at least some device functionality locally.

<FIG> is a flow chart of an example illustrating an example method <NUM> for virtualizing a physical device. The example method <NUM> can be performed, for example, by an AR/VR device (e.g., the AR headsets <NUM> and <NUM> in <FIG> and <FIG>), a devicelet (e.g., the devicelet <NUM> or <NUM>) fully or partially residing on the AR/VR device, or other data processing apparatus associated with the AR/VR device. For instance, some or all operations of the example method <NUM> can be implemented by a hosted devicelet (e.g., the devicelet <NUM>) or a remote devicelet (e.g., the client-side remote devicelet <NUM>), or a hybrid devicelet.

At <NUM>, a physical device is paired with the AR/VR device. The physical device can be a different, separate device from the AR/VR device. The physical device can be paired with the AR/VR device via wired or wireless technology to establish a communication channel between the physical device and the AR/VR device. For example, the physical device can be paired with the AR/VR device via Universal Serial Bus (USB) or Bluetooth connection.

The AR/VR device can include any AR or VR virtualized or partially virtualized devices. For example, the AR/VR device can wearable or non-wearable devices, such as, an AR headset, an AR/VR glasses/goggles/lens, or AR/VR enabled handheld devices. The physical device can include dummy or intelligent devices. For example, the physical device can include one or more of a keyboard, a mouse, a touch pad, a track pad, a touch screen, a camera, a sensor, a computer, a smartphone, a tablet, or an Internet of Things (IoT) device. The physical device can be virtualized by the AR/VR device, for example, by providing a virtual representation of the physical device to a user. In some implementations, the physical device can be virtualized by the AR/VR device before the AR/VR device receives a user interaction with the physical device. In some implementations, a user's interaction with the physical device can be in response to the virtual representation of the physical device to the user.

At <NUM>, first data of a user interaction with the physical device is received by the AR/VR device from the physical device. For example, the user interaction with the physical device can include a touch or slide of a touch pad, a press of a physical keyboard, or a gesture in front of a camera or a motion sensor, etc. The physical device can detect the user interaction, record the user action as the first data, and transmit the first data to the AR/VR device, for example, through the communication channel established between the paired physical device and AR/VR device.

At <NUM>, the first data of the user interaction with the physical device is converted into second data for representation of the user interaction with the physical device in a virtual representation of the physical device. In some implementations, converting the first data of the user interaction with the physical device into the second data further includes parsing the first data of the user interaction with the physical device, and determining from which physical device the user interaction is received. For example, the hardware management module <NUM> of the AR headset <NUM> may parse the first data and identify which physical device, among multiple devices that the AR headset <NUM> is connected to, the first data came from, for example, based on an identifier of the physical device included in the first data or certain specific format or feature of the first data.

In some implementations, converting the first data of the user interaction with the physical device into the second data can be performed, for example, by an emulation module (e.g., the device user interaction emulation <NUM> or <NUM>). For example, the emulation module can receive the first data and translate the first data of the user interaction into the second data that is suitable for representation of the user interaction with the physical device in the virtual representation of the physical device. In some implementations, the second data can be further input into, or processed by, one or more of an I/O API (e.g., the I/O API <NUM> or <NUM>), a virtual model of the physical device (e.g., the virtual model <NUM> or <NUM>), device hardware glue logic (e.g., the device hardware glue logic <NUM> or <NUM>) and the device core (e.g., the device core <NUM> or <NUM>) to provide the representation of the user interaction with the physical device in the virtual representation of the physical device.

As an example, if the physical device is a smartphone, the AR/VR device can provide the user with a virtual representation, such as, an AR/VR projection of the smartphone on a wall, a desk, or any other surface or in a <NUM>-dimensinoal space. The user interaction may include typing a text message using the physical keyboard or touch screen of the smartphone. As such, the first data of the user interaction with the physical device can include the corresponding key entries that the user pressed on the smartphone. The AR/VR device can receive the first data from the smartphone and parse the first data. An emulation module can map the first data into second data suitable for a virtual representation of the user's interaction with the smartphone. For example, the virtual representation of the user interaction can include displaying, in the AR/VR projection of the smartphone, a virtual hand typing the text message in the same manner as the user typing the text message using the physical keyboard or touch screen of the smartphone in real time.

In some implementations, converting the first data of the user interaction with the physical device into the second data can be performed, for example, by a hosted, remote, or hybrid devicelet. For instance, in the case of a remote devicelet, the conversion can be at least partially performed by a remote server (e.g., the remote server <NUM>). For example, the conversion of the first data of the user interaction with the physical device into the second data for representation of the user interaction with the physical device can include redirecting, to a remote server, the first data of the user interaction with the physical device; and receiving, from the remote server, the second data for representation of the user interaction with the physical device in the virtual representation of the physical device.

At <NUM>, the virtual representation of the physical device and the representation of the user interaction with the physical device are output by the AR/VR device. The virtual representation of the physical device and the representation of the user interaction with the physical device are rendered by an AR/VR display (e.g., the AR display <NUM> or <NUM>) of the AR/VR device. The representation of the user interaction with the physical device can be embedded or otherwise integrated in the virtual representation of the physical device.

At <NUM>, a tactile feedback in response to the user interaction with the physical device is output to the physical device. In some implementations, the tactile feedback can be provided in addition to the representation of the user interaction with the physical device in the virtual representation of the physical device. The additional tactile feedback allows the user to have more tangible and precise control of the physical device and, thus, enhance user experience.

In addition to the user interaction with the physical device (e.g., the smartphone), the user can interact with the virtual representation of the physical device (e.g., the AR/VR projection of the smartphone). For example, instead of pressing the buttons on the physical keyboard or touchscreen of the smartphone, the user may touch, point, or otherwise interact with the virtual keyboard or touchscreen of the AR/VR projection of the smartphone. In some implementations, the AR/VR device can detect such a second user interaction with the virtual representation of the physical device (e.g., from a camera, a motion sensor, or other hardware component of the AR/VR device). The AR/VR device can receive and parse third data of the second user interaction with the virtual representation of the physical device, for example, by the hardware management module <NUM> or <NUM>. The third data of the second user interaction with the virtual representation of the physical device can be converted into fourth data for a second representation of the second user interaction with the virtual representation of the physical device in the virtual representation of the physical device, for example, by one or more of an emulation component (e.g., the device user interaction emulation <NUM> or <NUM>), API (e.g., the movement API <NUM> or <NUM>), virtual model (e.g., the virtual model <NUM> or <NUM>), device hardware glue logic (e.g., the device hardware glue logic <NUM> or <NUM>) and the device core (e.g., the device core <NUM> or <NUM>). The second representation of the second user interaction with the virtual representation of the physical device can be output, for example, by the AR/VR display (e.g., the AR display <NUM> or <NUM>) of the AR/VR device, in the virtual representation of the physical device.

The AR/VR device includes a physical component (e.g., a gyroscope or compass) that does not exist in the physical device. The AR/VR device provides an emulation of the physical component of the AR/VR device in the virtual representation of the physical device so as to enhance the functionality of the virtual representation of the physical device. For instance, the user can interact with the emulation of the physical component of the AR/VR device in the virtual representation of the physical device and achieve the functionality that cannot be realized by the physical device itself. As an example, the user may move, rotate, or perform other actions with the AR/VR projection of the smartphone that may trigger an emulated gyroscope.

The AR/VR device detects such a third user interaction with the emulation of the physical component of the AR/VR device in the virtual representation of the physical device (e.g., from a camera, a motion sensor, or other hardware component of the AR/VR device). Such a third user interaction with the emulation of the physical component of the AR/VR device may be represented as fifth data that includes the movements, rotations, and other interactions of the virtual representation of the physical device. The AR/VR device can receive and parse fifth data of the third user interaction with the virtual representation of the physical device, for example, by the hardware management module <NUM> or <NUM>. The fifth data of the third user interaction with the emulation of the physical component of the AR/VR device can be converted into sixth data for a third representation of the third user interaction in the virtual representation of the physical device, for example, by one or more of an emulation component (e.g., the device user interaction emulation <NUM> or <NUM>), API (e.g., the movement API <NUM> or <NUM>), virtual model (e.g., the virtual model <NUM> or <NUM>), device hardware glue logic (e.g., the device hardware glue logic <NUM> or <NUM>) and the device core (e.g., the device core <NUM> or <NUM>). The third representation of the third user interaction with the emulation of the physical component of the AR/VR device in the virtual representation of the physical device can be output, for example, by the AR/VR display (e.g., the AR display <NUM> or <NUM>) of the AR/VR device, in the virtual representation of the physical device.

Some implementations of subject matter and operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Some implementations of subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage mediums for execution by, or to control the operation of, data processing apparatus. A computer storage medium can be, or can be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal.

The apparatus and execution environment can realize various, different, computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, as well as declarative or procedural languages. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory, or both. A computer includes a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. A computer may also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, flash memory devices, and others), magnetic disks (e.g., internal hard disks, removable disks, and others), magneto optical disks , and CD ROM and DVD-ROM disks.

Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), an inter-network (e.g., the Internet), a network comprising a satellite link, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

Claim 1:
A method (<NUM>) for virtualizing a physical device using an augmented reality, AR, or virtual reality, VR, device AR/VR device, the method comprising:
receiving (<NUM>), by the AR/VR device from the physical device, first data of a user interaction with the physical device;
converting (<NUM>) the first data of the user interaction with the physical device into second data for representation of the user interaction with the physical device in a virtual representation of the physical device;
displaying (<NUM>), by the AR/VR device, the virtual representation of the physical device and the representation of the user interaction with the physical device;
the method further comprising:
receiving, by the AR/VR device, a code of core functionalities (<NUM>) of the physical device, wherein the core functionalities (<NUM>) comprise a functionality related to a hardware component that does not physically exist on the physical device; and
emulating, by the AR/VR device, the functionality related to the hardware component that does not physically exist on the physical device, wherein the emulating comprises:
receiving, by the AR/VR device, fifth data of a user interaction with the virtual representation of the physical device, the fifth data comprising an emulated output of the hardware component that does not physically exist on the physical device in response to the user interaction with the virtual representation of the physical device;
passing, via a hardware emulation application program interface, API, corresponding to the functionality related to the hardware component that does not physically exist on the physical device, the emulated output to a hardware glue logic corresponding to the hardware component that does not physically exist on the physical device; and
converting, by the hardware glue logic, the emulated output into the functionality related to the hardware component that does not physically exist on the physical device.