Method and device for obtaining real time status and controlling of transmitting devices

A method and a device for controlling at least one device in a wireless communication system are provided. The method includes rendering a digital representation of a real world environment on a display unit, wherein the digital representation includes a graphical representation of the at least one device based on a status information of the at least one device; receiving a user-input indicating control information of the at least one device; and controlling one or more operations of the at least one device in the real world environment based on the control information.

PRIORITY

This application claims priority under 35 U.S.C. § 119(a) to an Indian Patent Application filed on Jan. 16, 2017 in the Indian Patent Office and assigned Serial No. 201711001610, the entire disclosure of which is incorporated herein by reference.

BACKGROUND

1. Field of the Disclosure

The present disclosure relates generally to Internet of Things (IoT) devices and virtual reality (VR), and more particularly, to obtaining real time status and controlling of IoT devices via VR.

2. Description of the Related Art

IoT refers to the interconnection of uniquely identifiable devices using a network. Devices, popularly referred to as IoT devices or as smart devices, are embedded with electronics, software, sensors, actuators, and network connectivity that enable these devices to collect data, exchange data and be controlled over a network. Such devices include, but are not limited to, sensors, smart automation devices, wearable devices, and a smart phone.

Currently, a user may track and manage IoT devices in a home environment using a smart phone. However, the number of IoT devices that may be tracked and managed is small, for example 10 devices. For example,FIGS. 1A, 1B, and 1Cillustrate an electronic device such as a smart phone100for tracking and managing IoT devices in a home environment.

Referring toFIG. 1A, the smart phone100displays a list101of home environments such as a kitchen, a master bedroom, a garage, a living room, a kid's room, and a laundry room, having IoT devices (hereinafter interchangeably referred to as smart devices) connected with the smart phone100. The user must select an environment from the list101, for viewing the IoT devices present in the selected environment. The user selects the “Master Bed Room” environment (as indicated by a dashed rectangle).

Referring toFIG. 1B, the smart phone100then displays a list102of smart devices available in the selected “Master Bed Room” environment. The list102of smart devices only displays an operating status as ON and OFF. To view further details about any smart device, the user must select the device from the list102of smart devices. Further, if there are many devices, then the user must scroll through the list102and select a device. This process is manual, tedious, and time consuming.

Referring toFIG. 1C, the smart phone100then displays details103of the selected Coffee Maker device. However, the details103are limited to a name and an internet protocol (IP) address of the device. The details103of the device fail to provide any real time information related to the device such as real time positioning, real time orientation, and network connectivity.

Further, with advancement in technology, augmented reality (AR) and VR technologies are gaining popularity. AR technologies modify a digital view of a real world environment by superimposing virtual objects on the digital view in real time. Examples of AR include AR 3D viewers, AR browsers, and AR games. VR technologies refer to the generation of a three-dimensional image or video of a real world environment with which a user may interact using a VR device. In one embodiment, a VR device may have VR processing and VR rendering capabilities.

InFIGS. 2A and 2B, a user (201) may wear a VR device202to enjoy a VR experience, wherein the VR device includes a smart phone203mounted in a secure manner in a mounting unit205provided in a head mounted device (HMD)204, wherein the smart phone203faces lenses of the HMD204, wherein, the VR device202allows more degrees of freedom for managing IoT devices. However, the user interaction on a mobile device is limited to the touch screen area.

Additional Benefits of using VR Device includes the VR environment being similar to the real world in order to artificially create a lifelike sensory experience, which may include sight, touch, hearing, and smell senses.

Navigating in a VR environment to check IoT devices would be a real lifelike experience without physically visiting those real places.

In a VR environment a user may switch from one location to any remote location.

While the general art proposes viewing information related to smart devices in real time by use of VR along with AR, general solutions require many cameras to be placed at various geographical locations to view live streaming. Further, a camera may only capture information related to external features of any device, such as size, shape, color, etc. In particular, general solutions cannot capture and provide live updates about information related to IoT devices such as internal functioning, connectivity status, and power consumption. In addition, general solutions consume much power and network bandwidth due to real time processing of live feed/streaming from cameras for generating VR/AR views. Some general solutions necessitate the creation of virtual counterparts of real world IoT devices in a VR environment.

Thus, there exists a need for a solution to overcome the above-mentioned deficiencies and enable tracking and management of IoT devices in a home environment in a user-friendly manner. Apart from the above, it is desirable that a solution minimizes factors such as overhead, power consumption and network bandwidth consumption.

SUMMARY

An aspect of the present disclosure provides an IoT device for obtaining real time status and controlling transmitting devices (e.g., IoT devices) in the real world via VR.

In accordance with an aspect of the present disclosure, a method for controlling at least one device in a wireless communication system is provided. The method includes rendering a digital representation of a real world environment on a display unit, wherein the digital representation includes a graphical representation of the at least one device based on a status information of the at least one device; receiving a user-input indicating control information of the at least one device; and controlling one or more operations of the at least one device in the real world environment based on the control information.

In accordance with another aspect of the present disclosure, a device for controlling at least one device in a wireless communication system is provided. The device includes a user interface unit configured to receive a user-input indicating control information of the at least one device; and at least one processor configured to render the digital representation of the real world environment on a display unit, the digital representation including a graphical representation of the at least one device based on a status information of the at least one device, and control one or more operations of the at least one device in the real world environment based on the control information.

Although embodiments of the present disclosure are described below, the present disclosure may be implemented using any number of techniques. The present disclosure is not intended to be limited to the embodiments, accompanying drawings, and techniques described below, but may be modified within the scope of the present disclosure as defined by the appended claims and their equivalents.

The term “some” as used herein indicates none, one, more than one, and all. Accordingly, the terms “none,” “one,” “more than one,” “more than one, but not all” and “all” are intended to be indicated by the term “some.” The term “some embodiments” may refer to no embodiments, one embodiment, several embodiments, and all embodiments. Accordingly, the term “some embodiments” is is intended to indicate no embodiment, one embodiment, more than one embodiment, and all embodiments.

The terminology employed herein is for describing, teaching and illuminating some embodiments and their features and elements but is not intended to limit, restrict or reduce the scope of the present disclosure as defined by the appended claims and their equivalents.

Any terms used herein such as but not limited to “includes,” “comprises,” “has,” “consists,” and grammatical variants thereof are not intended to specify an exact limitation or restriction of the present disclosure and are not intended to exclude the possible addition of one or more features or elements, unless otherwise stated, and are not intended to exclude the possible removal of one or more of the listed features and elements, unless otherwise stated with the limiting language “must comprise” or “must include.”

Whether or not a certain feature or element is limited to being used only once, the feature or element may be referred to as “one or more features,” “one or more elements,” “at least one feature,” and “at least one element,” respectively. Furthermore, the use of the terms “one or more” and “at least one” feature or element are not intended to preclude there being none of that feature or element, unless otherwise specified by limiting language such as “there must be one or more . . . ” or “one or more element is required.”

Unless otherwise defined, all terms used herein may be taken to have the same meaning as commonly understood by one having an ordinary skill in the art.

Reference is made herein to some “embodiments.” It is intended that an embodiment is an example of a possible embodiment of any features and/or elements presented in the appended claims. Some embodiments are described for the purpose of illuminating one or more of the potential ways in which certain features and/or elements of the attached claims fulfil the requirements of uniqueness, utility and non-obviousness.

Use of the phrases and/or terms such as but not limited to “a first embodiment,” “a further embodiment,” “an alternate embodiment,” “one embodiment,” “an embodiment,” “multiple embodiments,” “some embodiments,” “other embodiments,” “further embodiment”, “furthermore embodiment”, “additional embodiment” and variants thereof are not necessarily intended to refer to the same embodiments. Unless otherwise specified, one or more particular features and/or elements described in connection with one or more embodiments may be found in one embodiment, in more than one embodiment, in all embodiments, and in no embodiments. Although one or more features and/or elements may be described herein in the context of only a single embodiment, alternatively in the context of more than one embodiment, or further alternatively in the context of all embodiments, the features and/or elements may instead be provided separately or in any appropriate combination or not at all. Conversely, any features and/or elements described in the context of separate embodiments may alternatively be realized as existing together in the context of a single embodiment.

Any particular and all details set forth herein are used in the context of some embodiments and therefore are not intended to necessarily limit the present disclosure. The appended claims and their equivalents may be realized in the context of embodiments other than the ones described below.

In an embodiment of the present disclosure, a digital representation of the real world environment is obtained. The real world environment includes at least one transmitting device. The digital representation is indicative of a virtual realty environment corresponding to the real world environment. From the digital representation, the at least one transmitting device is identified and status information and/or location information of the at least one transmitting device is obtained. Thereafter, a modified digital representation is generated such that the modified digital representation includes a graphical representation of the at least one transmitting device in conjunction with the obtained status information and/or the obtained location information. The modified digital representation is then displayed or rendered on a VR enabled display unit. Thus, the modified digital representation indicates at position or location corresponding to the position or location of the transmitting device(s) at the real place along with real time status.

Further, the at least one transmitting device may be controlled via the digital representation. To this end, upon rendering the modified digital representation, a user-input indicative of control information of the at least one transmitting device is received through a VR enabled input unit. The control information is indicative of a variation in a value of location information and/or a device parameter of the at least one transmitting device. In accordance with the control information, one or more operations of the at least one transmitting device in the real world environment is controlled.

The advantages of the disclosure include, but are not limited to, viewing of the transmitting devices available in the real world environment via a VR space or digital representation. Further, real time status information and/or location information is provided on the digital representation. Furthermore, as the digital representation may be a virtual replica of the real world environment, navigating within the digital representation to interact with the transmitting devices provides a real lifelike experience. Further, the transmitting devices may be controlled through the digital representation via user-input received through a VR enabled input unit. For example, a user may interact with a transmitting device in a more natural and intuitive way. Furthermore, the user is not required to be physically present at the location to view the real time status information and/or location information or to operate the transmitting devices. Each of the above aspect contributes considerably to improving a user-experience.

FIGS. 3A and 3Bare block diagrams of a network environment300that includes an electronic device301, according to an embodiment of the present disclosure.

Referring toFIG. 3A, the electronic device301may include a bus302, a processor303, a memory304, an input/output (I/O) interface305, a VR enabled display unit306, a communication interface307, and a VR management module308The bus302may be a circuit that connects the foregoing components and allows communication (for example, control messages) between the foregoing components.

The processor303may, for example, receive instructions from other components (for example, the memory304, the I/O interface305, the VR enabled display unit306, and the communication interface307), interpret the received instructions, and execute computation or data processing according to the interpreted instructions. The processor303may control one or more other components of the electronic device301and/or processes an operation or data related to communication. The processor303may include one or more of a central processing unit (CPU), an application processor (AP), and a communication processor (CP).

The memory304may, for example, store instructions or data that are received from, or generated by, other components (for example, the I/O interface305, the VR enabled display unit306, the communication interface307, and the VR management module308). For example, the memory304may include programming modules such as a kernel309, a middleware310, an application programming interface (API)311, and applications312. Each of the foregoing programming modules may include software, firmware, hardware, or a combination of at least two of software, firmware, and hardware.

The kernel309may control or manage system resources (for example, the bus302, the processor303, or the memory304) that are used in executing operations or functions implemented in other programming modules such as the middleware310, the API311, and the applications312. In addition, the kernel309may provide an interface for allowing the middleware310, the API311, or the applications312to access and control or manage individual components of the electronic device301.

The middleware310may be a medium through which the kernel309may communicate with the API311or the applications312to transmit and receive data. In addition, the middleware310may perform control operations (for example, scheduling or load balancing) in regard to work requests by one or more applications312by, for example, assigning priorities for using system resources (e.g., the bus302, the processor303, and the memory304) of the electronic device301to the one or more applications312.

The API311is an interface that may control functions that the applications312provide at the kernel309or the middleware310. For example, the API311may include at least one interface or function (for example, a command) for file control, window control, video processing, or text control.

According to an embodiment of the present disclosure, the applications312may include a short message service (SMS)/multimedia messaging service (MMS) application, an email application, a calendar application, an alarm application, a health care application (for example, an application that measures an amount of exercise or a blood sugar level), or an environmental information application (for example, an application that provides information about air pressure, humidity, and temperature). Alternatively or additionally, the applications312may be related to information exchange between the electronic device301and an external electronic device (for example, an electronic device313). The information exchange-related application may be, for example, a notification relay application for transmitting certain information to the external electronic device or a device management application for managing the external electronic device.

For example, the notification relay application may include a function of transmitting notification information generated from another application (for example, an SMS/MMS application, an email application, a health care application, or an environment information application) to the external electronic device313. Alternatively or additionally, the notification relay application may receive notification information from the external electronic device313and transmit the received notification information to a user. The device management application may manage (for example, install, delete, and update) at least a part of functions of the external electronic device313communicating with the electronic device301(for example, turn-on and turn-off of the external electronic device313(or a part of its components) or control of the brightness (or resolution) of the display), an application executed in the external electronic device, or a service (for example, a call service or a message service) provided by the external electronic device.

According to an embodiment of the present disclosure, the applications312may include an application designated according to a property (for example, the type of the electronic device) of the external electronic device (for example, the electronic device313). For example, if the external electronic device is a digital audio player, the applications312may include an application related to music play. If the external electronic device is a mobile medical device, the applications312may include an application related to health care. According to an embodiment, the applications312may include at least one of an application designated in the electronic device301or an application received from a server314or the electronic device313. The server314may be a single server or a group of more than one server.

Further, according to an embodiment of the present disclosure, an electronic device, or a plurality of electronic devices, such as the external electronic device313and the server314, may perform some or all of the operations performed by the electronic device301. For example, when the electronic device301performs some functions or services automatically or by request, the electronic device301may request the external electronic device313and the server314to perform at least some of the functions related to the functions or services, in addition to or instead of performing the functions or services by itself. In this case, the external electronic device313and the server314may carry out the requested function or the additional function, and transfer the result to the electronic device301. The electronic device301may provide the requested functions or services based on the received result as is or after additionally processing the received result. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used.

The I/O interface305may receive a command or data from a user through an I/O device (for example, a sensor, a keyboard, or a touch screen) to the processor303, the memory304, the communication interface307, and the VR management module308, for example, through the bus302. For example, the I/O interface305may provide data of a user touch received through the touch screen to the processor303. Further, the I/O interface305may, for example, output a command or data received from the processor303, the memory304, the communication interface307, and the VR management module308through the bus302to the I/O device (for example, a speaker or a display). For example, the I/O interface305may output voice data processed by the processor303to a user through the speaker.

The VR enabled display unit306may display various types of information (for example, multimedia data or text data) to a user. The VR enabled display unit306may be configured to include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, a plasma cell display, an electronic ink array display, an electronic paper display, a flexible LCD, a flexible electro-chromic display, and a flexible electro wetting display.

The communication interface307may provide communication between the electronic device301and the electronic device313or the server314. For example, the communication interface307may be connected to a network315by wireless or wired communication and communicate with the external electronic device313or the server314over the network315. The wireless communication may be conducted in conformance to, for example, at least one of wireless fidelity (WiFi), Bluetooth (BT), near field communication (NFC), global positioning satellite (GPS), and cellular communication (for example, long term evolution (LTE), LTE-advanced (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), and global system for mobile communications (GSM)). The wired communication may be conducted in conformance to, for example, at least one of a universal serial bus (USB), a high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or a plain old telephone service (POTS).

According to an embodiment of the present disclosure, the network315may be a communication network, for example, at least one of a computer network, the Internet, an Internet of Things (IoT), and a telephone network. At least one of the applications312, the API311, the middleware310, the kernel309, and the communication interface307may support a protocol (for example, a transport layer protocol, a data link layer protocol, or a physical layer protocol) for communication between the electronic device301and the external device313.

According to an embodiment of the present disclosure, the device301enables viewing and controlling of transmitting devices through VR.

Referring toFIG. 3B, the device301is communicatively coupled with one or more transmitting devices (TDs)316-1,316-2, . . .316-N (hereinafter referred to transmitting device316for denoting a single transmitting device and transmitting devices316for denoting plurality of transmitting devices) operating in a real world environment317(represented by a dashed square). The transmitting devices316are typically embedded with electronics, software, sensors, actuators, and network connectivity that enable these devices to perform designated tasks and to collect and exchange date over the network315.

Such transmitting devices316include, but are not limited to, sensors, smart devices, wearable devices, and smart phone. Examples of sensors include, but are not limited to, proximity sensors and infrared sensors. Examples of smart devices include, but are not limited to, home automation devices such as a smart television (TV), a smart music system, smart speakers, smart sprinklers, a smart vacuum cleaner, a smart oven, and a smart lighting system. Examples of the wearable devices include, but are not limited to, smart watches, GPS trackers, and headphones. Examples of the real world environment317include, but are not limited to, a home, various rooms in a home, a vehicle, an office, a theatre, a museum, and other buildings.

Further, each of the transmitting devices316may be communicatively connected with other transmitting devices316in the real world environment317. For example, a smart door may be further communicatively connected with a smart lock, a smart key set, a corridor light, and a smart phone. In addition, a master transmitting device may be communicatively connected with the rest of the transmitting devices316. The master transmitting device controls the rest of the transmitting devices316The master transmitting device may be the transmitting device316. The smart phone may be a master transmitting device for a smart door, a smart lock, a smart key set, and a corridor light. The master transmitting device may be a different device.

According to an embodiment of the disclosure, the device301enables viewing and controlling of transmitting devices through VR. Thus, the VR management module308may perform, for example, an operation for synthesizing a digital representation of a real world environment depicting graphical representations of transmitting devices in the real world environment and their corresponding status information and/or location information, and operation for controlling of the transmitting devices via the digital representation.

In accordance with an embodiment of the present disclosure, the digital representation is indicative of a VR environment or alternatively is a virtual replica corresponding to the real world environment317. The digital representation is a direct representation of the real world environment317. Examples of such direct representation include, but are not limited to, an image, a panorama image, a video, a 360-degree image, a 360-degree panorama image, and a 360-degree video.

Referring toFIG. 4A, a first example digital representation400in format of an image depicting a front view of a real world kitchen is illustrated. Various transmitting devices401are present in the digital representation400. For example, transmitting device401-1indicates a smart oven, transmitting device401-2indicates a smart kettle, transmitting device401-3indicates a smart grinder, and transmitting device401-4indicates a smart coffee maker. In another embodiment, the digital representation is a processed representation of the real world environment317. Examples of such processed representation include, but not limited to, three-dimensional (3D) model created using 3D modelling software.

Referring toFIG. 4B, a second example digital representation402in format of 3D model created using 3D modelling software depicting front view of a real world living room is illustrated.

Referring toFIG. 3C, a block diagram of the VR management module308in the electronic device301is illustrated according to an embodiment of the present disclosure. The VR management module308includes a first digital representation unit (FDRU)318, a second digital representation unit (SDRU)319, an identification unit320, a status and location unit321, a rendering unit322, an input receiving unit323, and a control unit324. The FDRU318, the SDRU319, the identification unit320, the status and location unit321, the rendering unit322, the input receiving unit323, and the control unit324may be software modules. The FDRU318, the SDRU319, the identification unit320, the status and location unit321, the rendering unit322, the input receiving unit323, and the control unit324may be hardware modules. The FDRU318, the SDRU319, the identification unit320, the status and location unit321, the rendering unit322, the input receiving unit323, and the control unit324may be a combination of software and hardware modules. A single unit may perform functions of any combination of the units from the FDRU318, the SDRU319, the identification unit320, the status and location unit321, the rendering unit322, the input receiving unit323, and the control unit324.

In an embodiment of the present disclosure, the FDRU318obtains a digital representation of the real world environment317. The input receiving unit323receives a user-input to obtain the digital representation. The input receiving unit323receives the user-input through a VR enabled input unit325communicatively coupled with the device301. Examples of the VR enabled input unit325include, but are not limited to, a head mounted device, a smart glove, a joystick, a smart stylus, a smart touch interface, an eye gaze tracking input device, and a voice input device. The VR enabled input unit325is integrated with the electronic device301. The electronic device301is a standalone VR device having VR processing and VR rendering capabilities. The VR enabled input unit325is external to the electronic device301. The electronic device301such as a smart phone is coupled with the VR enabled input unit325such as an HMD.

In an embodiment of the present disclosure, the FDRU318obtains a digital representation by capturing the digital representation of the real world environment317in real time. The input receiving unit323receives a user-input as being a selection of an option to capture the digital representation in real time. The FDRU318captures the image of the real world using a VR enabled image-capturing unit in real time upon receiving the user-input. The VR enabled image-capturing unit may be internal to the device301. The VR enabled image-capturing unit may be external to the device301. The device301may access the VR enabled image-capturing unit directly. The device301may access the VR enabled image-capturing unit over the network315. A device having the VR enabled image-capturing unit may transmit the image to the device301.

In an embodiment of the present disclosure, the FDRU318obtains the digital representation by fetching the digital representation from a digital representations (DR) database326communicatively coupled with the device301. The DR database326includes a list of pre-stored digital representations327corresponding to real world environments. The DR database326may be external to the device301and may be accessed by the device301directly. The device301may access the DR database326over the network315. In accordance with a non-illustrated option, the DR database326may be internal to the device301.

The VR enabled image-capturing unit captures, processes (as deemed necessary), and stores the digital representations in the DR database326. While capturing and storing, location information and identification information of the corresponding real world environment is stored as metadata328in the DR database326. The location information may be obtained from one or more positioning systems329(hereinafter referred to positioning system329for denoting a single positioning system and positioning systems329for denoting plurality of positioning systems) over the network315. Examples of the positioning systems329include, but are not limited to, GPS, an indoor position system (IPS), and local positioning system (LPS). The IPS may be based on various technologies. Examples of such technologies include, but are not limited to, magnetic positioning, inertial measurements, WiFi-based positioning system (WPS), Bluetooth, choke point concepts, grid concepts, long range sensor concepts, angle of arrival, time of arrival, and received signal strength indication.

Further, a relative location and dimension information of the real world environment is stored as the metadata328in the DR database326. The relative location and dimension information is obtained from the captured digital representation. In addition, location information of the VR enabled image-capturing unit is stored in the metadata328. The metadata328may be stored in a tabular form, for example, as illustrated below Table 1.

Further, in an embodiment of the present disclosure, the input receiving unit323receives a user-input as being a selection of the real world environment317from a list of pre-stored real world environments. Upon receiving the user-input, the FDRU318fetches the digital representation corresponding to the selected real world environment317from the DR database326.

Referring toFIG. 4A, the image of the real world kitchen along with images of real world living room, bedroom, and dining room is pre-stored in the DR database326. The FDRU318fetches image of the real world kitchen in real time from the DR database326.

Further, the input receiving unit323may receive the user-input as being a navigation input on the digital representation other than the aforementioned inputs. In one embodiment, the user-input is navigation from a first digital representation of the real world environment to a second digital representation of the real world environment. In an example, the user-input is navigation from a front view digital representation of the kitchen to a left-side view digital representation of the kitchen. In an embodiment, the user-input is navigation from a digital representation of a first real world environment to a digital representation of a second real world environment. In an example, the user-input is navigation from digital representation of the kitchen to digital representation of the living room. In an embodiment, the user-input is navigation within the digital representation of the real world environment. In an example, the user-input may be zoom in or zoom out of digital representation of the kitchen. In these embodiments, the FDRU318obtains the digital representation either by capturing real digital representation or by fetching pre-stored digital representation, as described above.

Upon obtaining the digital representation, the identification unit320identifies the transmitting devices316from the digital representation. In one embodiment, the identification unit320identifies the transmitting devices316based on a user-input indicative of labelling the transmitting devices316on the digital representation. The user-input may be non-touch input and touch input. In such an embodiment, the input receiving unit323receives the user-input from the VR enabled input unit325.

In one embodiment, the identification unit320identifies the transmitting devices316based on metadata associated with the digital representation. Accordingly, the identification unit320fetches metadata from the metadata unit328in the DR database326to identify the transmitting devices316. In an example, the digital representation is a 3D model. As such, the metadata includes scaled-down information of real world in the 3D model as the relative location and dimension information. Based on a mapping of the relative location and dimension information and the location information of real world environment from the metadata from the metadata unit328, the identification unit320identifies the transmitting devices316on the digital representation.

In one embodiment, the identification unit320identifies the transmitting devices316based on processing of content within the digital representation and the metadata328. Accordingly, the identification unit320processes the content by using image/media recognition techniques. The identification unit320may further obtain location information of the transmitting devices316from the at least one positioning system329. The location information provides real-time location of the transmitting device316in the real world environment317in form of precise coordinates. Based on output of image/media processing and the location information, the identification unit320identifies the transmitting devices316within the digital representation.

In one embodiment, the identification unit320identifies the transmitting devices316based on the metadata in the metadata unit328and location information of a device capturing the digital representation of the real world environment or the VR enabled image-capturing unit. In one embodiment, the identification unit320identifies the transmitting devices316based on the metadata in the metadata unit328and location information of the transmitting devices316available in proximity to the location information of the VR enabled image-capturing unit. In one embodiment, the identification unit320obtains the location information of the transmitting devices316from at least one positioning system329. Based on the above-mentioned location information(s), the identification unit320identifies the transmitting devices316within the digital representation. In one embodiment, the identification unit320identifies the transmitting devices316based on combination of the above-mentioned information/processing.

Upon identification of the transmitting devices316, the status and location unit321obtains at least one of status information and location information of the transmitting devices316. The status information includes information corresponding to at least one device identifier and at least one device parameter of the transmitting device(s)316. The at least one device identifier includes a unique identification number, device name, unique address such IP address and MAC address, date and time stamp, and optionally user name, user photograph, and manufacturer. The at least one device parameter includes operational status, mode of operation, battery status, type of network connection, availability of said network connection, connection with one or more further transmitting devices, and status of said network connection with one or more further transmitting devices. For example, a smart door may have following values corresponding device parameters:

Operational Status or Door Status: Open/Close/Partially Open

Type of network connection type: WiFi

Availability of network connection or Connection Status: Active

Connection with further transmitting devices: Smart Lock, Smart Keys, and Corridor Light.

Connection Status with further transmitting devices: Active

In one embodiment, the status and location unit321obtains the status information directly from the transmitting devices316. In one embodiment, the transmitting devices316transmits the status information to the status and location unit321when a value of said at least one device parameter changes. In one embodiment, the transmitting devices316transmits the status information to the status and location unit321when a value of said at least one device parameter exceeds a predetermined threshold level. The status information may be sent/received either periodically or in real time.

In one embodiment, the status and location unit321obtains the status information from an intermediary device330communicatively coupled with the transmitting devices316. The intermediary device330may be any device having transmitting and receiving capabilities and is connected with the transmitting device(s)316and the device301. As such, the intermediary device330enables transmission of data by the transmitting device(s)316over the network315when the transmitting device(s)316are not able to connect with the network315directly. The intermediary device330may support various communication technologies such as Bluetooth, Zigbee, Z-Wave, 6LowPAN, Thread, WiFi, Mobile Cellular, NFC, Sigfox, Neul, LoRaWAN, Satellite, Ethernet, and HART. Example of the intermediary device330is an Internet gateway.

Further, in one embodiment, the intermediary device330may provide the status information in real time when requested by the status and location unit321. In one embodiment, the intermediary device330may provide the status information by periodically polling the transmitting devices316. In such an embodiment, the intermediary device330may provide the status information when a value of said at least one device parameter changes. In such implementation, the intermediary device330may also provide the status information when a value of said at least one device parameter exceeds a predetermined threshold level. In one example, the intermediary device330may obtain information from the transmitting device316. In another example, the intermediary device330may obtain from a second transmitting device connected with a first transmitting device to obtain the information of the first transmitting device and/or the second transmitting device.

Further, as described earlier, the location information of the transmitting device316provides real-time location of the transmitting device316in the real world environment317in form of precise coordinates. Accordingly, in one embodiment, the status and location unit321obtains the real-time location directly from the transmitting device316. In such an embodiment, the transmitting device316may include a location identification module (not shown in the figure) to identify and provide precise coordinates. The location identification module may be based on currently known location identification technologies such as GPS and IPS.

In one embodiment, the status and location unit321obtains the real-time location from the at least one positioning system329, as described earlier.

In one embodiment, the status and location unit321obtains the real-time location from the master transmitting device. In such an embodiment, the master transmitting device may include the location identification module in addition to the transmitting device316. Thus, the master transmitting device may obtain the real-time location from other transmitting devices316and then provide the real-time location to the status and location unit321.

In one embodiment, the status and location unit321obtains the real-time location from the intermediary device330, as described earlier.

In all the above embodiments, the transmitting devices316create data packet to transmit the status information, as known in the art, to either the intermediary device330or the status and location unit321. Similarly, the location information is transmitted using a data packet, as known in the art. It would be understood, that any other mechanism may be used to transmit the status information and the location information to corresponding recipients.

Further, upon obtaining the aforesaid information, the status and location unit321may store the status information and/or the location information in a status and location information (SLI) database331. In an example, the SLI database331may be external to the device301. In one option, the device301may access the SLI database331directly. In another option, the device301may access the SLI database331over the network315. In another example, the SLI database331may be internal to the device301. The status information and the location information may be stored as in a tabular form, as, for example, illustrated below in Table 2.

Further, based on the obtained status information and/or location information, the SDRU319creates a modified digital representation of the real world environment317. The modified digital representation includes a graphical representation of the identified transmitting devices316in conjunction with at least one of the status information and the location information.

In one embodiment, the status and location unit321may obtain only the status information. In such an embodiment, the SDRU319obtains graphical representations of the identified transmitting devices316and graphical representations indicative of the status information from a graphical representation (GR) database332. Examples of the graphical representations include, but not limited to, icons. The GR database332, as illustrated inFIG. 3E, includes a list of pre-stored graphical representations333(referred inFIG. 3Eas GRTD) of transmitting devices316mapped with corresponding device identifiers of the transmitting devices316. In a similar manner, as illustrated inFIG. 3E, the GR database332includes a list of pre-stored graphical representations334(referred inFIG. 3Eas GRSI) indicative of various types of status information. For example, a graphical representation of battery with low charge may be indicative of battery status with 10%. In an example, the GR database332may be external to the device301. In an example, the GR database332may be internal to the device301. In an example, the GR database332accessed over the network315.

Upon retrieving the graphical representation of the identified transmitting devices316and the status information, the SDRU319superimposes the graphical representation of the identified transmitting devices316and the status information on the digital representation to create the modified digital representation. The graphical representation of the identified transmitting devices316are positioned on the digital representation at a location mapping the location of the identified transmitting devices316in the real world environment317, as described earlier. Upon creation of the modified digital representation, the rendering unit322renders/displays the modified digital representation on a VR enabled display unit322coupled with the device301.

In one embodiment, the status and location unit321may obtain the status information and the location information. In such an embodiment, the SDRU319maps a real-time location of the identified transmitting devices316on the digital representation based on the location information. Upon mapping, the SDRU319obtains the graphical representations of the identified transmitting devices316and the graphical representations indicative of the status information from the GR database332. Thereafter, the SDRU319superimposes the graphical representations of transmitting devices316and the graphical representations of the status information on the digital representation at the mapped real-time location to create the modified digital representation. Such mapping of real-time location enables representing the transmitting devices316at exactly same location, as they are present in the real world environment317at time of obtaining the location information.

Further, the status information and/or location information may be represented via audio indicators. Examples of such audio indicators include, but not limited to, pre-stored speech notifications indicative of various types of status information and/or location information and pre-stored tones/chimes/music/notes indicative of various types of status information and/or location information. The audio indicators may be stored in the GR database332.

In one such an embodiment, the SDRU319obtains the audio indicators from the GR database332based on the status information and directs an audio output device located in the real world environment317to play the audio indicator. In another such an embodiment, the SDRU319obtains the audio indicators from the GR database332based on the status information and location information. In such an embodiment, the SDRU319may detect an audio output device located in the real world environment317that may be located in proximity to the location information and direct such detected audio output device to play the audio indicator. In one example, the audio device may itself be the transmitting device316. In one example, the audio device may be located in proximity to the transmitting device316and may be communicatively coupled with the device301. Examples of such audio output device include, but not limited to, wired speakers and wireless speakers.

Further, as described earlier, the status information and the location information is stored in the SLI database331. In addition, the status and location unit321receives the status information and the location information when a corresponding values changes or corresponding value exceeds predetermined threshold values. Thus, the SDRU319may also determine a variation in the status information and/or the location information. Upon determining the variation, the SDRU319again creates a further modified digital representation of the real world environment317, as described earlier. The further modified digital representation includes the graphical representation of the transmitting device316in conjunction with the determined variation. Further, in one embodiment, the SDRU319may direct the audio output device to play an audio indicator corresponding to the determined variation, as described earlier.

Thus, the present disclosure enables viewing of transmitting devices available in a real world environment via a virtual reality space or digital representation on a VR enabled display unit. This provides a better user-experience as a need for physically visiting a location of the transmitting devices is eliminated.

FIGS. 5A, 5B, 5C, 6A, 6B, 7A, 7B, 7C, and 7Dillustrate various examples of viewing transmitting devices via a digital representation, in accordance with one embodiment of the disclosure. In the examples, a display unit of the smart phone functions as the VR enabled display unit325and an HMD and/or the other devices such as smart gloves functions as a VR enabled input unit.

FIGS. 5A, 5B, and 5Cillustrate a first example500of viewing transmitting devices via a digital representation, in accordance with one embodiment of the disclosure. In the example, a VR enabled display unit501displays a digital representation. HMD502and/or other VR enabled input units (not shown in the figure) provide user-input.

Referring toFIG. 5A, the VR enabled display unit501displays a real image of a front view of a real world kitchen in a camera mode of the device301. The input receiving unit323receives a user-input via the HMD502and/or other VR enabled input units. The user-input is indicative of a selection of an option to capture the digital representation in real time.

Referring toFIG. 5B, upon receiving the user-input, the SDRU319displays a digital representation503on the VR enabled display unit501in a manner as described above. The digital representation503includes graphical representation(s)504of identified transmitting device(s) and graphical representation(s)505of corresponding status information. As illustrated, graphical representation504-1indicates smart oven and graphical representation505-1indicates ‘no network connection’ status information (e.g., no signal). The graphical representation504-1is positioned near an image of a kettle indicating a real world position of the kettle in the real world kitchen. Similarly, graphical representation504-2indicates a smart kettle and graphical representation505-2indicates ‘85%’ of battery status. Similarly, graphical representation504-3indicates smart grinder and graphical representation505-3indicates ‘10%’ of battery status. Similarly, graphical representation504-4indicates smart coffee maker. However, when status information for a transmitting device is not available, a corresponding graphical representation is not provided on the digital representation. Thus, the smart coffee maker does not have a graphical representation corresponding to status information.

Further, the graphical representation504-4indicates smart coffee maker is at location L1. However, while obtaining the status information and location information, location of the coffer maker changes from L1to L2inFIG. 5C. The change in location may be, for example, due to a person arriving in the kitchen and using the coffee maker. As such, the SRDU306maps the graphical representations504-4at the location L2.

Referring toFIG. 5C, the digital representation503includes the graphical representations504-4at location L2and a direction arrow indicating change in location from L1to L2.

FIGS. 6A and 6Billustrate a second example600of viewing transmitting devices via a digital representation, in accordance with one embodiment of the present disclosure. In the example, a VR enabled display unit601displays a digital representation. HMD602and/or other VR enabled input units (not shown in the figure) provide user-input.

Referring toFIG. 6A, the VR enabled display unit601displays a digital representation603including graphical representations of transmitting devices and graphical representations of corresponding status information. The digital representation603is a virtual replica of a front view of a real world kitchen, as illustrated inFIG. 5B.

As described above, a user may navigate virtually within the present environment for selecting an environment different from the present environment. In one embodiment, the user may navigate by providing input via the VR enabled input unit325. In one example of such an embodiment, the input may be a head motion along a direction of the different environment. In another example of such an embodiment, the input may be a hand movement or gesture along a direction of the different environment. In another embodiment, user-selectable options are provided on the digital representation.

The digital representation603further includes one or more environment selection options604. The environment selection options604may be represented in various forms such as icons and hyperlinks. The environment selection option604-1enables a selection of real world environment ‘balcony’. Similarly, environment selection option604-2enables a selection of real world environment ‘bedroom’.

The input receiving unit323then receives a user-input via the HMD603and/or other VR enabled input units. The user-input is indicative of navigation from a digital representation of a first real world environment to a digital representation of a second real world environment. The user-input is indicative of selection the environment selection option604-2.

Upon receiving the selection, the FDRU318obtains a digital representation of the bedroom as described earlier. Thereafter, the SDRU319renders a corresponding modified digital representation including graphical representations of transmitting devices present in the bedroom and corresponding status information, as described above.

Referring toFIG. 6B, the VR enabled display unit601displays a digital representation605of the real world bedroom. The digital representation605is a virtual replica of a front view of the real world bedroom. The digital representation605includes graphical representations of transmitting devices available in the bedroom and graphical representations of corresponding status information. Graphical representation606indicates smart locker and graphical representation607indicates a locked status.

FIGS. 7A, 7B, 7C, and 7Dillustrate a third example700of viewing transmitting devices via a digital representation, in accordance with one embodiment of the present disclosure. In the example, a VR enabled display unit701displays a digital representation. HMD702and/or other VR enabled input units provide user-input.

Referring toFIG. 7A, the VR enabled display unit701displays a digital representation703including graphical representations of transmitting devices and graphical representations of corresponding status information. The digital representation703is a virtual replica of a front view of a real world kitchen, as illustrated inFIG. 5B.

The input receiving unit323receives a user-input via the HMD703and/or other VR enabled input units. The user-input is indicative of navigation within a digital representation. In the example, the user-input is indicative of zooming out on the digital representation703to get a bird's eye view of all transmitting devices available in a house.

Upon receiving the selection, the FDRU318obtains digital representation of entire house. In an embodiment, such digital representation may be obtained by combining digital representations of all rooms in the house. Thereafter, the SDRU319renders a corresponding modified digital representation including graphical representations of transmitting devices present in all the rooms and corresponding status information, as described above.

Referring toFIG. 7B, the VR enabled display unit701displays a digital representation704of the real world house. The digital representation704is a virtual replica of a bird's eye view or top view of the real world house. The digital representation704includes graphical representations of transmitting devices available in the entire house and graphical representations of corresponding status information.

Further, the input receiving unit323may receive another user-input via the HMD703and/or other VR enabled input units on the digital representation704. The user-input is indicative of simultaneously displaying multiple locations having transmitting devices. The multiple locations may also include the present location being viewed through the digital representations703and704. Upon receiving the user-input, the FDRU318fetches the digital representations of the multiple locations from the DR database326, as described inFIG. 3A. The VR enabled display unit701then displays digital representations. In one example, the digital representation may be a subset of the other digital representation. In one example, the digital representation may be a combination of various digital representations. These digital representations are similar to the digital representations illustrated inFIGS. 4A and 4B.

Referring toFIG. 7C, the VR enabled display unit701displays digital representation705of first location such as kitchen and the digital representation706of second location such as house. Similarly, the VR enabled display unit701displays digital representation707of third location such as bedroom and digital representation708of fourth location such as dining hall. As may be observed, the digital representations of kitchen, bedroom, and dining hall are subsets of the digital representation of the house. Further, the VR enabled display unit701displays digital representation709of fifth location such as car and digital representation710of sixth location such as interior of the car at driver's location. As may be observed, the digital representation of interior of the car at driver's location is subset of the digital representation of the car.

The various digital representations of multiple locations may be arranged in any manner and/or using any size on the VR enabled display unit701based on a screen size and/or user preference(s). Such arrangement is performed using techniques as known in the art.

Further, the input receiving unit323may receive another user-input via the HMD703and/or other VR enabled input units on the digital representation704. The user-input is indicative of a selection of any one of the digital representations to view the transmitting devices present in the selected location. In the figure, the user-input is indicative of selection the digital representation709of car, i.e., the fifth location. Upon receiving the user-input, the SDRU319renders a corresponding modified digital representation including graphical representations of transmitting devices present in the selected location and corresponding status information, as described inFIG. 3B.

Referring toFIG. 7D, the VR enabled display unit701displays a digital representation711including graphical representations of transmitting devices and graphical representations of corresponding status information. The digital representation711is a virtual replica of the entire car.

In accordance with an embodiment, upon rendering the modified digital representation, the device301or the VR management module308, enables controlling of the identified transmitting devices316via the modified digital representation. Accordingly, the VR enabled display unit322displays a digital representation including the graphical representations of the identified transmitting devices316and corresponding status information and/or location information. Such digital representation may be interchangeably referred to as modified digital representation hereinafter.

Thereafter, the input receiving unit323receives a user-input through the VR enabled input unit325on the modified digital representation. The user-input is indicative of control information of the identified transmitting device316. The user-input may be in form of at least one of voice command, eye gaze, gesture input, touch input, motion of the VR enabled input unit325, position of the VR enabled input unit325, and spatial orientation the VR enabled input unit325. The control information may be indicative of variation in a value of location information of the transmitting devices316. The control information may also be indicative of variation in a value of at least one device parameter of the transmitting devices316.

Accordingly, the user-input may be received in two parts. As such, the input receiving unit323receives a first user-input through the VR enabled input unit325indicative of selection of one of the identified transmitting devices316. Upon receiving the first user-input, the control unit324fetches a graphical representation of a control panel associated with selected transmitting device316from the GR database332. The GR database332, as illustrated inFIG. 3E, includes a list335of pre-stored graphical representations of control panel (referred inFIG. 3Eas GRCP335) mapped with the corresponding device identifiers of the transmitting devices316.

Upon fetching the graphical representation of the control panel, the control unit324renders the graphical representation of the control panel on a current view of the modified representation on the VR enabled display unit306. To this end, the control unit324sends the graphical representation of the control panel and a corresponding command to the rendering unit322to render the graphical representation on the current view.

Upon rendering the control panel, the input receiving unit323receives second user-input through the VR enabled input unit. The second user-input is indicative of manipulating the graphical representation of the control panel. Based on the manipulation, the control unit324determines the control information. In an example, a control panel represents information as operation status being inactive/OFF for a smart vacuum cleaner on a digital representation of a real kitchen. In such example, the input receiving unit323receives second user-input as changing the operation status to active/ON. Accordingly, the control unit324determines the control information as ‘activating/switching ON the transmitting device’.

Upon determining, the control unit324renders the control information on the current view of the digital representation. To this end, the control unit324fetches graphical representation corresponding to the control information from the list334of pre-stored graphical representations in the GR database332. Upon fetching, the control unit324sends the graphical representations and a corresponding command to the rendering unit322to render the graphical representation on the current view.

Upon receiving the control information, the control unit324controls one or more operations of the transmitting devices316in the real world environment317in accordance with the control information. Accordingly, the control unit324transmits the control information to the selected transmitting device316in the real world environment317over the network315. Upon receiving the control information, the selected transmitting device316transmits an acknowledgement message. Further, the control unit324transmits the control information to the status and location unit321to update SLI database331based on the control information.

Furthermore, status information and/or location information of the selected transmitting device316may change based on the control information. Accordingly, in one embodiment, the status and location unit321may obtain further status information or modified/changed status information of the selected transmitting device316. In another embodiment, the status and location unit321may obtain further status information and real-time location of the selected transmitting device316. The status and location unit321obtains the status information and the real-time location in a manner as described during synthesis of environment. In addition, as described earlier, the status and location unit321update SLI database331.

Upon obtaining the further status information and/or the real-time location, the SDRU319fetches graphical representations of further status information from the list321of pre-stored graphical representations in the GR database332. Upon fetching, the SDRU319superimposes the graphical representation of further status information on the digital representation to modify the digital representation. In addition, the SDRU319may further map the graphical representations of further status information at the real time location, as described earlier. In one embodiment, the SDRU319superimposes the graphical representation on the current view of the digital representation. In the above example, the graphical representation indicative of ‘activated’ smart vacuum cleaner may be presented on the digital representation of the real kitchen itself. In another embodiment, the SDRU319superimposes the graphical representation on a different digital representation. In the above example, the graphical representation indicative of ‘activated’ smart vacuum cleaner may be presented on a new digital representation of the real kitchen such as left-side view of the kitchen.

Further, while viewing the digital representations, the device301enables the user to perform all other functions such calling, messaging, and emailing without interrupting the VR experience. As such, the SDRU319superimposes user-interfaces corresponding to the functions over the digital representations. In an example, a user-interface corresponding to incoming call is superimposed on a digital representation. The user-interface may include information pertaining to the incoming call and options to answer or reject the call. In another example, user-interface corresponding to messages or emails is superimposed on a digital representation. The user-interface may include information pertaining to the messages or emails and options to read, reply, and manage the messages or emails.

FIGS. 8A, 8B, 9A, 9B, 10A, 10B, 10C, 11A, 11B, 11C, 12A, and 12Billustrate various examples of controlling transmitting devices via a digital representation, in accordance with an embodiment of the present disclosure. In the examples, a display unit of the smart phone functions as the VR enabled display unit322and an HMD and/or the other devices such as smart gloves functions as a VR enabled input unit.

FIGS. 8A and 8Billustrate a first example800of controlling transmitting devices through digital representation, in accordance with another embodiment of the disclosure. In the example, a VR enabled display unit801displays a digital representation. HMD802and/or other VR enabled input units provide user-input. In the present example, an operation of the transmitting device is controlled to view further information.

The VR enabled display unit801displays a digital representation803corresponding to a front view of real world kitchen. The digital representation803includes graphical representation(s)804of transmitting device(s) and graphical representation(s)805of corresponding status information. The digital representation803further includes environment selection options806-1and806-2to enable selection of real world balcony and real world bedroom, respectively. The digital representation803further includes device selection option807.

Referring toFIG. 8A, graphical representation804-1indicates smart electric panel or transmitting device. A selection of the smart electric panel is received via the device selection option807(represented by a dashed line connecting804-1and807).

Referring toFIG. 8B, the control unit324provides a graphical representation808associated with a control panel of the smart electric panel in response to the selection. The graphical representation808provides status information and real-time location of the smart electric panel, as described above. The status information includes network address, manufacturer, and further transmitting devices connected with the smart electric panel such as smart oven, smart refrigerator, router, smart kettle, and smart coffee maker. Further, the control unit324provides graphical representation809(represented by dashed lines) to indicate active network connection between the connected devices.

FIGS. 9A and 9Billustrate a second example900of controlling transmitting devices through digital representation, in accordance with another embodiment of the disclosure. In the example, a VR enabled display unit901displays a digital representation. HMD902and/or other VR enabled input units provide user-input. In the present example, an operation of the transmitting device is controlled to view further information and to change operating status.

The VR enabled display unit901displays a digital representation903corresponding to a front view of real world kitchen. The digital representation903includes graphical representation(s)904of transmitting device(s) and graphical representation(s)905of corresponding status information. The digital representation903further includes environment selection options906-1and906-2to enable selection of real world balcony and real world bedroom, respectively. The digital representation903further includes device selection option907.

Referring toFIG. 9A, graphical representation904-1indicates smart oven or transmitting device and graphical representation905-1indicates ‘no network connection’ status information. Graphical representation904-2indicates smart grinder or transmitting device and graphical representation905-1indicates battery status as ‘10%’. A selection of the smart oven is received via the device selection option905(represented by a dashed line connecting904-1and907).

Referring toFIG. 9B, the control unit324provides a graphical representation908associated with a control panel of the smart oven in response to the selection. The graphical representation908provides status information and real-time location of the smart oven, as described above. A change in operation status906-1from inactive/OFF to active/ON is received via the graphical representation908. Accordingly, control information is determined and the graphical representation905-1is updated. The control information is further transmitted to the smart oven such that the smart oven becomes active to perform functions in the real world kitchen.

In a similar manner, two devices may be selected and corresponding status may be viewed. In an example, a selection of the smart oven and the smart grinder is received via the device selection option907. Upon selection, the graphical representations associated with control panels of the smart oven and the smart grinder are displayed.

FIGS. 10A, 10B, and 10Cillustrate a third example1000of controlling transmitting devices through digital representation, in accordance with another embodiment of the disclosure. In the example, a VR enabled display unit1001displays a digital representation. HMD1002and/or other VR enabled input units provide user-input. In the present example, an operation of the transmitting device is controlled to view further information and to change mode of operation.

The VR enabled display unit1001displays a digital representation1003corresponding to a front view of real world bedroom. The digital representation1003includes graphical representation(s)1004of transmitting device(s) and graphical representation(s)1005of corresponding status information. The digital representation1003further includes device selection option1006.

Referring toFIG. 10A, graphical representation1004-1indicates smart locker or transmitting device and graphical representation1005-1-1indicates mode of operation as ‘locked’ (represented by locked icon in circle). A selection of the smart locker is received via the device selection option1006(represented by a dashed line connecting1004-1and1006).

Referring toFIG. 10B, the control unit324provides a graphical representation1007associated with a control panel of the smart locker in response to the selection. The graphical representation1007provides a key pad to enter authentication information such as a personal identification number (PIN) or password. Accordingly, the control unit324determines the control information or the authentication information and transmits the authentication information to the smart locker. Upon receiving the authentication information, the smart locker validates and unlocks itself. The smart locker then transmits updated status information as mode of operation being ‘unlocked’ to the status and location unit321. Thereafter, the SDRU319modifies/updates the digital representation1003to include a graphical representation of updated the status information.

Referring toFIG. 10C, the VR enabled display unit1001displays the digital representation1003such that a graphical representation1005-1-2indicates mode of operation as ‘unlocked’ (represented by unlocked icon in circle).

FIGS. 11A, 11B, and 11Cillustrate a fourth example1100of controlling transmitting devices through digital representation, in accordance with another embodiment of the disclosure. In the example, a VR enabled display unit1101displays a digital representation. HMD1102and/or other VR enabled input units provide user-input. In the present example, an operation of the transmitting device is controlled to change mode of operation for future time.

The VR enabled display unit1101displays a digital representation1103corresponding to a front view of real world bedroom. The digital representation1103includes graphical representation(s)1104-1of transmitting device(s). For the sake of brevity, graphical representation(s) of corresponding status information and environment selection options are not illustrated. The digital representation1103further includes time progress bar1105to display time such as current time, past time, and future time. The digital representation1103further includes time selection options1106-1and1106-2inFIG. 11Bto select time such as past time and future time. The digital representation1103further includes device selection option1107inFIG. 11A.

Referring toFIG. 11A, graphical representation1104-1indicates smart oven or transmitting device. A selection of the smart oven is received via the device selection option1107(represented by a dashed line connecting1104-1and1107).

Referring toFIG. 11B, upon selection, the control unit324displays the time progress bar1105indicating current time 9:00 AM on the digital representation1103.

Referring toFIG. 11C, upon selection of a future time 2:00 PM on the time progress bar1105, the control unit324displays a graphical representation1108of a control panel associated with the smart oven. As the selection corresponds to the future time, the graphical representations of other transmitting devices are either removed or shown in different colour. Through the graphical representation1108, a change in mode of operation as ‘heating’ for 10 minutes is received. Accordingly, the control unit324determines and transmits control information to the smart oven. Upon receiving the control information, the smart oven sets a counter to operate at 2:00 PM for 10 minutes in heating mode. Consequently, the smart oven shares updated status information based on the control information at 2:00 PM and after 10 minutes with the status and location unit321. Thereafter, the SDRU319modifies the digital representation1103to include a graphical representation of updated the status information.

FIGS. 12A and 12Billustrate a fifth example1200of controlling transmitting devices through digital representation, in accordance with another embodiment of the disclosure. In the example, a VR enabled display unit1201displays a digital representation. HMD1202and/or other VR enabled input units provide user-input. In the present example, an operation of the transmitting device is controlled to view/access status of a transmitting device in past time.

The VR enabled display unit1201displays a digital representation1203corresponding to a front view of real world living room. The digital representation1203includes graphical representation(s)1204-1-1&1204-1-2of transmitting device(s). For the sake of brevity, graphical representation(s) of corresponding status information and environment selection options are not illustrated. The digital representation1203further includes time progress bar1205to display time such as current time, past time, and future time. The digital representation1203further includes time selection options1206-1and1206-2inFIG. 12Ato select time such as past time and future time. The digital representation1203further includes device selection option1207.

Referring toFIG. 12A, graphical representation1204-1-1indicates smart vacuum cleaner or transmitting device. A selection of the smart vacuum cleaner is received via the device selection option1207(represented by a dashed line connecting1204-1and1207). Upon selection, the control unit324displays the time progress bar1205indicating current time 6:00 PM on the digital representation1103. Thus, the graphical representation1204-1-1indicates current location and status of the smart vacuum cleaner at the current time, 6:00 PM.

Referring toFIG. 12B, upon selection of a past time 3:00 PM on the time progress bar1205, the control unit324determines control information as to identify location and/or status at the selected time, 3:00 PM. The control unit324then transmits the control information to the status and location unit321. The status and location unit321then fetches status information and/or location information from the SLI database310corresponding to the time 3:00 PM. Thereafter, the SDRU319fetches graphical representations corresponding to the fetched status and/or location information from the GR database332, as described above. The SDRU319then modifies the digital representation1203to include a graphical representation1204-1-2of past the status information and/or location information along with the graphical representation1204-1-1and corresponding time. Thus, the graphical representation1204-1-2indicates location and status of the smart vacuum cleaner at the past time, 3:00 PM. This enables viewing of status and/or location for both past and current times simultaneously.

Thus, the disclosure enables viewing of control information of the transmitting devices. Further, the disclosure enables controlling of the transmitting devices through the digital representation. This provides an ease of controlling as the user is not required to physically present to operate the transmitting devices, thereby considerably improving the user-experience.

FIGS. 13A, 13B, and 13Cillustrate an example1300of viewing and controlling transmitting devices through a digital representation obtained from an external device, in accordance with another embodiment of the present disclosure. In the example, a first electronic device1301is communicatively coupled with a second electronic device1302over a network (represented by a sync icon). A user of the first electronic device1301is currently present in a real world kitchen. A user of the second electronic device1302is currently present in a real world office.

Further, the second electronic device1302includes components and/or access databases as described above with reference toFIGS. 3A, 3B, 3C, 3D, and 3E. As such, the second electronic device1302includes a VR enabled display unit1303to display a digital representation and a HMD1304and/or other VR enabled input units to provide user-input. For the sake of brevity only the VR enabled display unit and the HMD are illustrated. The first electronic device1301includes a display unit1305, an image capturing unit, and other unit(s) as necessary. For the sake of brevity, only the display unit is illustrated.

Referring toFIG. 13A, the first electronic device1301displays a real image1306of a front view of the real world kitchen in a camera mode on the display unit1305. For the sake of brevity, the second electronic device1302is illustrated to be in screen-off state (represented by blank screen).

Referring toFIG. 13B, the first electronic device1301captures a digital representation1307, as described inFIG. 4A, of the real world kitchen in real time using the image-capturing unit. The first electronic device1301captures the digital representation1307upon receiving a corresponding user-input, as described above with reference toFIG. 3B.

The first electronic device1301further creates metadata corresponding to the digital representation1307. The metadata includes location information and identification information of the real world kitchen, and location information of transmitting devices available in proximity to the location information of the image-capturing unit or the first electronic device1301. The first electronic device1301transmits the digital representation1307and the metadata to the second electronic device1302over the network as known in the art.

Upon receiving the digital representation1307, the VR enabled display unit1303displays the digital representation1307. The input receiving unit receives a user-input via the HMD1304and/or other VR enabled input units. The user-input is indicative of a selection of an option to view transmitting devices available in the digital representation1307or in the real world kitchen.

Referring toFIG. 13C, upon receiving the user-input, the VR enabled display unit1303displays a modified digital representation1308in a manner as described above with reference toFIGS. 3B, 3D, and 3E. The modified digital representation1308includes graphical representation(s)1309of identified transmitting device(s) and graphical representation(s)1310of corresponding status information. As illustrated, graphical representation,1309-1indicates smart oven or transmitting device and graphical representation1310-1indicates ‘no network connection’ status information. The modified digital representation1308further includes device selection option1311. A selection of the smart oven is received via the device selection option1311(represented by a dashed line connecting1309-1and1311). For the sake of brevity, the first electronic device1301is illustrated to be in screen-off state (represented by blank screen) upon transmitting the digital representation1307.

Referring toFIG. 13D, the VR enabled display unit1303displays a graphical representation1312associated with a control panel of the smart oven in response to the selection, as described above with reference toFIGS. 3B, 3D, and 3E. The smart oven may be controlled in the real world kitchen by providing control information via the graphical representation1312, as described above with reference toFIGS. 3B, 3D, and 3E. The control information is further transmitted to the smart oven such that the smart oven becomes active to perform functions in the real world kitchen. For the sake of brevity, the first electronic device1301is illustrated to be in screen-off state (represented by blank screen) upon transmitting the digital representation1307.

FIG. 14is a block diagram of a distributed processing network environment1400that includes an electronic device1401according to an embodiment of the present disclosure.

Referring toFIG. 14, the electronic device1401may be the electronic device301illustrated inFIGS. 3A and 3B. The electronic device1401may include a bus, a processor, a memory, an input/output (I/O) interface, a VR enabled display unit, a communication interface, and a virtual reality (VR) management module1402, as illustrated inFIG. 3A. For the sake of brevity, only the VR management module1402has been illustrated.

The electronic device1401is communicatively coupled with positioning system(s)1403and transmitting device(s)1404the over a network1405, as described inFIG. 3B. The electronic device1401enables controlling of transmitting device(s)1404in a real world environment through VR. To this end, in the distributing processing network environment1400, as described earlier, another electronic device, or a plurality of electronic devices, may perform some of the operations performed by the electronic device1401. Accordingly, the electronic device1401is communicatively coupled with a device information determining unit1406, an external device, over the network1405.

The device information determining unit1406may include a bus, a processor, a memory, an input/output (I/O) interface, a display unit, a communication interface, and a management module1407. For the sake of brevity, only the management module1407has been illustrated. Further, the electronic device1401and the device information determining unit1406are communicatively coupled with a digital representations (DR) database, a status and location information (SLI) database, and a graphical representation (GR) database, as illustrated and described in reference toFIG. 3b. For the sake of brevity, the databases have not been illustrated.

In accordance with an embodiment of the present disclosure, the electronic device1401in conjunction with the device information determining unit1406enables viewing and controlling of transmitting devices through VR. Accordingly, the VR management module1402in the electronic device1402includes a second digital representation unit (SDRU)1408, a rendering unit (RU)1409, an input receiving unit (IRU)1410, and a control unit (CU)1411. Further, the management module1407of the device information determining unit1406includes a first digital representation unit (FDRU)1412, identification unit (IU)1413, and a status and location unit (SLU)1414.

In accordance with one embodiment of the present disclosure, the electronic device1401or specifically the VR management module1402synthesizes a digital environment of real world environment. Accordingly, the input receiving unit1410receives a user-input from a VR enabled input-receiving unit1415to synthesize a digital representation of the real world environment. Based on the user-input, the FDRU1412obtains a digital representation of the real world environment, as described above with reference toFIG. 3A. In an example, the FDRU1412obtains the digital representation of the real world environment when the electronic device1401captures an image of the real world environment, as described above with reference toFIGS. 5A and 5B. In another example, the FDRU1412obtains the digital representation of the real world environment when the electronic device1401receives an image of the real world environment from an external electronic device1416over the network1405, as described above with reference toFIGS. 13A and 13B.

Upon obtaining the digital representation, the identification unit1413identifies the transmitting device(s)1404from the digital representation and subsequently, the status and location unit1414obtains at least one of status information and location information of identifies the transmitting device(s)1404, as described above with reference toFIG. 3B. Upon receiving the status information and/or location information, the SDRU1408creates and renders a modified digital representation of the real world environment on the display unit. The modified digital representation includes a graphical representation of said identified transmitting device(s)1404in conjunction with the status information and/or location information, as described above with reference toFIG. 3B.

Further, upon rendering the modified digital representation, the input receiving unit1410receives a user-input on the modified digital representation from the VR enabled input unit1415. The user-input is indicative of control information corresponding to selected transmitting device(s) on the modified digital representation. Upon receiving the control information, the control unit1411transmits the control information to the selected transmitting device(s) to control their operation, as described above with reference toFIG. 3B. Thereafter, the status and location unit1414obtains updated status information and/or location information of the selected transmitting device(s) in accordance with the control information, as described above with reference toFIG. 3B. Based on the updated status information and location information, SDRU1408modifies/updates and renders the modified digital representation on the display unit, as described above with reference toFIG. 3B.

FIG. 15is a flowchart of a method1500for synthesizing digital representation of an environment for viewing transmitting devices, in accordance with one embodiment of the present disclosure. The order in which the method is described is not intended to be construed as a limitation, and any number of the described method blocks may be combined in any order to implement the method, or an alternative method. Additionally, individual blocks may be deleted from the method without departing from the spirit and scope of the subject matter described herein. Furthermore, the method may be implemented in any suitable hardware, software, firmware, or combination thereof

Referring toFIG. 15, at step1501, a digital representation of a real world environment is obtained. The digital representation includes at least one transmitting device. For example, the FDRU318obtains the digital representation.

At step1502, said at least one transmitting device is identified from said digital representation. For example, the identification unit320identifies the transmitting devices316from the digital representation.

At step1503, at least one of status information and location information of said at least one transmitting device is obtained. For example, the status and location unit321obtains the status information and/or the location information of the transmitting devices316.

At step1504, a modified digital representation of the real world environment is created. The modified digital representation includes a graphical representation of said at least one transmitting device in conjunction with the at least one of status information and location information. For example, the SDRU319creates the modified digital representation.

FIG. 16is a flowchart of a method1600for controlling the transmitting devices through the digital representation, in accordance with another embodiment of the present disclosure. The order in which the method is described is not intended to be construed as a limitation, and any number of the described method blocks may be combined in any order to implement the method, or an alternative method. Additionally, individual blocks may be deleted from the method without departing from the spirit and scope of the subject matter described herein. Furthermore, the method may be implemented in any suitable hardware, software, firmware, or combination thereof.

Referring toFIG. 16, at step1601, the digital representation of the real world environment is rendered on a virtual reality (VR) enabled display unit. The digital representation includes a graphical representation of said at least one transmitting device in conjunction with a status information of said at least one transmitting device. For example, the rendering unit322renders the digital representation on the VR enabled display unit306.

At step1602, a user-input is received through a VR enabled input unit. The user-input is indicative of control information of said at least one transmitting device. For example, the input receiving unit323receives the user-input from the VR enabled input unit325.

At step1603, one or more operations of said at least one transmitting device in the real world environment are controlled in accordance with the control information. For example, the control unit324transmits the control information to the transmitting device316to modify operation of the transmitting device316in accordance with the control information.

FIG. 17is a block diagram of an electronic device1700according to an embodiment of the present disclosure.

Referring toFIG. 17, the electronic device1700may be the VR enabled input unit325as illustrated inFIG. 3Band the VR enabled input unit315as illustrated inFIG. 13, in one example embodiment. In such example embodiment, the electronic device1700is external to a first electronic device and is communicatively coupled to the first electronic device. The electronic device1700may include at least one of a micro controller unit (MCU)1710, a communication module1720, a sensor module1730, an input module1740, an eye tracking module1750, a vibrator1752, an adjustable optics module1754, a power management module1760, and a battery1762.

The MCU1710may be a controller of the electronic device1700, for controlling other components (for example, the communication module1720, the sensor module1730, the input module1740, the eye tracking module1750, the vibrator1752, the adjustable optics module1754, and the power management module1760) by driving an operating system (OS) or an embedded software program. The MCU1710may include a processor and a memory.

The communication module1720may electrically connect other electronic device (for example, the electronic devices301and1301) to the electronic device1700by wired or wireless communication and perform data transmission and reception between the electronic devices. According to an embodiment, the communication module1720may include a USB module1721, a WiFi module1722, a BT module1723, an NFC module1724, and a GPS module1725. According to an embodiment, at least three of the USB module1721, the WiFi module1722, the BT module1723, the NFC module1724, and the GPS module1725may be included in a single integrated circuit (IC) or IC package.

The sensor module1730may measure a physical property or sense an operation state of the electronic device1700and convert the measured or sensed information to an electrical signal. The sensor module1730may include at least one of, for example, an accelerometer1731, a gyro sensor1732, a geomagnetic sensor1733, a magnetic sensor1734, a proximity sensor1735, a gesture sensor1736, a grip sensor1737, a biometric sensor1738, and an access sensor1739. The electronic device1700may sense a head motion of a user wearing the electronic device1700, using at least one of the accelerometer1731, the gyro sensor1732, and the geomagnetic sensor1733. The electronic device1700may sense whether the electronic device1700is worn or removed, using the proximity sensor1735or the grip sensor1737. According to an embodiment, the electronic device1700may determine whether the user wears the electronic device1700by at least one of Infrared (IR) recognition, pressure recognition, and sensing of a capacitance (or dielectric constant) variation involved in wearing of the electronic device1700. The gesture sensor1736may sense a hand or finger motion of the user and receive the hand or finger motion as an input to the electronic device1700. The electronic device1700may sense proximity of an object to the user by the proximity sensor1735. Alternatively or additionally, the sensor module1730may include a biometric sensor such as an e-node sensor, an Electromyography (EMG) sensor, an Electroencephalogram (EEG) sensor, an Electrocardiogram (ECG) sensor, an iris sensor, and a finger print sensor and thus may recognize vital information about the user. The sensor module1730may further include a control circuit for controlling at least one of internal sensors.

The input module1740may receive an input from the user. The input module1740may include a touch pad1741and a button1742. The touch pad1741may recognize a touch input in at least one of a capacitive manner, a resistive manner, an IR manner, and an ultrasonic manner. The touch pad1741may further include a control circuit. If the touch pad1741operates in the capacitive manner, the touch pad1741may recognize a physical contact or proximity. The touch pad1741may further include a tactile layer. In this case, the touch pad1741may provide a tactile response to the user. The button1742may be, for example, a physical button, an optical key, or a keypad.

The power management module1760may manage power of the electronic device1700. The power management module1760may include, for example, a power management integrated circuit (PMIC), a charger IC, or a battery gauge.

The PMIC may be mounted, for example, on an IC or a system on a chip (SOC) semiconductor. A battery may be charged wiredly or wirelessly. The charger IC may charge the battery and prevent introduction of overvoltage or overcurrent from a charger. According to an embodiment, the charger IC may operate wiredly and/or wirelessly. Wireless charging may be performed, for example, by magnetic resonance, magnetic induction, or electromagnetic waves. A circuit for wireless charging, for example, a coil loop, a resonance circuit, or a rectifier may be added.

The battery fuel gauge may measure, for example, a charge level, a voltage while charging, a current, or temperature of the battery1762. The battery1762may store electricity and supply power. The battery1762may include a rechargeable battery or a solar battery.

The eye tracking module1750may track the eyes of the user by at least one of an electrical ocular graph (EOG) sensor, a coil system, a dual purkinje system, a bright pupil system, and a dark pupil system. Further, the eye tracking module1750may include a micro camera for tracking the eyes.

The adjustable optics module1754may measure an inter-pupil distance (IPD) of the user so that the user may view an image suitable for the user's sight. The electronic device1700may adjust the distance between lenses according to the IPD of the user measured by the adjustable optics module1754. The electronic device1700may transmit the IPD of the user measured by the adjustable optics module1754to the first electronic device to adjust a displayed position of a screen on the display of the first electronic device.

The MCU1710may transmit a motion signal sensed through the motion sensor of the sensor module1730and transmit the motion signal to the first electronic device. The motion sensor may be at least one of the accelerometer1731, the gyro sensor1732, and the geomagnetic sensor1733.

The MCU1710may sense access of an object to the user of the electronic device1700through the access sensor1739and transmit an access sensing signal to the first electronic device. The MCU1710may measure a direction from which the object accesses the user of the second electronic device1700through the access sensor1739and transmit information indicating the direction to the first electronic device.

The access sensor1739may be a space recognition sensor such as an IR sensor, an ultrasonic sensor, a radio frequency (RF) sensor, or a radar. A Wisee sensor or an Allsee sensor may be used as the RF sensor. According to an embodiment, a wireless communication module may be used as the access sensor1739. The wireless communication module may be at least one of the WiFi module1722, the BT module1723, the NFC module1724, and the GPS module1725. When an object accesses the second electronic device, the received signal strength of a wireless communication signal received at the wireless communication module may get weak. If the received signal strength quickly decreases by a value larger than a predetermined threshold while the user of the second electronic device is stationary, the MCU1710may determine that the object is accessing. In addition, the MCU1710may determine a direction in which the received signal strength quickly decreases by the value larger than the predetermined threshold to be a direction from which the object is accessing.

FIG. 18is a block diagram of an electronic device1800according to an embodiment of the disclosure. The electronic device1800may be, for example, a whole or a part of the electronic device301illustrated inFIGS. 3A and 3B, and the electronic device1401illustrated inFIG. 14.

Referring toFIG. 18, the electronic device1800may include one or more application processors (APs)1810, a communication module1820, a subscriber identification module (SIM) card1824, a memory1830, a sensor module1840, an input device1850, a display1860, an interface1870, an audio module1880, a camera module1891, a power management module1895, a battery1896, an indicator1897, and a motor1898.

The AP1810may have a configuration equal or similar to the processor303as described above with reference toFIG. 3A. The AP1810may control one or more hardware or software components that are connected to the AP1810by executing an OS or an application program and may perform processing or computation of various types of data including multimedia data. The AP1810may be implemented, for example, as a SoC). According to an embodiment, the AP1810may further include a graphics processing unit (GPU). The AP1810may also include at least some of the components of the VR management module308as illustrated inFIG. 3C.

The communication module1820may have a configuration equal or similar to the communication interface307as described above with reference toFIG. 3A. The communication module1820may transmit and receive data in communication between the electronic device1800and other electronic devices. According to an embodiment, the communication module1820may include a cellular module1821, a WiFi module1823, a BT module1825, a GPS module1827, an NFC module1828, and a RF module1829.

The cellular module1821may provide services such as voice call, video call, SMS, or the Internet, via a communication network (for example, LTE, LTE-A, CDMA, WCDMA, UMTS, WiBro, or GSM). The cellular module1821may identify and authenticate electronic devices within a communication network, using a SIM card (for example, the SIM card1824). According to an embodiment, the cellular module1821may perform at least a part of the functionalities of the AP1810. For example, the cellular module1821may perform at least a part of multimedia control functionality.

According to an embodiment, the cellular module1821may include a communication processor (CP). The cellular module1821may, for example, be implemented as SoC. Although components such as the cellular module1821(for example, the CP), the memory1830, or the power management module1895are shown inFIG. 18as configured separately from the AP1810, the AP1810may include, or be integrated with, one or more of the foregoing components (for example, the cellular module1821).

According to an embodiment, the AP1810or the cellular module1821(for example, the CP) may process instructions or data received from at least one of a non-volatile memory or other components by loading the instructions or the data in a volatile memory. In addition, the AP1810or the cellular module1821may store at the non-volatile memory at least one of data received from at least one of other components or data generated by at least one of the other components.

Each of the WiFi module1823, the BT module1825, the GPS module1827, and the NFC module1828may include, for example, a processor that may process data received or transmitted by the respective modules. AlthoughFIG. 18shows the cellular module1821, the WiFi module1823, the BT module1825, the GPS module1827, and the NFC module1828as separate components, any combination (for example, two or more) of the cellular module1821, the WiFi module1823, the BT module1825, the GPS module1827, and the NFC module1828may be included in an IC or an IC package according to an embodiment. For example, at least some of the processors corresponding to the respective cellular module1821, the WiFi module1823, the BT module1825, the GPS module1827, or the NFC module1828may be implemented as a single SoC. For example, a CP corresponding to the cellular module1821and a WiFi processor corresponding to the WiFi module1823may be implemented as a single SoC.

The RF module1829may transmit and receive data, for example, RF signals. While not shown, the RF module1829may include a transceiver, a power amplifier module (PAM), a frequency filter, or a low noise amplifier (LNA). The RF module1829may further include one or more components for transmitting and receiving electro-magnetic (EM) waves in free space, such as conductors or conductive wires. AlthoughFIG. 18shows that the cellular module1821, the WiFi module1823, the BT module1825, the GPS module1827, and the NFC module1828share the single RF module1829, at least one of the cellular module1821, the WiFi module1823, the BT module1825, the GPS module1827, or the NFC module1828may transmit and receive RF signals via a separate RF module according to an embodiment.

The SIM card1824may be a card including a SIM, and may be configured to be inserted into a slot disposed at a specified location of the electronic device. The SIM card1824may include a unique identifier (for example, integrated circuit card identifier (ICCID)) or subscriber information (for example, international mobile subscriber identity (IMSI)).

The memory1830may include an internal memory1832or an external memory1834. The internal memory1832may be at least one of, for example, a volatile memory (for example, dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)) or a non-volatile memory (for example, one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, NAND flash memory, or NOR flash memory).

According to an embodiment, the internal memory1832may be a solid state drive (SSD). The external memory1834may be, for example, a flash drive (for example, a compact flash (CF) drive, a secure digital (SD), a micro secure digital (micro-SD), a mini secure digital (mini-SD), an extreme digital (xD), or a memory stick). The external memory1834may be operatively coupled to the electronic device1800via various interfaces. According to an embodiment, the electronic device1800may further include recording devices (or recording media) such as a hard disk drive (HDD).

In one embodiment, the internal memory1832may have a configuration equal or similar to the memory304as described above with reference toFIG. 3A. In one embodiment, the internal memory1832may have a configuration equal or similar to the DR database326as described above with reference toFIGS. 3B and 3D. In one embodiment, the internal memory1832may have a configuration equal or similar to the SLI database331as described above with reference toFIG. 3B. In one embodiment, the internal memory1832may have a configuration equal or similar to the GR database332as described above with reference toFIGS. 3B and 3D. In one embodiment, the external memory1834may have a configuration equal or similar to the DR database326as described in above with reference toFIG. 3B. In one embodiment, the external memory1834may have a configuration equal or similar to the SLI database331as described above with reference toFIG. 3B. In one embodiment, the external memory1834may have a configuration equal or similar to the GR database332as described above with reference toFIGS. 3B and 3D.

The sensor module1840may measure physical properties or detect operational states associated with the electronic device1800, and convert the measured or detected information into electric signals. The sensor module1840may include at least one of, for example, a gesture sensor1840A, a gyro sensor1840B, an atmospheric pressure sensor1840C, a magnetic sensor1840D, an accelerometer sensor1840E, a grip sensor1840F, a proximity sensor1840G, a color sensor1840H (for example, a red, green, blue (RGB) sensor), a biometric sensor1840I, a temperature/humidity sensor1840J, an illuminance sensor1840K, or an ultra violet (UV) light sensor1840M. Additionally or alternatively, the sensor module1840may include, for example, an electrical-nose sensor, an EMG sensor, an EEG sensor, an IR sensor, an iris sensor, or a finger print sensor. The sensor module1840may further include a control circuit for controlling one or more sensors included therein.

The input device1850may include a touch panel1852, a (digital) pen sensor1854, a key1856, or an ultrasonic input device1858. The touch panel1852may detect a touch input using at least one of, for example, capacitive, resistive, infrared, and ultrasonic methods. The touch panel1852may further include a control circuit. A capacitive-type touch panel may detect physical touch inputs or proximity inputs. The touch panel1852may further include a tactile layer, which may provide haptic feedback to the user.

The (digital) pen sensor1854may be implemented, for example, using methods identical to or similar to receiving a touch input from a user, or using a separate detection sheet. The key1856may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device1858may be a device configured to identify data by detecting, using a microphone (for example, a microphone1888), ultrasonic signals generated by a device capable of generating the ultrasonic signal. The ultrasonic input device1858may detect data wirelessly. According to an embodiment, the electronic device1800may receive a user input from an external device (for example, a computer or a server) connected to the electronic device1800using the communication module1820. The input device1850may further have a configuration equal or similar to the VR enabled input unit325as described above with reference toFIG. 3B.

The display module1860may include a panel1862, a hologram device1864, or a projector1866. The panel1862may be, for example, a LCD or an active-matrix organic light-emitting diode (AM-OLED) display. The panel1862may be configured to be, for example, flexible, transparent, or wearable. The panel1862and the touch panel1852may be implemented as a single module. The hologram device1864may utilize the interference of light waves to provide a three-dimensional image in empty space. The projector1866may provide an image by projecting light on a display. The display may be positioned, for example, inside or outside the electronic device1800. According to an embodiment, the display module1860may further include a control circuit for controlling the panel1862, the hologram device1864, or the projector1866. The display module1860may further have a configuration equal or similar to the VR enabled display unit306as described inFIG. 3a.

The interface1870may include, for example, a high-definition multimedia interface (HDMI)1872, a USB1874, an optical interface1876, or a D-sub connector1878. Additionally or alternatively, the interface1870may include, for example, a mobile high-definition link (MHL) interface, an SD/MMC, or an Infrared Data Association (IrDA) interface. The interface1870may be incorporated into, for example, the communication interface307as described above with reference toFIG. 3A.

The audio module1880may encode/decode a voice into an electrical signal, and vice versa. At least a part of components of the audio module1880may be incorporated in, for example, the I/O interface305as described above with reference toFIG. 3A. The audio module1880may process audio information input into, or output from, for example, a speaker1882, a receiver1884, an earphone1886, or the microphone1888.

The camera module1891may capture still images or a video. According to an embodiment, the camera module1891may include one or more image sensors (for example, a front sensor or a rear sensor), a lens, an image signal processor (ISP), or a flash (for example, a LED or a Xenon lamp).

The power management module1895may manage power of the electronic device1800. The power management module1895may include, for example, a PMIC, a charger IC, or a battery gauge.

The PMIC may be disposed, for example, in an IC or a SoC semiconductor. The charging method for the electronic device1800may include wired or wireless charging. The charger IC may charge a battery, or prevent excessive voltage or excessive current from a charger from entering the electronic device1800. According to an embodiment, the charger IC may include at least one of a wired charger IC or a wireless charger IC. The wireless charger IC may be, for example, a magnetic resonance type, a magnetic induction type or an electromagnetic wave type, and may include additional circuits for wireless charging, such as a coil loop, a resonance circuit, or a rectifier.

The battery gauge may measure, for example, a charge level, a voltage while charging, current, or temperature of the battery1896. The battery1896may store or generate electricity and supply power to the electronic device1800using the stored or generated electricity. The battery1896may include, for example, a rechargeable battery or a solar battery.

The indicator1897may indicate one or more states (for example, boot status, message status, or charge status) of the electronic device1800or a part of the electronic device1800(for example, the AP1810). The motor1898may convert an electrical signal into a mechanical vibration. The electronic device1800may include a device for supporting mobile TV (for example, a GPU). The device for supporting mobile TV may process media data compliant with, for example, digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or Media Flow.

Each of components of an electronic device described above according to the present disclosure may include one or more components, and each component's name may vary according to the type of the electronic device. The electronic device according to the present disclosure may include at least one of the above-described components, and some may be omitted or additional components may be included. In addition, some of the components of the hardware according to the present disclosure may be combined into a single component and perform functions identical to those of the respective components before their combination. Similarly, some of the components of the hardware according to the present disclosure may be split into a plurality of entities that collectively perform functions identical to those of the respective component before their split.

The term “module” as used herein may include its ordinary meaning including, but not limited to, for example, a unit of one, or a combination of two or more, hardware, software or firmware. The term “module” may be used interchangeably with a term such as unit, logic, logical block, component, or circuit. A module may be the smallest unit for performing one or more functions, or a portion thereof. A module may be implemented mechanically or electronically. For example, a module according to the present disclosure may include at least one of a known or to-be-developed application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or programmable logic device that perform certain operations.

According to various embodiments, at least a part of devices (for example, modules or their functions) or methods (for example, operations) according to the present disclosure may be implemented, for example, in the form of a programming module, as commands stored in a non-transitory computer-readable storage medium. When a command is executed by one or more processors (for example, the processor303), the one or more processors may execute a function corresponding to the command. The non-transitory computer-readable storage medium may be, for example, the memory304. At least a part of the programming module may be implemented (for example, executed) by the processor303. At least a part of the programming module may include, for example, a module, a program, a routine, a set of instructions, and/or a process to execute one or more functions.

The non-transitory computer-readable recording medium may include any kind of hardware device configured specially to store a program command (for example, a programming module). Examples of the hardware device may include magnetic media such as a hard disk, floppy disk, and a magnetic tape, optical media such as a compact disc read only memory (CD-ROM) and a digital versatile disc (DVD), magneto-optical media such as a optical disk, a ROM, a RAM, a flash memory, and the like. The program command may include a premium language code that may be executed in a computer using an interpreter as well as a mechanical code produced by a compiler. The above-mentioned hardware device may be implemented as one or more software modules to perform the operations of the present disclosure and vice versa.

While certain embodiments of the present disclosure have been illustrated and described herein, the present disclosure is not intended to be limited thereto. The present disclosure may be otherwise variously embodied, and practiced within the scope of the present disclosure as defined by the appended claims and their equivalents.