System and method of IOT device control using augmented reality

A method and system for remotely controlling an external device includes storing, in a computing device, a plurality of device profiles, each device profile including connection data, one or more control commands, and at least one associated image; receiving, by an imaging device interfaced with the computing device, a live rendering of a physical environment; detecting, by the computing device, a registered image in the live rendering of the physical environment; identifying, in the computing device, a specific device profile where the at least one associated image includes the detected registered image; establishing, by the computing device, a communication channel with an external device using the connection data included in the identified specific device profile; and electronically transmitting, by the computing device, at least one of the one or more control commands included in the identified specific device profile to the external device using the established communication channel.

FIELD

The present disclosure relates to the remote control of an external device via an augmented reality interface, specifically the display of a live rendering of an environment where external devices may be selectable for control through an augmented reality interface interacting with the live rendering to provide for more convenient and intuitive access to digital devices.

BACKGROUND

As technology develops and improves, more and more devices are created that are network-enabled in one manner or another, enabling the user to control or otherwise access the device remotely through a separate computing device. These network-enabled devices are often referred to as part of the “internet of things.” Examples of traditional types of digital devices that have begun to be adopted into the internet of things or are otherwise network-enabled include thermostats, doorbells, automobiles, refrigerators, light switches, door locks, and garage doors. Such devices are able to connect to a local area network provided by the device owner, or in some cases an alternative network that may provide access to the internet, such as a cellular communication network, where the device owner may be able to interact with the device via an application program executed on their own computing device or a web page.

The control of such devices for a user using their own computing device can offer a level of convenience, for instance, enabling the user to adjust the temperature in their house even while away from their house thanks to the internet connectivity. However, because each device might require the use of its own specialized application program or webpage, as a user purchases more and more internet of things devices, it begins to be inconvenient and time consuming to discover the digital devices to be managed, identify their capabilities, locate the proper application program associated therewith, and adjust settings and controls for every single device that needs to be managed. Furthermore, it may become difficult for the user to remember which application program corresponds to which device, or to identify the proper application program for a device, particularly if the device is one not interacted with often by the user.

Thus, there is a need for a technical solution to provide for greater convenience and intuitive interaction with internet of things devices by a user without the use of application programs and web pages that scale with the number of devices being controlled.

SUMMARY

The present disclosure provides a description of systems and methods for remotely controlling an external device. A user utilizes a computing device that is configured to connect to each internet of things device that the user wishes to be controlled. The computing device is able to capture a live rendering of the user's environment, such as through a camera located on the device, and enable the user to register images in the live rendering via an augmented reality interface that can be associated with each internet of things device. At subsequent times when the user, via the augmented reality interface, captures a live rendering of their environment that includes a registered image, the registered image is detected and enables the user to control, in the same application program that provides the augmented reality interface, the associated device. As a result, the user can control any and all devices via a single application program and interface, and select the devices through an intuitive interface or by simply pointing the camera at the device, such as by directly touching the device in the augmented reality interface, providing for easier selection of devices than traditional methods of hunting through a list of installed application programs and remembering which program corresponds to which internet of things device.

A method for remotely controlling an external device includes: storing, in a computing device, a plurality of device profiles, wherein each device profile includes connection data, one or more control commands, and at least one associated image; receiving, by an imaging device interfaced with the computing device, a live rendering of a physical environment; detecting, by the computing device, a registered image in the live rendering of the physical environment; identifying, in the computing device, a specific device profile of the plurality of device profiles where the at least one associated image includes the detected registered image; establishing, by the computing device, a communication channel with an external device using the connection data included in the identified specific device profile; and electronically transmitting, by the computing device, at least one of the one or more control commands included in the identified specific device profile to the external device using the established communication channel.

A system for remotely controlling an external device includes: a device database of computing device configured to store a plurality of device profiles, wherein each device profile includes connection data, one or more control commands, and at least one associated image; an imaging device interfaced with the computing device configured to receive a live rendering of a physical environment; a detection module of the computing device configured to detect a registered image in the live rendering of the physical environment; a querying module of the computing device configured to execute a query on the device database to identify a specific device profile of the plurality of device profiles where the at least one associated image includes the detected registered image; a communications interface of the computing device configured to establish a communication channel with an external device using the connection data included in the identified specific device profile; and a transmitting device of the computing device configured to electronically transmit at least one of the one or more control commands included in the identified specific device profile to the external device using the established communication channel.

A non-transitory computer readable media has instructions operable to cause one or more processors to perform the operations including: storing, in a computing device, a plurality of device profiles, wherein each device profile includes connection data, one or more control commands, and at least one associated image; receiving, by an imaging device interfaced with the computing device, a live rendering of a physical environment; detecting, by the computing device, a registered image in the live rendering of the physical environment; identifying, in the computing device, a specific device profile of the plurality of device profiles where the at least one associated image includes the detected registered image; establishing, by the computing device, a communication channel with an external device using the connection data included in the identified specific device profile; and electronically transmitting, by the computing device, at least one of the one or more control commands included in the identified specific device profile to the external device using the established communication channel.

DETAILED DESCRIPTION

System for Remote Control of External Devices Via Augmented Reality

FIG. 1illustrates a system100for the remote control of internet of things devices via the use of an augmented reality interface to provide a user control of various internet of things devices through a single application program via a series of registered images.

The system100may include a computing device102. The computing device102, discussed in more detail below, may be a device accessible by a user that may be configured to capture a live rendering of a physical environment104to display an augmented reality interface in relation thereto to enable the user to remotely control one or more internet of things devices, illustrated inFIG. 1as devices106,110, and112. The computing device102may be any type of computing device configured to capture a live rendering of a physical environment104and provide an interface to the user thereof for performing the functions discussed herein, such as a specifically configured cellular phone, smart phone, smart watch, wearable computing device, implantable computing device, desktop computer, notebook computer, tablet computer, smart television, augmented reality glasses, etc. The computing device102may include at least a camera or other type of optical imaging device configured to capture a live rendering of the physical environment104. In some cases, the live rendering may be such that, as the user moves the computing device102, the live rendering may update based on the movement of the computing device102, such as to display new portions of the physical environment104captured by the computing device102. For instance, the live rendering may be an image capture of the physical environment104based on what may be captured by a camera of the computing device102, where the image updates based on the movement of the camera of the computing device102.

The user may utilize the computing device102to control one or more of the external devices through images of the physical environment104as detected in the live rendering presented on the computing device102. In the system100, the computing device102may detect potential internet of things devices that may be controlled by the computing device102. In an exemplary embodiment, a single application program executed by the computing device102may be configured to capture and display the live rendering of the physical environment104and detect and control external devices as discussed herein. Such devices may be controlled either directly by the computing device102or via one or more intermediate devices. For instance, one external device may be a light, where the light may be controlled directly by the computing device102or where the light may be controlled through a separate device (e.g., installed at the light switch) where the computing device102may communicate therewith. In some cases, the computing device102may detect all internet of things or other network-enabled devices connected to an area network to which the computing device102is connected. In some instances, the user of the computing device102may be able to manually add internet of things devices for control via the computing device102, such as through dedicated application programs, web page interfaces, application programming interfaces, etc. In some cases, the user may be presented with devices directly viewable in the live rendering of the physical environment104, or in the direction which the computing device102is faced (e.g., would be viewable in live rendering beyond any walls, floors, or other impediments).

The user of the computing device102may register each external device they wish to control through the application program executed by the computing device102. Registration of a device for remote control may include the selection of the external device as well as the capturing of one or more images in the live rendering of the physical environment to be associated with the external device. Selection of the external device may be performed in any suitable method. In one instance, the application program executed by the computing device102may display a list of all detected external devices, where the user may be able to select a specific external device from the list. In another instance, the application program may be configured to attempt to identify a detected external device in the live rendering of the physical environment104, or in some cases in a captured image as selected by the user, such as using data associated with the external device and image recognition technology. In yet another instance, the user may manually add the external device to the application program, such as through its own associated application program, when registering the external device.

The one or more images may be captured in the application program via an augmented reality interface provided with the live rendering of the physical environment104. The user may, via the augmented reality interface, select any or all of the currently displayed live rendering as the captured image to be associated with a specific external device. In an exemplary embodiment, any captured image may be able to be associated with an external device. For instance, in the example illustrated inFIG. 1, the physical environment, the user may wish to control a light110. The user may capture an image of the light110itself in the live rendering of the physical environment104or may capture an image of a light switch106that is used to control the light110. In some cases, the light switch106may be an internet of things device, which may be the selected external device that the user wishes to associate with the captured image. In another example illustrated inFIG. 1, the physical environment104where the user is located may include a vent112. The user may capture an image of the vent112and associate it with a thermostat or other internet of things device that controls the climate or that the user otherwise associates with the vent, even if the device is located in a different portion of the environment, such as in a different room of a house. In some cases, the user may select a captured image that is not directly related to the external device they wish to control. For example, the user may capture an image of a movie poster on their wall, which they may associate with their refrigerator as the internet of things device to be controlled. In these cases, any image that may be captured of the live rendering of the physical environment104through the augmented reality interface may be associated with an external device for connection thereof. In some cases, the user may select multiple images for association with an external device, such as by capturing images of every vent and thermostat in a house for association with an internet-enabled climate control system.

The computing device102may register any captured image(s) with the selected external device. When the user wants to control a device, the user may access, in the computing device102, a live rendering of their physical environment104. The user may point the computing device102towards an object or area that the user associates with the device they wish to control, such that the live rendering captures that object or area. The computing device102may be configured to detect a previously registered image in the live rendering, corresponding to the object or area that the user associates with the device that was captured in the registration process. The detection of a previously registered image may be performed using any suitable image recognition method. In an exemplary embodiment, the computing device102may display, via the augmented reality interface, an indicator on the live rendering to indicate the detected, registered image. The indicator may be, for example, a green box that outlines or highlights the captured image. In cases where the registered image may be of the device itself or a specific object, the indicator may be an outline of the device or object. In some instances, the user may choose the indicator that may be used for a registered image, such as by selecting different shapes, colors, or symbols for the indicator for easier selection or recognition of the device to which a captured image corresponds.

The computing device102may detect the captured image, display the indicator to the user, and provide the user with the ability to remotely control the associated external device. For instance, in the example illustrated inFIG. 1, if the user navigates the computing device102to capture a live rendering of the physical environment104that includes the light switch106, the computing device102may detect the light switch106and enable the user to control the light110. In cases where a live rendering may include multiple captured images, the augmented reality interface may display an indicator for each detected, registered image, where the user may select (e.g., by tapping on the augmented reality interface on the desired indicator, selecting from a list of detected images, etc.) one of the associated devices for control. In some cases, the selection may be performed by simply pointing the computing device102toward a registered image or object. For example, the computing device102may display a crosshair that the user could aim toward the detected image. In instances where no registered image is detected in the live rendering, selection of an area or object by the user may initiate the registration process to register a new image (e.g., of the selected area or object) for association with an external device.

When an external device is to be controlled, the computing device102may establish a communication channel108with the external device. The communication channel108may utilize the Internet or any other suitable method of communication with the external device, such as via an alternative communication network such as the area network, Bluetooth, Zigbee, radio frequency, near field communication, etc. In some cases, multiple types of networks may be utilized in establishing the communication channel108. For instance, the computing device102may first establish communication with the external device via the internet, but may then connect directly to the external device via the area network, such as after first receiving connection data via the internet. The computing device102may utilize any protocols or standards in communicating with the external device as may be associated therewith, which may vary from one external device to another. In such cases, the computing device102may identify such data for communication when detecting the external device or when the external device is registered for control. In some instances, the application program used by the user may receive the communication data from a secondary application program on the computing device102, such as one that may be directly associated with the respective external device. In some embodiments, the computing device102may be configured to display messages or notifications to the user thereof regarding the establishing of the communication channel108. For instance, the computing device102may display a confirmation message if the connection is successfully established with the target external device, or may display an error message if the communication channel108is not establish. In some such instances, error messages may be accompanied by options for selection by the user to retry the connection or to adjust settings (e.g., authentication information that may be required by the external device). In some cases, such messages or notifications may be displayed on the live rendering. For instance, in one example, the indicator that indicates the detected image may reflect the connection status, such as a box outlining the detected image being initially gray, but turning green if the communication channel108is successfully established or turning red if the communication channel108is unsuccessful.

In one embodiment, the computing device102may establish the communication channel108directly to the external device via the application program that provides the augmented reality interface for selection of the external device. In another embodiment, the application program that provides the augmented reality interface, referred to herein as the “primary” application program may communicate with the external device through an application program directly associated with that external device, referred to herein as the “secondary” application program. For instance, the primary application program may be the augmented reality application program, whereas the secondary application program may be an application program that is provided by the manufacturer of the external device for control thereof. In such cases, the primary application may utilize an application programming interface of the secondary application program where the user may be able to modify control of the external device via the primary application program, but where the controls are transmitted to the external device via the communication channel108as established with the secondary application program.

Using the established communication channel108, the user of the computing device102may be able to control the external device that is associated with the detected, registered image in the live rendering of the physical environment104. Control of the external device may include the issuing of one or more instructions that are configured to modify operation of the external device in one manner or another, where such operations may be dependent on the type and capabilities of the external device. For instance, a light switch may be turned on, off, or the illumination level modified, while a thermostat may be used to turn on or off heat or air conditioning, change temperatures, etc. Such instructions may be input via any suitable method, such as voice, gesture (e.g., directly on the computing device102or viewable thereby, such as via a camera), direct input, movement of the computing device102itself, etc. For example, the user may point the computing device102at a light switch, where the computing device102will detect the light switch and identify the device associated therewith, and then say “on” to have the associated light turn on. In some embodiments, the user instructions for control may be transmitted directly from the primary application program to the external device via the communication channel108. In other embodiments, the user instructions for control may be transmitted from the primary application program to the secondary application program, which may forward the instructions to the external device via the communication channel108. The external device may receive the instructions and may modify operation accordingly. In some embodiments, the computing device102may be configured to display information regarding the submission of instructions to the external device, such as status information regarding the success or failure of the external device to receive and/or implement the instructions. For instance, the computing device102may display a confirmation message or error message if the external device succeeds or fails, respectively, at implementing the instructions submitted by the user.

In some embodiments, the computing device102may be configured to recommend device control options for the user. Such device control options may be based on past selections of the user, registered user preferences, options identified via artificial intelligence, selections of other users of the primary application program for the same device or similar devices, current device settings, etc. For example, if the device is a thermostat the computing device102may recommend that the temperature be set to a preferred temperature of the user, and/or may base a temperature recommendation on outside conditions. Additionally, the computing device102may be configured to provide other recommendations, such as for management of the external device. For instance, if the computing device102detects that the power level of the device is low, the computing device my recommend that a battery be replaced or the device recharged. In some embodiments, the user may define, or the computing device102may suggest based on past actions of the user, a series of device control options to execute. For example, a chain of actions may be set by the user to be performed when the user enters their house, such as to adjust temperature, turn on lights, and turn on music. In some cases, such a chain of actions may be initiated by detecting an image associated with one device involved in the chain of actions, or by detecting an image directly associated with the chain of actions.

In some instances, the application program may display an interface for control of an external device directly on the live rendering, such as via the augmented reality interface which may overlay or obscure the live rendering. For example, a portion of the screen used to display the live rendering may be dedicated to displaying controls that may be used by the user for the associated external device. In another instance, an interface may be displayed on the indicated captured image that the user may interact with to modify control of the external device. For example, if the associated external device is a thermostat, a virtual representation of the thermostat or the thermostat's interface may be displayed via the augmented reality interface on the captured image. In yet another instance, when an external device is selected for control via the detected, registered image, the primary application program may be configured to launch an instance of the secondary application program to enable the user to control the external device therefrom. In another instance, the primary application program may be configured to display an instance of the secondary application program as an overlay on the augmented reality interface, such as simulating control of the external device via the secondary application program on the detected image. In some cases, the computing device102may display contextual controls that are specific to the type of control and/or its values. For instance, Boolean values may be control via an on/off switch, checkbox, or other similar input type, while a control with a range of possible values may be controlled via a slider with directional buttons that may be grayed out or hidden if the value reaches a corresponding end of the range, where the slider may also be accompanied by a display of the corresponding value.

In some embodiments, the computing device102may be configured to display data regarding the current and/or adjusted control settings of the external device, as well as any additional settings or characteristics that may be useful to the user of the computing device102. For instance, if the external device is a light, the computing device102may display a status indicator that indicates the current status of the light (e.g., on, off, current intensity level, etc.). In such an instance, the computing device102may be configured to update the display following the issuing of control instructions by the user. For example, if the light is initially off, but is turned on by the user using the methods discussed herein, the computing device102may update the display to reflect the light being turned on. In another example, the external device may be a thermostat, which, as discussed above, may be controlled through a registered image of a vent as opposed to the thermostat itself. In order to assist the user in controlling the thermostat, the computing device102may display (e.g., accompanying the detected image, overlaid on the detected image, etc.) current status and settings data of the thermostat, such as the current temperature, desired temperature, heating/cooling schedules, etc. As with the light switch, the computing device102may update such a display when the settings of the device are adjusted by the user. In some cases, the computing device102may register itself with the external device (e.g., or secondary application associated with the external device) to receive updated status and settings information, which may be received as part of the established communication channel108or received separately based on the capabilities and configuration of the external device, such as through push notifications pushed the computing device102. In some instances, the computing device102may poll the external device or secondary application for updated settings and status information, which may be performed when the communication channel108is established, at a predetermined interval when the communication channel108is operable, once the communication channel108is closed, at a predetermined interval (e.g., which may be different than the above-referenced interval) when no direct communication channel108is established, etc. In these instances, such updated settings and status information may be reflected in the display of the computing device102for the external device, if applicable.

In some embodiments, the computing device102may be configured to provide a live rendering of a simulated or virtual representation of the physical environment104through which the user may be able to select external devices for control as an alternative to the real-world environment to which it corresponds. For instance, the computing device102may, as live renderings of a real-world environment are captured, generate a spatial representation of the physical environment104, the detected images, and the detected devices. In such embodiments, the user of the computing device102may use the simulated representation (e.g., via virtual reality) of the physical environment104to detect captured images and control external devices. For example, the user may, while physically in the physical environment104, register the vent112in the physical environment104to control their thermostat in their house. Then, the user may be at work but desire to control their home thermostat, where, in such a case, the computing device102may display a virtual representation of the physical environment104where the user may navigate in the virtual representation to locate the vent112, where the computing device102may detect the captured image associated therewith and provide the user with an interface to control the thermostat. In such embodiments, the virtual or simulated representation of the physical environment104may be operated analogous to the user and computing device102being physically located in the real-world environment. As discussed herein, “physical environment” may thus refer to real-world environments as well as virtual or spatial representations of a real-world environment. Methods for the creation of a spatial representation of an area including the location of external devices for control can be found in U.S. patent application Ser. No. 15/419,455, entitled “Spatial Representation of Network Elements,” by Charles Ross Denneler et al., filed on Jan. 30, 2017, which is herein incorporated by reference in its entirety.

Methods and systems discussed herein may enable a user to control a nearly unlimited number of internet of things devices via a single application program on a computing device102using an augmented reality interface on a physical environment104where the user may associate images captured in the physical environment104with the external devices. By using captured images and the live rendering of the physical environment104, the user may control the external devices without having to remember which secondary application program is associated with which external device, as well as without having to go through one secondary application program after another, even the locating of which can be time consuming, when multiple devices are to be controlled. In addition, enabling the user to associate their own images with external devices may provide even further ease of use to users, by enabling users to customize how and from where they control their external devices. Thus, the system100provides a technical solution that has significant improvements over existing methods for controlling external devices that is faster, more convenient, and more intuitive for users.

Computing Device

FIG. 2illustrates an embodiment of the computing device102in the system100. It will be apparent to persons having skill in the relevant art that the embodiment of the computing device102illustrated inFIG. 2is provided as illustration only and may not be exhaustive to all possible configurations of the computing device102suitable for performing the functions as discussed herein.

The computing device102may include a communications infrastructure202. The communications infrastructure202may be configured to transmit data between modules, engines, databases, memories, and other components of the computing device102for use in performing the functions discussed herein. The communications infrastructure202may be comprised of one or more communication types and utilize various communication methods for communications within a computing device. For example, the communications infrastructure202may be comprised of a bus, contact pin connectors, wires, etc. In some embodiments, the communications infrastructure202may also be configured to communicate between internal components of the computing device102and external components of the computing device102, such as externally connected databases, display devices, input devices, etc.

The computing device102may also include a communications interface204. The communications interface204may include one or more interfaces used to interact with and facilitate communications between the computing device102and one or more external devices via suitable communications mediums206. For instance, the communications interface204may interface with the communications infrastructure202and provide an interface204for connecting the computing device102to one or more communications mediums204for the electronic transmission or receipt of data signals that are encoded or otherwise superimposed with data for use in performing the functions discussed herein. Communications interfaces204may include universal serial bus (USB) ports, Personal Computer Memory Card International Association (PCMCIA) ports, PS/2 ports, serial ports, fiber optic ports, coaxial ports, twisted-pair cable ports, wireless receivers, etc. Communications mediums206may include local area networks, wireless area networks, cellular communication networks, the Internet, radio frequency, Bluetooth, near field communication, etc.

In some instances, the computing device102may include multiple communications interfaces204for electronically transmitting and receiving data signals via one or more communications mediums206, such as a first communications interface204configured to transmit and receive data signals via a local area network and a second communications interface204configured to transmit and receive data signals via the Internet. In some instances, the communications interface204may include a parsing module for parsing received data signals to obtain the data superimposed or otherwise encoded thereon. For example, the communications interface204may include (e.g., or otherwise have access to, such as via the communications infrastructure204) a parser program configured to receive and transform the received data signal into usable input for the functions performed by the processing device to carry out the methods and systems described herein.

The communications interface204may be configured to establish communication channels108with external devices, such as the light switch106and light110illustrated inFIG. 1, via communication channels108established therewith. In some cases, the communications interface204may provide for the transmission of user instructions as controls to the external device from the primary application program. In other cases, the communications interface204may provide for the transmission of user instructions from the secondary application program, where the communications infrastructure202of the computing device102may facilitate communications between the primary and secondary application programs.

The computing device102may also include memory208. The memory208may be configured to store data for use by the computing device102in order to perform the functions discussed herein. The memory208may be comprised of one or more types of memory using one or more suitable types of memory storage, such as random access memory, read-only memory, hard disk drives, solid state drives, magnetic tape storage, etc. In some cases, the memory208may be non-transitory computer readable media, which may be configured to store data suitable for execution by the computing device102in performing the functions discussed herein. The memory208may store data in any suitable type of configuration, such as in one or more lists, databases, tables, etc., which may store the data in a suitable data format and schema. In some instances, the memory208may include one or more relational databases, which may utilize structured query language for the storage, identification, modifying, updating, accessing, etc. of structured data sets stored therein.

The memory208of the computing device102may include a device database210. The device database210may be configured to store a plurality of device profiles212using a suitable storage format and schema. For example, the device database210may be a library of device profiles212or other types of data storage, such as may utilize a repository for the storage of device data. Each device profile212may be a structured data set configured to store data related to an external device that may be controlled via the methods discussed herein. Each device profile212may be configured to store, for instance, one or more captured images that have been associated with the related device during the registration process, communication data for establishing a communication channel108with the external device, data associated with communicating with the device's secondary application program, control settings or options, interface data for displaying an interface related to the control thereof, etc. The memory208may also be configured to store program data for application programs to be executed by a processor220of the computing device102, such as program code for the first application program and any secondary application programs. In some embodiments, the memory208may also be configured to store spatial data for use in providing virtual or simulated representations of a physical environment104. The memory208may also store algorithms for use in performing image recognition for detecting captured images in a live rendering of a physical environment104as discussed herein.

The computing device102may also include the processor220. The processor220may be configured to perform the functions of the computing device102discussed herein as will be apparent to persons having skill in the relevant art. In some embodiments, the processor220may include and/or be comprised of a plurality of engines and/or modules specially configured to perform one or more functions of the processing device, such as a querying module222, detection module224, etc. As used herein, the term “module” may be software or hardware particularly programmed to receive an input, perform one or more processes using the input, and provides an output. The input, output, and processes performed by various modules will be apparent to one skilled in the art based upon the present disclosure. The processor220as discussed herein may be a single processor, a plurality of processors, or combinations thereof, which may also include processors that may have one or more processor “cores.” Operations performed by the processor220or modules included therein may be performed as a sequential process and/or be performed in parallel, concurrently, and/or in a distributed environment. In some embodiments the order of operations may be rearranged without departing from the spirit of the disclosed subject matter. The processor220and the modules or engines included therein may be configured to execute program code or programmable logic to perform the functions discussed herein, such as may be stored in the memory208and/or a secondary memory230, discussed in more detail below.

The processor220of the computing device102may include a querying module222. The querying module222may be configured to execute queries on databases to identify information. The querying module222may receive one or more data values or query strings, and may execute a query string based thereon on an indicated database, such as the device database210, to identify information stored therein. The querying module222may then output the identified information to an appropriate engine or module of the computing device102as necessary. The querying module222may, for example, execute a query on the device database210to identify a device profile212that includes a captured image that matches an area or object in a live rendering being captured of a physical environment104, representative of the detection of that captured image in the physical environment104.

The processor220of the computing device102may also include a detection module224. The detection module224may be configured to detect captured images in live renderings of a physical environment104, or, if applicable, in a simulated or virtual representation of a previously captured physical environment104. The detection module224may be configured to utilize image recognition techniques and algorithms to detect an area or object in a live rendering of a physical environment104that has been previously captured. The detection module224may also be configured to detect external devices that may be eligible for control via the methods discussed herein, such as by monitoring for external devices located on an area network to which the computing device102is connected or by monitoring for transmissions from external devices that may be received via the communications interface204. In some embodiments, the detection module224may also be configured to detect an external device in a live rendering of a physical environment104based on data associated with the external device, including image data, signal strength and/or direction, image recognition, etc.

In some embodiments, the computing device102may also include a secondary memory230. The secondary memory230may be another memory in addition to the memory208that may be used to store additional data for use in performing the functions of the computing device102as discussed herein. In some embodiments, the secondary memory230may be a different format or may use a different data storage method and/or schema than the memory208. The secondary memory230may be any suitable type of memory, and, in some instances, may include multiple types of memory. For instance, the secondary memory230may be comprised of a hard disk drive232and one or more interfaces234, where the interfaces234are configured to transmit data to and receive data from one or more removable storage units236. Removable storage units236may include, for example, floppy disks, compact discs, digital video discs, Blu-ray discs, removable hard drives, flash drives, universal serial bus drives, etc.

In some cases, the computing device102may also include a display interface238. The display interface may be configured to interface the computing device102with one or more display devices240. The display devices240may be devices configured to display data received from the computing device102. Display devices240may be any suitable type of display, including, for example, liquid crystal displays, light emitting diode displays, thin film transistor display, capacitive touch displays, etc. In some instances, the computing device102may include one or more display interfaces238, which may interface with one or more display devices240. Display devices240may, for example, display live renderings of a physical environment104being captured by the computing device102, which may also include an augmented reality interface displayed thereon that may assist the user in capturing images for association with external devices, displaying indicators of detected, registered images, displaying lists of external devices for selection, displaying controls for an external device, displaying an interface for controlling an external device, etc. In some embodiments, the display device240may be configured to display a virtual or spatial representation of a physical environment104for use in performing the functions discussed herein.

The computing device102may also include an input/output interface242. The input/output interface242may be configured to interface the computing device102with one or more input devices244and/or output devices246for the transmission to and receipt of data from the respective devices. The input/output interface242may include any suitable type of interface, and in some instances may include multiple types of interfaces, such as for interfacing with multiple types of input devices244and/or output devices246. Input devices244may include any suitable type of device for inputting data to an computing device102, such as a keyboard, mouse, microphone, camera, light field camera, touch screen, click wheel, scroll wheel, remote control, accelerometer, compass, gyroscope, sensor, etc. Input devices244may be configured to receive input from a user of the computing device102, such as for the selection of an area or object in a live rendering of a physical environment104for use as a captured image associated with an external device. Input devices224may also be used by the user to select an external device for association with a captured image or for the adding of an external device into the device database210for future control thereof. The input devices224may also be configured to receive user instructions for the control of an external device, such as may be transmitted thereto via a communication channel108for the control thereof. Output devices246may include any suitable type of device for outputting data from an computing device102, such as a speaker, remote control, headset, tactile feedback device, etc.

Process for Registration of Images for Control of External Devices

FIG. 3illustrates an example process in the system100for the registration of one or more captured images in a live rendering of a physical environment104to be associated with an external device for the control thereof using an augmented reality interface.

In step302, an external device106may broadcast a communication signal that may be detected by other devices, such as the computing device102. In some cases, the broadcast may be a signal emitted by the external device106through one or more communication networks. In other cases, the broadcast may be the connection of the external device106to a communication network where it may be discoverable by other devices. For instance, the broadcast of the external device106may be represented by its connection to an area network that includes the computing device102. In step304, a receiver of the communications interface204of the computing device102may receive the broadcasted signal of the external device106and thereby detect (e.g., via the detection module224) that the external device106exists on the area network and is eligible for control.

In step306, an input device244of the computing device102may capture a live rendering of a physical environment104. In step308, a display device240of the computing device102may display the live rendering of the physical environment104to a user of the computing device102. In an exemplary embodiment, the live rendering of the physical environment104may update in real-time as the computing device102is moved or adjusted, such as based on movement of an optical imaging device (e.g., a camera) thereof. In embodiments where a virtual representation of the physical environment104is used, the virtual representation may be moved based on movement of the computing device102, such as to simulate movement of the computing device102in the real physical environment104.

In step310, an input device244of the computing device102may receive an instruction from a user thereof selecting a portion of the live rendering. The selected portion of the live rendering may be captured by the computing device102to serve as a captured image that is to be associated with an external device, such as the external device106. In some cases, the live rendering may be an image of the external device106or a component of the external device106, or, in other cases, may have no direct relation to the external device106. In step312, the display device240of the computing device102may display a list of all external devices106detected by the computing device102that the user may select to associate with the captured image. In step314, the input device244may receive input of the user's selection, selecting one of the external devices106for registration as being associated with the captured image. In step316, the querying module222of the computing device102may execute a query on the device database210in the memory208of the computing device102to insert a new device profile212therein related to the selected external device106that includes the captured image and any data associated with the external device106for communication therewith and control thereof. In instances where a device profile212already exists for the external device106(e.g., an image has already been registered as associated with the external device106), then the newly captured image may be added to the existing device profile212.

Process for Remote Control of an External Device Via Augmented Reality

FIG. 4illustrates an example process in the system100ofFIG. 1for the remote control of an external device, such as the light switch106or light110via the use of augmented reality in the computing device102.

In step402, an input device244of the computing device102may capture a live rendering of a physical environment104in which the computing device102is located, such as by using a camera mounted on or as part of the computing device102. In step404, the display device240of the computing device102may display the live rendering to a user of the computing device102. In an exemplary embodiment, the live rendering may be updated in real-time as the computing device102is moved or orientation of the computing device102is changed. In step406, the detection module224of the computing device102may use image recognition tools and algorithms to detect a registered image in the live rendering of the physical environment104. In some cases, the detection module224will continually attempt to detect a registered image in the live rendering as the live rendering is updated in real-time, until such a time when an attempted detection is successful.

Once a registered image has been detected, then, in step408, the querying module222of the computing device102may execute a query on the device database210of the memory208of the computing device102to identify a device profile212stored therein that includes the registered image that was detected by the detection module224. Once the device profile212is identified, then, in step410, the computing device102may identify (e.g., via communication with the external device106or second application program associated therewith) the current state of the external device106. The state of the external device106may include the current control options, settings, or other data associated with the external device106. In step412, the display device240of the computing device102may display options for controlling the external device106related to the identified device profile212, where the options may be based on data stored in the device profile212. For instance, the device profile212may directly store the device control options, or may store data for use by the primary application program of the computing device102to communicate with a secondary application program associated with the external device106to obtain device control options therefrom. In some embodiments, the device control options may be displayed concurrent with the live rendering of the physical environment104. In other embodiments, the device control options may be overlaid on the live rendering, such as directly on top of the detected registered image. In some cases, the overlay may be a virtual representation of the external device106or an interface thereof.

In step414, the input device244of the computing device102may receive input from the user selecting one or more device control options for the external device106. Input may be received from the user via any suitable method, such as through voice, gestures directly on or viewable by the computing device102, preset movements of the computing device102, direct entry (e.g., typing) in the computing device102, or a combination thereof. In step416, the communications interface204of the computing device102may electronically transmit the selected device control options to the external device106via a communication channel108associated therewith established between the computing device102and the external device106, either directly or indirectly (e.g., via the Internet or one or more other suitable communication networks). In some embodiments, the computing device102may be configured to provide recommendations for device control options to the user. In such an embodiment, the recommendations may be based on past selections by the user, user preferences, artificial intelligence, selected options by other users of the same device or type of device, etc. In step418, the external device106may receive the selected device control options then, in step420, adjust operation of the device based on the user selections. As a result, the external device106may be remotely controlled by the user as initiated via the viewing of a registered image via the computing device102using the augmented reality interface of the primary application program thereof.

Exemplary Method for Remotely Controlling an External Device

FIG. 5illustrates a method500for the remote control of an external internet of things device by a computing device through the detection of an image associated therewith in a live rendering of a physical environment.

In step502, a plurality of device profiles (e.g., device profiles212) may be stored (e.g., in a device database210) in a computing device (e.g., the computing device102), wherein each device profile includes connection data, one or more control commands, and at least one associated image. In step504, a live rendering of a physical environment (e.g., the physical environment104) may be received by an imaging device (e.g., an input device244) interfaced with the computing device. In step506, a registered image may be detected (e.g., by the detection module224) by the computing device in the live rendering of the physical environment.

In step508, a specific device profile of the plurality of device profiles may be identified (e.g., via a query executed by a querying module222) in the computing device where the at least one associated image includes the detected registered image. In step510, a communication channel (e.g., the communication channel108) may be established by the computing device with an external device (e.g., an external device106) using the connection data included in the identified specific device profile. In step512, at least one of the one or more control commands included in the identified specific device profile may be electronically transmitted (e.g., via the communications interface204) by the computing device to the external device using the established communication channel.

In one embodiment, the received live rendering of the physical environment may be displayed by a display device (e.g., the display device240) interfaced with the computing device. In a further embodiment, the detected registered image may be indicated on the displayed live rendering of the physical environment. In an even further embodiment, the one or more control commands included in the identified specific device profile may be displayed, by the display device, as superimposed on the indicated detected registered image.

In some embodiments, the method500may further include receiving, by an input device (e.g., an input device244) interfaced with the computing device, a user instruction associated with the at least one of the one or more control commands, wherein the user instruction is received prior to electronic transmission of the at least one of the one or more control commands to the external device. In one embodiment, the external device may be an internet-enabled computing device. In some embodiments, the live rendering of the physical environment may be a virtual representation of the physical environment based on stored three-dimensional spatial data.