Device control method and system thereof

The present disclosure provides a device control method, which comprises capturing, by a portable device, a first image of a specific area containing a number of electronic devices connected to a communication network; transmitting a connection signal to a selected electronic device on a device list containing all the electronic devices connected to a communication network; determining whether the selected electronic device is in the first image captured based on whether a response has been received from the selected electronic device; identifying the selected electronic device on the device list and the relative position of the selected electronic device on the first image captured upon determined that the selected electronic device is in the first image; and continuing the device identification process until all the connected electronic devices on the device list has been verified, to identify the relative positions of all electronic devices contained in the first image.

TECHNICAL FIELD

The instant disclosure relates generally to a device control method and a system thereof pertains particularly to a device control method and a system thereof utilizing image recognition.

BACKGROUND

As communication and Internet of Things (IoT) technologies advanced, it has been desirable to control various electronic devices in a local area setting using a single mobile device, such as a smart phone or tablet, via network communication. Current device control mechanisms have some drawbacks. For instance, a user of a mobile device might not have a visualization for the devices linked, but rather a device list. Thus, a user can't easily identify the device to be controlled. Moreover, in order to properly setup communication channel between the mobile device and the device to be controlled, a user has to manually perform lengthy operations on the mobile device and maybe even on the device to be controlled, which may include performing device searching procedure, manually selecting the devices to be controlled and setting communication parameters and the like, while the user's operation on the device to be communicated with may include waiting for the selected device to be ready, setting communication channel parameters and the like. In additional, different electronic devices might use different control setup procedures, making the device management process even more difficult. If the user makes any errors during the control setup operation, the user might have to start over. These causes inconvenience to the user.

DETAILED DESCRIPTION

For consistency of purpose and ease of understanding, like features are identified (although, in some instances, not shown) with like numerals in the exemplary figures. However, the features in different embodiments may differ in other respects, and thus shall not be narrowly confined to what is shown in the figures.

Embodiments of the present disclosure provide various methods, apparatuses, and systems for providing user-behavior-oriented automated control system using cloud computing, digital communication, and Internet of Things (IoT) technologies. The following descriptions describes various embodiments related to a user behavior-oriented automated control methods, apparatuses, systems with a surface mounted multifunctional intelligent modular system merely for illustration purposes and therefore does not limit the present disclosure.

Exemplary embodiments of the present disclosure that are described in the context of a functional computer processing systems for device control. The present disclosures may also be embodied in a computer readable product disposed on data bearing media for use with any suitable computational and data processing device with communication capabilities. Such data bearing media may be transmission media or recordable media for machine-readable information, including magnetic media, optical media, or other suitable media. Examples of recordable media include magnetic disks in hard drives or diskettes, compact disks for optical drives, magnetic tape, and others as known to those of skill in the art.

Persons skilled in the art shall immediately recognize that any computer system having suitable programming means will be capable of executing the steps of the method of the present disclosure as embodied in a computer readable product. Persons skilled in the art will recognize immediately that, although some of the exemplary embodiments described in this specification are oriented to software installed and executing on computer hardware, nevertheless, alternative exemplary embodiments implemented as firmware or as hardware or as a combination of hardware and software are well within the scope of the present disclosure.

Device Control System Exemplary Embodiment

The present disclosure provides a device control system and a device control method for constructing a physical-to-virtual spatial relationship for electronic devices to be linked and controlled. A photographic representation for the electronic devices is provided on a user interface such that a user can easily locate the electronic devices to be controlled and conduct various operation without having to go through complex device linking and control procedures.

FIG. 1Ais a schematic illustration of an exemplary device control system provided in accordance with exemplary embodiments of the present disclosure. Particularly,FIG. 1Ashows an exemplary device control system1(herein after the control system1), which comprises a surface mounted multifunctional intelligent modular system (herein after the modular system3) mounted on a mounting surface (e.g., a wall) in a manner that all elements in the modular system3is observable and operable by a user2, a portable device10, and a router device20.

The modular system3comprises a plurality of electronic devices having communication capabilities (e.g., functional module units30a-g) mounted to the supporting surface (e.g., a wall) through one or more discrete mounting member (not shown). The functional module units30a-ginclude but are not limited to a TIME display/control device30b(herein after the TIME30b), one or more sets of speakers30a,30d, a fragrance apparatus30e, an air purifier30g, and an alarm30f.

In the instant embodiment, the functional module units30a-gof the modular system3and the portable device12are connected to a communication network through the router device20. The portable device10is linked to each of the functional module units30a-gvia the communication network. The functional module units30a-gand the portable device10are operable to communicate and perform data/signal transmission over the communication network.

In some exemplary embodiment, the speakers30a,30d, the fragrance apparatus30e, the air purifier30g, and the alarm30fofFIG. 1Aare physically connected to the TIME30bvia cable. The TIME30bis configured to control the operations of the speakers30a,30d, a fragrance apparatus30e, the air purifier30g, and the alarm30fbased on user-input or on a default setting (e.g., standby or sleep mode, scheduled operations, or the like).

In some exemplary embodiment, the TIME30b, the speakers30a,30d, the fragrance generator30e, the air purifier30g, and the alarm30fofFIG. 1Aare connected to a remote cloud server for retrieving user-preference control parameters and configure accordingly. Each of the TIME30b, the speakers30a,30d, the fragrance generator30e, the air purifier30g, and the alarm30fmay further feed the operation status information back to the remote cloud server.

In some exemplary embodiment, each of the exemplary functional module units is configured to have a flat and substantially rectangular structural profile that resembles a wall tile. The flat and geometrically regular structural configuration of the functional module units allows them to be arranged in close proximity to each other in an organized manner, so as to effectively cover a wide span of the wall. The flat construction of the functional module units also facilitates efficient space usage upon installation to the wall as the thin profile of the module units are made to reduce necessary space requirement.

In some exemplary embodiment, each of the exemplary functional module units defines a substantially rectangular-shaped viewing face arranged to face toward away the mounting surface (e.g., wall) and operatively accessible to a potential user (e.g., user2) situated in a living/interactive environment. The planar profile (e.g., the viewing face) of the functional module units may be substantially identical.

Specifically, the router device20is operable to route data among the electronics devices (e.g., functional module units30a-g) and the portable device10through the communication network. In one exemplary embodiment, the router device20can comprises of necessary hardware and firmware component for operatively monitoring and routing data over a local area network.

The portable device10is equipped with image capturing function and communication capability. The portable device10is operable by the user2. The portable device10is operable to construct the physical to virtual spatial relationship of the electronic devices (e.g., functional module units30a-g), i.e., to map the physical position of electronic devices to a virtual user interface, through the uses of image capturing and image recognition for the user2to control and manage using the portable device10.

More specifically, the portable device10has at least one image sensor12and the image sensor12has a sensing range12covering a specific area, such as a surface of a wall, an area of ceiling, or a room space. In the instant embodiment, the sensing range121encompasses at least the wall surface having the modular system3mounted thereon.

Moreover, the portable device10has a device list stored in a built-in memory, wherein the device list contains all the electronic devices in the communication network and linked to or linkable to the portable device10. In some exemplary embodiment, the device list may be implemented in a tabular form listing each of the electronic devices in the communication network and the associated identification number. The built-in memory may be implemented by a static random access memory (SRAM), a dynamic random access memory (e.g., DRAM, SDRAM, DDR, DDRII), a Flash memory (e.g., NAND Flash, NOR Flash), Read only memory (e.g., ROM, EPROM, EEPROM).

The portable device10further has an application program for device control (e.g., a smart device control application program) installed thereon for a built-in processor of the portable device10to execute.

The user2can initiate the application program on the portable device10via user operation in such a manner that the image sensor12to capture the specific area or a target area while the portable device10is held at hand. The image sensor12in the instant embodiment operates in an image capturing mode and captures the specific area at a constant frame rate (e.g., 5˜10 frames per second). In another exemplary embodiment, the image sensor12may be configured to operate in a video capturing mode and can capture the specific area at a constant frame rate (e.g., 24˜30 frames per second).

The constant frame rate of the image sensor12may be pre-configured before factory shipment of the portable device10or may be configured by the user2via user operations.

The communication network topology adopted in the exemplary device control system ofFIG. 1Ais a star network topology, referring toFIG. 1B, i.e., all the electronic devices (e.g., functional modular units30a-gor functional modular units32a-c) denoted as “D” and the portable device10denoted as “M” are connected to a special gateway node denoted as “R” (i.e., the router). However, it should be noted that the usage of the network is to provide a data communication path for the electronic devices (e.g., functional modular units30a-gor functional modular units32a-c) and the portable device10for data communication, sharing of data as well as to perform control operation with each other, and the exact network topology adopted may vary depending upon the actual structure of the system and type of the functional modular units. The present disclosure is not limited thereto.

Referring again toFIG. 1B, in other embodiments, the communication network topology used by the system for linking the electronic devices (e.g., functional modular units30a-gor functional modular units32a-c) and the portable device10may be configured in any form, for example, a bus network topology, a ring network topology, a mesh network topology, a fully connected network topology, a tree or a hierarchical network topology or the like. InFIG. 1B, the electronic devices are denoted as “D”, while the portable device10is denoted as “M”.

For instance, under the bus network topology structural, all the electronic devices (e.g., functional modular units30a-gor functional modular units32a-c) may be configured to connect to a single bus or to Ethernet cable, with a network routing node (e.g., the router device20) connected thereto. The portable device10may communicate with electronic devices through the router using correspond MAC or IP addresses. For another instance, under the ring network topology structural, the electronic devices and the portable device10are connected in a loop or ring form. For still another instance, under the tree network topology structural, a root node (e.g. a central controller of the system or the TIME30b) is connected to the one or more functional modular units (e.g., the lighting modular units, the speakers30a,30d, the air purifier30g, the alarm clock30f, or the like) that are in lower level or the rank in the hierarchy by cables. The portable device10and the electronic devices connected in tree network topology form may be configured to connect to a single bus via cables or may be connected to a network routing device (e.g., the router device20) via wireless connection.

Based on the above elaboration, those skilled in the art shall be able to establish the communication network based on network system and operational requirements, and thus further descriptions are hereby omitted.

In the embodiment illustrated byFIG. 1, the image sensor12of the portable device10captures at least one image of the wall surface with the modular system3mounted thereon and generates one or more first images containing all the functional modular units30a-g. The processor of the portable device10at same time is driven to execute a device recognition and control procedure while performing image capturing function to creating a physical to virtual spatial relationship for the electronic devices in the first image. i.e., maps the physical position of the device to the virtual position of the device displayed on the display of the portable device10.

Upon execution of the device recognition and control procedure, the portable device10transmits a connection signal through the communication network to a selected electronic device on the device list. The portable device10may transmit the connection signal in the order as listed in the device list or to a user-specific electronic device. The connection signal comprises at least an acknowledgement request, wherein the acknowledgement request asked the electronic device to provide a response. In one exemplary embodiment, the connection signal comprises an acknowledgement request and an identification association with the selected electronic device.

The portable device10further determines whether the selected electronic device appears in the first image captured by determining whether the response which is responsive to the connection signal has being received from the selected electronic device within a preset period (e.g., one or two seconds). In one embodiment, the portable device10determines whether the selected electronic device has provided an optical feedback e.g., a visible light pattern response or invisible light pattern response within the preset period using the image sensor12.

As illustrated inFIG. 2, the user (not shown inFIG. 2) hold the portable device10and enables the image sensor12to continue capturing images containing the electronic devices (e.g., the functional modular units32a-c) in the specific area at a constant frame rate, the first image (e.g., the image with the functional modular units32a-c) will be displayed on a touch display14of the portable device10with the functional modular units32a-cdisplayed as object A32a′, object B32b′, and object C32c′ respectively.

Upon receiving the connection signal, each of the electronic devices (e.g., the functional modular units32a-c) operatively provides the optical feedback response, e.g., a visible light pattern response (e.g., a geometric shape lighting signal, a colored pattern signal, a flashing signal, or a unique shape of lighting) or invisible light pattern response (e.g., an infrared signal) for the image sensor12of the portable device10to capture.

For instance, the functional modular unit32amay be configured to transmit an invisible light response (e.g., infrared light signal201) for the image sensor12to capture and recognize upon receiving the connection signal. Alternatively, the functional modular unit32bmay be configured to transmit a visible light pattern response (e.g., square-frame shaped lighting203) for the portable device10to capture and recognize or the functional modular unit32cmay be configured to transmit a visible light pattern response (e.g., flashing triangular-shaped lighting205) for the portable device10to capture and recognize.

In some exemplary embodiment, the electronic devices may each have a unique 1D or QR barcode representation and lighting the barcode representation or the portable device10to capture and recognize in response to receiving the connection signal.

It is worth mentioning that techniques such as feature recognition and objects' position identification in image captured are known image processing technique, thus further descriptions are hereby omitted.

When the portable device10does not detect an optical feedback from the selected electronic device, the portable device10determines that the selected electronic device is not in the first image and continue transmits the connection signal to the next electronic device on the device list until all the presence of electric devices of the device list has in the first image has been verified.

Conversely, when the portable device10detects the optical feedback from the selected electronic device within the preset period, the portable device10determines that the selected electronic device (e.g., the functional modular units32a-c) is in the first image. The portable device10operatively identifies the relative position of the electronic device with respect to another electric device (e.g., neighboring electric device) within the first image and stores the relative position information in the built-in memory.

For instance, the portable device10can map the functional modular units32ato the left most object (e.g., object A32a′), the functional modular units32bto the center object (e.g., object B32b′), and the functional modular units32cto the right most object (e.g., object C32c′), thereby creating physical to virtual spatial relationship amount the electronic devices.

In short, the portable device10is capable to provide a visual representation of the electronic devices shown in the first image for the user to view and easily locate the target electronic device to operate, improving user experience.

FIGS. 3A and 3Billustrate a device selection and control operation in accordance with exemplary embodiments of the instant disclosure. After the execution of the device recognition and control procedure, the application program creates a virtual layout of the electronic devices in the first image on the touch display14of the portable device10as illustrated inFIG. 3A. The user of the portable device10thereafter can select an electronic device (e.g., the functional modular unit32c) to control by selecting or clicking the object C32c′ (e.g., graphic icon) via the touch display14of the portable device10to initiate a user control interface for the electronic device (e.g., the functional modular unit32c).

An exemplary user control interface141for the selected electronic device (e.g., the functional modular unit32c) may be illustrated byFIG. 3B. Take the selected electronic device (e.g., the functional modular unit32c) to be a speaker as an illustrative example, the user control interface141may include on/off power control, up/down volume control.

In some exemplary embodiment, the device list may further include the control information (or control protocols) associated with each of the electronic devices (e.g., the functional modular units), the processor of the portable device10is operable to interpret the control information retrieved from the device list, execute a control routines, and remotely control the operation of the respective electronic device. The portable device10may obtain the corresponding control information from the respective electronic device through the communication network during connection establishing process or from a cloud server.

In the instant embodiment, the image sensor12can be implemented by an infrared image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, or a charge-coupled device (CCD) image sensor depend upon the type of portable device12and operational requirements.

In some exemplary embodiment, the portable device10may be a smart phone, a tablet, a PDA, a digital camera, a camcorder, or any other wearable device with image capturing function and the application program for device control installed thereon.

In some exemplary embodiment, the application program for device control may be pre-stored in the device before factory shipment. In some exemplary embodiment, the application program for device control may be retrieved and downloaded by the user from an external cloud server through the internet.

It is worth to note thatFIG. 1A,FIG. 2,FIG. 3A, andFIG. 3Bare merely used for illustrating the device control system and the device control operation thereof, and thus should not be used to limit the scope of the present disclosure.

Device Recognition and Control Method Exemplary Embodiment

From the aforementioned exemplary embodiment, the present disclosure may generalize a device recognition and control method which can be applied to the device control system illustrated in the aforementioned embodiment.FIG. 4shows a flowchart diagram illustrating a device control method provided in accordance to an exemplary embodiment of the present disclosure. The device control method can be executed by a processor of the aforementioned portable device having a built-in image sensor. The portable device also has an application program for device control installed thereon, which may be downloaded from an external server.

In block401, the processor of the portable device initiates an application program for device control (e.g., a smart device control application program) by a corresponding user operation.

In block403, the processor of the portable device searches over a communication network connected thereto through the router device for all of connected electronic devices. Specifically, the portable device first connects to a communication network (e.g., through a router device) to establish connections to each of electronic devices. Next, the processor of the portable device searches over a communication network to locate all the electronic devices that are connected to the communication network.

In some embodiments, the portable device communicates with the router and retrieves the information on the electronic devices connected to the communication network.

In block405, the processor of the portable device creates a device list of all connected electronic devices in the communication network and stores the device list in a built-in memory. The built-in memory in one exemplary embodiment may be allocated by the internal memory of the processor.

In block407, the portable device was held by the user in such a manner that the image sensor captures images of a specific area (e.g., a wall surface or a room space) with the built-in image sensor. The processor of the portable device operatively causes the image sensor to capture at least one first image of the electronic devices at a constant frame rate.

The specific area contains a number of the electronic devices, wherein the number of the electronic devices in the specific area (i.e., appeared in the first image) is less than or equal to the total number of electronic devices on the device list. The size of the specific area depends on the sensing range of the built-in image sensor. The constant frame rate of the built-in image sensor may be pre-configured before factory shipment of the portable device or may be configured by the user via user operation.

In block409, the processor of the portable device causes the portable device to transmit a connection signal through the communication network to the selected one electronic device on the device list, wherein the connection signal comprises an acknowledgement request. In some exemplary embodiment, the connection signal comprises an acknowledgement request and an identification of the selected electronic device.

In block411, the processor of the portable device determines whether the selected electronic device is in the first images captured by determining whether a response has being received from the selected electronic device within a preset period (e.g., one or two seconds). More specifically, the processor of the portable device determines whether an optical feedback response (e.g., a visible light pattern response or invisible light pattern response) from the selected electronic device has been detected within the preset period using the image sensor based on image detection results (i.e., based on at least one another first image captured.

Upon receiving the connection signal from the portable device, each of the electronic devices (e.g., the functional modular units30a-gofFIG. 1A) operatively provides the optical feedback response, e.g., a visible light pattern response (e.g., a geometric shape lighting signal, a colored pattern signal, a flashing signal, or a unique shape lighting signal) or invisible light pattern response (e.g., infrared signal) for the image sensor of the portable device to capture.

In some exemplary embodiment, the electronic devices may each have a unique 1D or QR barcode representation and lighting the barcode representation or the portable device to capture and recognize in response to receiving the connection signal.

When the processor of the portable device does not detect the optical feedback response responsive to the connection signal from the selected electronic device within the preset period based on the image detection result, the processor of the portable device determines that the selected electronic device is not in the first image and executes block415. Conversely, when the processor of the portable device detects the optical feedback responsive to the connection signal from the selected electronic device within the preset period based on the image detection result, the processor of the portable device determines that the selected electronic device (e.g., the functional modular units32a-c) is in the first image and executes block413.

In block413, the processor of the portable device operatively identifies the relative position of the electronic device with respect to another electric device in the first image and stores the relative position information in the built-in memory and executes block419.

In block415, the processor of the portable device determines whether all of connected electronic devices on the device list has been verified. When the processor of the portable device determines that not all of connected electronic devices on the device list has been verified, the processor of the portable device executes block417. On the other hand, when the processor of the portable device determines that all of connected electronic devices on the device list has been verified, the processor of the portable device executes block419.

In block417, the processor of the portable device causes the portable device transmit the connection signal to the next electronic device on the device list and returns to block411, i.e., the processor of the portable device continue the connection signal transmission process until all the presences of electric devices of the device list has in the first image are verified.

In block419, the processor of the portable device causes a display of the portable device to display a user interface containing relative position information for the electronic device in the first image for the user to view and control the operation of the electronic devices in the first image via user operation (e.g., a touch input). The processor of the portable device transmit the control signal to the respective electronic device for configuration the operation of the selected electronic device based on a user-input operation (e.g., a touch input)

In some exemplary embodiment, the processor of the portable device10is operable to retrieve the control information associated with the selected electronic device from the device list and execute a control routine to remotely control the operation of the selected electronic device.

In some exemplary embodiment, after the physical-virtual position relationship of the electronic devices in the first image has been identified, the electronic devices may be grouped in a desired manner.FIG. 5shows a flowchart diagram illustrating device grouping method provided in accordance with exemplary embodiments of the instant disclosure.FIG. 6shows a schematic illustration of device grouping operation provided in accordance with exemplary embodiments of the instant disclosure.

In block510, upon receiving a user selection operation of at least two electronic devices appeared in the first image captured while still capturing the image of the specific area, the processor of the portable device causes the image sensor of the portable device to capture at least one second image containing the at least two electronic devices from the electronic devices (e.g., the electronic devices60a-e) appeared in the first image, e.g., a first group image containing60c′,60d′,60e′ and a second group image containing60a′,60b′. The number of electronic devices appeared in the second image is less than or equal to the number of the electronic devices appeared in the first image.

In block520, the processor of the electronic device defines the electronic devices in the second image as functional groups e.g., electronic devices60c-ecorresponding60c′,60d′,60e′ as the first functional group and electronic devices60a-bcorresponding60a′,60b′ as the second functional group. The processor of the electronic device may define the functional group by linked the operations of the two electronic devices together for the user to control or create a group control user interface on the display of the portable device for the user to view and operate therefrom.

In block530, the processor of the electronic device configure the operations of the electronic devices based on a user input operation e.g., a touch input.

For instance, the user may configure one of the electronic devices in the functional group as a master device and the remaining electronic device as slave devices. The electronic device that is configured to be the master device is operable to control the operations of the slave devices based on user input.

For another instance, by grouping the selected electronic devices into a target group (e.g., functional group), the user may control the operations of the electronic devices at the same time, such as power on/off at the same time or operate at same conditions (e.g., scheduling operation time)

The present disclosure further provides an implementation for the portable device. The portable device comprises a memory, a processor, an image sensor, a display, and a communication unit. The memory, the image sensor, the display, and the communication unit are coupled to the processor.

More specifically, the image sensor operatively captures an image of a specific area, wherein the image containing a number of connected electronic devices in the specific area. The number of connected electronic devices contained in the specific area (i.e., appeared in the first image) is less than or equal to the total number of electronic devices on a device list.

The communication unit operatively connects to the communication network by connecting to a router (e.g., the router device20ofFIG. 1A) in a wired or wireless manner. The memory stores a device list containing all connected electronic devices in the communication network, the images captured by the image sensor, operational data for the processor to execute the device recognition and control procedure.

The display operates display a visual representation of electronic devices connected to the communication network (of which the portable device is connected to) and appeared in the image captured for the user to view.

The processor, which is the operational core for the portable device, is configured to operatively executes the device recognition and control procedure for identifying the presence and the associated position of the electronic devices in the specific area and creating a virtual device control information on the display, which maps the physical to virtual spatial relationship of the electronic devices in the image captured, for the user of the portable device to operate and control.

The processor may be implemented by a processing chip such as a digital signal processor or embedded microcontroller, programmed with necessary firmware for controlling the operations of the memory, the image sensor, the display and the communication unit and executing the device recognition and control procedure. The memory may be a memory section of the processing chip, a volatile memory chip, or a nonvolatile memory chip such as a flash memory chip, a read-only memory chip, or a random access memory chip. The communication unit may a communication processing chip capable of connecting to a local communication network through the router and a remote cloud server through the internet.

In some exemplary embodiments, the portable device is operable to connect the electronic device that is captured in the first image but is not yet connected to the communication network that the portable device is currently connected to. More specifically, when the processor of the portable device determines that there is an electronic device in the first image that is not on the device list, the portable device operatively perform a device search via near field scanning or Bluetooth scanning techniques and locate the respective electronic device for paring. After paired, the processor of the portable device transmits the communication network information (such as the ip address of the router device) to the respective electronic device for the respective electronic device to connect to the communication network.

Accordingly, the present disclosure provides a device control mechanism, which not only is able to create a visual representation that maps the physical positions of the electronic devices to be controlled detected to the virtual layout displayed on the device for the user to easily view and control electronic device operations, the device control mechanism also incorporates a device grouping mechanism by image capturing for the user to flexibly and quickly control the operation of a group electronic devices. Thereby, reducing the device control time and increase user experience.