Patent Description:
Demand for automation of residential and commercial environments has resulted in a dramatically expanding market of "smart" doorbells, thermostats, lighting, security systems, audio, video, and numerous other devices. These devices are offered by a substantial number of manufacturers who, for competitive or performance reasons, have adopted different control protocols that are not generally compatible with each other. Thus, a "smart" thermostat offered by one manufacturer does not operate in accordance with a widely adopted standard, and often will not interoperate with "smart" lighting offered by a different manufacturer. Among other disadvantages, this lack of interoperability limits the choices of an end user in selecting devices for a particular automation environment. Another disadvantage is that an end user is prevented from effectively controlling all of the devices in an environment with a single control device (e.g., remote control, smartphone, or tablet).

As a result, an end user is often forced to either select one manufacturer, even though that manufacturer does not offer all of the "smart" devices of interest to the end user, or select multiple manufacturers at increased cost and lack of interoperability.

<CIT> discloses systems and methods for coordinating home automation activity in communication with a storage database in order to finally generate a graphically depiction identifying the number of events occurring during discrete time intervals for a period of time. <CIT> discloses techniques for utilizing a WPAN (e.g., BLE) and an in-home LAN (e.g., Wi-Fi network) together to provision and/or configure hardware elements of a home automation system.

In accordance with one aspect of the invention, an automation control apparatus is provided, according to independent claim <NUM>.

In accordance with another aspect of the invention, a method of controlling devices in an automation environment is provided, according to independent claim <NUM>.

The invention description below refers to the accompanying drawings, of which:.

<FIG> shows an automation environment <NUM> which includes a multi-role automation host <NUM>. A first network cloud <NUM> is accessible by multi-role automation host <NUM>. Multi-role automation host <NUM> includes a system configuration (not shown) which, in turn, includes a component or device profile (not shown) associated with each profiled device 108a. 108b controlled by multi-role automation host <NUM>. In general, a device profile is a file or other data structure which describes the inputs, outputs, and available set of commands for the associated device. Through either internal storage, network cloud <NUM>, or another external source, multi-role automation host <NUM> must have access to an associated device profile in order to control a given device.

Examples of a system configuration, component profile, and hardware and software suitable for multi-role automation host <NUM> may be found in <CIT> incorporated by reference above.

A user may interact with, and effectively control, multi-role automation host <NUM> by operating a user device 116a running a host control application <NUM> which is compatible with multi-role automation host <NUM>. User device 116a may be implemented with a smartphone, tablet, or other device capable of running host control application <NUM> and having appropriate network connectivity to communicate with multi-role automation host <NUM>. Alternatively, a user may interact with and effectively control multi-role automation host <NUM> using a user remote 118a which is compatible with multi-role automation host <NUM>. Multi-role automation host <NUM> may, for example, be based upon a Smart Host Model SHC-S2-<NUM>, a surround sound soundbar, a "smart" Multistat thermostat, a "smart" audio speaker (as described below in connection with <FIG>), a "smart" amplifier (as described below in connection with <FIG>), or other devices offered by Savant Systems, LLC of Hyannis, MA.

A second network cloud <NUM> is accessible by an automation hub <NUM>. Unlike multi-role automation host <NUM> which is capable of controlling any profiled device, automation hub <NUM> is conventionally capable of controlling only devices 114a-114c which have been certified as control protocol compliant (i.e., devices which have been certified as compliant with the control protocol specified by the manufacturer of automation hub <NUM>).

A user may interact with and effectively control automation hub <NUM> by operating a user device 116b running a hub control application <NUM> which is compatible with automation hub <NUM>. User device 116b may be implemented with a smartphone, tablet, or other device capable of running hub control application <NUM> and having appropriate network connectivity to communicate with automation hub <NUM>. Alternatively, a user may interact with and effectively control automation hub <NUM> using a user remote 118b which is compatible with automation hub <NUM>. Automation hub <NUM> may, for example, be based on an Apple TV®, HomePod™, or iPad® running HomeKit®, all of which are offered by Apple Inc. of Cupertino, CA.

Multi-role automation host <NUM> includes an automation bridge <NUM>. In one embodiment, an SDK available from Apple Inc. may be used to create automation bridge <NUM> and embed it in multi-role automation host <NUM>. As described in greater detail below, automation bridge <NUM> functions, in part, to communicate (bidirectionally) with automation hub <NUM> using the same manufacturer-specified control protocol as used by certified control protocol compliant devices 114a-114c. Automation bridge <NUM> also functions to convert control protocol messages received from automation hub <NUM> to an appropriate form for processing by multi-role automation host <NUM> (and vice versa). Advantageously, through such communication and protocol translation, and depending upon the type of device, profiled devices 108a. 108b as well as user remote 118a are exposed to automation hub <NUM>. That is, profiled devices 108a. 108b and user remote 118a, even though not certified as compliant with the control protocol used by automation hub <NUM>, become visible to and controllable by automation hub <NUM> by virtue of multi-role automation host <NUM>.

<FIG> shows a block diagram of a powered surround sound soundbar <NUM> which may be used to implement multi-role automation host <NUM> (<FIG>). A microprocessor <NUM> is coupled in bidirectional communication with an audio digital signal processor (DSP) <NUM>. Audio DSP <NUM> is also coupled to a TosLink <NUM>. Audio DSP <NUM> is coupled to digital to analog converters (not shown) whose outputs are coupled to an amplifier <NUM> whose outputs are coupled to various audio speakers <NUM>.

Microprocessor <NUM> is also coupled in bidirectional communication with a WISA module <NUM> as well as a WiFi dual band module <NUM>. In addition, microprocessor <NUM> is coupled to an LAN/AVB port <NUM>, a microphone <NUM>, control ports <NUM> which may include RS232 control portsinfrared (IR) control ports, or other control ports. An AC/DC power supply <NUM> provides power to soundbar <NUM>.

<FIG> shows a message flow diagram illustrating an example of higher level messaging needed to make profiled devices 108a. 108b and user remote 118a (<FIG>) visible to and controllable by automation hub <NUM>. Host control application <NUM>, running on user device 116a, initiates a Discovery protocol <NUM> with multi-role automation host <NUM>. Host control application <NUM> next sends a Host Provision message <NUM>. In response, multi-role automation host <NUM> proceeds to Query <NUM> its configuration to identify any (profiled) device(s) that is of an appropriate type to be supported/controlled by automation hub <NUM>. Assuming at least one profiled device of an appropriate type is identified, multi-role automation host <NUM>, through bridge <NUM>, transmits an Add Bridged Devices to Hub message <NUM> to automation hub <NUM>.

Subsequently, host control application <NUM> issues a Control Device message <NUM> to multi-role automation host <NUM> which responds by exerting control <NUM> over the profiled device to which message <NUM> is directed. In contrast, when hub control application <NUM> issues a Control Device message <NUM> that is directed to a profiled device, automation hub <NUM> responds by forwarding a Control Request message <NUM> to multi-role automation host <NUM>. In turn, multi-role automation host <NUM> responds by exerting control <NUM> over the profiled device to which message <NUM> is directed.

<FIG> shows a message flow diagram illustrating an example of lower level messaging that occurs when hub control application <NUM> and automation hub <NUM> are exerting control over profiled device 108a that has become visible through multi-role automation host <NUM> as described above. In this example, automation hub <NUM> represents an Apple TV®, HomePod™, or iPad® running HomeKit®. Hub control application <NUM> issues a HMActionSet execute message <NUM> to automation hub <NUM> which, in turn, issues a Device Characteristic Write message <NUM> to multi-role automation host <NUM>. Device Characteristic Write message <NUM> could operate, for example, to dim a lamp.

In response, multi-role automation host <NUM> locates the profile corresponding with the device <NUM> to which message <NUM> is directed, and then sends a profiled command <NUM> to device 108a. Profiled command <NUM> is in substance comparable to message <NUM> but is prepared by multi-role automation host <NUM> in a form that is expected and understood by device 108a. Following receipt and execution of profiled command <NUM>, device 108a sends a command response <NUM> to multi-role automation host <NUM> which, in turns, sends a Characteristic Update message <NUM> to automation hub <NUM>.

Turning now to <FIG>, an automation environment <NUM> is shown. In general, each component or device shown in <FIG> may include comparable structure and function to the corresponding component or device shown in <FIG> and described above with one exception. A multi-role automation hub <NUM>, in its role as a hub, functions in a substantially comparable manner to automation hub <NUM> (<FIG>) described above. However, multi-role automation hub <NUM>, in an additional role, produces an enhanced on screen display (EOSD) for a television <NUM>.

In a preferred embodiment, as referenced above, automation hub <NUM> may be based on Apple TV® running HomeKit®. While Apple TV® includes a limited on screen display which allows a user to select audio/video to play, or power on/off an attached TV, that limited on screen display is not capable of controlling other devices. However, Apple TV® does include a high performance graphics processing capability.

In accordance with one aspect of the present invention, an application is provided which runs on Apple Inc. 's tvOS®, and leverages the high performance graphics processing built into Apple TV® to produce an EOSD <NUM> as shown in <FIG>. By placing an Apple TV® into a lockdown mode, the Apple TV® will automatically boot up into the application which produces EOSD <NUM>.

As shown in <FIG>, EOSD <NUM> may be formatted as three horizontal bands of rotating icons, graphics, images, video or other content. For example, band <NUM> displays a series of "Scenes" corresponding to various rooms or areas of a home. When a user selects a particular "Scene" (by interacting with EOSD <NUM> using user devices 516a or 516b, or user remotes 518a or 518b (<FIG>)) to center and select the corresponding image, previously defined conditions or states of lighting, devices, shades, media, or any other resource that is under the control of multi-role automation host <NUM> or multi-role automation hub <NUM>, will be set in the selected room or area.

Band <NUM> of EOSD <NUM> displays icons corresponding to various systems and services. When a user selects a centered "Security" icon, EOSD <NUM> transitions to a screen shown in <FIG> in which band <NUM> displays live feeds from several closed circuit security cameras. Similarly, as shown in <FIG>, when a user selects a centered "Lighting" icon, EOSD <NUM> transitions to a screen shown in <FIG> in which band <NUM> displays true images of the current lighting conditions of individual rooms or lamps along with a vertical slider bar control. Exemplary techniques for generating and displaying such true images of current lighting conditions are described in <CIT>, and co-pending application Serial No. <CIT>, both of which are incorporated by reference above.

EOSD <NUM> may also provide advanced video display management including windowing, tiling, and picture-in-picture among others. Details regarding hardware and software for generating EOSD <NUM> may be found in the patents and pending patent applications incorporated by reference above.

<FIG> shows a message flow diagram illustrating an example of lower level messaging that occurs when multi-role automation hub <NUM> with EOSD <NUM> interacts with multi-role automation host <NUM>. Host control application <NUM> issues a Control Profiled Device message <NUM> to multi-role automation host <NUM>. Because message <NUM> is directed to a profiled device, multi-role automation host <NUM> proceeds to Execute Profiled Device Command <NUM>.

Subsequently, host control application <NUM> issues a Control Certified Device message <NUM>. Because message <NUM> is not directed to a profiled device, multi-role automation host <NUM> issues a Forward Request message <NUM>, via bridge <NUM>, to multi-role automation hub <NUM>. Multi-role automation hub <NUM>, in turn, issues a Control Request message <NUM> to automation hub <NUM> which proceeds to Execute Certified Device Command <NUM>.

In accordance with one aspect of the invention, an application is provided which runs on Apple Inc. 's tvOS®, and provides the necessary functionality to transform an Apple TV® into multi-role automation host. Other devices which are controllable by a dedicated remote may also be suitable to serve as a multi-role automation host. In general, the application embodies all necessary automation host functionality including the ability to recognize and control profiled devices. Because of the native functionality of an Apple TV®, this type of multi-role automation host is capable of effectively controlling a mix of profiled devices and certified devices without an additional hub or other intermediate device. In addition, the capability of this type of multi-role automation host may be further enhanced by the addition of AVB network connectivity. Such connectivity, in the case of an Apple TV® multi-role automation host, permits distribution of music in digital form directly to AVB enabled speakers, soundbars or other devices.

In accordance with another aspect of the invention, multiple Apple TVs may be pooled together to become a shared resource for multiple TVs. As shown in <FIG>, an automated environment <NUM> includes a group of TVs 712a-712c, each of which is coupled to a video switch <NUM>. A multi-role automation host <NUM> is coupled to video switch <NUM>, as are multiple Apple TVs 718a and 718b, and multiple hubs with enhanced OSD 726a and 726b (which are substantially similar to hub with enhanced OSD <NUM> (<FIG>) described above).

In general, in response to commands from multi-role automation host <NUM>, video switch <NUM> functions to set up and tear down video signal connections or paths between various ones of TVs 712a-712c, and various ones of Apple TVs 718a-718b or hubs with enhanced OSD 726a-726b. Multi-role automation host <NUM> may also command video switch <NUM> to set up a video signal connection between a single Apple TV 718a-718b and multiple TVs 712a-712c. Similarly, multi-role automation host <NUM> may also command video switch <NUM> to set up a video signal connection between a single hub with enhanced OSD 726a-726b and multiple TVs 712a-712c.

<FIG> is a message flow diagram illustrating communication among various devices in <FIG> needed to display an enhanced OSD on one of TVs 712a-712c. The communication begins with a user (not shown) pressing a button on user remote <NUM> which causes transmission of an OSD Request for Zone X message <NUM> to multi-role automation host <NUM>. Multi-role automation host <NUM> responds first by Dequeue Menu Token Y which, in effect, dedicates a particular Apple TV 718a-718b (associated with the token) to the requested OSD session. Next, multi-role automation host <NUM> transmits an Activate OSD message <NUM> to a Menu Resource Y <NUM> (i.e., a particular Apple TV associated with Token Y). This is followed by a Start Stream <NUM> message <NUM> from multi-role automation host <NUM> to a Menu Resource Y's Video Transmitter <NUM> which, in turn, is followed by a Start Stream <NUM> With Resized Video message <NUM> to Current Transmitter Used By Zone X <NUM>.

Next, multi-role automation host <NUM> sends a message <NUM> to Zone X Video Receiver <NUM> which sets up picture in picture and assigns the appropriate transmitters. At this point, the enhanced OSD appears in a picture in picture <NUM>. A User OSD Navigation message <NUM> is then transmitted from user remote <NUM> to Menu Resource Y <NUM>. This is followed by a Cancel or Select message <NUM> from user remote <NUM>. If a Cancel message <NUM> was transmitted, an OSD Deactivated message <NUM> is transmitted from Menu Resource Y to multi-role automation host <NUM>. This is followed by Menu Token Y being returned to the Queue <NUM> because the OSD session is over.

Subsequently, a message <NUM> is transmitted from multi-role host <NUM> to restore full screen video for the original transmitter. This is followed by a Stop Stream <NUM> message <NUM> to Current Transmitter <NUM>, a Stop Stream <NUM> message to Menu Resource Y's Video Transmitter <NUM>, and restoration of full screen video <NUM>.

In addition, similar to the enhanced OSD example described above, other services available on an Apple TV 718a-718b may be shared with multiple TVs 712a-712c. Thus, services such as Netflix, Weather, or many others may be shared with multiple TVs 712a-712c by placing an Apple TV in lockdown mode and transmitting the selected video over IP, taking advantage of video switch <NUM> to set up the appropriate video connections.

In accordance with another aspect of the invention, a ceiling or wall mountable speaker, such as those disclosed in co-pending application Serial No. <CIT> and incorporated by reference above, or bookshelf or other speaker, is provided enhanced functionality and becomes a multi-role speaker. The previously disclosed speakers include a LAN/AVB port through which both digital audio and power over Ethernet (PoE) are received. As shown in <FIG>, a multi-role speaker <NUM> includes a LAN/AVB port <NUM> coupled to a microprocessor <NUM>. A microphone <NUM> is coupled to microprocessor <NUM> as is a digital signal processor (DSP) <NUM>. An amplifier <NUM> is coupled to DSP <NUM> as well as speaker <NUM>. Multi-role speaker <NUM> also includes a WISA module <NUM> and a WiFi dual band module <NUM>, both of which provide wireless connectivity. Through its WISA module <NUM> or LAN/AVB port <NUM>, multi-role speaker <NUM> may distribute digital audio in either of two forms: <NUM>-bit, <NUM> form (used in Airplay <NUM>), or higher fidelity <NUM>-bit, <NUM> form.

Through its WiFi dual band module <NUM>, multi-role speaker <NUM> provides support for Airplay <NUM> as well as serving as a wireless access point. To provide whole house audio, speakers are frequently distributed throughout most rooms or other areas of a house. By providing such speakers with wireless network connectivity so that they become multi-role speakers, robust wireless network coverage is attained without the necessity of separate wireless access points. Through its WISA module <NUM>, multi-role speaker <NUM> may be paired with a WISA speaker <NUM> to provide stereo sound. Paired WISA speaker <NUM> includes a microcontroller <NUM> which is coupled to a WISA module <NUM> and a DSP <NUM>. DSP <NUM> is coupled to an amplifier <NUM>, which in turn is coupled to a speaker <NUM>. In general, paired WISA speaker <NUM> receives digital audio from multi-role speaker <NUM> via WISA module <NUM>.

Within an automation environment <NUM> (<FIG>), multi-role speaker <NUM> is capable of serving in any combination of four roles: audio speaker; multi-role automation host <NUM> (<FIG>); wireless access point; and pairing with WISA speaker <NUM>.

<FIG> shows a multi-role amplifier <NUM> which includes a microprocessor <NUM> coupled to a DSP <NUM> which in turn is coupled to an amplifier <NUM>. Microprocessor <NUM> is coupled, respectively, to a LAN/AVB port <NUM>, an analog input <NUM>, a TosLink input <NUM>, and control ports <NUM> which may include RS232, IR, or other control ports. An AC/DC power supply <NUM> provides power to multi-role amplifier <NUM>. Multi-role amplifier <NUM> provides an RCA pre-amp output <NUM> and may be used to drive external passive speakers <NUM>.

Within an automation environment <NUM> (<FIG>), multi-role amplifier <NUM> is capable of serving as both an audio amplifier and a multi-role automation host <NUM> (<FIG>).

Claim 1:
An automation control apparatus (<NUM>), said apparatus comprising a multi-role automation host (<NUM>), said multi-role automation host having connectivity for communicating with a first user device (116a, 118a), said first user device having a set of commands (<NUM>) to which said multi-role automation host is responsive;
said multi-role automation host including an automation bridge (<NUM>), said automation bridge communicating bidirectionally with one or more devices including a first automation hub (<NUM>) using a predetermined control protocol, said automation bridge converting control protocol messages received from said first automation hub to an appropriate form for processing by said multi-role automation host, said multi-role automation host operating in a first role in which said multi-role automation host directly controls one or more profiled devices (108a, 108b), each of said one or more profiled devices having an associated profile which is accessible by said multi-role automation host, each said associated profile including information which describes a set of device commands to which said associated profiled device is responsive;
said multi-role automation host operating in a second role in which said multi-role automation host, through said automation bridge, directly controls a first set (114a) of one or more devices each of which is certified compliant with said predetermined control protocol, and, through said automation bridge, indirectly controls through said first automation hub a second set (114b, 114c) of one or more devices each of which is certified compliant with said predetermined control protocol; and
a second user device (116b, 118b) having a set of commands (<NUM>, <NUM>) to which said first automation hub is responsive, said second user device in conjunction with said first automation hub and said multi-role automation host effectively controlling at least one of said one or more profiled devices.