Patent Publication Number: US-10775960-B2

Title: User generated virtual room-based user interface

Description:
RELATED APPLICATIONS 
     The present application is a continuation-in-part of U.S. patent application Ser. No. 13/551,289, filed Jul. 17, 2012 by Robert P. Madonna et al., entitled “Virtual Room-Based Light Fixture and Device Control”, which is itself a continuation of U.S. patent application Ser. No. 12/792,236, filed on Jun. 2, 2010 by Robert P. Madonna et al., and entitled “Virtual Room-Based Light Fixture and Device Control”, which claims the benefit of U.S. Provisional Patent Application Ser. No. 61/183,825, filed Jun. 3, 2009 by Robert P. Madonna et al., entitled “Virtual Room-Based Light Fixture and Device Control”, the contents of all of which are incorporated by reference herein in their entirety. 
    
    
     BACKGROUND 
     Technical Field 
     The present disclosure relates generally to device control and more specifically to generation of a user interface to control devices coupled to a home automation or electronic device control system. 
     Background Information 
     As homes and other structures become larger, and become filled with more devices, such as light fixtures, motor-actuated devices and electronic devices, the burden of controlling these devices increases. Traditionally, many devices have been controlled by mechanical switches. While mechanical switches are reliable and cost-effective, they have many limitations, especially when there are many devices located in the same room of a structure. For example, a large room may include a number of light fixtures, such as lamps, overhead fixtures, task lights, etc.; a number of motor-operated devices, such as automatic window shades, automatic doors, etc.; and a number of electronic devices, such as display screens (e.g., televisions and monitors), audio/video (a/v) devices, computer devices, etc. To control all these devices, a large number of mechanical switches may be needed. As the number of mechanical switches increases within the room, usability decreases. Mechanical switches often are unlabeled, or if labeled, marked with only cryptic descriptions (e.g., “Lamp 1”, “Lamp 2”, “Window Shade 1”, etc.). A user may be forced to memorize which of the many mechanical switches available in the room controls which device. A new guest in the room, who has not memorize this relationship, typically must rely upon trial and error to determine which mechanical switch controls which device, flipping switches until they happen upon the result they desire. 
     A variety of types of home automation and electronic device control systems have been developed that attempt to improve upon the shortcomings of mechanical switches. Such systems typically include one or more controllers that manage the operation of devices. The controllers may be interacted with via user interface devices, such as dedicated touch-screen control units, that provide a user interface for controlling the devices. The user interface may include an array of touch-sensitive buttons, where each button controls a device or a group of devices. 
     However, such button-centric on-screen user interfaces share many of the same shortcomings as mechanical switches. While the buttons are shown on a screen, rather than existing as physical objects like mechanical switches, they operate very similarly. By looking at an on-screen button, it may not be apparent what the button does. While a label may be provided, often such labels are short and cryptic, given the constraints of screen space. A new guest to a room, similar to the case with mechanical switches, may have to touch each on-screen button to discover by trial and error what button achieves what result. 
     Recently, a virtual-room based device control solution has been developed that addresses many of the shortcomings of mechanical switches and button-centric on-screen user interfaces. This virtual-room based device control solution provides a virtual room-based user interface that includes one or more virtual rooms. By interacting with the virtual rooms, a user may indicate desired state changes for devices within the physical room. 
     However, while this type of solution solves many of the shortcomings of mechanical switches and button-centric on-screen user interfaces, it presents new challenges. One challenge is how to generate the virtual rooms in an economical manner. Unlike many other types of user interface screens, virtual rooms are unique. A single screen cannot be centrally designed and then merely distributed; each virtual room generally needs to be custom created for a corresponding physical room of a structure. 
     Further, virtual rooms may need to be re-created on an ongoing basis. Since a virtual room is intended to correspond to a physical room, if the appearance of the physical room changes, the corresponding virtual room may need to be re-created to provide a satisfactory user experience. For example, if a user rearranges furniture, repaints walls, or otherwise changes the appearance of the physical room, the corresponding virtual room may need to be re-created to reflect these changes. The expense incurred creating and recreating virtual rooms poses a barrier to their more wide-spread use in device control user interfaces. 
     Accordingly, there is a need for improved techniques to generate virtual rooms that may address some or all of these challenges. 
     SUMMARY 
     In one embodiment, a technique is provided to rapidly and economically generate a virtual room of a virtual room-based user interface that control devices coupled to a home automation or electronic device control system. The technique may be implemented on a mobile device, for example, a tablet computer, by a user who lacks specialized training, for example, a homeowner or other end-user. 
     For a given virtual room of the virtual room-based user interface, an application executing on the mobile device prompts a user to select one or more services provided by devices (e.g., light fixtures, motor-actuated devices, or electronic devices) within the physical room, to be controllable via the virtual room. The selected services may be a subset of all services that are possible by the devices within the physical room. Each service may have at least one active state and an inactive state, the natures of which generally depend on the particular service. The configuration application executing on the mobile device then obtains a plurality of images (e.g., photographs) of the physical room in different states, where each image shares substantially the same perspective (e.g., is captured from the same location in the same direction). The plurality of images may include an all-off image, in which all of the one or more of the services are in the inactive state, and one or more service-specific images, in which a single service is in the active state and other services are in the inactive state. In one alternative, the configuration application obtains the plurality of images by capturing each image using a camera of the mobile device, while the configuration application deactivates and then sequentially activates services within the room. In another alternative, the configuration application obtains the plurality of images by prompting the user to assign previously-captured images from an image library. 
     After the plurality of images of the physical room have been obtained, the configuration application executing on the mobile device aligns the images to create an aligned image stack. Such alignment compensates for minor differences in perspective between the plurality of images. The alignment may be a manual process, based on input received on the mobile device from the user. Alternatively, the alignment may be an automatic process, utilizing an image feature alignment algorithm, to create the aligned image stack. The configuration application executing on the mobile device then designates one or more hit boxes about devices shown in the images of the aligned image stack. Each hit box corresponds to a service provided by the respective device. The designation of hit boxes may be based upon input on the mobile device from the user. 
     Based upon the aligned image stack and the designated one or more hit boxes, the configuration application generates a virtual room of the virtual room-based user interface, that is usable to control the one or more devices within the physical room and thereby the one or more services they provide. The generated virtual room may be displayed to the user on the mobile device, and the user prompted to test its functionality. Should the functionality differ from the user&#39;s expectations, the user may cause the configuration application to repeat one or more of the sequence of steps to change the virtual room. Otherwise, the virtual room may be finalized and used along with other virtual rooms (created in a similar manner) in the virtual room-based user interface. 
     It should be understood that a variety of additional features and alternative embodiments may be implemented. This Summary is intended simply as an introduction to the reader, and does not indicate or imply that the examples mentioned herein cover all aspects of the invention, or are necessary or essential aspects of the invention. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The description below refers to the accompanying drawings, of which: 
         FIG. 1  is a block diagram of a generalized architecture for a home automation or electronic device control system configured to control devices within a physical room of a structure; 
         FIG. 2  is a block diagram of an example mobile device; 
         FIG. 3  is a front view of an example mobile device showing an example virtual room of an example virtual room-based user interface displayed upon a touch-sensitive screen; 
         FIG. 4  is a flow diagram of an example sequence of steps for generating a given virtual room of a virtual room-based user interface; 
         FIG. 5  is a front view of an example mobile device showing an example service selection screen; 
         FIG. 6  is a front view of an example mobile device showing an example image capture screen, according to one alternative; 
         FIG. 7  is a front view of an example mobile device showing an example image assignment screen, according to another alternative; 
         FIG. 8  is a front view of an example mobile device showing an example image qualification screen, where images are manually aligned; 
         FIG. 9  is a front view of an example mobile device showing an example marking screen, where hit boxes are defined; and 
         FIG. 10  is a front view of an example mobile device showing an example marking screen, where a virtual room is tested. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  is a block diagram of a generalized architecture for a home automation or electronic device control system  100  configured to control devices within a physical room of a structure. The devices may include light fixtures, motor-actuated devices, electronic devices, and/or other types of devices. The light fixtures may be lamps, overhead fixtures, task lights or other types of lighting devices. The motor-actuated devices may be automatic window shades, automatic doors, motorized television lifts, or other type of devices moved by motors. The electronic devices may be display screens (e.g., televisions and monitors), audio/video (a/v) devices, computer devices, or other types of devices utilizing electronics. 
     The example system  100  may be composed of a number of components, including controllers, user interface devices, and control modules. The controllers may include a device controller  110  and a host controller  120 . The user interfaces devices may include dedicated touch panel units  180  and mobile devices  200 . The control modules may include load modules  140 . Each of these components may interoperate to control devices within the structure. 
     More specifically, the device controller  110  may provide control and communication functionality, and supports, via a module bus  130 , the load modules  140 . The device controller  110  may be responsible for issuing control commands to individual load modules  140  to cause them to change the state of specific devices coupled to them. The host controller  120  may be coupled to the device controller  110 , for example, by a wired local area network (LAN)  150 . The host controller  120  may be configured to control and monitor operations of the device controller  110 , as well as to provide high-level control and user interface support functions. These user interface support functions may support, among other things, a virtual room-based user interface for controlling the devices. 
     One or more wireless access points  160  may be coupled to the LAN  150 , and support, for example, a wireless local area network (WLAN)  170  used to communicate with dedicated touch panels  180  and mobile devices  200 . Alternatively, or additionally, dedicated touch panels  180  and mobile devices  200  may communicate with the host controller  120  via wired connections to the wired LAN  150 . As discussed in more detail below, the dedicated touch panels  180  and mobile devices  200  may present a virtual room-based user interface, via which a user may indicate desired changes to the state of devices within the structure. 
     As used herein, the term “mobile device” refers to a multi-purpose electronic device that is adapted to be transported on one&#39;s person and includes a wireless communication interface and a touch sensitive display screen. A mobile device may, but need not, include an integrated camera. Devices such as tablet computers (e.g., the iPad® tablet available from Apple, Inc.) and smartphones (e.g., the iPhone® smartphones available from Apple, Inc., and Android® smartphones available from various suppliers), are considered mobile devices. Devices such as desktop computers are generally not considered mobile devices. The mobile devices  200  of the system  100  may be used in a control role, to provide the virtual room-based user interface to a user, or in a configuration role, to support generation of the virtual room-based user interface. An individual mobile device may be used in only one, or in both, of these roles. 
       FIG. 2  is a block diagram of an example mobile device  200 . The mobile device  200  may include a processor  210 , a memory  220 , a wireless network interface  230 , a touch-sensitive display screen  240 , a camera  250 , as well as other hardware. The processor  210  may include logic configured to execute software and manipulate data structures. The memory  220  may include a plurality of storage locations for storing the software and the data structures. The wireless network interface  230  may facilitates communication over a WLAN and/or cellular broadband data network. The touch-sensitive display screen  240  may receive user input in the form of gestures (e.g., touches, swipes, etc.) from a user. 
     An operating system  260  resident in memory  220 , may functionally organize the mobile device  200 . The operating system  260  may be an IOS® operating system available from Apple, Inc., an Android® operating system available from Google, Inc., or another type of operating system. A configuration application  270  may be executed in conjunction with the operating system  260 , to permit the mobile device  200  to operate in a configuration role, to generate a virtual room of a virtual room-based user interface. The configuration application  270  may display a configuration user interface on the touch sensitive display screen  240 , and receive user input thereon. In addition, a control application (not shown) may also be stored in the memory  220 , and executed to permit the mobile device  200  to operate in a control role. The control application (not shown) may display the virtual room-based user interface on the touch sensitive display screen  240 , and receive user input thereon, to permit a user to indicated desired changes to the state of devices within the structure. 
     The virtual room-based user interface may include one or more virtual rooms that each show a photo-realistic depiction of a corresponding physical room of the structure. The photo-realistic depiction may show boundaries of the physical room (e.g., the walls, ceiling, floor, etc.), furnishings present in the physical room (e.g., sofas, chairs, beds, wall-hangings, etc.), and controllable devices present in the physical room (e.g., light fixtures, motor-actuated devices, electronic devices, and/or other types of devices). The photo-realistic depictions of the devices are preferably shown at locations within the virtual room corresponding to the device&#39;s actual locations within the physical room. The virtual room may operate as an interface environment. By interacting with (e.g., touching) photo-realistic depictions of the devices within the virtual room, a user may indicate changes to the state of services provided by the corresponding devices in the physical room. As the state of devices is changed so they provide different services within the physical room, the appearance of the virtual room may be dynamically updated to reflect the changes. In such manner, the virtual room may be updated to reflect back to the user a photo-realistic depiction of the present condition of the corresponding physical room, such that what a user views within the virtual room will mirror, or at least resemble, their experience within the corresponding physical room. Such an interface may be highly intuitive, allowing a user to readily indicate control selections without having to understand complicated control schemes. 
       FIG. 3  is a front view  300  of an example mobile device showing an example virtual room  310  of an example virtual room-based user interface displayed upon a touch-sensitive screen of the mobile device. The example virtual room  310  corresponds to a living room within an example home. However, it should be understood that such choice of room for this example is arbitrary. The example virtual room  310  includes photo-realistic depictions of boundaries of the physical room, such as the ceiling  315  and walls  320 ; photo-realistic depictions of furnishings present in the physical room, such as a sofa  330 , coffee table  340 , and end table  350 ; as well as photo-realistic depictions of controllable devices within the physical room, such as a chandelier  360 , lamps  370 ,  375 , recessed cans  380 , and a television  390 . Each of these devices  360 - 390  provides a service having states, which include at least one active state and an inactive state. For example, the chandelier  360 , lamps  370 ,  375 , and recessed cans  380  provide a lighting service that includes an active state of “on” (and potentially a plurality of active sub-states representing different lighting levels), and an inactive state of “off”. Similarly, the television  390  provides a viewing service that includes an active state of “on” (and potentially a plurality of active sub-states representing different video sources, channels, volume levels, etc. being displayed/used), and an inactive state of the television being “off”. 
     By interacting with (e.g., touching) photo-realistic depictions of the devices  360 - 390  within the virtual room  310 , a user may indicate changes to the state of the services provided by the corresponding devices in the corresponding physical room. These changes may be implemented by the home automation or electronic device control system  100 . For example, by touching the photo-realistic depiction of the chandelier  360 , the user may cause the chandelier within the physical room to be illuminated (or otherwise controlled). Similarly, by touching the photo-realistic depiction of the television  390 , the user may cause the television within the physical room to be powered on (or otherwise controlled). For more advanced control, one or more menus (not shown) may be presented within the virtual-room based user interface, for example, overlaid upon a portion of the virtual room  310 , to permit further control selections. 
     As the devices  360 - 390  provide different services, the appearance of the virtual room  310  may be dynamically updated to reflect the changes. For example, when the chandelier is turned on, the photo-realistic depiction of the chandelier  360  within the virtual room  310  is changed to depict the chandelier in an illuminated state. Similarly, if the television is powered on, the photo-realistic depiction of the television  390  within the virtual room  310  is changed to depict the television in a powered on state. In this manner, the virtual room  310  may minor, or at least resemble, the present appearance of the physical room. 
     As discussed above, it has previously been a challenge to generate virtual rooms in an economical manner. To address this challenge, the configuration application  270  may implement a sequence of steps that enable a user who lacks specialized training, for example, a homeowner or other end-user, to generate virtual rooms and thereby construct a virtual room-based user interface. This may obviate the need for custom programming and configuration by an installer. The sequence of steps may be repeated for each desired virtual room. 
       FIG. 4  is a flow diagram of an example sequence of steps  400  for generating a virtual room of a virtual room-based user interface. At step  410 , the configuration application  270  may prompt a user to select a virtual room to generate. At step  420 , the configuration application  270  may display an instructional animation or video providing guidance on how to prepare the physical room, what to do if certain issues are encountered, and other topics. For example, the instructional video may discuss the use of ambient lighting, how to maintain consistency among images, how to estimate the number of images needed, what to do if all the services cannot fit into one image, etc. 
     At step  430 , the configuration application  270  prompts the user to select one or more services provided by devices (e.g., light fixtures, motor-actuated devices, or electronic devices) to be controllable via the virtual room. The selected services may be a subset of all services that are possible by the devices within the physical room. Further, text descriptions, animations and/or video may guide the user through the selection process. 
       FIG. 5  is a front view of an example mobile device showing an example service selection screen  500  provided by the configuration application  270 . A guide portion  510  of the screen may indicate the steps used in generating a virtual room, with a present step being to select services  512 . A description portion  520 , and an animation/video portion  530 , may provide text and pictorial guidance to the user. Finally, a selection portion  540  may provide a listing of services, from which a user can select desired services (e.g., by touch) to be controllable. Each service may correspond to a respective device and share a name with the device. 
     At step  440 , the configuration application  270  obtains a plurality of images (e.g., photographs) of the physical room in different states, where each image shares substantially the same perspective (e.g., is captured from the same location in the same direction). The plurality of images may include an all-off image, in which all of the services are in the inactive state, and one or more service-specific images, in which a single service is in the active state and other services are in the inactive state. 
     In one alternative, the configuration application  270  may obtain the plurality of images by capturing each image using the camera  250  of the mobile device  200 , while the configuration application  270  deactivates, and then sequentially activates, services within the room. In another alternative, the configuration application  270  may obtain the plurality of images by prompting the user to assign previously-captured images from an image library. 
       FIG. 6  is a front view of an example mobile device showing an example image capture screen  600  provided by the configuration application  270 , according to one alternative. A guide portion  610  of the screen may be expanded to show image capture steps. A user may select from the guide portion  610  using options  612 . The options may include an automated mode, where the configuration application  270  controls camera operation, and a manual mode, where the user controls the camera operations. A description portion  620  may instruct the user to stand in one location and point the camera  250  in one direction. The user may position the camera  250  by looking at an image capture portion  640  of the screen  600 . If the automated mode is selected, the configuration application  270  may rapidly cycle through service activations, while activating the camera  250  when appropriate. For example, the configuration application  270  may communicate with the host controller  120  and device controller  110  to cause all services to be in the inactive state, and then trigger the camera  250  to capture an all-off image. The configuration application  270  may sequentially cause the single services to be activated, and trigger the camera  250  to capture a service-specific image each time. 
     If the manual mode is selected, the configuration application  270  may control service activation, but the user may control camera functions. For example, the configuration application  270  may communicate with the host controller  120  and device controller  110  to cause all services to be in the inactive state, and then prompt the user to use the camera  250  to take an all-off image. The user may capture the image by interacting with a camera button  650 . The configuration application  270  may sequentially activate single services, and prompt the user each time to use the camera button  650  to take a service-specific image. After capturing each image, a user may have the option to retake the image before proceeding onward. The user may base the decision of whether or not to retake the image multiple factors. One factor may be the similarity of perspective to previously captured images, such as the all-off image. An accuracy meter  630  may be provided to indicate such similarity to the user. In this manner, images may be obtained by image capture. 
     As mentioned above, the image may also be obtained from pre-captured images.  FIG. 7  is a front view of an example mobile device showing an example image assignment screen  700 . A guide portion  710  of the screen may show that the present step it to assign images  712 . A description portion  720  and a video/animation portion  730  may provide instructions regarding image assignment. An assignment portion  740  may display previously-captured images. The user may be prompted to select each image, and to assign each image to a respective role, for example, either as the all-off image or as a certain one of the service-specific images. When so assigned, a name of the role may be displayed below each image. In some cases the name may coincide with the name of the device. 
     At step  450 , the configuration application  270  aligns the images to create an aligned image stack. Such alignment compensates for minor differences in perspective between the plurality of images. The alignment may be a manual process, based on input received on the mobile device  200  from the user. Alternatively, the alignment may be an automatic process, utilizing an image feature alignment algorithm executing on the mobile device  200 , to create the aligned image stack. 
       FIG. 8  is a front view of an example mobile device showing an example image qualification screen  800 , where images are manually aligned. A guide portion  810  of the screen  800  may show that the present step is to qualify/align images  812 . A description portion  820  and a video/animation portion  830  may provide instructions regarding image alignment. An alignment portion  840  of the screen may allow a user to manually move images to align them. For example, the all-off image may be displayed, and sequentially each service-specific image may be overlaid upon it. The user may be prompted to adjust the alignment of the service-specific image, for example, by dragging  842  or using directional buttons  844 , so that features coincide with the all-off image, to create the aligned image stack. 
     At step  460 , the configuration application  270  designates one or more hit boxes about devices shown in the images of the aligned image stack. Hit boxes may be regions that when selected (e.g., touched) cause a state of a service to be changed. While hit boxes may be delineated by visible boundaries as they are designated, when incorporated into a final virtual room, they are generally invisible to the user. 
       FIG. 9  is a front view of an example mobile device showing an example marking screen  900 , where hit boxes are defined. A guide portion  910  of the screen may show that the present step it to mark hit boxes  912 . A description portion  920  and a video/animation portion  930  may provide instructions regarding hit box creation. A hit box creation portion  940  may allow a user to draw, size, and position shapes representing the hit box  942 - 948  over respective devices. For example, a user may touch the screen and drag diagonally to create a hit box. The user may then assign that hit box to a particular service provided by a device. A user may also resize and adjust the hit box. In one implementation, hit boxes are not allowed to overlap each other. 
     At step  470 , based upon the aligned image stack and the designated one or more hit boxes, the configuration application  270  generates a virtual room of the virtual room-based user interface. The configuration application  270  may employ a number of filtering routines, image adjustment routings, and other software processes as part of step  470 . Further details of example operations that may be utilized are provided in U.S. patent application Ser. No. 12/792,236, which was incorporated by reference above. 
     At step  480 , the configuration application  270  displays the generated virtual room for testing. Should the functionality differ from user-expectations, the user may cause the configuration application  270  to repeat one or more of the sequence of steps, to change the virtual room. Otherwise, the virtual room may be finalized. 
       FIG. 10  is a front view of an example mobile device showing an example testing screen  1000 , where a virtual room is tested. A guide portion  1010  of the screen may show that the present step  1012  it to test the virtual room. The virtual room  1050  may be shown in a test portion  1040 . A description portion  1020  and a video/animation portion  1030  may provide instructions regarding the testing. For example, the user may be instructed to interact with the virtual room  1050 , and touch the photo-realistic depiction for each of the devices (where the now-invisible hit boxes are located). The user may then observe whether the services provided in the physical room meet their expectations. If not, the user may cause the configuration application  270  to repeat an earlier step by selecting a back control  1042 . If so, the user may finalize the virtual room by selecting a confirmation control  1044 . 
     In summary, a technique is described herein that allows rapid and economical generation of virtual rooms for a virtual room-based user interface that controls devices coupled to a home automation or electronic device control system. While the above description provides certain specific examples, it should be apparent that a number of modifications and/or additions may be made thereto. 
     For instance, while it is discussed above that steps may be performed on a mobile device  200 , it should be understood that at least some of the steps may be performed on other types of electronic devices, which lack the characteristics of a mobile device. For example, at least some of the steps may be performed on a desktop computer. Further, steps may be performed on several devices working in conjunctions. For example, some steps may be performed on a desktop computer, while other steps, for example those related to image capture, may be performed by a stand-alone digital camera. A wide variety of alternative arrangements are possible. 
     Further, while it is discussed above that a substantially photo-realistic depiction may be captured by a camera, a substantially photo-realistic depiction may be otherwise created. For example, a photo-realistic depiction may be rendered from a model by a computer-aided-design (CAD) application. 
     Still further, while reference is made above to a “physical room”, it should be understood that a physical room need not coincide with a “room” as labeled on a floor plan or blueprint of the structure. A physical room may be a portion of a room as labeled on a floor plan or blueprint, a combination of multiple such rooms, or other logical partition of a structure defined by one or more boundaries. 
     Additionally, it should be understood that the steps described above may be implemented in hardware, software (embodied as a non-transitory computer-readable medium containing program instructions), firmware, or a combination thereof. A non-transitory computer-readable medium may take the form of a memory, such as a Random Access Memory (RAM), a disk, such as a CD-ROM, or other tangible storage medium. 
     In general, it should be understood that the above descriptions are meant to be taken only by way of example.