Patent Publication Number: US-2018052520-A1

Title: System and method for distant gesture-based control using a network of sensors across the building

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation-in-part of U.S. Ser. No. 15/241,735 filed Aug. 19, 2016, which is incorporated herein by reference in its entirety. 
    
    
     BACKGROUND 
     The subject matter disclosed herein generally relates to controlling in-building equipment and, more particularly, to gesture-based control of the in-building equipment. 
     Traditionally, a person&#39;s interaction with in-building equipment such as an elevator system, lighting, air conditioning, electronic equipment, doors, windows, window blinds, etc. depends on physical interaction such as pushing buttons or switches, entering a destination at a kiosk, etc. Further, a person&#39;s interaction with some in-building equipment is designed to facilitate business management applications, including maintenance scheduling, asset replacement, elevator dispatching, air conditioning, lighting control, etc. through the physical interaction with the in-building equipment. For example, current touch systems attempt to solve requesting an elevator from locations other than at the elevator through, for example, the use of mobile phones, or with keypads that can be placed in different parts of the building. The first solution requires the users to carry a mobile phone and install the appropriate application. The second solution requires installation of keypads which is costly and not always convenient. 
     With advances in technology, systems requiring less physical interaction can be implemented such as voice or gesture controlled systems with different activation systems. For example, an existing auditory system can employ one of two modes to activate a voice recognition system. Typically, a first mode includes a user pushing a button to activate the voice recognition system, and a second mode includes the user speaking a specific set of words to the voice recognition system such as “OK, Google”. However, both activation methods require the user to be within very close proximity of the in-building equipment. Similarly, current gesture-based systems require a user to approach and be within or near the in-building equipment, for example, the elevators in the elevator lobby. 
     None of these implementations allow for calling and/or controlling of in-building equipment such as an elevator from a particular location and distance away. 
     BRIEF DESCRIPTION 
     A gesture-based interaction system for communicating with an equipment-based system according to one, non-limiting, embodiment of the present disclosure includes a sensor device configured to capture at least one scene of a user to monitor for at least one gesture of a plurality of possible gestures conducted by the user and output a captured signal; and a signal processing unit including a processor configured to execute recognition software, a storage medium configured to store pre-defined gesture data, and the signal processing unit is configured to receive the captured signal, process the captured signal by at least comparing the captured signal to the pre-defined gesture data for determining if at least one gesture of the plurality of possible gestures are portrayed in the at least one scene, and output a command signal associated with the at least one gesture to the equipment-based system. 
     Additionally to the foregoing embodiment, the plurality of possible gestures includes conventional sign language applied by the hearing impaired, and associated with the pre-defined gesture data. 
     In the alternative or additionally thereto, in the forgoing embodiment, the plurality of possible gestures includes a wake-up gesture to begin interaction, and associated with the pre-defined gesture data. 
     In the alternative or additionally thereto, in the forgoing embodiment, the gesture-based interaction system includes a confirmation device configured to receive a confirmation signal from the signal processing unit when the wake-up gesture is received and recognized, and initiate a confirmation event to alert the user that the wake-up gesture was received and recognized. 
     In the alternative or additionally thereto, in the forgoing embodiment, the plurality of possible gestures includes a command gesture that is associated with the pre-defined gesture data. 
     In the alternative or additionally thereto, in the forgoing embodiment, the gesture-based interaction system includes a display disposed proximate to the sensor device, the display being configured to receive a command interpretation signal from the signal processing unit associated with the command gesture, and display the command interpretation signal to the user. 
     In the alternative or additionally thereto, in the forgoing embodiment, the plurality of possible gestures includes a confirmation gesture that is associated with the pre-defined gesture data. 
     In the alternative or additionally thereto, in the forgoing embodiment, the sensor device includes at least one of an optical camera, a depth sensor, and an electromagnetic field sensor. 
     In the alternative or additionally thereto, in the forgoing embodiment, the wake-up gesture, the command gesture, and the confirmation gesture are visual gestures. 
     In the alternative or additionally thereto, in the forgoing embodiment, the equipment-based system is an elevator system, and the command gesture is an elevator command gesture and includes at least one of an up command gesture, a down command gesture, and a floor destination gesture. 
     A method of operating a gesture-based interaction system according to another, non-limiting, embodiment includes performing a command gesture by a user and captured by a sensor device; recognizing the command gesture by a signal processing unit; and outputting a command interpretation signal associated with the command gesture to a confirmation device for confirmation by the user. 
     Additionally to the foregoing embodiment, the method includes performing a wake-up gesture by the user captured by the sensor device; and acknowledging receipt of the wake-up gesture by the signal processing unit. 
     In the alternative or additionally thereto, in the forgoing embodiment, the method includes performing a confirmation gesture by the user to confirm the command interpretation signal. 
     In the alternative or additionally thereto, in the forgoing embodiment, the method includes recognizing the confirmation gesture by the signal processing unit by utilizing recognition software and pre-defined gesture data. 
     In the alternative or additionally thereto, in the forgoing embodiment, the method includes recognizing the command gesture by the signal processing unit by utilizing recognition software and pre-defined gesture data. 
     In the alternative or additionally thereto, in the forgoing embodiment, the method includes sending a command signal associated with the command gesture to an equipment-based system by the signal processing unit. 
     In the alternative or additionally thereto, in the forgoing embodiment, the wake-up gesture and the command gesture are visual gestures. 
     In the alternative or additionally thereto, in the forgoing embodiment, the sensor device includes an optical camera for capturing the visual gestures and outputting a captured signal to the signal processing unit. 
     In the alternative or additionally thereto, in the forgoing embodiment, the signal processing unit includes a processor and a storage medium, and the processor is configured to execute recognition software to recognize the visual gestures. 
     The foregoing features and elements may be combined in various combinations without exclusivity, unless expressly indicated otherwise. These features and elements as well as the operation thereof will become more apparent in light of the following description and the accompanying drawings. It should be understood, however, that the following description and drawings are intended to be illustrative and explanatory in nature and non-limiting. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The foregoing and other features, and advantages of the present disclosure are apparent from the following detailed description taken in conjunction with the accompanying drawings in which: 
         FIG. 1  is a block diagram of a gesture and location recognition system for controlling in-building equipment in accordance with one or more embodiments; 
         FIG. 2  is a block diagram of a gesture and location recognition system for controlling in-building equipment in accordance with one or more embodiments; 
         FIG. 3  is a diagram of a building floor that includes the gesture and location recognition system in accordance with one or more embodiments; 
         FIG. 4  depicts a user interaction between the user and the gesture and location recognition system in accordance with one or more embodiments; 
         FIG. 5  depicts a user gesture in accordance with one or more embodiments; 
         FIG. 6  depicts a two-part user gesture that is made up of two sub-actions in accordance with one or more embodiments; 
         FIG. 7  depicts a two-part user gesture that is made up of two sub-actions in accordance with one or more embodiments; 
         FIG. 8  depicts a state diagram for a gesture being processed in accordance with one or more embodiments; 
         FIG. 9  depicts a state diagram for a gesture being processed in accordance with one or more embodiments; 
         FIG. 10  is a flowchart of a method that includes gesture at a distance control in accordance with one or more embodiments; and 
         FIG. 11  is a flowchart of a method of operating a gesture-based interaction system. 
     
    
    
     DETAILED DESCRIPTION 
     As shown and described herein, various features of the disclosure will be presented. Various embodiments may have the same or similar features and thus the same or similar features may be labeled with the same reference numeral, but preceded by a different first number indicating the figure to which the feature is shown. Thus, for example, element “a” that is shown in FIG. X may be labeled “Xa” and a similar feature in FIG. Z may be labeled “Za.” Although similar reference numbers may be used in a generic sense, various embodiments will be described and various features may include changes, alterations, modifications, etc. as will be appreciated by those of skill in the art, whether explicitly described or otherwise would be appreciated by those of skill in the art. 
     Embodiments described herein are directed to a system and method for gesture-based interaction with in-building equipment such as, for example, an elevator, lights, air conditioning, doors, blinds, electronics, copier, speakers, etc., from a distance. According to other embodiments, the system and method could be used to interact and control other in-building equipment such as transportation systems such as an escalator, on-demand people mover, etc. at a distance. One or more embodiments integrate people detection and tracking along with spatio-temporal descriptors or motion signatures to represent the gestures along with state machines to track complex gesture identification. 
     For example, the interactions with in-building equipment are many and varied. A person might wish to control the local environment, such as lighting, heating, ventilation, and air conditioning (HVAC), open or close doors, and the like; control services, such as provision of supplies, removal of trash, and the like; control local equipment, such as locking or unlocking a computer, turning on or off a projector, and the like; interact with a security system, such as gesturing to determine if anyone else is on the same floor, requesting assistance, and the like; or interact with in-building transportation, such as summoning an elevator, selecting a destination, and the like. This latter example of interacting with an elevator shall be used as exemplary, but not limiting, in the specification, unless specifically noted otherwise. 
     In one embodiment, the user uses a gesture-based interface to call an elevator. Additionally, the gesture based interface is part of a system that may also include a tracking system that extrapolates the expected arrival time (ETA) of the user to the elevator being called. The system can also register the call with a delay calculated to avoid having an elevator car wait excessively, and tracks the user sending changes to the hall call if the ETA deviates from the latest estimate. In an alternative embodiment, the remote command for the elevator exploits the user looking at the camera when doing the gesture. In yet another alternative embodiment, the remote command for the elevator includes making a characteristic sound (e.g., snapping fingers) in addition to the gesture. The detection and tracking system may use other sensors (e.g., Passive Infrared (PIR)) instead of optical cameras or depth sensors. The sensor can be a 3D sensor, such as a depth sensor; a 2D sensor, such as a video camera; a motion sensor, such as a PIR sensor; a microphone or an array of microphones; a button or set of buttons; a switch or set of switches; a keyboard; a touchscreen; an RFID reader; a capacitive sensor; a wireless beacon sensor; a pressure sensitive floor mat, a gravity gradiometer, or any other known sensor or system designed for person detection and/or intent recognition as described elsewhere herein. While predominately taught with respect to an optical image or video from a visible spectrum camera, it is contemplated that a depth map or point cloud from a 3D sensor such as a structured light sensor, LIDAR, stereo cameras, and so on, in any part of the electromagnetic or acoustic spectrum, may be used. 
     Additionally, one or more embodiments detect gestures using sensors in such a way that there is a low false positive rate by a combination of multiple factors. Specifically, a low false positive rate can be provided because a higher threshold for a positive detection can be implemented because a feedback feature is also provided that allows a user to know if the gesture was detected. If it was not because of the higher threshold the user will know and can try again to make a more accurate gesture. For example, a specific example of the factors can include: the system making an elevator call only when it has a very high confidence on the gesture being made. This allows the system to have a low number of false positives at the cost of missing the detection of some gestures. The system compensates this factor by communicating to the user whether the gesture has been detected or not. Other means of reducing the false positive rate without many missed detections are provided in one or more embodiments herewith. For example, one or more embodiments include exploiting the orientation of the face (people will typically look at the camera, or sensor, to see if their gesture was recognized), or using additional sources of information (the user might snap the fingers while doing the gesture for example, and this noise can be recognized by the system if the sensor has also a microphone). Accordingly, one or more embodiments include being able to call the elevator through gestures across the building, and providing feedback to the user as to whether or not the gesture has been made. 
     In accordance with other embodiments, calling an elevator from a large distance, i.e., in parts of the building that are far from the elevator is provided. This system and method allows for the optimization of elevator traffic and allocation, and can reduce the average waiting time of users. Further, according to another embodiment, the system does not require a user to carry any device or install any additional hardware. For example, a user may make a gesture with the hand or arm to call the elevator in a natural way. To detect these gestures, this embodiment may use an existing network of sensors (e.g., optical cameras, depth sensors, etc.) already in place throughout the building, such as security cameras. According to one or more embodiments, the sensor can be a 3D sensor, such as a depth sensor; a 2D sensor, such as a video camera; a motion sensor, such as a PIR sensor; a microphone or an array of microphones; a button or set of buttons; a switch or set of switches; a keyboard; a touchscreen; an RFID reader; a capacitive sensor; a wireless beacon sensor; a pressure sensitive floor mat, a gravity gradiometer, or any other known sensor or system designed for person detection and/or intent recognition as described elsewhere herein. 
     Accordingly, one or more embodiments as disclosed herewith provide a method and/or system for controlling in-building equipment from distant places in the building. For example, the user knows that he/she has called an elevator, for example by observing a green light that turns on close to the sensor as shown in  FIG. 4 . 
     Turning now to  FIG. 1 , a block diagram of a system  100  with gesture at a distance control is shown in accordance with one or more embodiments. The system  100  includes at least one sensor device  110 . 1  that is located somewhere within a building. According to one embodiment, this sensor device  110 . 1  is placed away from the elevator lobby elsewhere on the floor. Further this sensor device  110 . 1  can be a video camera that can capture a data signal that includes video sequences of a user. The system  100  may further include other sensor devices  110 . 2  through  110 . n  that are provided at other locations throughout the building floor. The system  100  also includes an equipment-based system  150  that may be an on-demand transportation system (e.g., elevator system). It is contemplated and understood that the equipment-based system  150  may be any system having equipment that a person may desire to control based on gestures via the gesture system  100 . 
     The elevator system  150  may include an elevator controller  151  and one or more elevator cars  152 . 1  and  152 . 2 . The sensor devices  110 . 1 - 110 . n  all are communicatively connected with the elevator system  150  such that they can transmit and receive signals to the elevator controller  151 . These sensor devices  110 . 1 - 110 . n  can be directly or indirectly connected to the system  150 . As shown the elevator controller  151  also functions as a digital signal processor for processing the video signals to detect if a gesture has been provided and if one is detected the elevator controller  151  sends a confirmation signal back to the respective sensor device, e.g.,  110 . 2  that provided the signal that contained a gesture. The sensor device  110 . 2  can then provide a notification to the user  160  that the gesture was received and processed and that an elevator is being called. According to another embodiment, a notification device such as a screen, sign, loudspeaker, etc. (not shown) that is near the sensor provides the notification to the user  160 . Alternatively, notice can be provided to the user by sending a signal to a user mobile device that then alerts the user. Another embodiment includes transmitting a notice signal to a display device (not shown) that is near the detected location of the user. The display device then transmits a notification to the user. For example, the display device can include a visual or auditory display device that shows an image to a user or gives a verbal confirmation sound, or other annunciation, that indicates the desired notification. The user  160  may then travel to the elevator car  152 . 1  or  152 . 2 . Further, the elevator controller  151  can also calculate an estimated time of arrival based on which sensor device  110 . 2  provided the gesture. Accordingly, an elevator call can be tailored to best suit the user  160 . 
     Turning now to  FIG. 2 , a block diagram of a system  101  with gesture at a distance control is shown in accordance with one or more embodiments. This system is similar to that shown in  FIG. 1  in that includes one or more sensor devices  110 . 1 - 110 . n  connected to an equipment-base system  150  (e.g., elevator system). The elevator system  150  may include an elevator controller  151  and one or more elevator cars  152 . 1  and  152 . 2 . The system  101  also includes a separate signal processing unit  140  that is separate from the elevator controller  151 . The signal processing unit  140  is able to process all the received data from the sensor device and any other sensors, devices, or systems and generate a normal elevator call that can be provided to the elevator system  150 . This way the system can be used with already existing elevator systems without a need to replace the elevator controller. Additionally the signal processing unit  140  can be provided in a number of locations such as, for example, within the building, as part of one of the sensor devices, off-site, or a combination thereof. Further the system can include a localization device  130  such as a device detection scheme using wireless routers, or another camera array in the building or some other form of detecting location. This localization device  130  can provide a location of the user to the signal processing unit  140 . For example, a localization device  130  can be made up of wireless communication hubs that can detect signal strength of the user&#39;s mobile device which can be used to determine a location. According to another embodiment, the localization device can use one or more cameras placed throughout the building at known locations that can detect the user passing through the camera&#39;s field of view and can therefore identify where the user is within the building. Other known localization devices can be used as well such as a depth sensor, radar, lidar, audio echo location systems, and/or an array of pressure sensors installed in the floor at different locations. This can be helpful if the image sensor device  110  that receives the gesture from the user  160  is at an unknown location, such as a mobile unit that moves about the building or if the sensor device  110  is moved to a new location and the new location has not yet been programmed into the system. 
       FIG. 3  is a diagram of a building floor  200  that includes the user  260 , who makes a gesture, and gesture and location detecting system that includes one or more sensors  210 . 1 - 210 . 3  that each have a corresponding detection field  211 . 1 - 211 . 3  and an in-building equipment such as an elevator  250  in accordance with one or more embodiments of the present disclosure. The user  260  can make a gesture with their hand, head, arm and/or otherwise to indicate his/her intention to use an elevator  250 . As shown, the gesture is captured by a sensor ( 210 . 1 ) as the user is within the detection field  211 . 1  (e.g., optical camera, infrared camera, or depth sensor) that covers the entrance area, and the system calls the elevator  250  before the user  260  arrives to the elevator door. According to other embodiments, the sensor can be a 3D sensor, such as a depth sensor; a 2D sensor, such as a video camera; a motion sensor, such as a PIR sensor; a microphone or an array of microphones; a button or set of buttons; a switch or set of switches; a keyboard; a touchscreen; an RFID reader; a capacitive sensor; a wireless beacon sensor; a pressure sensitive floor mat, a gravity gradiometer, or any other known sensor or system designed for person detection and/or intent recognition as described elsewhere herein. Accordingly, according to one or more embodiments, the sensor captures a data signal that can be at least one of a visual representation and/or a 3D depth map, etc. 
     Thus, as shown, a gesture can be provided that is detected by one or more sensors  210 . 1 - 210 . 3  which can be cameras. The cameras  210 . 1 - 210 . 3  can provide the location of the user  260  and the gesture from user  260  that can be processed to determine a call to an elevator  250 . Processing the location and gesture can be used to generate a user path  270  through the building floor to the elevator. This generated, expected path  270  can be used to provide an estimated time of arrival at the elevators. For example, different paths through a building can have a corresponding estimate travel time to traverse. This estimated travel time value can be an average travel time detected over a certain time frame, it can be specific to a particular user based on their known speed or average speed over time, or can be set by a building manager. Once a path  270  is generated for a user, the path  270  can be analyzed and matched with an estimate travel time. A combination of estimated travel times can be added together if the user takes a long winding path for example, or if the user begins traveling part way along a path, the estimate can be reduced as well. With this estimated time of arrival the elevator system can call an elevator that best provides service to the user while also maintaining system optimization. As shown in  FIG. 3 , a small floor plan is provided where the user is not very far away from the elevator. Accordingly,  FIG. 3  shows the same idea as a bigger plan that may correspond, for example, to a hotel. 
     According to another embodiment, the user  260  may enter the sensor  210 . 1  field of detection  211 . 1  and not make a gesture. The system will not take any action in this case. The user  260  may then travel into and around the building. Then at some point the user  260  may decide to call an elevator  250 . The user can then enter a field of detection, for example the field of detection  211 . 2  for sensor  210 . 2 . The user  260  can then make a gesture that is detected by the sensor  210 . 2 . When this occurs, the sensor  210 . 2  can analyze or transmit the signal for analysis. The analysis includes determining what the gesture is requesting and also the location of the user  260 . Once these are determined a path  270  to the user requested elevator  250  can be calculated along with an estimate of how long it will take the user to travel along the path  270  to reach the elevator  250 . For example, it can be determined that the user  260  will take 1 minute and 35 seconds to reach the elevator  250 . The system can then determine how far vertically the nearest elevator car is, which can be for example 35 seconds away. The system can then determine that calling the elevator in one minute will have it arrive at the same time as the user  260  arrives at the elevator  250 . 
     According to another embodiment, a user may move along the path  270  only to decide to no longer take the elevator. The sensors  210 . 1 - 210 . 3  can detect another gesture from the user cancelling the in-building equipment call. If the user  260  does not make a cancelation gesture, the sensors  210 . 1 - 210 . 3  can also determine that the user is no longer using the elevator  250  for example by tracking that the user has diverged from the path  270  for a certain amount of time and/or distance. The system can, at first detecting this divergence from the path  270 , provide the user  260  additional time in case the user  260  plans to return to the path  270 . After a predefined amount of time, the system can determine that the user no longer is going to use the elevator  250  and can cancel the elevator call. According to another embodiment, the system may indicate to the user by sending a notification to the user that the previous gesture based call has been cancelled. 
     As shown in  FIG. 3 , and according to another embodiment, an estimated path  270  that the user  260  would follow to take the elevator  250  is shown. This path  270  can be used to calculate an estimated time to arrive at the elevator  250 , which can use that information to wait and then call a particular elevator car at the particular time. For example, a user  260  may provide a gesture that they need an elevator using a sensor  210 . 1 - 210 . 3 . The sensors  210 . 1 - 210 . 3  can then track the user&#39;s movements and calculate a real-time estimate based on the current speed of the user which can be adjusted as the user moves through the building. For example, if a user is moving slowly the time before calling an elevator  250  will be extended. Alternatively if a user  260  is detected as running for the elevator  250  a call can be made sooner, if not immediately, to have an elevator car there sooner based on the user&#39;s traversal through the building. 
     After detecting the gesture, the system can call the elevator right away or, alternatively, it can wait for a short time before actually calling it. In the latter case, the system can use the location of the sensor that captured the gesture to estimate the time that it will take the user to arrive to the elevator, in order to place the call. 
       FIG. 4  depicts an interaction between a user  460  and the detection system  410  in accordance with one or more embodiments of the present disclosure. Particularly,  FIG. 4  illustrates an interaction between a user  460  and a sensor  411  (e.g., optical camera, depth sensor, etc.) of the system. For example, the user  460  makes a gesture  461 , and the system  410  confirms that the elevator has been called by producing a visible signal  412 . Specifically, as shown the gesture  461  is an upward arm movement of the user&#39;s  460  left arm. According to other embodiments, this gesture can be a hand waving, a movement of another user appendage, a head shake, a combination of movements, and/or a combination of movements and auditory commands. Examples of a confirmation include turning on a light  412  close to the sensor  411  capturing the gesture  461 , or emitting a characteristic noise, verbalization, or other annunciation that the user  460  can recognize. According to other embodiments, the system  410  can provide a signal that is transmitted to a user&#39;s  460  mobile device or to a display screen in proximity to the user. Providing this type of feedback to the user  460  is useful so that the person  460  may repeat the gesture  461  if it was not recognized the first time. According to other embodiments, the confirmation feedback can be provided to the user  460  by other means such as by an auditory sound or signal or a digital signal can be transmitted to a user&#39;s personal electronic device such as a cellphone, smartwatch, etc. 
       FIG. 5  depicts a user  560  and a user gesture  565  in accordance with one or more embodiments of the present disclosure. As shown the user  560  raises their left arm making a first gesture  561  and also raises their right arm making a second gesture  562 . These two gestures  561 ,  562  are combined together to create the user gesture  565 . When this gesture  565  is detected the system can then generate a control signal based on the known meaning of the particular gesture  565 . For example, as shown, the gesture  565  has the user  560  raising both arms which can indicate a desire to take an elevator up. According to one or more embodiments, a simple gesture can be to just raise one an arm, if the user wants to go up, or make a downwards movement with the arm if the intention is to go down. Such gestures would be most useful in buildings having traditional two button elevator systems, but may also be useful in destination dispatch systems. For example, a user may make a upward motion indicting a desire to go up and also verbally call out the floor they desire. According to other embodiment, the user may raise their arm a specific distance that indicates a particular floor or, according to another embodiment, raise and hold their arm up to increment a counter that counts the time that is then translated to a floor number. For example a user may hold their arm up for 10 seconds indicating a desire to go to floor  10 . According to another embodiment, the user may use fingers to indicate the floor number, such as holding up 4 fingers to indicate floor  4 . Other gestures, combination of gestures, and combination of gestures and auditory response, are also envisioned that can indicated a number of different requests. 
     According to one or more embodiments, an issue with simple gestures is that they can be accidentally be performed by people. In general, simple gestures can lead to a higher number of false positives. In order to avoid this, one or more embodiments can require the user to perform more complex gestures, e.g., involving more than one arm, as illustrated in  FIG. 5 . In this case, the gesture recognition system is based on detecting an upward movement on both sides  561 ,  561  of a human body  560 . According to other embodiments, other possibilities are to perform the same simple gesture twice, so that the system is completely sure that the user  560  intends to use the elevator. Additionally, alternative embodiments might require the user  560  to make some characteristic sound (e.g., snapping fingers, or whistling) in addition to the gesture. In this case the system makes use of multiple sources of evidence (gesture and audio pattern), which significantly reduces the number of false positives. 
       FIG. 6  depicts a two-part user gesture that is made up of two sub-actions in accordance with one or more embodiments. Decomposition of the gesture into a temporal sequence of movements or “sub-actions” is shown. In this example, the gesture (action) consists of bending the arm upwards. This action produces a series of consecutive movements, called sub-actions. Each movement or sub-action is associated with a specific time period and can be described by a characteristic motion vector in a specific spatial region (in the example, the first sub-action occurs at low height and the second one at middle height). Two different approaches can be utilized to capture this sequence of sub-actions, as illustrated in  FIG. 7  and  FIG. 8 . 
       FIG. 7  depicts a two-part user gesture (sub-action  1  and sub-action  2 ) that is made up of two sub-actions in accordance with one or more embodiments. Specifically, as shown, a sub-action  1  is a up and outward rotating motion starting from a completely down position to a halfway point were the user arm is perpendicular with the ground. The second sub-action  2  is a second movement going up and rotating in toward the user with the user hand rotating in toward the user. As shown the vector for each sub-action is shown as a collection of sub vectors for each frame the movement passes through. These sub-actions can then be concatenated together into an overall vector for the gesture. 
     Further,  FIG. 8  depicts a state diagram for a gesture being processed with one or more embodiments. In order to account for a dynamic gesture, which produces different motion vectors at consecutive times, one can break down the gesture into a sequence of sub-actions, as illustrated in  FIG. 6  for the rising arm gesture. Based on this, one can follow a number of approaches, such as the two described herewith, or others not described.  FIG. 7  illustrates the first type of approach. It consists of building a spatial-temporal descriptor of a Histogram of Optical Flow (HOF), obtained by concatenating the feature vectors at consecutive frames of the sequence. Other descriptors may be used as well, for example HOOF, HOG, SIFT, SURF, ASIFT, other SIFT variants, a Harris Corner Detector, SUSAN, FAST, a Phase Correlation, a Normalized Cross-Correlation, GLOH, BRIEF, CenSure/STAR, ORB, and the like. The concatenated feature vector is then passed to a classifier which determines if it corresponds to the target gesture.  FIG. 8  illustrates the second type of approach, where one makes use of a state machine that allows one to account for the recognition of the different sub-actions in the sequence. Each state applies a classifier that is specifically trained to recognize one of the sub-actions. Although only two sub-actions are used in this example, the gesture can be broken down into as many sub-actions as desired, or a single sub-action can be used (i.e., the complete gesture), which corresponds to the case of not using a state machine and, instead, just using a classifier. 
       FIG. 7  includes an illustration of concatenating feature vectors over time in order to capture the different motion vectors produced by the gesture over time. The concatenated descriptor captures this information and it can be regarded as a spatial-temporal descriptor of the gesture. Shown are only two frames in the illustration, but more could be used. The concatenated feature vectors can belong to contiguous frames or to frames separated by a given elapse of time, in order to sample the trajectory of the arm appropriately. 
       FIG. 8  includes an example of state machine that can be used to detect a complex gesture as consecutive sub-actions (see  FIG. 7 ), where each sub-action is detected by a specialized classifier. As shown, a system starts in state 0 where no action is recognized, started, or partially recognized at all. If no sub-action is detected the system will continue to remain in state 0 as indicated by the “no sub-action” loop. Next, when a sub-action  1  is detected the system moves into state 1 in which the system now has partially recognized a gesture and is actively searching for another sub-action. If no sub-action is detected for a set time then system will return the state 0. However, if the system does recognize the sub-action  2 , the system will transition to state 2 which is a state where the system had detected the completed action and will respond in accordance with the detected action. For example, if the system detected the motion shown in  FIG. 5 or 7 , the system, which can be an elevator system, would call an elevator car to take the user upward in the building. 
     In practice, the output of the classification process will be a continuous real value, where high values indicate high confidence that the gesture was made. For example, when detecting a sub-action that is a combination of six sub-vectors from each frame, it is possible that only 4 are detected meaning a weaker detection was made. In contrast if all six sub-vectors are recognized then a strong detection was made. By imposing a high threshold on this value, the system can obtain a low number of false positives, at the expense of losing some true positives (i.e., valid gestures that are not detected). Losing true positives is not critical because the user can see when the elevator has been actually called or when the gesture has not been detected, as explained above (see  FIG. 4 ). This way, the user can repeat the gesture if not detected the first time. Furthermore, the system may contain a second state machine that allows one to accumulate the evidence of the gesture detection over time, as illustrated in  FIG. 8 . 
       FIG. 9  depicts a state diagram for a gesture being processed with one or more embodiments. Particularly,  FIG. 9  includes an example of state machine that allows one to increase the evidence of the gesture detector over time. In state 0, if the user performs some gesture, three things might happen. In the first case, the system recognizes the gesture with enough confidence (confidence&gt;T 2 ). In this case the machine moves to state 2 where the action (gesture) is recognized and the system indicates this to the user (e.g., by turning on a green light, see  FIG. 4 ). In the second case, the system might detect the gesture but not be completely sure (T 1 &lt;confidence&lt;T 2 ). In this case, the machine moves to state 1, and does not tell the user that the gesture was detected. The machine expects the user to repeat the gesture. If, after a brief elapse of time, the system detects the gesture with confidence&gt;T 1 ′, the action is considered as recognized and this is signaled to the user. Otherwise, it comes back to the initial state. In state 1, the system can accumulate the confidence obtained in the first gesture with the one of the second gesture. Finally, in the third case the gesture is not detected at all (confidence&lt;T 1 ) and the state machine simply waits until the confidence is greater than T 1 . 
     In one or more embodiments, in order to increase the accuracy of the system, one can leverage information such as that the user is looking at the sensor (e.g., optical camera) when doing the gesture. This can be detected by detecting the face under a certain orientation/pose. This type of face detection is relatively accurate given the current technology, and can provide an additional evidence that the gesture has been made. Another source of information that can be exploited is the time of the day when the gesture is done, considering that people typically use the elevator at specific times (e.g., when entering/leaving work, or at lunch time, in a business environment). As discussed above, one might also ask the user to produce a characteristic sound while doing the gesture, for example snapping the fingers while doing the gesture. This sound can be recognized by the system if the sensor has an integrated microphone. 
       FIG. 10  is a flowchart of a method  1100  that includes gesture at a distance control in accordance with one or more embodiments of the present disclosure. The method  1100  includes capturing, using a sensor device, a data signal of a user and detects a gesture input from the user from the data signal (operation  1105 ). Further, the method  1100  includes calculating a user location based on a sensor location of the sensor device in a building and the collected data signal of the user (operation  1110 ). The method  1100  goes on to include generating, using a signal processing unit, a control signal based on the gesture input and the user location (operation  1115 ). Further, the method  1100  includes receiving, using in-building equipment, the control signal from the signal processing unit and controlling the in-building equipment based on the control signal (operation  1120 ). According to another embodiment, the method can include receiving, using an elevator controller, the control signal from the signal processing unit and controlling the one or more elevator cars based on the control signal. 
     In an alternative embodiment, sensors such as Passive Infrared (PIR) can be used instead of cameras. These sensors are usually deployed to estimate building occupancy, for example for HVAC applications. The system can leverage the existing network of PIR sensors for detecting gestures made by the users. The PIR sensors detect movement, and the system can ask the user to move the hand in a characteristic way in front of the sensor. 
     In an additional embodiment, the elevator can be called by producing specific sounds (e.g., whistling three consecutive times, clapping, etc.) and in this case the system can use a network of acoustic microphones across the building. Finally, as explained above, the system can fuse different sensors, by requiring the user to make a characteristic sound (e.g., whistling twice) while performing a gesture. By integrating multiple evidence, the system can increase significantly the accuracy of the system. 
     According to one or more embodiments, a gesture and location recognition system for controlling in-building equipment can be used a number of different ways by a user. For example, according to one embodiment, a user walks up to a building and is picked up by a camera. The user then waves their hands, gets a flashing light acknowledging the hand waving gesture was recognized. The system then calculates an elevator arrival time estimate as well as a user&#39;s elevator arrival time. Based on these calculations the system, places an elevator call accordingly. Then the cameras placed throughout the building that are part of the system track the user through the building (entrance lobby, halls, etc.) to elevator. The tracking can be used to update the user arrival estimate and confirm the user is traveling the correct direction toward the elevators. Once the user arrives at the elevators, the elevator car that was requested will be waiting or will also arrive for the user. 
     According to another embodiment, a user can approach a building, is picked up by a building camera, but can decide to make no signal. The system will not generate any in-building equipment control signals. The system may continue tracking the user or it may not. The user can then later be picked in a lobby at which point the user gestures indicating a desire to user, for example, an elevator. The elevator system can chime an acknowledging signal, and the system will then call an elevator car for the user. Another embodiment includes a user that leaves an office on the twentieth floor and a hall camera picks up the user. At this point the user makes a gesture, such as clapping their hands. The system detects this gesture and an elevator can be called with a calculated delay and an acknowledgment sent. Further, cameras throughout the building can continue to track the user until the user walks into the elevator. 
     Referring to  FIG. 2 , the system  101  may be a gesture-based interaction system that may be configured to interact with an elevator system  150 . The gesture-based interaction system  101  may include at least one localization device  130 , at least one sensor device  110 , at least one confirmation device  111  (i.e., also see visible signal  412  in  FIG. 4 ), and a signal processing unit  140 . The devices  110 ,  111 ,  130 , and  140  may communicate with each other and/or with the elevator controller  151  over pathways  102  that may be hardwired or wireless. The sensor device  110  may be an image sensor device, and/or may include a component  104  that may be at least one of a depth sensor, an e-field sensor and an optical camera. The sensor device  110  may further include an acoustic microphone  106 . The localization device  130  may be an integral part of the sensor device  110  or may be generally located proximate to the sensor device  110 . The signal processing unit  140  may also be an integral part of the sensor device  110  or may be remotely located from the device  110 . In one embodiment, the signal processing unit  140  may be an integral part of the elevator controller  151  and/or a retrofit part. The signal processing unit  140  may include a processor  108  and a storage medium  112 . The processor  108  may be a computer-based processor (e.g., microprocessor), and the storage medium  112  may be a computer writeable and readable storage medium. Examples of an elevator system  150  may include elevators, escalators, vehicles, rail systems, and others. 
     The component  104  of the sensor device  110  may include a field of view  114  (also see field of views  211 . 1 ,  211 . 2 ,  211 . 3  in  FIG. 3 ) configured, or generally located, to image or frame the user  160 , and/or record a scene, thus monitoring for a visual gesture (see examples of visual gesture  461  in  FIG. 4 , and visual gestures  561 ,  562  in  FIG. 5 ). The microphone  106  may be located to receive an audible gesture  116  from the user  160 . Examples of visual gestures may include a physical pointing by the user  160 , a number of fingers, a physical motion such as ‘writing’ a digit in the air, conventional sign language known to be used for the hearing impaired, specific signs designed for communicating with specific equipment, and others. These gestures, and others, may be recognized by Deep Learning, Deep Networks, Deep Convolutional Networks, Deep Recurrent Neural Networks, Deep Belief networks, Deep Boltzmann Machines, and so on. Examples of audible gestures  116  may generally include the spoken language, non-verbal vocalizations, a snapping of the fingers, a clap of the hands, and others. A plurality of gestures, which may be various visual gestures, various audible gestures, and/or a combination of both, may be associated with a plurality of transportation commands. Examples of transportation commands may include elevator commands, which may include up and/or down calls, destination floor number (e.g., car call or Compass call), a need to use the closest elevator (i.e. for users with reduced mobility), hold the doors open (i.e., for loading the elevator car with multiple items or cargo), and door open and/or door close. 
     More specifically, the user  160  may desire a specific action from the elevator system  150 , and to achieve this action, the user  160  may perform at least one gesture that is recognizable by the signal processing unit  140 . In operation, the component  104  (e.g., optical camera) of the sensor device  110  may monitor for the presence of a user  160  in the field of view  114 . The component  104  may take a sequential series of scenes (e.g., images where the component is an optical camera) of the user  160  and output the scenes as a captured signal (see arrow  118 ) to the processor  108  of the signal processing unit  140 . The microphone  106  may record or detect sounds and output an audible signal (see arrow  119 ) to the processor  108 . 
     The signal processing unit  140  may include recognition software  120  and pre-defined gesture data  122 , both being generally stored in the storage medium  112  of the signal processing unit  140 . The pre-defined gesture data  122  is generally a series of data groupings with each group associated with a specific gesture. It is contemplated and understood that the data  122  may be developed, at least in-part, through learning capability of the signal processing unit  140 . The processor  108  is configured to execute the recognition software  120  and retrieve the pre-defined gesture data  122  as needed to recognize a specific visual and/or audible gesture associated with the respective scene (e.g., image) and audible signals  118 ,  119 . 
     More specifically, the captured signal  118  is received and monitored by the processor  108  utilizing the recognition software  120  and the pre-defined gesture data  122 . In one embodiment, if the gesture is a visual gesture of a physical motion (e.g., movement of a hand downward), the processor  108  may be monitoring the captured signal  118  for a series of scenes taken over a prescribed time period. In another embodiment, if the visual gesture is simply a number of fingers being held upward, the processor  108  may monitor the captured signal  118  for a single recognizable scene, or for higher levels of recognition confidence, a series of substantially identical scenes (e.g., images). 
     Referring to  FIG. 11 , one example of a method of operating the gesture-based interaction system  101  to interact with an elevator system  150  is illustrated. At block  600 , once a user  160  is within the field of view  114  of the component  104  (e.g., optical camera), and/or is within range of the microphone  106 , the user  160  may initiate a wake-up gesture (i.e., visual and/or audible gesture) to begin an interaction. At block  602 , the processor  108  of the signal processing unit  140  acknowledges it is ready to receive a command gesture by the user  160  by, for example, sending a confirmation signal (see arrow  124  in  FIG. 2 ) to the confirmation device  111  that then initiates a confirmation event. Examples of the confirmation device  111  may be a local display or screen capable of displaying a message as the confirmation event, a device that turns on lights in the area as the confirmation event, an transportation call light adapted to illuminate as the confirmation event, an audio confirmation, and other devices. 
     At block  604 , the user  160  performs a command gesture (e.g., up or down call, or a destination floor number), through, for example, a visual gesture. At block  606 , the component  104  (e.g., optical camera) of the sensor device  110  captures the command gesture and, via the captured signal  118 , the command gesture is sent to the processor  108  of the signal processing unit  140 . 
     At block  608 , the processor  108  utilizing the recognition software  120  and the predefined gesture data  122 , attempts to recognize the gesture. At block  610 , the processor  108  of the signal processing unit  140  sends a command interpretation signal (see arrow  126  in  FIG. 2 ) to the confirmation device  111  (e.g., a display) which may request gesture confirmation from the user  160 . It is contemplated and understood that the confirmation device  111  for the wake-up confirmation may be the same device, or may be a different and separate device from the display that receives the command interpretation signal  126 . 
     At block  612 , if the signal processing unit  140  has interpreted the command gesture correctly, the user  160  may perform a confirmation gesture to confirm. At block  614 , if the signal processing unit  140  did not interpret the command gesture correctly, the user  160  may re-perform the command gesture or perform another gesture. It is contemplated and understood that the gesture-based interaction system  101  may be combined with other forms of authentication for secure floor access control and VIP service calls. 
     At block  616  and after the user performs a final gesture indicating confirmation that the signal processing unit correctly interpreted the previous command gesture, the signal processing unit  140  may output a command signal  128 , associated with the previous command gesture, to the elevator controller  151  of the elevator system  151 . 
     It is contemplated and understood that if the confirmation gesture is not received, and/or if the user explicitly wants to gesture that the command was not properly understood, the system may first, time out, then may provide a ready-to-receive-command signal that signifies the system is ready to receive another attempt at a user gesture. The user may know that the system remains awake because the system may indicate the same acknowledging receipt state after a wake-up gesture. However, after a longer timeout, if the user does not appear to make any further gestures, the system may return to a non-awake state. It is further contemplated that while waiting of for a confirmation gesture, the system may also recognize a gesture that signifies the user&#39;s attempt to correct the system interpretation of a previous gesture. When the system receives this correcting gesture, the system may immediately turn off the previous, and wrongly interpreted, command interpretation signal and provides a ready-to-receive-command signal once again. 
     Advantageously, embodiments described herein provide a system that allows users to call the equipment-based system (e.g., elevator system) from distant parts of the building, contrary to current systems that are designed to be used inside or close to the elevator. One or more embodiments disclosed here also allow one to call the elevator without carrying any extra equipment, just by gestures, contrary to systems which require a mobile phone or other wearable or carried devices. One or more embodiments disclosed here also do not require the installation of hardware. One or more embodiments are able to leverage an existing network of sensors (e.g., CCTV optical cameras or depth sensors). Another benefit of one or more embodiments can include seamless remote summoning of an elevator without requiring users to have specific equipment (mobile phones, RFID tags, or other device) with automatic updating of a request. The tracking may not need additional equipment if an appropriate video security system is already installed. 
     While the present disclosure has been described in detail in connection with only a limited number of embodiments, it should be readily understood that the present disclosure is not limited to such disclosed embodiments. Rather, the present disclosure can be modified to incorporate any number of variations, alterations, substitutions, combinations, sub-combinations, or equivalent arrangements not heretofore described, but which are commensurate with the scope of the present disclosure. Additionally, while various embodiments of the present disclosure have been described, it is to be understood that aspects of the present disclosure may include only some of the described embodiments. 
     The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. 
     The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the disclosure. The embodiments were chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand various embodiments with various modifications as are suited to the particular use contemplated. 
     The present embodiments may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure. 
     Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions. 
     The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions. 
     The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. 
     Accordingly, the present disclosure is not to be seen as limited by the foregoing description, but is only limited by the scope of the appended claims.