Patent Description:
Two-dimensional (2D) and three-dimensional (3D) sensors may be used in an effort to capture passenger behaviors. Both types of sensors are intrinsically flawed. For example, 2D sensors that operate on the basis of color or intensity information may be unable to distinguish two passengers wearing similar colored clothing or may be unable to discriminate between a passenger and an object in the background of similar color. 3D sensors that provide depth information may be unable to generate an estimate of depth in a so-called "shadow region" due to a difference in distance between an emitter/illuminator (e.g., an infrared (IR) laser diode) and a receiver/sensor (e.g., an IR sensitive camera). What is needed is a device and method of sufficient resolution and accuracy to allow explicit and implicit gesture-based control of a conveyance. An explicit gesture is one intentionally made by a passenger intended for communication to the conveyance controller. An implicit gesture is where the presence or behavior of the passenger is deduced by the conveyance controller without explicit action on the passenger's part. This need may be economically, accurately, and conveniently realized by a particular gesture recognition system utilizing distance (called hereafter the "depth").

<CIT> discloses a method for controlling a transportation system, by detecting that at least one user is moving into a target area, detecting a gesture of the user, and outputting a corresponding control command to the control module.

According to the invention there is provided a method according to claim <NUM>.

According to the invention there is provided a system according to claim <NUM>.

Additional embodiments are described below.

Referring to <FIG>, an exemplary computing system <NUM> is shown. The system <NUM> is shown as including a memory <NUM>. The memory <NUM> may store executable instructions. The executable instructions may be stored or organized in any manner and at any level of abstraction, such as in connection with one or more applications, processes, routines, procedures, methods, functions, etc. As an example, at least a portion of the instructions are shown in <FIG> as being associated with a first program 104a and a second program 104b.

The instructions stored in the memory <NUM> may be executed by one or more processors, such as a processor <NUM>. The processor <NUM> may be coupled to one or more input/output (I/O) devices <NUM>. In some embodiments, the I/O device(s) <NUM> may include one or more of a keyboard or keypad, a touchscreen or touch panel, a display screen, a microphone, a speaker, a mouse, a button, a remote control, a joystick, a printer, a telephone or mobile device (e.g., a smartphone), a sensor, etc. The I/O device(s) <NUM> may be configured to provide an interface to allow a user to interact with the system <NUM>.

The memory <NUM> may store data <NUM>. The data <NUM> may include data provided by one or more sensors, such as a 2D or 3D sensor. The data is processed by the processor <NUM> to obtain depth information for intelligent crowd sensing for elevator control. The data is associated with a depth stream that is combined (e.g., fused) with a video stream for purposes of combining depth and color information.

The system <NUM> is illustrative. In some embodiments, one or more of the entities may be optional. In some embodiments, additional entities not shown may be included. For example, in some embodiments the system <NUM> may be associated with one or more networks. In some embodiments, the entities may be arranged or organized in a manner different from what is shown in <FIG>.

Turning now to <FIG>, a block diagram of an exemplary system <NUM> in accordance with one or more embodiments is shown. The system <NUM> includes one or more sensors, such as a sensor <NUM>. The sensor <NUM> is used to provide a structured-light based device for purposes of obtaining depth information.

The sensor <NUM> includes an emitter <NUM> and a receiver <NUM>. The emitter <NUM> is configured to project a pattern of electromagnetic radiation. The pattern comprises an array of dots, lines, shapes. The emitter <NUM> is configured to project a pattern of electromagnetic radiation in a non-visible infrared frequency range of near infrared or far infrared. The sensor <NUM> is configured to detect the pattern using a receiver <NUM>. The receiver <NUM> may include a complementary metal-oxide-semiconductor (CMOS) image sensor or other electromagnetic radiation sensor with a corresponding filter.

The pattern is projected onto a scene <NUM> that that includes one or more objects, such as objects <NUM>-<NUM>. The objects <NUM>-<NUM> may be of various sizes or dimensions, of various colors, reflectances, light intensities, etc. A position of one or more of the objects <NUM>-<NUM> may change over time. The pattern received by the receiver <NUM> may change size and position based on the relative position of the objects <NUM>-<NUM> relative to the emitter <NUM>. The pattern may be unique per position in order to allow the receiver <NUM> to recognize each point in the pattern to produce a depth stream containing depth information. A pseudo random pattern may be used in some embodiments. In other exemplary embodiments, the depth information is obtained using a time-of-flight camera, a stereo camera, laser scanning, light detection and ranging (LIDAR), or phased array radar.

Sensor <NUM> also includes an imager <NUM> to generate at least one video stream of the scene <NUM>. The video stream may be obtained from a visible color, UV, or IR camera. Multiple sensors may be used to cover a large area, such as a hallway or a whole building. It is understood that the imager <NUM> need not be co-located with the emitter <NUM> and receiver <NUM>. For example, imager <NUM> may correspond to a camera focused on the scene, such as a security camera.

In exemplary embodiments, the depth stream and the video stream may be fused. Fusing the depth stream and the video stream involves registering or aligning the two streams, and then processing the fused stream jointly. Alternatively, the depth stream and the video stream may be processed independently, and the results of the processing combined at a decision or application level.

Turning now to <FIG>, an environment <NUM> is shown. The environment <NUM> may be associated with one or more of the systems, components, or devices described herein, such as the systems <NUM> and <NUM>. A gesture is recognized for control of a conveyance device (e.g., an elevator). The gesture may be recognized by the gesture recognition device <NUM>.

A gesture recognition device <NUM> may include one or more sensors <NUM>. Gesture recognition device <NUM> may also include system <NUM>, that executes a process to recognize gestures. System <NUM> may be located remotely from sensors <NUM>, and may be part of a larger control system, such as conveyance device control system.

Gesture recognition device <NUM> may be configured to detect gestures made by one or more passengers of the conveyance device. For example, a "thumbs-up" gesture <NUM> may be used to replace or enhance the operation of an 'up' button <NUM> that may commonly be found in the hallway outside of an elevator or elevator car. Similarly, a "thumbs-down" gesture <NUM> may be used to replace or enhance the operation of a 'down' button <NUM>. The gesture recognition device <NUM> detects a gesture based on a combination of a depth stream and a video stream.

While the environment <NUM> is shown in connection with gestures for selecting a direction of travel, other types of commands or controls may be provided. For example, a passenger may hold up a single finger to indicate that she wants to go one floor up from the floor on which she is currently located. Conversely, if the passenger holds two fingers downward that may signify that the passenger wants to go down two floors from the floor on which she is currently located. Of course, other gestures may be used to provide floor numbers in absolute terms (e.g., go to floor #<NUM>).

An analysis of passenger gestures may be based on one or more techniques, such as dictionary learning, support vector machines, Bayesian classifiers, etc. The techniques apply to a combination of depth information and video information, including color information.

Turning now to <FIG>, a method <NUM> is shown. The method <NUM> may be executed in connection with one or more systems, components, or devices, such as those described herein (e.g., the system <NUM>, the system <NUM>, the gesture recognition device <NUM>, etc.). The method <NUM> is used to detect a gesture for purposes of controlling a conveyance device.

In block <NUM>, a depth stream is generated by receiver <NUM> and in block <NUM> a video stream is generated from imager <NUM>. In block <NUM>, the depth stream and the video stream are processed, for example, by system <NUM>. Block <NUM> includes processing the depth stream and video stream to derive depth information and video information. The depth stream and the video stream may be aligned and then processed, or the depth stream and the video stream may be independently processed. The processing of block <NUM> may include a comparison between the depth information and the video information with a database or library of gestures.

In block <NUM>, a determination may be made whether the processing of block <NUM> indicates that a gesture has been recognized. If so, flow may proceed to block <NUM>. Otherwise, if a gesture is not recognized, flow may proceed to block <NUM>.

In block <NUM>, the conveyance device is controlled in accordance with the gesture recognized in block <NUM>.

The method <NUM> is illustrative. In some embodiments, one or more blocks or operations (or a portion thereof) may be optional. In some embodiments, the blocks may execute in an order or sequence different from what is shown in <FIG>. In some embodiments, additional blocks not shown may be included. For example, in some embodiments, the recognition of the gesture in block <NUM> may include recognizing a series or sequence of gestures before flow proceeds to block <NUM>. In some embodiments, a passenger providing a gesture may receive feedback from the conveyance device as an indication or confirmation that one or more gestures are recognized. Such feedback may be used to distinguish between intended gestures relative to inadvertent gestures.

In some instances, current technologies for 3D or depth sensing may be inadequate for sensing gestures in connection with the control of an elevator. Sensing requirements for elevator control may include the need to accurately sense gestures over a wide field of view and over a sufficient range to encompass, e.g., an entire lobby. For example, sensors for elevator control may need to detect gestures from <NUM> meters (m) to <NUM> and at least a <NUM>° field of view, with sufficient accuracy to be able to classify small gestures (e.g., greater than <NUM> pixels spatial resolution corresponding to a person's hand with <NUM> depth measurement accuracy).

Depth sensing may be performed using one or more technical approaches, such as triangularization (e.g., stereo, structured light) and interferometry (e.g., scanning LIDAR, flash LIDAR, time-of-flight camera). These sensors (and stereo cameras) may depend on disparity as shown in <FIG> uses substantially the same terminology and a similar analysis to <NPL>. A structured light projector 'L' may be at a distance (or aperture) 'a' from a camera 'C'. An object plane, at distance 'Zk', may be at a different depth than a reference plane at a distance `zo'. A beam of the projected light may intersect the object plane at a position 'k' and the reference plane at a position 'o'. Positions 'o' and 'k', separated by a distance 'A' in the object plane, may be imaged or projected onto an n-pixel sensor with a focal length `f and may be separated by a distance 'b' in the image plane.

In accordance with the geometry associated with <FIG> described above, and by similar triangles, equations #<NUM> and #<NUM> may be constructed as:
<MAT>
<MAT>.

Substituting equation #<NUM> into equation #<NUM> will yield equation #<NUM> as:
<MAT>.

Taking the derivative of equation #<NUM> will yield equation #<NUM> as:
<MAT>.

Equation #<NUM> illustrates that the change in the size of the projected image, 'b', may be linearly related to the aperture 'a' for constant f, z<NUM>, and zk.

The projected image may be indistinct on the image plane if it subtends less than one pixel, as provided in equation #<NUM>:<MAT>.

Equation #<NUM> shows that the minimum detectable distance difference (taken in this example to be one pixel) may be related to the aperture 'a' and the number of pixels 'n'.

Current sensors may have a range resolution of approximately <NUM> centimeter (cm) at a range of <NUM>. The cross-range and range resolutions may decrease quadratically with range. Therefore, at <NUM>, current sensors might have a range resolution of greater than <NUM><NUM>, which may be ineffective in distinguishing anything but the largest of gestures.

Current sensors at <NUM> and with <NUM> pixels across a <NUM>° field of view, may have approximately <NUM>/pixel spatial resolution horizontally, and <NUM>/pixel vertically. For a small person's hand (approximately <NUM> millimeters (mm) by <NUM>), current sensors may have approximately 22x32 pixels on target. However, at <NUM>, current sensors may have approximately <NUM>/pixel or <NUM>. <NUM> pixels on target. Such a low amount of pixels on target may be insufficient for accurate gesture classification.

Current sensors cannot be modified to achieve the requirements by simply increasing the aperture 'a' because this would result in a non-overlapping of the projected pattern and infrared camera field of view close to the sensor. The non-overlapping would result in an inability to detect gestures when close to the sensor. As it is, current sensors cannot detect depth at a distance of less than <NUM>.

Current sensors cannot be modified to achieve the requirements by simply increasing the focal length 'f since a longer focal length may result in a shallower depth of field. A shallower depth of field may result in a loss of sharp focus and a resulting inability to detect and classify gestures.

Current sensors or commercially available sensors may be modified relative to an off-the-shelf version by increasing the number of pixels 'n' (see equation <NUM> above). This modification is feasible, given a low sensor resolution and the availability of higher resolution imaging chips.

Another approach is to arrange an array of triangulation sensors, each of which is individually insufficient to meet the desired spatial resolution while covering a particular field of view. Within the array, each sensor may cover a different field of view such that, collectively, the array covers the particular field of view with adequate resolution.

In some embodiments, elevator control gesture recognition may be based on a static 2D or 3D signature from a 2D or 3D sensing device, or a dynamic 2D/3D signature manifested over a period of time. The fusion of 2D and 3D information may be useful as a combined signature. In long-range imaging, a 3D sensor alone might not have the desired resolution for recognition, and in this case 2D information extracted from images may be complementary and useful for gesture recognition. In short-range and mid-range imaging, both 2D (appearance) and 3D (depth) information may be helpful in segmentation and detection of a gesture, and in recognition of the gestures based on combined 2D and 3D features.

In some embodiments, behaviors of passengers of an elevator may be monitored, potentially without the passengers even knowing that such monitoring is taking place. This may be particularly useful for security applications such as detecting vandalism or violence. For example, passenger behavior or states, such as presence, direction of motion, speed of motion, etc., may be monitored. The monitoring may be performed using one or more sensors, such as a 2D camera/receiver, a passive IR device, and a 3D sensor.

In some embodiments, gestures may be monitored or detected at substantially the same time as passenger behaviors/states. Thus, any processing for gesture recognition/detection and passenger behavior/state recognition/detection may occur in parallel. Alternatively, gestures may be monitored or detected independent of, or at a time that is different from, the monitoring or detection of the passenger behaviors/states.

In terms of the algorithms that may be executed or performed, gesture recognition may be substantially similar to passenger behavior/state recognition, at least in the sense that gesture recognition and behavior/state recognition may rely on a detection of an object or thing. However, gesture recognition may require a larger number of data points or samples and may need to employ a more refined model, database, or library relative to behavior/state recognition.

While some of the examples described herein related to elevators, aspects of this disclosure may be applied in connection with other types of conveyance devices, such as a dumbwaiter, an escalator, a moving sidewalk, a wheelchair lift, etc..

Embodiments may be implemented using one or more technologies. In some embodiments, an apparatus or system may include one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus or system to perform one or more methodological acts as described herein. Various mechanical components known to those of skill in the art may be used in some embodiments, within the scope of the appended claims.

Claim 1:
A method (<NUM>) comprising:
generating a depth stream (<NUM>) from a scene (<NUM>) associated with a conveyance device;
processing, by a computing device (<NUM>), the depth stream to obtain depth information;
generating a video stream (<NUM>) from the scene (<NUM>) from an imager (<NUM>);
characterized by:
projecting, by an emitter (<NUM>), a pattern of electromagnetic radiation in a non-visible frequency range of near or far infrared onto the scene (<NUM>) comprising a plurality of objects (<NUM>, <NUM>, <NUM>), wherein the pattern comprises an array of dots, lines or shapes, and wherein the depth stream is generated, by a receiver (<NUM>), by detecting the emitted pattern;
processing (<NUM>), by the computing device (<NUM>), the video stream to obtain color information,
recognizing (<NUM>) a gesture (<NUM>, <NUM>) based on the depth information and the color information; and
controlling (<NUM>) the conveyance device based on the gesture (<NUM>, <NUM>).