Patent Description:
A passenger conveyor (such as an escalators or a moving walkway) is increasingly widely applied in public places such as subways, shopping malls, airports, and the like, and the running safety thereof becomes increasingly important.

During running of a passenger conveyor, abnormalities such as reverse running, speed anomaly, Step missing, Landing Plate displacement or missing may occur, causing major safety accidents; or during a repair operation of the passenger conveyor, no Barrier is disposed due to a non-standard operation while the passenger conveyor is in a repair running state, and in this case, if a passenger enters the passenger conveyor, a major safety accident also occurs easily.

<CIT> describes a method and device for detecting abnormality of an escalator step.

According to a first aspect of the present invention, a monitoring system of a passenger conveyor is provided in accordance with claim <NUM>
One or more examples of such a system are provided by claims <NUM> to <NUM>.

According to another aspect of the present invention, a monitoring method of a passenger conveyor is provided in accordance with claim <NUM>. One or more examples of such a method are provided by claims <NUM> to <NUM>.

According to still another aspect of the present invention, a passenger conveying system in accordance with claim <NUM> is provided, including a passenger conveyor and the foregoing monitoring system.

The foregoing features and operations of the present invention will become more obvious according to the following description and the accompanying drawings.

The following detailed description with reference to the accompanying drawings will make the foregoing and other objectives and advantages of the present invention more complete and clearer, wherein identical or similar elements are represented using identical reference signs.

The present invention is now described more completely with reference to the accompanying drawings. Exemplary embodiments of the present invention are illustrated in the accompanying drawings. However, the present invention may be implemented in lots of different forms, and should not be understood as being limited to the embodiments described herein. On the contrary, the embodiments are provided to make the disclosure thorough and complete, and fully convey the concept of the present invention to those skilled in the art. In the accompanying drawings, identical reference signs represent identical elements or parts, and therefore, descriptions thereof are omitted.

Some block diagrams in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in a software form, or implemented in one or more hardware modules or integrated circuits, or in different processing apparatuses and/or microcontroller apparatuses.

In the present invention, passenger conveyors include Escalators and Moving Walkways. In the embodiments illustrated below, an escalator is used as an example to describe the monitoring system and monitoring method of the embodiments of the present invention in detail. However, it should be understood that, the monitoring system and monitoring method for the escalator in the following embodiments may also be analogically applied to moving walkways. Adaptive improvements or the like that may need to be performed can be obtained by those skilled in the art with the teachings of the embodiments of the present invention.

It should be noted that, in the present invention, a "normal state" of a monitored object of the passenger conveyor refers to a working condition state in which the monitored object at least does not bring potential safety hazards to passengers; on the contrary, an "abnormal state" refers to a working condition state in which the monitored object at least may bring potential safety hazards to passengers, for example, a working condition state that does not conform to related standards or regulations. Those skilled in the art may define the "normal state" and "abnormal state" in advance according to a specific application environment and a specific monitored object.

<FIG> is a schematic structural diagram of a monitoring system of a passenger conveyor according to a third embodiment of the present invention. The monitoring system according to the embodiment shown in <FIG> may be used to detect whether a state of a Barrier <NUM> of an escalator <NUM> is normal, for example, whether the barrier is missing, in a maintenance and repair working condition. If the barrier <NUM> is missing or displaced at an improper position, a passenger may enter the escalator <NUM> in the maintenance and repair working condition, which easily causes a major safety accident, and this needs to be completely avoided.

The monitoring system in the embodiment shown in <FIG> specifically includes a sensing apparatus <NUM> and a processing apparatus <NUM> coupled to the sensing apparatus <NUM>, and the escalator <NUM> includes a passenger conveyor controller <NUM>, a driving part <NUM> such as a motor, and an alarm unit <NUM>.

The sensing apparatus <NUM> is specifically an Imaging sensor or a Depth sensing sensor, or a combination thereof. According to a specific requirement and a monitoring range of a sensor, the escalator <NUM> may be provided with one or more sensing apparatuses <NUM>, for example, <NUM><NUM> to <NUM>n, N being an integer greater than or equal to <NUM>. The sensing apparatus <NUM> is mounted in such a manner that it can clearly and accurately acquire a monitored object of the escalator <NUM>, and the specific mounting manner and mounting position thereof are not limited. In this embodiment, there are two sensing apparatuses <NUM>, which are disposed approximately above entry/exit regions (<NUM> and <NUM>) at two ends of the escalator <NUM> respectively, to sense barriers <NUM> in the entry/exit regions (<NUM> and <NUM>). It should be understood that, to accurately sense the barrier <NUM>, an imaging sensor or a depth sensing sensor of a corresponding type may be selected according to a specific application environment, and even a corresponding lighting lamp or the like may be configured above the position where the barrier <NUM> should be disposed.

The sensing apparatus <NUM> may be an imaging sensor of a 2D image sensor of various types. It should be understood that, any image sensor capable of capturing an image frame including pixel brightness information can be applied herein. Certainly, an image sensor capable of capturing an image frame including pixel brightness information and color information (such as RGB information) may also be applied herein.

The sensing apparatus <NUM> may be any 1D, 2D, or 3D depth sensor or a combination thereof. Such a sensor is operable in an optical, electromagnetic, or acoustic spectrum that can generate a depth map (which is also known as a point cloud or an occupancy grid) of a corresponding dimension. Various depth sensing sensor technologies and devices include, but are not limited to: structured light measurement, phase shift measurement, time-of-flight measurement, a stereo triangulation device, an optical triangulation device plate, a light field camera, a coded aperture camera, a computational imaging technology, simultaneous localization and map-building (SLAM), an imaging radar, an imaging sonar, an echolocation device, a scanning LIDAR, a flashing LIDAR, a passive infrared (PIR) sensor, and a small focal plane array (FPA), or a combination including at least one of the foregoing. Different technologies may include active (transmitting and receiving signals) or passive (only receiving signals) technologies and are operable in a band of an electromagnetic or acoustic spectrum (such as visual and infrared). Depth sensing can achieve a particular advantage over conventional 2D imaging. The use of infrared sensing can achieve a particular benefit over visible spectrum imaging such that alternatively or additionally, the sensor can be an infrared sensor having one or more pixel spatial resolutions, for example, a passive infrared (PIR) sensor or a small IR focal plane array (FPA).

It should be noted that, there may be qualitative and quantitative differences between a 2D imaging sensor (such as a conventional monitoring camera) and the 1D, 2D or 3D depth sensing sensor to the extent that the depth sensing provides numerous advantages. In 2D imaging, reflected color (a mixture of wavelengths) from the first object in each radial direction of the imager is captured. A 2D image, then, may include a combined spectrum of source lighting and a spectral reflectivity of an object in a scene. The 2D image may be roughly interpreted as a picture by a person. In the 1D, 2D or 3D depth sensing sensor, there is no color (spectrum) information. More specifically, a distance (depth, range) to a first reflection object in a radial direction (1D) or directions (2D, 3D) from the sensor is captured. The 1D, 2D and 3D technologies may have inherent maximum detectable range limits and may have relatively lower spatial resolution than typical 2D imagers. In terms of relative immunity to ambient lighting problems, compared to conventional 2D imaging, the use of 1D, 2D, or 3D depth sensing may advantageously provide improved operations, and better separation and better privacy protection of shielded objects. The use of infrared sensing can achieve a particular benefit over visible spectrum imaging. For example, it is possible that a 2D image cannot be converted into a depth map, and a depth map does not have a capability of being converted into a 2D image (for example, artificial allocation of continuous colors or brightness to continuous depths may cause a person to roughly interpret a depth map in a manner somewhat akin to how a person sees a 2D image, while the depth map is not an image in a conventional sense).

When the sensing apparatus <NUM> is specifically a combination of an imaging sensor and a depth sensing sensor, the sensing apparatus <NUM> may be an RGB-D sensor, which may simultaneously acquire RGB information and depth (D) information.

During inspection and repair, the sensing apparatus <NUM> is triggered to sense a region where the barrier <NUM> of the escalator <NUM> should be located and obtain multiple data frames in real time; if sequence frames are acquired through sensing by the imaging sensor, the sequence frames are multiple image frames, and each pixel therein has, for example, corresponding brightness information and color information; if the sequence frames are acquired through sensing by the depth sensing sensor, the sequence frames are multiple depth maps, and each pixel or occupancy grid therein also has a corresponding depth dimension (reflecting depth information).

If it is necessary to monitor the barrier <NUM> all the time in the inspection and repair operation condition, multiple sensing apparatuses <NUM><NUM> to <NUM>n all work simultaneously to acquire corresponding sequence frames, and each frame is transmitted to the processing apparatus <NUM>. The processing apparatus <NUM> is responsible for performing data processing on each frame, and finally obtains information indicating whether the barrier <NUM> of the escalator <NUM> is in a normal state, for example, determine whether the barrier is missing.

Specifically, the processing apparatus <NUM> is configured to include a background acquisition module <NUM> and a foreground detection module <NUM>. In a first case, the background acquisition module <NUM> may acquire a background model by using one or more 2D images or 3D depth maps when the barrier <NUM> is in a normal state, and in a second case, the background acquisition module <NUM> may acquire a background model by using one or more 2D images or 3D depth maps when the barrier <NUM> is not disposed in a region where the barrier <NUM> needs to be disposed. Therefore, in the first case, the background acquisition module <NUM> acquires a first background model by learning a data frame that is acquired when the barrier <NUM> is correctly disposed; in the second case, the background acquisition module <NUM> acquires a second background model by learning a data frame that is acquired when no barrier <NUM> is disposed. The first background model and the second background model may be established before maintenance and repair of the escalator <NUM>, and the processing apparatus <NUM> is initialized during this period to obtain the background models. The background models may be established through learning by using, but not limited to, a Gaussian Mixture Model, a Code Book Model, Robust Principle Components Analysis (RPCA), and the like. A background model obtained by learning image frames acquired by the imaging sensor is a typical brightness background model or chromaticity background model; a background model obtained by learning frames acquired by the depth sensing sensor is a typical depth background model.

It should be understood that, at the subsequent barrier monitoring stage, the background model may be adaptively updated. When an application scenario, a sensor type, or a barrier type changes, a corresponding background model may be acquired again by means of learning at the initialization stage.

The foreground detection module <NUM> is configured to subtract, from the background model, a data frame acquired in real time to obtain a foreground object (taking the second background model as an example), or subtract the background model from a data frame acquired in real time to obtain a foreground object (taking the first background model as an example), for example, the foreground object is obtained by using a differential method. Taking the second background model as an example, if the barrier <NUM> is missing, a portion of the data frame corresponding to the barrier <NUM> is compared with a corresponding portion of the background model, and the obtained foreground object also includes a feature reflecting that the barrier <NUM> is missing.

In an embodiment, the foreground detection module <NUM> may eliminate noise of the foreground object by using some filtering technologies. For example, noise is eliminated by using erosion and dilation image processing technologies, so that the foreground object is obtained more accurately. It should be noted that, the filtering may include convolution about a spatial, temporal, or spatial-temporal kernel.

The processing apparatus <NUM> further includes a foreground feature extraction module <NUM>. The foreground feature extraction module <NUM> extracts a corresponding foreground feature from the foreground object. In order to monitor the barrier of the escalator <NUM>, the extracted foreground feature includes features such as the shape and color of the foreground object (the barrier <NUM> is generally yellow, which is significantly different from colors of other objects), and may even include information such as the size and/or position. Taking the data frame acquired by the imaging sensor as an example, the color information is embodied by the chromaticity of pixels of the foreground object, and the shape, size and position information is embodied by brightness value changes of the pixels in the foreground object. Taking the data frame acquired by the depth sensing sensor as an example, the shape, size and position information is embodied by depth value changes of occupancy grids in the foreground object.

Further, the processing apparatus <NUM> further includes a barrier working condition judgment module <NUM>. The working condition judgment module <NUM> judges whether the barrier <NUM> is in a normal state based on the foreground feature. Specifically, comparison judgment may be performed on the foreground feature by using the background model. For example, by comparing the shape, size and position features of the foreground object with shape, size and position features about the barrier in the background model, it is judged whether the barrier is missing; or by comparing the shape, size, position and color features of the barrier with shape, size, position and color features about the barrier in the background model, it is judged whether the barrier is missing. It should be noted that, the shape, size and color feature information about the barrier in the background model may be implemented in the background acquisition module <NUM>.

By taking the background model being the first background model as an example, assuming that the foreground feature is mainly a foreground feature of a foreground object of maintenance personnel, and by comparing the foreground feature with the feature information about the barrier <NUM> in the first background model, it can be judged that the foreground feature is unrelated to the barrier <NUM>. In this case, the barrier working condition judgment module <NUM> can easily judge that the barrier <NUM> is in a normal state. Assuming that the foreground feature is mainly a foreground feature of a foreground object of a barrier, by comparing the foreground feature with the feature information about the barrier <NUM> in the first background model, it can be judged that the foreground feature is the same or substantially the same as the barrier <NUM>, and the barrier working condition judgment module <NUM> can easily judge that the barrier <NUM> is in a missing state.

By taking the background model being the second background model as an example, assuming that the foreground feature is a foreground feature of the foreground object of the barrier <NUM>, by comparing the foreground feature with the feature information about the barrier <NUM> in the second background model, it can be judged that the foreground feature is related to the barrier <NUM>, and it is further judged that a position feature in the foreground feature is substantially the same as the position information about the barrier <NUM> in the second background model; in this case, the barrier working condition judgment module <NUM> can easily judge that the barrier <NUM> is in a normal state. Assuming that the foreground feature is mainly a foreground feature of a foreground object of maintenance personnel, by comparing the foreground feature with the feature information about the barrier <NUM> in the second background model, it can be judged that the foreground feature is unrelated to the barrier <NUM>, and the barrier working condition judgment module <NUM> can easily judge that the barrier <NUM> is in a missing state.

Further, taking barrier missing in 2D image processing as an example, a current 2D image frame is compared with the first background model to obtain the foreground object of the barrier <NUM>, and based on the 2D image of the object, features such as the position, color and shape of the object are also extracted, which are further compared with the background model, for example, features such as shapes and colors are compared. Therefore, it can be judged that the barrier is missing, thus directly judging that the barrier <NUM> is in an abnormal state.

Further, taking barrier missing in depth map processing as an example, a current 3D depth map frame is compared with the first background model (a model based on depth information), to obtain the foreground object of the barrier <NUM>. Features such as the position and 3D shape of the object are also extracted, which are further compared with the background model, for example, features such as 3D shapes corresponding to the same position are compared. Therefore, it can be judged that a particular position lacks a barrier, thus directly judging that the barrier <NUM> is in an abnormal state.

Further, the barrier working condition judgment module <NUM> may be configured to determine, only when judgment results based on multiple (such as two) continuous data frames are that a same barrier <NUM> is missing, that the current barrier <NUM> is in the abnormal state, which helps improve the judgment accuracy.

In still another embodiment, by taking the second background model as an example, when it is determined that the foreground object of the barrier is included, whether the barrier <NUM> is in a normal state in which the barrier is at a corresponding correct position is judged based on the position feature of the foreground object of the barrier, i.e., whether the barrier <NUM> is placed at a proper position is judged. The position feature of the foreground object of the barrier <NUM> is obtained by the foreground feature extraction module <NUM>. When the data frame is obtained through sensing by the depth sensing sensor, the position feature of the foreground object of the barrier <NUM> is a 3D position feature. Specifically, the foreground feature extraction module <NUM> may extract 3D position features corresponding to multiple corner points (such as points at four corners) of the foreground object of the barrier <NUM> and/or a 3D position feature corresponding to the center of the foreground object of the barrier <NUM>. With these position features, whether the barrier <NUM> is placed at a proper position can be judged more accurately.

When the barrier working condition judgment module <NUM> in the processing apparatus <NUM> in the foregoing embodiment determines that the monitored barrier <NUM> is in an abnormal state (for example, the barrier is missing), a corresponding signal may be sent to the passenger conveyor controller <NUM> of the escalator <NUM>, so that corresponding measures are taken. For example, the controller <NUM> further sends a signal to a braking part (not shown in the figure) to perform braking. The processing apparatus <NUM> may further send a signal to an alarm unit <NUM> mounted above the escalator <NUM>, to remind the passenger to watch out, and remind the maintenance personnel to arrange the barrier <NUM>, for example, a message such as "the barrier is missing; do not take the escalator; please arrange a barrier immediately" is broadcast. Certainly, the processing apparatus <NUM> may further send a signal to an elevator maintenance center of a building, to prompt that on-site processing needs to be performed in time. Measures taken when it is found that the barrier <NUM> of the escalator <NUM> is in an abnormal state are not limited.

The monitoring system in the embodiment shown in <FIG> can automatically monitor the state of the barrier <NUM> during inspection and maintenance of the escalator <NUM>, the monitoring is accurate and abnormal conditions of the barrier can be found in time, helping prevent safety accidents.

A method procedure of barrier monitoring based on the monitoring system in the embodiment shown in <FIG> is illustrated according to <FIG>. A working principle of the monitoring system according to the embodiment of the present invention is further described with reference to <FIG> and <FIG>.

First, in step S31, entry/exit regions (<NUM> and <NUM>) of a passenger conveyor is sensed by using an imaging sensor and/or a depth sensing sensor to acquire a data frame. During acquisition of a background model through learning, a data frame corresponding to the first background model is acquired through sensing when the barrier <NUM> is correctly arranged, and a data frame corresponding to the second background model is acquired through sensing when the barrier <NUM> is not arranged; in other cases, the data frame is acquired at any time in a maintenance and repair working condition, for example, <NUM> data frames are acquired per second, and a predetermined number of data frames are acquired at predetermined intervals or acquired continuously, for use in subsequent analysis.

Further, in step S32, taking the second background model as an example, the background model is acquired based on a data frame sensed when the barrier is not arranged. This step is accomplished in the background acquisition module <NUM>, and may be implemented at the initialization stage of the system. If both the imaging sensor and the depth sensing sensor are used, background models are acquired by learning separately based on data frames acquired by the imaging sensor and the depth sensing sensor respectively.

Further, in step S33, a data frame sensed in real time is compared with the background model to obtain a foreground object. This step is accomplished in the foreground detection module <NUM>, and the foreground object may be sent to the barrier working condition judgment module <NUM> for analysis.

Further, in step S34, a corresponding foreground feature is extracted from the foreground object. This step is accomplished in the foreground feature extraction module <NUM>, and the extracted foreground feature includes the shape and position of the foreground object, and even further includes information such as color and/or size.

Further, in step S35, it is judged whether it is in a normal state in which the barrier has been arranged, and if the judgment result is "no", it indicates that the barrier <NUM> is in an abnormal state currently, and step S36 is performed, to trigger alarming and escalator braking operations. Step S35 and step S36 are accomplished in the barrier working condition judgment module <NUM>. The specific judgment method is already disclosed in the description about the barrier working condition judgment module <NUM>.

So far, the barrier monitoring process in the maintenance and repair working condition in the foregoing embodiment has basically ended, and this process may be repeated and continuously performed, to continuously monitor the barrier of the escalator <NUM>. It should be noted that, if both the imaging sensor and the depth sensing sensor are used, the imaging sensor and the depth sensing sensor may separately acquire respective data frames, and in steps S32 to S35, processing is separately performed on the respective data frames; and when the judgment result of the data frame corresponding to either of the imaging sensor and the depth sensing sensor is "no" in step S35, step S36 is performed. In another alternative embodiment, the process of steps S22 to S25 may be performed on data from the imaging sensor and the depth sensing sensor respectively by using one or more of the following technologies: Bayesian estimation, Maximum likelihood (ML), Maximum a priori (MAP), Non-linear least squares, and the like. In this way, defects of monitoring by either of the imaging sensor and the depth sensing sensor can be avoided, ensuring that the abnormal state of the barrier is detected timely and reliably.

It should be noted that, the processing apparatus (<NUM>) in the monitoring system in the embodiments shown in <FIG> may be disposed in a control center of a building, or may be integrated with the controller <NUM> of the escalator <NUM> or the like, and the specific setting form is not limited. Moreover, more than one monitoring system in the embodiments shown in <FIG> may be integrated for implementation, and share the sensing apparatus <NUM>, thereby simultaneously monitoring more than one of the landing plate, the step, the barrier used in the maintenance and repair working condition, and the speed of the step, to reduce costs.

It should be noted that the elements disclosed and depicted herein (including flowcharts and block diagrams in the accompanying drawings) imply logical boundaries between the elements. However, according to software or hardware engineering practices, the depicted elements and the functions thereof may be implemented on machines through a computer executable medium. The computer executable medium has a processor capable of executing program instructions stored thereon as a monolithic software structure, as standalone software modules, or as modules that employ external routines, code, services, and so forth, or any combination thereof, and all such implementations may fall within the scope of the appended claims.

Although the different non-limiting implementation solutions have specifically illustrated components, the implementation solutions of the present invention are not limited to those particular combinations. It is possible to use some of the components or features from any of the non-limiting implementation solutions in combination with features or components from any other non-limiting implementation solutions within the scope of the appended claims.

Although particular step sequences are shown, disclosed, and claimed, it should be appreciated that the steps may be performed in any order, separated or combined, unless otherwise indicated and will still benefit from the present disclosure.

Claim 1:
A monitoring system of a passenger conveyor (<NUM>), wherein the passenger conveyor is an escalator (<NUM>) or a moving walkway, the monitoring system comprising:
an imaging sensor (<NUM>) and/or a depth sensing sensor (<NUM>) configured to sense (S31) a monitored object (<NUM>) of the passenger conveyor (<NUM>) to acquire a data frame;
a processing apparatus (<NUM>) configured to analyze the data frame to monitor whether the monitored object (<NUM>) is in a normal state, the processing apparatus (<NUM>) being configured to comprise:
a background acquisition module (<NUM>) configured to acquire (S32) a background model based on a data frame that is sensed when the monitored object (<NUM>) is in a normal state;
a foreground detection module (<NUM>) configured to compare (S33) a data frame sensed in real time with the background model to obtain a foreground object; and
a working condition judgment module (<NUM>) configured to perform data processing at least based on the foreground object to judge whether the monitored object (<NUM>) is in a normal state;
wherein the processing apparatus (<NUM>) further comprises:
a foreground feature extraction module (<NUM>) configured to extract a corresponding foreground feature from the foreground object according to the monitored object (<NUM>);
wherein the working condition judgment module (<NUM>) is further configured to judge, based on the foreground feature, whether the monitored object (<NUM>) is in a normal state and wherein a comparison judgment is performed on the foreground feature by using the background model; and
characterized in that
the monitored object comprises a barrier (<NUM>) used in the passenger conveyor (<NUM>) in a maintenance and repair working condition, and the working condition judgment module (<NUM>) is further configured to judge that the barrier (<NUM>) is in an abnormal state when the barrier (<NUM>) is missing and/or placed at an improper position.