Patent Description:
<CIT> discloses a head-mounted display (HMD) device for providing holographic instruction, which is portable by a user and can be used in various applications in various areas including a work area.

The online video source of document XP054979268 discloses the use of a Head Mounted Display (HMD) in an elevator facilitating construction and maintenance. Another online video of document XP054980973 discloses a further embodiment of the HMD configured with an guidance application for maintenance.

The present invention relates to a system and a method according to the appended claims.

The invention is a system comprising features of claim <NUM>.

In further embodiments of the system, the work area can be on top of an elevator car, in an elevator pit, in an elevator hoistway, inside an elevator cab or in an escalator pit.

In accordance with one or more embodiments or any of the system embodiments above, the interface can comprise a speaker or a light and the outputs can comprise an audio alarm or a visual alarm.

In accordance with one or more embodiments or any of the system embodiments above, the interface can comprise a portable device that wirelessly communicates with the processor.

In accordance with one or more embodiments or any of the system embodiments above, the system can comprise a database providing additional information to the one or more pattern recognition systems of the processor.

In accordance with one or more embodiments or any of the system embodiments above, the database can store at least one of installation procedures, maintenance procedures, safety benchmarks, warning conditions, notification scenarios, and pattern recognition system configurations.

In accordance with one or more embodiments or any of the system embodiments above, the database can store data related to the sensor signals.

In accordance with one or more embodiments or any of the system embodiments above, the one or more pattern recognition systems can comprise at least one state machine that determines whether a sequence of steps was followed.

In accordance with one or more embodiments or any of the system embodiments above, the one or more pattern recognition systems can utilize pattern classifiers, pattern templates, and training data to determine a location and/or an orientation of a body of the mechanic.

The invention further comprises a method with features of claim <NUM>.

In accordance with one or more embodiments or any of the method embodiments above, the work area can be on top of an elevator car, in an elevator pit, in an elevator hoistway, inside an elevator cab or in an escalator pit.

In accordance with one or more embodiments or any of the method embodiments above, the interface can comprise a speaker or a light and the outputs can comprise an audio alarm or a visual alarm.

In accordance with one or more embodiments or any of the method embodiments above, the interface can comprise a portable device that wirelessly communicates with the processor.

In accordance with one or more embodiments or any of the method embodiments above, a database can provide additional information to the one or more pattern recognition systems of the processor.

In accordance with one or more embodiments or any of the method embodiments above, the database can store at least one of installation procedures, maintenance procedures, safety benchmarks, warning conditions, notification scenarios, and pattern recognition system configurations.

In accordance with one or more embodiments or any of the method embodiments above, the database can store data related to the sensor signals.

In accordance with one or more embodiments or any of the method embodiments above, the one or more pattern recognition systems can comprise at least one state machine that determines whether a sequence of steps was followed.

In accordance with one or more embodiments or any of the method embodiments above, the one or more pattern recognition systems can utilize pattern classifiers, pattern templates, and training data to determine a location and/or an orientation of a body of the mechanic.

A system, method, and computer program product (herein generally a system) for monitoring a work area for safe and efficient operation is provided. In this regard, the system monitors human positions with respect to a path of a moving object in the work area. In turn, the system can generate outputs, e.g., provide notifications to warn humans and stop the moving object to avoid contact when the humans are in the path. Further, the system monitors that proper maintenance or repair procedures (e.g., a set of complex steps) are being performed correctly and in the correct order. In turn, the system can generate outputs, e.g., provide notifications to warn humans that an unsafe work area has been created due to a step being performed incorrectly or out-of-order. The system can be used to verify that a human is wearing required safety equipment (a. personal protective equipment).

For example, with respect to when a mechanic, construction/ installation personnel, and factory worker (generally referred to herein as mechanic) performing installation, maintenance, or repairs an elevator hoistway or an escalator pit, that mechanic must follow a set of complex steps that ensures their safety (e.g., a step of inserting a mechanical blocking device, a step of opening a safety switch before closing hoistway door, etc.) and work efficiency (e.g., a step of adjusting a work tool for up-to-date procedures). During each of the set of complex steps, the system can utilize three-dimensional sensors to detect a mechanic's body orientation and location (even in an unlit environment) and detect other objects (e.g., a door position or a counterweight presence) to ensure no body part is extending outside a safety zone or piece of equipment is creating an unsafe environment.

Turning now to <FIG>, a system <NUM> is depicted according to one or more embodiments. The system <NUM> comprises a facility <NUM> (including work areas <NUM> and <NUM>), sensors <NUM> and <NUM>, a processor <NUM>, an interface <NUM>, databases <NUM> and <NUM>, and a network <NUM>. The system <NUM> is an example and is not intended to suggest any limitation as to the scope of use or operability of embodiments described herein (indeed additional or alternative components and/or implementations may be used). Further, while single items are illustrated for items of the system <NUM>, these representations are not intended to be limiting and thus, any item may represent a plurality of items. In general, the system <NUM> executes model sensing and activity determination to implement safety and efficiency at the facility <NUM>.

The facility <NUM> can be any site comprising one or more work areas <NUM> and <NUM> for a mechanic to perform installation, maintenance, and/or repairs. Examples of the facility <NUM> include but are not limited to an office building, a high-rise, a mall, a transportation station, a school, and a hotel. Examples of the work areas <NUM> and <NUM> can be the physical space surrounding an elevator or an escalator of the facility <NUM>. For instance, work areas with respect to an elevator include, but are not limited to, within an elevator hoistway (e.g., on top of an elevator car), inside an elevator cab, in the elevator pit, in a machine room, a utility room, and inside the elevator itself. Further, a work area with respect to an escalator can include, but is not limited to, an escalator pit.

The sensors <NUM> and <NUM> can be any electro-mechanical component that detect events in an environment (e.g., the work areas <NUM> and <NUM>) and generate an electrical signal (e.g., sensors signals and/or sensor data) as a function of the events. With respect to the elevator embodiment described herein, the sensor <NUM> can be located on top of the elevator car to detect events within the elevator hoistway and above the elevator car, while the sensor <NUM> can be located in the elevator pit to detect events within the elevator hoistway and below the elevator car. Note that an event can include a status of a piece of equipment (e.g., a machine that may continue to be powered on), a phase of a maintenance procedure, an operation of a piece of equipment, and/or events or statuses relevant to a safety of a mechanic.

In accordance with one or more embodiments, the sensors <NUM> and <NUM> are three-dimensional depth sensors that determine locations (e.g., x, y, z points) for a body of a mechanic (e.g., appendages or torso) and for other objects. In one embodiment, the sensors <NUM> and <NUM> function with precision in any light condition. For example, as a three-dimensional depth sensor, the sensors <NUM> and <NUM> can provide a scene with near-infrared light that returns to the sensors <NUM> and <NUM> distorted depending upon where things are in the scene. That is, the distorted near-infrared light returning back from the scene can be considered the coded light that the sensors <NUM> and <NUM> can read using triangulation, time of flight determination, and signal processing algorithms. In this way, the sensors <NUM> and <NUM> can interpret and identify people (the mechanic), their body properties (sensing of the position and orientation of the mechanic's body), their movements and gestures (whether the mechanic has moved outside a safety zone); interpret and classify of objects (such as a hoist rope); and interpret and identify walls, floor, and vacant space. Note that the system <NUM> leverages a connection between the sensors <NUM> and <NUM> and what a human person is doing in the work areas <NUM> and <NUM>. In this way, the sensors <NUM> and <NUM> can determine positions of key body joints and other body parts (e.g., whether each eye is opened, where someone is looking a certain direction) to distinguish if a human is putting some part of their body in an unsafe location.

In accordance with one or more embodiments, the sensors <NUM> and <NUM> can be any depth sensor in a vicinity of the work area, such as time-of-flight sensors. A time-of-flight sensor transmits a signal and utilizes a receiver to determine a distance traveled by the signal based on a time between transmission and receipt. Example of time-of-flight sensors include, but are not limited to, any electromagnetic wave (e.g., RADAR) or acoustic signal (e.g., ultrasonic) sensor. For instance, personal protective equipment can be detected by the sensors <NUM> and <NUM>, and the system <NUM> can subsequently determine that the personal protective equipment is being worn properly.

Note that a plurality of sensors can be employed at each location to improve a viewing angle when objects, like hoist ropes and mechanical equipment, interfere with a field of view of a single sensor. Other examples of the sensors <NUM> and <NUM> include, but are not limited to, video sensors, infrared sensors, depth sensors, motion sensors, and floor mounted pressure sensors.

The processor <NUM> can include any processing hardware, software, or combination of hardware and software utilized by the system <NUM> to carry out computer readable program instructions by performing arithmetical, logical, and/or input/output operations. The processor <NUM> can include a central processing unit and a memory, where the memory stores program code executable by the central processing unit to cause the processor <NUM> to facilitate monitoring of the work areas <NUM> and <NUM> for safe and efficient operation by receiving/processing sensor data and additional information (from the databases <NUM> and <NUM>) and determining whether the mechanic is following the correct sequence of operations within a prescribed area (e.g., the work areas <NUM> and <NUM>). In accordance with one or more embodiments, the processor <NUM> operates one or more pattern recognition systems (in parallel) that receive as inputs the sensor signals and change their state based those sensor signals. For instance, in accordance with one or more embodiments, a pattern recognition system can be a state machine. For example, in accordance with one or more embodiments, at least two state machines can respectively operate in parallel to detect whether a proper process is being followed and whether objects (including the mechanic themselves) are out of harm's way. Note that a state machine is an example of a pattern recognition systems and that state machines are well-suited for recognizing that a sequence of steps have been followed where the inputs could be the recognition of a certain step (e.g., mechanic has blocked open the door) or the change in status of a relevant state (e.g., power is switched on or off, body part is within or outside the path of a counterweight) and states and transitions between states triggered by the inputs define whether or not a proper sequence of steps has been followed.

The interface <NUM> can be any output mechanism for generating outputs, e.g., warning (or alerting) humans, including the mechanic, that are in a path of a moving object and/or that are in an unsafe work area created due to an installation procedure or a maintenance procedure being performed incorrectly or out-of-order. An example of the interface <NUM> can include, but is not limited to, a speaker, and an example of the warning can include an audio alarm. Examples of the interface <NUM> can also include a wearable device (e.g., a head-mounted display or optical head-mounted display); a visual alarm (e.g., a flashing light, signage, or displayed text); and a database that tracks activity for subsequent analysis (e.g., audit purposes, learning what efficient workers do, learning where people spend the most time, etc.). The interface <NUM> can generate outputs, e.g., output the warnings based on each state of the one or more pattern recognition systems. An example of a human being in a path of a moving object includes when a mechanic has inadvertently positioned themselves in a path of an elevator counterweight. An example of an unsafe work area created due to a maintenance procedure being performed incorrectly or out-of-order includes the closing of the safety chain during a procedural step that requires the safety chain to be open. Note that the interface <NUM> can be located at a place where the mechanic would be instantaneously notified, i.e., located within the work areas <NUM> and <NUM>. Thus the interface <NUM> can be portable (e.g., a beacon or other wearable device), while wirelessly communicating with the processor <NUM>, such that it can move with the mechanic between the work areas <NUM> and <NUM>. In one embodiment, the interface <NUM> might be a mobile phone carried by the mechanic. In an embodiment, a plurality of interfaces <NUM> can be employed in the system <NUM> (e.g., one for each work area).

In accordance with one or more embodiments, the sensors <NUM> and <NUM> and the interface <NUM> can communicate (whether through wired or wireless communications) with the processor <NUM> that is located outside the work areas <NUM> and <NUM>, such as within a machine room controller or in a cloud computing infrastructure. In one embodiment, the processor <NUM> may be located within the work area <NUM> and/or work area <NUM>. In this way, the interface <NUM> can provide immediate feedback with respect to when the sensors <NUM> and <NUM> detect that a human or object is in a path of a moving object and/or an unsafe work area has been created.

In accordance with one or more embodiments, the sensor <NUM> can be integrated with the processor <NUM> and the interface <NUM> in the same housing to create a first sensing and warning device. Further, in one embodiment, the sensor <NUM> can be separately integrated with the processor <NUM> and the interface <NUM> in the same housing to create a second sensing and warning device. The first and second sensing device can communicate with each other (whether through wired or wireless communications) and provide immediate feedback to a mechanic with respect to when a human or object is in a path of a moving object and/or an unsafe work area has been created. In one embodiment, the sensors <NUM> and <NUM>, processor <NUM>, and interface <NUM>, may be joined in any combination of housings or separate.

Each database <NUM> and <NUM> can be a computer and/or data server that stores data for use by the processor <NUM>. The databases <NUM> and <NUM> can communicate over the network <NUM>. The network <NUM> is a system of computing components and electrical and radio connections that allow and support communications with nodes thereon (e.g., the database <NUM> and <NUM> and the processor <NUM>). In accordance with one or more embodiments, the database <NUM> can store at least one of installation procedures, maintenance procedures (e.g., sequence of actions for repairing rail alignments, wiring inside a hoistway, malfunctioning junction boxes), safety benchmarks (e.g., location ranges for the work areas <NUM> and <NUM>), warning conditions, notification scenarios, and pattern recognition system configurations (e.g., pattern recognition systems that utilize pattern classifiers, pattern templates, and training data to detect a state of the system <NUM>, facility <NUM>, and/or work areas <NUM> and <NUM>; spatiotemporal pattern recognition systems implemented as a state machine). The database <NUM> can store data related to the sensor signals (e.g., generated by the sensors <NUM> and <NUM>), whether in the form of raw sensor signals and/or historical sensor data derived from the raw sensor signals. In one embodiment, the database <NUM> and database <NUM> can be combined into a single database.

In accordance with one or more embodiments, the system <NUM> can be integrated in a work or safety suit worn by a human. The work or safety suit can include markers, which facilitate identifying each part of the human's body. Examples of markers include fiducial markers, such as quick-response codes printed on or affixed to the work or safety suit. The sensors <NUM> and <NUM> can detect the markers to identify and locate each part of the body.

In accordance with one or more embodiments, the system <NUM> can utilize a combination of one or more beacons worn at specific locations of the humans' body (e.g., a helmet, bracelets, on a belt, on one or both boots, etc.). The one or more beacons can broadcast signals to one or more receivers (e.g., the sensors <NUM> and <NUM>). The one or more receivers and/or the one or more beacons themselves can receive and process the signals to determine the position of the one or more beacons.

In accordance with one or more embodiments, the system <NUM> can utilize wearable sensors, such as an accelerometer. Like the beacons, the wearable sensors can be worn at specific locations of the humans' body (e.g., a helmet, bracelets, on a belt, on one or both boots, etc.). The wearable sensors can provide signals that are used to determine the position and orientation of the wearable sensor (and hence the attached body part).

Turning now to <FIG>, a process flow <NUM> is depicted according to one or more embodiments. The process flow <NUM> is described with respect to <FIG> above.

The process flow <NUM> begins at block <NUM>, where the processor <NUM> receives sensor data. The sensor data can be the electrical signals outputted by the sensor <NUM> in the work area <NUM> to the processor <NUM>. Similarly, the sensor data can be the electrical signals outputted by the sensor <NUM> in the work area <NUM> to the processor <NUM>. The sensor data can be raw sensor signals or sensor information prepacked by the sensors <NUM> and <NUM> themselves.

At block <NUM>, the processor <NUM> analyzes the sensor data to interpret an object status (e.g., an event). The object status can include one or more appendages of a mechanic (e.g., processing the sensor data/signals to determine a location and/or an orientation of a body) and other objects within the work areas <NUM> and <NUM>. In accordance with one or more embodiments, the processor <NUM> performs analytic operations to interpret the status of each object detected by the sensors <NUM> and <NUM>. The objects can include a door wedge, a counterweight, and human appendages. The status can respectively indicate whether the door wedge is not properly inserted, whether a counterweight is in motion, whether a human appendage is in a path of the counterweight (or outside a safety barrier), or any other safety or job-related item.

At block <NUM>, the processor <NUM> compares a sequence of changes for the object status with respect to an operation model. For example, the processor <NUM> can compare the sequence of status changes, such as a motion path of human appendage, with respect to an operation model defined by one or more pattern recognition systems. At block <NUM>, the processor <NUM> signals when a proper sequence is not followed.

Turning now to <FIG>, a process flow <NUM> is depicted according to one or more embodiments. The process flow <NUM> is described with respect to <FIG> above. The process flow <NUM> begins at block <NUM>, where the processor <NUM> receives sensor data. At block <NUM>, the processor <NUM> analyzes the sensor data to interpret an object status (e.g., an event). Further, the processor <NUM> can analyze the sensor data to determine at which phase of a maintenance procedure a mechanic is currently on (e.g., an event).

At block <NUM>, the processor <NUM> receives additional information. The information can be derived from one or more databases <NUM> and <NUM>. The information can be derived from other components of the system, such as when a top of car stop button is engaged and/or when a data gathered or produced by an elevator/escalator controller. The information can include, but is not limited to, general maintenance standards data (that a certain component can only be used X times or needs to be replaced after Y days) and data specific to equipment (such as the date that a component was last replaced or the number of cycles that the equipment has run since the last maintenance procedure). The receipt of additional status information can be optional, as indicated by the outline of block <NUM> being a dashed-line.

At decision block <NUM>, the processor <NUM> determines whether proper operation exists with respect to a sequence of changes for the object status and the additional information. For example, a state machine of the processor <NUM> can utilize the additional information in combination with the sensor signals to determine whether the mechanic has correctly advanced to a next phase of the maintenance procedure.

The process flow <NUM> proceeds to block <NUM> if the proper operation does not exist with respect to the sequence of changes for the object status and to the additional information (as indicated by the NO arrow). At block <NUM>, the processor <NUM> signals an output when the proper sequence is not followed.

The process flow <NUM> proceeds to block <NUM> if the proper operation exists with respect to the sequence of changes for the object status and to the additional information (as indicated by the YES arrow). At block <NUM>, the processor <NUM> issues a permissive signal to proceed when all steps have been followed.

In accordance with one or more embodiments, the system <NUM> can determine that any sequence of steps was properly followed (such as an explicit list of steps). For instance, timing-dependent procedures can be by an explicit lists of steps or pattern recognition systems with timing requirement. For example, after a mechanic applies a chemical of a first type, the mechanic must wait for at least X minutes before applying a second coat of the chemical.

Claim 1:
A system comprising:
at least one sensor (<NUM>, <NUM>) which is to be located within a work area (<NUM>, <NUM>) of a facility (<NUM>), wherein, when the at least one sensor (<NUM>, <NUM>) is located within the work area (<NUM>, <NUM>), the at least one sensor (<NUM>, <NUM>) is configured to detect events in the work area (<NUM>, <NUM>) and to output sensor signals in accordance with the events detected in the work area (<NUM>, <NUM>), wherein the events comprise movements or gestures by a body of a mechanic,
a processor (<NUM>) communicatively coupled to the at least one sensor (<NUM>, <NUM>) and processing the sensor signals utilizing one or more pattern recognition systems; and
an interface (<NUM>) communicatively coupled to the processor (<NUM>) and generating outputs based on each state of the one or more pattern recognition systems,
wherein the one or more pattern recognition systems comprise at least one state machine that determines whether a sequence of steps was followed;
wherein the at least one sensor (<NUM>, <NUM>) is configured to determine positions of body joints and other body parts to distinguish if the mechanic is putting some part of its body in an unsafe location;
wherein the at least one sensor (<NUM>, <NUM>) comprises a three-dimensional depth sensor; and
wherein the interface (<NUM>) is configured to provide feedback with respect to when the at least one sensor (<NUM>, <NUM>) detects that the mechanic or an object is in a path of a moving object.