Patent ID: 12243403

DESCRIPTION OF EMBODIMENTS

An embodiment of the present invention will be described in detail below with reference to the drawings. Note that the drawings are only schematically illustrated to the extent that the present invention can be sufficiently understood. Hence, the present invention is not limited only to the illustrated examples. Further, in the drawings, the same reference numerals are given to common components and similar components, and redundant description thereof will be omitted.

<Configuration of Safety Monitoring Device>

Hereinafter, a configuration of a safety monitoring device10according to the present embodiment will be described with reference toFIGS.1and2.FIG.1is a configuration diagram of a safety monitoring system S including the safety monitoring device10according to the present embodiment.FIG.2is a configuration diagram of the safety monitoring device10.

As illustrated inFIG.1, the safety monitoring system S includes one or a plurality of safety monitoring devices10, one or a plurality of lidars51, one or a plurality of monitoring cameras61, and one or a plurality of Web terminals62.

The safety monitoring device10is a device that monitors safety of a sensing target area (measurement space) of a safety sensor. The safety monitoring device10includes a personal computer (PC) or the like.

The lidar51is a three-dimensional sensor that is installed around the measurement space so as to be used as a safety sensor and senses an object.

The monitoring camera61is a photographing unit that is counterposed around the measurement space and photographs the measurement space. The monitoring camera61may be configured to include a setting reception unit61a, a setting processing unit61b, and a notification unit61clike a monitoring camera61z. The monitoring camera61zis an external device on which a program (not illustrated) for performing camera video processing is mounted. The setting reception unit61ais a means that receives setting of camera video processing. The setting processing unit61bis a means that executes the camera video processing received by the setting reception unit61awith respect to the video data. The setting processing unit61bmay be set to determine the safety of the measurement space. The notification unit61cis a means that notifies the safety monitoring device10and the Web terminal62of the video subjected to the camera video processing and the determination result of the safety of the measurement space.

The Web terminal62is an external device on which a program (not illustrated) for performing camera video processing is mounted. In the present embodiment, description is given assuming that that the Web terminal62includes a notification acquisition unit62a. The notification acquisition unit62ais a means that acquires a video subjected to the camera video processing and a determination result of safety of the measurement space from the safety monitoring device10or the monitoring camera61z.

As illustrated inFIG.2, the safety monitoring device10includes a control unit11, a storage unit30, and a communication unit39. Further, the safety monitoring device10further includes an operation unit41, a display42(display unit), a warning lamp43, and a speaker44(sounding unit).

The control unit11includes an area setting unit12, an alarm activation type setting unit13, a video acquisition unit14, a lidar data acquisition unit15, a moving body recognition analysis unit16, a result output unit17, a video processing unit18, a recording management unit19, a video display processing unit20, a safety sensor input unit21, a safety monitoring unit22, and an event list processing unit23.

The area setting unit12is a means that sets an area (for example, alarm activation target area, alarm activation exclusion area, or the like, to be described below) having an arbitrary attribute with respect to a sensing target area (measurement space) of the lidar51, which is a safety sensor.

The alarm activation type setting unit13is a means that sets an arbitrary alarm activation function type for each area.

The video acquisition unit14is a means that acquires video data of the measurement space.

The lidar data acquisition unit15is a sensor data acquisition unit that acquires sensor data from the lidar51, which is a safety sensor, arranged around the measurement space.

The moving body recognition analysis unit16is a means that extracts a moving body from three-dimensional data on the basis of sensor data measured by the safety sensor (the lidar51in the present embodiment), recognizes the motion of the moving body, analyzes a traveling direction, and issues a warning. The moving body recognition analysis unit16includes a foreground data extraction unit16a, a clustering processing unit16b, a bounding box generation unit16c, an angle smoothing unit16d, an image recognition unit16e, and an alarm activation analysis unit16f.

The foreground data extraction unit16ais a means that extracts foreground data from the input three-dimensional data.

The clustering processing unit16bis a means that groups data on the basis of similarity between data by unsupervised learning.

The bounding box generation unit16cis a means that generates a bounding box. Here, the “bounding box” means a minimum rectangular or cuboidal region surrounding a certain object.

The angle smoothing unit16dis a means that performs averaging processing with the number of frames or a processing method specified in advance.

The image recognition unit16eis a means that recognizes an image from the video data. The image recognition unit16eis a means that performs person determination by face identification, sex determination, posture determination, and the like.

The alarm activation analysis unit16fis a means that performs processing according to the type of the alarm activation function using the result of moving body detection in the clustering processing and issues a warning.

The result output unit17is a means that sends an issued warning as a result of sensing of the moving body by the moving body recognition analysis unit16to the video processing unit18as event data.

The video processing unit18is a means that creates a display screen to be displayed on the display42. The video processing unit18can perform processing such as superimposing the event data on the video data. For example, the video processing unit18can perform arbitrary camera video processing such as superimposing a person sensing frame on the camera video.

The recording management unit19is a means that records and manages the video data in the storage unit.

The video display processing unit20is a means that processes a file divided at predetermined time intervals to create a video and displays the video on the display42.

The safety sensor input unit21is a means that inputs a signal output from a safety sensor such as a lidar, a camera, a human sensor, or a touch sensor.

The safety monitoring unit22is a means that monitors entry of an object into a monitoring region. The event list processing unit23is a means that creates and manages an event list selectable for each event. Note that the event list includes alarm activation data, custom data, and the like as events.

The storage unit30stores a control program Pr, setting recording data31, video recording data32, and processed image data33.

The control program Pr is a program that causes a computer to function as the safety monitoring device10.

The setting recording data31is data set in the safety monitoring device10via a user interface. The video recording data32is video data photographed by the monitoring camera61.

The processed image data33is image data obtained by performing arbitrary processing on the video data.

<Setting of Area>

Hereinafter, setting of an area will be described with reference toFIG.3.FIG.3is an explanatory diagram of an alarm activation target area and an alarm activation exclusion area.

The area setting unit12of the safety monitoring device10sets the alarm activation target area and the alarm activation exclusion area for the sensing target area (measurement space).

The sensing target area is a measurement space in which an object is measured by the lidar51. In other words, the sensing target area is an area where the movement of the moving body present within a sensible range of the lidar51is monitored by the safety monitoring device10.

The alarm activation target area is an area in which an event is issued (warning is issued) when an entering person is sensed. The event is data indicating that a warning has been issued. The safety monitoring device10continues to issue a warning while an entering person is present in the alarm activation target area. Further, the safety monitoring device10does not issue a warning even when a moving body is sensed outside the alarm activation target area. A plurality of alarm activation target areas can be set. Setting of the alarm activation target area is valid only in the sensing target area. Note that when the alarm activation target area is set to overlap the alarm activation exclusion area, the alarm activation exclusion area is prioritized, and thus the safety monitoring device10does not issue a warning.

The alarm activation exclusion area is an area in which an event is not issued (warning is not issued) even when an entering person is sensed. The alarm activation exclusion area is designated when sensing of a moving body is intentionally excluded from the monitoring target. The safety monitoring device10does not sense an entering person or issue an event (issue a warning) within the range designated as the alarm activation exclusion area even within the range of the sensing target area or the alarm activation target area. For example, when there is a tree in the sensing target area and there is a possibility that the tree is erroneously sensed as an entering object due to swaying of the tree by wind, the safety monitoring device10can prevent erroneous sensing by setting the area around the tree as the alarm activation exclusion area. A plurality of alarm activation exclusion areas can be set.

As illustrated inFIG.3, the area setting unit12can set a plurality of alarm activation target areas for one sensing target area (measurement space). In the example illustrated inFIG.3, two alarm activation target areas A61aand A61bare set for the sensing target area (measurement space) indicated by the two-dot chain line. Note that although only one alarm activation exclusion area A62is set in the example illustrated inFIG.3, the area setting unit12can set a plurality of alarm activation exclusion areas for one sensing target area (measurement space).

In the example illustrated inFIG.3, when the person moves along a walking route R11, a portion where the walking route R11and the alarm activation exclusion area A62overlap is set as an exclusion portion B62. Then, a portion where the walking route R11and an alarm activation target area A61aoverlap, the portion being a portion not overlapping the alarm activation exclusion area A62is set as an alarm activation portion B61a. Furthermore, a portion where the walking route R11and the alarm activation target areas A61boverlap is set as an alarm activation portion B61b. The safety monitoring device10issues an event (issues a warning) when or while a person enters the alarm activation portions B61aand B61b.

FIGS.4A to4Cillustrate an example of setting of an area.FIGS.4A to4Care each an explanatory diagram of a display screen100of the safety monitoring device10. The safety monitoring device10displays the display screen100on the display42and accepts the setting of area by a safety monitoring person. Here, the “safety monitoring person” means a person who monitors the safety of each measurement space by the safety monitoring device10.

In the example illustrated inFIG.4A, the display screen100includes a 3D viewer unit101, a setting unit102, a lidar selection unit103, an alarm activation target area tab104, and the like.

The 3D viewer unit101is a field for displaying a video of the measurement space photographed by the monitoring camera61and an image obtained by processing the video.

The setting unit102is a field for performing various settings such as an area and the type of an alarm activation function.

The lidar selection unit103is an instruction unit that selects three-dimensional data (distance measurement data) acquired by an arbitrary lidar51.

The alarm activation target area tab104is a tab for setting an alarm activation target area.

In the example illustrated inFIG.4B, the screen of the alarm activation target area tab104is configured to include a block tab104a, a validity check box104b, an area selection unit104c, a display color setting unit104d, and the like.

The block tab104ais a tab for setting the specifications of the bounding box to be added to an object for which alarm activation is performed. Here, the “bounding box” means a minimum rectangular or cuboidal region surrounding a certain object.

The validity check box104bis a field for designating addition of a bounding box to an object for which alarm activation is performed.

The area selection unit104cis a field for selecting an area to be set as the alarm activation target area. The display color setting unit104dis a field for setting an alarm activation target area frame101a(seeFIG.4C) shown in the 3D viewer unit101.

When the safety monitoring person selects (designates) an arbitrary area using the area selection unit104c, the safety monitoring device10sets the selected area as the alarm activation target area. At this time, the safety monitoring device10changes the color of the alarm activation target area frame101a(seeFIG.4C) shown in the 3D viewer unit101to the color set by the display color setting unit104d.

<Alarm Activation Function>

The safety monitoring device10has various types of alarm activation functions. Here, for example, a case where the safety monitoring device10uses the three types of alarm activation functions described below will be described as an example.

(1) Entry sensing: issuing a warning to all moving bodies that have entered an alarm activation target area (entry prohibited area).

(2) Entry sensing (person): issuing a warning only to a “person” among moving bodies that have entered an alarm activation target area (entry prohibited area).

(3) Proximity sensing: issuing a warning when a “person” and a “large object” that have entered an alarm activation target area (entry prohibited area) become equal to or less than a preset certain distance.

Hereinafter, a usage example of the alarm activation function will be described with reference toFIGS.5and6.FIG.5is an explanatory diagram illustrating an example of an area.FIG.6is an explanatory diagram illustrating an example of selection of an alarm activation function by a user interface.

For example, in the example illustrated inFIG.5, three areas: a first area, a second area, and a third area are set for one measurement space.

The first area is an empty area. Therefore, the safety monitoring person can set an arbitrary alarm activation function for the first area according to the operation. Thus, for example, the safety monitoring person can set “(1) entry sensing” for the first area and set the first area so that any moving body is a sensing target.

The second area is a forklift work place, and is an area that is dangerous for a person entering. Thus, the safety monitoring person sets “(2) entry sensing (person)” for the second area.

The third area is a crane work site, and is an area that is dangerous when a person comes closer to the crane with respect to a certain distance R. Thus, the safety monitoring person sets “(3) proximity sensing” for the third area in order to issue a warning to a person who is closer to the crane with respect to the certain distance R.

As illustrated inFIG.6, the display screen100(seeFIG.4A) of the safety monitoring device10is configured such that the setting unit102can select one of these three types of alarm activation functions.

Further, the display screen100(seeFIG.4A) of the safety monitoring device10is configured to be able to select a plurality of alarm activation target areas (entry prohibited areas), and each area can be independently set as an area having an arbitrary attribute.

<Creation of Bounding Box>

Hereinafter, the creation of the above-described “bounding box” will be described with reference toFIGS.7to10C.FIG.7is an explanatory diagram illustrating an example of point cloud data of a bus and a bounding box.FIG.8is an explanatory diagram illustrating an example of coordinates and a configuration of a bounding box.FIG.9is an explanatory diagram illustrating an example of a bottom surface of a bounding box.FIGS.10A to10Care each an explanatory diagram of a method for obtaining a rectangle of a plane (x, z), which is a bottom surface of a bounding box.

The bounding box is created, for example, by the method described in Patent Literature 1 (WO 2018/084146 A) described above.FIG.7illustrates an example of point cloud data (L-shape shape data) of a bus measured by the lidar51and a bounding box having a rectangular shape formed so as to surround the point cloud data. Further,FIG.8illustrates an example of coordinates and configuration of a bounding box having a rectangular shape formed along the three-dimensional shape of the bus.

In a three-dimensional coordinate system (x, y, z), a horizontal plane is defined as (x, z), and a height direction is defined as a y coordinate. The bounding box includes a minimum rectangular or cuboidal region surrounding a certain object. A length in a long direction of the bottom surface is a width W (Width), a length in a narrow direction is a depth D (Depth), and a length from the bottom surface to the top surface is a height H (Height).

FIG.9illustrates an example of the bottom surface of the bounding box of the bus disposed to be inclined with respect to the X axis. In the example illustrated inFIG.9, the angle formed with the X axis is indicated as an angle θ.

FIGS.10A to10Ceach illustrate a method for obtaining a rectangle of a plane (x, z), which is a bottom surface of a bounding box. In the drawings, a pixel point indicated by a black circle (hereinafter, referred to as a “black circle pixel point”) represents each pixel of a point cloud subjected to clustering processing.

InFIGS.10A to10C, first, straight lines passing through black circle pixel points and having an inclination of an angle θ are drawn, and the outermost straight line is doubled in the X-axis direction of the bounding box at the angle θ. Next, straight lines passing through black circle pixel points and having an inclination of an angle θ+90° are drawn, and the outermost straight line is doubled in the Z-axis direction of the bounding box at the angle θ. Finally, a rectangle surrounded by these four straight lines is defined as a bounding box at the angle θ.

This procedure is repeated θ=0°, Δθ, 2×Δθ, 3×Δθ, . . . , (n−1)×Δθ (here, n is a maximum integer satisfying ((n−1)×Δθ)<90°), and a bounding box is obtained (calculated) at each inclination.

Then, among the obtained bounding boxes of each inclination, a bounding box having the largest correlation with the point cloud data is set as an optimal bounding box of the point cloud data. In this way, a bounding box of each monitoring target is created.

<Relationship between Moving Body and Bounding Box>

Hereinafter, the relationship between the moving body and the bounding box will be described with reference toFIGS.11to13B.FIG.11is an explanatory diagram illustrating an example of a bounding box.FIG.12is an explanatory diagram illustrating an example of determination conditions of a person and a large object.FIG.13Ais an explanatory diagram illustrating an example of attributes (in the case of a large object) in consideration of a classification result and continuity of a detected moving body.FIG.13Bis an explanatory diagram illustrating an example of attributes (in the case of a person) in consideration of a classification result and continuity of a detected moving body.

As illustrated inFIG.11, the safety monitoring device10senses a moving body105with the lidar51, and generates a bounding box105box having a cuboidal region (rectangular region) shape so as to just surround (surround with a minimum volume) the moving body105.

The safety monitoring device10determines whether the moving body105surrounded by the bounding box105box is a person or a large object larger than the person from the size (width, depth, height) of the bounding box105box. The determination is made on the basis of, for example, determination conditions illustrated inFIG.12.

In the example illustrated inFIG.12, values of the width W, the depth D, and the heights H, Ht, and Hb of the moving body105are illustrated as determination conditions. Note that the height H is a distance from the top surface to the lower surface of the bounding box105box. Further, the height Ht is a distance from a ground G to the top surface of the bounding box105box. Further, the height Hb is a distance from the ground G to the lower surface of the bounding box105box.

For example, in each item of the width W, the depth D, and the heights H, Ht, and Hb of the moving body105shown in a certain frame of a series of video data, the safety monitoring device10classifies the moving body105as “person” in a case where all the values conform to the determination conditions of “person” described below. However, the ranges described below are merely an example, and can be changed depending on the operation.

(Determination Conditions of “Person”)

Width W: a range from a minimum value of 0.3 [m] to a maximum value of 1.2 [m].Depth D: a range from a minimum value of 0.2 [m] to a maximum value of 1.0 [m].Height H: a range from a minimum value of 0.5 [m] to a maximum value of 2.3 [m].Height Ht: a range from a minimum value of 0.5 [m] to a maximum value of 3.0 [m].Height Hb: a range from a minimum value of −1.0 [m] to a maximum value of 0.8 [m].

Specifically, in the width W, the depth D, and the heights H, Ht, and Hb of the moving body105shown in a certain frame of a series of video data, the safety monitoring device10classifies the moving body105as “person” in a case where all the values of the items are within the ranges described below.0.3 [m]≤width W≤1.2 [m]0.2 [m]≤depth D≤1.0 [m]0.5 [m]≤height H≤2.3 [m]0.5 [m]≤height Ht≤3.0 [m]−1.0 [m]≤height Hb≤0.8 [m]

Further, for example, in each item of the width W, the depth D, and the heights H, Ht, and Hb of the moving body105, the safety monitoring device10classifies the moving body105as “large object” in a case where each value conforms to determination conditions of “large object” described below. However, the ranges described below are merely an example, and can be changed depending on the operation.

(Determination Conditions of “Large Object”)

In the items of the width W, the depth D, and the heights H, Ht, and Hb of the moving body105, all of the items are equal to or larger than the minimum value of “person”, and one or more items are equal to or larger than the maximum value of “person”.

Specifically, in the width W, the depth D, and the heights H, Ht, and Hb of the moving body105of a certain frame (bounding box105box), when the value of each item satisfies a first condition and a second condition described below, the safety monitoring device10classifies the moving body105as “large object”.

(First Condition)

All values of the items satisfy the conditions described below.0.3 [m]≤width W0.2 [m]≤depth D0.5 [m]≤height H0.5 [m]≤height Ht−1.0 [m]≤height Hb
(Second Condition)

The value of any one or more items satisfies the conditions described below.1.2 [m]≤width W1.0 [m]≤depth D2.3 [m]≤height H3.0 [m]≤height Ht0.8 [m]≤height Hb

Further, in each item of the width W, the depth D, and the heights H, Ht, and Hb of the moving body105, the safety monitoring device10classifies the moving body105as unknown (others) in a case where each value does not conform to the determination conditions of person or the determination conditions of large object.

Note that the safety monitoring device10preferably sets a threshold value to be large in consideration of a case where a plurality of persons is detected as one object. Further, the safety monitoring device10preferably sets a threshold value to be large in consideration of a case where the moving body105appears to float from the ground, such as a case where only the upper body of a person is visible.

Further, as described above, the safety monitoring device10can perform classification into three types: “person”, “large object”, and “unknown (others)” for each frame. However, the moving body sensed by the lidar51is split at the time of clustering processing due to measurement variation, or the size varies. Therefore, a large object may instantaneously become a mass of a person's size. In this way, there is a possibility that a large object is classified as “person”. Thus, for example, as illustrated inFIG.13A, the safety monitoring device10preferably classifies the attribute of the moving body105as “large object (large moving body)” in a case where the size of the moving body105is continuously maintained for a plurality of arbitrarily determined frames. In this way, the safety monitoring device10can prevent a large object from being classified as “person”.

Note that, in the example illustrated inFIG.13A, the safety monitoring device10classifies the attribute of the moving body105as “large object (large moving body)” in a case where the size of the moving body105is continuously maintained for three frames. Thereafter, the safety monitoring device10maintains the attribute of the moving body105as “large object (large moving body)” unless three frames of another attribute (for example, “person”) size continue.

Further, there is a case where a moving body501classified as “person” instantaneously becomes a mass having a size larger than a person (or a mass having a size smaller than a person) due to the influence of some noise. In this case, for example, as illustrated inFIG.13B, since the continuity of the moving body501is interrupted, it is sufficient if the attribute of the moving body501is reviewed only when a plurality of arbitrarily determined frames (for example, three frames) continues.

Note that, in the lidar51at 10 fps (frames/second), since the time of one frame is 0.1 seconds, three frames correspond to 0.3 seconds. Here, the case where three frames are continuous has been described, but the optimum number of frames may be selected according to the site.

Further, the safety monitoring device10may provide an alarm activation target area (entry prohibited area) within the measurement range of the lidar51, and issue a warning only when the moving body501identified as “person” enters the alarm activation target area.

<Operation of Safety Monitoring Device>

Hereinafter, the operation of the safety monitoring device10will be described with reference toFIG.14.FIG.14is a flowchart illustrating an operation at the time of person sensing and alarm activation.

As illustrated inFIG.14, in step S101, the safety monitoring person operates the safety monitoring device10while viewing the display screen100(seeFIG.4A) and sets an area having an arbitrary attribute with respect to the measurement space. In response to this, the area setting unit12of the safety monitoring device10sets one or a plurality of alarm activation target areas and one or a plurality of alarm activation exclusion areas for one measurement space.

Next, in step S102, the safety monitoring person operates the safety monitoring device10while viewing the display screen100(seeFIG.4A) and designates the type of an alarm activation function with respect to each alarm activation target area. In response to this, the alarm activation type setting unit13of the safety monitoring device10sets the type of the alarm activation function with respect to each alarm activation target area. Thereafter, the safety monitoring person operates the safety monitoring device10and instructs the safety monitoring device10to start monitoring the measurement space.

Then, in step S103, the video acquisition unit14of the safety monitoring device10acquires video data of each measurement space from each monitoring camera61.

Next, in step S105, the lidar data acquisition unit15of the safety monitoring device10acquires distance measurement data (θ(i), φ(j), R) as three-dimensional data (sensor data) from the lidar51.

Next, in step S110, the foreground data extraction unit16aof the moving body recognition analysis unit16of the safety monitoring device10extracts foreground data from three-dimensional data by the background subtraction method. That is, the foreground data extraction unit16aextracts the foreground data by calculating (three-dimensional data−reference background data).

Next, in step S115, the clustering processing unit16bof the moving body recognition analysis unit16of the safety monitoring device10extracts a cluster of point cloud data (moving bodies) from the foreground data on the basis of the distance data of the point cloud sensed using the three-dimensional data, and performs the clustering processing.

Next, in step S120, the bounding box generation unit16cof the moving body recognition analysis unit16of the safety monitoring device10creates a bounding box by creating a cuboidal region necessary to surround the cluster of the point cloud data (moving bodies) subjected to the clustering processing.

Next, in step S125, the alarm activation analysis unit16fof the moving body recognition analysis unit16of the safety monitoring device10confirms whether or not the moving body conforms to the determination conditions of person from the size (width, depth, height) of the bounding box. In a case where it is confirmed that the moving body conforms to the determination conditions of person, the alarm activation analysis unit16fadds (sets) a flag (person flag) representing a person to the moving body. In this way, the safety monitoring device10recognizes a person with respect to the moving body.

Next, in step S130, the alarm activation analysis unit16fof the moving body recognition analysis unit16of the safety monitoring device10senses the same size of the same moving body in a next frame. Then, when the size of the moving body continuously conforms to the size of the person for the designated number of frames, the alarm activation analysis unit16fsets the attribute of the moving body to “person”.

Next, in step S135, the alarm activation analysis unit16fof the moving body recognition analysis unit16of the safety monitoring device10checks whether or not the moving body has entered the alarm activation target area. Then, in a case where there is an entering moving body, and when the attribute of the entering moving body is “person”, the alarm activation analysis unit16fissues a warning.

<Distance Sensing Method>

Hereinafter, a distance sensing method will be described with reference toFIG.15.FIG.15is an explanatory diagram illustrating an example of a distance between a “person” and a “large object” viewed from above.

The safety monitoring device10can obtain (calculate) the distance between the classified “person” and “large object” as described below.

As illustrated inFIG.15, the bottom surface of a bounding box106box of a person106does not become an extreme rectangle. Therefore, the safety monitoring device10sets a circumscribed circle surrounding the bounding box106box as the region of the person106.

In a case where the horizontal axis is the X axis and the vertical axis is the Z axis, the coordinates of a center point106oof the bounding box106box of the person106can be expressed as (ADx, ADz). Then, the radius of the circumscribed circle surrounding the bounding box106box is set as AD_R.

The bottom surface of a large object105ahas a rectangular shape with four points a, b, c, and d as vertices. The safety monitoring device10obtains (calculates) a distance between a line segment ab of the large object105aand the center point106oof the person106, a distance between a line segment bc of the large object105aand the center point106oof the person106, a distance between a line segment cd of the large object105aand the center point106oof the person106, and a distance between a line segment da of the large object105aand the center point106oof the person106. Then, the safety monitoring device10sets the minimum value of the obtained (calculated) values as a distance Dmin between each of the line segments ab, bc, cd, and da and the center point106oof the person106.

The safety monitoring device10can obtain (calculate) the distance between the person106and the large object105aby (Dmin−AD_R).

It is sufficient if the safety monitoring device10calculates the distances of all the combinations between all the detected persons106and the large object105ain each frame, and issue a warning when the calculated distances are shorter than the designated threshold value distance.

At that time, the safety monitoring device10may issue different types of warnings depending on the calculated distances. For example, in a case where the calculated distances are threshold value distances Th1, Th2, and Th3 (not illustrated) described below, the warnings described below can be issued. Note that the threshold value distances Th1, Th2, and Th3 (not illustrated) have a relationship of Th1>Th2>Th3.(1) In the case of (Dmin−AD_R)<Th1, the safety monitoring device10sets the level as a caution level. In this case, the safety monitoring device10causes the warning lamp43to light up yellow.(2) In the case of (Dmin−AD_R)<Th2, the safety monitoring device10sets the level as a warning level. In this case, the safety monitoring device10causes the warning lamp43to light up red.(3) In the case of (Dmin−AD_R)<Th3, the safety monitoring device10sets the level as a danger level. In this case, the safety monitoring device10causes the warning lamp43to blink in red and causes the speaker44to sound an alarm sound.

Hereinafter, the operation of the safety monitoring device10will be described with reference toFIG.16.FIG.16is a flowchart illustrating an operation at the time of person and large object sensing and alarm activation.

As illustrated inFIG.16, the operation at the time of person and large object sensing and alarm activation is different from the operation at the time of person sensing and alarm activation illustrated inFIG.14on the points described below.(1) Point that determination processing of step S121is performed after step S120.(2) Point that the processing proceeds to step S125when “Yes” is determined in the determination of step

S121.(3) Point that the safety monitoring device10performs the processing of step S145and step S150when “No” is determined in the determination of step S121.(4) Point that step S135is deleted, and instead, the processing of step S155is performed after step S130or step S150.

Except for these differences, the operation at the time of person and large object sensing and alarm activation illustrated inFIG.16is the same as the operation at the time of person sensing and alarm activation illustrated inFIG.14.

Specifically, as illustrated inFIG.16, after step S120, in step S121, the alarm activation analysis unit16fof the moving body recognition analysis unit16of the safety monitoring device10determines whether or not the object is a person.

When it is determined in step S121that the object is a person (“Yes”), the processing proceeds to step S125. In this case, the processing from step S125to step S130is performed similarly to the operation at the time of person sensing and alarm activation illustrated inFIG.14. After step S130, the processing proceeds to step S155.

On the other hand, when it is determined in step S121that the object is not a person (“No”), the processing proceeds to step S145.

In step S145, the alarm activation analysis unit16fof the moving body recognition analysis unit16of the safety monitoring device10confirms that the moving body conforms to the determination conditions of large object from the size (width, depth, height) of the bounding box, and adds (sets) a flag (large object flag) representing a large object to the moving body. In this way, the safety monitoring device10recognizes a large object with respect to the moving body.

Next, in step S150, the alarm activation analysis unit16fof the moving body recognition analysis unit16of the safety monitoring device10senses the size of the same moving body in a next frame. Then, when the size of the moving body continuously conforms to the determination conditions of large object for the designated number of frames, the alarm activation analysis unit16fsets the attribute of the moving body to “large object”. After step S150, the processing proceeds to step S155.

In step S155, the alarm activation analysis unit16fof the moving body recognition analysis unit16of the safety monitoring device10checks whether or not the moving body has entered the alarm activation target area. Then, when both the “person” and the “large object” enter the alarm activation target area, the alarm activation analysis unit16fmeasures a distance between the “person” and the “large object”, and issues a warning that the “person” is in a dangerous state when the distance is shorter than a certain distance.

FIG.17is an explanatory diagram illustrating an example of a warning screen when the person106sensed by the lidar51enters an alarm activation target area (entry prohibited area) of a camera video. The safety monitoring device10projects the three-dimensional data acquired from the lidar51onto the two-dimensional video data (camera video) acquired from the monitoring camera61. The safety monitoring device10is set such that the three-dimensional data and the video data match in positional relationship. Then, as illustrated inFIG.17, the safety monitoring device10senses a moving body, adds the bounding box106box to the moving body, and recognizes the moving body as the person106on the basis of the size of the bounding box106box. Note that it is sufficient if the safety monitoring device10changes the color of the bounding box106box of the corresponding person106in the camera video to red or superimpose a red frame on the portion of the corresponding person106in order to issue a warning when the person106enters the alarm activation target area (entry prohibited area).

Further, the safety monitoring device10can be configured to constantly record (video-record) the video data (camera video) acquired from the monitoring camera61in the storage unit30. Such a safety monitoring device10can create an alarm activation list by leaving the video data (camera video) at the moment of issuing the warning in the record without erasing the video data. In this case, the safety monitoring device10can designate and reproduce (display on the display42) the video data (camera video) at the time of alarm activation for a necessary time on the basis of the alarm activation list.

Further, the safety monitoring device10can be configured to operate as illustrated, for example, inFIG.18A or18B.FIGS.18A and18Bare operation explanatory diagrams of the safety monitoring device10.

As illustrated inFIG.18A, the safety monitoring device10acquires three-dimensional data (distance measurement data) measured by the lidar51with the lidar data acquisition unit15. The lidar data acquisition unit15outputs the acquired three-dimensional data (distance measurement data) to the moving body recognition analysis unit16.

In the moving body recognition analysis unit16, the alarm activation analysis unit16fsenses the moving body on the basis of the alarm activation function designated for each area. Then, the alarm activation analysis unit16fdetermines whether or not any area is in a situation where a warning should be issued. In a case where there is an area where a warning should be issued, the moving body recognition analysis unit16outputs a lighting instruction signal to the warning lamp so as to light the lamp according to the level of the warning. In response to this, the warning lamp lights the lamp Note that the warning lamp may include a plurality of lamps and may be designated for each area.

Further, the moving body recognition analysis unit16outputs a result of sensing of the moving body to the video processing unit18via the result output unit17. In response to this, the video processing unit18creates the display screen100(seeFIG.4A) to be displayed on the display42. At that time, the video processing unit18may perform arbitrary camera video processing such as, for example, superimposing a person sensing frame on the camera video.

Further, as illustrated inFIG.18B, the safety monitoring device10may be configured to include an image recognition unit18afor camera video processing in the video processing unit18. On the basis of the video data (camera video) photographed by the monitoring camera61, the image recognition unit18aperforms person determination by face identification, sex determination, posture determination, and the like with respect to the person recognized by the moving body recognition analysis unit16. In this way, the safety monitoring device10can determine the danger level more finely.

<Regarding Smoothing of Bounding Box>

The safety monitoring device10preferably has a bounding box smoothing function.FIG.19is an operation explanatory diagram of the safety monitoring device10.

As illustrated inFIG.19, the safety monitoring device10outputs the three-dimensional data (distance measurement data) acquired from the lidar51with the lidar data acquisition unit15to the moving body recognition analysis unit16. In the moving body recognition analysis unit16, the foreground data extraction unit16aextracts foreground data by taking a difference between three-dimensional data (distance measurement data) as input data and background data acquired in advance.

Next, the clustering processing unit16bextracts point cloud data having a certain cluster from the foreground data and sets the point cloud data as a moving body candidate. Then, the clustering processing unit16bverifies continuity with all the frames from among the point cloud data set as the moving body candidate, regards point cloud data having continuity as a moving body, and allocates an ID. Thereafter, the bounding box generation unit16cgenerates a bounding box for the point cloud data regarded as a moving body.

The angle θ between the frames on the (x, z) plane of the bottom surface of the created bounding box is a discrete value for each Δθ. Thus, the angle smoothing unit16dperforms averaging processing with the number of frames or a processing method specified in advance. At that time, the safety monitoring person may make a change or the like in the number of frames and the processing method using the operation unit41.

Further, the safety monitoring device10simultaneously displays the video data (camera video) and the bounding box obtained as the point cloud data on the display42. Therefore, the safety monitoring device10can show the safety monitoring person the portion without point cloud data on the back side with respect to the lidar51to such an extent that the approximate outer shape is visually visible by the bounding box.

For example, a case where the input data from the lidar51is 10 fps (frames/second) will be described.FIG.20is an explanatory diagram illustrating an example of angle smoothing processing.FIG.20illustrates an example of a case where the angle θ of the bounding box of the same object is 4θ=6° for each frame, and the angle θ is obtained by a simple average formula of five frames.

The obtained angle θ is discretely any one angle of 0°, 6°, 12°, 18°, . . . , 78°, and 84°. In the illustrated example, an example in which the angle θ gradually bends from 6° to 12° is illustrated. In the illustrated example, the angle θ is either 6° or 12° due to the variation of the point cloud. The movement of the bounding box in such a state visually appears to be lagging.

Here, in a case where the angle of an i-th frame is θ(i) and the average angle is θave(i), the simple average formula of five frames is Formula (1) described below.
θave(i)=(θ(i−4)+θ(i−3)+θ(i−2)+θ(i−1)+θ(i))÷5  (1)

The average angle illustrated inFIG.20is an angle of a calculation result according to Formula (1) above. The average angle illustrated inFIG.20is an angle of smooth change Therefore, the movement of the bounding box has a stable appearance and is relatively close to the actual movement of the moving body (for example, a bus or the like). However, for example, since angle data between the past four frames and the current frame is used, the movement of the bounding box is a slightly delayed motion. The smoothness of the movement of the bounding box can be improved by increasing the number of smooth frames, but the responsiveness is lowered. Thus, it is preferable to select the number of smooth frames according to the situation of a moving body to be detected. For example, since a bus, a truck, or the like hardly turns suddenly, a larger number of frames may be selected, and a smaller number of frames may be selected for a car that often turns such as a forklift.

FIG.21is an explanatory diagram illustrating an example of angle smoothing processing similarly toFIG.20. However, whileFIG.20illustrates an example of a case of being obtained by the simple average formula of five frames,FIG.21illustrates an example of a case of being obtained by a weighted average formula of five frames weighted by Formula (2) described below. The weighted average formula is a formula that puts weight on a current frame.

Here, in a case where the angle of an i-th frame is θ(i) and the average angle is θave(i), the weighted average formula of five frames weighted by the weighted average formula is Formula (2) described below.
θave(i)=(1×θ(i−4)+2×θ(i−3)+3×θ(i−2)+4×θ(i−1)+5×θ(i))±15  (2)

FIG.22is an explanatory diagram illustrating a setting example of angle smoothing processing.FIG.22illustrates an example of a case where averaging processing is set by a graphical user interface (GUI), which is not illustrated, by operating the operation unit41.

In the example illustrated inFIG.22, the display screen is configured to be able to set the execution of the angle smoothing processing to “Yes” or “No” by using a radio button. Further, the display screen is configured to be able to set execution of the angle smoothing processing in a case where the aspect ratio (width/depth) of the moving body is less than a certain value. Further, the display screen is configured to be able to set that the angle smoothing processing is executed when the size (width), which is an execution target, is less than a certain size. Note that as the value of the size (width), which is an execution target, is set to be larger, the angle smoothing unit16dfrequently executes the angle smoothing processing with respect to the moving body. Further, the display screen is configured to be able to set execution of the angle smoothing processing in a case where the number of averaged frames is less than a certain number. In the illustrated example, five frames is selected as the number of averaged frames, but this value can be freely changed. Further, the display screen is configured such that either the simple average or the weighted average can be selected by using a radio button.

Note that when the center coordinates of the bottom surface of the bounding box in the i-th frame are (Cx(i), Cz(i)), the angle smoothing processing of Cxave(i) and Czave(i) is performed by Formulae (3) and (4) below.
Cxave(i)=(Cx(i−4)+Cx(i−3)+Cx(i−2)+Cx(i−1)+Cx(i))÷5  (3)
Czave(i)=(Cz(i−4)+Cz(i−3)+Cz(i−2)+Cz(i−1)+Cz(i))÷5  (4)

Note that, in the above calculation example, the angle smoothing unit16dperforms the angle smoothing processing using the simple average, but may perform the angle smoothing processing using the weighted average. Further, the safety monitoring device10may perform smoothing processing of the center coordinates simultaneously with angle smoothing processing using a graphical user interface, which is not illustrated. Further, the safety monitoring device10may be provided with a graphical user interface, which is not illustrated, independently of angle.

<Regarding Exclusion of Video Data in which Important Mark is Set at Time of Automatic Video-Record Erasure>

It is sufficient if the safety monitoring device10is configured to constantly video-record (store in the storage unit30) the video data (camera video) acquired from the monitoring camera61and erase the video data after a lapse of a certain period.FIG.23is an operation explanatory diagram of such a safety monitoring device10.

As illustrated inFIG.23, the safety monitoring device10converts the video data (camera video) acquired from the monitoring camera61into manageable data such as MxPEG by the video processing unit18and outputs the data to the recording management unit19.

The recording management unit19records the converted data in a storage means (the storage unit30or an external recording medium, which is not illustrated) by dividing the converted data into files (file data) at predetermined time intervals so that the database management is enabled. Further, the recording management unit19acquires a file from the video processing unit18on the basis of an instruction from the safety monitoring person, and outputs the acquired file to the video display processing unit20. Further, the recording management unit19reads a file of a designated time from a file group recorded in the storage means (the storage unit30or the external recording medium, which is not illustrated), and outputs the file to the video display processing unit20. The video display processing unit20creates an image on the basis of the file and displays the image on the display42.

Further, the safety monitoring device10acquires a sensing signal of an object in the measurement space from a safety sensor50such as a lidar, a camera, a human sensor, or a touch sensor with the safety sensor input unit21, and outputs the sensing signal to the safety monitoring unit22. The safety monitoring unit22senses entry of a person, entry of a dangerous object, or the like on the basis of the sensing signal, and issues an event. That is, the safety monitoring unit22monitors entry of an object into a monitoring region of the safety sensor50. Then, the safety monitoring unit22outputs the event to the event list processing unit23. At this time, the safety monitoring unit22sets the time when the object enters the monitoring region of the safety sensor50as the start time of the event, and sets the time when the object leaves the monitoring region or the time when a certain time has elapsed from the start time of the event as the end time of the event. Then, the safety monitoring unit22notifies the event list processing unit23of the start time and the end time of the event.

In response to this, the event list processing unit23creates and updates the event list, and displays a display screen110illustrated, for example, inFIG.24, on the display42.FIG.24is an explanatory diagram illustrating an example of the display screen110. In the example illustrated inFIG.24, the display screen110has a configuration in which the video of each measurement space photographed by the monitoring camera61and the updated event list are arranged at the same time. The safety monitoring person operates the operation unit41while viewing the display screen110. The event list processing unit23edits the event in accordance with the operation of the safety monitoring person.

Note that, in the example illustrated inFIG.24, the display screen110has a configuration in which a video display unit111that displays a video is arranged on the left side, and an event list display unit112that displays an event list is arranged on the right side.

In the example illustrated inFIG.24, the video display unit111is divided into four screens. The safety monitoring device10is configured to be able to connect up to four monitoring cameras61, and displays the video acquired from each monitoring camera61on an arbitrary screen of the video display unit111. Note that the safety monitoring device10may selectively use the four screens of the video display unit111according to the operation. For example, the safety monitoring device10may display a live video from the monitoring camera61on the upper left screen of the video display unit111, and display a video-record reproduction video from the storage means on the upper right screen of the video display unit111.

Further, the safety monitoring device10is configured to be able to reproduce the video relating to each event of the event list displayed on the event list display unit112on the video display unit111. For example, the safety monitoring person selects an event to be reproduced from the event list displayed on the event list display unit112, and double-clicks the event with a mouse or drags the event to a reproduction screen on the video display unit111. In this way, the safety monitoring device10reproduces the video related to the selected event on the video display unit111.

FIG.25is an explanatory diagram illustrating an example of an event list. In the example illustrated inFIG.25, an event list120is configured to include, from the left, an important mark field121, a time stamp field122, a camera name field123, a type field124, and a description field125.

The important mark field121is a field to which an important mark indicating an important event is added. In the illustrated example, the important mark has a star shape, but the shape of the important mark is not limited to the star shape and may be any shape.

The time stamp field122is a field for displaying the start time of an event.

The camera name field123is a field for displaying the name of the camera that has photographed a video.

The type field124is a field for displaying an event type (Intruder, CUSTOM, or the like).

The description field125is a field for displaying description regarding an event.

Note that the event list120can be configured to display the end time, an occurrence cause, and the like of an event in addition to the above.

The event list120is configured such that the important mark can be turned on/off by clicking the important mark field121of an arbitrary event with a mouse. The safety monitoring person reproduces the video of an arbitrary event, performs video confirmation, and adds an important mark to the important mark field121of the event when determining that an incident (important occurrence) has occurred in the video being reproduced. The safety monitoring device10stores the video related to the event to which the important mark is added so that the video is not erased.

The safety monitoring device10can perform editing processing on an event by the safety monitoring person arbitrarily selecting one event from the event list120and performing an editing operation on an event editing screen130illustrated inFIG.26.FIG.26is an explanatory diagram illustrating an example of the event editing screen130.

In the example illustrated inFIG.26, the event editing screen130is configured to include a camera name field131, a user name field132, an occurrence date and time field133, an end date and time field134, a description field135, a type field136, a mark field137, a cause field138, a measure field139, and an importance field140.

The camera name field131is a field for displaying the name of the camera that has photographed a video.

The user name field132is a field for displaying a user name.

The occurrence date and time field133is a field for displaying the occurrence date and time of an event.

The end date and time field134is a field for displaying the end date and time of an event.

The description field135is a field for inputting description regarding an event.

The type field136is a field for inputting an event type (Intruder, CUSTOM, or the like).

The mark field137is a field for displaying the type of an important mark.

The cause field138is a field for inputting the cause of an incident that has occurred.

The measure field139is a field for inputting a measure for an incident that has occurred.

The importance field140is a field for inputting the importance of an incident that has occurred.

The event editing screen130illustrated inFIG.26has contents reflecting detailed data of an event to which the important mark illustrated inFIG.25is added. In the example illustrated inFIG.26, the occurrence date and time (start time) of an event displayed in the occurrence date and time field133is 2020/01/17 12:05:41. Further, the end date and time (end time) of an event displayed in the end date and time field134is 2020/1/17 12:11:20. In the case of reproducing the video data of the event, it is sufficient if the safety monitoring person takes a certain time (for example, 10 seconds before and after) longer than the occurrence date and time (start time) and the end date and time (end time) of the event and reproduce the video data. For example, it is sufficient if the safety monitoring person edits the occurrence date and time field133and the end date and time field134on the event editing screen130, sets the occurrence date and time of the event to 2020/1/17 12:05:31, sets the end date and time of the event to 2020/1/17 12:11:30 and reproduce the video. Note that a certain time (for example, 10 seconds before and after) applied to change the date and time can be arbitrarily changed.

The safety monitoring person can edit the occurrence date and time (the start time of the event) in the occurrence date and time field133and the end date and time (the end time of the event) in the end date and time field134from the event editing screen130.

Further, although not illustrated, the safety monitoring device10can limit the end time of the event to the time up to “the start time of the event+the longest time of the event” by setting the longest time of the event.

Further, the safety monitoring device10can be configured to operate as described below. For example, a monitoring area is set in advance in the lidar51, a time when a person enters the monitoring area is set as the occurrence date and time (start time) of the event, and a time when the person leaves the monitoring area is set as the end date and time (end time) of the event. Then, the safety monitoring device10can acquire the occurrence date and time data and the end date and time data of the event from the lidar51at the time of event registration, and record and manage the acquired date and time data in the storage unit30.

Further, the safety monitoring device10can limit the end time of the event to the time up to “the start time of the event+the longest time of the event” in a case where the moving body stays in the monitoring area for a long time.

Further, the safety monitoring device10may be configured to be able to edit other items from the event editing screen130.

Further, the safety monitoring device10may be configured to be able to create a new event by reproducing a video at an arbitrary time by using the event editing screen130.

FIG.27is an explanatory diagram illustrating an example of a save format of video data in a storage means.FIG.27illustrates an example in which SQLite, which is one of open source relational database management systems (RDBMS), is used for file management.

As illustrated inFIG.27, in the storage unit30, a database management file called “archive.db” is created under “Video folder”, and a date folder indicating a date (for example, “2020 Jan. 17 folder”, “2020 Jan. 18 folder”, “2020 Jan. 19 folder”), or the like is created once a day.

For example, a video data file compressed in H.264 format is created under each date folder. In the example illustrated inFIG.27, for example, “video_20200117_000000.mp4”, “video_20200117_001000.mp4”, and the like are created as the video data file. “video_20200117_000000.mp4” is a video data file for 10 minutes from 2020/1/17 0:00:00. Further, “video_20200117_001000.mp4” is a video data file for 10 minutes from 2020/1/17 0:10:00. In the present embodiment, description is given assuming that a video data file is created at intervals of 10 minutes as described above. As a result, 144 files of “24 hours×(6 files/1 hour)” are created per day.

In the case of a normal XGA format (1024×768 pixels), the data amount of video-recorded data for 10 minutes is about 500 MB, and the data amount of video-recorded data for one month is about 2 TB. Therefore, when a large capacity hard disk drive (HDD) is used as a recording means (the storage unit30or the external recording medium, which is not illustrated), the safety monitoring device10can constantly save the video-recorded data for about 30 days.

In the present embodiment, the safety monitoring device10constantly saves video-recorded data (video data file) for 30 days, and automatically erases the video-recorded data (video data file) for one day, which exceeds 30 days from video-recording, once a day. At this time, the safety monitoring device10automatically deletes the event, which exceeds 30 days from issuance, from the event list.

However, the safety monitoring device10saves the video-recorded data (video data file) related to the event to which the important mark illustrated inFIGS.25and26is added without erasing the video-recorded data.

For example, if the important mark is not added to the video-recorded data in which the time stamp illustrated inFIG.25is 2020/01/17 12:05:41, the video-recorded data is automatically erased when the save period of 30 days is exceeded. However, an important mark is added to the video-recorded data in which the time stamp illustrated inFIG.25is 2020/01/17 12:05:41. Therefore, even at the timing of the automatic erasure (that is, for example, even if the save period from video-recording exceeds 30 days), the video-recorded data is continuously saved without being erased. Further, the event corresponding to the video-recorded data to which such an important mark is added is left without being deleted from the event list.

The safety monitoring device10checks the event to which the important mark is added at the time of automatic erasure of the video-recorded data once a day, obtains the reproduction time of the event to which the important mark is added, and saves the video-recorded data (video data file) including the time without erasing. For example, here, it is assumed that the safety monitoring device10saves the video-recorded data (video data file) including 2020 Jan. 17 12:05:31 to 12:11:30 without erasing the video-recorded data with respect to the video-recorded data in which the time stamp illustrated inFIG.25is 2020 Jan. 17 12:05:41.

In the example illustrated inFIG.27, the video-recorded data including 2020/1/17 12:05:31 to 12:11:30 includes two files: “Video_20200117_120000.mp4” and “Video_20200117_121000.mp4”. As illustrated in FIG.28, even at the time of automatic erasure of video-recorded data, the safety monitoring device10leaves two files related to the event to which the important mark is added, and automatically erases the other files.

Note that the video data file that has not been erased even at the time of automatic erasure of video-recorded data is erased at the time of automatic erasure of video-recorded data performed subsequently when the important mark is turned off (deleted) from the event by the subsequent editing. Further, in this case, the event for which the important mark is turned off (deleted) is also deleted from the event list.

<Main Features of Safety Monitoring Device>

(1) As illustrated inFIG.2, the safety monitoring device10according to the present embodiment is configured to include the area setting unit12capable of setting a plurality of alarm activation target areas with respect to the sensing target area of the safety sensor, the alarm activation type setting unit13capable of setting the type of an arbitrary alarm activation function with respect to each of the alarm activation target areas, and the moving body recognition analysis unit16that recognizes the motion of the moving body on the basis of sensor data measured by the safety sensor.

At a construction site, a manufacturing site, and the like, various management areas are assumed, such as an area where a heavy machine (for example, a crane) works, an area where a work vehicle such as a forklift moves around, and an area where all moving bodies should not enter for vibration suppression, security, and the like. The safety monitoring device10according to the present embodiment can set such various management areas as an alarm activation target area, and can set a plurality of alarm activation target areas with respect to the sensing target area of the safety sensor. Further, the safety monitoring device10can set the type of an arbitrary alarm activation function for each of the alarm activation target areas. That is, the safety monitoring device10can independently set different alarm activation functions for each of the plurality of alarm activation target areas. Therefore, the safety monitoring device10can realize an optimum monitoring method. Further, the safety monitoring device10can satisfactorily grasp a relationship between a person and an object.

(2) As illustrated inFIG.4A, the safety monitoring device10according to the present embodiment includes the sensor data acquisition unit (lidar data acquisition unit15) that acquires three-dimensional data (distance measurement data) as sensor data from the lidar51, which is a safety sensor. The moving body recognition analysis unit16may be configured to extract a moving body from the sensing target area on the basis of the three-dimensional data and sense traveling of the moving body to each of the alarm activation target areas.

Such a safety monitoring device10according to the present embodiment can set a plurality of areas with measurement accuracy under the same condition by using the safety sensor (for example, the lidar51) capable of measurement of three-dimensional data. Therefore, the safety monitoring device10can ensure stable sensing accuracy. Further, in a case where a condition for monitoring is changed, the safety monitoring device10can cope with only by changing the setting of the area without a physical change such as reattachment of the sensor.

(3) Preferably, the alarm activation type setting unit13of the safety monitoring device10according to the present embodiment may be configured to be able to select any two or more of: a function of issuing a warning to all moving bodies; a function of issuing a warning only to a person; a function of issuing a warning in a case where a distance between a person and a large object larger than the person is within a certain distance; a function of issuing a warning in a case where a person enters the traveling direction of a large object; an alarm activation function of issuing a warning only to a moving body at a certain speed or higher; an alarm activation function of issuing a warning only to a moving body at a certain speed or lower; a function of combining two or more arbitrary functions to obtain a logical sum; and a function of combining two or more arbitrary functions to obtain a logical product, as the type of the alarm activation function.

Such a safety monitoring device10according to the present embodiment can use various types of alarm activation functions according to the operation. For example, in an area where a heavy machine (for example, a crane) moves, it is preferable to issue a warning when a person approaches the heavy machine. Further, in an area where a work vehicle such as a forklift moves, it is preferable to perform monitoring so that a worker does not enter. Furthermore, in an area where a heavy machine or the like running on a rail moves, it is preferable to issue a warning when a person enters the traveling direction. Further, in an area where all moving bodies should not enter for vibration suppression, security, or the like, it is preferable to monitor the movement of all moving bodies. Further, depending on the area, there is an area in which it is preferable to perform monitoring so as to issue a warning when the moving body enters beyond the speed limit. The safety monitoring device10can realize appropriate warning issuance in such an area. Further, depending on the area, by setting a condition for issuing a warning to a logical product condition of a certain speed or more of a person and a large object (for example, a car), the safety monitoring device10can be set to issue a warning when the large object (for example, a car) enters the area beyond the speed limit. Further, by setting a condition for issuing a warning to a logical sum condition of speeds of a person and a large object, the safety monitoring device10can also be set to issue a warning when the large object enters the area.

(4) As illustrated inFIGS.5and6, the safety monitoring device10according to the present embodiment may be configured such that the type of the alarm activation function can be designated for each area via a user interface.

In a construction site, a manufacturing site, and the like, the environment changes from moment to moment according to the progress of work. Therefore, it is preferable that the setting of the area and the selection of the alarm activation function can be arbitrarily and easily changed according to the situation of the site. The safety monitoring device10according to the present embodiment can realize an environment in which the setting can be arbitrarily and easily changed via a user interface.

(5) As illustrated inFIGS.12to14, the moving body recognition analysis unit16of the safety monitoring device10according to the present embodiment may be configured to extract point cloud data representing a moving body from three-dimensional data measured with the lidar51, which is a safety sensor, create a bounding box having a size necessary for surrounding the point cloud data, determine a minimum value and a maximum value of each of five pieces of data, that is, the width, the depth, and the height of the bounding box, and the height of the bottom surface and the height of the top surface of the bounding box, as the size of a person by using any one or more or all of the five pieces of data, determine that the size of the moving body is within the determined range as the determination condition of person, and identify the moving body as a person when the size of the moving body continuously conforms to the determination condition for an arbitrarily determined certain time.

Such a safety monitoring device10according to the present embodiment can sense a person by a simple method without using a known image processing method or a machine learning-based method by sensing from the size of the bounding box.

Note that the point cloud data of the moving body measured by the lidar51is subjected to clustering processing and processed as one mass (object). At that time, since a large object (for example, a car or the like) often has an uneven shape, there is a possibility that the large object is not stably recognized as one object but is recognized as being divided into a plurality of masses. In this case, the size of one divided mass may be the same as the size of a person. Accordingly, when the attribute of the moving body is simply determined only by the size of the person, there is a possibility that a large object is erroneously determined as a person. On the other hand, since the state in which the large object is divided into a plurality of masses and recognized is often resolved in a short period, the safety monitoring device10according to the present embodiment can determine that the moving body (mass) is a person when the state in which the size of the moving body (mass) is determined to be the size of the person continues for a certain time. Even when a large object is erroneously determined as a person on the basis of the size of the moving body (mass), such a safety monitoring device10can greatly reduce erroneous determination of the large object as a person by determining whether or not the moving body (mass) is a person on the basis of both the size of the moving body (mass) and the duration of the size.

Further, in a construction site, a manufacturing site, or the like, a worker walks on the ground. In such an environment, a phenomenon in which a large object is divided into a plurality of masses and recognized, and the size of one divided mass becomes the size of a person often occurs at a place higher than the ground. Therefore, the safety monitoring device10according to the present embodiment adds, as a condition, that the height of the bottom surface of the bounding box is equal to or less than a certain height. Further, since the height of a person is about 2 m at the highest, the safety monitoring device10according to the present embodiment can prevent a moving body other than a person from being erroneously recognized as a person by determining that the moving body is not a person when the height of the top surface of the bounding box is higher than an arbitrarily determined certain height.

Note that, since the safety monitoring device10according to the present embodiment senses a person according to the size of the bounding box, it is possible to include a moving body other than a person by adjusting the size. For example, the safety monitoring device10can also adjust the size of the bounding box so as to include a person carrying a load, a plurality of persons in a lump, a small car, an object carried by a crane, and the like.

(6) As illustrated inFIG.12, the moving body recognition analysis unit16of the safety monitoring device10according to the present embodiment identifies a large object larger than a person by using any one or more or all of the five pieces of data including the width W, the depth D, and the height H of the bounding box, and the height Hb of the bottom surface and the height Ht of the top surface of the bounding box. Then, as illustrated inFIG.15, the moving body recognition analysis unit16may be configured to calculate, as the distance between the person and the large object, a distance (Dmin−AD_R) obtained by subtracting the radius of the circumscribed circle of the bottom surface or the top surface of the bounding box of the person from the minimum value of the distance between the center point106oof the bottom surface or the top surface of the bounding box106box of the person and each side of the bottom surface or the top surface of the bounding box105box of the large object, and issue a warning when the distance between the person and the large object becomes smaller than an arbitrarily determined certain distance.

In obtaining the distance between a person and a large object, in a three-dimensional distance, there is a case where a dangerous object hangs over the head of the person, such as a crane vehicle. In such a case, even when the distance between the person and the crane vehicle is larger than a predetermined distance, there is a possibility that it is not safe. It is desirable that the safety monitoring device10according to the present embodiment does not overlook such a case. Thus, it is sufficient if the moving body recognition analysis unit16of the safety monitoring device10according to the present embodiment calculates the distance using the bottom surface (or the top surface) of each bounding box. As illustrated inFIG.15, the moving body recognition analysis unit16obtains a distance from the center to each line segment of the bottom surface of the bounding box105box of the large object with reference to the center point106oof the bottom surface of the bounding box106box of the person, and calculates a distance (Dmin−AD_R) obtained by subtracting the radius of the circumscribed circle of the bottom surface of the bounding box106box of the person from the minimum value as the distance between the person and the large object. In this way, the safety monitoring device10can calculate distances of all combinations between a plurality of persons and a large object present in the alarm activation target area (entry prohibited area), and issue a warning when the distances of the combinations become smaller than a certain distance. Such a safety monitoring device10can perform monitoring without omission so as not to overlook a risk even when the risk is easily overlooked in a case where a dangerous object hangs over the head of a person such as, for example, a crane vehicle as described above.

(7) As illustrated inFIGS.15to18B, the moving body recognition analysis unit16of the safety monitoring device10according to the present embodiment may be configured to issue a warning when a time during which a distance between a person and a large object becomes smaller than a certain distance continues for an arbitrarily determined certain time.

In a construction site, a manufacturing site, or the like, when a person approaches a large object and stays near a certain distance from the large object, there is a possibility that a large number of warnings is issued continuously. Thus, the safety monitoring device10according to the present embodiment can prevent a large number of warnings from being issued continuously by generating a warning in the case of continuation for a certain time.

(8) The moving body recognition analysis unit16of the safety monitoring device10according to the present embodiment may be configured to stop the warning when the distance between the person and the large object becomes larger than a certain distance.

When a person approaches a large object and then moves away from the large object and the distance between the person and the large object becomes larger than a certain distance, the person is in a state of retreating to a safety zone. Thus, it is sufficient if the safety monitoring device10according to the present embodiment stops the warning when the distance between the person and the large object becomes larger than a certain distance.

(9) The moving body recognition analysis unit16of the safety monitoring device10according to the present embodiment may be configured to stop a warning when a time during which a distance between a person and a large object becomes larger than a certain distance continues for arbitrarily determined certain or more time.

In a case where a person approaches a large object and then moves away from the large object and the person stays near a certain distance, there is a possibility that a large number of warnings is continuously issued before the person retreats to a safety zone. Thus, it is sufficient if the safety monitoring device10according to the present embodiment stops the warning when the time during which the distance between the person and the large object becomes larger than a certain distance continues for arbitrarily determined certain or more time.

(10) The moving body recognition analysis unit16of the safety monitoring device10according to the present embodiment may be configured to change the alarm level according to the distance between the person and the large object.

For example, when a person approaches an object, such a safety monitoring device10according to the present embodiment can change a warning level such as a caution level, a warning level, or a danger level according to the approach distance. At that time, for example, the safety monitoring device10may change an alarm lamp from yellow to red when the level of warning changes from the caution level to the warning level, and further may sound an alarm sound (buzzer sound) through the speaker44when the level of warning changes from the warning level to the danger level.

(11) The moving body recognition analysis unit16of the safety monitoring device10according to the present embodiment may be configured to calculate the speed of the moving body from the distance over which the center position of the bounding box has moved for a certain time.

Such a safety monitoring device10according to the present embodiment can grasp the timing at which the moving body enters the area by specifying the speed of the moving body from the distance over which the center position of the bounding box has moved for a certain time (for example, 0.5 seconds).

(12) The safety monitoring device10according to the present embodiment may be configured to include the video processing unit18that superimposes the position of the moving body sensed by the safety sensor (for example, the lidar51) on the video data.

For example, in a case where the moving body is a person for which a warning is issued, such a safety monitoring device10according to the present embodiment superimposes a warning mark as the position of the moving body on the corresponding person, thereby allowing the safety monitoring person to visually recognize the danger quickly. In this way, the safety monitoring person can quickly take some measures.

(13) As illustrated inFIG.2, the safety monitoring device10according to the present embodiment includes the recording management unit19that records and manages video data obtained by photographing the whole or a part of the sensing target area in the storage unit30, and the event list processing unit23that adds an event including alarm activation data to an event list and manages the event when a warning is issued. The recording management unit19constantly records the video data in the storage unit30. Then, when an arbitrary alarm activation portion is designated from the event list, the safety monitoring device10may be configured such that the event list processing unit23displays video data of a certain time before and after of the designated alarm activation portion on the display42(display unit).

Such a safety monitoring device10according to the present embodiment records and saves the time point of alarm activation and creates an event list so that the reproduction portion can be designated so as to allow the safety monitoring person to grasp the behavior of the person to which a warning has been issued. Then, when an arbitrary event is designated, the safety monitoring device10can reproduce the video data of the time before and after corresponding to the designated event. In this way, the safety monitoring person can verify the danger degree by confirming the video data of the time before and after corresponding to the designated event. Further, in this way, the safety monitoring person can also provide the workers at the site with risk prediction education.

(14) The event list processing unit23of the safety monitoring device10according to the present embodiment may be configured to be able to filter and display alarm activation data (events including alarm activation data) for each type with respect to the event list displayed on the display42(display unit).

Such a safety monitoring device10according to the present embodiment can filter and display alarm activation data (events including alarm activation data) for each type. Therefore, for example, when only the function of issuing a warning only to a person is extracted as the filtering, the safety monitoring device10can easily extract video-recorded data suitable for the condition. Further, the safety monitoring device10can facilitate confirmation work of the video data by the safety monitoring person by managing and saving the alarm activation data (events including alarm activation data) for each alarm activation type. Further, the safety monitoring device10can, for example, limit video-recording of camera video to only when an alarm is issued. Furthermore, the safety monitoring device10can reduce the save capacity by selectively leaving a certain time before and after in the record according to the type of the alarm activation function. Further, the safety monitoring device10can facilitate confirmation work of the video data by the safety monitoring person by managing and saving the video data for each alarm activation type.

(15) As illustrated inFIG.2, the safety monitoring device10according to the present embodiment includes the recording management unit19that records and manages the video data of the sensing target area in the storage unit30. As illustrated inFIGS.24and25, the recording management unit19may be configured to record the video data before and after a base point in the storage unit30, the base point being a time point at which an occurrence for which a warning is issued occurs according to the type of the alarm activation function.

Such a safety monitoring device10according to the present embodiment records the video data before and after the base point in the storage unit30, the base point being a time point at which an occurrence for which a warning is issued occurs. Therefore, the safety monitoring device10can manage the video data for each event that has occurred. Further, the safety monitoring device10can facilitate confirmation work of the video data by the safety monitoring person by managing and saving the video data for each event that has occurred.

(16) The alarm activation type setting unit13of the safety monitoring device10according to the present embodiment may be configured to be able to designate processing of sensing a feature of a moving body from video data of the sensing target area as an alarm activation function.

For example, fine features of a moving body cannot be acquired only by sensing the moving body using the lidar51. Thus, the safety monitoring device10according to the present embodiment may be configured to be able to designate processing of sensing a feature of a moving body from video data. In this way, the safety monitoring device10can acquire fine features of the moving body such as human face recognition, sex, and the like from the video acquired from the camera as the alarm activation function. Since such a safety monitoring device10can set the condition for determining the attribute of the moving body in detail, it is possible to improve the accuracy of sensing the attribute of the object.

(17) The safety monitoring device10according to the present embodiment may be configured such that the processing of sensing a moving body includes any one of person identification by face recognition, sex determination, and posture determination.

Since such a safety monitoring device10according to the present embodiment can set the condition for determining the attribute of the moving body in detail, it is possible to improve the accuracy of sensing the attribute of the object.

(18) The safety monitoring device10according to the present embodiment may be configured to notify an external device (Web terminal62, monitoring camera61z, or the like) equipped with a program for performing camera video processing of warning data sensed by a set alarm activation function.

The external device (Web terminal62, monitoring camera61z, or the like) may manage the event and the video data independently of the operation of the safety monitoring device10. Further, there is a case where it is easier to manage the event and the video data when the program executed by the safety monitoring device10and the program for displaying the camera video are configured separately. Thus, the safety monitoring device10according to the present embodiment may be configured to notify an external device equipped with a program for performing camera video processing of warning data sensed by a set alarm activation function. In this way, the safety monitoring device10can cause, for example, an external device to perform camera video processing and cause the external device to create an image in which an image of warning data overlaps the camera video. At that time, it is sufficient if the safety monitoring device10converts the position data (three-dimensional data) of the moving body sensed by the safety monitoring device10into two-dimensional data of camera video and causes an external device to create an image such that the position of the camera video corresponds to the position of the image of the warning data.

(19) As illustrated inFIGS.19to22, the safety monitoring device10according to the present embodiment may be configured to include a smoothing means (angle smoothing unit16d) that smooths the angle of the bounding box.

When the bounding box is generated for each frame (for example, every 0.1 seconds), the obtained angle θ varies due to the variation in the distance measurement data, and as a result, there is a possibility that the variation increases when the distance between two objects (for example, between a person and a car) is obtained. Thus, the safety monitoring device10according to the present embodiment can stabilize a change in the angle θ resulting from the movement of the car by smoothing the angle of the bounding box with the smoothing means (angle smoothing unit16d). Therefore, the safety monitoring device10can stably obtain a distance between two objects (for example, between a person and a car). Further, the safety monitoring device10can display a visually natural motion video when displaying a video showing a moving body on the display42(display unit).

(20) The safety monitoring device10according to the present embodiment includes the video processing unit18that performs processing corresponding to the type of the alarm activation function on the three-dimensional data. As illustrated inFIGS.19to22, the video processing unit18of the safety monitoring device10according to the present embodiment may be configured to simultaneously display the point cloud data and the bounding box on the display42(display unit).

Such a safety monitoring device10according to the present embodiment can cause the safety monitoring person to visually recognize what point cloud the actual measurement data is by simultaneously displaying the point cloud data after the clustering processing and the bounding box. Further, in a case where any accident occurs, the safety monitoring device10allows the safety monitoring person to easily grasp the situation of an accident site by allowing the safety monitoring person to visually recognize the point cloud data of the accident site and the bounding box simultaneously.

(21) As illustrated inFIGS.19to22, it is sufficient if the smoothing means (angle smoothing unit16d) of the safety monitoring device10according to the present embodiment is configured to smooth and calculate the angle of the bottom surface of the bounding box.

Since the angle is discrete, the calculation amount increases and the calculation time increases when the resolution is improved when generating the bounding box. Thus, it is sufficient if the smoothing means (angle smoothing unit16d) of the safety monitoring device10according to the present embodiment is configured to smooth and calculate the angle of the bottom surface of the bounding box. The safety monitoring device10can obtain the angle with an appropriate calculation amount by smoothing the discrete angle of the bottom surface of the bounding box. Therefore, the safety monitoring device10can reduce the calculation amount and shorten the calculation time. Note that the calculation amount at the time of generating the bounding box is determined by the number of point clouds and the number of repetitions n of calculation based on the discrete angle Δθ. Further, the calculation time is determined by performance of a central processing unit (CPU) of a personal computer (PC) constituting the safety monitoring device10.

(22) As illustrated inFIGS.19to22, the angle of the bottom surface of the bounding box discretely calculated by the smoothing means (angle smoothing unit16d) of the safety monitoring device10according to the present embodiment may be configured to be arbitrarily changeable via a user interface.

Such a safety monitoring device10according to the present embodiment can select the value of the discrete angle Δθ by preparing a user interface, for example, so that the optimum discrete angle Δθ with respect to the angle θ of the bottom surface of the bounding box can be determined.

(23) As illustrated inFIGS.20and21, any one or both of the smoothing method of the smoothing means (angle smoothing unit16d) of the safety monitoring device10according to the present embodiment and the value of the smoothing time may be configured to be arbitrarily changeable via a user interface.

The angle smoothing processing is performed using past data from the present time point. At that time, the safety monitoring device10according to the present embodiment can select, for example, a method of simply performing averaging, a method of performing weighted averaging in which larger weighting is performed on those closer to the present time, or the like. In this way, the safety monitoring device10can perform smoothing processing of an angle suitable for the three-dimensional data input from the lidar51. Note that smoothing a large amount of data can improve the smoothness of the video, but the responsiveness of calculation with respect to a change in input data decreases. Therefore, the safety monitoring device10may be configured to be able to select a suitable time according to the input data in consideration of responsiveness.

(24) The smoothing means (angle smoothing unit16d) of the safety monitoring device10according to the present embodiment may be configured to smooth the center coordinates of the bottom surface of the bounding box.

The point cloud data varies for each frame. Therefore, not only the angle but also the center coordinates of the bounding box (the center coordinates of the rectangle of the bottom surface) vary. In order to suppress the variation, smoothing of the center coordinates of the bounding box is effective. Thus, the smoothing means (angle smoothing unit16d) of the safety monitoring device10according to the present embodiment may be configured to smooth the center coordinates of the bottom surface of the bounding box. Note that, here, the center coordinates of the bounding box have been described as the center coordinates of the rectangle of the bottom surface, but the center coordinates of the bounding box can be processed as the center coordinates of a cuboidal region (rectangular region). In this case, smoothing of the center point of the cuboidal region (rectangular region) is also effective.

(25) As illustrated inFIGS.23to28, the safety monitoring device10according to the present embodiment includes the recording management unit19that divides video data into files at predetermined time intervals and records the files in the storage unit30, and the event list processing unit23that creates and manages an event list selectable for each event. The event list processing unit23may be configured to perform, for the event selected by the safety monitoring person, either control not to automatically erase the file including the video within a video reproduction time range of the event or control to re-save the file in another region of the storage unit30.

Such a safety monitoring device10according to the present embodiment has data from which a video reproduction time range is known in each event, and can specify a file (video-recorded data) the video of which is to be reproduced from the data. Then, the safety monitoring device10is configured to be able to determine whether or not it is a file (video-recorded data) in which an incident (important occurrence) has occurred and to select the absence of automatic erasure when it is determined that the file should be left. In this way, the safety monitoring device10can perform management such that the file in which the incident (important occurrence) has occurred is not erased even if the file (video-recorded data) is set to be automatically erased on the basis of a determined rule. Such a safety monitoring device10can accumulate data related to similar incidents by leaving a file in which an incident (important occurrence) has occurred, and can cause the safety monitoring person to consider accident prevention measures.

(26) As illustrated inFIG.28, the event list processing unit23of the safety monitoring device10according to the present embodiment has a function of automatically erasing a file on the basis of a predetermined rule. The predetermined rule may be configured such that a certain amount of free capacity of a recording means (the storage unit30or the external recording medium, which is not illustrated) is secured or a certain time elapses.

Such a safety monitoring device10according to the present embodiment can automatically erase the file on the basis of the predetermined rule. Note that, in a case where a file is automatically erased on the basis of the predetermined rule, it is preferable that the safety monitoring device10perform management such that a certain amount of free capacity can be secured in a configuration in which the capacity of the HDD, which is the recording means, is fully utilized. Further, in the configuration in which the file is automatically erased after a lapse of a certain time within a range in which the capacity of the HDD, which is the recording means, is large, it is clear how long data remains, and it is easy to manage the data, so that the safety monitoring device10may use the data depending on the use.

(27) As illustrated inFIG.26, the event list processing unit23of the safety monitoring device10according to the present embodiment may be configured to acquire sensor data including one or both of the start time and the end time of an event occurring in the sensing target area from the safety sensor and determine a video reproduction time range of the event list.

Such a safety monitoring device10according to the present embodiment can reproduce the file (video data) in the video reproduction time range determined for each event, and thus can reproduce the file (video data) for a suitable time. Note that the safety monitoring device10can set, for example, the time when a person enters a warning issuing area as the start time of the event, and the time when the person leaves the warning issuing area as the end time of the event list. Such a safety monitoring device10can manage whether or not the person who has entered the dangerous area has evacuated from the dangerous area to a safe area. Note that the safety monitoring device10according to the present embodiment may be configured to be able to record data of one or both of the start time and the end time of the event depending on the use.

(28) As illustrated inFIGS.23to28, the event list processing unit23of the safety monitoring device10according to the present embodiment may be configured to have a function of additionally registering an arbitrary occurrence having occurred in the past as a new event in the event list.

Such a safety monitoring device10according to the present embodiment can additionally register an arbitrary occurrence having occurred in the past as a new event in the event list according to the determination of the safety monitoring person. For example, even in the case where an event the safety monitoring device10has overlooked an occurrence in the past, when the safety monitoring person views the video of the file left in the recording means and determines that an incident (important occurrence) has occurred, the safety monitoring device10can additionally register the incident as a new event in the event list. In this case, the safety monitoring device10leaves the file (video-recorded data) in which the incident (important occurrence) has occurred without erasing the file.

(29) As illustrated inFIGS.23to28, the event list processing unit23of the safety monitoring device10according to the present embodiment may be configured to have a function of editing the start time and the end time of the event.

Such a safety monitoring device10according to the present embodiment can arbitrarily edit the start time and the end time of the event (that is, the reproduction start time and the reproduction end time of the file related to the event). For example, a case where an incident (important occurrence) occurs before a person enters a dangerous area, or a case where a dangerous situation continuously occurs even after a person who has entered the dangerous area leaves the dangerous area is assumed. In such a case, the safety monitoring device10can leave the file of the time zone by changing the start time and/or the end time of the event.

(30) The safety monitoring device10according to the present embodiment includes the video processing unit18that performs processing corresponding to the type of the alarm activation function with respect to the video data of the sensing target area. As illustrated inFIGS.23to28, the video processing unit18of the safety monitoring device10according to the present embodiment may be configured to create a display screen for simultaneously displaying a video and an event list on the display42(display unit), and when a specific event is designated by the safety monitoring person from the event list via a graphical user interface, display a video including a video reproduction time range of the designated event on the display42(display unit).

Such a safety monitoring device10according to the present embodiment can reproduce a video of an arbitrary event on the basis of the event list. The safety monitoring person can efficiently confirm the degree of danger at the site by viewing the reproduced video. Such a safety monitoring device10can facilitate the management of the event by simultaneously displaying the video and the event list on the display42(display unit), and can improve the work efficiency such as the event display operation and the event editing.

(31) As illustrated inFIGS.23to28, in the safety monitoring device10according to the present embodiment, the event list may be configured such that an important mark for selecting the presence or absence of automatic erasure of a file is added for each event via a graphical user interface.

Such a safety monitoring device10according to the present embodiment can add an important mark for each event. Therefore, for example, a safety manager can easily change the setting of each event from the presence of the automatic erasure to the absence by adding an important mark to each event. In this way, the safety manager can easily leave the file in which the incident (important occurrence) has occurred.

(32) As illustrated inFIGS.23to28, the event list processing unit23of the safety monitoring device10according to the present embodiment may be configured to perform control not to automatically erase a file including a period from a certain time before the start time of an event for which the absence of automatic erasure is selected at the timing of selecting the presence or absence of automatic erasure of the file and an event to which an important mark is added in advance to a certain time after the end time.

Such a safety monitoring device10according to the present embodiment can perform control not to automatically erase the files of the event for which the absence of automatic erasure is selected and the event to which the important mark is added in advance. Note that the safety monitoring device10can add or delete an important mark from the present time point to the timing of automatic erasure of the file. Further, the safety monitoring device10can change the time of each event from the present time point to the timing of the automatic erasure of the file.

(33) As illustrated inFIGS.23to28, the event list processing unit23of the safety monitoring device10according to the present embodiment may be configured to perform control to erase the file of the event in a case where the event for which the absence of the automatic erasure is selected is changed to the event for which the presence of the automatic erasure is selected or in a case where the important mark is removed from the event to which the important mark is added in advance.

Such a safety monitoring device10according to the present embodiment can perform control to erase the file of the event when the selection of the event is changed to the event for which the presence of the automatic erasure is selected or when the important mark is removed. In this way, the safety monitoring device10can increase the free capacity of the HDD. That is, the safety monitoring device10is configured to leave the video data file related to the event to which the important mark is added without erasing the video data file even when a predetermined time elapses after the video-recording. However, in a case where the safety monitoring person confirms the risk of each event on the basis of the event list, even if an important mark is added to the event, it may be determined that the risk is low. In such a case, the safety monitoring device10can erase the video data file related to the event from which the important mark is removed by removing the important mark. In this way, the safety monitoring device10can increase the free capacity of the HDD.

(34) As illustrated inFIGS.23to28, the event list processing unit23of the safety monitoring device10according to the present embodiment may be configured to perform control such that while an event of a file is deleted from the event list in a case where a file for which a predetermined time has elapsed since recording is automatically erased, an event for which the absence of automatic erasure is selected at the timing of selecting the presence or absence of automatic erasure of the file and an event to which an important mark is added in advance are not deleted from the event list.

Such a safety monitoring device10according to the present embodiment can prevent an event for which the absence of automatic erasure is selected and an event to which an important mark is added from being erroneously erased. Note that the event list and the video data file have a one-to-one correspondence. Therefore, the event corresponding to the video data file is deleted from the event list at the same time as the deletion of the video data file to be deleted. Therefore, the safety monitoring person can prevent an event that should not be erased from being erroneously erased.

(35) The safety monitoring device10according to the present embodiment includes the safety monitoring unit22that monitors entry of an object into a monitoring region of a safety sensor. The safety monitoring unit22may be configured to notify the event list processing unit23of the start time and the end time of the event by setting the time when the object enters the monitoring region as the start time of the event and setting the time when the object leaves the monitoring region or the time when a certain time has elapsed from the start time of the event as the end time of the event.

Such a safety monitoring device10according to the present embodiment can automatically sense the start time and the end time of the event and notify the event list processing unit23of the start time and the end time of the event. The safety monitoring device10can monitor the dangerous area three-dimensionally regardless of day or night by using, for example, the lidar51, which is a safety sensor.

As described above, the safety monitoring device10according to the present embodiment can satisfactorily grasp the relationship between a person and an object.

Note that the present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the gist of the present invention.

For example, the above-described embodiment has been described in detail for easy understanding of the gist of the present invention. Therefore, the present invention is not necessarily limited to those including all the components described above. Further, in the present invention, another component can be added to a certain component, or some components can be changed to other components. Further, in the present invention, some components can be eliminated

Furthermore, for example, in the above-described embodiment, the safety monitoring device10monitors the safety of an area on the basis of three-dimensional data, but can monitor the safety of an area on the basis of two-dimensional data.

REFERENCE SIGNS LIST

10Safety monitoring device11Control unit12Area setting unit13Alarm activation type setting unit14Video acquisition unit15Lidar data acquisition unit (sensor data acquisition unit)16Moving body recognition analysis unit16aForeground data extraction unit16bClustering processing unit16cBounding box generation unit16dAngle smoothing unit (smoothing means)16e,18aImage recognition unit16fAlarm activation analysis unit17Result output unit18Video processing unit19Recording management unit20Video display processing unit21Safety sensor input unit22Safety monitoring unit23Event list processing unit30Storage unit31Setting recording data32Video recording data33Processed image data39Communication unit41Operation unit42Display (display unit)43Warning lamp44Speaker (sounding unit)50Safety sensor51Lidar61Monitoring camera (photographing unit)61zMonitoring camera (photographing unit, external device)61aSetting reception unit61bSetting processing unit61cNotification unit62Web terminal (external device)62aNotification acquisition unit100,110Display screen1013D viewer unit101aAlarm activation target area frame102Setting unit103Lidar selection unit104Alarm activation target area tab104aBlock tab104bValidity check box104cArea selection unit104dDisplay color setting unit105Moving body105aLarge object105box,106box Bounding box106Person106oCenter point106rRadius111Video display unit112Event list display unit120Event list121Important mark field122Time stamp field123Camera name field124Type field125Description field130Event editing screen131Camera name field132User name field133Occurrence date and time field134End date and time field135Description field136Type field137Mark field138Cause field139Measure field140Importance fieldA51Sensing target areaA61a, A61bAlarm activation target areaA62Alarm activation exclusion areaB61a, B61bAlarm activation portionB62Exclusion portionD DepthG GroundH, Hb, Ht HeightPr Control programR Certain distanceR11Walking routeS Safety monitoring systemSu1, Su2, Su3, Su4, Su5, Su6, Su8, Sub SurfaceW Width