Patent Description:
As a related art, <CIT> discloses an in-vehicle monitoring apparatus that prevents a passenger in a vehicle from falling down in public passenger transportation. The in-vehicle monitoring apparatus described in <CIT> is mounted on a vehicle, such as a bus, that transports a passenger in a state of sitting or standing. The vehicle includes a plurality of in-vehicle cameras and various kinds of in-vehicle sensors. The in-vehicle camera captures an image of a seat, a passenger, baggage, and the like in the vehicle. The in-vehicle monitoring apparatus tracks movement of a passenger from the passenger getting on a bus to moving at a boarding position, by using a video of the in-vehicle camera. <CIT> and <CIT> are also related prior art.

In <CIT>, the in-vehicle sensor includes an ultrasonic sensor, a pressure sensor, and a contact sensor. The ultrasonic sensor is used for detecting a position and a pose of a passenger in a vehicle. The pressure sensor is installed in a seat surface of a seat, and is used for detecting whether a passenger is sitting on the seat. In addition, the pressure sensor or the contact sensor is installed in a strap and a handrail, and is used for detecting whether a passenger is holding on to the handrail or the strap.

The in-vehicle monitoring apparatus grasps a boarding state of a passenger in a vehicle by using the in-vehicle camera and the in-vehicle sensor. In addition, the in-vehicle monitoring apparatus grasps a traveling state of the vehicle by using an out-of-vehicle camera and a vehicle sensor. The in-vehicle monitoring apparatus determines a risk level indicating a risk of a passenger falling, based on the boarding state of the passenger and a traveling state of the vehicle.

The in-vehicle monitoring apparatus performs notification related to safety of a passenger according to the risk level. For example, the in-vehicle monitoring apparatus converts an image captured by using the in-vehicle camera into an overhead image, and displays the converted image on a monitor. The in-vehicle monitoring apparatus displays, on a display screen of the monitor, each of a passenger sitting, a passenger standing on an aisle, and a passenger not moving for a long time, with different marks. In addition, the in-vehicle monitoring apparatus displays a passenger standing on the aisle with holding on to a handrail or a strap and a passenger standing on the aisle without holding on to the handrail or the strap, in a distinguished manner.

In riding bus business, a driver visually confirms whether a passenger is in a stable pose, that is, whether a standing passenger is holding on to a pole or the like. There is a possibility that a need for stable pose determination of a passenger increases as one axis of passenger watching which focuses on not only current manual driving by a driver but also a future autonomous driving age.

The in-vehicle monitoring apparatus described in <CIT> displays a boarding state of each passenger on a monitor. By referring to the monitor, a driver can grasp a passenger not holding on to a handrail or a strap, or the like. However, in <CIT>, a sensor such as a contact sensor is installed in a handrail and a strap, and the in-vehicle monitoring apparatus determines, by using the sensor, whether a passenger is holding on to the handrail or the strap. Therefore, in <CIT>, a sensor needs to be installed in all handrails or straps that can be held by a passenger.

In view of the above circumstances, an object of the present disclosure is to provide a system, an information processing apparatus, a method, and a computer-readable medium that are capable of determining whether a passenger is holding a pole without installing a sensor such as a contact sensor in each of a plurality of places where a passenger may hold.

In order to achieve the above object, the present disclosure provides, as a first aspect, an information processing apparatus as defined in claim <NUM>.

The information processing apparatus includes: a skeleton information acquisition means for analyzing an in-vehicle video acquired from a camera that captures an image of a vehicle interior of a moving object, and acquiring skeleton information of each of one or more passengers boarding on the moving object; a holding determination means for determining whether the passenger is holding a pole, based on skeleton information of the passenger and a position of the pole in the moving object; and an alert output means for outputting an alert when it is determined that the passenger is not holding the pole.

The present disclosure provides, as a second aspect, a system as defined in claim <NUM>. The system includes a moving object having a camera that captures an image of a vehicle interior, and an information processing apparatus that monitors a passenger of a moving object. The information processing apparatus includes: a skeleton information acquisition means for analyzing an in-vehicle video acquired from the camera, and acquiring skeleton information of each of one or more passengers boarding on the moving object; a holding determination means for determining whether the passenger is holding a pole, based on skeleton information of the passenger and a position of the pole in the moving object; and an alert output means for outputting an alert when it is determined that the passenger is not holding the pole.

The present disclosure provides, as a third aspect, an information processing method as defined in claim <NUM>.

The information processing method includes: analyzing an in-vehicle video acquired from a camera that captures an image of a vehicle interior of a moving object, and acquiring skeleton information of each of one or more passengers boarding on the moving object; determining whether the passenger is holding a pole, based on skeleton information of the passenger and a position of the pole in the moving object; and outputting an alert when determining that the passenger is not holding the pole.

The present disclosure provides, as a fourth aspect, a computer-readable medium as defined in claim <NUM>.

The computer-readable medium stores a program for causing a processor to execute processing of: analyzing an in-vehicle video acquired from a camera that captures an image of a vehicle interior of a moving object, and acquiring skeleton information of each of one or more passengers boarding on the moving object; determining whether the passenger is holding a pole, based on skeleton information of the passenger and a position of the pole in the moving object; and outputting an alert when determining that the passenger is not holding the pole.

A system, an information processing apparatus, a method, and a computer-readable medium according to the present disclosure can determine whether a passenger is holding a pole without installing a sensor such as a contact sensor in each of a plurality of places where the passenger may hold.

Prior to the description of example embodiments of the present disclosure, an outline of the present disclosure will be described. <FIG> illustrates a system according to the present disclosure. A system <NUM> includes a moving object <NUM> and an information processing apparatus <NUM>. The moving object <NUM> includes a camera <NUM> that captures an image of a vehicle interior. The information processing apparatus <NUM> monitors a passenger boarding on the moving object <NUM>. The information processing apparatus <NUM> includes a skeleton information acquisition means <NUM>, a holding determination means <NUM>, and an alert output means <NUM>.

The skeleton information acquisition means <NUM> analyzes an in-vehicle video acquired from the camera <NUM>, and acquires skeleton information of each of one or more passengers boarding on the moving object <NUM>. The holding determination means <NUM> determines whether a passenger is holding a pole, based on the skeleton information of the passenger and a position of the pole in the moving object <NUM>. The alert output means <NUM> outputs an alert when it is determined that the passenger is not holding the pole.

In the present disclosure, the holding determination means <NUM> determines whether a passenger is holding a pole by using the skeleton information of the passenger acquired from a video of the camera <NUM>. In the present disclosure, a video of the camera <NUM> is used for determination of whether a passenger is holding a pole. Therefore, the present disclosure can determine whether a passenger is in a safe boarding pose without installing a sensor such as a contact sensor at each of a plurality of places where the passenger may hold. The information processing apparatus <NUM> outputs an alert when it is determined that a passenger is not holding a pole. In this way, the information processing apparatus <NUM> can contribute to safe operation of the moving object <NUM>.

Hereinafter, the example embodiments of the present disclosure will be described in detail. <FIG> illustrates a system according to a first example embodiment of the present disclosure. A system (passenger monitoring system) <NUM> includes an information processing apparatus <NUM>, a remote monitoring apparatus <NUM>, and a moving object <NUM>. In the system <NUM>, the information processing apparatus <NUM> and the remote monitoring apparatus <NUM> are connected to the moving object <NUM> via a network <NUM>. The network <NUM> includes, for example, a wireless communication network by using a communication line standard such as long term evolution (LTE). The network <NUM> may include a wireless communication network such as WiFi (registered trademark) or a fifth generation mobile communication system. The system <NUM> corresponds to the system <NUM> illustrated in <FIG>.

The moving object <NUM> is configured as, for example, a vehicle such as a riding bus that travels with a passenger boarding thereon. The moving object <NUM> is not limited to a riding bus, and may be a vehicle, such as a street train, on which a passenger can stand and board. The moving object <NUM> includes one or more in-vehicle cameras <NUM> that capture an image of a vehicle interior. In <FIG>, an example in which the moving object <NUM> includes two in-vehicle cameras <NUM> that capture the vehicle interior from a front and a back of the vehicle is illustrated. The moving object <NUM> corresponds to the moving object <NUM> illustrated in <FIG>. The in-vehicle camera <NUM> corresponds to the camera <NUM> illustrated in <FIG>.

The information processing apparatus <NUM> is an apparatus that monitors a passenger of the moving object <NUM>. The information processing apparatus <NUM> receives a video of the in-vehicle camera <NUM> from the moving object <NUM> via the network <NUM>. The information processing apparatus <NUM> analyzes the received video of the in-vehicle camera <NUM>, and watches whether a passenger is in a safe boarding state. The information processing apparatus <NUM> transmits a result of watching of a passenger to the moving object <NUM>. The information processing apparatus <NUM> corresponds to the information processing apparatus <NUM> illustrated in <FIG>.

The remote monitoring apparatus <NUM> is an apparatus that remotely monitors operation of the moving object <NUM>. The information processing apparatus <NUM> may transmit a result of watching of a passenger to the remote monitoring apparatus <NUM>, in addition to or instead of transmitting the result of watching of the passenger to the moving object <NUM>. Note that, when it is not necessary to remotely monitor the operation of the moving object <NUM>, the remote monitoring apparatus <NUM> can be omitted. In other words, the system <NUM> may not include the remote monitoring apparatus <NUM>.

<FIG> illustrates a configuration example of the moving object <NUM>. The moving object <NUM> includes the one or more in-vehicle cameras <NUM>, a vehicle information acquisition unit <NUM>, a communication apparatus <NUM>, and a display apparatus <NUM>. In the moving object <NUM>, these components are configured to be communicable with one another via an in-vehicle local area network (LAN), a controller area network (CAN), or the like.

Each of the in-vehicle cameras <NUM> is a camera that captures an image of inside of the moving object <NUM>. In particular, the in-vehicle camera <NUM> captures an area in which a passenger boards. The in-vehicle camera <NUM> is disposed, for example, on a ceiling of a vehicle interior of the moving object <NUM>. The vehicle information acquisition unit <NUM> acquires various kinds of information of the moving object <NUM>. The vehicle information acquisition unit <NUM> acquires information such as a vehicle speed, a steering angle, an opening degree of an accelerator pedal, and a depression amount of a brake pedal from, for example, a vehicle sensor of the moving object <NUM>. In addition, the vehicle information acquisition unit <NUM> acquires information such as an operation state of a direction indicator and an opening/closing state of a door.

The communication apparatus <NUM> is configured as an apparatus that performs wireless communication between the moving object <NUM> and the network <NUM> (refer to <FIG>). The communication apparatus <NUM> includes an antenna for wireless communication, a transmitter, and a receiver. The communication apparatus <NUM> transmits a video of the in-vehicle camera <NUM> to the information processing apparatus <NUM> and the remote monitoring apparatus <NUM> via the network <NUM>.

In addition, the communication apparatus <NUM> transmits various kinds of information of the moving object <NUM> to the information processing apparatus <NUM> and the remote monitoring apparatus <NUM> via the network <NUM>. The communication apparatus <NUM> transmits, for example, the vehicle speed, the operation state of the direction indicator, and the opening/closing state of the door acquired by the vehicle information acquisition unit <NUM> as vehicle information to the information processing apparatus <NUM> and the remote monitoring apparatus <NUM>. The communication apparatus <NUM> may transmit position information of the moving object <NUM> to the information processing apparatus <NUM> and the remote monitoring apparatus <NUM>. Note that, information transmitted to the information processing apparatus <NUM> and information transmitted to the remote monitoring apparatus <NUM> may not be the same information.

The moving object <NUM> may include a peripheral monitoring sensor that monitors a peripheral situation of the moving object <NUM>, which is not illustrated in <FIG>. The peripheral monitoring sensor includes, for example, a camera, a radar, a light detection and ranging (LiDAR), or the like. The peripheral monitoring sensor may include, for example, a plurality of cameras that capture a front, a back, a right side, and a left side of the vehicle. The communication apparatus <NUM> may transmit a video or the like of outside of the vehicle acquired by the peripheral monitoring sensor to the remote monitoring apparatus <NUM> via the network <NUM>. The remote monitoring apparatus <NUM> displays the video received from the moving object <NUM> and various kinds of information on a display apparatus. A monitoring person refers to the video and various kinds of information displayed on the display apparatus, and performs remote monitoring of the moving object <NUM>.

<FIG> illustrates a configuration example of the information processing apparatus <NUM>. The information processing apparatus <NUM>, which is a passenger watching apparatus, includes a video acquisition unit <NUM>, a vehicle information acquisition unit <NUM>, a skeleton information acquisition unit <NUM>, a pole position acquisition unit <NUM>, a holding determination unit <NUM>, and an alert output unit <NUM>. The video acquisition unit <NUM> receives an in-vehicle video captured by using the in-vehicle camera <NUM> from the moving object <NUM>. The vehicle information acquisition unit <NUM> receives vehicle information from the moving object <NUM>. Each of the video acquisition unit <NUM> and the vehicle information acquisition unit <NUM> may receive the in-vehicle video and the vehicle information from a plurality of the moving objects <NUM>.

The skeleton information acquisition unit <NUM> analyzes a video (image) of the in-vehicle camera <NUM>, and acquires skeleton information of each of one or more passengers boarding on the vehicle. When the moving object <NUM> includes a plurality of in-vehicle cameras <NUM>, the skeleton information acquisition unit <NUM> performs analysis on each of the videos of the plurality of in-vehicle cameras. For example, the skeleton information acquisition unit <NUM> estimates a two-dimensional skeleton structure of a person (passenger) in an image from the video (image) of the in-vehicle camera <NUM> acquired by the video acquisition unit <NUM>. The skeleton information acquisition unit <NUM> estimates the skeleton structure for each passenger included in the image. For example, the skeleton information acquisition unit <NUM> detects a skeleton structure of each passenger, based on a feature such as a joint of each passenger, by using a skeleton estimation technique using machine learning. For example, a known skeleton estimation technique such as OpenPose may be used for detecting the skeleton structure. The skeleton information acquisition unit <NUM> corresponds to the skeleton information acquisition means <NUM> illustrated in <FIG>.

<FIG> illustrates one example of a skeleton structure detected by the skeleton information acquisition unit <NUM>. For example, the skeleton information acquisition unit <NUM> extracts an image of each passenger from an in-vehicle video of the moving object <NUM>. The skeleton information acquisition unit <NUM> detects, as a skeleton structure, coordinate data of a joint point of a passenger from the extracted image. In <FIG>, the detected joint points are indicated by black circles. The skeleton information acquisition unit <NUM> detects a position of a joint point <NUM> of a wrist and a position of a joint point <NUM> of an elbow in particular, from the detected skeleton structure. The skeleton information acquisition unit <NUM> acquires information indicating the skeleton structure of the passenger as skeleton information.

The pole position acquisition unit (pole position acquisition means) <NUM> acquires, as pole position information, information indicating a position of a pole existing inside of the moving object <NUM>. The pole position acquisition unit <NUM> acquires pole position information from, for example, an in-vehicle video of the moving object <NUM>. The pole position acquisition unit <NUM> identifies a region of a pole included in an in-vehicle video, for example, based on a color. In Japan, a "standard specification low-floor bus" determines that a specific color (orange color) is used for a color of a vertical grip bar (pole). The pole position acquisition unit <NUM> may identify a region of a bar-shaped object having an orange color as a region of a pole. For example, the pole position acquisition unit <NUM> may acquire the pole position information from an in-vehicle video in a state where a passenger is not boarding. Alternatively, the pole position acquisition unit <NUM> may read pre-registered pole position information from a not-illustrated storage device.

The holding determination unit <NUM> determines whether a passenger is holding a pole, based on the skeleton information of the passenger acquired by the skeleton information acquisition unit <NUM> and the pole position information acquired by the pole position acquisition unit <NUM>. The holding determination unit <NUM> may further determine whether a passenger is holding at least one of a pole and a strap by further using positional information of the strap and the like. The holding determination unit <NUM> corresponds to the holding determination means <NUM> illustrated in <FIG>.

For example, the holding determination unit <NUM> determines whether a passenger is holding a pole, based on a distance between the joint point <NUM> of the wrist (refer to <FIG>) and the pole in an in-vehicle video. The holding determination unit <NUM> derives, for example, a shortest distance from the joint point <NUM> of the wrist to the pole. When the derived shortest distance is equal to or less than a predetermined distance threshold value, the holding determination unit <NUM> determines that the passenger is holding the pole.

The holding determination unit <NUM> may determine whether a passenger is holding a pole, based on a relationship between a position of the joint point <NUM> of the wrist and a position of the joint point <NUM> of the elbow, and a position of the pole. For example, the holding determination unit <NUM> estimates a range in which a palm exists, based on the position of the joint point <NUM> of the wrist and the position of the joint point <NUM> of the elbow. The holding determination unit <NUM> may estimate whether the passenger is holding the pole according to whether the pole exists in the estimated range.

<FIG> illustrates an example in which it is determined whether a passenger is holding a pole, based on the positions of the joint points of the wrist and elbow and the pole. The holding determination unit <NUM> derives a length L of a straight line connecting the joint point <NUM> of the wrist and the joint point <NUM> of the elbow. The holding determination unit <NUM> places a point <NUM>, on a straight line passing through the joint point <NUM> of the wrist and the joint point <NUM> of the elbow, at a position separated from the joint point <NUM> of the wrist by a distance (kL) acquired by k times (<NUM><k<<NUM>) the distance L on an opposite side of the joint point <NUM> of the elbow. The holding determination unit <NUM> estimates that a palm exists between the joint point <NUM> of the wrist and the point <NUM>. The holding determination unit <NUM> determines whether a straight line connecting the joint point <NUM> of the wrist and the point <NUM> intersects a pole <NUM>. When the pole <NUM> exists between the joint point <NUM> of the wrist and the point <NUM>, the holding determination unit <NUM> determines that the passenger is holding the pole <NUM>.

<FIG> illustrates another example not according to the invention in which it is determined whether a passenger is holding a pole, based on the positions of the joint points of the wrist and elbow and the pole. In this example, the holding determination unit <NUM> sets a fan-shaped region <NUM> passing through the point <NUM> on the opposite side of the joint point <NUM> of the elbow, with the joint point <NUM> of the wrist as a base point. The holding determination unit <NUM> estimates that a palm exists in a range of the region <NUM>. The holding determination unit <NUM> determines whether a part of the pole <NUM> is included in the region <NUM>. When a part of the pole <NUM> is included in the region <NUM>, the holding determination unit <NUM> determines that the passenger is holding the pole <NUM>.

Herein, in an in-vehicle video, a front pole and a back passenger may overlap with each other, and although the passenger does not actually hold the pole, a joint point of a wrist of the passenger and the pole may overlap with each other. <FIG> schematically illustrates a passenger and a pole in the moving object <NUM>. Herein, it is assumed that there are three poles being poles <NUM> to <NUM> inside of the moving object <NUM>. In addition, it is assumed that there are two passengers being passengers <NUM> and <NUM> in the moving object <NUM>. It is assumed that the passenger <NUM> holds the pole <NUM>. On the other hand, it is assumed that the passenger <NUM> has something in his/her hand and does not hold any pole. The in-vehicle camera <NUM> captures an image of the vehicle interior of the moving object <NUM> from the front of the moving object <NUM>.

<FIG> schematically illustrates an in-vehicle video captured by the in-vehicle camera <NUM>. In the example illustrated in <FIG>, the passenger <NUM> is holding the pole <NUM>, so that a joint point of a wrist is close to the pole <NUM>. In this case, the holding determination unit <NUM> can determine that the passenger <NUM> is holding the pole <NUM>, based on a distance between the joint point of the wrist and the pole <NUM>. On the other hand, for the passenger <NUM>, the passenger <NUM> does not hold the pole, but in the in-vehicle video, the vicinity of a wrist of the passenger <NUM> overlaps with the pole <NUM>. Therefore, there is a possibility that the holding determination unit <NUM> erroneously determines that the passenger <NUM> is holding the pole <NUM>.

In order to avoid the above-described erroneous determination, the holding determination unit <NUM> may extract a pole that can be held by each passenger among the poles inside the moving object <NUM>, and determine whether each passenger is holding the extracted pole. The holding determination unit <NUM> specifies, for example, a position of each passenger on a floor of the vehicle interior of the moving object <NUM> and a position of each pole on the floor. For example, the holding determination unit <NUM> specifies the position of each passenger on the floor and the position of each pole on the floor by using floor information for identifying which part in the video corresponds to the floor. Alternatively, the position of each passenger on the floor can be specified from a video of the in-vehicle camera <NUM> by using an angle of view, a focal length, and the like of the in-vehicle camera <NUM>. Similarly, the position of each pole on the floor can be specified from a video of the in-vehicle camera <NUM> by using the angle of view, the focal length, and the like of the in-vehicle camera <NUM>. The position of each pole on the floor may be prepared in advance.

The holding determination unit <NUM> extracts a pole that can be held by each passenger, based on, for example, a positional relationship between the position of each passenger on the floor and the position of each pole on the floor. The holding determination unit <NUM> derives, for example, a distance between each passenger and each pole on the floor. The holding determination unit <NUM> extracts, from a position of a passenger on the floor, a pole existing in a range where an arm of the passenger reaches, as a pole that can be held by the passenger. The holding determination unit <NUM> may estimate a height of a passenger from an estimation result of the skeleton structure, and estimate a length of an arm of the passenger from the estimated height. When a pole does not exist in the range in which an arm of a passenger reaches, the holding determination unit <NUM> determines that there is no pole that the passenger can hold.

<FIG> schematically illustrates a positional relationship between a passenger and a pole on a floor. For the passenger <NUM>, the holding determination unit <NUM> sets, around the passenger <NUM>, a range <NUM> in which an arm of the passenger <NUM> reaches according to an estimated arm length. The holding determination unit <NUM> extracts the pole <NUM> in which a position of the pole on the floor is included in the range <NUM> as a pole that can be held by the passenger <NUM>. The holding determination unit <NUM> determines whether the passenger <NUM> is holding the pole <NUM>, based on the position of the joint point of the wrist of the passenger <NUM> and the position of the pole <NUM>. The holding determination unit <NUM> excludes the poles <NUM> and <NUM> from determination.

For the passenger <NUM>, the holding determination unit <NUM> sets, around the passenger <NUM>, a range <NUM> in which an arm of the passenger <NUM> reaches according to an estimated arm length. In <FIG>, any of positions of the poles <NUM> to <NUM> on the floor is outside the range <NUM>. In this case, the holding determination unit <NUM> determines that there is no pole that the passenger <NUM> can hold. As described above, by limiting a pole to be determined, based on a positional relationship between a passenger and the pole on the floor, erroneous determination can be suppressed even when the passenger and the pole overlap with each other on an in-vehicle video.

Note that, it is conceivable that a foot of a passenger standing on board is in contact with the floor. Therefore, it is conceivable that the holding determination unit <NUM> can specify a position of the passenger on the floor from an in-vehicle video. However, a pole may be fixed to a seat or a wall, and is not necessarily attached to the floor. When the pole is not attached to the floor, the holding determination unit <NUM> may virtually extend the pole in a direction of a floor surface, and thereby specify a position of the pole on the floor.

<FIG> schematically illustrates an in-vehicle video when a pole is not in contact with a floor. In <FIG>, a pole <NUM> is fixed to a ceiling and a side wall of the vehicle interior, and is not in contact with a floor surface. In this case, the holding determination unit <NUM> virtually extends a portion of the pole <NUM> in a vertical direction to the floor surface, and specifies an intersection point between the extended portion of the pole <NUM> and the floor surface, as a position <NUM> of the pole <NUM> on the floor. The holding determination unit <NUM> may determine whether the passenger <NUM> can hold the pole <NUM>, based on a positional relationship between the position of the passenger <NUM> on the floor and the position <NUM>.

Returning to <FIG>, the alert output unit <NUM> acquires a determination result from the holding determination unit <NUM>. The alert output unit <NUM> transmits, to the moving object <NUM>, a warning indicating that there is a passenger in an unsafe pose when the passenger is determined to be standing in the moving object <NUM> and not holding a pole or the like. The alert output unit <NUM> may transmit information indicating a position of a passenger determined that the passenger is not holding the pole or the like to the moving object <NUM>. The alert output <NUM> can transmit a warning to the remote monitoring apparatus <NUM>, in addition to or instead of the moving object <NUM>. The alert output unit <NUM> corresponds to the alert output means <NUM> illustrated in <FIG>.

The alert output unit <NUM> may acquire vehicle information from the vehicle information acquisition unit <NUM>, and determine whether a predetermined alert condition is satisfied, based on the vehicle information. For example, the alert output unit <NUM> specifies an operation state of the moving object <NUM>, based on the vehicle information. The alert output unit <NUM> may determine whether the specified operation state indicates an operation state set as an alert condition. When the alert condition is satisfied, the alert output unit <NUM> may transmit a warning to the moving object <NUM> or the like.

For example, the alert output unit <NUM> specifies whether the moving object <NUM> is traveling, based on the vehicle information including the vehicle speed. The alert output unit <NUM> may transmit a warning to the moving object <NUM> when it is determined that there is a passenger who is not holding a pole or the like while the moving object <NUM> is traveling. In addition, the alert output unit <NUM> may specify whether to be immediately before the moving object <NUM> starts, based on the vehicle information including an opening/closing state of a door. The alert output unit <NUM> specifies, for example, that it is immediately before the moving object <NUM> starts, when the door is closed at a bus stop. The alert output unit <NUM> may transmit a warning to the moving object <NUM> when it is determined that there is a passenger who is not holding a pole or the like immediately before the moving object <NUM> starts.

In the moving object <NUM>, the communication apparatus <NUM> (refer to <FIG>) receives information indicating presence of a passenger in an unsafe pose, a position thereof, and the like. The display apparatus <NUM> displays presence of a passenger in an unsafe pose, a position thereof, and the like, and notifies a driver of the moving object <NUM> of danger. As necessary, the driver makes an announcement such as "please hold on to a handrail". Alternatively, the moving object <NUM> may make an announcement such as "please hold on to a handrail" by using a not-illustrated automatic broadcasting apparatus. The driver does not cause the moving object <NUM> to start until a passenger holds the handrail when danger is notified immediately before start.

Next, an operation procedure will be described. <FIG> illustrates an operation procedure (information processing method) in the information processing apparatus (passenger watching apparatus) <NUM>. The video acquisition unit <NUM> receives a video (in-vehicle video) of the in-vehicle camera <NUM> from the moving object <NUM> (step S1). The video acquisition unit <NUM> outputs the received video to the skeleton information acquisition unit <NUM> and the pole position acquisition unit <NUM>. The vehicle information acquisition unit <NUM> receives vehicle information acquired by the vehicle information acquisition unit <NUM> (refer to <FIG>) or the like from the moving object <NUM> (step S2). The vehicle information acquisition unit <NUM> outputs the received vehicle information to the alert output unit <NUM>.

The skeleton information acquisition unit <NUM> acquires skeleton information of each passenger from the in-vehicle video received in step S1 (step S3). In step S3, the skeleton information acquisition unit <NUM> acquires, in particular, a position of a joint points of a wrist and a position of a joint point of an elbow of a passenger. The pole position acquisition unit <NUM> acquires a position of a pole from the in-vehicle video (step S4).

Based on the skeleton information acquired in step S3 and the position of the pole acquired in step S4, the holding determination unit <NUM> determines whether a passenger is holding a pole or the like (step S5). The holding determination unit <NUM> outputs a result of determination to the alert output unit <NUM>.

Based on the result of the determination in step S5, the alert output unit <NUM> determines whether there is a passenger who is not holding a pole or the like (step S6). When the alert output unit <NUM> determines that there is no passenger who has not held a pole or the like, processing ends. The alert output unit <NUM> determines whether a predetermined alert condition is satisfied, based on the vehicle information received in step S2 (step S7). When determining that the predetermined alert condition is satisfied, the alert output unit <NUM> transmits a warning indicating presence of a passenger in an unsafe pose to the moving object <NUM> and the remote monitoring apparatus <NUM> (step S8). The passenger watching apparatus <NUM> repeatedly performs the processing from steps S1 to S8.

Note that, in the above-described operation procedure, a part of steps S1 to S8 is not necessarily performed in above-described order. For example, either step S1 or step S2 may be performed first. In addition, the vehicle information received in step S2 is used for determining whether the alert condition is satisfied in step S7. Therefore, step S2 may be performed at any timing from step S3 to immediately before step S7. Alternatively, step S2 may be performed in parallel with any of the steps from step S1 to step S6. In addition, either step S3 or step S4 may be performed first, or may be performed in parallel.

In the present example embodiment, the skeleton information acquisition unit <NUM> acquires skeleton information of a passenger from a video of the in-vehicle camera <NUM> of the moving object <NUM>. The holding determination unit <NUM> determines whether a passenger is holding a pole by using the skeleton information of the passenger and position information of the pole. In the present example embodiment, whether a passenger is holding a pole is determined by using the skeleton information acquired from a video. In this way, even when a sensor such as a contact sensor or a pressure sensor is not installed at a plurality of places in a vehicle, the passenger watching apparatus <NUM> can determine whether a passenger is holding a pole or the like. In addition, in the present example embodiment, it is possible to find a passenger who is not holding a pole in the moving object <NUM> without depending on visual observation of a driver.

In the present example embodiment, the alert output unit <NUM> outputs an alert to the moving object <NUM> and the remote monitoring apparatus <NUM> when it is determined that a passenger is not holding a pole. When a passenger is not holding a pole or the like, a driver or the like of the moving object <NUM> is warned, so that the driver can know that there is a passenger who is not holding a pole without confirming by visual observation. The driver can cause the moving object <NUM> to reduce a risk of a passenger falling in the moving object <NUM>, for example, by slowing down as needed or prompting the passenger to hold the pole. In addition, in the present example embodiment, the alert output unit <NUM> outputs an alert when a predetermined alert condition is satisfied. In this case, the alert output unit <NUM> can output an alert when a risk of occurring falling or the like is high when a passenger is not holding a pole.

Next, a second example embodiment of the present disclosure will be described. A configuration of a system according to the second example embodiment of the present disclosure may be similar to the configuration of the information processing apparatus (passenger watching apparatus) <NUM> described in the first example embodiment illustrated in <FIG>. In the present example embodiment, a moving object is configured to be capable of autonomous driving. Other configurations may be similar to those described in the first example embodiment.

<FIG> illustrates a configuration of a moving object used in the second example embodiment. In the present example embodiment, a moving object 200a includes an autonomous driving electric control unit (ECU) <NUM> and a broadcasting apparatus <NUM>, in addition to the configuration of the moving object <NUM> used in the first example embodiment illustrated in <FIG>. The autonomous driving ECU <NUM> is an electric control unit that controls autonomous driving of the moving object <NUM>. The autonomous driving ECU <NUM> acquires sensor information from a not-illustrated peripheral monitoring sensor and a not-illustrated vehicle sensor, and controls autonomous traveling of the moving object <NUM>, based on the acquired sensor information. The broadcasting apparatus <NUM> performs various types of broadcasting inside the moving object <NUM>.

In the present example embodiment, an alert output unit <NUM> (refer to <FIG>) of a passenger watching apparatus <NUM> outputs an alert to the moving object 200a and a remote monitoring apparatus <NUM>. In the moving object 200a, a communication apparatus <NUM> receives an alert from the passenger watching apparatus <NUM>. When an alert is received, the autonomous driving ECU <NUM> changes a parameter of autonomous driving from a parameter of the autonomous driving before the alert is received. For example, when an alert is received, the autonomous driving ECU <NUM> increases a safety level of the moving object <NUM>. The autonomous driving ECU <NUM> lowers a speed of the moving object <NUM>, for example, by increasing the safety level. Alternatively, a change in the speed of the moving object <NUM> is made gradual. The autonomous driving ECU <NUM> does not cause the moving object <NUM> to start when an alert is received while stopping at a bus stop or the like. When an alert is received, the broadcasting apparatus <NUM> makes an announcement such as "please hold on to a handrail", and prompts a passenger to hold a pole.

The remote monitoring apparatus <NUM> receives an alert from the passenger watching apparatus <NUM>. When an alert is received, the remote monitoring apparatus <NUM> notifies a monitoring person that there is a passenger who is not holding a pole in the moving object 200a. The monitoring person can know that there is a passenger having a risk of falling in the moving object 200a. In this case, the monitoring person may switch driving of the moving object 200a from autonomous driving to remote driving, and remotely operate the moving object 200a.

In the present example embodiment, the moving object 200a is configured to be capable of autonomous driving. In the present example embodiment, when there is a passenger who is not holding a pole in the moving object 200a, an alert is output to the remote monitoring apparatus <NUM>. In this way, a monitoring person can watch whether the moving object 200a to be performed autonomous driving is safely operated. In addition, when an alert is output, the broadcasting apparatus <NUM> broadcasts a passenger to hold a pole in the moving object 200a. When the passenger who has heard the broadcasting holds the pole, the moving object 200a can be safely operated.

Note that, in each of the above example embodiments, an example in which the passenger watching apparatus <NUM> is connected to the moving object <NUM> via the network <NUM> has been described. However, the present disclosure is not limited thereto. For example, the passenger watching apparatus <NUM> may be mounted on the moving object <NUM>. In addition, the passenger watching apparatus <NUM> does not necessarily have to be configured as one apparatus, and may be configured by using a plurality of apparatuses. For example, the passenger watching apparatus <NUM> may be configured by using a first apparatus including the video acquisition unit <NUM>, the skeleton information acquisition unit <NUM>, and the pole position acquisition unit <NUM>, and a second apparatus including the vehicle information acquisition unit <NUM>, the holding determination unit <NUM>, and the alert output unit <NUM>. In that case, the first apparatus may be mounted on the moving object <NUM>, and the second apparatus may be connected to the first apparatus via the network <NUM>.

The passenger watching apparatus <NUM> may receive in-vehicle videos from a plurality of the moving objects <NUM>, and determine whether a passenger is holding a pole for each moving object. In this case, in order to reduce a processing load in the passenger watching apparatus <NUM>, at least a part of acquisition of skeleton information, acquisition of position information of a pole, and determination of holding may be performed in each moving object <NUM>.

Next, a physical configuration of the passenger watching apparatus <NUM> will be described. In the present disclosure, the passenger watching apparatus <NUM> may be configured as a computer apparatus (server apparatus). <FIG> illustrates a configuration example of a computer apparatus that can be used as the passenger watching apparatus <NUM>. A computer apparatus <NUM> includes a control unit (CPU: central processing unit) <NUM>, a storage unit <NUM>, a read only memory (ROM) <NUM>, a random access memory (RAM) <NUM>, a communication interface (IF) <NUM>, and a user interface <NUM>. The passenger watching apparatus <NUM> may be configured as an electronic control unit including the CPU <NUM>, the ROM <NUM>, and the RAM <NUM>.

The communication interface <NUM> is an interface for connecting the computer apparatus <NUM> and a communication network with each other via a wired communication means, a wireless communication means, or the like. The user interface <NUM> includes, for example, a display unit such as a display. In addition, the user interface <NUM> includes an input unit such as a keyboard, a mouse, and a touch panel.

The storage unit <NUM> is an auxiliary storage device capable of holding various types of data. The storage unit <NUM> is not necessarily a part of the computer apparatus <NUM>, and may be an external storage device or a cloud storage connected to the computer apparatus <NUM> via a network.

The ROM <NUM> is a non-volatile memory device. For example, a semiconductor memory apparatus such as a flash memory having a relatively small capacity is used for the ROM <NUM>. A program executed by the CPU <NUM> may be stored in the storage unit <NUM> or the ROM <NUM>. The storage unit <NUM> or the ROM <NUM> stores various programs for achieving a function of each unit in the passenger watching apparatus <NUM>, for example.

The program can be stored by using various types of non-transitory computer-readable media, and supplied to the computing apparatus <NUM>. The non-transitory computer-readable medium includes various types of tangible storage media. Examples of the non-transitory computer-readable medium include a magnetic recording medium such as, for example, a flexible disk, a magnetic tape, or a hard disk, a magneto-optical recording medium such as, for example, a magneto-optical disk, an optical disk medium such as a compact disc (CD) or a digital versatile disk (DVD), and a semiconductor memory such as a mask ROM, a programmable ROM (PROM), an erasable PROM (EPROM), a flash ROM, or a RAM. In addition, the program may also be supplied to a computer by using various types of transitory computer-readable media. Examples of the transitory computer-readable medium include an electric signal, an optical signal, and an electromagnetic wave. The transitory computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

The RAM <NUM> is a volatile memory device. Various types of semiconductor memory devices such as a dynamic random access memory (DRAM) or a static random access memory (SRAM) are used for the RAM <NUM>. The RAM <NUM> may be used as an internal buffer for temporarily storing data and the like. The CPU <NUM> loads a program stored in the storage unit <NUM> or the ROM <NUM> to the RAM <NUM>, and executes the program. By the CPU <NUM> executing the program, the function of each unit in the passenger watching apparatus <NUM> can be achieved. The CPU <NUM> may include an internal buffer capable of temporarily storing data and the like.

Claim 1:
An information processing apparatus (<NUM>) comprising
skeleton information acquisition means (<NUM>) for analyzing an in- vehicle video acquired from a camera (<NUM>) configured to capture an image of a vehicle interior of a vehicle, and acquiring skeleton information of each of one or more passengers boarding on the vehicle;
holding determination means (<NUM>) for determining whether the passenger is holding a pole, based on skeleton information of the passenger and a position of the pole in the vehicle; and
alert output means (<NUM>) for outputting an alert when determining that the passenger is not holding the pole,
wherein:
the skeleton information includes information indicating a position of a joint point (<NUM>) of a wrist of the passenger and information indicating a position of a joint point (<NUM>) of an elbow of the passenger, and
the holding determination means determines that the passenger is holding the pole, when the pole exists, in the in-vehicle video, between a point (<NUM>) on a straight line passing through the joint point of the elbow and the joint point of the wrist, being separated from the joint point of the wrist, on an opposite side of the joint point of the elbow, by a distance acquired by multiplying a distance (L) between the joint point of the elbow and the joint point of the wrist by a predetermined coefficient k, where <NUM><k<<NUM>, and the joint point of the wrist.