Patent Description:
A wheel loader operating in a mine and a quarry perform excavation work of a blasting rock and the like. During such work, for example, a tire may come into contact with a sharp rock and cause tire damage. A tire damage is, for example, scratching or puncturing a tire. Although an operator allows the wheel loader to travel while checking whether there is any rock on a road surface which may cause the tire damage, it is difficult to visually recognize a condition of the road surface from an operator's seat in some cases. Therefore, in a work vehicle disclosed in Patent Document <NUM>, a road surface condition in front of a front tire is imaged by a camera installed in front of a front accelerator, and the operator can check the road surface condition by a monitor provided in a cab.

Patent Document <NUM> describes a vehicle operation management device for a vehicle, in particular, a mine transport vehicle. The system has the capability of detecting a contact between a tire of the vehicle and an object on the road such as a stone. In particular, in case of a collision between the vehicle and the object, it is determined whether or not the object may damage a tire. The determination is based on a contact waveform including collision peak data.

Patent Document <NUM> describes an operation system for a transport vehicle such as mine vehicle (dump truck). The system is capable of determining a contact between a tire of the vehicle and an obstacle on the road, which may potentially cause damage and thus reduce the life-cycle of tires, and record respective information. For this purpose, the system is equipped with front cameras such as a stereo camera system.

Patent Documents <NUM> and <NUM> thus describe systems for detecting a contact between a tire of a vehicle and an obstacle and determining whether this may cause damage, after the contact.

Patent Document <NUM> describes a tire pattern determination device and a vehicle type determination device, in particular, for toll collection facilities. A toll booth system includes inter alia a tire pattern determination device capable of determining a tire pattern from passing by vehicles.

According to the work vehicle disclosed in Patent Document <NUM>, the operator can visually recognize whether a rock causing the tire damage exists in a traveling direction by viewing the monitor. However, in this configuration, it is first necessary for the operator to visually recognize a rock causing the tire damage on the monitor regardless of the presence or absence of a rock to be paid attention for the tire damage, and the tire damage can be avoided only when the operator can determine whether or not the visually recognized rock is likely to cause the tire damage. During excavation work, the operator's line of sight is forward, and there is a problem that it is difficult for the operator to keep viewing the monitor always. In addition, when the frequency of looking the monitor is increased, the operation of the work vehicle becomes slow and the productivity is lowered.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a road surface condition monitoring system, a work vehicle, a road surface condition monitoring method, and a program which enable an operator to easily monitor a road surface condition.

According to the aspect of the present invention, the operator can easily monitor the road surface condition.

In the drawings, the same or corresponding configurations are represented by the same reference numerals, and description thereof will not be repeated.

<FIG> is a perspective view showing a wheel loader <NUM> as an example of a work vehicle according to the present embodiment. <FIG> is a side view of the wheel loader <NUM> shown in <FIG>. <FIG> is a side view of the wheel loader <NUM> (when a bucket <NUM> is moved upward) shown in <FIG>. <FIG> is a front view of the wheel loader <NUM> shown in <FIG>. <FIG> is a view of the wheel loader <NUM> shown in <FIG> as viewed from diagonally below. <FIG> is a perspective view of a tire <NUM> shown in <FIG>. <FIG> is a schematic diagram showing an imaging region of a camera according to the present embodiment.

As shown in <FIG> and the like, the wheel loader <NUM> includes a vehicle body <NUM> and work equipment <NUM> supported by the vehicle body <NUM>. <FIG> schematically shows a rock pile <NUM>, which is a work object of the wheel loader <NUM>, and a rock <NUM>, which is located away from the rock pile <NUM> and has a possibility of damaging the tire <NUM>.

The vehicle body <NUM> has a cab <NUM>, a traveling mechanism <NUM>, and an engine (not shown) that generates power for driving the traveling mechanism <NUM>. The cab <NUM> is provided with a driver's seat (not shown). The wheel loader <NUM> is operated by an operator who is seated in the driver's seat in the cab <NUM>. A driving operation device operated by the operator is disposed around the driver's seat. The driving operation device includes, for example, a shift lever, an accelerator pedal, a brake pedal, and a work equipment lever for operating the work equipment <NUM>. The operator operates the driving operation device, to control the traveling speed of the wheel loader <NUM>, switch between forward and reverse, and operate the work equipment <NUM>.

The traveling mechanism <NUM> has wheels <NUM> that can rotate around a rotating shaft DX. The tire <NUM> is mounted on each of the wheels <NUM>. The wheels <NUM> include two front wheels 5F and two rear wheels 5R. The tires <NUM> include a right front tire 6FR and a left front tire 6FL mounted on the front wheels 5F, and a right rear tire 6RR and a left rear tire 6RL mounted on the rear wheels 5R. In the following, the right front tire 6FR and the left front tire 6FL may be collectively referred to as a front tire 6F, and the right rear tire 6RR and the left rear tire 6RL may be collectively referred to as a rear tire 6R. The traveling mechanism <NUM> can travel on a road surface RS.

In the following description, a direction parallel to the rotating shaft DX when the wheel loader <NUM> travels in a straight advancing state is appropriately referred to as a vehicle width direction of the vehicle body <NUM>, a direction parallel to the vertical axis orthogonal to the road surface RS is appropriately referred to as an up-down direction of the vehicle body <NUM>, and a direction orthogonal to both the rotating shaft DX and the vertical axis is appropriately referred to as a front-rear direction of the vehicle body <NUM>.

The tire <NUM> has, for example, a block pattern (also referred to as a tread pattern) 6P as shown in <FIG>. The block pattern 6P is a pattern formed of a groove <NUM> or the like carved in a tread <NUM> which is a portion where the tire <NUM> is in contact with the road surface RS. In the example shown in <FIG>, the block pattern 6P is a pattern (lug type pattern) in which a plurality of the grooves <NUM> are alternately carved on the left and right at substantially right angles to a circumferential direction of the tire <NUM>.

In the present embodiment, the direction in which the work equipment <NUM> is present is the front, and the opposite direction to the front is the rear, with respect to the operator who is seated in the driver's seat of the cab <NUM>. One in the vehicle width direction is the right, and the opposite direction to the right is the left. The front wheels 5F are disposed in front of the rear wheels 5R. The front wheels 5F are disposed on both sides of the vehicle body <NUM> in the vehicle width direction. The rear wheels 5R are disposed on both sides of the vehicle body <NUM> in the vehicle width direction.

The work equipment <NUM> has an arm <NUM> movably connected to the vehicle body <NUM>, a bucket <NUM> which is an excavation member movably connected to the arm <NUM> via a link <NUM>, and a bell crank <NUM>.

The arm <NUM> is operated by power generated by a lift cylinder <NUM> (<FIG>). The lift cylinder <NUM> is a hydraulic cylinder that generates power for moving the arm <NUM>. One end portion of the lift cylinder <NUM> is connected to the vehicle body <NUM>, and the other end portion of the lift cylinder <NUM> is connected to the arm <NUM>. Two lift cylinders <NUM> are provided. One lift cylinder <NUM> is provided on the right side of the center in the vehicle width direction, and the other lift cylinder <NUM> is provided on the left side of the center in the vehicle width direction. When the operator operates the work equipment lever, the lift cylinder <NUM> expands and contracts. As a result, the arm <NUM> moves in the up-down direction.

The bucket <NUM> is an excavation member having teeth 12B. The excavation member may be a blade having a blade edge. The bucket <NUM> is connected to a tip portion of the arm <NUM> and is connected to the vehicle body <NUM> via the arm <NUM>. The bucket <NUM> is operated by power generated by a bucket cylinder <NUM>. The bucket cylinder <NUM> is a hydraulic cylinder that generates power for moving the bucket <NUM>. A central portion of the bell crank <NUM> is rotatably connected to the arm <NUM>. One end portion of the bucket cylinder <NUM> is connected to the vehicle body <NUM>, and the other end portion of the bucket cylinder <NUM> is connected to one end portion of the bell crank <NUM>. The other end portion of the bell crank <NUM> is connected to the bucket <NUM> via the link <NUM> (<FIG>). One bucket cylinder <NUM> is provided. The bucket cylinder <NUM> is disposed at the center in the vehicle width direction. When the operator operates the work equipment lever, the bucket cylinder <NUM> expands and contracts. As a result, the bucket <NUM> swings. The bucket <NUM> swings in front of the vehicle body <NUM>.

As shown in <FIG> and <FIG>, end portions 12E on both sides of the bucket <NUM> in the vehicle width direction are disposed outside the tire <NUM> in the vehicle width direction. That is, a distance in the vehicle width direction between the right end portion 12E and the left end portion 12E of the bucket <NUM> is larger than a distance in the vehicle width direction between an outer surface of the right tire <NUM> and an outer surface of the left tire <NUM>.

<FIG> is a front view showing the wheel loader <NUM> according to the present embodiment, and shows a state in which the bucket <NUM> is moved upward. In the present embodiment, the traveling mechanism <NUM> has a power transmission mechanism <NUM> that transmits the power generated by the engine to the front wheels 5F, and a housing <NUM> (also referred to as an axle case) that houses at least part of the power transmission mechanism <NUM>. The engine is disposed in a rear part of the vehicle body <NUM>. The power generated by the engine is transmitted to the left and right front wheels 5F via a differential gear of the power transmission mechanism <NUM>. The differential gear is housed in a spherical portion 8B of the housing <NUM>. In the following description, the spherical portion 8B of the housing <NUM> that houses the differential gear is appropriately referred to as an axle ball 8B. The axle ball 8B is disposed at the center in the vehicle width direction. In addition, the axle ball 8B is disposed below the bucket cylinder <NUM>. An axle housing 8C, which is a cover of the axle ball 8B (housing <NUM>), is provided above the axle ball 8B. The housing <NUM> includes a housing 8F for the front wheels 5F and a housing 8R for the rear wheels 5R (<FIG>).

As shown in <FIG>, a road surface condition monitoring system <NUM> according to the present embodiment includes a camera <NUM>, a computer <NUM>, a buzzer <NUM>, and a monitor <NUM>. The camera <NUM> is installed in the axle housing 8C, for example, as shown in <FIG> and <FIG>. The computer <NUM>, the buzzer <NUM>, and the monitor <NUM> are installed in the cab <NUM>.

As shown in <FIG>, the camera <NUM> acquires image data of a region <NUM> between the bucket <NUM> and the front tire 6F. In the present embodiment, the imaging region of the camera <NUM> is the region <NUM> of the road surface RS between the front tire 6F and the bucket <NUM> in a ground contact state in contact with the road surface RS. The camera <NUM> is not limited to being installed in the axle housing 8C and may be installed in a boom connector <NUM> shown in <FIG>, for example. The boom connector <NUM> is a member that connects the left and right arms <NUM> by welding. In addition, the number of the cameras <NUM> is not limited to one and may be a plurality. For example, as shown in <FIG>, the camera can be attached to the back side of the bucket <NUM> (the upper surface of the bucket facing the cab <NUM>) (camera 20a), attached to the top (camera 20b) or back (camera 20e) of a front fender <NUM>, attached to the top, bottom, or side of lighting <NUM> (camera 20c), attached to the top 3a of the ceiling of the cab <NUM> (camera 20d), or attached to the cover of the housing 8R (camera 20f). The cameras <NUM> and 20a to 20f can be mounted on the wheel loader <NUM> via, for example, a bracket, and the bracket may be provided with an adjustment mechanism capable of adjusting the imaging direction. The cameras <NUM> and 20a to 20f shown in <FIG> are represented by two rectangles, and the imaging direction of each camera is from a large rectangle to a small rectangle.

The imaging region is not limited to the region <NUM>, and for example, part or the entirety of a region <NUM>, a region <NUM>, and a region <NUM> may be used as the imaging region. The region <NUM> is a certain region behind the front tire 6F. The region <NUM> is a certain region in front of the rear tire 6R. The region <NUM> is a certain region behind the rear tire 6R. Each of the regions <NUM> to <NUM> can be a region including part of the tire <NUM> and part of the road surface RS. In addition, an imaging region of the camera may be provided for each of the left and right tires <NUM>.

Next, the road surface condition monitoring system <NUM> shown in <FIG> will be described with reference to <FIG> and <FIG>. <FIG> is a block diagram showing an example of the road surface condition monitoring system <NUM> according to the present embodiment. <FIG> is a flowchart showing an operation example of the road surface condition monitoring system <NUM> shown in <FIG>. <FIG> is a schematic diagram showing an example of an image for learning according to the present embodiment. <FIG> and <FIG> are schematic diagrams showing an example of a camera image according to the present embodiment.

As shown in <FIG>, the road surface condition monitoring system <NUM> includes the camera <NUM>, the computer <NUM>, the buzzer <NUM>, and the monitor <NUM> whose installation positions are described with reference to <FIG> and the like.

The computer <NUM> has a processing device <NUM>, a storage device <NUM>, and an input/output device <NUM>. The processing device <NUM> includes hardware such as a central processing unit (CPU), a storage device, and an input/output device inside, and operates by executing, for example, a program stored in the internal storage device. The processing device <NUM> has an image processing unit <NUM> and an image recognition unit <NUM> as functional components composed of a combination of hardware and software such as a program. The storage device <NUM> stores a trained model <NUM> and the like used by the image recognition unit <NUM> in image recognition processing. The input/output device <NUM> inputs an image signal captured by the camera <NUM> and stores the input image signal in a predetermined storage device or outputs the input image signal to the image processing unit <NUM>, superimposes, for example, an image signal indicating a predetermined determination result of the image recognition unit <NUM> on the input image signal and outputs the superimposed image signal to the monitor <NUM>, or outputs a signal indicating a predetermined determination result of the image recognition unit <NUM> to the buzzer <NUM>. That is, the input/output device <NUM> is a device that executes a transmission control of the image signal, a control of display contents displayed on the monitor <NUM> based on the image signal, and a control of sound contents output to the buzzer <NUM> based on the image signal.

The image processing unit <NUM> receives the image signal captured by the camera <NUM>. The image signal is input to the image processing unit <NUM> via the input/output device <NUM>, performs predetermined image processing (for example, resolution conversion and image quality adjustment), and stores the image-processed image signal in a predetermined storage device. The image recognition unit <NUM> receives the image-processed image signal by the image processing unit <NUM>, determines whether or not the image captured by the camera <NUM> contains rocks or the like to be paid attention which may damage the tire <NUM>, and decides information to be output from the buzzer <NUM> or the monitor <NUM> based on the determined result.

The camera <NUM> has a video camera function of acquiring moving image data in the imaging region <NUM>. The image data (moving image data) acquired by the camera <NUM> is input to the input/output device <NUM>.

The buzzer <NUM> as a sound output device outputs information according to a control signal output from the input/output device <NUM>. The buzzer <NUM> generates, for example, an alarm sound audible to the operator in the cab <NUM>. A speaker may be used instead of the buzzer <NUM>, and sound may be output from the speaker by changing it to an alarm sound as information.

The monitor <NUM> is a display device such as a liquid crystal display or an organic electroluminescence display, and displays, for example, an image (moving image or still image) that can be visually recognized by the operator in the cab <NUM> according to the image signal output from the input/output device <NUM>. The monitor <NUM> may use a display device such as a head-up display capable of displaying an image or information on a windshield of the cab <NUM>. The monitor <NUM> may be a single display device or may be composed of a plurality of display devices. In addition, the display device and the sound output device may be integrated. For example, a liquid crystal display and a speaker may be integrated.

As shown in <FIG>, the monitor <NUM> is disposed in the cab <NUM> of the vehicle body <NUM>. The monitor <NUM> displays, for example, the moving image data acquired by the camera <NUM> in real time, or displays information according to the determination result of the image recognition unit <NUM>. Although the operator of the cab <NUM> can visually recognize the bucket <NUM>, the arm <NUM>, the bucket cylinder <NUM>, and the like via the windshield <NUM>, it is difficult to visually recognize the condition of the road surface RS. In particular, it is difficult to directly visually check the condition of the road surface RS in front of the front tire 6F. In addition, when the bucket <NUM> is grounded, the condition of the road surface RS on a lower surface of the bucket <NUM> and the condition of the road surface RS in front of the bucket <NUM> are also difficult to be visually recognized by the operator of the cab <NUM>. In any case, the condition of the road surface in front of the front tire <NUM> becomes less visible as it comes closer to the front tire <NUM>. On the other hand, when the monitor <NUM> displays the moving image data acquired by the camera <NUM> in real time, the operator of the cab <NUM> views the monitor <NUM> provided in the cab <NUM> and can visually recognize, for example, the condition of the road surface RS (region <NUM>) between the bucket <NUM> and the front tire 6F.

Next, an operation example of the road surface condition monitoring system <NUM> shown in <FIG> will be described with reference to <FIG>. The process shown in <FIG> is repeatedly executed by the computer <NUM> at a predetermined cycle. When the process shown in <FIG> is started, the input/output device <NUM> acquires the image signal output by the camera <NUM> for one or a plurality of frames and stores the image signal in a predetermined storage device (Step S11). In Step S11, the input/output device <NUM> may perform, for example, a process of outputting the image signal input from the camera <NUM> to the monitor <NUM> as it is in response to an instruction from the image recognition unit <NUM>. In this case, an image captured by the camera <NUM> can be displayed in real time on the monitor <NUM> in response to the instruction from the image recognition unit <NUM>.

Next, the image processing unit <NUM> inputs the image signal stored in the predetermined storage device in Step S11, performs predetermined image processing thereon, and then stores the image signal in the predetermined storage device again (Step S12).

Next, the image recognition unit <NUM> executes image recognition processing on the image signal of one or a plurality of frames stored in the predetermined storage device, and determines whether or not the region <NUM> includes the rock <NUM> to be paid attention which may damage the tire <NUM> (Step S13). When the image recognition unit <NUM> determines in Step S13 that the rock <NUM> to be paid attention is included ("YES" in Step S13), the image recognition unit <NUM> outputs an instruction to the input/output device <NUM> to issue information (alarm sound) indicating the determination result from the buzzer <NUM> or to display information (alarm image) indicating the determination result on the monitor <NUM> (Step S14), to alert the operator, and the process shown in <FIG> ends. On the other hand, when the image recognition unit <NUM> does not determine in Step S13 that the rock <NUM> to be paid attention is included ("NO" in Step S13), the process shown in <FIG> ends.

Here, an example of the determination processing (image recognition processing) in Step S13 will be described. The determination processing by the image recognition unit <NUM> can be processing of determining whether the image signal to be determined is classified into an image containing a rock to be paid attention or an image not containing a rock to be paid attention, by using the trained model <NUM> stored in the storage device <NUM>. The trained model <NUM> is a trained model using a neural network such as a convolution neural network (CNN) as an element, and weighting coefficients between neurons in each layer of the neural network are optimized by machine learning so that a solution obtained for a large number of input data is output. The trained model <NUM> is composed of, for example, a combination of a program that performs an operation from input to output and a weighting coefficient (parameter) used for the operation.

The trained model <NUM> can be generated as follows, for example. That is, for example, as shown in <FIG>, a plurality of pieces of image data <NUM> including the rock pile <NUM> and the rock <NUM> to be paid attention, a plurality of pieces of image data <NUM> including the rock pile <NUM> without the rock <NUM> to be paid attention, and a plurality of pieces of image data <NUM> including neither the rock pile <NUM> nor the rock <NUM> to be paid attention are prepared. Then, the plurality of pieces of image data <NUM> are defined as data including the rock <NUM> to be paid attention and the rock pile <NUM>. In addition, the plurality of pieces of image data <NUM> are defined as data including the rock pile <NUM> without the rock <NUM> to be paid attention. In addition, the plurality of pieces of image data <NUM> are defined as data including neither the rock pile <NUM> nor the rock <NUM> to be paid attention. The defined plurality of pieces of image data <NUM>, <NUM>, and <NUM> are prepared as a data set <NUM> for learning. As described above, in the data set for learning <NUM>, classification (labeling) such as the presence or absence of the rock <NUM> to be paid attention for each image data is linked as incidental information. The trained model <NUM> is generated by machine learning by supervised learning using the data set <NUM> for learning. In the example shown in <FIG>, the pieces of image data <NUM> to <NUM> are all images including part of the tire <NUM> (block pattern 6P). In this case, it is considered that the size and orientation of the rock <NUM> can be easily grasped with reference to the tire <NUM> (block pattern 6P) as compared with a case where part of the tire <NUM> (block pattern 6P) is not included, and the learning accuracy can be improved. The initial classification (labeling) of the image data included in the data set <NUM> for learning can be performed, for example, manually or by image recognition processing such as pattern matching based on the conditions described below.

The rock <NUM> to be paid attention can be defined as, for example, a rock having a certain size or larger or a rock having a sharp edge angle, and defined as being in a case where it is assumed that the tire <NUM> is highly likely to be damaged when traveling toward the rock, based on the relative position and the relative orientation with respect to the tire <NUM>. In addition, the rock <NUM> to be paid attention can be defined as a rock having a certain size or larger or a rock having a sharp edge angle and defined as being in a case where the position of the rock <NUM> does not exist on a slope of the rock pile <NUM> (or exists on a flat surface).

A rock smaller than a certain size has a high possibility of avoiding the damage by flexibility of the tire <NUM> and is not the rock to be paid attention, but the rock having a certain size or larger has a high possibility of causing the damage due to the weight of the wheel loader <NUM> when the wheel loader <NUM> goes over the rock. In addition, when a possibility of damaging the tire <NUM> is considered with respect to a rock having a round shape, the rock having a round shape has a low possibility of piercing the tire <NUM> or cutting the tire <NUM> and is not a rock to be paid attention.

The rock pile <NUM> is an object in which a plurality of rocks, earth, and the like are accumulated, and is an area to be worked by the work equipment <NUM>, so that the tire <NUM> does not normally enter the rock pile <NUM>. Therefore, the rock located on the slope of the rock pile <NUM> can be excluded from the rock <NUM> to be paid attention since it does not lie on the road surface RS. Note that the "rock pile" is merely one aspect of "an area where the tires of the work vehicle do not enter". For example, "rock loaded in a dump truck" may be defined as "an area where the tires of the work vehicle do not enter" in the same manner as the "rock pile". The fact that the rock does not exist on the slope of the rock pile <NUM> is used as a condition for determining the rock <NUM> to be paid attention. In this way, for example, when rocks are scattered all over a mine, it is possible to avoid determining that the rocks are the rock <NUM> to be paid attention, and as a result, it is possible to prevent unnecessary alarm from being issued.

As described above, in the data set <NUM> for learning used when the trained model <NUM> is constructed by machine learning, the condition relating to the size of the rock (monitoring object) or the shape of the rock (shape with a sharp edge angle) is included in the information for defining whether or not the image data includes the rock <NUM> to be paid attention, so that the image recognition unit <NUM> (damage determination unit) can use information on the shape or size of the rock (monitoring object) included in the image captured by the camera <NUM> as an element for the determination.

In addition, for the data set used when the trained model <NUM> is constructed by machine learning, the condition relating to the monitoring object information (relative position and relative orientation, or whether or not the position is on the slope of the rock pile <NUM>) of the rock (monitoring object) with respect to the tire <NUM> is included in the information defining whether or not the image data includes the rock <NUM> to be paid attention, so that the image recognition unit <NUM> (damage determination unit) can use, as an element for the determination, information on the relative position and the relative orientation of the rock (monitoring object) and whether or not the rock is on the slope, with respect to the tire <NUM> included in the image captured by the camera <NUM>.

In addition, in the data set used when the trained model <NUM> is constructed by machine learning, part of the tire <NUM> (block pattern 6P) is included, so that the image recognition unit <NUM> (damage determination unit) can use tire information on the shape of the block pattern 6P or the size of the block pattern 6P of the tire <NUM> as an element for the determination.

Regarding the data set <NUM> for learning, it is desirable to prepare the data set <NUM> for learning at night so that the determination processing can be performed even though an image in which a rock is illuminated by a light source mounted on the work vehicle is acquired during night work. In this case, this system functions effectively even during night work. In addition, the data set <NUM> for learning at night may be created based on an image that reproduces the appearance at night by performing image processing such as color tone correction based on an image acquired during daytime.

Due to the material characteristics of the tire, the tire is more susceptible to damage in rainy weather than in fine weather. For example, a first trained model <NUM> based on the data set for learning <NUM> in rainy weather and a second trained model <NUM> based on the data set for learning <NUM> other than in rainy weather may be prepared, and the second trained model <NUM> may be switched to the first trained model <NUM> in rainy weather (for example, a raindrop sensor). For example, when a raindrop sensor is installed outside the cab <NUM> and the raindrop sensor detects rainfall, a detection signal may be output to the computer <NUM>, and switching from the second trained model to the first trained model may be performed by the input/output device <NUM> in response to the input of the detection signal.

The image of the data set for learning <NUM> may be artificially created by using software for creating computer graphics. In addition, the tire need not be reflected in the image of the data set for learning <NUM>.

As the image recognition technique, any image recognition technique such as pattern matching can be used in addition to the trained model such as CNN. In addition, in the data set for learning, for example, an image generated by using the image generation technique such as auxiliary classifier generative adversarial Network (ACGAN) can be used.

<FIG> and <FIG> are schematic diagrams showing an example of a captured image by the camera <NUM> according to the present embodiment. An image <NUM> shown in <FIG> includes the rock pile <NUM> surrounded by a chain line frame and one rock <NUM> surrounded by a broken line frame. The rock <NUM> shown in the example of this captured image is a rock to be paid attention for the damage to the tire <NUM> and exists on a flat surface (road surface RS). The image <NUM> (image-processed) is input to the trained model <NUM>, so that the image recognition unit <NUM> can obtain the determination result that the image includes the rock <NUM> to be paid attention by using the trained model <NUM>. In addition, an image <NUM> shown in <FIG> includes the rock pile <NUM> surrounded by a chain line frame and four rocks <NUM> surrounded by a broken line frame. The rock <NUM> shown in the example of this captured image is a rock to be paid attention for the damage to the tire <NUM> and exists on a flat surface (road surface RS). The image <NUM> (image-processed) is input to the trained model <NUM>, so that the image recognition unit <NUM> can obtain the determination result that the image includes the rock <NUM> to be paid attention by using the trained model <NUM>.

As described above, the road surface condition monitoring system <NUM> can identify whether or not the monitoring object is a rock to be paid attention by the image recognition technique by using the image captured by the in-vehicle camera <NUM> as an input to the computer <NUM>. The rock to be paid attention in the present embodiment can be a rock having a certain size or larger, a rock with a sharp edge angle, a rock existing on a flat surface not on a rock pile, and the like. According to the present embodiment, when there is the rock to be paid attention, the buzzer <NUM> issues a warning and the monitor <NUM> displays a warning, and the operator can be alerted. When the operator receives the alert, the operator can take measures to prevent the front tire 6F from being damaged by viewing the moving image data displayed on the monitor <NUM>. The operator can take measures to prevent the front tire 6F from being damaged by, for example, confirming that the rock <NUM> lies in front of the front tire 6F by viewing the moving image data displayed on the monitor <NUM> in accordance with the received alert, operating a brake to stop the wheel loader <NUM> or operating a steering to change the traveling direction of the wheel loader <NUM> so that the front tire 6F does not ride on the rock <NUM> to be paid attention.

According to the present embodiment, without looking the monitor <NUM> always or frequently, the operator need only view the monitor <NUM> when the rock <NUM> to be paid attention exists in the vicinity of the tire <NUM> and can perform an operation to surely avoid the damage to the tire <NUM>. That is, according to the present embodiment, the operator can normally execute excavation work and the like without concentrating on the image of the camera <NUM> and can carefully execute the traveling operation and take necessary measures to avoid the tire damage only when the rock <NUM> to be paid attention comes closer to the tire <NUM>. According to the present embodiment, it is possible to monitor the road surface condition even though the operator does not always or frequently view the monitor <NUM>, and to improve workability and productivity.

Next, with reference to <FIG>, a basic configuration example of the embodiment including the above-described embodiment will be described. <FIG> is a block diagram showing the basic configuration example of the embodiment including the above-described embodiment. The same reference numerals are appropriately used for the same or corresponding configurations as those shown in <FIG> and <FIG>. Hereinafter, a basic configuration example of the embodiment including a modification example of the above-described embodiment will be described.

A road surface condition monitoring system <NUM> shown in <FIG> includes a road surface condition acquisition unit <NUM>, a storage unit <NUM>, a damage determination unit <NUM>, and an output unit <NUM> as functional components composed of, for example, hardware such as a computer and its peripheral devices and software such as a program. In addition, the storage unit <NUM> stores reference information <NUM>.

The road surface condition acquisition unit <NUM> acquires the monitoring object information on at least the shape or size of the monitoring object <NUM> existing in the area including the road surface RS in the direction in which the work vehicle <NUM> travels by driving of the traveling mechanism <NUM>, which mounts a tire, of the work vehicle <NUM>. The work vehicle <NUM> can be a tire-based work vehicle such as a wheel loader, a motor grader, or a dump truck. The monitoring object information may further include information on the relative position and the relative orientation of the monitoring object with respect to the tire <NUM>. The monitoring object information may further include information on the existence position of the monitoring object with respect to the tire <NUM>. The road surface condition acquisition unit <NUM> can be a camera (monocular, stereo, infrared ray), a radar scanner, or the like. The monitoring object is, for example, a rock. Note that holes in the road surface that cause the tire damage can also be monitored. In addition, when a wheel loader is operated at an industrial waste treatment plant, the object to be monitored may be a sharp metal object. Of course, even such a metal object may cause the tire damage. Further, the road surface is not limited to the soil road surface, but may be a road surface paved with asphalt or concrete.

The storage unit <NUM> stores the reference information <NUM> for determining whether or not the tire <NUM> is likely to be damaged. The reference information <NUM> is a trained model if the determination is made by artificial intelligence (AI), pattern data if the determination is made by image processing (pattern matching), waveform data if the determination is made by a radar scanner, and the like.

The damage determination unit <NUM> determines, based on the monitoring object information acquired by the road surface condition acquisition unit <NUM> and the reference information stored in the storage unit <NUM>, whether the tire <NUM> is to be damaged when the tire <NUM> comes into contact with the monitoring object <NUM> by the driving of the traveling mechanism <NUM>. The damage determination unit <NUM> may determine whether the tire is to be damaged by using the tire information on the shape of the block pattern or the size of the block pattern of the tire. The damage determination unit <NUM> makes the determination based on any of the determination by artificial intelligence (AI), the determination by pattern matching by image processing, the determination by analysis of a reception signal of a radar scanner, and the like.

Then, the output unit <NUM> outputs the result determined by the damage determination unit <NUM>. When there are a plurality of the tires <NUM>, the output unit <NUM> may output information indicating which tire <NUM> is to be damaged as a result of the determination by the damage determination unit <NUM>. In this case, the road surface condition acquisition unit <NUM> may have a number corresponding to the number of the tires <NUM>. The output unit <NUM> can perform a sound output from a speaker in the cab, an image output to a monitor, an output to a head-up display, an output by vibration of an operation lever, and the like.

In a remote operation of the work vehicle <NUM>, the result determined by the damage determination unit <NUM> may be output to a place away from the work vehicle <NUM>. The result output by the output unit <NUM> may be a result indicating that there is no risk of the tire damage due to an object. That is, the output unit <NUM> may output not only that there is an object on the road surface RS but also that there is no object on the road surface RS (having no sharp rock or hole or metal object that may damage the tire), and the output unit <NUM> outputs the result of the "road surface condition monitoring".

According to the road surface condition monitoring system <NUM> shown in <FIG>, the road surface condition can be monitored even though the operator does not view the monitor always or frequently.

The output unit <NUM> may output a signal for controlling the brake of the work vehicle.

In addition, the output unit <NUM> may output a signal for controlling the steering of the work vehicle <NUM>. When the damage determination unit <NUM> determines that the monitoring object may damage the tire <NUM>, the actuator controls the steering based on a signal transmitted from the output unit <NUM> to the actuator. The work vehicle <NUM> can automatically turn in a direction to avoid damage to the tire <NUM> by controlling the steering of the work vehicle <NUM>.

In addition, the output unit <NUM> may output a signal for controlling the engine speed of the work vehicle <NUM>. When the damage determination unit <NUM> determines that the monitoring object may damage the tire <NUM>, the output unit <NUM> outputs a signal for instructing reduction of the engine speed. The signal is transmitted to a controller that executes engine control, and the controller can reduce an output of the engine and reduce the speed of the work vehicle <NUM>.

In addition, the output unit <NUM> may output a signal for controlling a posture of the bucket of the work vehicle <NUM>. When the damage determination unit <NUM> determines that the monitoring object may damage the tire <NUM>, a hydraulic valve that controls the operation of the work equipment <NUM> can automatically lower the bucket of the work vehicle <NUM> based on a signal transmitted from the output unit <NUM> to the hydraulic valve, to avoid the contact between the monitoring object and the tire <NUM>.

The road surface condition monitoring system <NUM> may have all the configurations in the work vehicle <NUM>, but for example, when the work vehicle <NUM> is provided with a device capable of being remotely operated and the work vehicle <NUM> is remotely operated, a configuration of part (for example, a display device or a sound output device connected to the output unit <NUM>) of the output unit <NUM> or the like other than the road surface condition acquisition unit <NUM> may be provided at a remote location among the components of the road surface condition monitoring system <NUM>. For example, it is assumed that the output unit <NUM> includes a display device (not shown) at a remote location. Information output from the output unit <NUM> is transmitted to a display device (not shown) at a remote location via wireless communication or the like, and the display device displays or outputs information (alarm) on the alert about the monitoring object to be paid attention. When the operator who executes the remote operation recognizes the display or output of the display device and confirms the monitoring object, the steering operation or brake operation of the work vehicle <NUM> can be executed by the remote operation such that the tire <NUM> of the work vehicle <NUM> does not come into contact with the monitoring object.

A vehicle speed sensor may be provided, and the output unit <NUM> receiving a signal indicating the vehicle speed of the work vehicle from the vehicle speed sensor may switch information for the alert according to the vehicle speed. For example, when the work vehicle travels at a high speed faster than a predetermined speed, the output unit <NUM> may output a warning with a high alarm level, and when the work vehicle <NUM> travels at a low speed lower than a predetermined speed, the output unit <NUM> may output a warning with a low alarm level.

The correspondence between the configuration of the embodiment described with reference to <FIG> and <FIG> and the configuration shown in <FIG> is as follows. The road surface condition monitoring system <NUM> shown in <FIG> and <FIG> corresponds to the road surface condition monitoring system <NUM> shown in <FIG>. Part of the combination of the camera <NUM> and the input/output device <NUM> shown in <FIG> corresponds to the road surface condition acquisition unit <NUM> shown in <FIG>. The image recognition unit <NUM> shown in <FIG> corresponds to the damage determination unit <NUM> shown in <FIG>. The storage device <NUM> shown in <FIG> corresponds to the storage unit <NUM> shown in <FIG>. The trained model <NUM> shown in <FIG> corresponds to the reference information <NUM> shown in <FIG>. The combination of part of the input/output device <NUM>, the buzzer <NUM>, and the monitor <NUM> shown in <FIG> corresponds to the output unit <NUM> shown in <FIG>. The rock <NUM> to be paid attention shown in <FIG> corresponds to the monitoring object <NUM> shown in <FIG>.

Hereinabove, the embodiments of the invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to those embodiments and also includes changes within the scope of the appended claims.

In addition, some or all of programs executed by a computer in the above embodiment can be distributed via a computer-readable recording medium or a communication line.

Claim 1:
A road surface condition monitoring system comprising:
a road surface condition acquisition unit (<NUM>) configured to acquire monitoring object information on at least a shape or size of a monitoring object (<NUM>, <NUM>) existing in an area including a road surface (RS) in a direction in which a work vehicle (<NUM>) travels by driving of a traveling mechanism (<NUM>) of the work vehicle (<NUM>), the traveling mechanism (<NUM>) mounting a tire (<NUM>);
a storage unit (<NUM>) configured to store reference information (<NUM>) for determining whether the tire (<NUM>) is to be damaged; and
an output unit (<NUM>, <NUM>);
characterized by
a damage determination unit (<NUM>) configured to determine, before the tire (<NUM>) comes into contact with the monitoring object (<NUM>, <NUM>), based on the monitoring object information and the reference information (<NUM>), whether the tire (<NUM>) is to be damaged when the tire (<NUM>) comes into contact with the monitoring object (<NUM>, <NUM>) by the driving of the traveling mechanism (<NUM>); and in that
the output unit (<NUM>, <NUM>) is configured to output a result determined by the damage determination unit (<NUM>).