PARKING ASSISTANCE DEVICE

A parking assistance device performs a process for parking a vehicle in a parking area based on a captured image obtained by an imaging device mounted on the vehicle, and includes a storage part that stores the captured image; a first processing part that detects parking areas present in an imaging area of the imaging device, based on a low-resolution image obtained by reducing the resolution of the captured image; a selecting part that selects a target parking area from the detected parking areas when the parking areas have been detected by the first processing part; an extracting part that extracts an extraction image including the target parking area from the captured image stored in the storage part; and a second processing part that generates, based on the extraction image, target information indicating a target stop location used upon parking the vehicle in the target parking area.

TECHNICAL FIELD

The present disclosure relates to a parking assistance device.

BACKGROUND ART

A device that assists in parking a vehicle uses a technique for recognizing parking areas based on a result of analysis of a captured image obtained by an imaging device mounted on the vehicle.

CITATIONS LIST

Patent Literature

Patent Literature 1: JP 2018-041176 A

SUMMARY OF THE DISCLOSURE

Technical Problems

To improve the quality of parking assistance control, there is a need to improve the accuracy of recognition of parking areas. To improve the accuracy of recognition of parking areas, it is effective to use a high-resolution image as an analysis target, but the computational load increases as the resolution of an image increases, increasing the possibility of causing a reduction in processing speed, an increase in cost, etc.

Hence, one problem of the present disclosure is to provide a parking assistance device that can improve the accuracy of recognition of parking areas while suppressing an increase in computational load.

Solutions to Problems

One aspect of the present disclosure is a parking assistance device that performs a process for parking a vehicle in a parking area based on a captured image obtained by an imaging device mounted on the vehicle, and the parking assistance device includes: a storage part that stores the captured image; a first processing part that detects parking areas present in an imaging area of the imaging device, based on a low-resolution image obtained by reducing resolution of the captured image; a selecting part that selects a target parking area from the detected parking areas when the parking areas have been detected by the first processing part; an extracting part that extracts an extraction image including the target parking area from the captured image stored in the storage part; and a second processing part that generates, based on the extraction image, target information indicating a target stop location used upon parking the vehicle in the target parking area.

According to the above-described configuration, upon detecting (searching for) parking areas present around the vehicle, a low-resolution image is used, and upon generating target information indicating a target stop location used upon parking the vehicle in a target parking area selected from the detected parking areas, an extraction image extracted from a captured image is used. By this, while an increase in computational load is suppressed, the accuracy of recognition of parking areas is improved, enabling an improvement in the quality of parking assistance control.

In addition, the first processing part may detect one or more parking areas using a first trained model and based on a result of analysis of the low-resolution image, and the second processing part may detect the target parking area using a second trained model and based on a result of analysis of the extraction image, the second trained model being different from the first trained model.

Like the above-described configuration, by performing, using different trained models, a process for detecting one or more parking areas from a captured image and a process for detecting one target parking area from an extraction image, the overall process can be made efficient.

In addition, the second trained model may analyze a resized version of the extraction image, the extraction image being resized to a predetermined size.

By this, analysis can be performed on an extraction image of a fixed size regardless of the shape of a target parking area.

In addition, the parking assistance device may include: a first processor that generates an instruction signal for a traveling mechanism of the vehicle based on the target information; and a second processor that is connected to the first processor such that the first processor and the second processor can perform transmission and reception of information with each other, and that performs a process using the first trained model and a process using the second trained model.

Like the above-described configuration, by allowing different processors to perform a process of controlling a traveling mechanism and a process of recognizing parking areas using trained models, load concentration on a specific processor can be avoided.

DESCRIPTION OF EMBODIMENTS

An exemplified embodiment of the present disclosure is described below. Configurations of the embodiment shown below and the effects, results, and advantageous effects that are brought about by the configurations are examples. Aspects of the present disclosure can also be implemented by other configurations than the configurations disclosed in the following embodiment, and at least one of various advantageous effects and derivative advantageous effects based on a basic configuration can be obtained.

FIG.1is a diagram showing an example of a configuration of a vehicle1according to the embodiment. The vehicle1is controlled by a parking assistance device according to the present embodiment, and is a four-wheel automobile having a pair of left and right front wheels3F and a pair of left and right rear wheels3R.

The vehicle1has a plurality of (four in the present embodiment) imaging devices15ato15d. The imaging devices15ato15deach have an area around the vehicle1as their imaging areas. The imaging device15ais provided at an end part2eon a rear side of a vehicle body2(e.g., at the bottom of a rear trunk door) and captures an image of an area behind the vehicle1. The imaging device15bis provided on a door mirror2gat an end part2fon a right side of the vehicle body2and captures an image of an area on the right side of the vehicle1. The imaging device15cis provided at an end part2con a front side of the vehicle body2(e.g., on a front bumper) and captures an image of an area in front of the vehicle1. The imaging device15dis provided on a door mirror2gat an end part2don a left side of the vehicle body2and captures an image of an area on the left side of the vehicle1. The imaging devices15ato15deach obtain a captured image (image data) corresponding to their imaging areas. When the imaging devices15ato15ddo no need to be distinguished from each other, the imaging devices15ato15dmay be collectively referred to as imaging devices15.

FIG.2is a block diagram showing a configuration of a vehicle control system20according to the embodiment. The vehicle control system20according to the present embodiment includes the imaging devices15, a braking mechanism22, a driving mechanism24, a steering mechanism26, a transmission mechanism28, a vehicle speed sensor30, a monitor device32, a parking assistance device34, and an intra-vehicle network36.

The braking mechanism22is a mechanism for decelerating the vehicle1. The braking mechanism22includes a braking part40, a braking control part42, and a braking-part sensor44. The braking part40includes, for example, brake pads, brake discs, and a brake pedal. The braking control part42is, for example, an electronic device including a central processing unit (CPU), etc. The braking control part42controls the braking part40based on an instruction signal from the parking assistance device34to control deceleration of the vehicle1. The braking-part sensor44is, for example, a sensor that detects the position of the brake pedal, and outputs a result of the detection to the intra-vehicle network36.

The driving mechanism24is a mechanism for driving (accelerating) the vehicle1. The driving mechanism24includes a driving part46, a drive control part48, and a driving-part sensor50. The driving part46includes, for example, an internal combustion engine, an electric motor, and an accelerator pedal. The drive control part48is, for example, an electronic device including a CPU, etc. The drive control part48controls the driving part46based on an instruction signal from the parking assistance device34to control the speed or acceleration of the vehicle1. The driving-part sensor50is, for example, a sensor that detects the position of the accelerator pedal, and outputs a result of the detection to the intra-vehicle network36.

The steering mechanism26is a mechanism for changing a traveling direction of the vehicle1. The steering mechanism26includes a steering part52, a steering control part54, and a steering-part sensor56. The steering part52includes, for example, a steering wheel. The steering control part54is, for example, an electronic device including a CPU, etc. The steering control part54controls the steering part52based on an instruction signal from the parking assistance device34to control the traveling direction of the vehicle1. The steering-part sensor56is, for example, a sensor that detects the turning angle (steering angle) of the steering wheel, and outputs a result of the detection to the intra-vehicle network36.

The transmission mechanism28is a mechanism for changing the transmission gear ratio of the vehicle1. The transmission mechanism28includes a transmission part58, a transmission control part60, and a transmission-part sensor62. The transmission part58includes, for example, an automatic transmission, a continuously variable transmission, a manual transmission, and a gearshift. The transmission control part60is, for example, an electronic device including a CPU, etc. The transmission control part60controls the transmission part58based on an instruction signal from the parking assistance device34to control the transmission gear ratio of the vehicle1. The transmission-part sensor62is, for example, a sensor that detects the position of the gearshift, and outputs a result of the detection to the intra-vehicle network36.

The vehicle speed sensor30is, for example, a sensor that detects the amount of rotation of wheels3of the vehicle1or the number of rotations per unit time made by the wheels3. The vehicle speed sensor30outputs a result of the detection to the intra-vehicle network36.

The monitor device32is a user interface provided in a dashboard, etc., in the interior of the vehicle1. The monitor device32includes a display part64, an audio output part66, and an operation input part68.

The display part64displays an image based on image data transmitted from the parking assistance device34. The display part64is, for example, a display device such as a liquid crystal display or an organic EL display. The display part64displays an image that allows a user to perform, for example, an operation for switching between autonomous driving and manual driving or for selecting a target parking area.

The audio output part66outputs audio based on audio data transmitted from the parking assistance device34. The audio output part66is, for example, a speaker.

The operation input part68accepts passenger's input. The operation input part68is, for example, a touch panel. The operation input part68is provided on a display screen of the display part64. The operation input part68is configured to allow an image displayed on the display part64to pass therethrough. By this, the operation input part68allows a passenger to visually recognize an image displayed on the display screen of the display part64. The operation input part68accepts an instruction inputted by the passenger touching a location corresponding to an image displayed on the display screen of the display part64, and transmits the instruction to the parking assistance device34. Note that the operation input part68is not limited to a touch panel and may be a hard switch of a push-button type, etc.

The parking assistance device34is an electronic control unit that performs various types of processes for assisting in traveling upon parking the vehicle1in a parking area. The parking assistance device34according to the present embodiment includes a CPU34a(first processor), a read only memory (ROM)34b, a random access memory (RAM)34c, a display control part34d, an audio control part34e, a solid state drive (SSD)34f, and a field programmable gate array (FPGA)34g(second processor). The CPU34a, the ROM34b, and the RAM34cmay be integrated in the same package.

The CPU34areads a program stored in a non-volatile storage device such as the ROM34b, and performs various types of computation processes and control processes in accordance with the program. Upon providing parking assistance by autonomous driving (including semi-autonomous driving) of the vehicle1, the CPU34aaccording to the present embodiment generates an instruction signal for controlling a traveling mechanism (at least one of the braking mechanism22, the driving mechanism24, the steering mechanism26, and the transmission mechanism28) of the vehicle.

The ROM34bstores a program, parameters required to execute the program, etc. The RAM34ctemporarily stores various types of data used in computations performed by the CPU34a.

The display control part34dis an integrated circuit that mainly performs a process on image data obtained by the imaging devices15among computation processes performed by the parking assistance device34. The display control part34dperforms, for example, data conversion of an image for display to be displayed on the display part64and data conversion upon outputting image data from the imaging devices15to the CPU34a. The audio control part34eis an integrated circuit that mainly performs a process on audio to be outputted by the audio output part66among computation processes performed by the parking assistance device34. The SSD34fis a non-volatile, rewritable storage device, and maintains data even when the power to the parking assistance device34is turned off. The SSD34fat least temporarily stores image data (captured images) obtained by the imaging devices15.

The FPGA34gmainly performs an analysis process for recognizing parking areas, based on a captured image obtained by an imaging device15among computation processes performed by the parking assistance device34. The FPGA34gaccording to the present embodiment is hardware that analyzes a captured image using trained models created by predetermined machine learning (deep learning).

The intra-vehicle network36includes, for example, a controller area network (CAN) and a local interconnect network (LIN). The intra-vehicle network36connects the braking mechanism22, the driving mechanism24, the steering mechanism26, the transmission mechanism28, the vehicle speed sensor30, the operation input part68of the monitor device32, the parking assistance device34, etc., to each other such that they can mutually perform transmission and reception of information.

Note that a hardware configuration shown inFIG.2is an example and the configuration of the parking assistance device34is not limited to the one described above.

FIG.3is a block diagram showing an example of a functional configuration of the parking assistance device34according to the embodiment. The parking assistance device34according to the present embodiment includes an obtaining part101, a storage part102, a resolution reduction part103, a first processing part104, a selecting part105, an extracting part106, a resizing part107, a second processing part108, and a travel control part109. These functional components are implemented by cooperation of hardware such as that described above and software such as a program.

The obtaining part101obtains a captured image obtained by capturing an image of an area around the vehicle1.

The storage part102stores the captured image obtained by the obtaining part101.

The resolution reduction part103generates a low-resolution image obtained by reducing the resolution of the captured image stored in the storage part102(obtained by the obtaining part101).

The first processing part104detects parking areas present in an imaging area of an imaging device15based on the low-resolution image generated by the resolution reduction part103, and generates parking area information about the parking areas. The parking areas are areas in which the vehicle1can be parked and are, for example, areas marked off by parking lines in a parking lot. The parking area information includes information that allows estimation of the location, area, shape, etc., of each parking area and includes, for example, information indicating the locations of points of intersection of parking lines, directions in which the parking lines extend, etc. The first processing part104according to the present embodiment analyzes the low-resolution image using a first model111which is a trained model, and detects (searches for) parking areas based on a result of the analysis.

When parking areas have been detected by the first processing part104, the selecting part105selects one target parking area from the parking areas. The target parking area is a parking area in which the vehicle1is finally parked, and may be automatically selected by the parking assistance device34based on a predetermined condition or may be manually selected by a user. The selecting part105generates target parking area information about the selected target parking area. The target parking area information includes, for example, information indicating the location, area, shape, etc., of the target parking area.

Based on the target parking area information generated by the selecting part105, the extracting part106extracts an extraction image including the target parking area from the captured image stored in the storage part102. For example, the extraction image is extracted by cutting an image of a partial area including the target parking area from the captured image.

The resizing part107changes (resizes) the size of the extraction image extracted by the extracting part106to a predetermined size.

The second processing part108generates target information indicating a target stop location used upon parking the vehicle1in the target parking area, based on the extraction image having been resized by the resizing part107. The target information includes, for example, information that allows estimation of a positional relationship between the vehicle1and the target stop location and includes, for example, information indicating the locations of points of intersection of parking lines of the target parking area, directions in which the parking lines extend, etc.

The second processing part108according to the present embodiment has a second model121. The second model121is a trained model different from the first model111. The second processing part108according to the present embodiment analyzes the extraction image (resized extraction image) using the second model121, detects (captures) a target stop location in the target parking area based on a result of the analysis, and generates target information. In addition, the second model121may analyze the extraction image further using auxiliary information indicating the location of the target parking area which is included in the parking area information generated by the first processing part104(first model111).

Based on the target information generated by the second processing part108, the travel control part109generates a control signal for controlling a traveling mechanism (at least one of the braking mechanism22, the driving mechanism24, the steering mechanism26, and the transmission mechanism28) of the vehicle1.

According to the above-described configuration, upon detecting (searching for) parking areas present around the vehicle, a low-resolution image is used, and upon generating target information indicating a target stop location used upon parking the vehicle1in a target parking area selected from the detected parking areas, an extraction image extracted from a captured image is used. By this, while an increase in computational load is suppressed, the accuracy of recognition of parking areas is improved, enabling an improvement in the quality of parking assistance control.

FIG.4is a diagram showing examples of a captured image201and a low-resolution image202according to the embodiment. The captured image201exemplified here is an image captured by the imaging device15aprovided at the end part2eon the rear side of the vehicle1. The captured image201can be, for example, an image whose number of pixels is about 1280*800. The captured image201includes six parking areas211A to211F. Of the six parking areas211A to211F, two parking areas211A and211B are vacant and four parking areas211C to211F have other vehicles parked therein. The parking areas211A to211F may be hereinafter collectively referred to as parking areas211.

In the middle ofFIG.4there is exemplified a low-resolution image202obtained by reducing the resolution of the captured image201. The low-resolution image202is obtained by converting the entire captured image201into an image having resolution with a smaller number of pixels than that of the captured image201. The first processing part104detects candidates for parking areas in which the vehicle1can be parked, based on a result of analysis obtained by inputting such a low-resolution image202to the first model111. Here, as shown at the bottom ofFIG.4, the two vacant parking areas211A and211B are detected as candidates, and parking area information about the parking areas211A and211B is generated.

FIG.5is a diagram showing examples of extraction images231A and231B and an analysis result image251according to the embodiment. Here, a case is exemplified in which of the parking areas211A and211B detected as described above, the parking area211A closest to the vehicle1is selected as a target parking area225. In this case, the extraction image231A including the target parking area225(in the upper right ofFIG.5) is extracted from a captured image201stored in the storage part102(in the upper left ofFIG.5). Upon the extraction, the extraction image231A is extracted so as to include the entire target parking area225and so as not to include the entire other parking areas (the parking areas211B to211F adjacent to the target parking area225). The extraction image231A is resized to the extraction image231B of a predetermined size (in the lower right ofFIG.5). By inputting the resized extraction image231B to the second model121, detection of a target stop location for the target parking area225is performed.

Note that although, in this example, the extraction image231A before resizing is rectangular and the extraction image231B after resizing is square, the form of resizing is not limited thereto. The shape of the extraction image231A before resizing varies depending on the shape, etc., of the target parking area225. By performing resizing as described above, analysis by the second model122can be performed using the extraction image231B of a fixed size regardless of the shape of the target parking area225.

In the lower left ofFIG.5there is exemplified the analysis result image251in which a result of analysis performed on the extraction image231B is reflected in the captured image201. The analysis result image251includes information indicating the location, area, shape, etc., of the target parking area225, and based on such an analysis result image251, target information for determining a target stop location of the vehicle1, etc., can be generated.

FIG.6is a flowchart showing an example of a process performed by the parking assistance device34according to the embodiment. When the obtaining part101obtains a captured image201, the captured image201is stored in the storage part102(S101). The resolution reduction part103reduces the resolution of the captured image201obtained by the obtaining part101(stored in the storage part102), thereby generating a low-resolution image202(S102).

The first processing part104inputs the low-resolution image202to the first model111, thereby generating parking area information about parking areas211present in an imaging area (an area around the vehicle1) (S103). When the parking areas211have been detected by the first processing part104, the selecting part105selects, based on the parking area information, a target parking area225that matches a predetermined condition from the one or more parking areas211(S104). The extracting part106extracts an extraction image231A including the target parking area225from the captured image201stored in the storage part102(S105). The resizing part107resizes the extraction image231A to an extraction image231B of a predetermined size (S106). The second processing part108inputs the resized extraction image231B to the second model121, thereby generating target information indicating a target stop location for the target parking area225(S107). The travel control part109controls travel of the vehicle1based on the target information (S108), and if the vehicle1has reached the target stop location (S109: Yes), then the present routine ends. If the vehicle1has not reached the target stop location (S109: No), then an extraction image231A including the target parking area225is extracted and resized again, using a detected target location, a vehicle's moving distance, etc., obtained at the last point in time, and step S107(regeneration of target information) is performed.

A program that causes the parking assistance device34to perform various types of processes such as those described above may be stored as a file in an installable format or an executable format in a computer-readable storage medium such as a CD-ROM, a CD-R, a memory card, a digital versatile disk (DVD), or a flexible disk (FD), and provided as a computer program product. In addition, the program may be stored on a computer connected to a network such as the Internet, and provided by downloading the program via the network. In addition, the program may be provided or distributed via a network such as the Internet.

According to the above-described embodiment, upon detecting (searching for) one or more parking areas211present around the vehicle1, a low-resolution image202is used, and upon detecting (capturing) a target parking area225selected from the one or more parking areas211, high-resolution extraction images231A and231B are used. By this, while an increase in computational load is suppressed, the accuracy of recognition of parking areas is improved, enabling an improvement in the quality of parking assistance control.

Although the embodiment of the present disclosure has been described above, the above-described embodiment is presented as an example and is not intended to limit the scope of the claimed invention. The new embodiment can be implemented in various other modes, and various omissions, substitutions, or changes can be made without departing from the spirit of the disclosure. The embodiment and variants thereof are included in the scope and spirit of the disclosure and included in the various inventive aspects described in the claims and in the range of equivalency thereof.

REFERENCE SIGNS LIST