DISPLAY PROCESSING APPARATUS, MOVABLE APPARATUS, AND DISPLAY PROCESSING METHOD

A display processing apparatus includes a memory storing instructions, and a processor that executes the instructions to display a guidance image configured to guide a route to a destination area for a user, in a display area superimposed on an external world so that the guidance image is superimposed on the external world, acquire three-dimensional information on an object existing between the user and the destination area after the user specifies the destination area in a captured image of a location including the destination area, and generate, using the three-dimensional information, the guidance image including a shielded area image that displays the route and the destination area that are respectively shielded from the user by the object.

BACKGROUND

Technical Field

One of the aspects of the embodiments relates to a navigation technology that guides a route to a destination.

Description of Related Art

Some navigation apparatuses for guiding a route to a destination for drivers of vehicles such as automobiles and motorcycles use a head-up display that displays a guidance image such as a route guidance arrow in a display area that can see-through-display a route such as a road in the external world of the vehicle. Since the guidance image is superimposed on the route in the external world, the driver can check the route while viewing the external world.

In a case where an arrow indicating a turn, such as a left turn ahead of the road sign in the external world viewable through the display area is superimposed on the road sign, the apparatus disclosed in Japanese Patent Laid-Open No. 2018-173399 does not display an arrow portion superimposed on the sign and facilitates understanding that the turning position is located ahead of the sign.

However, a driver may not be able to view a post-turn route or destination due to buildings or other obstructions. In this case, the driver has difficulty in checking whether or not he is to actually turn according to the arrow displayed in the display area, and consequently may not be able to smoothly turn or may go past the turning position.

SUMMARY

A display processing apparatus according to one aspect of the embodiment includes a memory storing instructions, and a processor that executes the instructions to display a guidance image configured to guide a route to a destination area for a user, in a display area superimposed on an external world so that the guidance image is superimposed on the external world, acquire three-dimensional (3D) information on an object existing between the user and the destination area after the user specifies the destination area in a captured image of a location including the destination area, and generate, using the three-dimensional information, the guidance image including a shielded area image that displays the route and the destination area that are respectively shielded from the user by the object. A movable apparatus, a wearable apparatus, or a terminal device having the above display processing apparatus also constitutes another aspect of the embodiment. A display processing method corresponding to the above display processing apparatus also constitutes another aspect of the embodiment.

DESCRIPTION OF THE EMBODIMENTS

In the following, the term “unit” may refer to a software context, a hardware context, or a combination of software and hardware contexts. In the software context, the term “unit” refers to a functionality, an application, a software module, a function, a routine, a set of instructions, or a program that can be executed by a programmable processor such as a microprocessor, a central processing unit (CPU), or a specially designed programmable device or controller. A memory contains instructions or programs that, when executed by the CPU, cause the CPU to perform operations corresponding to units or functions. In the hardware context, the term “unit” refers to a hardware element, a circuit, an assembly, a physical structure, a system, a module, or a subsystem. Depending on the specific embodiment, the term “unit” may include mechanical, optical, or electrical components, or any combination of them. The term “unit” may include active (e.g., transistors) or passive (e.g., capacitor) components. The term “unit” may include semiconductor devices having a substrate and other layers of materials having various concentrations of conductivity. It may include a CPU or a programmable processor that can execute a program stored in a memory to perform specified functions. The term “unit” may include logic elements (e.g., AND, OR) implemented by transistor circuits or any other switching circuits. In the combination of software and hardware contexts, the term “unit” or “circuit” refers to any combination of the software and hardware contexts as described above. In addition, the term “element,” “assembly,” “component,” or “device” may also refer to “circuit” with or without integration with packaging materials.

Referring now to the accompanying drawings, a description will be given of embodiments.

First Embodiment

FIG.1illustrates the configuration of a navigation system10including a display processing apparatus30according to a first embodiment. The navigation system10includes the display processing apparatus30, a GPS sensor33, a display device34, and an on-board (or in-vehicle) camera35each mounted on an automobile (vehicle)20as a movable apparatus.

The GPS sensor33acquires self-location information on an automobile20(that is, a driver as a user) and inputs it into the display processing apparatus30.

The display device34is a head-up display that projects and displays a guidance image, which will be described below, on a display area set so that the display area is superimposed on the external world on the windshield of the automobile20through which the external world of the vehicle can be viewed. The driver can view the displayed guidance image while viewing the outside of the vehicle (external world) through the windshield. Thereby, the driver can drive the automobile20while viewing the guidance image superimposed on the route, such as a road, in the external world and provided with route guidance. The display device34may use one that projects and displays a guidance image onto a display area set on a transparent member placed between the windshield and the driver so that the display area is superimposed on the external world.

The on-board camera35moves together with the automobile20(that is, the driver) and images the external world. The imaging angle of view of the on-board camera35is set in accordance with the driver's field of view (FOV) relative to the external world. The external world image obtained by the on-board camera35is used to set the position, size, etc. of the display area on the windshield of the guidance image by the display device34.

The navigation system10is used by a driver who downloads captured images of various locations viewable on websites on the Internet into a terminal device50such as a smartphone or a tablet computer. The captured images include images captured by an external camera (such as a surveillance camera, and cameras installed on an aircraft and satellite). The driver can specify as a region of interest (ROI) a destination area (such as a parking lot) toward which the automobile20is heading from among locations (such as the entire facility represented by one address) included in the captured image downloaded by the terminal device50. Information such as the position (coordinates) and shape of the designated destination area is input from the terminal device50to the display processing apparatus30through communication using Bluetooth (registered trademark) or the like.

The display processing apparatus30can acquire three-dimensional map data of the above various locations stored in the external server60via communication. The three-dimensional map data can use, for example, data published as a three-dimensional city model by Project PLATEAU (trademark) in Japan.

The display processing apparatus30includes a computer (such as an Electronic Control Unit (ECU)) including at least one processor, memory, etc., and has a three-dimensional information acquiring unit31as an information acquiring unit and an image generator32as an image processing unit. The three-dimensional information acquiring unit31specifies a building, a signboard, and another object existing between the automobile20and the destination area on the three-dimensional map data based on the user's position information obtained from the GPS sensor33and the position information on the destination area. The three-dimensional information on the specified object is acquired from the three-dimensional map data.

The image generator32generates a guidance image. The guidance image includes a visible area image that is displayed and superimposed on the route viewable by the driver in the external world, and a shielded area image that is displayed and superimposed on the object so as to indicate a route or destination area that is shielded by an object (shield or obstruction) and is not visible to the driver. The image generator32generates a shielded area image based on the three-dimensional information on the shield acquired by the three-dimensional information acquiring unit31.

The image generator32detects the route and shields in the external world image acquired by the on-board camera35by template matching, AI processing using a machine learning model, etc. The display position and display size of the visible area image and the shielded area image on the display area (windshield) by the display device34are set in accordance with these positions and sizes.

FIG.2conceptually illustrates how the navigation system10according to this embodiment is used.FIG.2illustrates a relationship between a surface imaged by the on-board camera35, that is, a surface viewed by the driver (referred to as an FOV surface hereinafter)70, a destination area80, and an object85. The FOV surface70is located between the destination area80(having a three-dimensional shape in this example) and the object85closer to the automobile20.

As illustrated inFIG.2, in a case where the object85exists as a shield between the automobile (user's vehicle)20and the destination area80, the driver cannot directly see the destination area80and the route around it. On the other hand, a location including the destination area80and its surroundings is imaged by an external camera90, and the driver can confirm and specify the destination area80in the captured image. Therefore, once the position of the user's vehicle (user's position information), information on the position and shape of the destination area80specified in the captured image, and three-dimensional information (information on the position and three-dimensional shape) of the object85are known, it can be determined whether or not the destination area80is blocked (invisible) by the object85as the shield for the user.

In a case where the object85is determined to be the shield, a shielded area image is displayed as a guidance image illustrating a map75of the destination area80and a map of the nearby route so that the shielded area image is superimposed on the map71of the object85in the display area corresponding to the FOV surface70. Thereby, the driver can visually recognize the destination area80and the nearby route.

FIGS.3to6illustrate examples of guidance images displayed by the display device34.FIGS.3to6illustrate the vicinity of the destination area80where a distance to the destination area80is less than a predetermined distance, which the driver views the route (road)101in the external world and the surrounding facilities102and103through the windshield (the external world image from the board camera35).

As illustrated by the broken line inFIG.3, the destination area80is a parking lot near the entrance of the site of the facility102, and most of it is hidden by a wall104of the facility102. Conventionally, as illustrated inFIG.3, an arrow image (visible area image)120as a guidance image is superimposed and displayed only on the route101that is visible to the driver in the display area. However, it is difficult for a driver who can hardly view the destination area80to check whether or not he is actually allowed to turn according to the arrow image120and enter from the entrance of the facility102. As a result, he is likely to go past the entrance, or the destination area80near the wall104even if the user enters through the entrance.

On the other hand,FIG.4according to this embodiment displays and superimposes an arrow image (shielded area image)131for guiding the route to the destination area80on the wall104, and also displays and superimposes a frame image (shielded area image)132indicating the destination area80on the wall104. Thereby, the driver can enter through the entrance of facility102and be confident that he should park in the parking lot near the wall104.

By starting to display the arrow image131and the frame image132before the user turns (turns left) near the destination area80, the driver can be smoothly guided to the destination area80without hesitation as to whether or not he is allowed to turn. In particular, the arrow image131and the frame image132may be displayed before the user turns multiple times near the destination area80.

The colors of the wall104and surroundings may be detected from the external world image from the on-board camera35, and the arrow image131and the frame image132may be displayed in colors different from the colors of the wall104and surroundings. For example, the arrow image131and the frame image132may be displayed in complementary colors to the average color of the surroundings of the wall104and others.

FIG.4illustrates the arrow image131so as to connect it with the arrow image120and to provide the route guidance without making the driver feel uncomfortable. The arrow image131may be displayed so that it is not connected with the arrow image120as long as the driver does not feel uncomfortable.

FIG.4highlights the part of the destination area80that is visible to the driver by a frame image (visible area image)133connected with the frame image132. Thereby, the driver can recognize the entire destination area80including the part visible to the driver.

FIGS.5and6illustrate a modification toFIG.4. InFIG.4, the display form (color, line type, etc.) of each of the arrow image131and the frame image132is the same as that of the arrow image120. On the other hand, the display form of each of the arrow image131and the frame image132may be different from that of the arrow image120.

InFIG.5, the color of each of the arrow image131and the frame image132is different from that of the arrow image120. The arrow image131and the frame image132may be displayed with double lines without filling. InFIG.6, the arrow image131and the frame image132are indicated by dashed lines to differ from the arrow image120indicated by solid lines. Thereby, the driver can clearly recognize that the route and the destination area80indicated by the arrow image131and the frame image132are not actually visible.

InFIGS.5and6, the frame image133, which is the visible area image, is displayed in the same manner as the arrow image120, which is also the visible area image, but the display form of the frame image133may be the same as that of a certain frame image132.

FIGS.7C,7D, and7Eillustrate display examples of the destination area80(parking lot P) by the display device34.FIGS.7C,7D, and7Eomit an arrow image for route guidance to the destination area80.

FIG.7Aillustrates a captured image obtained by the external camera (surveillance camera)90set on the opposite side of the road140from the destination area80, as illustrated inFIG.7B. The destination area80is located between two buildings141and142built along the road140and is separated from the road140by one parking lot. Thus, most of the destination area80is difficult to see from the user's vehicle approaching from the front inFIG.7A, as it is blocked by the building141(and the automobile148already parked in the parking lot on the roadside).

FIG.7Cillustrates a display example that superimposes on the building141abroken-line frame image (shielded area image)143aillustrating the destination area80that is invisible due to the building141, and a broken-line arrow image (shielded area image)143bindicating the route to the destination area80that is also invisible due to the building141.FIG.7Cillustrates the visible part of the destination area80as a broken-line frame image (visible area image)144connected with the frame image143aas the invisible part, and the visible part of the route as a solid-line arrow image (visible area image)149aconnected to the arrow image143b.

Similarly toFIG.7C,FIG.7Dillustrates an example that displays and superimposes the broken-line frame image143aand the arrow image143bon the building141and a solid-line frame image144. Even this example displays the arrow image149. InFIG.7D, a dotted-line stereoscopic image (shielded area image)145illustrating a portion of the three-dimensional shape (rectangular parallelepiped shape herein) of the destination area80as the parking lot that is invisible by the building141is displayed and superimposed on the building141, and a visible portion is displayed as a solid-line stereoscopic image (visible area image)146. Due to such a display, the driver can stereoscopically recognize the destination area80.

Similarly toFIG.7D,FIG.7Eillustrates an example that displays frame images143ato146, and a captured image147inFIG.7Aacquired by the external camera90as Picture-in-Picture (PIP) display. By displaying the captured image147, the driver can reach the destination area80while clearly recognizing the actual shape of the destination area80and the positional relationship among the buildings141and142and the automobile148.

A flowchart inFIG.8illustrates the processing (display processing method) executed by the display processing apparatus30according to a program.

A program (application) illustrated inFIG.11is installed in the terminal device50. This program includes step S31of acquiring a captured image (seeFIG.7A) on a website40through the Internet, step S32of causing the user to specify a destination area in the captured image by finger touch or the like, and step S33of transmitting information on the specified destination area to the display processing apparatus30.

In step S1ofFIG.8, the display processing apparatus30communicates with the terminal device50and requests the driver to specify a destination area. Then, in a case where the destination area is specified by the driver in step S2, information on the destination area is acquired from the terminal device50.

Next, in step S3, the image generator32identifies a field of view (FOV) that is estimated to be viewed by the driver based on the imaging angle of view of the on-board camera35and the external world image and sets the position and size of the display area where the display device34can display the guidance image according to the field of view.

Next, in step S4, the image generator32determines whether or not the destination area enters the field of view based on the user's vehicle position information from the GPS sensor33and the field of view information specified in step S3. Here, the “destination area enters the field of view” means that the destination area is included in the field of view of the user's vehicle on the two-dimensional map, regardless of whether the destination area is actually visible to the driver. In a case where the destination area enters the field of view, the flow proceeds to step S5, and in a case where the destination area does not enter the field of view, the flow proceeds to step S8.

Next, in step S5, the three-dimensional information acquiring unit31acquires from the above three-dimensional map data three-dimensional information on an object existing between the user's vehicle position and the destination area position that can be specified from the user's vehicle position information and the destination area position information from the GPS sensor33, respectively.

Next, in step S6, the image generator32determines whether an object existing between the user's vehicle and the destination area is a shield that makes the destination area invisible to the driver, based on the user's vehicle position information, the position and shape information on the destination area, and the three-dimensional information acquired in step S5. More specifically, for example, whether or not the object is a shield is determined by checking the space ID of a voxel of the object that can be acquired from the three-dimensional information. In a case where the object is the shield, the flow proceeds to step S7; in a case where the object is not the shield, the flow proceeds to step S8.

In step S7, the image generator32generates a guidance image that includes a visible area image to be displayed on the route that is illustrated in the external world image from the on-board camera35(visible to the driver) and a shielded area image that displays a route and a destination area that is not illustrated in the external world image (invisible to the driver). Then, the flow proceeds to step S9. At this time, the destination area may be displayed in the visible area image, or only one of the route and the destination area may be displayed in the shielded area image. As described with reference toFIGS.5and6, the visible area image and the shielded area image may have different display forms.

On the other hand, in step S8, the image generator32generates a guidance image of only the visible area image to be displayed on the route illustrated in the external world image from the on-board camera35. Then, the flow proceeds to step S9. Again, the destination area may be displayed using the visible area image.

In step S9, the image generator32displays the guidance image generated in step S7or S8in the display area through the display device34.

Next, in step S10, the image generator32determines whether or not to end the display of the guidance image. More specifically, the image generator32determines whether or not the user's vehicle has approached the destination area based on the user's vehicle position information from the GPS sensor33. In a case where the display of the guidance image is to end because the user's vehicle has approached the destination area, this flow ends; otherwise, the flow returns to step S3.

This embodiment can display the destination area and route that are not visible to the driver due to a shield and thus can provide route guidance that is easier for the driver to understand.

In this embodiment, three-dimensional information on an object is acquired from three-dimensional map data and it is determined whether the object is a shield. On the other hand, in a case where the user's vehicle has a function of detecting a three-dimensional object using Light Detection and Ranging (LIDAR), etc., a three-dimensional structure of an object that exists between the user's vehicle and the destination area may be acquired as three-dimensional information using the above function. Then, by using the projection transformation of this three-dimensional structure into the display area, whether the object shields the destination area may be determined.

In an image captured by the external camera, a final destination area may be specified as at least one relay point between a departure point and the destination area (such as a parking lot for taking a break or checking the destination near an expressway exit). In this case, a shielded area image can be displayed near the relay point and near the final destination area.

A shielded area image may be displayed according to the constraint condition on the entry into the destination area. For example, in a case where the destination area is a parking lot, the shielded area image may be displayed so as to provide route guidance (parking guidance) based on forward and backward parking restrictions, restrictions on the vehicle size that can be parked, and restrictions from the sizes and situations of the vehicles parked in the adjacent parking lots.

Second Embodiment

A description will now be given of a second embodiment. A flowchart inFIG.9illustrates the processing that the display processing apparatus30executes according to a program in the second embodiment. In this embodiment, steps S1-S4are the same as steps S1-S4in the first embodiment (FIG.8). A description will be given of a case where a plurality of objects exist between the user's vehicle and the destination area.

In a case where the display processing apparatus30determines that the destination area enters the field of view in step S4, the flow proceeds to step S15. In step S15, the three-dimensional information acquiring unit31acquires three-dimensional information on each of a plurality of (n) objects that exist between the user's vehicle position and the destination area position, which can be specified from the user's vehicle position information from the GPS sensor33and the position information on the destination area.

In the next step S16, the image generator32acquires information on the shape of the portion of the destination area that is shielded (hidden) by the k-th (k=1 to n) object from the user's vehicle side. Information on the shape of the shielded portion can be acquired using the user's vehicle position information, the position and shape information on the destination area, and the three-dimensional information acquired in step S5.

Next, in step S17, the image generator32determines whether or not there is a part hidden by the k-th object in the destination area in step S16(that is, the object is a shield), and in a case where there is that part, the flow proceeds to step S18; otherwise, the flow proceeds to step S19.

In step S18, the image generator32generates a guidance image that includes a visible area image to be displayed on the route in the external world image from the on-board camera35and a shielded area image to be displayed and superimposed on the k-th object (shield). Then, the flow proceeds to step S9.

On the other hand, in step S19, the image generator32determines whether or not k is n. In a case where k is n, the flow proceeds to step S8; in a case where k is not n, the flow proceeds to step S20to increment k by 1. Then, the flow returns to step S16.

In step S8, the image generator32generates a guidance image of only the visible area image to be displayed on the route illustrated in the external world image from the on-board camera35, similarly to step S8of the first embodiment. Then, the flow proceeds to step S9.

In step S9, the image generator32displays the guidance image generated in step S21or S8on the display area through the display device34, similarly to step S9of the first embodiment. In step S10, similarly to step S10of the first embodiment, it is determined whether or not to end the display of the guidance image. If not, the flow returns to step S3, and if it does, the flow ends.

Thus, in a case where there are a plurality of objects between the user's vehicle and the destination area, this embodiment determines whether the object is a shield in order from the object closest to the user's vehicle. Then, this embodiment displays and superimposes the shielded area image on the object that is determined to be a shield first among the plurality of objects, and displays only the visible area image if none of the objects is a shield.

Even if the driver cannot view the destination area or route due to any one of shields among the plurality of objects between the user's vehicle and the destination area, this embodiment can display them. Thereby, this embodiment can provide route guidance that can be easily understood by the driver.

In each of the above embodiments, the display processing apparatus is installed in an automobile, but it may also be installed in various movable apparatuses other than automobiles (such as a ship and aircraft).

The display processing apparatus may be installed in a wearable apparatus that a user wears in front of his eyes, such as a glasses-type device using the augmented reality (AR) technology. As illustrated inFIG.10A, a guidance image is displayed in a display area set on a lens201placed in front of the user's eyes in a wearable apparatus200so that the display area is superimposed on the external world.

Each of the above embodiments has discussed a guidance image displayed and superimposed on the external world in a display area that is superimposed on the actual external world. However, the superimposition on the external world is not limited to superimposition on the actual external world but also includes superimposition on an external world image of the vehicle generated by imaging. That is, as illustrated inFIG.10B, a display area302may be set on a screen301of a terminal device300such as a smartphone or tablet so that the display area302is superimposed on the external world image, and a guidance image may be displayed in the display area302so that the guidance image is superimposed on the external world image.

OTHER EMBODIMENTS

Each embodiment can provide route guidance that can be easily understood by a user even if the user cannot view the route or destination area due to a shield.

This application claims the benefit of Japanese Patent Application No. 2022-185311, filed on Nov. 18, 2022, which is hereby incorporated by reference herein in its entirety.