Patent Description:
These days, camera monitor systems (CMSs) can be mounted, and the CMSs are started to be actually introduced to automobiles. In the CMSs, an image captured by a camera mounted in an automobile is displayed on a monitor at a driver seat, and the situation around the automobile, for example, the situation of the rear (rear direction) of the automobile can be checked.

For example, <CIT> discloses a camera monitor system (CMS) in a host vehicle. In the CMS, an image of the rear of a vehicle captured by a rear camera installed in the rear of the vehicle, images of the right rear and left rear captured by side cameras installed at rearview mirror positions on the right and left sides of the vehicle, or a composite image thereof are displayed on a display unit.

<CIT> discloses a system providing images of blind spots, detecting a driver's line of sight and displaying the image corresponding to the blind spot in that direction on respective display devices.

<CIT> discloses a display control device processing the captured image of a plurality of cameras in a car. The field of view of the displayed image is adapted to the driver's head movement.

A camera monitor system (CMS) enables a surrounding state, which has been checked with a back view mirror such as a so-called traditional rearview mirror and side mirror of a car, to be checked by using a camera captured image.

For example, three images of a rear captured image, a left rear captured image, and a right rear captured image are individually displayed on a display unit (monitor) capable of being observed by a driver. The rear captured image is captured by a rear camera that images the rear of a vehicle. The left rear captured image and a right rear captured image are respectively captured by a left side camera and a right side camera installed in the vicinity of left and right side mirror positions of the vehicle. Alternatively, a single composite image generated by appropriately synthesizing these three images is displayed on the monitor.

In a case where one continuous composite image is generated and displayed from three images captured by the rear camera and left and right side cameras in such an image display system, however, a subject captured in any of the three images sometimes disappear, and is not displayed. This is because the images captured by the three cameras of the rear camera, the L side camera, and the R side camera have different viewpoints, and this is caused by, for example, processing of conversion to an image from one viewpoint, for example, a viewpoint of the rear camera, at the time when a composite image is generated.

If a composite image is generated with respect to the viewpoint of the rear camera at the time when a composite image is generated, a subject captured by a left side camera and a right side camera, such as, for example, a motorcycle behind a large vehicle in the immediate rear and a vehicle approaching at a high speed from a distance, disappears from the composite image. This is because, for example, the motorcycle is in a region behind the large vehicle, that is, an occlusion region from the viewpoint of the rear camera.

The present disclosure has been made in view of, for example, the above-described problem, and an object of thereof is to provide an image processing apparatus, a moving apparatus, a method, and a program in which a driver can almost continuously (seamlessly) check images of the rear side observed from various viewpoints similarly to the case of checking a traditional rearview mirror and side mirror by sequentially switching and displaying images captured by a plurality of cameras having different imaging viewpoints in accordance with the behavior of the driver, for example, movements of the head.

A first aspect of the present disclosure is in an image processing apparatus including a data processing unit that executes image display control of:.

Moreover, a second aspect of the present disclosure is in a moving apparatus including:.

Moreover, a third aspect of the present disclosure is in an image processing method executed in an image processing apparatus,
in which a data processing unit executes image display control of:.

Moreover, a fourth aspect of the present disclosure is in a display image control method executed in a moving apparatus, including:.

Moreover, a fifth aspect of the present disclosure is in a program for executing image processing in an image processing apparatus, causing a data processing unit to execute image display control of:.

Note that the program of the present disclosure can be provided by a storage medium or a communication medium capable of providing various program codes in a computer readable format to an information processing apparatus and a computer system capable of executing various program codes. Processing in accordance with a program is performed on an information processing apparatus and a computer system by providing such a program in a computer-readable format.

Other objects, features, and advantages of the present disclosure will be apparent from more detailed description based on the later-described embodiments of the present disclosure and the accompanying drawings. Note that a system in the specification has configuration of a logical set of a plurality of apparatuses. The system is not limited to a system in which apparatuses having each configuration are placed in the same housing.

According to the configuration of one embodiment of the present disclosure, configuration, in which images output to a display unit are switched and displayed in accordance with the behavior of a driver, such as movements of the head of the driver, is achieved.

Specifically, for example, driver information indicating the behavior of a driver of a moving apparatus and images captured by a plurality of cameras that images the situation around the moving apparatus from different viewpoints are input. Images output to the display unit are switched in accordance with driver information. The plurality of cameras is, for example, a plurality of rear cameras installed in the rear of the moving apparatus. For example, a direction of the face or line-of-sight of the driver is detected. An image in a direction corresponding to the detected direction of the face or line-of-sight of the driver is selected as an output image, and displayed on the display unit. Alternatively, an image in a direction indicated by a gesture of the driver is selected, and displayed on the display unit.

The configuration achieves configuration in which images output to the display unit are switched and displayed in accordance with the behavior of the driver, such as movements of the head of the driver.

Note that the effects described in the specification are merely illustration and not limitation, and additional effects may be exhibited.

Details of an image processing apparatus, a moving apparatus, a method, and a program of the present disclosure will be described below with reference to the drawings. Note that the description will be given in accordance with the following items.

First, a camera monitor system (CMS) mounted in an automobile will be outlined with reference to <FIG> and subsequent figures.

<FIG> outlines a CMS mounted in an automobile <NUM>. The CMS displays an image captured by a camera installed on an automobile on a display unit that a driver can observe.

The automobile <NUM> in <FIG> includes a rear camera <NUM>, a left side camera <NUM>, and a right side camera <NUM>. The rear camera <NUM> is installed in the rear of the automobile <NUM>. The left side camera <NUM> is installed on the left side of the automobile <NUM>. The right side camera <NUM> is installed on the right side of the automobile <NUM>. Each of these cameras captures an image behind the automobile <NUM>, observed from the installation position of each camera.

Images captured by these cameras or a composite image generated on the basis of these captured images are displayed on a display unit <NUM> inside the automobile <NUM>. A driver of the automobile <NUM> can check the state behind the automobile <NUM> by looking at a display image on the display unit <NUM>.

A specific image example displayed on the display unit <NUM> will be described with reference to <FIG> and subsequent figures. <FIG> illustrates the state behind the automobile <NUM>. In <FIG>, the automobile <NUM> is traveling in a direction of an arrow (forward) in the figure.

In this state, the three cameras of the rear camera <NUM>, the left side camera <NUM>, and the right side camera <NUM> capture an image with each camera position as a viewpoint position.

<FIG> illustrates examples of images, which the driver of the automobile <NUM> can check, displayed on the display unit <NUM>. <FIG> illustrates an example in which images captured by three cameras of the rear camera <NUM>, the left side camera <NUM>, and the right side camera <NUM> are individually displayed on the display unit <NUM>. On the display unit <NUM>, a rear camera image <NUM> captured by the rear camera <NUM> is displayed in the center of the display unit <NUM>, a left side camera image <NUM> captured by the left side camera <NUM> is displayed on the left side of the display unit <NUM>, and a right side camera image <NUM> captured by the right side camera <NUM> is displayed on the right side of the display unit <NUM>.

The driver can check the surrounding situation in a manner similar to the case of looking at a rearview mirror and right and left side mirrors by looking at these three images.

In a case where three images are displayed side by side in this way, however, the driver mainly pays attention only to the central image corresponding to the rearview mirror. If the driver is not careful of the right and left images, the driver tends to overlook information mainly reflected in the right and left images.

In the example in <FIG>, a bicycle reflected in the left side camera image <NUM> is hidden behind a large vehicle in the central rear camera image <NUM>, and almost invisible. In such a case, if the driver pays attention only to the rear camera image <NUM> displayed in the center of the display unit <NUM>, the driver is at risk of being unaware of the bicycle.

Although <FIG> illustrates an example in which images captured by the cameras are individually displayed on the display unit <NUM>, a configuration, in which a composite image is generated by combining these three images to one image and displayed on the display unit <NUM>, has been proposed and used. <FIG> illustrates an example of display of a composite image.

<FIG> illustrates one example of a composite image <NUM> displayed on the display unit <NUM>. The composite image <NUM> is generated with the viewpoint of the rear camera <NUM> as a reference. The rear camera <NUM>, the left side camera <NUM>, and the right side camera <NUM> capture images from different viewpoints, so that, for example, processing of converting a viewpoint position of each image, processing of converting the size of an object (subject), and processing of adjusting a position are necessary for generating one composite image. Specifically, image conversion processing such as, for example, affine transformation is performed.

Multiple images are joined by such image conversion processing to generate one composite image. <FIG> illustrates an example of a composite image generated by converting images captured by the left side camera <NUM> and the right side camera <NUM> into a captured image from the viewpoint of the rear camera <NUM>. In a case where such a composite image is generated, however, a subject captured by the left side camera or the right side camera may disappear.

Although the bicycle serving as a subject can be clearly confirmed in the above-described left side camera image <NUM> in <FIG>, the bicycle is hidden behind the vehicle, and almost invisible in the composite image <NUM> in <FIG>. This is because the bicycle is in a region behind the large vehicle in front of the vehicle, that is, an occlusion region from the viewpoint of the rear camera.

The driver cannot confirm the bicycle approaching from behind even if looking at the composite image <NUM>. If the driver drives in a manner of, for example, suddenly turning left, an accident such as contact against the bicycle approaching from behind may occur.

Next, a configuration example of the image processing apparatus and the moving apparatus of the present disclosure will be described with reference to <FIG> and subsequent figures. The present disclosure solves the problem described with reference to <FIG>, for example. Specifically, for example, images captured by a plurality of cameras having different imaging viewpoints are sequentially switched and displayed in accordance with the behavior of a driver, such as, for example, movements of a head, face, and line-of-sight.

<FIG> illustrates one configuration example of the automobile <NUM> (moving apparatus) of the present disclosure. The automobile <NUM> in <FIG> includes the left side camera <NUM> on the left side of the automobile <NUM> and the right side camera <NUM> on the right side of the automobile <NUM>, in a manner similar to that described above with reference to <FIG>. Three rear cameras, that is, a rear central (C) camera <NUM>, a rear left (L) camera <NUM>, and a rear right (R) camera <NUM> are installed in the rear of the automobile <NUM>. The rear C camera <NUM> is installed in the center of the rear of the automobile <NUM>. The rear L camera <NUM> is installed on the left side of the rear of the automobile <NUM>. The rear R camera <NUM> is installed on the right side of the rear of the automobile <NUM>.

Note that, although three cameras are installed in the rear of the automobile <NUM> in the example, a plurality of various cameras other than three can be used. An example of using three or more multiple cameras will be described later.

<FIG> illustrates an example of imaging regions of three cameras installed in the rear of the automobile <NUM>. <FIG> illustrates the imaging regions of three cameras of the rear central (C) camera <NUM>, the rear left (L) camera <NUM>, and the rear right (R) camera <NUM>. The rear C camera <NUM> is installed in the center of the rear of the automobile <NUM>. The rear L camera <NUM> is installed on the left side of the rear of the automobile <NUM>. The rear R camera <NUM> is installed on the right side of the rear of the automobile <NUM>.

As illustrated in <FIG>, the three cameras have setting in which the rear of the automobile is imaged from different viewpoints, and parts of the imaging region overlap with each other. In the configuration of the present disclosure, images captured by a plurality of cameras having such different imaging viewpoints are sequentially switched and displayed in accordance with the behavior of the driver, for example, movements of the head. The movements of the head are detected for detecting a movement comparable to action and motion of the driver at the time of looking at a traditional rearview mirror and side mirror and using the sign as a trigger for switching display.

<FIG> illustrates an imaging environment similar to that in <FIG> described above. The automobile <NUM> is traveling in the direction of an arrow (forward) in the figure.

A large vehicle is traveling behind the automobile <NUM>, and a bicycle is traveling in the left rear of the large vehicle.

<FIG> illustrates examples of images, which the driver of the automobile <NUM> can check, displayed on the display unit <NUM>. The left side camera image <NUM> captured by the left side camera <NUM> is displayed on the left side of the display unit <NUM>. The right side camera image <NUM> captured by the right side camera <NUM> is displayed on the right side of the display unit <NUM>.

An image display region in the central part of the display unit <NUM> is set as a rear-camera-image switching display region <NUM>. Images captured by three cameras, that is, the rear central (C) camera <NUM>, the rear left (L) camera <NUM>, and the rear right (R) camera <NUM>, installed in the rear of the automobile <NUM> are sequentially switched and displayed on the rear-camera-image switching display region <NUM> in accordance with the behavior of the driver, for example, movements of the head. The rear C camera <NUM> is installed in the center of the rear of the automobile <NUM>. The rear L camera <NUM> is installed on the left side of the rear of the automobile <NUM>. The rear R camera <NUM> is installed on the right side of the rear of the automobile <NUM>.

An example of image display on the rear-camera-image switching display region <NUM>, that is, an image switching display example will be described with reference to <FIG>.

<FIG> illustrates correspondence data between movements of the head of a driver <NUM> (direction of the face or line-of-sight) and the example of image display on the rear-camera-image switching display region <NUM>. Three examples of image display are given below.

<FIG> illustrates (R) display example of a rear R camera captured image in a case where the face or line-of-sight of the driver <NUM> is in the right direction. In this way, in a case where the driver <NUM> of the automobile <NUM> looks at the right, an image captured by the rear right (R) camera <NUM>, which is installed on the right side of the rear of the automobile <NUM>, is displayed on the rear-camera-image switching display region <NUM> of the display unit <NUM>.

Furthermore, <FIG> illustrates (C) display example of a rear C camera captured image in a case where the face or line-of-sight of the driver <NUM> is substantially in the front direction. In this way, in a case where the driver <NUM> of the automobile <NUM> looks at the front, an image captured by the rear central (C) camera <NUM>, which is installed in the center of the rear of the automobile <NUM>, is displayed on the rear-camera-image switching display region <NUM> of the display unit <NUM>.

Moreover, <FIG> illustrates (L) display example of a rear L camera captured image in a case where the face or line-of-sight of the driver <NUM> is in the left direction. In this way, in a case where the driver <NUM> of the automobile <NUM> looks at the left, an image captured by the rear left (L) camera <NUM>, which is installed on the left side of the rear of the automobile <NUM>, is displayed on the rear-camera-image switching display region <NUM> of the display unit <NUM>.

Note that, in a case where the driver <NUM> exhibits behavior of, for example, first looking at the right, then looking at the front, and finally looking at the left, display images on the rear-camera-image switching display region <NUM> of the display unit <NUM> are sequentially switched as follows. First, an image captured by the rear right (R) camera <NUM> is displayed, switched to an image captured by the rear central (C) camera <NUM>, and then switched to display an image captured by the rear left (L) camera <NUM>.

A driver information detection unit that detects the behavior of the driver <NUM> is installed in the automobile <NUM>. An image processing apparatus of the automobile <NUM> performs control to switch display images on the display unit <NUM> on the basis of detection information from the driver information detection unit.

An image of a bicycle can be clearly confirmed in (L) display example of the rear L camera captured image described on the left side among three images in <FIG>. This is because the rear left (L) camera <NUM> captures an image from the left side of the rear of the automobile <NUM>, and can capture an image of the bicycle that is not hidden behind the large vehicle in front of the bicycle.

The driver <NUM> can clearly confirm that the bicycle is behind the large vehicle by looking at the image.

Note that, although each image in <FIG> may be displayed on a central region between the left side camera image <NUM> and the right side camera image <NUM> together with these two images as in the display example of the display unit <NUM> described above with reference to <FIG>, only an image in <FIG> may be independently displayed on the display unit <NUM> in accordance with the direction of the face or line-of-sight of the driver <NUM> without displaying the left side camera image <NUM> and the right side camera image <NUM>.

Note that <FIG> illustrates setting of switching display images on the display unit <NUM>, that is, images on the rear-camera-image switching display region <NUM>, which is a central region of the display unit <NUM> in <FIG>, in accordance with the direction of the face or line-of-sight of the driver <NUM>, the position of the rear-camera-image switching display region <NUM> may be changed in accordance with the direction of the face or line-of-sight of the driver <NUM>, for example. That is, the position of the rear-camera-image switching display region <NUM> is changed to a position that matches the direction of the face or line-of-sight of the driver <NUM>.

The image position change display example will be described with reference to <FIG>. Similarly to <FIG>, <FIG> illustrates correspondence data between movements of the head of a driver <NUM> (direction of the face or line-of-sight) and the example of image display on the rear-camera-image switching display region <NUM>. Three examples of image display are given below.

The display unit <NUM> in <FIG> is set on an upper part of a front panel of an automobile, and has a display region that is long in the right and left direction.

<FIG> illustrates (L) display example of the rear L camera captured image in a case where the face or line-of-sight of the driver <NUM> is in the left direction. In a case where the driver <NUM> looks at the left, an image captured by the rear left (L) camera <NUM>, which is installed on the left side of the rear of the automobile <NUM>, is displayed on a left end part of the display unit.

Furthermore, <FIG> illustrates (C) display example of the rear C camera captured image in a case where the face or line-of-sight of the driver <NUM> is substantially in the front direction. In a case where the driver <NUM> looks at the front, an image captured by the rear central (C) camera <NUM>, which is installed in the center of the rear of the automobile <NUM>, is displayed in the central part of the display unit.

Furthermore, <FIG> illustrates (R) display example of the rear R camera captured image in a case where the face or line-of-sight of the driver <NUM> is in the right direction. In a case where the driver <NUM> looks at the right, an image captured by the rear right (R) camera <NUM>, which is installed on the right side of the rear of the automobile <NUM>, is displayed on a right end part of the display unit.

<FIG> illustrates an example in which the position of the rear-camera-image switching display region <NUM> is changed to a position that matches the direction of the face or line-of-sight of the driver <NUM>. Such display position control allows the driver <NUM> to display an image which the driver <NUM> wants to see in the direction of line-of-sight of his/her own. Consequently, the driver <NUM> can certainly confirm the image in the direction in which the driver <NUM> is paying attention, while keeping the line-of-sight in the direction.

Note that the control of the image display position is also executed on the basis of the detection information from the driver information detection unit, which detects the behavior of the driver <NUM>, installed in the automobile <NUM>.

Moreover, separately from the direction of the face or line-of-sight of the driver <NUM>, display images may be changed in accordance with indication with, for example, a head movement sequence, a hand, or a finger of the driver <NUM>.

An example of the configuration in which display images are changed in accordance with indication with the hand or finger of the driver <NUM> will be described with reference to <FIG>.

<FIG> illustrates (L) display example of the rear L camera captured image in a case where a finger of driver <NUM> indicates the left direction. In a case where the finger of the driver <NUM> indicates the left, an image captured by the rear left (L) camera <NUM>, which is installed on the left side of the rear of the automobile <NUM>, is displayed on the display unit <NUM>.

Furthermore, <FIG> illustrates (C) display example of the rear C camera captured image in a case where the finger of the driver indicates substantially the front. In a case where the finger of the driver <NUM> indicates the front, an image captured by the rear central (C) camera <NUM>, which is installed in the center of the rear of the automobile <NUM>, is displayed on the display unit <NUM>.

Moreover, <FIG> illustrates (R) display example of the rear R camera captured image in a case where the finger of the driver <NUM> indicates the right. In a case where the finger of the driver <NUM> indicates the right, an image captured by the rear right (R) camera <NUM>, which is installed on the right side of the rear of the automobile <NUM>, is displayed on the display unit <NUM>.

<FIG> illustrates an example in which images displayed on the rear-camera-image switching display region <NUM> are changed in accordance with indication (gesture) with a hand or finger of the driver <NUM>. Note that the motion sequence of a head may be detected as a gesture. Such control enables the driver <NUM> to change images without significantly changing the line-of-sight direction.

<FIG> illustrates a configuration example for image switching display control provided in the automobile <NUM>, which is a moving apparatus. As illustrated in <FIG>, the automobile <NUM> includes a driver information detection unit <NUM> that detects the direction of the face or line-of-sight of the driver <NUM> or indication (gesture) with a hand or finger. Specifically, the driver information detection unit <NUM> includes, for example, a camera or a movement sensor. The detection information from the driver information detection unit <NUM> is input to a data processing unit <NUM>.

The data processing unit <NUM> performs processing of switching images displayed on the display unit <NUM>, and changes a display position on the basis of the detection information from the driver information detection unit <NUM>.

Note that a user (driver <NUM>) may perform operation of switching images displayed on the display unit <NUM>. For example, as illustrated in <FIG>, for example, a user operation unit may be displayed on an operation unit set in the vicinity of the display unit <NUM> or a display unit of touch panel type. The user may operate the operation unit to switch display images.

(L) display example of the rear L camera captured image in <FIG> is an example of a display image on the display unit <NUM> in a case where the user moves the operation unit to the left. In this case, an image captured by the rear left (L) camera <NUM>, which is installed on the left side of the rear of the automobile <NUM>, is displayed on the display unit <NUM>.

(C) display example of the rear C camera captured image is an example of a display image on the display unit <NUM> in a case where the user moves the operation unit to the center. In this case, an image captured by the rear central (C) camera <NUM>, which is installed in the center of the rear of the automobile <NUM>, is displayed on the display unit <NUM>.

(R) display example of the rear R camera captured image is an example of a display image on the display unit <NUM> in a case where the user moves the operation unit to the right. In this case, an image captured by the rear right (R) camera <NUM>, which is installed on the right side of the rear of the automobile <NUM>, is displayed on the display unit <NUM>.

Note that setting similar to that described above with reference to <FIG> can be adopted. In the setting, each image in <FIG> is displayed together with the left side camera image <NUM> and the right side camera image <NUM> as the display example of the display unit <NUM> described above with reference to <FIG>. Furthermore, only an image in <FIG> may be independently displayed on the display unit <NUM> in accordance with the direction of the face or line-of-sight and indication of the driver <NUM> without displaying the left side camera image <NUM> and the right side camera image <NUM>.

The image switching display example described with reference to <FIG> is a configuration example in which images captured by three cameras of, that is, the rear left (L) camera <NUM>, the rear central (C) camera <NUM>, and the rear right (R) camera <NUM> installed in the rear of the automobile <NUM> are switched and displayed on the basis of the position of the head/face, line-of-sight direction, or indication of the driver.

Moreover, an image captured by the left side camera <NUM> and an image captured by the right side camera <NUM> may be added to images captured by these three cameras as a target of switching images.

The specific example will be described with reference to <FIG>.

<FIG> illustrates an example of a display image switching sequence of the display unit <NUM> based on, for example, the direction of the face or line-of-sight of the driver <NUM>, indication, or input to an operation unit.

For example, as the direction of the face or line-of-sight of the driver <NUM> changes from right to left, display images on the display unit <NUM> are sequentially switched in the following order.

Moreover, as the direction of the face or line-of-sight of the driver <NUM> changes from left to right, display images on the display unit <NUM> are sequentially switched in the order of (<NUM>) to (<NUM>) above.

The embodiment described with reference to <FIG> is a configuration example in which images captured by three cameras of, that is, the rear left (L) camera <NUM>, the rear central (C) camera <NUM>, and the rear right (R) camera <NUM> installed in the rear of the automobile <NUM> are switched and displayed on the basis of the direction of the face or line-of-sight of a driver, gesture indication with, for example, a head, or input to the operation unit.

Any multiple number other than three of cameras can be installed in the rear of the automobile <NUM>.

<FIG> illustrates an example in which five cameras are installed in the rear of the automobile <NUM>. <FIG> illustrates an example similar to the embodiment described with reference to <FIG>. In the example, the rear left (L) camera <NUM>, the rear central (C) camera <NUM>, and the rear right (R) camera <NUM> are installed. Moreover, a rear central left (CL) camera <NUM> and a rear central right (CR) camera <NUM> are provided. The rear CL camera <NUM> captures an image from a middle viewpoint between the rear L camera <NUM> and the rear C camera <NUM>. The rear CR camera <NUM> captures an image from a middle viewpoint between the rear C camera <NUM> and the rear R camera <NUM>.

<FIG> illustrates an example of imaging regions of these five cameras. <FIG> illustrates the imaging regions of the five cameras of the rear left (L) camera <NUM>, the rear central left (CL) camera <NUM>, the rear central (C) camera <NUM>, the rear central right (CR) camera <NUM>, and the rear right (R) camera <NUM>. These five cameras are installed from the left side to the right side of the rear of the automobile <NUM>.

As illustrated in <FIG>, the five cameras image the rear of the automobile from different viewpoints. Parts of the imaging region overlap with each other. Smoother image switching display is made possible by sequentially switching and displaying images captured by the five cameras having these different imaging viewpoints in accordance with the behavior of a driver, for example, movements of the head.

<FIG> illustrates an example of switching images in a case where more cameras (c1 to cn = <NUM>) are disposed in the rear of the automobile <NUM> and images are switched.

A driver can observe images smoothly changing like a moving image by sequentially switching manual images captured by a plurality of cameras in this way.

<FIG> illustrates an example in which the number of cameras installed in the rear of the automobile <NUM> is set smaller than that in the example in <FIG>. Although images may be switched with a small number of cameras in this way, parallax difference between the cameras causes intermittent images. In such a case, processing of calibrating each camera captured image is performed to set and display an infinite point <NUM> of each image at a fixed position of the image. The processing enables inhibition of subject blurring generated at the time of switching images, and enables an image viewer (driver) to observe a smoothly changing image without unnatural change.

Furthermore, image correction of sweeping and shifting an image in a lateral direction at a neighboring point in an image in accordance with movements of the head of the driver may be performed. The processing enables the driver to visually and intuitively recognize the switching between the camera images.

Furthermore, in a case where a small number of cameras are installed in the rear of the automobile <NUM>, and the parallax difference between the cameras causes intermittent images, a virtual viewpoint image of a middle viewpoint between two adjacent cameras may be synthesized and displayed on the basis of images captured by the two adjacent cameras.

That is, image display control of switching an image output to the display unit to any of images captured by a plurality of cameras or a virtual viewpoint composite image is performed.

<FIG> illustrates an example of generating a virtual viewpoint image. Here, (<NUM>) camera C3 captured image and (<NUM>) camera C4 captured image in <FIG> are images actually captured by adjacent cameras C3 and C4.

Virtual viewpoint composite images of (<NUM>) and (<NUM>) are generated on the basis of these two actually captured images.

Here, (<NUM>) virtual viewpoint composite image is a composite image generated in the proportion of (<NUM>) camera C3 captured image of <NUM>% and (<NUM>) camera C4 captured image of <NUM>%.

Smoothly changing images can be presented for a driver by sequentially switching and displaying the four images (<NUM>) to (<NUM>) in <FIG>.

What is important for the driver is, however, not to visually check the details of an image at a halfway viewpoint between cameras but to judge a rear situation. The driver sometimes cannot judge a switched situation by just instantly viewing an acquired image, at a rear C camera position, captured by the rear C camera <NUM> at the center of the rear of the car and an image taken from a camera disposed on a side surface. Switching and presenting a plurality of halfway images for the driver enables the driver to sensorily and naturally recognize image switching. Thus, confusion in thinking at the time when instant situation judgment is necessary can be avoided.

Switching based on, for example, the orientation of a face and direction of line-of-sight, switching in accordance with a so-called gesture such as hand or neck shaking, and the like can be used for switching images. Switching with a gesture can give an advantage that it is unnecessary to maintain an improper posture. Note, however, that, in a case where an image is switched without depending on the orientation of a face or the direction of a line-of-sight, the observation posture of the driver is the same. Thus, information indicating switching and transition of an image, for example, auxiliary information such as a switching mark is preferably displayed.

In the processing of the present disclosure, a means for properly checking the rear without occlusion is provided by providing camera images of the rear of a vehicle obtained from a plurality of different viewpoints. The camera images having the different viewpoints are made to have continuity. An image on the halfway of switching is simulatively generated by using human visual characteristics, and presented on a monitor. In this way, the driver can sensorily grasp viewpoint movements from the image. In formation can be provided while reduction or loss in grasping a situation is inhibited. That is, ergonomically, viewpoints can be grasped well by the driver viewing an image that warps between changing cameras having different viewpoints.

<FIG> illustrates an example in which a camera position icon <NUM> is superimposed and displayed on an image displayed on the display unit <NUM>. The camera position icon <NUM> indicates a camera imaging viewpoint. In this way, displaying the camera position icon <NUM>, which indicates the position of a camera that has captured an image, together with the image enables the driver to immediately check an image captured position, and more quickly recognize the position of a subject in the image.

<FIG> illustrates an example in which a rear left end camera image and a left side camera image are displayed side by side on the display unit <NUM>. The rear left end camera image is an image captured by a rear left end camera installed at a left end of the rear of the automobile <NUM>. The left side camera image is an image captured by the left side camera <NUM>. Such display enables the situation on the left side of the automobile <NUM> to be certainly grasped.

Note that, similarly, displaying an image captured by a rear right end camera installed at a right end of the rear of the automobile <NUM> and an image captured by the right side camera <NUM> side by side enables the situation on the right side of the automobile <NUM> to be certainly grasped.

Although <FIG> illustrates an example in which a pictogram representing a camera is displayed in the vicinity of a vehicle bumper, a simple color marker, the entire screen, or right and left frames may be displayed thick, and made blinking. Flow zebra caution may be simply displayed. Another caution calling display may be made as long as the driver can intuitively understand camera arrangement at the time of looking at the screen. Furthermore, a plurality of displays may be combined. Furthermore, configuration, in which switching to a display method that does not impair intuitive understanding can be made, can be adopted in accordance with the preference of the driver.

Note that, for example, as illustrated in <FIG>, the following two installation aspects are mainly adopted as an aspect of installing a plurality of cameras disposed in the rear of the automobile <NUM>.

Here, (a) radial arrangement is an example in which cameras are disposed on a curve having a predetermined curvature, and is characterized by being able to capture an image in a wider range. In contrast, (b) linear arrangement has an advantage that, although the imaging range is narrowed, an amount of deformation of an object in each image is small, and an amount of correction processing for a display image is reduced. Note that, although, contrary to the radial type of (a), arrangement in a direction of convergence to a rear neighboring point is possible in a convergence arrangement, the installation aspect has a disadvantage that continuity in directions to the left side camera <NUM> and the right side camera <NUM> cannot be obtained. An image of actually installed camera is used as a projection image of a central projection image, and it is not necessary to dispose an optical axis in each direction. An image may be converted from a fisheye projection image, and image conversion may be used so that an optical axis of a virtual central projection image corresponds to an optical axis in accordance with each installation aspect.

Next, an image processing apparatus and a sequence of processing executed by a moving apparatus of the present disclosure will be described with reference to the flowchart of <FIG>.

The processing of the flow in <FIG> and subsequent figures is executed in a moving apparatus or an image processing apparatus mounted in the moving apparatus. For example, the processing is performed under the control of a data processing unit including a processor such as a CPU. The processor has a function of executing a program in accordance with the program stored in a storage unit in an apparatus.

The processing of each step of the flow in <FIG> will be described below.

First, in Step S101, the data processing unit detects a face, line-of-sight direction, or a gesture of a driver, or information regarding operations of the driver.

This is processing of, for example, the data processing unit <NUM> in <FIG> inputting detection information from the driver information detection unit <NUM> and information regarding operations of the driver to the operation unit.

Next, in Step S102, the data processing unit determines whether or not the input of a face, change of a line-of-sight direction, or a gesture of a driver, or information regarding operations of the driver is detected.

If the input is detected, the processing proceeds to Step S103. If the input is not detected, the detection processing of Step S101 is continued.

Next, in Step S103, the data processing unit selects a camera captured image in a direction in accordance with the face, the line-of-sight direction, or the gesture of the driver, or the operation information, which has been input in Step S102.

Note that, for example, if there is no image that completely matches a line-of-sight direction of the driver, an image in a closest direction is selected.

For example, if the face of, the line-of-sight, or the gesture the driver, or the input to the operation unit is in a left direction, an image captured by a camera (rear L camera) on the left side of the rear of the automobile <NUM> is selected.

Furthermore, for example, if the face, the line-of-sight, or the gesture of the driver, or the input to the operation unit is in a central direction, an image captured by a camera (rear C camera) in the center of the rear of the automobile <NUM> is selected. Alternatively, for example, if the face, the line-of-sight, or the gesture of the driver, or the input to the operation unit is in a right direction, an image captured by a camera (rear R camera) on the right side of the rear of the automobile <NUM> is selected.

Next, in Step S104, the data processing unit displays the image that has been selected in Step S103 on the display unit.

Note that various settings can be adopted for an image displayed on the display unit. The various settings include, for example, a setting in which only a selected rear camera captured image is displayed and a setting in which two side camera images captured by right and left side cameras are displayed together.

In the flow in <FIG>, processing of selecting an image in the nearest direction is performed in a case where, for example, there is no image that completely matches a line-of-sight direction of the driver at the time when a camera captured image in a direction in accordance with the face, the line-of-sight direction, or the gesture of the driver, or the operation information, which has been input in Step S102, is selected.

Next, a sequence of processing of synthesizing images in the line-of-sight direction of the driver in such a case, that is, in a case where there is no image that completely matches the line-of-sight direction of the driver will be described with reference to the flowchart of <FIG>.

First, in Step S201, the data processing unit detects a face, a line-of-sight direction, or a gesture of a driver, or information regarding operations of the driver.

Next, in Step S202, the data processing unit determines whether or not the input of a face, change of a line-of-sight direction, a gesture of a driver, or information regarding operations of the driver is detected.

If the input is detected, the processing proceeds to Step S203. If the input is not detected, the detection processing of Step S201 is continued.

Next, in Step S203, the data processing unit determines the presence or absence of a camera captured image in a direction in accordance with the face, change of the line-of-sight direction, or the gesture of the driver, or the information regarding operations of the driver, which has been input in Step S202.

If there is a captured image in the matching direction, the processing proceeds to Step S204.

In contrast, if there is no captured image in the matching direction, the processing proceeds to Step S206.

In a case where it is determined in Step S203 that there is the camera captured image in a direction in accordance with the face, change of the line-of-sight direction, or the gesture of the driver, or the information regarding operations of the driver, which has been input in Step S202, the processing of Steps S204 and S205 is executed. The display is mainly suitable for a case where cameras are adjacently disposed at the time of switching and an image is not described in a bound. If imaging directions of switching images are far away from each other, switching in accordance with the later-described Step S206 is performed.

In Step S204, a camera captured image in a direction in accordance with the face, the line-of-sight direction, or the gesture of the driver, or the operation information, which has been input in Step S202, is selected.

For example, if the face, the line-of-sight, or the gesture of the driver, or the input to the operation unit is in a left direction, an image captured by a camera (rear L camera) on the left side of the rear of the automobile <NUM> is selected.

Next, in Step S205, the data processing unit displays the image that has been selected in Step S204 on the display unit.

In contrast, in a case where it is determined in Step S203 that there is no camera captured image in a direction in accordance with the face, change of the line-of-sight direction, or the gesture of the driver, or the information regarding operations of the driver, which has been input in Step S202, the processing of Steps S206 and S207 is executed.

First, in Step S206, processing of generating, as a composite image, a camera captured image in a direction in accordance with the face of, change of the line-of-sight direction, or the gesture of the driver, or the information regarding operations of the driver, which has been input in Step S202 is executed.

The composite image generating processing corresponds to, for example, the image synthesizing processing described above with reference to <FIG>.

The image synthesizing processing can be performed by, for example, simply adding a weight to images actually captured by adjacent cameras. Furthermore, a middle virtual image may be generated by detecting a moving object for each region and performing correction on the basis of the detection information. Note that information loss of a driver is prevented by presenting a composite image of a middle viewpoint between actually captured images to the driver. Specifically, for example, if the captured image is switched from the rear central (C) camera <NUM> to a side camera, the image greatly changes in an instant. The composite image is effective in preventing the information loss of the driver, which may occur in such a case.

Note that, for example, input of the face, the line-of-sight direction, or gesture of the driver, or image selection indication in Step S206 makes various movements such as moderate movements and sharp movements. Steps <NUM> and <NUM> are executed as steps of generating one or two or more plurality of middle images, that is, composite images in accordance with the detected movement.

Next, in Step S207, the data processing unit displays the composite image that has been generated in Step S206 on the display unit. In addition, the driver can grasp the camera position of the display image by superimposing and displaying the description camera position on a screen of the display unit <NUM> in <FIG> at the time when the composite image in Step S207 is output to the display unit. Note that various settings can be adopted for an image displayed on the display unit. The various settings include, for example, a setting in which only a selected rear camera captured image is displayed and a setting in which two side camera images captured by right and left side cameras are displayed together.

Furthermore, in the description of the sequence, images are switched by, for example, movement of face, line-of-sight, and head based on the will of the driver, a gesture, or an operation. In contrast, as described above with reference to <FIG>, <FIG>, and <FIG>, an object behind a vehicle in the rear, that is, an object in an occlusion region may be detected, and a warning prompting a driver to visually check details may be displayed as a trigger for the driver to check a side camera image at the time when the driver looks at the rear central (C) camera <NUM>. The warning includes a zebra pattern and a blinking superposition warning display at the rear object boundary that generates the occlusion.

That is, if, for example, a large vehicle approaches behind, and only an image captured by a central camera is used, an approaching vehicle or the like behind the large vehicle is easily overlooked due to the occlusion. In such a case, a warning indicating that an object is in the occlusion region is displayed. The warning prompts the driver to perform checking with images captured by cameras having different viewpoints. The processing enables the driver to change the camera viewpoints and certainly check the rear situation even in a case where there is occlusion, thereby improving safety.

Note that an image may be switched in configuration in which different camera captured images in a single bound are provided. Preferably, an image having a virtual viewpoint between cameras is output between different camera captured images, and display with image transition and continuity is performed. The processing prevents loss in grasping a situation of the driver, and achieves safe checking of the rear side.

Next, a hardware configuration example of an image processing apparatus that executes the above-described processing will be described.

<FIG> illustrates a hardware configuration example of the image processing apparatus.

A central processing unit (CPU) <NUM> functions as a data processing unit that executes various pieces of processing in accordance with a program stored in a read only memory (ROM) <NUM> or a storage unit <NUM>. For example, processing is performed in accordance with the sequence described in the above-described embodiment.

For example, a program to be executed by the CPU <NUM> and data are stored in a random access memory (RAM) <NUM>. These CPU <NUM>, ROM <NUM>, and RAM <NUM> are mutually connected by a bus <NUM>.

The CPU <NUM> is connected to an input/output interface <NUM> via the bus <NUM>. An input unit <NUM> and an output unit <NUM> are connected to the input/output interface <NUM>. The input unit <NUM> includes, for example, a situation data acquisition unit such as various switches, a keyboard, a touch panel, a mouse, a microphone, a sensor, a camera, and a GPS. The output unit <NUM> includes, for example, a display and a speaker.

The CPU <NUM> inputs, for example, a command and situation data input from the input unit <NUM>, executes various pieces of processing, and outputs the processing result to, for example, the output unit <NUM>.

The storage unit <NUM> connected to the input/output interface <NUM> includes, for example, a hard disk, and stores a program to be executed by the CPU <NUM> and various pieces of data. A communication unit <NUM> functions as a transmission/reception unit for data communication via a network such as the Internet or a local area network, and communicates with an external apparatus.

Claim 1:
An image processing apparatus comprising
a plurality of cameras comprising a plurality of rear cameras and a side camera;
a data processing unit that executes image display control of:
inputting driver information indicating behavior of a driver of a moving apparatus, wherein the driver information includes information obtained by detecting a direction of a face or line-of-sight of the driver, or a gesture of the driver, or an operation of the driver;
inputting images captured by the plurality of cameras that images a situation around the moving apparatus from different viewpoints;
determining the presence or absence of an image captured by a camera in a direction in accordance with the face, change of the line-of-sight direction, or the gesture of the driver, or the information regarding operations of the driver; and
if an absence of an image captured by the plurality of cameras in the detected direction is determined, switching an image output to a display unit to a virtual viewpoint composite image in accordance with the driver information, wherein the virtual viewpoint composite image is a composite image based on images captured by two adjacent cameras of the plurality of cameras, and is a composite image corresponding to a captured image from a virtual viewpoint between imaging viewpoints of the two adjacent cameras of the plurality of cameras.