IMAGE PROCESSING DEVICE AND IMAGE PROCESSING METHOD

An image processing device connected to a display device having a non-planar screen. The image processing device generates a parameter for generating a display-target image in accordance with information on an eye position; generates a display-target image from an outside-world image based on the generated parameter; and outputs the generated display-target image to the display device. The parameter includes: an element of a projective transformation matrix for transforming the coordinates of the outside-world image from those in the first coordinate system corresponding to the real space into those in the coordinate system (a second coordinate system) of the outside-world image; and an element of a projective transformation matrix for transforming, into the coordinates of the screen, the coordinates of a part of the outside-world image corresponding to the area of the reference plane to be displayed on the screen.

TECHNICAL FIELD

The present invention relates to an image processing device and image processing method.

BACKGROUND

An image processing device capable of displaying an image that expands the field of vision of a driver is known. A specific configuration of this type of image processing device is described in, for example, Patent Document 1 (JP 2007-096638 A).

The image processing device described in Patent Document 1 converts the coordinates of a captured image of the outside of a vehicle by an image capture device to display, on a screen of a display device, an image equivalent to the one that the driver views the outside of the vehicle from his/her eye-point through the installation region of a display device as if the installation region were empty.

Some display devices installed in a pillar portion in a vehicle cabin have a display device using a flexible-shape display with a curved-surface screen. A screen having such a complicated shape needs an enormous amount of computation to render a vehicle's exterior image that is deemed natural when the occupant of the vehicle views it. What is thus required is an image processing device having high processing capacity, which requires an increased cost.

Therefore, in light of the foregoing, an object of the present application is to provide an image processing device and an image processing method that can reduce the load of processing a moving body's outside-world image to be displayed on a non-planar screen.

SUMMARY

An image processing device according to an embodiment of the present application is a device disposed in a moving body and connected to a display device having a screen at least a part of which is non-planar. The image processing device includes an eye-point information acquisition unit that acquires information on an eye position of an occupant of a moving body, an image acquisition unit that acquires an outside-world image showing an outside world of the moving body, a parameter generation unit that generates a parameter for generating a display-target image to be displayed on a display device from the outside-world image according to the information on the eye position, an image generation unit that generates the display-target image from the outside-world image based on the parameter generated by the parameter generation unit, and an image output unit that outputs the display-target image generated by the image generation unit to the display device. The parameter includes at least the following two elements. A first element is an element of a projective transformation matrix for transforming the coordinates on a reference plane set in a real space from a first coordinate system corresponding to the real space into a second coordinate system which is the coordinate system of the outside-world image. A second element is an element of a projective transformation matrix for transforming the coordinates of the screen into the coordinates of the outside-world image of a part of the reference plane corresponding to the area to be displayed on the screen. The latter conversion is performed by specifying the coordinate on the reference plane, which is the intersection of the extension line of the line segment connecting the eye position in the first coordinate system and the coordinate on the screen disposed in the first coordinate system based on screen data indicating a shape and a position of the screen, and the reference plane set in the first coordinate system.

According to an embodiment of the present application, an image processing device and an image processing method are provided that are capable of reducing a load of the processing executed to display an outside-world image of a moving body on a non-planar screen.

DETAILED DESCRIPTION OF EMBODIMENTS

The following description relates to an image processing device, an image processing method, and an image processing program according to one embodiment of the present application. Note that common or corresponding elements are marked with the same or similar reference codes, and duplicate descriptions are simplified or omitted as appropriate.

FIG.1is a block diagram illustrating a configuration of a display control system1according to one embodiment of the present application.FIG.2is a block diagram illustrating a configuration of a display control system1.

The display control system1includes an electronic control unit (ECU)10, an interior camera20, a driver monitoring system (DMS)30, an exterior camera40, a human machine interface (HMI)50, and display devices60Rand60L. Note thatFIG.1andFIG.2illustrate main components necessary for the description of the present embodiment, and some components, e.g., a housing, are omitted from the drawing as appropriate.

The display control system1is a system incorporated in a vehicle traveling on a road (an example of a moving body). Note that the configurations inFIG.1andFIG.2are illustrated merely by way of examples. For example, the DMS30may be incorporated into the ECU10. In other words, there is a high degree of freedom in the configuration of the display control system1, and various design changes are possible.

The ECU10is an exemplary image processing device and has a processor100and a storage device200. Note that the ECU10may be a navigation device or a device that forms part of an in-vehicle infotainment (IVI) device. The image processing device is not limited to an on-board device such as the ECU10. The image processing device may be in a different form such as a smartphone, a feature phone, a tablet-type terminal device, a personal computer (PC), a personal digital assistant (PDA), a portable navigation device (PND), a handheld game device, and the like.

The processor100executes an image processing program200A stored in the storage device200. In other words, the processor100is an exemplary computer that executes the image processing program200A.

The processor100includes a Random Access Memory (RAM), a flash Read-Only Memory (ROM), and the like, and controls the entire display control system1. For example, the processor100deploys various programs, including the image processing program200A stored in the storage device200, on the RAM serving as a work area and controls the display control system1in accordance with the deployed programs.

The processor100is a single processor or a multiprocessor, for example, and includes at least one processor. When configured to include a plurality of processors, the processor100may be packaged as a single device, or it may include multiple devices housed within the ECU10but physically separated from each other.

The image processing program200A stored in the storage device200is a program to be executed by the processor100, which is an exemplary computer connected to the display devices60Rand60L, each of which is disposed in the vehicle, i.e., a moving body, and has a screen at least a part of which is non-planar. The image processing program200A causes the processor100to execute a series of processes: acquiring information representing the eye-point of an occupant of the moving body (note that as used herein, the term “eye-point” represents the eye position of the occupant); acquiring an outside-world image showing an outside world of the moving body; generating, in accordance with the information on the eye position, a parameter for generating, from the outside-world image, a display-target image to be displayed on the display devices60Rand60L; generating the display-target image from the outside-world image based on the generated parameter; and outputting the generated display-target image to the display devices60Rand60L. The parameter mentioned above includes at least the following two elements. A first element is an element of a projective transformation matrix for transforming the coordinates of the outside-world image from a first coordinate system corresponding to the real space into a second coordinate system which is the coordinate system of the outside-world image. A second element is an element of a projective transformation matrix for transforming the coordinates of the screen into the coordinates of the outside-world image of a part of the reference plane corresponding to the area to be displayed on the screen. The latter conversion is performed by specifying the coordinates, on the reference plane, of the intersection, with the reference plane set in the first coordinate system, of the extension line of the line segment connecting the eye position in the first coordinate system and the position with coordinates on the screen arranged in the first coordinate system based on the screen data indicating the shape and the position of the screen.

In other words, the image processing program200A causes the processor100to execute an image processing method that includes the above series of processes.

Executing image processing using the image processing program200A makes it possible to reduce the load of processing that is executed to display a moving body's outside-world image on a non-planar screen.

FIG.3is a diagram depicting an area around a driver seat of a vehicle in which the display control system1is incorporated.FIG.3is a view seen obliquely forward from the rear seat of the vehicle. The vehicle's interior ofFIG.3includes the interior camera20, a display device60C, the display device60R, and the windshield300. The display device60Ris embedded in a right front pillar portion on the right side of a windshield300. The display device60Lis embedded in a left front pillar portion on the left side of the windshield300.

The interior camera20is installed near the display device60C, for example. The interior camera20captures an image of, for example, an occupant2seated in the driver seat.

The DMS30performs face recognition and eye-point detection of the occupant2using an interior image P20captured by the interior camera20. By way of example, the DMS30uses known image recognition technology to execute the processing for recognizing the position of the face of the occupant2, the orientation of the face, each part of the face including the eyes, motions of the occupant2with respect to the face, and the like. Some of the motions of the occupant2with respect to the face include, for example, winking, nodding, or the like.

The DMS30uses the results of face recognition to detect eye-point coordinates PV, which represent the eye position of the occupant2. Then, the DMS30outputs the coordinates to the processor100. The eye-point coordinate PV may be the coordinate of a dominant eye (either the right eye or the left eye) of the occupant2, or may be the midpoint of a line segment connecting the left and right pupils. The occupant2can, for example, operate the HMI50to pre-input his/her own dominant eye. Note that the DMS30may directly detect the eye-point coordinates PV of the occupant2from the interior image P20without using the results of face recognition. Conversely, the DMS30may detect the coordinates of the occupant's parts other than the eye position and/or detect the facial contours, and estimate a standard eye position corresponding thereto as the eye-point coordinates PV of the occupant2.

The exterior camera40captures an image of the outside field of the vehicle. By way of example, the exterior camera40captures an image of the field in front of and the fields on the sides of the vehicle. The imaging field of view of the vehicle exterior camera40includes a blind area hidden behind the right front pillar portion and a blind area hidden behind the left front pillar portion when viewed from the eye-point of the occupant2, i.e., from the eye position of the occupant2(for example, a position slightly in front of the headrest of the driver's seat). The exterior camera40outputs the captured vehicle's exterior image P40to the processor100.

The exterior camera40, which is an example of an image capture device, may be a camera equipped with a wide-angle lens and thus being capable of capturing a wide angle of view for the purpose of capturing a wide area. The wide-angle lens that the exterior camera40is equipped with is, for example, a fish-eye lens. Therefore, the vehicle's exterior image P40captured by the exterior camera40is an image with distortion (barrel distortion).

The exterior camera40may include either only a single camera or a plurality of cameras. The vehicle's exterior image P40may, for example, be a combination of: the image captured by a front camera operable to image the field in front of the vehicle; and each of the images captures by the corresponding one of a pair of left and right side cameras operable to image the fields on their respective sides of the vehicle.

The HMI50may presumably be any of various user interfaces of hardware, software, or a combination thereof. By way of example, the HMI50is a mechanical switch key installed on a dashboard or a remote controller. When the display device60Cis equipped with a touch panel, the graphical user interface (GUI) provided in a touch panel environment also forms a part of the HMI50. The occupant2can operate the display control system1via the HMI50.

The display device60Cis, for example, a liquid crystal display (LCD) device with a touch panel and is installed on the dashboard. The display devices60Rand60Lare also LCD devices, and are installed in the right front pillar portion and in the left front pillar portion, respectively. These display devices are not limited to LCD devices, but may be another form of display devices, for example, an organic electro luminescence (EL) display device or the like.

In the illustration ofFIG.3, the linearly extending object OJ exists outside the vehicle. In this example, a high-precision output image PRIMGcorresponding to the eye-point of the occupant2is displayed on the screen60ARof the display60Rinstalled in the right front pillar portion. Therefore, as illustrated inFIG.3, a part of the object OJ which is visible via the windshield300, and a part of the object OJ which is displayed on the screen60AR(referred to as OJ' for the convenient sake) are aligned with each other in a straight line. The occupant2can visually recognize, via the screen60AR, the blind area that cannot otherwise be seen directly.

InFIG.3, the center line of the object OJ is denoted by a reference symbol LC. As illustrated inFIG.3, the center line LC of the object OJ coincides with the center line of the object OJ' displayed on the screen60AR. As the object OJ and the object OJ' appears to be a continuous linear-shaped object, the output image PRIMGappearing on the screen60ARcan be perceived by the occupant2as a natural image.

In this way, the screen60ARdisplays the output image PRIMGthat is similar to the scene of the outside of the vehicle viewed from the eye-point of the occupant2as if the same scene were viewed through the installation region of the screen60AR. The screen60ALof the display apparatus60Lalso displays an output image PLIMGthat is similar to the scene of the outside of the vehicle viewed from the eye-point of the occupant2as if the same scene were viewed through the installation region of the screen60AL.

Hereinafter, for convenience's sake, the display device60Rand the display device60Lmay be collectively referred to as a “display device60.” The screen60ARand the screen60ALmay be collectively referred to as a “screen60A.” The output image PRIMGand the output image PLIMGmay be collectively referred to as an “output image PIMG.”

In the present embodiment, the occupant2can be made to perceive the scene of the outside the vehicle shown in the screen60A as a part of the scene of the real world. Since the occupant2can view the scene behind the front pillar portion, the occupant2can be made to perceive as if the front pillar portion were transparent.

It should be noted that the display device60is a curved-screen display device having a curved screen. In other words, the display device60is an exemplary display device disposed in a vehicle (an example of a moving body) and having a screen at least a part of which is non-planar.

Conventionally, an enormous amount of computation is necessary for a display device having a complicated screen shape such as a curved-screen display device to draw a vehicle's exterior image that is deemed natural. What is thus required is an image processing device having high processing capacity, which needs an increased cost.

Therefore, for the purpose of reducing the load of processing on the processor100to display, on the display apparatus60, the high-precision image PIMGmatching the eye-point of the occupant2, the ECU10according to the present embodiment has the following configuration.

The processor100includes, as functional blocks, an eye-point information acquisition unit100A, an image acquisition unit100B, a parameter generation unit1000, an image generation unit100D, and an image output unit100E. Each functional block is realized by the image processing program200A executed by the processor100. Each functional block may be partially or wholly implemented by hardware such as a dedicated logic circuit, or the like.

The eye-point information acquisition unit100A acquires information on the eye-point of the occupant2. By way of example, the eye-point information acquisition unit100A acquires the eye-point coordinates PV from the DMS30. It should be noted that the eye-point coordinates PV may be set, for example, by an operation on the HMI50by the occupant2. In this case, the DMS30can be omitted from the display control system1.

The image acquisition unit100B acquires a vehicle's exterior image P40captured by the exterior camera40(an example of an outside-world image showing the outside world of the moving body).

The parameter generation unit100C generates, in accordance with the eye-point coordinates PV (an example of information on the eye-point), a parameter PMT for generating, from the vehicle's exterior image P40, an output image PIMG(an example of a display-target image) to be displayed on the display apparatus60. More specifically, the parameter PMT generates, from the vehicle's exterior image P40, a first parameter PMT for generating: the output image PRIMGto be displayed on the display device60R; and a second parameter PMT for generating the output image PLIMGto be displayed on the display device60L.

Based on the first and second parameters PMT generated by the parameter generation unit1000, the image generation unit100D generates the output images PRIMGand PLIMGfrom the vehicle's exterior image P40.

The image output unit100E outputs the output images PRIMGand PLIMGgenerated by the image generation unit100D to the display devices60Rand60L, respectively.

The output image PRIMGas illustrated inFIG.3is displayed on the screen60ARof the display device60R. Although not illustrated inFIG.3, as is the case with the screen60AR, the screen60ALof the display apparatus60Lalso displays the output image PLIMGthat is similar to the scene of the outside of the vehicle viewed from the eye-point of the occupant2as if the same scene were viewed through the installation region of the screen60AL.

As will be described in detail later, the parameter PMT includes at least the following two elements. A first element is an element of a projective transformation matrix for transforming the coordinates on a reference plane set in a real space from a first coordinate system corresponding to the real space into a second coordinate system which is the coordinate system of the vehicle's exterior image P40. A second element is an element of a projective transformation matrix for transforming the coordinates of the screen60AR(or the screen60AL) into the coordinates of the vehicle's exterior image P40of a part of the reference plane corresponding to the area to be displayed on the screen60AR(or the screen60AL). The latter conversion is performed by specifying the coordinates, on the reference plane, of the intersection, with the reference plane set in the first coordinate system, of the extension line of the line segment connecting the eye-point coordinates PV in the first coordinate system and the coordinates on the screen60AR(or the screen60AL) arranged in the first coordinate system based on the screen data indicating the shape and the position of the screen60AR(or the screen60AL).

In other words, the processor100generates the above-described parameter PMT including the elements of the projective transformation matrix in real time in accordance with the eye-point coordinates PV. Use of such a parameter PMT enables the high-accuracy output image PIMGthat is changing in real time in accordance with the eye-point of the occupant2to be displayed on the screen60ARand the screen60ALof their respective display devices60Rand60L, which are curved-screen display devices, at a lower computation cost than the cost needed in the conventional technique.

FIGS.4to6are diagrams providing additional descriptions to the image processing to be performed by the processor100(in particular, the parameter generation unit100C and the image generation unit100D). A method of generating the output image PRIMGto be outputted to the display device60Rwill be described with reference to these figures. A method of generating the output image PLIMGis not provided to avoid redundancy in description.

The vehicle coordinate system is an exemplary first coordinate system corresponding to the real space, and is represented by three axes of XV, YV, and ZVas illustrated inFIG.4, for example. The vehicle coordinate system is a coordinate system based on the vehicle in which the display control system1is incorporated, and is a 3D coordinate system with the position where the vehicle is located as the origin O. The axis XVextends in a vehicle width direction. The axis YVextends in a vehicle up-down direction. The axis ZVextends in a vehicle front-rear direction. The unit for each axis is millimeters (mm).

The image coordinate system is an exemplary second coordinate system that is a coordinate system of an outside-world image, and is represented by two axes of a horizontal axis XCand a vertical axis YCas illustrated inFIG.4, for example. The image coordinate system is a coordinate system based on the vehicle's exterior image P40, and is a two-dimensional coordinate system with the origin O at the upper left corner of the vehicle's exterior image P40. The unit for each axis is pixels (px).

The reference plane SM is a virtual plane set in the vehicle coordinate system, and is a vertical plane that is parallel to the axis YVas illustrated inFIG.4, for example. By way of example, a vertical plane separated forward from the vehicle by a predetermined distance is set as the reference plane SM. In addition, for example, the processor100may perform image recognition processing, and a position of a vertical surface (a wall surface or the like) recognized by this processing may be set as the reference plane SM. The reference plane SM is not limited to a vertical plane. A horizontal plane (e.g., a ground surface or the like) may be set as the reference plane SM. Alternatively, an obliquely inclined plane may be set as the reference plane SM.

The screen60ARillustrated inFIG.5is arranged in the vehicle coordinate system and is a plane that the output image PRIMGis projected onto.

It should be noted that the storage device200of the ECU10stores a screen-data database200B. The screen-data database200B includes screen data of each screen such as the screen60ARand the screen60AL. The screen data include information on the shape and the position of the screen in the vehicle coordinate system. The screen60ARillustrated inFIG.5is defined in the vehicle coordinate system on the basis of the screen data of the screen60AR.

The coordinates of the screen60ARin the vehicle coordinate system are obtained by geometric calculation based on, for example, minimum information required to define a curved surface. The coordinates of the screen60ARmay be obtained by referring to the coordinate array elements in the vehicle coordinate system. In any case, the coordinates of the screen60ARare obtained from the screen data of the screen60AR.

The shape (including the size) of the display device is different from one product to another, for example. In addition, the installation position of the display device is different from one vehicle model to another. Hence, the screen-data database200B may store a plurality of types of screen data corresponding to display devices of various shapes and of various installation positions. By storing various types of screen data in advance, the screen data to be applied in the image processing conducted by the processor100(in other words, the screen to be arranged in the vehicle coordinate system based on the screen data) can be switched in accordance with the product and the model of the vehicle.

The screen-data database200B is an exemplary storage unit that stores a plurality of types of image data.

The screen data may be provided on a network instead of locally, for example. In this case, the processor100accesses the screen-data database on the network via a moving body wireless communication unit (not illustrated) to download the screen data.

FIGS.7and8are conceptual diagrams describing an example of screen data D1of the screen60AR.

As conceptually illustrated inFIG.7, the screen data D1of the screen60ARinclude: at least two point data P1and P2(three-dimensional coordinate data) defining a vertical side of the screen60AR; curve data C1defining a curved surface (for example, data on a Bezier curve drawn either in a two-dimensional space or in a three-dimensional space); and rotation angle data R1, which are data on the angle of rotation relative to the curve.

In the example ofFIG.7, the point data P1and P2are used to define the position and the length of one vertical side of the screen60ARin the vehicle coordinate system. For convenient sake, a line segment connecting the point data P1and the point data P2is denoted by a reference symbol LA. The curve data C1are used to define the shape and the length of the curved upper and lower sides that are orthogonal to the vertical side. For convenient sake, the curve defined by the curve data C1is denoted by the reference symbol C2. The rotation angle data R1is used to define the orientation of each of the curved upper and lower sides. By connecting the end points P3and P4of the curve data C1, information on the shape and the position of the screen60ARis obtained.

In this way, the screen data of the substantially rectangular screen60ARformed in a curved-surface shape that is curved in the vertical-side direction (an example of the first direction) include: information on the curved shape in the vertical-side direction; information on the linear shape in the perpendicular-side direction (an example of the second direction) that is orthogonal to the vertical-side direction; and information on the installation angle (that is, the curved data C1, the point data P1and P2, and the rotation angle data R1).

In order to define a more accurate curved-surface shape, as conceptually illustrated inFIG.8, the shape and the position of the screen60ARare obtained by geometric calculation based on the curve data C1, the point data P1and the point data P2, and the rotation angle data R1.

In this geometric calculation, an array of coordinates of the grid points on the screen60AR(in other words, the curved surface defined by the screen data D1) is obtained. In order to reduce the load of processing, an array of the coordinates of the grid points corresponding to some pixels discretely arranged at regular intervals are obtained instead of the array of the coordinates of the grid points corresponding to all pixels. The array of coordinates of the grid points obtained here correspond to the coordinate group D indicated by each grid point on the output image PRIMGillustrated inFIG.6.

As illustrated inFIG.8, the curve C2defined by the curve data C1is divided into n equal parts. This value of n is obtained by subtracting 1 from the number of grid points in the perpendicular-side direction. The coordinates of both end points of the curve C2and the coordinates of each division point of the curve C2divided into n equal parts are calculated.

The division point group CV1on the curve C2calculated above is replicated. The division point group CV1of the replication source is moved so that one end point of the division point group CV1of the replication source is positioned at the point data P1, and the division point group CV1of the replication is moved so that one end point of the division point group CV1of the replication is positioned at the point data P2. It should be noted that the direction of the division point group CV1of the replication source is determined so that the line segment LA connecting the point data P1and the point data P2is orthogonal to the curve C2formed by the division point group CV1. The replicated division point group CV2is oriented in the same orientation as that of the division point group CV1of the replication source.

The line segment LA is equally divided into m segments. This value of m is obtained by subtracting 1 from the number of grid points in the vertical-side direction. The coordinates of both end points of the line segment LA and the coordinates of each division point of the line segment LA divided into m equal parts are calculated.

The division point group CV2on the line segment LA calculated above is replicated. To be more specific, the division point group CV2is replicated so that the point data P1are located at the other end point of one division point group CV1and the point data P2are located at the other end point of the other division point group CV1.

A line segment LB is defined along the vertical sides by connecting the opposed arrangement points of a pair of division point groups CV1. A line segment LD is defined along the perpendicular sides by connecting the opposed arrangement points of a pair of division point groups CV2.

The coordinates of all the grid points are calculated where each line segment LB and each line segment LD intersect, that is, an array of the coordinates of the grid points is calculated.

The array of coordinates of the grid points is rotated in the vehicle coordinate system in accordance with the rotation angle data R1. As a result, the array of coordinates of the grid points is obtained as the screen data D1including information on the shape and the position of the screen60AR.

In the image processing by the processor100, the output image PRIMGis generated from the vehicle's exterior image P40using the following equation. In addition, a coordinate group D' of the vehicle's exterior image P40which corresponds to the grid point group (coordinate group D) of the output image PRIMG, and which satisfies the following equation (1), is calculated.

D′=f⁡(HD)Equation⁢(1)D: a coordinate group representing a pixel on the output image PRIMGH: projective transformation matrix M×projective transformation matrix Nf: function for transforming the coordinates on the vehicle's exterior image P40after removing the barrel distortion into the corresponding coordinates on the vehicle's exterior image P40before removing the barrel distortionD': a coordinate group on the vehicle's exterior image P40corresponding to the coordinate group D

The formula (1) will be described specifically below.

The coordinate group D is a matrix in which coordinates of a two-dimensionally expanded coordinate system (an example of a third coordinate system) of the screen60ARare horizontally arranged. The coordinate system of the screen60ARis indicated by two axes, XDon the horizontal axis and YDon the vertical axis. The coordinate system of the screen60ARis a two-dimensional coordinate system based on the screen60AR, with the origin O in the upper left corner of the screen60AR. The unit for each axis is pixels (px).

The coordinate group D does not represent a coordinate group of all pixels, but it represents a coordinate group of some pixels corresponding to the screen data D1(i.e., an array of coordinates of grid points corresponding to some pixels discretely arranged at regular intervals). In addition, the processor100holds in advance the correspondence between the coordinates of each grid point indicating the coordinate group D and the corresponding coordinates GP in the vehicle coordinate system (in other words, grid points on the screen60ARdefined by the screen data D1).

The projective transformation matrix H is a product of the projective transformation matrix M and the projective transformation matrix N.

The projective transformation matrix N is used to transform the grid points on the screen60ARdefined by the screen data D1(in other words, the coordinates of the grid points representing the coordinate group D) into the coordinates on the reference plane SM set in the vehicle coordinate system.

InFIG.5, the projection center T indicates the projection center of the output image PRIMGprojected on the screen60AR. In the present embodiment, the eye-point coordinates PV of the occupant2in the vehicle coordinate system are set as the projection center T.

The processor100specifies which area on the reference plane SM set in the vehicle coordinate system is to be projected onto the screen60ARwith the projection center T used as the reference point. In particular, the processor100obtains a line segment L1connecting the projected center T and one of the grid points GP indicating the coordinates of the screen60AR. The processor100obtains coordinates CP of a point on the reference plane SM at which the extension line L1′ of the line segment L1intersects the reference plane SM. By obtaining the coordinates CP on the reference plane SM corresponding to each grid point GP, the area HC corresponding to the screen60ARis specified.

It should be noted that the shape of the reference plane SM and that of the area HC illustrated inFIG.5are merely conceptual and not accurate.

Hence, the projective transformation matrix N is a factor for projectively transforming the grid points GP on the screen60ARinto the coordinates CP on the reference plane SM. The coefficients of the projective transformation matrix N are determined in accordance with the projection center T (eye-point coordinates PV). In a case where the reference plane SM dynamically changes in accordance with an object appearing in the vehicle's exterior image P40instead of being fixed at a specified position in the vehicle coordinate system, the coefficients of the projective transformation matrix N are determined in accordance with the projection center T (eye-point coordinates PV) and the reference plane SM.

In this way, the processor100performs projective transformation of each grid point GP into its corresponding coordinates CP using the projective transformation matrix N.

It should be noted that the homogeneous coordinates (xV, yV) on the reference plane SM of the vehicle coordinate system are denoted by the reference symbol PV. The homogeneous coordinates representing coordinates (xD, yD) on the screen60ARcorresponding to the coordinates (xV, yV) are denoted by the reference symbol PD. The relationship between the homogeneous coordinates PVand PDis expressed by the following equation (2).

Using the projective center T (the eye-point coordinates PV) as the center of the projection, a projective transformation matrix is calculated that is to be used when the points of the grid points GP on the screen60ARis projected onto the reference plane SM as the plane of projection. The projective transformation matrix thus obtained is used as the projective transformation matrix N. It should be noted that values λVand λDrepresent the magnification factors at their respective homogeneous coordinates PVand PD. Regardless of the values λVand λD, other than 0, each homogeneous coordinate represents the same coordinate on each coordinate system.

Each grid point GP (in other words, the corresponding coordinates of the coordinate group D) is transformed into a coordinates CP on the reference plane SM by the above equation (2) using the projective transformation matrix N.

Using the projective transformation matrix M, individual coordinates CP on the reference plane SM are transformed into coordinates CP' on the vehicle's exterior image P40from which the barrel distortion has been removed (in other words, into coordinates in the image coordinate system).

It should be noted that the homogeneous coordinates (xV, yV) on the reference plane SM of the vehicle coordinate system are denoted by the reference symbol PV. The homogeneous coordinates representing coordinates (xC, yC) of the image coordinate system corresponding to the coordinates (xV, yV) are denoted by the reference symbol Pc. The relationship between the homogeneous coordinates PVand PCis expressed by the following equation (3).

Pairs of points PVand PCcorresponding to each other are identified by actual measurement. By substituting the coordinates of these identified points into the equation (3) and solving a simultaneous equations involving individual elements of the projective transformation matrix M, the projective transformation matrix M is calculated. Values λVand λCrepresent the magnification factors at their respective homogeneous coordinates PVand PC. Regardless of the values λVand λC, other than 0, each homogeneous coordinate represents the same coordinate on each coordinate system. The projective transformation matrix M is calculated by assuming that the values λVand λCare 1.

Individual coordinates CP on the reference plane SM set in the vehicle coordinate system are transformed into coordinates CP' in the image coordinate system by the above-mentioned equation (3) using the projective transformation matrix M.

The product H of the projective transformation matrix M and projective transformation matrix N (i.e., H=MN) becomes a projective transformation matrix that transforms the coordinates (xC, yC) of the image coordinate system into the coordinates (xD, yD) of the screen60AR. The relational expression using the projective transformation matrix H is as expressed in the equation (4) below. By obtaining the projective transformation matrix H in advance, the coordinate transformation between the coordinates of the image coordinate system and the coordinates of the screen60ARis easily performed.

Further, a function f obtained in advance is used to transform the vehicle's exterior image P40after the removal of the barrel distortion into the vehicle's exterior image P40before the removal of the barrel distortion, that is, the original image captured by the exterior camera40. Thus, the coordinate group D' of the vehicle's exterior image P40corresponding to the grid point group (coordinate group D) of the output image PRIMGis calculated. In short, the above-mentioned equation (1) is used to calculate the coordinate group D' of the vehicle's exterior image P40corresponding to the grid point group (coordinate group D) of the output image PRIMG.

In this way, in the present embodiment, the projective transformation matrix H is used to perform high-precision coordinate transformation among the image coordinate system, the vehicle coordinate system, and the coordinate system of the screen60AR. Thus, generated from the vehicle's exterior image P40is a highly accurate output image RIMGthat is adapted to the eye-point of the occupant2. This allows the scene that the occupant2visually recognizes through the windshield300and the output image PRIMGdisplayed on the screen60ARto be displayed continuously, as depicted inFIG.3, for example. As a result, the output image PRIMGdisplayed on the screen60ARcan be perceived by the occupant2as a natural image.

Hence, the parameter PMT includes at least the following two elements. A first element is an element of a projective transformation matrix for transforming the coordinates on a reference plane set in a real space from a first coordinate system corresponding to the real space into a second coordinate system which is the coordinate system of the vehicle's exterior image P40(the first element being the projective transformation matrix M). A second element is an element of a projective transformation matrix for transforming the coordinates of the screen60ARinto the coordinates of the vehicle's exterior image P40of a part of the reference plane SM corresponding to the area HC to be displayed on the screen60AR(the second element being the projective transformation matrix N). The latter conversion is performed by specifying the coordinates, on the reference plane SM, of the intersection, with the reference plane SM set in the first coordinate system, of the extension line L1′ of the line segment L1connecting the eye-point coordinates PV in the first coordinate system and the coordinates on the screen60ARarranged in the first coordinate system based on the screen data indicating the shape and the position of the screen60AR. In addition, the parameter PMT further includes an element (function f) for removing the barrel distortion of the vehicle's exterior image P40to transform the coordinates of the vehicle's exterior image P40from the second coordinate system to the first coordinate system.

In the present embodiment, use of such a parameter PMT can reduce the cost for the computation that is executed to perform highly accurate coordinate transformation between the coordinate systems and thus to display a natural vehicle's exterior image reproducing a real scene on a screen (non-planar surface) having a complicated shape such as the screen60AR. Hence, the processing load on the processor100is reduced greatly.

As described above, the parameter PMT is used to transform the coordinates of the vehicle's exterior image P40from the image coordinate system into the vehicle coordinate system and to transform the coordinates within the area HC into the coordinates on the screen60AR, provided that the target of the transformations described above is some of the pixels discretely arranged at regular intervals.

The image generation unit100D performs interpolation processing based on information on the coordinate group D' of the vehicle's exterior image P40corresponding to the grid point group (the coordinate group D) of the output image PRIMG(the above-mentioned information being an example of information on some of the pixels included in the display-target image) generated based on the parameter PMT. The image generation unit100D thus acquires information on all the pixels of the display-target image.

FIG.9is a conceptual diagram illustrating an example of interpolation processing executed by the image generation unit100D. As conceptually illustrated inFIG.9, the image generation unit100D performs interpolation processing based on the information on the coordinate group D' of the vehicle's exterior image P40corresponding to the grid point group (coordinate group D) of the output image PRIMG.

The image generation unit100D performs interpolation of pixel coordinates by, for example, a known method. To interpolate the information of the missing pixel, the image generation unit100D uses such methods as the nearest neighbor interpolation, the bilinear interpolation, the bicubic interpolation, or the like. Thus, obtained is the information on all pixels to be displayed on the screen60AR.

In the present embodiment, processing involving high-precision coordinate transformation is executed only for some of the pixels. Therefore, the processing load on the processor100is greatly reduced as compared to the case where the coordinate transformation processing or the like is executed for all the pixels.

FIG.10is a flowchart illustrating image processing executed by a processor100. For example, when the display control system1starts up, the system begins executing the image processing illustrated inFIG.10. The image processing is repeated at a prescribed rate (e.g., a plurality of times per second) until, for example, the display control system1stops.

It should be noted that the embodiments are not limited by the manner in which the processing units of the flowchart are divided or the names of the processing units. In addition, the order of processes illustrated in the flowchart is not limited to the illustrated order.

As illustrated inFIG.10, the processor100acquires the eye-point coordinates PV of the occupant2detected by the DMS30(Step S101).

Thus, in Step S101, the processor100operates as an eye-point information acquisition unit100A to acquire the eye-point coordinates PV of the occupant2(an example of information on the eye positions of the occupant).

The processor100acquires the vehicle's exterior image P40captured by the exterior camera40(Step S102).

Thus, in Step S102, the processor100operates as an image acquisition unit100B to acquire the vehicle's exterior image P40(an example of an outside-world image) showing the outside world of the vehicle (an example of a moving body).

The processor100generates the parameter PMT in accordance with the eye-point coordinates PV acquired in Step S101(Step S103).

As described above, in Step S103, the processor100operates as the parameter generation unit100C operable to generate, in accordance with the eye-point coordinates PV, the parameters PMT for generating, from the vehicle's exterior image P40, the output images PRIMGand PLIMGto be displayed on the display devices60Rand60L.

Based on the parameter PMT generated in Step S103, the processor100generates the output images PRIMGand PLIMGfrom the vehicle's exterior image P40(Step S104). The above-described pixel interpolation processing is executed in Step S104, for example.

As described above, in Step S104, the processor100operates as the image generation unit100D operable to generate the output images PRIMGand PLIMGfrom the vehicle's exterior image P40based on the parameter PMT generated by the parameter generation unit100C.

The processor100outputs the output images PRIMGand PLIMGgenerated in Step S104to the display devices60Rand60L(Step S105).

Thus, in Step S105, the processor100operates as the image output unit100E operable to output, to the display devices60Rand60Lthe output images PRIMGand PLIMG, respectively, generated by the image generating unit100D.

During the execution of the image processing illustrated inFIG.10, if the eye-point coordinate PVmoves, for example, due to the occupant2moving his/her body, the scenes outside the vehicle displayed on the screen60ARand60AL(the output image PRIMGand the output image PLIMG) change in real-time in accordance with the movement of the eye-point coordinates PV. Hence, the occupant2can experience a state where the occupant2visually recognizes the outside of the vehicle by viewing through the transparent front pillar portion.

In the image processing illustrating inFIG.10, the parameter PMT is used to generate the output images PRIMGand PLIMG. Thus, achieved is a reduced cost for the computation that is executed to perform highly accurate coordinate transformation between the coordinate systems and thus to display a natural vehicle's exterior image reproducing a real scene on a screen (non-planar surface) having a complicated shape such as the screen60AR. Hence, the processing load on the processor100is reduced greatly.

FIG.11is a block diagram illustrating a configuration of a display control system1according to Modified Example 1 of the present application. As illustrated inFIG.11, the display control system1according to Modified Example 1 includes a sensor70.

In Modified Example 1, the display devices60Rand60Lare movably installed in the vehicle cabin. For example, the display devices60Rand60Lare respectively installed on the right front pillar portion and the left front pillar portion via well-known mechanical mechanisms, thereby enabling the display devices60R and60L to perform tilting, panning, sliding, and the like motions.

For example, when the occupant2operates the HM150, the display devices60Rand60Lare driven to perform such motions as tilting, panning, sliding, and the like motions. The sensor70detects the motions of the display devices60Rand60Lrelative to the right front pillar portion and the left front pillar portion, respectively. Motions of the display devices60Rand60Lin turn, change the positions of the screen60ARand the screen60ALto be arranged in the vehicle coordinate system.

Hence, the processor100updates the screen data D1in accordance with the motions of the display devices60Rand60Ldetected by the sensor70. For example, the processor100updates the information on the positions of the display devices60Rand60Lincluded in the screen data D1. The update of the screen data D1, in turn, causes the update of the parameter PMT.

In this way, the parameter PMT is updated in accordance with the motions of the display devices60Rand60L. Hence, even with the motions of the screen60ARand the screen60AL, it is possible to display a natural vehicle's exterior image that reproduces an actual scene while reducing the computation cost.

The description provided thus far is a description of exemplary embodiments of the present invention. The embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the technical concept of the present invention. For example, appropriate combinations of embodiments and the like that are explicitly indicated by way of example in the specification or obvious embodiments and the like are also included in the embodiments of the present application.

Although a single ECU10executes various processes in the above-described embodiment, the configuration of the present invention is not limited to this. In another embodiment, a plurality of ECUs may be configured to share and execute various processes. Execution of distributed processing by a plurality of ECUs improves, for example, the processing speed.

For example, the display control system1may be configured to include: a main ECU that executes various processes; and a sub ECU that holds the screen-data database and that mediates communications between the main ECU and the display devices60Rand60L.

One sub ECU as described above may be provided for each display device. In other words, the display control system1may be configured to include: a sub ECU that mediates communications between the main ECU and the display device60Rand a different sub ECU that mediates communications between the main ECU and the display device60L.

A single sub ECU may be provided for a plurality of display devices. In other words, the display control system1may be configured to include a sub ECU that mediates communications between the main ECU and each of the display devices60Rand60L.

The image processing according to the present embodiment can be applied not only to the display devices60Rand60Lbut also to the display device60Cinstalled on the dashboard. Even when the display device60Cis, for example, a curved-screen display device, the load of the processing executed to display the output image PIMGcan be reduced from the corresponding load in the conventional cases.

The vehicle's exterior image P40is not limited to a real-time captured image, but may be, for example, an image that was captured in the past.

The screen to display the output image PIMGthereon is not limited to the curved-screen display exemplified as the screen60A, and may have a more complicated three dimensional shape. In addition, the screen to display the output image PIMGthereon may not be such a screen as an LCD screen or an organic EL display screen but may be a projection surface (for example, a surface of a structure including unevenness) that a projector can project images onto.

DESCRIPTION OF REFERENCE NUMERALS