Patent ID: 12223584

DETAILED DESCRIPTION OF EMBODIMENTS

With reference to [FIG.1], in most common ADAS installation1000in a transport vehicle, the ADAS comprises a digital image recording device1001located behind a demarcated view zone1003of the windshield1002of a vehicle. The digital image recording device1001is configured to real-time record or acquire images/videos of the landscape1005of surrounding environment of the vehicle. These images/videos are fed to the processing devices of the ADAS for object detection and classification, e.g. cars, roads, trees, people, houses . . . . They may be also directly displayed, e.g. on a screen, to the driver to help him to perform some manoeuvres.

On the windshield1002, the view zone1003through which the digital image recording device1001is recording images/videos is usually demarcated by glaze or enamel stripes that are used as ornament and/or for concealing some elements such as fixing joints and electronics.

The inclination angle between the median axis (C) of the demarcated zone1003and the optical axis (B) of the digital image recording device1001may be equal to or may differ from the inclination angle between the median axis (C) of the demarcated zone1003and the median axis (A) of the vehicle chassis, which generally is horizontal. In most transport vehicles, the windshield1002is not vertical but inclined at a certain angle which is the installation angle of the windshield1002on the vehicle frame. In other words, with respect to the digital image recording device1001, the windshield1002is inclined at an angle which corresponds to its condition of use.

In this configuration, the quality of the recorded images or/videos may partly depend on the optical quality of the demarcated view zone1003of the windshield1002. In particular, the windshield1002must be devoid of optical distortions. However, as efficient as they could be, windshields still remain optical systems in themselves with their own optical properties and limitations, and their effects on the recorded images/videos remain to be evaluated for each of them.

Furthermore, as a windshield1002is often inclined with respect to the digital image recording device1001, the distance between the surface of its demarcated view zone1003and the lens of the digital image recording device1001, in one hand, and the length travelled by light ray inside the windshield1002itself, in other hand, vary vertically, i.e. along the (C) axis. Because the optical path length is not constant along this axis, further optical distortions, and/or deflections, may occur which have also to be considered while evaluating the effects of windshields on the image/video quality.

With reference to [FIG.1] to3, in a first aspect of the disclosure, there is provided a computer implemented method3000for generating a map of point spread function convolutional kernels PSF-CK which simulates the effects of optical distortions of a windshield on the image recording quality of a digital image recording device,wherein said method3000takes as input a measured optical quality function13001related to the optical distortions of the windshield1002,wherein said method3000provides as output O3001a map of point spread function convolutional kernels PSF-CK simulating the effects of optical distortions of a windshield on the image recording quality of a digital image recording device,wherein said method3000comprises the following steps:(a) modelling S3001at least one sheet2002of transparent mineral glass comprising two main parallel faces2002a,2002b, wherein the surface of at least one of said two main faces2002a,2002bis textured with the measured quality function13001, and wherein the sheet2002of mineral glass is placed in front of a grid2001of point sources2001(1)-2001(n) and is inclined, in respect to said grid2001, with an installation angle (AÔC) of said windshield1002in a transporting vehicle;(b) calculating S3002, with a stochastic ray tracing method, the global illuminance GI arriving through the modelled inclined sheet2002of transparent mineral glass from the grid2001used as a light source2003;(c) computing S3003the projection O3001of the global illuminance GI for each point source2001(1)-2001(n) of the grid2001from the view frustum2004fof a virtual camera2004with an optical camera model OCM, wherein said virtual camera2004is placed in front of the opposite main face2002bof the sheet2002of transparent mineral glass with respect to the grid2001used as a light source2003and at a position corresponding to an installation position of the digital image recording device1001, wherein the so projected grid of point sources is a map of point spread function convolutional kernels, wherein each point spread function convolutional kernel is associated to a point source of the grid2001.

In the field of ray tracing, ‘global illumination’ is a well-defined expression. It encompasses all kinds of illumination, whether direct or indirect, coming from a light source, i.e. the image2001in the present invention. Direct light is the light coming directly form the light source and indirect light is the reflected, refracted and/or diffused light from surfaces and/or volumes in the scene.

The frustum2004fof the virtual camera2004corresponds to the field of view of said camera2004in the ray tracing scene. It is usually represented as a kind of pyramid of vision which is considered representative of the field of view of a real camera. Concretely, it constitutes the region of the ray tracing scene which may appear on the screen. ‘Frustum’ is a well-defined term in the art.

In the context of the disclosure, optical distortions are to be understood as encompassing all the optical aberrations of the windshield which may affect the optical path of light so that the magnification varies across the field of view of said windshield at a given working distance. In other words, optical distortions are all aberrations which induce deviation from rectilinear projection; projection in which straight lines remain straight across the field of view. At some extent, optical distortions may also comprise blur as long as it comes from the optical particularities of the windshield itself. Optical distortions can come from variations in refractive index, in surface roughness/profile, or in thickness of the windshield. They may also occur from defects.

As already mentioned, the measured optical quality function13001of the windshield1002may come from real-time measurements performed on production line of windshields. In this context, it may be the measured transmitted wavefront error of the windshield, the measured surfaces profiles and/or the measured distribution of complex refractive index.

In a preferred embodiment, the optical quality function13001related to the optical distortions is the measured transmitted wavefront error, whose measurement can be relatively easily implemented on production lines of windshields since it can be performed quickly with acquisition rates which are compatible with throughputs of most production lines.

At step (a), at least one of the faces of the modelled sheet2002of transparent mineral glass is textured with the measured quality function13001. The texture of the textured surface may be modelled in different ways. For instance, it may be modelled with a bump map or a displacement map which is representative of the measured optical quality function.

At step (b), the global illuminance GI is calculated with a stochastic ray tracing method. Compared to conventional, i.e. non-stochastic, ray tracing methods which rely on a given number of light rays to be drawn for each pixel, stochastic ray tracing methods allow to focus calculation on light rays reaching the virtual camera and not to those light rays going nowhere. They thus need less computer resource and computing time. In the context of the disclosure, besides these advantages, using a stochastic ray tracing method may also allow to get the same level of accuracy as with conventional ray tracing methods. Monte Carlo ray tracing methods, in particular Metropolis Light Transport ray tracing methods, provide valuable results.

Different stochastic ray methods are available in the art. Any of them may be used at step (b) providing it is adapted to calculate the global illuminance arriving through the modelled inclined sheet of mineral glass. In particular, a stochastic path tracing method may be used, as it allows to get a high level of precision and sharpness.

Several physically based rendering engines are available in the art to implement stochastic ray tracing in step (b). Examples of engines are LightTools, Indigo renderer or LuxCoreRender.

According to the disclosure, the 3D projection O3001of the global illuminance GI is performed with an optical camera model OCM. Any adapted optical camera model OCM, either realistic or simulated, may be used.

In particular, the optical camera model OCM may be obtained by a real modelling of each interface of the different optics of the camera lens (optical design). For instance, the optical camera model (OCM) may be modelled as a combination of a sensor and an optical system.

An example of optical system may be an objective lens system as described in U.S. Pat. No. 3,451,745 A, and an example of sensor may be a digital image sensor.

In one preferred example embodiment, the optical camera model OCM may be represented as a recording sensor Nx,Ny pixels in size and optical system, wherein the half size Sx, Sy, Sz of the grid3001of point source may be calculated with the following formulas

{Sx=D×tan⁢(H×π180)⁢wherein⁢H=H⁢F⁢O⁢V2⁢and⁢⁢V=NxNy×H,whereinSy=D×tan⁢(V×π180)Sz=D
HFOV is the horizontal view frustum2004fof a virtual camera2004and D is the distance between the sensor and the grid of point sources.

As illustrative example embodiments, the recording sensor may have a 1280×960 px in size and the pixel may be one pixel for 3.75 μm.

In these illustrative example embodiments, the optical system may be a modelled objective lens system as described in U.S. Pat. No. 3,451,745 A with a focal length of 15.5 mm, an aperture size of 0.8 mm, a horizontal field of view of 21.7 mm and a diameter size for the lens of 50 mm.

In certain embodiments, the grid3001may comprise a point source for each pixel of the sensor or one same point source for a given number of pixels of the sensor.

On [FIG.2], the grid2001is at a finite or fixed distance from the virtual camera2004or the modelled sheet2002, and the map of point spread function convolutional kernels may be only computed for this given distance. In particular, the grid2001may be in the view frustum2004fof said virtual camera2004.

Alternatively, in certain embodiments, the steps of the method3000may be reiterated for grid of point sources located at different distances from the virtual camera2004in order to form a 3D map of point source function convolutional kernels for a given region of the view frustum2004fof said camera2004. These embodiments allow to construct a map of point source function convolutional kernels for each distance so that to provide a 3D matrix of point source function convolutional kernels.

In these embodiments, the so obtained 3D matrix may be used to model more realistic configurations in respect to what a digital image recording device from an ADAS may acquire, or view, from a real landscape. In particular, distance-dependent effects can be taken into account.

According to the disclosure, in step (c), the projected grid of point sources is a map of point spread function convolutional kernels, and each point spread function convolutional kernel is associated to a point source of the grid2001. The number of point sources2001(1)-2001(n) of the grid2001thus determines the number of point spread function convolutional kernels in the map of point spread function convolutional kernels.

The areas between the point sources2001(1)-2001(n) of the grid2001, i.e. the areas which do not contain point sources, are not projected and do not provide information regarding the value of the point spread function kernels in the corresponding areas in the projected grid of point sources.

The number of point sources of the grid may then be increased or decreased depending on the desired image for the map of point spread function convolutional kernels. The higher the number of point sources2001(1)-2001(n) in the grid2001, the higher the number of projected point sources in the projected grid and the higher the number of point spread function convolutional kernels in the map of point spread function convolutional kernels, and inversely.

For instance, when a high resolution is desired, the number of point sources2001(1)-2001(n) in the grid2001may be increased so that the number of corresponding point spread function convolutional kernels is sufficient to be considered as representative of a high resolution image, the number of point sources2001(1)-2001(n) in the grid2001then being a matter of choice.

However, one drawback of such approach is that, depending on the desired image resolution, the number of light rays to compute the global illuminance GI at step (b) may rapidly increase the workload on the computing facilities and more computing resources may be needed.

Furthermore, increasing the number of light rays to compute may lead to unwanted overlaps in the projected grid, i.e. same area is projected twice or even more, and to unnecessary waste computing resources. This is now a matter of concern as reducing our energy consumption and our global ecologic footprint are nowadays mandatory.

In that respect, in advantageous embodiments, the method3001may further comprise, after step (c), a step of interpolating point source function convolutional kernels for additional non projected point sources located between source points2001(1)-2001(n) of the grid2001from the point spread function convolutional kernels associated to neighbour projected point sources of said non projected point sources.

Interpolating the point spread function convolutional kernels for additional non projected point sources may be an efficient way to increase the resolution of the map point spread function convolutional kernels while alleviating the computing workload and reducing the computing time.

In further advantageous embodiments, the interpolating step is performed from only selected parameters of the point spread function convolutional kernels associated to neighbour projected point sources.

As illustrative example embodiments, the parameters which may be interpolated may be lateral spatial shifts, the maximum intensity, the area and/or diameter of the neighbour point spread function convolutional kernels, or a combination thereof. As further example embodiments, the parameters may also be the eccentricity, the major and minor axis length, the moments of the neighbour point spread function convolutional kernels, or a combination thereof.

While computing S3003the projection O3001of the global illuminance GI for each point source2001(1)-2001(n) of the grid2001at step (c), the projected grid may be laterally shifted comparing to the original grid2001.

In this respect, in advantageous embodiment, the method (3000) further comprises a step of computing a map of position shifts, wherein each position shifts of said map is the position shift between a point source of the grid and the corresponding point spread function convolutional kernel, wherein the map of point spread function convolutional kernels comprises said map of position shifts as output.

In certain embodiments, with reference to [FIG.4], the grid2001may be digitally preprocessed as an environment map3001projected onto the inside surface of an environment sphere centered on the virtual camera2004.

In the context of the disclosure, an environment map is a map projected on the scene coordinate system in order to make it the background of the scene in a ray tracing method. The environment map converts pixel locations in an image into angles of incidence. This makes any object to appear at an infinite location.

As an environment map, the grid2001may be projected on all or part of the inside surface of the sphere with spherical mapping technique, so that the image2001used as light source2003corresponds to an infinitely distant illumination which is afterwards refracted, reflected and/or diffused on the modelled sheet of transparent mineral glass. Main advantage may be that the light rays coming from each pixel of the image2001used as a light source2003form a rectilinear and parallel beam from an angular direction, as if the image was located at infinite distance. The computing workload may be alleviated and the computing time may be reduced without prejudicing the accuracy of the output image O3001.

Placing the image2001of the landscape1005at finite or infinite distance in the scene of ray tracing is a matter of choice and will depend on the specifications of both the digital image recording device and the ADAS.

At step (a), the modelled sheet2002of transparent mineral glass may represent the whole surface of the windshield1002, and the surface of the at least one of two main faces2002a,2002bof said modelled sheet2002may be textured with the measured quality function13001of the whole surface of said windshield1002.

In certain embodiments, in step (a), the sheet2002of transparent mineral glass is modelled so that said sheet2002of transparent mineral glass represents only the demarcated zone1003of the windshield1002in front of which a digital image recording device1001may be placed. As the demarcated zone1003is the region of interest for the recording of image through the windshield1002by the digital image recording device1001. Therefore, it may be advantageous regarding computing time and speed of simulation to model only this demarcated zone1003.

Alternatively, in other embodiments, in step (a), the sheet2002of mineral glass is modelled so that the global illuminance GI calculated at step (b) is only the global luminance GI arriving through the part of modelled sheet2002of mineral glass which corresponds to a demarcated zone1003of the windshield1002through which a digital image recording device1001may record an image.

As an illustrative example, in step (a), the modelled sheet2002of transparent mineral glass may be modelled to represent the whole surface of the windshield1002, and the surface of at least one of two main faces2002a,2002bof said modelled sheet2002may be textured with the measured optical13001quality function of the whole surface of said windshield1002. The portion of the surface of the modelled sheet2002which does not correspond to the demarcated zone1001may then be blackened in order to prevent part of the light coming from the light source from going through that portion. The global illuminance GI calculated at step (c) is then global luminance GI arriving through the non-blackened portion of the surface which corresponds to the demarcated zone1003.

In certain embodiments, in step (a), a second sheet of transparent mineral glass and a polymeric interlayer may be modelled so that the first and second sheets of transparent mineral glass are assembled to each other by the polymeric interlayer, for instance a PVB interlayer, to form a laminated glazing, such as a windshield. The optical quality function13001of the windshield1002may then be the measured optical quality function of the windshield or a calculated combination of measured optical quality functions of each sheet of transparent mineral glass and the polymeric interlayer.

With reference to [FIG.5], in a second aspect of the disclosure, there is provided a computer implemented method5000for simulating the effects of the optical distortions of a windshield1002on the image recording quality of a digital image recording device1001,wherein said method5000takes as input an image15001of a landscape likely to be recorded by a digital image recording device1001through said windshield1002, and a map15002of point spread functions convolutional kernels which is generated with a computer implemented method according to any embodiments of the first aspect of the disclosure for said windshield1002, andwherein said method5000provides as output an image O5001of said input image15001of a landscape as viewed through said windshield1002,wherein said method5000comprises a step S5001of computing a convolution operation of the image15001of a landscape with the map15002of point spread functions convolutional kernels.

The convolutional operation may be implemented with any adapted computing methods, in particular those that are well-known in the art.

Object detection and classification algorithms implemented in data processing systems of ADAS are often sensitive to colours, i.e. light wavelengths, and may perform better in colour filtered, for instance red or green filtered image. On the other hand, colour filtering may help to reduce image artefacts or anomalies, in particular for night-time images.

Furthermore, digital image recording device may sometimes comprise colour filter array on some pixels. On this scope, the image2001may advantageously undergo some prior image processing in order to improve the image quality and/or extract wavelength specific information from said image and/or to render the effect of colour filter arrays.

In certain advantageous embodiments, before the convolution operation, the image15001of landscape may be digitally preprocessed, first with a colour filter array, for instance a clear-red-clear-clear or a clear-clear-red-clear filter array, and/or, second, with a demosaicing algorithm, in particular with a nearest-neighbour interpolation kernel.

The first and second aspects of the disclosure are computer implemented. Accordingly, with reference to [FIG.6], in a third aspect of the disclosure, there is provided a data processing system6000comprising means for carrying out a method according to any embodiments of the first aspect of the disclosure and/or a method according to any embodiments of the second aspect of the disclosure.

Example of means for carrying out the method is a device6001which can be instructed to carry out sequences of arithmetic or logical operations automatically to perform tasks or actions. Such device, also called computer, may comprise one or more Central Processing Unit (CPU) and at least a controller device that are adapted to perform those operations. It may further comprise other electronic components like input/output interfaces6003, non-volatile or volatile storages devices6003, and buses that are communication systems for the data transfer between components inside a computer, or between computers. One of the input/output devices may be user interface for human-machine interaction, for example graphical user interface to display human understandable information.

As image processing and ray tracing may often require high computational power to process large amounts of data, the data processing system6000may advantageously comprise one or more Graphical Processing Units (GPU) whose parallel structure makes them more efficient than CPU, in particular for image processing in ray tracing.

Nevertheless, in the context of the disclosure, it is worth to mention that one objective is to provide a method which is able to simulate accurately the effect of optical distortion of windshield without requiring high computing and data storage resources for, in particular, real-time applications such as in on-board systems in which the effects of the distortions of a windshield may have to be simulated on-the-fly. Accordingly, this may be achieved by the method according to the second aspect of the disclosure as once the point spread functions map has been computed, said method requires low computing resources and may be executed on low power computer board such as single-board computer.

A fourth aspect of the disclosure, there is provided a computer program15001comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to any embodiments of the first aspect of the disclosure and/or a method according to any embodiments of the second aspect of the disclosure.

Any kind of programming language, either compiled or interpreted, can be used to implement the steps of the method of the invention. The computer program can be part of a software solution, i.e. part of a collection of executable instructions, code, scripts or the like and/or databases.

In certain embodiment, the computer program may be stored is on a computer-readable medium6003. Accordingly, such computer-readable medium6002may comprise instructions which, when executed by a computer, cause the computer to carry out the method according to any of the embodiments described herein.

The computer-readable storage6002may be preferably a non-volatile storage or memory, for example hard disk drive or solid-state drive. The computer-readable storage can be removable storage media or a non-removable storage media as part of a computer.

Alternatively, the computer-readable storage6002may be a volatile memory inside a removable media. This can ease the deployment of the invention into many production sites.

The invention according to the first and second aspects of the disclosure is well adapted to be used in a process for evaluating the optical quality of windshields for a use with a digital image recording device, in particular with a digital image recording device of an automated driving and advanced safety system.

The effects of the optical distortions may then be simulated for each windshield of a batch of windshields from their respective measured quality function and a reference image of landscape or a set of reference images. Once the effects of the optical distortions are simulated, the windshields may be compared, and those less altering the image quality regarding required specifications may be selected for use with a digital image recording device.

Alternatively, some criteria may be defined on the simulated images or on the output of ADAS algorithms fed with the simulated images in order to reject non-compliant windshields.

The invention according to the first and second aspects of the disclosure may be as well adapted to be in a process for calibrating digital image recording device of an automated driving and advanced safety system.

For instance, the effects of optical distortions of a reference windshield or a windshield from a set of reference windshields, i.e. windshields whose optical quality is considered to fulfil requirements for a use with a digital image recording device, may be simulated from a set of images of landscapes. The output images may then be used to calibrate and/or select the relevant features of a digital image recording device.

In this context, in a fifth aspect of the invention, with reference to [FIG.7], there is provided a process7000for evaluating the performances of an object detection and classification algorithm of an automated driving and advanced safety system, wherein said process7000comprises the following steps:(a) providing S7001a set17001of measured optical quality functions related to the optical distortions of a set of windshields1002;(b) providing S7002a set17002of images2001of landscapes1005likely to be recorded by a digital image recording device1001through the windshield1002of said set of windshields1002;(c) using S7003a computer implemented method5000according to the second aspect of the disclosure for each image2001of the set of images2001of landscapes1005and each measured optical quality function13001of the set measured optical quality functions13001related to the optical distortions, in order to provide a set O7001of images2001of a landscape as viewed through the windshield1002of said set of windshields1002;(d) feeding S7004the set of images O7001obtained at step (c) to an object detection and classification algorithm of an automated driving and advanced safety system;(e) monitoring S7005performance parameters O7002in the object detection and classification of said algorithm while the algorithm is processing the set of images fed at step (d).