Method for harmonizing images acquired from non overlapping camera views

An image processing method for harmonizing images acquired by a first camera and a second camera connected to a vehicle and arranged in such a way as their fields of view cover a same road space at different times as the vehicle travels along a travel direction is disclosed. The method includes: acquiring by a selected camera, a first image at a first time; selecting a first region of interest bounding a road portion from the first image; sampling the first region of interest; acquiring by the other camera, a second image in such a way that the road portion is included in a second region of interest; sampling the second region of interest; and determining one or more correction parameters for harmonizing images acquired by the first and second cameras, based on a comparison between the image content of the first and second regions of interest.

FIELD

The present invention relates to a method for harmonizing images acquired by two or more cameras connected to a vehicle and having fields of view that do not overlap.

BACKGROUND

It is known for vehicles to be provided with multi-camera automotive vision systems, including a number of cameras disposed at the front, rear and on the left and right side mirrors of the vehicle for capturing images of the environment surrounding the vehicle.

Images from these cameras are typically relayed to an electronic control unit, ECU, comprising a processor which, among various tasks, processes the images before providing one or more processed images to a display or windscreen located within a cabin of the vehicle, to provide assistance to the driver of the vehicle.

Different types of views can be generated by the vehicle ECU by combining the input images received from the multiple cameras, before being displayed to the driver or used for other applications, such automatic or semi-automatic vehicle operations. In particular, regions of interest from the input camera images can be first remapped to a targeted viewport and then merged, thus generating a mosaic image that represents a view from a particular selected 3D point in the environment surrounding the vehicle. For example, a virtual camera can be placed above the vehicle, looking from a top central position, and input camera textures corresponding to regions of interest of the camera images are projected to a surface viewport that corresponds to a flat 2D plane, and merged to generate a mosaic view showing an upper side of the vehicles as well as the environment surrounding the vehicle, that is usually referred to as a top-view (or bird eye view). Other merged views can be generated using the rendering capabilities of the vehicle ECU, such as multi dimensional views (e.g., 3D Bowl views, where a bowl shaped projective 2D is used instead of a flat 2D plane).

Each vehicle camera has its own lens, image sensor and, in many implementations, an independent image signal processing chain (therefore, the vehicle cameras can have a different exposure or gain control, a different white balance or the like). Furthermore, each vehicle camera is facing in different directions and is seeing different areas of the vehicle environment.

As such, brightness and colour hues can be slightly different for the images acquired by each vehicle camera, due the limitations (lens, image sensor, etc) and different orientations of the vehicle cameras. These slight differences in colour and brightness negatively affect the visual quality of displayed merged views, thus jeopardizing the driver's illusion of a view corresponding to a virtual camera in a 3D point surrounding the vehicle.

In order to improve the visual quality of merged views, brightness and colour harmonization is generally applied. In particular, harmonization between two vehicle cameras having overlapping field of views is achieved using a common ground region captured by the camera as a reference for harmonization. For instance, harmonization between the front and right side mirror cameras can be achieved using a corner road region, where the fields of view of these cameras overlap.

For example, WO2018/087348 (Ref: SIE0426) discloses a method for harmonizing brightness and colour of a composite image of the environment surrounding a vehicle, using histograms representing the luminance Y and chromatic values U, V of the merged input images.

It is further known for vehicles to be provided with hitches allowing them to tow a trailer. It will be appreciated that trailers (as well objects transported by the trailer, where applicable) lead to large blind-spots for a driver as they are not able to see most areas around the trailer and in particular, it can be difficult for unskilled drivers to attempt to reverse a vehicle with a trailer without assistance. It is therefore known for such trailers to also incorporate a rear camera directed rearwardly of the trailer (as well as in some cases trailer side cameras pointing outwardly from respective sides of the trailer). Images from these cameras can be acquired by a controller within the trailer and provided to the vehicle ECU to produce an enhanced display to assist a driver. In some cases, any trailer camera can be connected directly to the vehicle ECU.

For example, the vehicle ECU can generate an invisible trailer view by merging images acquired by the vehicle rear camera and the trailer rear camera. In particular, the invisible trailer view is built by remapping regions of interest of the images acquired by the rear cameras to a target viewport (Invisible Trailer viewport), and merging the remapped regions of interest. In this way, a rear view is provided to the driver, via the vehicle display or windscreen, where the trailer becomes virtually invisible. For example, WO2021/032434 (Ref: 2019PF00307) discloses generating a first camera image from a rear camera of a vehicle and a second camera image from a rear camera of a trailer hitched to the vehicle. An invisible trailer view is generated by superimposing these camera images, such that the second camera image covers a subsection of the first camera image depending on a hitch angle between the vehicle and the trailer.

Other applications can require a merging of the images acquired by the rear cameras of the vehicle and hitched trailer. For example, DE102019133948 (Ref: V25-2103-19DE) discloses the use of the multiple cameras of a vehicle and trailer combination to construct a 3D view of the environment surrounding the vehicle and trailer combination, for display to a driver.

In the merged views generated for assisting a driver of a vehicle combined with a trailer, brightness and/or colour disparities can be visible to the driver in the merged areas of the images acquired by the rear cameras of the vehicle and hitched trailer. However, these cameras are positioned such that their fields of view do not overlap in such a way as to cover a common ground portion. Thus, there is no reliable common reference for harmonizing the rear cameras of the vehicle and hitched trailer.

Similarly, there is no common ground available as a reliable reference for harmonizing images acquired by the rear and front cameras of a vehicle (with or without trailer).

SUMMARY

According to the present invention there is provided a method according to claim1, for harmonizing images acquired by a first camera and a second camera connected to a vehicle, having fields of view that do not overlap at a same time.

Embodiments of the invention are based on the realization that, although the first and second cameras can not capture a same road portion at a given time, the first and second camera can capture a same road portion at different times as the vehicle proceeds along a travel direction and this can be advantageously used as reliable reference for harmonizing a view including merged images captured by the first a second cameras. For example, the harmonized view can be an invisible trailer view, in embodiments where the first and second cameras are a rear camera of a vehicle and a rear camera of a trailer towed by the vehicle. In other embodiments, the first and second cameras can be the front and rear cameras of a vehicle.

In more detail, embodiments of the invention involve sampling at least one first region of interest, ROI, from a first image acquired, at a first time, by one of the first and second cameras selected based on a determined direction of the vehicle. The first ROI is defined in the first image to include a reference road portion within the captured scene. A second ROI is sampled from a second image acquired by the other camera at a second time such that, according to a monitored travelled distance of the vehicle after the first time, the second ROI can also include the reference road portion. One or more correction parameters for harmonizing images acquired by the first and second cameras are then determined based on a comparison between the image data within the sampled ROIs.

In some embodiments, the image data within the first and second ROIs is compared after conversion into a YUV format. In these embodiments, a difference between luma values Y estimated for the first and second ROIs is compared to a threshold with the purpose of determining whether these ROIs actually include a same road reference portion, based on the realization that if the ROIs include different objects within the imaged scene (e.g. because an object has moved onto or over or left the reference road portion before the acquisition time of the second image), the difference between the estimated luma values Y is significant.

Further aspects of the invention include an automotive multi-camera vision system, a combination of a vehicle and trailer or a vehicle, and computer program product configured to execute the method according to the invention.

DESCRIPTION OF THE EMBODIMENTS

Referring now toFIG.1, there is shown a combination of a car11with a trailer12, including a multi-camera vision system configured to perform an image harmonization method according to an embodiment of the present invention.

The car11is provided with a hitch13allowing the car11to tow objects, such as the trailer12illustrated inFIG.1. In particular, the trailer12is hooked to the hitch13via a drawbar14, in such a way that a hitch angle can be present between the car10and the trailer12(when the car10is towing the trailer12moving along a curved trajectory).

It is to be noted that the trailer12illustrated inFIG.1is just a non-limitative example of several types of trailers that can towed by the car10—other examples include a caravan or a horsebox. As such, in the present application any object towed by the car10is referred to as a trailer. As such, in cases where a trailer is used as a platform for transporting an object that occludes the rear view of the driver (e.g., a boat trailer carrying a boat, or a bike trailer carrying bikes), for the purposes of the present application the transported object is considered as part of the towed trailer. It is to be noted the car11illustrated inFIG.1is also just a non-limitative example of several types of vehicles that can tow a trailer (such as a truck or tractor). Thus, the connecting means for connecting the trailer to the vehicle can be different than the hitch13and drawbar14illustratedFIG.1.

The multi-camera vision system includes a plurality of cameras disposed at the front (FV camera), rear (RV camera) and on the left and right side mirrors (ML and MR cameras) of the vehicle for capturing images of the environment surrounding the vehicle. The side cameras need not necessarily be located on the mirrors, and these can be located at any location suitable for acquiring an image from the environment to the sides of a vehicle.

The system further includes a trailer rear camera (TR camera) directed rearwardly of the trailer12(and in some cases, can also include side cameras pointing outwardly from respective sides of the trailer12). As such, as illustrated inFIG.1, the fields of view FOV1and FOV2of the vehicle camera RV and trailer camera TR do not overlap, at a given time, on a common portion of a road18along which the car11is travelling. This does not exclude that FOV1, FOV2can overlap at some regions of the scene, but these common regions are not uniform and variable and therefore, are not reliable for image harmonization of the cameras RV, TR.

The system further comprises a vehicle ECU15running an application configured to receive images acquired by the vehicle cameras FV, RV, MR, ML, and a controller16within the trailer12that is configured to collect the images acquired by the trailer camera TR (as well by the trailer side cameras, if present). The images collected by the trailer controller16are streamed or otherwise provided, through either a wired or wireless connection, to the vehicle ECU15. In some cases, any trailer camera can be connected directly to the vehicle ECU15.

A processor of the vehicle ECU15is configured to process the images received from the vehicle and trailer cameras FV, RV, MR, ML, TR with the purpose of providing processed images to a display22or windscreen located within a cabin of the car11. Such camera information can also be processed by the ECU15to perform autonomous or semi-autonomous driving, parking or braking of the vehicle as well as for example, storing streams of images captured by one or more of the cameras as dashcam or security footage for later retrieval.

The ECU15(or another processing unit within the car11) can also estimate a distance travelled by the car11over time, by processing sensor data provided by odometry sensors (schematically represented and cumulatively indicated with numeral reference17inFIG.1). For example, the odometry sensors17can include sensors that measure the space travelled by some of the wheels and steering angle. In addition or as an alternative to using the sensor data provided by the sensors17, the change in position of the car11over time can be estimated using GPS tracking.

With reference now toFIG.2, an image harmonization method100operable within the system is disclosed.

At method step101, the travel direction of the car11is determined, by using for example the odometry sensors17and/or GPS location tracking information.

With reference toFIG.3A, a case is first considered where, at the determination of step101, the car11is moving forward along a substantially straight direction indicated by the longitudinal axis20. InFIG.3A, the field of view FOV1of the RV camera covers a first road portion50and a second road portion51within the gap between the car11and the trailer12, beside respectively a first side500and a second side510of the trailer drawbar14. As such, these road portions50,51are not occluded by the trailer12or drawbar14and therefore, can be captured by the RV camera and sampled as reliable reference road portions for the purpose of harmonizing the cameras FV, RV (as it will be disclosed in more detail). It is to be noted that in this application, a reference road portion encompasses not only paved road portion, but any portion of ground where the vehicle can have a uniform colour and texture (e.g. a portion of a highway, street, country road, pitch).

Responsive to a determination that the car11is travelling along the forward direction illustrated inFIG.3A, the RV camera is selected to acquire a first image at a first time, t1, (step102), corresponding to the position of the car11illustrated inFIG.3A.FIG.4Aillustrates the image200acquired by the RV camera at t1.

Then, two ROIs201,202are selected to be sampled from the image200(step103). In particular, the ROIs201,202are positioned and dimensioned within the acquired image200in such a way as to correspond to the road portions50and51, respectively, beside the drawbar14.

One exemplary method to select the two ROIs201,202is now disclosed.

When the vehicle ECU15receives the image200acquired at t1, the ECU15is configured to check two ROIs201,202where road portions beside the sides500,510of the drawbar14are expected to be included, assuming that the trailer12is substantially aligned to the car11along a longitudinal axis.

For example, the ECU15is configured to define these ROIs201,202by knowing an image area occupied by the trailer12and drawbar14, when the trailer12is substantially aligned to the car11. In one implementation, the ECU15can learn this area by detecting the trailer12and drawbar14within a set of images acquired by the RV camera and including the trailer12aligned with the car11. This provides for a high degree of accuracy of ROI (position and size), however, it will be appreciated that this approach adds complexity in terms of implementation. Alternatively, the ECU15can estimate this area by knowing dimensional parameters of the vehicle11and drawbar14(e.g., at least the width of the vehicle11and the length of the drawbar14). This information can be provided to the ECU15in various ways, including: receiving this information from a user's input, receiving a scan of the trailer12and drawbar14, or obtaining vehicle CAD data possibly through a network connection. In any case, a default ROI position can be determined based on a known position of the camera RV on the vehicle11from the vehicle CAD, as well as a known width for the vehicle11(which can also be obtained from the vehicle CAD). This in turn indicates a shortest length for a suitable drawbar—these are supposed to be at least half as long as the vehicle width. This allows a default position for the ROIs to be determined with minimum user input and processing power required.

Then, the ECU15determines whether any of the checked ROIs201,202includes a portion of the drawbar14or the trailer12(due to steering the car11at the image acquisition time t1). In an embodiment, the ECU15applies image detection on the ROIs201,202to detect whether any of these ROIs201,202contains a portion of the drawbar14or the trailer12. In another embodiment, the ECU15uses odometry data provided by the sensors17and/or GPS location information to measure a steering angle of the car11at the image acquisition time t1, and compare the measured angle with a threshold. Responsive to a determination that the measured steering angle has a value below the threshold (including a null value), the ECU15determines that none of the ROIs201,202contains a portion of the drawbar14or the trailer12. In addition or as an alternative, a similar determination can be performed by the ECU15using a measured hitch angle between the longitudinal axes of the car11and the trailer12. This angle can be detected in any number of ways, for example using image information from the acquired image200to detect a rotation of the trailer12around a vertical axis passing through the hitch14. Equally, image information from the vehicle mirror cameras ML, MR can detect features from the surface of the trailer moving laterally within their respective fields of view to estimate the relative angle of the vehicle and trailer. Other techniques for determining the relative angle of the vehicle and trailer include using information from rear facing ultrasonic or radar sensors mounted to the rear of the vehicle11(where changing differences measured by the sensors signal changes in the relative angle of the car11and trailer12).

With reference back to the image200illustrated inFIG.4A, the ECU15determines that none of the checked ROIs201,202contains a portion of the drawbar14or the trailer12. Responsive to this determination, the ROIs201,202are selected to be sampled from the image200(step103).

It is to be further noted fromFIG.4Athat the selected ROIs201,202are defined to correspond to road portions50,51that are separated from the respective sides500,510of the drawbar14in such a way as to be minimally if at all affected by the shadow projected by the drawbar14(and by the trailer12), at any time of the day and lighting condition.

With reference back toFIG.3A, it is to be further appreciated that if at t1the car11is steering to the right, instead of moving forward along a straight trajectory, the steering angle can be such that only the road portion50beside the left side500of the drawbar14is viewable within an image acquired by the RV camera at t1. Similarly, if at t1the car11is steering to the left, the steering angle can be such that only the road portion51beside the right side510of the drawbar14is viewable within an image acquired by the RV camera at t1.

In these cases, the method step103includes selecting only one of the ROIs201,202, corresponding to the road portion50,51that can be captured by the RV camera according to the steering direction.

In other embodiments, when the ECU15receives an image acquired by the RV camera at t1, the ECU15can perform detection of the trailer12and drawbar14to determine the image area occupied by the trailer12and drawbar14, and select accordingly one or more ROIs201,202around the detected area that can include respective road portions50,51beside the sides500,510of the drawbar14. In some other embodiments, the selection of the ROIs can be based on a detection of road portions within the captured scene, e.g., by using a texture-oriented method or by evaluating the pixel intensity.

Furthermore, although the above disclosed embodiments are based on sampling road portions50,51beside the drawbar14from the image acquired by the RV camera at acquisition time t1, it will be appreciated that, in addition or as an alternative, also road portions viewable within the field of view of the FOV1of the RV camera beside the trailer12can be sampled as references for image harmonization. In this case, the trailer's shadow projection on the road18is to be considered in the selection of the ROIs (as the trailer's shadow projection can cover one of the surrounding road portions depending on the orientation of the sun, as can be seen inFIG.4A).

The description of method100now continues referring back to the case where the two ROIs201,202are selected at method step103to be sampled from the image200illustrated inFIG.4A. Nevertheless, the following disclosure applies also to the case where, at method step103, only one ROI201,202is selected (due to steering).

The selected ROIs201,202are sampled from the image200(step104) and the respective image data stored within a memory of the system or other storage means accessible by the system (e.g. a database or server that can be accessed by the system via network connection).

Then, at method step105, a distance travelled by the car11after the acquisition time t1of image200is monitored to determine a second t2to acquire a second image by the TR camera of the trailer12, in such a way that the same road portions50,51corresponding to the ROIs201,202sampled from the image200(acquired by the RV camera of the car11) can be included in corresponding ROIs defined in the second image. The travelled distance can be monitored using the odometry data provided by the sensors17and/or GPS tracking information.

For example,FIG.3Billustrates that the car11has moved further along to the forward direction from the position illustrated inFIG.3A, covering a distance dx such that each of the road portions50,51is viewable within the field of view FOV2of the TR camera (if not occluded by an object moving into the scene during the time for travelling distance dx).

A time t2is determined, corresponding to travelled distance dx, and an image300is acquired by the TR camera at t2(step106). The acquired image300is illustrated inFIG.4B, including two ROIs301,302where the road portions50,51are expected to be included according to the travelled distance dx (since in the example ofFIG.4Bno objects are occluding the road portions50,51, the ROIs301,302actually include the road portions50,51).

With reference back toFIGS.3A-3B, it is to be noted that when the cameras RV and TR are at the same height above the road surface and at the same relative angle to the road surface and have the same projection model, the travelled distance dx can be about equal to a distance D between the RV and TR cameras (that in turn substantially corresponds to a sum of the lengths of the trailer12and the drawbar14). In this case, with reference toFIGS.4B-4A, the ROIs301,302can be defined at a position within the image300and with a pixel area that substantially correspond to the position within the image200and the pixel area of the ROIs201,202. This improves the comparability of the image data within the sampled ROIs201,202with the image data within the sampled ROIs301,302. It will therefore be appreciated that if the relative positions, heights and/or projection models of the cameras RV and TR differ, then distance dx will need to change accordingly and/or the ROIs201,202and301,302will need to be mapped to one another differently.

In any case, the determined acquisition time t2for the TR camera can correspond to a travelled distance either greater or less than dx, as long as the road portions50,51can still be viewable within the field of view FOV2of the TR camera.

After the acquisition of the image300at t2, the ROIs301,302are sampled (step107), and the respective image data stored within the memory of the system (or other storage means accessible by the system).

With reference back to the initial method step101, an operation of the method100is now disclosed in response to determining that the direction of the car11is a reverse direction.

In particular, with reference toFIG.5A, a case is considered where the car11is reversing along a substantially straight direction corresponding to the longitudinal axis30, at the determination of step101. InFIG.5A, the field of view FOV2of the TR camera covers a first road portion60and a second road portion61within the gap between the car11and the trailer12, that can become also viewable within the field of view FOV1of the RV camera beside the sides500,510of the drawbar14, as the car11proceeds in the reverse direction (as illustrated inFIG.5B). As such, the road portions60,61will be not occluded by the trailer12or the drawbar14and, therefore, can be sampled at different times by both the TR and RV cameras and used as reliable reference for image harmonization.

Responsive to the determination that the car11is travelling along the reverse direction illustrated inFIG.5A, the TR camera is selected to acquire a first image at a first time, t1, (step108), corresponding to the position of the car11illustrated inFIG.5A.FIG.6Aillustrates an image400acquired by the TR camera at t1.

Then, two ROIs401,402are selected to be sampled from the image400(step109). In particular, the ROIs401,402are positioned and dimensioned within the acquired image400in such a way as to include the road portions60and61.

One exemplary method to select the two ROIs401,402is now disclosed.

When the vehicle ECU15receives the image400acquired at t1, the ECU15is configured to check two ROIs401,402corresponding to road portions that can be viewable by the RV camera beside the sides500,510of the drawbar14, assuming that the trailer12is substantially aligned to the car11along a longitudinal axis. For example, the ECU15is configured to define these ROIs401,402by knowing the image area that is occupied by the trailer12and trailer drawbar14, when the trailer12is substantially aligned with the car11.

Then, the ECU15determines whether the car11is reversing along a substantially straight trajectory. For example, the ECU15uses the odometry data provided by the sensors17and/or GPS tracking information to measure a steering angle of the car11or a hitch angle between the car11and the trailer12, at the image acquisition time t1, and compare the measured angle with a threshold. Responsive to a determination that the measured steering angle or hitch angle has a value below the threshold (including a null value), the ECU15determines that the car11is reversing along a straight direction. Responsive to this determination, the ECU selects the two ROIs401,402to be sampled from the image400.

With reference back toFIG.5A, it is to be appreciated that if at t1the car11is reversing while steering to the right, instead of reversing along a straight trajectory, the steering angle can be such that only the road portion60can become viewable within the field of view FOV1of the RV camera, beside the side500of the drawbar14, as the car11proceeds in the reversing direction. Similarly, if at t1the car11is reversing while steering to the left, instead of reversing along a straight trajectory, the steering angle can be such that only the road portion61can become viewable within the field of view FOV1of the RV camera, beside the side510of the drawbar14, as the car11proceeds in the reversing direction.

In these cases, the method step109includes selecting only one of the ROIs401,402, corresponding to the road portion60,61that can be captured also by the RV camera according to the steering direction.

The description of method100now continues referring back to the case where two ROIs401,402are selected, at method step109, to be sampled from the image400illustrated inFIG.6A. Nevertheless, the following disclosure applies also to the case where only one ROI is selected to be sampled at method step109(because of steering).

The ROIs401,402are sampled from the image400(step110) and the respective image data stored within the memory of the system (or other storage means accessible by the system).

Then, at method step111, a distance travelled by the car11after the acquisition time t1of image400is monitored to determine a second time t2to acquire a second image by the RV camera of the car11, such that the road portions60,61corresponding to the ROIs401,402sampled from the image400(acquired by the TR camera) can be included in corresponding regions of interest defined in the second image.

For example,FIG.5Billustrates that the car11has moved further along to the reverse direction from the position illustrated inFIG.5A, covering a distance dx such that the road portions60,61are each viewable within the field of view FOV1of the RV camera (if not occluded by an object moving into the scene during the time for travelling distance dx). A time t2is determined, corresponding to travelled distance dx, and an image600is acquired by the RV camera at t2(step112).

The acquired image600is illustrated inFIG.6B, and includes two ROIs601,602beside the sides500,510of the imaged drawbar14, where the road portions60,61are expected to be included according to the travelled distance dx (since in the example illustrated inFIG.6Bno objects are occluding the road portions60,61, the ROIs601,602actually include these portions60,61).

With reference back toFIGS.5A-5B, it is to be noted that the road portions60,61are captured, at t1, by the TR camera at a close acquisition distance that is about the acquisition distance between the same road portions60,61and the RV camera, at t2. In this way, the possibility of having an object occluding one of the portions60,61at t1is reduced. Furthermore, with reference toFIGS.6A-6B, the ROIs401,402are defined at a position within the image400and with a pixel area that substantially corresponds to the position within the image600and the pixel area of the ROIs601,602.

The method100then proceeds by sampling the ROIs601,602from the image600(step113), and the respective image data are stored within the memory of the system (or other storage means accessible by the system).

A harmonization process according to the execution of the method100is now disclosed for simplicity only with reference to the ROIs201,202,301,302sampled as per the operation of steps102-107of the method100(following the determination of a forward direction at initial step101). It is to be noted that the principles of this disclosure equally apply to the operation of the harmonization process based on the ROIs401,402,601,602sampled as per the operation of steps108-113of the method100(following the determination of a reverse direction at initial step101).

The image data of the sampled ROIs201,202(extracted from the image200acquired by the RV camera at t1) and the image data of the sampled ROIs301,302(extracted from the image300acquired by the TR camera at t2) are retrieved from the memory of the system (or other storage means accessible by the system) and provided to a harmonisation network (that can be implemented by the vehicle ECU15or another processing unit of the system), where the retrieved image data are converted into a YUV format (step114) if this has not been done already.

Then, luminance components Y1and Y2are estimated from the pixel data of the ROIs201,202, as well as luminance components Y3and Y4being estimated from the pixel data of the ROIs301,302(step115). Various methods can be used to estimate Y1to Y4. For example, some techniques to estimate Y1to Y4, based on histograms generated to describe the luminance of the ROIs201,202,301,302, are disclosed in WO2018/087348 referenced above (including a non-segmentation based approach, a histogram-segmentation based approach, and a bi-modal histogram segmentation approach).

Based on the appreciation that luminance values of different imaged objects are significantly different, a difference between the estimated Y1and Y3of the ROIs201,301is compared to a threshold for the purpose of verifying whether both these ROIs201,301include the same reference road portion50(step116).

Responsive to a determination that the absolute value of Y1−Y3is below the threshold, it is assumed that this minor difference is due to a lack of brightness harmonization between the RV and TR cameras. As such, the image data within the ROIs201,301is verified to belong to the same reference road portion50.

Responsive to a determination that the absolute value of Y1−Y3exceeds the threshold, the image data within the ROIs201,301is determined to belong to different imaged objects. For example, this can correspond to the case where an object (such as another vehicle or person) moved into the road portion50between the acquisition times t1and t2of the images200,300from which the ROIs201,301are extracted. In another case, an object can cover the road portion50at t1, and move away from the portion50between the acquisition times t1-t2.

A similar verification is performed to verify whether both the two ROIs202,302include the same reference road portion51, by comparing a difference between Y2and Y4with the threshold (Step116).

Responsive to a determination that the absolute value of at least one of the differences Y1−Y3and Y2−Y4is below the threshold, such a difference is used to determine correction parameters for harmonizing the brightness of images acquired by the RV camera of the car11and the TR camera of the trailer12(step117). Various methods can be applied to determine the brightness correction parameters based on luminance difference values, such as the method disclosed in WO2018/087348. Once determined, the brightness correction parameters can be stored in the memory of the system (or any other storage means accessible by the system).

Furthermore, chrominance values U1, V1and U2, V2are estimated from the pixel data of the ROIs201,202, as well as chrominance values U3, V3and U4, V4being estimated from the pixel data of the ROIs301,302. The values of the differences U1−U3, V1−V3are used to determine correction parameters for harmonizing the colour of images acquired by the RV and TR cameras (step117). Various methods can be applied to determine the colour correction parameters based on luminance difference values, such as the method further disclosed in WO2018/087348. Once determined, the colour correction parameters can be stored in the memory of the system (or any other storage means accessible by the system).

In some embodiments, each of the differences U1−U3, V1−V3is used to calculate colour parameters only upon verification that its value is below a threshold.

Furthermore, although the above disclosed embodiment is based on a comparison between Y, U, V values estimated for describing the whole data content of the ROIs201,202,301,302, in other embodiments the ROIs201,202,301can be divided in sub-regions for which respective Y, U, V are estimated and compared to determine the harmonization parameters. The subregions can correspond to single pixels or group of pixels within the ROIs201,202,301,302.

After calculation of the Y, U, V correction parameters, the method100can be re-executed at a later stage, starting again from step101to determine the direction of the car11. For example, the system can be configured to initiate the method periodically (and/or triggered by a specific driving activity/environment condition). In this way the stored harmonization correction parameters are updated over time.

With reference back to method step116, the method100is also re-executed after a determination that both the differences Y1−Y3and Y2−Y4have an absolute value exceeding the threshold (and this determination can trigger the re-execution of the method100).

The determined harmonization correction parameters can then be retrieved by the system, when required to be applied (step118) in the process of generating a combined view included merged images acquired by the RV and TR cameras, such as an invisible trailer view to be displayed on the main display22of the car11or on a windscreen that provides a digital rear mirror. In some embodiments, the harmonization correction parameters are applied to at least one of the images acquired by the RV and TR cameras before these images are merged into the combined view. In other embodiments, the harmonization correction parameters are applied to the combined view, particularly in the merging region between the images acquired by the RV and TR cameras.

Other combined views can benefit from applying the harmonization correction parameters obtained by the operation of method100, such as a top view of the environment surrounding the trailer12, that can be displayed on the display22and used to perform autonomous or semi-autonomous operations, or provide a footage that can be stored and retrieved at a later stage (e.g. for investigation after an accident, or theft of the trailer's content).

With reference back to method step116, if both the differences Y1−Y3and Y2−Y4are determined to have an absolute value exceeding the threshold, no updated harmonization parameters are available to the system to harmonize a combined view. Thus, the system can determine whether correction parameters previously generated and stored according to the operation of method100are available (step119). Responsive to a positive determination, the system can apply the previous correction parameters to harmonize the combined view (step120). Responsive to a negative determination (e.g., because only one iteration of the method100has been performed or the previous parameters are not retrievable), no harmonization is applied (step121—and in this case, the negative determination can trigger the re-execution of the method100).

Although the execution of the method100has been disclosed to harmonize the RV and TR cameras of the car11and the trailer12, the same principles can be similarly applied to harmonize the front FV camera and rear camera RV of the car11(or other vehicle, with or without a trailer), based on sampling the same road portions by the FV and RV cameras as the car11travels along a travel direction, and using the sampled road portions as common reference for harmonization.