Patent Description:
There are a variety of known methodologies for performing object display and detection using the front, rear, and side cameras of a vehicle and standard camera images. These methodologies allow such objects to be displayed to a vehicle occupant on an in-vehicle display screen, and an artificial intelligence (AI) algorithm can be applied to the standard camera images to allow a driver assist (DA) or autonomous driving (AD) system to recognize, segment, annotate, process, respond, and/or react to the objects. Often, multiple front, rear, and side camera images are combined into a single camera image on the in-vehicle display screen, providing the vehicle occupant with a surrounding view that is essentially unobstructed by the vehicle itself. One challenge faced is that the front and rear cameras, as well as side cameras, are often fisheye cameras.

A standard camera lens, also referred to as a rectilinear camera lens, is a camera lens that reproduces straight lines as straight lines. A fisheye camera lens, also referred to as a curvilinear camera lens or an omnidirectional camera lens, on the other hand, reproduces straight lines as curved lines, i.e., it provides a convex non-rectilinear appearance.

The front portion of a fisheye camera lens utilizes a cupola or domed shaped front end, and the fisheye camera lens derives its name from being similar in appearance to the eye of a fish. A fisheye camera lens is an example of a panoramic or hemispherical camera lens that has a field of view that is e.g. <NUM>°, <NUM>°, or <NUM>°. A fisheye camera lens thus has a wider field of view than a rectilinear camera lens, and has the ability to capture large dimensions of a specified area in one shot. Instead of producing images with straight lines of perspective (i.e., rectilinear images), the fisheye camera lens produces images with convex non-rectilinear lines of perspective. Thus, a fisheye camera lens provides images with altered or inaccurate views, i.e., with visual distortion. There are several types of fisheye camera lenses, such as a circular fisheye camera lens, a full-frame fisheye camera lens, a panomorph fisheye camera lens, an omnidirectional camera lens, etc..

Thus, fisheye camera lenses are widely used as visual sensors in DA and AD systems because of their good coverage (i.e., wide field of view). However, this comes with costs: the unnatural images caused by distortion, especially at the outer edges of the images. This not only makes it difficult for a vehicle occupant to understand and interpret the content of such images, but also causes problems for AI algorithms that process the images. Most computer vision and machine learning (ML) algorithms are designed for and trained on non-fisheye (i.e., standard, undistorted) datasets, which makes them sub-optimal or even prone to failure when performing on the highly distorted images captured by fisheye camera lenses.

Some existing approaches undistort images at the cost of losing a large portion of the cameras' field of view, which defeats the purpose of using the fisheye camera lens to begin with. A vehicle detection system is an exemplary system that uses fisheye camera lenses for detecting surrounding vehicles. <FIG> and <FIG> illustrate the operation of such a vehicle detection system, where <FIG> illustrates the vehicle detection system applying a distorted image and <FIG> illustrates the vehicle detection system applying an undistorted image. In more detail, <FIG> and <FIG> illustrate the enhancement of detection performance for a single-shot detection (SSD) object detection algorithm for AD using a conventional fisheye camera lens methodology to minimize distortion, which comes at the cost of losing a large portion of the field of view. <FIG> illustrates the curvilinear image captured by a fisheye camera lens located at the side (i.e., flank) of a vehicle. The bounding boxes around two of the vehicles represent that they are the vehicles that are detected. The vehicles that do not have any bounding boxes around them are not detected by the vehicle detection system due to the distortions in the image. <FIG> illustrates the undistorted image, where a large portion of the field of view is lost, but where all the vehicles in the remaining field of view are detected by the vehicle detection system, as indicated by the bounding boxes around the vehicles. <FIG> and <FIG> serve the purpose of highlighting two facts: (<NUM>) mitigating distortion improves the performance of software such as the vehicle detection system and (<NUM>) mitigating distortion undesirably reduces the field of view.

One important object detection function is lane marking detection. Such lane marking detection is typically carried out using a standard front or rear camera image, or a side fisheye camera image that is undistorted using a conventional methodology, thereby sacrificing field of view and detection scope. A significant problem arises, however, under low-standing sun or glare conditions, when a standard front camera image can be obscured, for example. The present invention provides systems and methods that address this and other problems. <CIT> describes image processing algorithms for extracting natural looking panoramic images and distortion-free rectilinear images from images acquired using a camera equipped with a wide-angle lens which is rotationally symmetric about an optical axis and devices implementing such algorithms. <NPL>, XP055699004 describes mathematically precise image-processing algorithms for extracting panoramic images from fisheye images.

The present invention provides systems and methods for vehicle lane marking and other object detection using side fisheye cameras and three-fold de-warping. Three-fold de-warping is applied to a side fisheye camera image to create straight, rectilinear side, front, and rear camera images that are readily displayed and understood by a vehicle occupant and/or processed as a suitable dataset for an AI algorithm in a DA or AD system. This three-fold de-warping of the side fisheye camera image preserves field of view, such that all surrounding lane markings and other objects can be viewed and/or detected. Advantageously, the side fisheye camera image is typically not obscured by low-standing sun or glare conditions (at least not on both sides of a vehicle), and can be used when the vehicle is traveling towards or away from the sun or glare source, as a replacement for or complement to the images obtained from typical front or rear camera methodologies. The three-fold de-warped, straight, rectilinear side, front, and rear camera images obtained from the side fisheye camera image are ideally suited for use as a dataset for any variety of AI algorithms, again as a replacement for or complement to the typical front or rear camera methodologies. Thus, the systems and methods for vehicle lane marking and other object detection using side fisheye cameras and three-fold de-warping provided herein can be used as a replacement or substitute for conventional front or rear camera methodologies depending upon vehicle operating and camera visibility conditions, or can be used as a verification of or complement to such conventional front or rear camera methodologies.

In one embodiment, provided herein is a method of handling images of surroundings of a vehicle, the method comprising: obtaining an image of the surroundings of the vehicle, wherein the image is obtained from at least one side image capturing device mounted in or on the vehicle, and wherein the at least one side image capturing device comprises a curvilinear camera lens; correcting at least a part of distortions in the image to obtain a corrected image; rotationally transforming a left part of the corrected image using a first rotational transformation to obtain a first transformed image; rotationally transforming a right part of the corrected image using a second rotational transformation to obtain a second transformed image, wherein the first and second transformed images are consecutive or adjacent images or image parts; and performing object detection using both the first transformed image and the second transformed image to perform object detection for the image. Optionally, the step of obtaining the image of the surroundings of the vehicle using the at least one side image capturing device is performed only after determining that one of a front or a rear image capturing device is obscured. Alternatively, the step of obtaining the image of the surroundings of the vehicle using the at least one side image capturing device is performed simultaneously with a step of obtaining another image of the surroundings of the vehicle using at least one of a front of a rear image capturing device. Preferably, the surroundings of the vehicle include one or more lane markings. The first and second transformed images are provided to an artificial intelligence algorithm operable for performing lane marking detection using the first and second transformed images. The method further includes removing redundant overlapping areas from at least one of the first and second transformed images. The first transformed image is mapped on one planar surface and the second transformed image is mapped on another planar surface. The method further includes displaying at least one of the first and second transformed images to a user of the vehicle.

In another embodiment, provided herein is a computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method comprising the following steps: obtaining an image of the surroundings of the vehicle, wherein the image is obtained from at least one side image capturing device mounted in or on the vehicle, and wherein the at least one side image capturing device comprises a curvilinear camera lens; correcting at least a part of distortions in the image to obtain a corrected image; rotationally transforming a left part of the corrected image using a first rotational transformation to obtain a first transformed image; rotationally transforming a right part of the corrected image using a second rotational transformation to obtain a second transformed image, wherein the first and second transformed images are consecutive or adjacent images or image parts; and performing object detection using both the first transformed image and the second transformed image to perform object detection for the image. Optionally, the step of obtaining the image of the surroundings of the vehicle using the at least one side image capturing device is performed only after determining that one of a front or a rear image capturing device is obscured. Alternatively, the step of obtaining the image of the surroundings of the vehicle using the at least one side image capturing device is performed simultaneously with a step of obtaining another image of the surroundings of the vehicle using at least one of a front or a rear image capturing device. Preferably, the surroundings of the vehicle include one or more lane markings. The first and second transformed images are provided to an artificial intelligence algorithm operable for performing lane marking detection using the first and second transformed images. The method further includes the step of removing redundant overlapping areas from at least one of the first and second transformed images. The first transformed image is mapped on one planar surface and the second transformed image is mapped on another planar surface. The method further includes the step of displaying at least one of the first and second transformed images to a user of the vehicle.

In a further embodiment, provided herein is a system for handling images of surroundings of a vehicle, the system comprising:at least one side image capturing device mounted in or on the vehicle operable for obtaining an image of the surroundings of the vehicle, wherein the at least one side image capturing device comprises a curvilinear camera lens;a correcting module executed on a processor operable for correcting at least a part of distortions in the image to obtain a corrected image (s302);a transforming module executed on the processor operable for rotationally transforming a left sub-image of the corrected image using a first rotational transformation to obtain a first transformed image (s303); andthe transforming module executed on the processor operable for rotationally transforming a right sub-image of the corrected image using a second rotational transformation to obtain a second transformed image (s304), wherein the first and second transformed images are consecutive or adjacent images or image parts;wherein the processor is operable to perform object detection using both the first transformed image and the second transformed image to perform object detection for the image. Optionally, obtaining the image of the surroundings of the vehicle using the at least one side image capturing device is performed only after determining that one of a front or a rear image capturing device is obscured. Alternatively, obtaining the image of the surroundings of the vehicle using the at least one side image capturing device is performed simultaneously with obtaining another image of the surroundings of the vehicle using at least one of a front or a rear image capturing device. Preferably, the surroundings of the vehicle include one or more lane markings, wherein the first and second transformed images are provided to an artificial intelligence algorithm operable for performing lane marking detection using the first and second transformed images.

The present invention is illustrated and described herein with reference to the various drawings, in which like reference numbers are used to denote like system components/method steps, as appropriate, and in which:.

The drawings provided herein are not necessarily to scale and the dimensions of certain features may be exaggerated for the sake of clarity. Emphasis is instead placed on illustrating the principles of operation of the exemplary embodiments provided herein.

Thus, the systems/methods provided herein provide a means for de-warping and undistorting images (e.g., <NUM>-degree images and <NUM>-degree images) that enhance the visualizing of lane markings and other objects for a vehicle occupant (e.g., a vehicle driver), as well as the performance of conventional computer vision and ML algorithms.

<FIG> is a schematic diagram illustrating a vehicle <NUM>. The vehicle <NUM> can be any vehicle, for instance, a car, truck, van, bus, motorcycle, passenger transport, etc. The vehicle <NUM> can be at least partly autonomous or self-driven, it can be completely autonomous or self-driven, or it can be non-autonomous or self-driven.

The vehicle <NUM> utilizes at least one image capturing device <NUM>. <FIG> illustrates an example where the vehicle <NUM> utilizes three or four image capturing devices <NUM>, but any other suitable number of image capturing devices <NUM> is applicable. In <FIG>, the image capturing devices <NUM> are located at the front of the vehicle <NUM>, at one or both sides of the vehicle <NUM>, and at the rear of the vehicle <NUM>. Thus, the image capturing devices <NUM> can be referred to as being forward looking, sideways looking, and backwards looking. At least the side image capturing devices <NUM> include a fisheye camera lens and can also be referred to as a fisheye camera or an ultra-wide angle camera. Such side image capturing devices <NUM> can be adapted to capture an image of at least <NUM> degrees of the surroundings of the vehicle <NUM>, for example. In some embodiments, the image capturing device <NUM> may be adapted to capture an image of <NUM> degrees of the surroundings of the vehicle <NUM>, for example.

A fisheye camera lens can produce strong visual distortion in obtained images, including wide panoramic or hemispherical images. Fisheye camera lenses are designed to achieve extremely wide angles of view, but instead of images with straight lines of perspective (i.e., rectilinear images) as obtained by rectilinear camera lenses, fisheye camera lenses use mapping, which gives images a characteristic convex non-rectilinear appearance.

The method performed by the vehicle system for handling images of the surroundings of the vehicle <NUM>, including lane markings and other objects, is described with reference to the flowchart depicted in <FIG>. The method performed by the vehicle system includes at least one of the following steps, which steps can be carried out in any suitable order.

Step <NUM>: The vehicle system obtains an initial image <NUM> of the surroundings of the vehicle <NUM>. The image <NUM> is obtained from at least one image capturing device <NUM> mounted in or on the vehicle <NUM>. The image capturing device <NUM> includes a fisheye camera lens, and therefore the obtained image can be referred to as a fisheye camera image. The image <NUM> can be obtained by receiving it directly from the image capturing device <NUM>. In another embodiment, the image capturing device <NUM> captures the image <NUM> and stores it in a memory, and then the vehicle system obtains the image <NUM> from the memory. The image may be obtained upon request from the vehicle system, on a periodic basis, or continuously. As described herein above, the initially obtained image <NUM> includes distortions. The image <NUM> can be of at least <NUM> degrees of the surroundings of the vehicle <NUM>, for example. The image capturing device <NUM> can be a fisheye camera accordingly.

Step <NUM>: The vehicle system then corrects at least a part of the distortions in the image <NUM> to obtain a corrected image. This correcting process can also be referred to as de-warping or base de-warping. A base de-warp can be carried out by any appropriate camera calibration algorithms or models executed by the vehicle system that obtains a mapping from warped coordinates to de-warped coordinates, as well as the intrinsic parameters of the image capturing device <NUM>. De-warping can be described as correcting the obtained image <NUM> to reverse the effects of geometric distortions caused by the image capturing device <NUM>, e.g., the fisheye camera lens of the mage capturing device <NUM>.

Step <NUM>: The vehicle system rotationally transforms the corrected image using a first rotational transformation to obtain a first transformed image <NUM>.

Step <NUM>: The vehicle system rotationally transforms the corrected image using a second rotational transformation to obtain a second transformed image <NUM>. The first and second rotational transformations are different from each other, in that the first and second transformed images <NUM>, <NUM> are preferably consecutive images. The term "consecutive images" can refer to images that are following, sequential, serial, succeeding, adjacent, etc. For example, the first transformed image <NUM> and the second transformed image <NUM> are consecutive in that the first transformed image <NUM> represents the right part of the obtained image and the second transformed image <NUM> represents the left part of the obtained image. When the two transformed images <NUM>, <NUM> are placed together, they form one image which corresponds to the obtained image, but instead it provides an undistorted image relative to the initial image <NUM>.

The first transformed image <NUM> can be mapped on one planar surface and the second transformed image <NUM> can be mapped on another planar surface, as illustrated in <FIG>. The two planar surfaces can be located next to each other, i.e., they are consecutive or adjacent surfaces. <FIG> illustrates a three-dimensional (3D) view of the two-fold form/pattern. <FIG> also illustrates a 3D view of the two-fold form/pattern in another view.

Steps <NUM> and <NUM> can be referred to as a two-fold mapping, where two rotational transformations are applied separately after the base de-warp. This generates two different views. The amount of rotation applied in each rotational transformation is set or can be adjusted such that the de-warped images look natural and as if they are captured by two cameras facing different directions or having different orientations. The structure of the two-fold is demonstrated in <FIG>.

Step <NUM>: The vehicle system can remove redundant overlapping areas from at least one of the first and second transformed images <NUM>, <NUM>. This step includes applying appropriate cropping to remove part of the redundant overlapping areas between the first and second transformed images <NUM>, <NUM>. This step can also remove a part of any highly distorted areas (usually at the edges of the views). Some overlapping areas may be preserved between the first and second transformed images <NUM>, <NUM> to allow for potential stitching/porting of algorithm results between the first and second transformed images <NUM>, <NUM>.

The first and second transformed images <NUM>, <NUM> (possibly also after removal of the redundant overlapping areas) can be referred to as the resulting undistorted, de-warped images. The resulting undistorted, de-warped images allow for more natural and understandable views in at least two directions defined by the planar surfaces.

Step <NUM>: The vehicle system provides the first and second transformed images <NUM>, <NUM> as input to another vehicle system for further processing. Such other vehicle system may be, for example, a lane detection system, a vehicle detection system, a crash avoidance system, an AD system, etc. The first and second transformed images <NUM>, <NUM> allow for "general" ML/computer vision algorithms/models to be applied by the vehicle system or by other vehicle systems. Here, "general" algorithms/models may refer to those designed for and/or trained on images that are usually captured by standard cameras, i.e., non-fisheye/non-omnidirectional cameras.

Step <NUM>: The vehicle system can display at least one of the first and second transformed images <NUM>, <NUM> to a vehicle occupant. In one embodiment, all transformed images <NUM>, <NUM> can be displayed on a display screen in the vehicle <NUM> at the same time. In another embodiment, one image <NUM>, <NUM> can be displayed at a time, and the vehicle occupant can then switch between the different images.

The method of <FIG> is described with at least two rotational transformations. <FIG> provides an example with three rotational transformations. In other words, <FIG> illustrates how a two-dimensional (2D) image captured by a <NUM>-degree fisheye camera <NUM> is mapped onto three planar surfaces. The three rotational transformations provide a first transformed image <NUM>, a second transformed image <NUM>, and a third transformed image <NUM>. The vehicle system rotationally transforms the corrected image using a third rotational transformation to obtain the third transformed image <NUM>. The first, second, and third rotational transformations are different from each other, in that the first, second, and third transformed images <NUM>, <NUM>, <NUM> are consecutive or adjacent images. The left part of <FIG> illustrates the obtained image <NUM>, i.e., the original view, with the dashed lines indicating the splitting of such image <NUM> into three folds. The right part of <FIG> illustrates the three transformed images <NUM>, <NUM>, <NUM>, i.e., the top view of the formation of the three planar surfaces.

<FIG> presents the three-fold pattern in 3D from a different perspective as compared to <FIG>.

<FIG>, <FIG>, <FIG> and <FIG> illustrate examples of resulting images. The figures illustrate images taken from a left-facing image capturing device <NUM> mounted in or on a vehicle <NUM> going on an expedition on a highway, for example. The image capturing device <NUM> is exemplified with a <NUM>-degree fisheye camera. <FIG> and <FIG> illustrate the result of applying the three-fold de-warping. <FIG> is the original input image captured by the <NUM>-degree fisheye camera <NUM>. <FIG> illustrates the three resulting images that are mapped onto three planar surfaces. <FIG> and <FIG> illustrate another such result. <FIG> is the original input image captured by the <NUM>-degree fisheye camera <NUM>. <FIG> illustrates the three resulting images that are mapped onto three planar surfaces.

The original images before three-fold de-warping (i.e., before steps <NUM>-<NUM>) are shown in <FIG> and <FIG>. <FIG> and <FIG> show the images after de-warping (i.e., after step <NUM>, and possibly also after step <NUM>). In <FIG> and <FIG>, the left and right images synthesize the views of the forward and backward facing cameras, which gives a vehicle occupant a better sense of position and direction, as compared to the previously distorted area at the edges. It can be clearly seen that the images in <FIG> and <FIG> are much more intuitive after the de-warping.

In both <FIG> and <FIG>, the lane markings are highly distorted, whereas they are undistorted in <FIG> and <FIG>. The curved lane markings are corrected to be straight, and this is desired and will improve the performance of other vehicle systems, such as e.g. an AD system (in this case, a lane detection system) that utilizes these images. Also, it should be noticed that in the right image of <FIG>, the previously sideways rotated truck is now upright positioned, which in this case helps a vehicle detection system.

The embodiments provided herein aim at achieving undistorted images without losing much field of view of the image capturing device <NUM>, so that the images can be better used by both vehicle occupants (e.g., users <NUM>) and vehicle systems (e.g., AI/ML/computer vision algorithms for DA and AD).

Steps <NUM>, <NUM>, and <NUM> will now be described in more detail using three rotational transformations as an example.

Step <NUM>: The base de-warp can be described as the process of estimating a set of intrinsic related parameters K, ξ, and D, as well as a set of extrinsic related parameters r and t, from a set of images containing a calibration pattern, such as a chessboard. Here, K is a generalized image capturing device matrix, ξ is a single value parameter, D includes the distortion coefficients, and r and t characterize rotations and translations between the set of images and the image capturing device <NUM>, respectively. K, ξ, and D are used to undistort the images taken by the image capturing device <NUM>. The image capturing device matrix for the rectified images Knew is usually a scaled identity matrix.

Steps <NUM> and <NUM>: A rotational transformation is applied after the base de-warp in step <NUM> by multiplying a rotational matrix R with the image capturing device matrix for rectified images Knew: <MAT>.

The new image capturing device matrix KR replaces Knew, and is used together with the previous K, ξ and D to obtain the rotated views of the undistorted images (i.e., the first, second, and third transformed images <NUM>,<NUM>,<NUM>).

Here, the rotational transformation can be decomposed into three rotational transformations around x (horizontal), y (vertical), and z (optical axis of the fisheye camera lens in the image capturing device <NUM>) axes. <MAT> <MAT> <MAT> <MAT>.

For the left fold among the three-fold de-warps (the angles are in radians): <MAT> <MAT> <MAT>.

For the center fold among the three-fold de-warps (the angles are in radians): <MAT> <MAT> <MAT>.

For the right fold among the three-fold de-warps (the angles are in radians): <MAT> <MAT> <MAT>.

To perform the method steps shown in <FIG> for handling images of the surroundings of a vehicle <NUM>, the vehicle system can utilize an arrangement as shown in <FIG>.

The vehicle system is adapted to, e.g., by means of an obtaining module <NUM>, obtain an image of the surroundings of the vehicle <NUM>. The image is obtained from at least one image capturing device <NUM> mounted on/to the vehicle <NUM>. The image capturing device <NUM> includes a fisheye camera lens. The image can be of at least <NUM> degrees of the surroundings of the vehicle <NUM>. The image capturing device <NUM> can be a fisheye camera. The obtaining module <NUM> can also be referred to as an obtaining unit, an obtaining means, an obtaining circuit, means for obtaining, etc. The obtaining module <NUM> can be comprised in a processor <NUM> of the vehicle system. In some embodiments, the obtaining module <NUM> can be referred to as a receiving module.

The vehicle system is adapted to, e.g., by means of a correcting module <NUM>, correct at least a part of distortions in the image to obtain a corrected image. The correcting module <NUM> can also be referred to as a correcting unit, a correcting means, a correcting circuit, means for correcting, etc. The correcting module <NUM> can be comprised in the processor <NUM> of the vehicle system.

The vehicle system is adapted to, e.g. by means of a transforming module <NUM>, rotationally transform the corrected image using a first rotational transformation to obtain a first transformed image. The transforming module <NUM> may also be referred to as a transforming unit, a transforming means, a transforming circuit, means for transforming etc. The transforming module <NUM> may be or comprised in the processor <NUM> of the vehicle system.

The vehicle system is adapted to, e.g., by means of the transforming module <NUM>, rotationally transform the corrected image using a second rotational transformation to obtain a second transformed image. The first and second rotational transformations are different from each other, in that the first and second transformed images are consecutive or adjacent images. The first transformed image can be mapped on one planar surface and the second transformed image can be mapped on another planar surface.

The vehicle system can be adapted to, e.g., by means of a removing module <NUM>, remove redundant overlapping areas from at least one of the first and second transformed images. The removing module <NUM> can also be referred to as a removing unit, a removing means, a removing circuit, means for removing, etc. The removing module <NUM> can be comprised in the processor <NUM> of the vehicle system.

The vehicle system can be adapted to, e.g., by means of a providing module <NUM>, provide the first and second transformed images as input to another vehicle system for further processing. The providing module <NUM> can also be referred to as a providing unit, a providing means, a providing circuit, means for providing, etc. The providing module <NUM> can be comprised in the processor <NUM> of the vehicle system. In some embodiments, the providing module <NUM> can be referred to as a transmitting module.

The vehicle system can be adapted to, e.g., by means of a displaying module <NUM>, display at least one of the first and second transformed images to a user <NUM> of the vehicle <NUM>. The images can be displayed on a display in the vehicle <NUM>. The displaying module <NUM> can also be referred to as a displaying unit, a displaying means, a displaying circuit, means for displaying, etc. The displaying module <NUM> can be comprised in the processor <NUM> of the vehicle system.

In some embodiments, the vehicle system includes the processor <NUM> and a memory <NUM>. The memory <NUM> stores instructions executable by the processor <NUM>. The memory <NUM> can include one or more memory units. The memory <NUM> is arranged to be used to store data, received data streams, power level measurements, images, parameters, distortion information, transformation information, vehicle information, vehicle surrounding information, threshold values, time periods, configurations, schedulings, and applications to perform the methods herein when being executed by the vehicle system.

The embodiments herein for handling images of the surroundings of a vehicle <NUM> can thus be implemented through one or more processors, such as a processor <NUM> in the vehicle system arrangement depicted in <FIG>, together with computer program code for performing the functions of the embodiments herein. The processor can be, for example, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC) processor, a Field-Programmable Gate Array (FPGA) processor, or a microprocessor. The program code mentioned above can also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the vehicle system. One such carrier can be in the form of a CD ROM disc. It is however feasible with other data carriers, such as a memory stick. The computer program code can furthermore be provided as pure program code on a server and downloaded to the vehicle system. A computer program can include instructions which, when executed on at least one processor, cause the at least one processor to carry out the method described above. A carrier can include the computer program, and the carrier may be one of an electronic signal, an optical signal, a radio signal, or a computer readable storage medium.

Those skilled in the art will also appreciate that the obtaining module <NUM>, the correcting module <NUM>, the transforming module <NUM>, the removing module <NUM>, the providing module <NUM>, and the displaying module <NUM> described above can refer to a combination of analog and digital circuits, and/or one or more processors configured with software and/or firmware, e.g., stored in a memory, that when executed by the one or more processors, such as the processor <NUM>, perform as described above. One or more of these processors, as well as the other digital hardware, can be included in a single ASIC, or several processors and various digital hardware can be distributed among several separate components, whether individually packaged or assembled into a system-on-a-chip (SoC).

The following terminologies are used interchangeably herein: "de-warping", "undistortion", and "mapping". These all describe the process of some geometric transformation of an image, usually from the two-dimensional (2D) images captured by the image capturing device <NUM> to at least two planar images that do not have distortion effects introduced by the image capturing device <NUM>.

"Computer vision and machine learning algorithms" refer to general algorithms that use images captured by the image capturing device <NUM> as input, and output decisions that are relevant for DA and/or AD, based on machine learning/artificial intelligence technology. Some examples are lane marking detection, vehicle detection, pedestrian detection, distance measurement, etc..

Directions as used herein, e.g., horizontal, vertical, and lateral relate to when the vehicle system is mounted in the vehicle <NUM>, which stands on essentially flat ground. The vehicle system can be manufactured, stored, transported, and sold as a separate unit. In that case, the directions may differ from when mounted in the vehicle <NUM>.

The present invention thus provides systems and methods for vehicle lane marking and other object detection using side fisheye cameras and three-fold de-warping. Three-fold de-warping is applied to a side fisheye camera image to create straight, rectilinear side, front, and rear camera images that are readily displayed and understood by a vehicle occupant and/or processed as a suitable dataset for an AI algorithm in a DA or AD system. This three-fold de-warping of the side fisheye camera image preserves field of view, such that all surrounding lane markings and other objects can be viewed and/or detected. Advantageously, the side fisheye camera image is typically not obscured by low-standing sun or glare conditions (at least not on both sides of a vehicle), and can be used when the vehicle is traveling towards or away from the sun or glare source, as a replacement for or complement to the typical front or rear camera methodologies. The three-fold de-warped, straight, rectilinear side, front, and rear camera images obtained from the side fisheye camera image are ideally suited for use as a dataset for any variety of AI algorithms, again as a replacement for or complement to the typical front or rear camera methodologies. Thus, the systems and methods for vehicle lane marking and other object detection using side fisheye cameras and three-fold de-warping provided herein can be used as a replacement or substitute for conventional front or rear camera methodologies depending upon vehicle operating and camera visibility conditions, or can be used as a verification of or complement to such conventional front or rear camera methodologies.

<FIG> shows a front view out of a vehicle with low standing sun causing a glare that can prevent conventional front camera images from being used to detect lane markings and other objects surrounding the vehicle. Under such conditions, the vehicle system can default to the use of one or both side fisheye cameras to obtain images used to detect such lane markings an other objects, via the process described in detail above. The side fisheye camera(s) can also be used as a substitute for, as a complement to, or as a confirmation for the conventional front camera under normal, non-obscured conditions as well.

<FIG> shows a resulting side fisheye camera image <NUM> prior to processing. It can be seen that the lane markings are curved, which is not preferred for vehicle occupant display and/or AI/ML processing, however the lane markings are not obscured by the sun, and typically will not be, at least not on both sides of a vehicle simultaneously. This side fisheye camera image <NUM> is then processed as described in detail above.

Claim 1:
A method of handling images of surroundings of a vehicle (<NUM>), the method comprising:
obtaining an image (<NUM>) of the surroundings of the vehicle (<NUM>) (s301), wherein the image (<NUM>) is obtained from at least one side image capturing device (<NUM>) mounted in or on the vehicle (<NUM>), and wherein the at least one side image capturing device (<NUM>) comprises a curvilinear camera lens;
correcting at least a part of distortions in the image (<NUM>) to obtain a corrected image (s302);
rotationally transforming a left part of the corrected image using a first rotational transformation to obtain a first transformed image (<NUM>) (s303);
rotationally transforming a right part of the corrected image using a second rotational transformation to obtain a second transformed image (<NUM>) (s304), wherein the first and second transformed images (<NUM> and <NUM>) are consecutive or adjacent images or image parts; and
performing object detection using both the first transformed image and the second transformed image to perform object detection for the image.