Patent ID: 12190599

DETAILED DESCRIPTION

FIG.1shows a schematic function block diagram of an image processing system100, shown in turn to comprise an ego localization component102, an object region computation component104, an image transformation component106and an image recognition component108.

The image processing system100receives an image sequence111containing views of a known object at different scales. For example, in a driving context, the known object could be a static object (such as a traffic light, road sign, or other traffic control object) that a sensor-equipped vehicle is approaching. As successive images of the object are captured from the vehicle, the scale of the object will increase across the sequence of images as the object is approached (that is, the object will appear larger in later images captured closer to the object).

A core function of the image processing system is to allow the object view in each of the images to be rescaled to essentially a fixed scale. The image transformation component106applied an image transformation to each image in order to compute a transformed image. The transformation comprises image rescaling, and may also include other types of transformation.

In the examples described below, a crop region of variable size is computed around the object in each of the images, and the transformed image is a cropped and rescaled image of that region is generated from the original image.

The rescaling is such that the vertical and horizontal extent of the object view in the transformed image as measured in pixels (its pixel width and height in the transformed image) is essentially the same in all of the cropped and rescaled images, and essentially independent of the distance of the object from the sensor-equipped vehicle when the original image was captured (object distance), and essentially independent of its pixel width and height in the original image prior to transformation.

The ability to provide fixed-scale images of an object view—irrespective of the scale at which it was originally captured—has significant benefits in an image recognition context.

For example, when training a machine learning (ML) image recognition component to extract information about a particular type of object from images in which the object might appear at different scales, a sufficient number of training images will be needed that capture the object at a representative range of scales. By contrast, if the image recognition component108ofFIG.1is implemented using ML model(s), those models need only be capable of recognizing features of the object at essentially a fixed scale. This, in turn, reduces the training data requirements, making it possible to achieve a given level of performance with less training data.FIG.6shows examples of transformed images obtained using the described image processing techniques, for a range of object distances, annotated with 2D bounding box detections. The bounding boxes were detected using an extremely light-weight convolutional neural network (CNN), made possible by the automatic rescaling of the object views to fixed pixel dimensions.

Moreover, if the object is known to appear in the transformed images at substantially a fixed scale, this potentially means that simpler ruled-based image processing could be applied by the image recognition component108in order to recognize the relevant features of the object. For example, in traffic light detection context, it may be possible to implement rules-based detection based on appropriate assumptions about the pixel sizes of the component lights in the fixed-scale cropped images.

Cropping is not essential—for example, CNNs can receive an image of arbitrary size (pixel dimensions), and are based on convolutions applied uniformly across the area of the image. what is material in that context is the rescaling that significantly simples the pattern recognition task that the CNN needs to learn. Nevertheless, cropping can yield efficiency benefits (the image recognition can be performed using fewer computational resources because there is less extraneous image content to process). The selective image cropping can also potentially improve the reliability of the image recognition process by reducing the amount of extraneous visual information that the image recognition component108needs to consider. Removing image data outside of the crop error prevents such data from causing a false detection or other image recognition error.

The following examples implement the rescaling by computing a variable-sized crop region Rnfor each image n, containing the object view in the original image n, and rescaling the portion of the image within the crop region Rn. This results in a transformed image having fixed pixel dimensions M×N. The size of the crop region relative to the original image n is computed in the manner described below, to ensure that the rescaled object view also has essentially fixed pixel dimensions m×n in the M×N transformed image—seeFIG.2Band accompanying description below. However, it will be appreciated that underlying principles of the rescaling can be implemented in other ways, with or without cropping.

One application of the system100is traffic light detection, where the aim is to detect a current state of a set of traffic lights from a discrete set of possible states. This is a significantly simpler problem when traffic lights are always known to appear at essentially the same scale, irrespective of how close or far away they were when the original image was captured.

In order to determine an appropriate crop region for each image, the system100uses a “world model”112that encodes knowledge about the location of objects within a world frame of reference (the world). The world model122encodes a degree of external knowledge of the known object that allows a suitable crop area (region of interest) to be determined before the desired image recognition is applied. In the following examples, that knowledge includes the location of the object in the world frame of reference, and knowledge of its size or dimensions encoded in a predetermined object model O (which takes the form of a simple 3D template for modelling the object).

A function of the ego object localization component102is to determine, for each image, an image capture location in the world (i.e. in the same world frame of reference). References below to localization mean ego localization in this sense (unless otherwise indicated). That is, an estimated location, in the world, of an image capture system (camera) when it captured the image. In a driving context, this would be camera of a sensor-equipped vehicle.

Once the locations of the camera and the object are known, this allows the location and extent (approximate dimensions) of the object within the image plane of the image to be determined via projection into the image plane of the image, for the purpose of computing a suitable crop area containing the object view.

Localization is performed using sensor data110, which could include the image sequence111itself (vision-based localization) and/or other type(s) of sensor data, such as one or more lidar, radar, satellite navigation (e.g. GPS), and IMU (inertial measurement unit) data etc. There are many known localization methods that can be used for this purpose depending on the type of sensor data110that is available, in the field of autonomous driving and elsewhere. Localization an image capture device in this context means determining its location and orientation (pose) in some fixed world (global) frame of reference of the world model122. This could, for example, be geographic coordinates (e.g. latitude, longitude), or whatever fixed frame of reference the world model122is defined in.

The world model112ofFIG.1is shown to comprise a high-definition (HD) map112aof static objects within a driving area. HD maps are maps having a sufficient level of detail, precision and accuracy for use in autonomous driving. The HD map112aallows a known static object of interest to be located in the world. HD maps that describe road structure and surrounding objects with centimetre-level accuracy are available.

The present techniques can be deployed in both “online” and “offline” contexts. In an online context, the image processing may be implemented in real time, to allow e.g. an autonomous vehicle or other robotic system to make perception-driven decisions. For example, in an autonomous driving context, the techniques may be used to provide real-time traffic light detection to allow a planner to plan suitable maneuvers as the vehicle approaches a set of traffic lights.

Offline contexts include the generation of training data, and in that case cropped, fixed-scale images can be derived using the present techniques to be used for training the image recognition component108.

Another offline context is scenario extraction, where the aim is to extract a relatively high-level scenario that can be deployed in a simulator. For example, in a traffic light detection context, the image recognition component108could perform traffic light detection (analogous to the online application) to allow potentially changing traffic light states to be captured in the extracted scenario such that they can be subsequently re-created in a simulator.

In an offline context, the image processing system100can be usefully deployed within an annotation system to facilitate automatic or semi-automatic image annotation. Example annotation applications are described below with reference toFIGS.5and6.

FIG.2shows a flowchart for an image processing method implemented by the image processing system100. To further aid understanding,FIG.2also shows a schematic illustration of certain operations performed at each step.

At step202, time sequence of images111is received. In an online context, images of the sequence may be received in real-time as they are captured, with the subsequent steps performed in real time for each successive image. In an offline context, the method may or may not be implemented in real-time depending on the context.

Three possible images of the sequence are depicted, as captured at time instants ta, tband tcrespectively (the notation tnis used to denote the capture time of image n). In this example, the images111are captured by a vehicle as it approaches a known object200, which is depicted as a set of traffic lights. As the vehicle moves closer to the traffic lights200, the size of the traffic lights200in the images relative to the area of the images (the scale of the traffic lights within the images) increases.

At step204, localization is performed, in order to determine an image capture location of each of the images in the world (one form of localization data). For the aforementioned images a, b and c, the image capture locations are denoted by xa, xband xcrespectively and, in the present example, these take the form of 6D poses, encoding a spatial position and orientation of the camera at the respective time instants ta, tband tc, in a 3D world frame of reference. As noted, the localization data may be extracted from the image(s)111themselves (vision-based localization) and/or other data of the sensor data110.

A location X of the known object in the world frame of reference is known from the world model112. This means the location of the known object200relative to each image capture location xnis known.

FIG.2Ashows an expanded top-down view of the world coordinate system, in which the image capture location xnfor image n and the object location X are defined. The world frame of reference could be 2D, providing a top-down (bird's-eye view) representation of known object(s) in the world, in which case xnand X are 2D coordinates in the plane of the world coordinate system. Alternatively, the word frame of reference could be 3D (as depicted in the perspective view of step206inFIG.2), in which case xnand X are 3D coordinates in 3D space. In any event, the world frame of reference spans the direction Nnperpendicular to the image plane In. The image capture location xnand object location X lie outside of the image plane In, and an object distance dnis defined as the distance of the object200from the image capture location xnin the direction Nnperpendicular to the image plane In. This object distance dndefines the scale of the object200in the original image n, prior to rescaling.

Returning toFIG.2, at step206, the object location X and image capture location xnare used to locate the object200within image n, i.e. within an image plane Inof the image. The location (spatial position and orientation) of the image plane Inin the 3D world frame of reference is defined by the 6D camera pose xn. In this example, the world model112comprises a 3D model O representing the known object200, which is projected into the image plane Inbased on the object location X and the image capture location xn. The projection of the object model O in the image plane Inis denoted by Pn. The size of the projection Pnrelative to the dimensions of image n (its scale in the original image n) will depend on the location of the object relative to the image capture location xn.

A detailed object model O is not required for this purpose. For example, in many practical applications (including traffic light detection), a simple model such as a cuboid of approximately the correct size may be sufficient.

The object projection Pndefines a crop area Rncontaining the view of the object within the image In. Note that this has been not been detected from the content of the image n itself, but rather has been inferred from the world model based on ego localization. Depending on the type of localization that is used (and whether or not it is vision-based), a degree of image processing may be performed as part of the localization of step202. However, this is for the purpose of determining the image capture location n rather than for the purpose of recognizing the object200in the content of the image n. The crop area Rnis instead computed using the external knowledge encoded in the world model112, so that image recognition can be subsequently applied to the cropped image.

In the present example, that external knowledge is the location of the object200in the world that is encoded in the HD map112a, and its approximate dimensions encoded in the object model O.

At step208, each image n is cropped and rescaled, i.e. a rescaled and cropped image of the crop area Rnis generated, by extracting the subset of image data from the original image n that is contained within the crop region Rn.

This results in a cropped image Cn, containing a view of the object200as essentially a fixed scale, that is essentially independent of the object location X and the image capture location xn.

By way of example,FIG.2depicts cropped images Ca, Ccfor images a and c. In the original images a, c the scale of the object200is different because they have been captured at different distances from it. However, within the cropped images Ca, Cc, the scale of the object200is essentially the same, because that effect has been compensated by adjusting their respective crop areas Ra, Rcto account for the differences in the image capture locations xa, xc.

FIG.2Bshows further details of the cropping and rescaling operation applied to example images. The variable-sized crop region Rnfor each image n is rescaled to a transformed image Cnof fixed pixel dimensions M×N.

Here, the transformed image is relatively low resolution, such that, with sufficiently accurate localization, it may be possible to achieve the same object dimensions (m×n pixels) across the transformed images to within one pixel or so. As noted above, relatively low resolution images are sufficient for certain image recognition tasks in autonomous driving, such as the detection of visual signalling states designed to be readily perceptible (even from a distance and/or in poor weather, lighting conditions etc.).

The crop region Rnis computed as a function of the object projection Pn. For example, the centre of the crop region Rncould be defined to lie at the center of the object projection Pn, with a with and height that is some fixed multiple of the width and height of the object projection Pnrespectively. This means that, when the crop region is rescaled to M×N pixels, the pixels dimensions m×n of the object across all images will be essentially the same.

Other image processing can also be performed, using the object projection Pnas a reference. For example, a rotation applied to the image can be used to compensate for rotation of the object projection Pnin the image plane In.

If part of the crop region lies outside of the area of the original image n, as depicted for image c, pixels201of the transformed image Ccoutside of the original image can be set e.g. to black.

The position of the object200within the crop region Rcis a function of orientation localization error, i.e. error in an estimated orientation of the camera, whereas the scale of the object in the original image will be a function of position. Orientation error can mean that the object200is not centered in the transformed images, but this is immaterial provided a suitably large crop region is used (large enough to accommodate a realistic range of orientation localization errors). In some practical contexts, including autonomous driving using state of the art localization, it may be possible to achieve a higher level of accuracy and precision on position localization than orientation localization, which is acceptable in the present context because the main cause of rescaling errors would be position localization errors. In other words, a reduction in position localization error yields a reduction in rescaling errors, which is the material factor in this context of a light-weight image recognition component108. The cropping is a secondary element to improve efficiency, imply that orientation localization error is also a secondary consideration.

FIG.2Cshows an example of a transformed image Cnsupplied to the image recognition component108. The output of the image recognition component108comprises a detected 2D bounding box202for the object200in the transformed image In, and an associated state detection204(probabilistic/score-based or deterministic).FIG.3consider an annotation system for efficiently generating suitable training images for training the image recognition component108on this image recognition task.

FIG.3shows a schematic block diagram of an annotation system500that incorporates the described image transformation techniques. The annotation system500can be used to create annotated images suitable for training an image recognition component108of the kind described above with reference toFIGS.1and3, or for other purposes (such as scenario extraction). The annotation system500is shown to comprise a localisation component102, an object region computation component104and an image transformation component106. These are implemented within the annotation system500for the purpose of generating transformed (rescaled and cropped) images, so that the transformed images can then be annotated. Although deployed in the context of annotation, these components operate in exactly the same way as the equivalent components ofFIGS.1and3. For that reason, the same reference numerals are used, and all of the above description applied to these components in the context ofFIG.3. Similarly, reference numerals110,111and112are used to denote, respectively, the sensor data, image sequence and world model on which these operate, noting that, in the annotation system500, the images111and the sensor data110are used to generated training images or other transformed images to be annotated for some other use.

An annotation component504outputs annotation data for annotating a given transformed image as generated by the image transformation component106. A manual modification component is provided to allow a human annotator (user) to apply manual corrections to the annotation data.

A user interface501is shown to have inputs connected to respective outputs of the image transformation component106and the annotation component504to allow transformed images to be annotated in accordance with the annotation data. An output of the user interface (UI)501is shown connected to an input of a manual modification component502, representing the ability of the system500to receive manual annotation inputs at the UI501for applying manual corrections to the annotation data.

Reference numeral510is used to denote a transformed image generated by the image transformation component106according to the principles described above (equivalent to the transformed image Cnabove).

Transformed images are stored with their annotation data in an image database511, where they can be subsequently accessed.

FIG.3Ashows a schematic annotation view provided by the UI501on which a transformed image510is displayed. In the example ofFIG.3A, the annotation data comprises a 2D bounding box that is to define the location and extent of a view of an object200in the transformed image510. Recall the purpose of the rescaling by the image transformation component106is to rescale the object view to a fixed scale, such that is has essentially fixed pixel dimensions m×n in the transformed image510. Provided the world model112and ego localisation are sufficiently accurate, it should therefore be possible to predetermine the bounding box as m×n pixels in all cases. Recall also that the primary source of rescaling error is position localisation error. As noted, the accuracy and precision of position localisation that can now be achieved in the context of autonomous driving is more than sufficient to provide highly accurate rescaling in the present context.

In the absence of orientation localisation error, it should also be the case that the view of the object200appears in the transformed image510at the location of the object projection Pnused to generate the transformed image510(its expected location), e.g. the center point of the transformed image510if the original image is cropped to a region centred on the object projection Pn. However, orientation localisation error can have the effect of causing deviation in the actual location of the object200in the cropped image510from its expected location. With current state of the art vehicle localization, orientation localization error is generally expected to be higher than position error. In this case, a manual correction to the location of the bounding box may be needed (even if no correction of its dimensions is needed). The example ofFIG.3Ashows a bounding box512initially located at the centre of the cropped image510(the default or assumed bounding box). Reference numeral514denotes a corrected 2D bounding box, as defined by applying a manual correction to the default bounding box512, in order to better align it with the actual location of the object200in the image. In this particular example, no resizing of the default bounding box512is necessary, however, options may be provided for adjusting the size of the bounding box (typically by a small amount) if necessary.

Summarizing the above, in the context of annotation, an effect of the rescaling and cropping is to minimise the extent of manual corrections that are needed in the majority of cases. Because the image has been cropped and rescaled, the bounding box can be initially assumed to have a certain size and to be at a certain location relative to the transformed image510that should at least be relatively close to the actual object200in the cropped image.

FIG.4shows an extension of the annotation system to further improve the efficiency at which images can be annotated. In the system ofFIG.4, the user need only apply manual corrections to a relatively small number of transformed images, and the resulting corrected bounding boxes can then be interpolated or extrapolated through a sequence of transformed images to automatically determine corrected bounding boxes for other images of the transformed sequence.

InFIG.4, the annotation component504is shown to additionally comprise an interpolation component602that computes interpolated annotation data based on the user's manual corrections.

FIG.4Aschematically illustrates the principles of interpolated annotation by example. InFIG.4A, transformed images Ca, Cband Ccof the image sequence111are shown. In this example, the user applies manual corrections to transformed images Caand Cc, each of which consists, in the depicted example, of a translation of the bounding box to the correct object location in each of those images. The resulting correcting bounding boxes are then linearly interpolated and/extrapolated to compute interpolated bounding boxes for images between Caand Cc(including image Cbor images before Caand/or after Cc).

As noted, errors in the location of the bounding box—that is, deviation between the actual location of the view of the object200in the transformed image510from the location of the object projection Pn—arise from orientation localisation. Provided orientation error changes in an essentially linear manner across a reasonable sub-sequence of the images, then a simple linear interpolation of the corrected bounding boxes will suffice to provide accurate interpolated or extrapolated bounding boxes. Even if the orientation error is relatively large, provided it is essentially linear every reasonable number of images, the present interpolation techniques will be highly effective.

In other words, the transformation of the images using the above-described object projection techniques largely accounts for any non-linear effects of the vehicle's motion within the world. Therefore, the ability to use linear interpolation effectively in this context is a consequence of the way the images are transformed using the object projection techniques and world model112described above.

Once interpolated and/or extrapolated bounding boxes have been computed, the user can scroll through the sequence of images, overlaid with the interpolated or extrapolated bounding boxes as applicable, and if the user observes a point at which the interpolated or extrapolated bounding boxes are starting to deviate significantly from the object locations, he or she can apply a further correction that can then be used to interpolate or extrapolate beyond that image.

Overall, the effect is to provide a highly efficient annotation image annotation interface where a small number of manual corrections can be propagated via linear interpolation through a relatively large number of images in a sequence to provide high quality annotation data for the transformed images.

For any given transformed image Cb, an interpolated or extrapolated bounding box could be determined based on the corrected bounding boxes defined for images Caand Ccby linearly interpolating coordinates of those bounding boxes based on time index of image Cb(time b) relative to the time indexes of image Caand Cc(times a and c respectively). That is, based on the position if image Cbin the transformed sequence, relative to the images Caand Cc. The coordinates could, for example, be defining corner points (such as top right and bottom left, or top left and bottom right) or, if the dimensions of the bounding box are unmodified, a single coordinate (e.g. center point or single corner point).

FIG.5shows another extension of the annotation system500, in which manual corrections of the kind described with reference toFIG.4Aare used to refine the original ego localisation data in an offline context. In this respect, an output of the manual modification component502is shown connected to an input of the object region computation component104, and an output of the latter is shown connected to an input of the ego localisation component102, representing the ability of the system500to correct the ego localization data generated by the ego localization component102based on the user's bounding box corrections.

In the context of the annotation system500ofFIG.5, the world model112is assumed to be ground truth, i.e. no attempt is made to correct the world model112and it is assumed that any manual correction that the annotator has been required to make has arisen as a consequence of ego localisation error. Taking the world model112as a fixed ground truth, in turn, allows the ego localisation data to be refined based on the user's manual corrections. One way to achieve this is, once the user has defined manual corrections over a sufficient number of images, to back project the resulting corrected 2D bounding boxes into 3D space, and use that information to correct the ego localisation data in 3D space so that is it now consistent with the user's corrections. This is essentially the reverse of the projection from 3D into 2D space that is used to perform the rescaling and transformation. If those transformations were to be re-performed based on the corrected ego localisation data, the images should be rescaled and cropped in a way that is now substantially consistent with the user's manual corrections, i.e. the object would be exactly the right size and centred in the cropped image if the corrected ego localisation data were to be used to re-perform those operations.

For the reasons explained above, with current ego localization technology, it is expected that those corrections would mainly be to orientation, i.e. correcting orientation data of the ego localization data. In some cases, the system could be limited to orientation corrections (i.e. the ego position data is also taken as ground truth), which it be possible to implement based on a single correction, without needing to re-project into 3D space.

FIG.6shows a selection of transformed and cropped images that have been obtained using the described techniques, over a range of object distances from 8 metres to around 100 metres. These images are images of traffic lights captured from a sensor equipped vehicle, based on ego localisation and an HD map. It can be seen that the localisation has been performed with sufficient position accuracy to achieve consistent rescaling of the traffic light objects over this range of object distances, and that the depicted crop region is sufficient to accommodate the range of orientation localisation errors encountered over these images.

The images are shown annotated with 2D bounding boxes that have been defined using the techniques described with reference toFIGS.3A and4A.

Summarizing the above:

1. Rescaling errors are caused primarily by position localization errors;2. Cropping errors, i.e. deviation of the object from the center of the cropped images, are caused by orientation localization error, which may be larger—but this can be accommodated using a sufficiently large crop region;3. For those reasons, manual corrections might only be needed to correct the location (rather than the size) of the 2D bounding box—in practice, it has been found that rescaling errors are negligible with state of the art localization and with HD maps that are available today;4. InFIG.4(interpolation)—the linear interpolation of the user's bounding boxes works because the cropping/rescaling has removed most of the non-linear effects of the ego vehicle's motion.5. InFIG.5(refining ego localization data):a. a full implementation—where both ego orientation and position are refined—might require multiple corrections, projected back into 3D spaceb. but a simpler implementation, e.g. where ego orientation is corrected, could be done feasibly on a single image, without reprojection into 3D space.

The above considered annotation of transformed images. In this case, the image is transformed (e.g. scaled and cropped) to match the image to predefined annotation data (the m×n 2D bounding box assumed to lie at the center of the transformed image). However, the annotation techniques can be applied without such transformations. For example, the object projection can be used to annotate the object view within the original image. The above interpolation/extrapolation principles can still be applied in this context—each bounding box projection provides a “baseline” location in that frame, from which corrections to other frames can be extrapolated. This could, for example, be based on a manual correction vector, applied to a reference point (e.g. center point) of the object projection Pnas follows:use the world model to automatically compute an object projection Pnfor every frame n;the user would then correct the projection for, say, two frames—e.g. if the crop region for image n is determined automatically as (an, bn) where anis e.g. the top left corner and bnis the bottom right corner, the user would correct this for frames m and n as:
(am, bm)←(am, bm)+(Δam, Δbm)
(an, bn)←(an, bn)+(Δan, Δbn)

The object regions for other frames can them be automatically corrected as
(am+a, bm+a)=(am+a, bm+a)+(Δam+a, Δbm+a)

where (am+a, bm+a) is the initial crop region that you get from the world model, and (Δam+a, Δbm+a) is derived via linear interpolation or extrapolation of the correction vectors (Δam, Δbm), (Δan, Δbn) defined by the user from frames m and n.

References herein to components, functions, modules and the like, denote functional components of a computer system which may be implemented at the hardware level in various ways. This includes the components depicted inFIGS.1,3,5and6. A computer system comprises one or more computers that may be programmable or non-programmable. A computer comprises one or more processors which carry out the functionality of the aforementioned functional components. A processor can take the form of a general-purpose processor such as a CPU (Central Processing unit) or accelerator (e.g. GPU) etc. or more specialized form of hardware processor such as an FPGA (Field Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit). That is, a processor may be programmable (e.g. an instruction-based general-purpose processor, FPGA etc.) or non-programmable (e.g. an ASIC). Such a computer system may be implemented in an onboard or offboard context.

Practical applications of image recognition include autonomous vehicles and other robotic systems. The present techniques could also be implemented in simulation, e.g. for the purpose of testing and/or training components. In this context, the techniques could be applied to simulated (synthetic) image data generated using suitable sensor models, using simulated ego localization data