Patent Description:
The invention is described herein with reference to a data collection vehicle recording a route. However, the skilled person will appreciate that the invention is more widely applicable and may use route images collected in any way, including collating images from a variety of different sources.

Further, the invention is described herein in relation to autonomous or semi-autonomous vehicles driving through urban environments, but the skilled person will appreciate that the path proposals identified may be used for other purposes (for example, to identify a route for a person to walk), and that the techniques may be applied to non-urban environments. Nonetheless, the invention is currently expected to have particular utility in the field of autonomous vehicles driving in urban environments.

Road scene understanding is a critical component for decision making and safe operation of autonomous vehicles in urban environments. Given the structured nature of on-road driving, all autonomous vehicles must follow the "rules of the road"; crucially, driving within designated lanes in the correct direction and negotiating intersections.

Traditional methods of camera-based drivable path estimation for road vehicles involve pre-processing steps to remove shadow and exposure artefacts (see, for example, <NPL>, and<NPL>. ), extraction of low-level road and lane features (see, for example<NPL>, and<NPL>), fitting road and lane models to feature detections (see, for example, <NPL>, and <NPL>), and temporal fusion of road and lane hypotheses between successive frames (see, for example,<NPL>, and <NPL>).

While effective in well-maintained road environments, these approaches suffer in the presence of occlusions, shadows and changing lighting conditions, unstructured roads and areas with few or no markings (see <NPL>). Robustness can be significantly increased by combining images with radar (see <NPL>) or LIDAR (see <NPL>) but at an increased sensor cost.

More recently, advances in image processing using deep learning (see <NPL>) have led to impressive results on the related problem of semantic segmentation, which aims to provide per-pixel labels of semantically meaningful objects for input images (see, for example <NPL>, <NPL> and the paper of V. Badrinarayanan et al cited above). Deep networks make use of the full image context to perform semantic labelling of road and lane markings, and hence are significantly more robust than previous feature-based methods. However, for automated driving these approaches depend on large-scale manually-annotated road scene datasets (notably CamVid (<NPL>) and Cityscapes (<NPL>), consisting of <NUM> and <NUM>,<NUM> labelled frames respectively), which are time-consuming and expensive to produce.

The challenges in building large-scale labelled datasets have led some researchers to consider virtual environments, for which ground truth semantic labels can be rendered in parallel with synthetic camera images. Methods using customised video game engines have been used to produce hundreds of thousands of synthetic images with corresponding ground truth labels (see <NPL>, and <NPL>). While virtual environments allow large-scale generation of ground truth semantic labels, they present two problems: firstly, rendering pipelines are typically optimised for speed and may not accurately reflect real-world images (both above approaches suggest rendered images are used only for augmenting real-world datasets and hence manual labelling is still necessary for at least a sub-set of the datasets); secondly, the actions of the vehicle and all other agents in the virtual world must be pre-programmed and may not resemble real-world traffic scenarios.

A recent method uses sparse 3D prior information to transfer labels to real-world 2D images (see <NPL>) but requires sophisticated 3D reconstructions and manual 3D annotations.

Some approaches have proposed bypassing segmentation entirely and learning a direct mapping from input images to vehicle behaviour (see <NPL>, and <NPL>). These methods also use the driver of the data collection vehicle to generate a label for each image, so generating the supervised labels for the network (e.g. a single steering angle value per image) and have recently demonstrated impressive results in real-world driving tests (see <NPL>), but it is not clear how this approach generalises to scenarios where there are multiple possible drivable paths to consider (e.g. intersections). This approach uses a convolutional neural network to map raw pixels from a single front-facing camera directly to steering commands; there is no segmentation of a proposed path extending into the future/image.

Current commercial systems that perform driver assistance and on-road autonomy typically depend on visual recognition of lane markings and explicit definitions of lanes and traffic rules, and therefore rely on simple road layouts with clear markings (e.g. well-maintained highways). See, for example, <NPL>, and <NPL>.

To extend these systems beyond multi-lane highways to complex urban environments and rural or undeveloped locations without clear or consistent lane markings, an alternative approach is proposed.

The document by<NPL>, discloses a method of generating data for autonomous robot navigation system using odometry data collected along a path where the robot has previously driven.

According to a first aspect of the invention, there is provided a method of training a segmentation unit according to claim <NUM>.

The method may require little or no supervision, and/or may not require manual labelling of images.

The method comprises obtaining data from a data collection vehicle driven through an environment, the data comprising:.

The method comprises using the obstacle sensing data to label one or more portions of at least some of the images as obstacles.

The method comprises using the vehicle odometry data to label one or more portions of at least some of the images as the path taken by the vehicle through the environment.

Conveniently, the training dataset is created from the labelled images, and may or may not be constituted by the labelled images (i.e. the creation of the training dataset may simply comprise collating the labelled images, without the addition of any further data or image processing, or there may be additional data).

The method may further comprise a calibration process to allow the odometry data and the obstacle sensing data to be matched to the images.

According to another aspect of the invention, there is provided a segmentation unit trained in accordance with claim <NUM>.

According to another aspect of the invention, there is provided an autonomous vehicle as defined by claim <NUM>.

The vehicle comprises a sensor arranged to capture images of an environment around the autonomous vehicle.

A route of the autonomous vehicle through the environment is determined by the segmentation unit using images captured by the sensor. In some embodiments, additional systems may be used to make a decision based on the path proposals from the segmentation unit.

The autonomous vehicle may only comprise a monocular camera, as monocular camera data is sufficient for the trained segmentation unit.

The person skilled in the art will appreciate that, in some embodiments, the data collection vehicle may also be the autonomous vehicle of the further aspect of the invention. In such embodiments, obstacle sensing data and odometry data may or may not be recorded whilst driving autonomously. A different vehicle is therefore not required.

Features described in relation to one of the above aspects of the invention may be applied, mutatis mutandis, to the other aspect of the invention. Further, the features described may be applied to the or each aspect in any combination.

There now follows by way of example only a detailed description of embodiments of the present invention with reference to the accompanying drawings in which:.

In the figures, like reference numerals are used to reference like components.

Embodiments of the invention are described in relation to a sensor <NUM> mounted upon a vehicle <NUM>, as is shown in <FIG>. The skilled person would understand that the vehicle <NUM> could be replaced by a plane, boat, aerial vehicle or robot, or by a person carrying a sensor <NUM>, amongst other options. In still other embodiments, the sensor used may be stationary. Further, any feature or combination of features described with respect to one embodiment may be applied to any other embodiment.

The embodiment being described utilises a weakly-supervised approach <NUM> to segmenting path proposals for a road vehicle <NUM> in urban environments given a single monocular input image <NUM>. Weak supervision is used to avoid expensive manual labelling by using a more readily available source of labels instead. In the embodiment being described, weak supervision involves creating labels of the proposed path in images <NUM> by leveraging the route actually travelled by a road vehicle <NUM>. In the embodiment being described, the labels are pixel-wise; i.e. pixels of an image <NUM> are individually labelled.

The approach <NUM> is capable of segmenting a proposed path <NUM> for a vehicle <NUM> in a diverse range of road scenes, without relying on explicit modelling of lanes or lane markings. The term "path proposal" is defined as a route a driver would be expected to take through a particular road and traffic configuration.

The approach <NUM> described herein uses the path taken 14a by the data collection vehicle <NUM> as it travels through an environment to implicitly label proposed paths <NUM> in the image <NUM> in the training phase, but may still allow a planning algorithm to choose the best path <NUM> for the a route in the deployment phase.

A method <NUM> of automatically generating labelled images <NUM> containing path proposals <NUM> by leveraging the behaviour of the data collection vehicle driver along with additional sensors 12a, 12b mounted to the vehicle <NUM>, is described, as illustrated in <FIG> and <FIG>. The skilled person will appreciate that the data collection vehicle <NUM> could be an autonomous vehicle, with no driver, in some embodiments.

Using this approach <NUM>, vast quantities of labelled training data <NUM> can be generated without any manual annotation, spanning a wide variety of road and traffic configurations under a number of different lighting and weather conditions limited only by the time spent driving the data collection vehicle <NUM>. This labelled training data <NUM> can be thought of as weakly-supervised input for training a path segmentation unit <NUM>. In this case, the only "supervision" or supervisory signal is the behaviour of the data collection vehicle driver; the driver itself may be an autonomous unit. In particular, the only "supervision" or supervisory signal used, by the embodiment being described to label training date, may be the movements of the data collection vehicle; manual seeding or labelling of training images may therefore be substantially or completely avoided.

Embodiments of the invention as disclosed herein relate not only to the method of generating a training dataset <NUM> described, but also to the resultant training dataset itself, and to applications of that dataset. The method <NUM> described allows a training dataset to be generated without any manual labeling - either of each training image, or of one training image (or a subset of training images) which is then used as a seed which allows labels to be propagated to other images.

The skilled person will appreciate that a set of labeled images <NUM> generated by the method <NUM> disclosed herein may form part of a training dataset which also includes manually labeled images, images labeled by a different technique and/or unlabeled images.

The training dataset <NUM> produced is arranged to be used in autonomous route determination. In particular, the training dataset shows examples of paths 14a within images <NUM> which were chosen by a driver (or an autonomous vehicle <NUM>). A segmentation unit <NUM> (such as SegNet - <NPL>. ) trained on the training dataset <NUM> is therefore taught to identify a path <NUM> within an image <NUM> that would be likely to be chosen by a driver.

In <FIG> and <FIG>, the data collection vehicle <NUM> is equipped with a camera 12a and odometry and obstacle sensors 12b. The vehicle <NUM> is used to collect data <NUM>, <NUM>, <NUM> during normal driving (first (ie leftmost) part of <FIG>). In the embodiment being described, the data <NUM>, <NUM>, <NUM> comprises odometry data <NUM>, obstacle data <NUM> and visual images <NUM>. In other embodiments, other data may generated and for example, it is possible that the visual images, in particular, may be replaced with other representations of the environment, such as LiDAR scans, or the like.

The visual images <NUM> obtained by the data collection vehicle <NUM> are described as training images <NUM> as these are labelled as described below and then used to train one or more systems or devices to identify path proposals in other images (that is proposed paths for a vehicle to traverse the environment contained within the visual images <NUM>).

In the embodiment being described the training images <NUM> are used to train a segmentation framework (in particular a deep semantic segmentation network <NUM>), which may be a neural network such that the trained segmentation framework can predict routes likely to be driven by a driver who contributed to the training dataset in new/unknown scenes.

The segmentation framework, or the output therefrom, can then be used to inform a route planner. The skilled person will appreciate that route planners generally operate by minimising a cost function so as to provide a recommended route. In some embodiments, the segmentation framework is arranged to output a suitable cost function for a route planner, or suitable parameters for a cost function of a route planner.

The training data <NUM> can therefore be used to train a system able to predict routes likely to be driven by the original driver through a scene at hand. That system can then be used to inform a trajectory planner/route planner (e.g. via mapping the route proposal into a planner cost function). The planner may or may not be separate from the trained system.

In alternative embodiments, a single sensor <NUM> may provide both odometry <NUM> and obstacle <NUM> data. In yet further embodiments, more sensors may be provided.

The odometry <NUM> and obstacle data <NUM> is projected into the training images <NUM> to generate weakly-supervised labels <NUM> relevant for traversing through the environment, such as used in autonomous driving. The chosen labels are "Unknown", "Obstacles" and "Path" in the embodiment being described. In alternative or additional embodiments, more, fewer or different labels may be used. For example, "Unknown" may not be used, and/or "Obstacles" may be subdivided by obstacle type. The labels are used to classify regions of the training images <NUM>.

In the Figures, diagonal lines sloped at an acute angle to the horizontal, as measured from the right hand side (///), are used to mark regions identified as "Unknown" <NUM>, diagonal lines sloped at an obtuse angle to the horizontal, as measured from the right hand side (\\\), are used to mark regions identified as "Obstacles" 15a, and broken diagonal lines are used to mark regions identified as "Path" 14a.

The labelled images <NUM> are, in the embodiment being described, used to train a deep semantic segmentation network <NUM>. The skilled person will appreciate that, although a deep semantic network <NUM> was used in the embodiment being described, other machine learning systems may be used as a segmentation unit <NUM> instead of or as well as a deep semantic network <NUM> (any of these may be referred to as a segmentation network <NUM>). The skilled person will appreciate that, in various embodiments, the segmentation unit <NUM> may be implemented in software and may not have unique hardware, or vice versa.

At run-time, a vehicle <NUM> equipped with only a monocular camera 12a can perform live segmentation of the drivable path eg 14a and obstacles 15a using the trained segmentation network <NUM> (second part of <FIG>), even in the absence of explicit lane markings. The skilled person will appreciate that the vehicle <NUM> used at run-time may be the same as that used for data collection, or a different vehicle. Further, although a monocular camera 12a is sufficient, alternative or additional sensors may be used.

In the embodiment being described, the data was used to train an off-the-shelf deep semantic segmentation network <NUM> (e.g. SegNet, see the paper of V. Badrinarayanan et al. cited above) to produce path proposal segmentations <NUM> using only a monocular input image <NUM> (e.g. a photograph). The deep semantic segmentation network <NUM> may then be used as part of, or to feed into, a route planner, which may be an in-car or portable device used to suggest routes to a driver/user.

The approach <NUM> was evaluated using two large-scale autonomous driving datasets: the KITTI dataset (see <NPL>), collected in Karlsruhe, Germany, and the large-scale Oxford RobotCar Dataset (http://robotcar-dataset. uk), consisting of over <NUM> of recorded driving in Oxford, UK, over the period of a year.

For each of these datasets, additional sensors 12a, 12b on the vehicle <NUM> and the trajectory taken by the driver were used as the weakly-supervised input to train a pixel-wise semantic classifier. Segmentation results are presented on the KITTI Road (<NPL>), and with Object and Tracking benchmarks, and the performance under different lighting and weather conditions is investigated using the Oxford dataset.

In the following section, an embodiment of the invention for generating weakly-supervised training data <NUM> for proposed path segmentation using video and other sensor data recorded from a (in the embodiment being described) manually-driven vehicle <NUM> is outlined. In some embodiments, the vehicle <NUM> used to record video and other sensor data (the data collection vehicle <NUM>) may be autonomously driven, but embodiments of the invention are described below in relation to a manually driven data collection vehicle <NUM> as this was the arrangement used for the test results described. The skilled person will appreciate that other factors specific to the test performed may be varied or eliminated in other embodiments of the invention.

As regions within an image <NUM> corresponding to the route taken 14a by the vehicle <NUM> and to obstacles 15a are identified and marked, the image <NUM> is described as being segmented, forming a segmented image <NUM>. The segmented images <NUM> can be used as training data.

Once trained on the training data <NUM>, a segmentation unit <NUM> can then segment new images <NUM> in the same way, so forming new segmented images <NUM>.

The segmented images <NUM> formed at run-time may optionally be added to the training data <NUM> and used in the training phase thereafter, optionally subject to driver/user approval.

In addition to a monocular camera 12a to collect input images <NUM>, <NUM> (i.e. images of the environment around the data collection vehicle <NUM>), the approach <NUM> of the embodiment being described uses the following two capabilities of the data collection vehicle <NUM>:.

A vehicle odometry system 12a and an obstacle sensing system 12b may be mounted on, or integral with, the data collection vehicle <NUM>. In either case, the sensing systems 12a, 12b move with the vehicle <NUM> and may therefore be described as being onboard the vehicle <NUM>.

Note that the vehicle odometry <NUM> and obstacle sensing <NUM> capabilities used for collecting training data <NUM>, <NUM> are not required when using the training data, nor when operating an autonomous vehicle <NUM> using a segmentation unit <NUM> trained on the training data; the resulting network can operate with only a monocular input image <NUM>, although the skilled person will appreciate that additional sensors and/or data inputs may also be used.

<FIG> illustrates the sensor extrinsics for a vehicle <NUM> equipped with a stereo camera 12a and LIDAR sensor 12b. The skilled person will appreciate that other camera types may be used to collect the input images <NUM> in other embodiments, and/or that a camera 12b forming part of a visual odometry system 12b may provide the images <NUM>.

<FIG> shows the data collection vehicle <NUM> equipped with a camera C 12a and obstacle sensor L 12b, e.g. a LIDAR scanner. The extrinsic transform GCL between the camera 12a and LIDAR scannner 12b is found using a calibration routine. The contact point c{l,r} of the left (l) and right (r) wheels on the ground relative to the camera frame C is also measured at calibration time. At time t, the LIDAR scanner observes a number of points <MAT> on obstacles <NUM>, including other vehicles <NUM> on the road. The relative pose GCtCt+<NUM> of the camera between time t and t+<NUM> is determined using vehicle odometry, e.g. using a stereo camera. The relative pose GCtCt+<NUM> can be used to determine the motion of the vehicle <NUM>.

In the embodiment being described, pixels of images are assigned to, and labelled as being part of, one or more classes (ie they have class labels associated with them). To generate class labels for pixels in the input image(s) <NUM>, recorded data from the data collection vehicle <NUM> driven by a human driver in a variety of traffic and weather conditions is used in the embodiment being described. The classes/labels described herein correspond to obstacles <NUM> in the environment which are in front of the data collection vehicle <NUM>, path taken <NUM> by the data collection vehicle <NUM> through the environment, and "unknown" for the remainder <NUM>, i.e. for any unlabeled area(s). In alternative embodiments, fewer classes may be used (e.g. not using "unknown") and/or more classes may be used (e.g. separating dynamic and static obstacles, or obstacles in front of the vehicle from other obstacles).

In the embodiment being described, each pixel is assigned to a class <NUM>, <NUM>, <NUM>. In alternative embodiments, some pixels may be unassigned, and/or averages may be taken across pixel groups. In this way, different portions of an image <NUM>, <NUM> are labeled as belonging to different classes (or not labeled, in some embodiments). There may be more than one portion of an image <NUM>, <NUM> relating to the same class within a single image; for example, an obstacle on the road as well as obstacles on either side of the road. In some images <NUM>, <NUM>, there may be no portion(s) in a particular class; for example where the image represents a portion of the environment fully within the vehicle's path, with no obstacles <NUM> or unknown areas <NUM>.

The general approach of methods that learn to drive by demonstration is used for the embodiment described herein (see, for example <NPL>, and <NPL>), and it is assumed that the proposed path <NUM> corresponds to the one chosen by the driver of the data collection vehicle <NUM> in each scenario. Labels <NUM>, <NUM>, <NUM> are then generated by projecting the future path of the vehicle <NUM> into each image <NUM>, over which object labels as detected by the LIDAR scanner are superimposed as follows. "Future" in this context means after the image in question <NUM> was taken - the "future" path is the path driven by the vehicle <NUM>, as recorded by the odometry system 12b during the training phase (first part of <FIG>), and is the proposed path for a vehicle <NUM> to take in the deployment phase (second part of <FIG>).

The segmentation unit <NUM> segments new images <NUM> (i.e. images not forming part of the training dataset) provided to it in accordance with its training, thereby marking proposed paths <NUM> within the new images <NUM>.

The segmentation unit <NUM> may be onboard an autonomous (or semi-autonomous) vehicle <NUM> and arranged to receive images <NUM> taken by a camera 12a on the autonomous vehicle <NUM> and to process those images so as to propose a path <NUM> by segmentation <NUM> of the image. The route proposals <NUM> may be provided in real-time so as to enable the output of the segmentation unit <NUM> to be used in directing the vehicle <NUM>.

The embodiments being described use real images <NUM> and vehicle path data <NUM> for the training dataset/to train the system. The supervisory signal as to a proposed path <NUM> is the path 14a actually driven by the data collection vehicle <NUM>, projected into the image <NUM> as a label (shown in <FIG>. That combined with the obstacle labels <NUM> allows an informative training image <NUM> and general representation to be generated.

The embodiments described allow for multiple proposed paths <NUM> (for example left and right at an intersection) and obstacle segmentations <NUM>. In some embodiments, an additional system may be used after the segmentation to decide how to utilise the path proposals <NUM>.

The skilled person will appreciate that the labeled images <NUM> produced may be used as a test dataset as well as, or instead of, as a training dataset. A trained segmentation unit <NUM> could be given the images <NUM> of the test dataset without segmentation information <NUM> and the output of the trained segmentation unit can then be compared to the segmented images <NUM> of the test dataset to assess the performance of the segmentation unit <NUM>. Any features described with respect to the training data set may therefore be applied equally to a test data set.

Proposed path projection: To project the future path 14a of the vehicle <NUM> into the current frame <NUM>, the size of the vehicle <NUM> and the points of contact with the ground during the trajectory are used. The position of the contact points c{l,r} of the front left and right wheels on the ground relative to the camera C may be determined as part of a calibration procedure. The position of the contact point c{l,r} in the current camera frame Ct after k frames is then found as follows: <MAT> where K is the perspective projection matrix for the camera C and GCtCt+k is the SE(<NUM>) chain of relative pose transforms formed by vehicle odometry from frame t to frame t + k as follows: <MAT>.

Proposed path pixel labels <NUM> are then formed by filling quadrilaterals in image coordinates corresponding to sequential future frames. The vertices of the quadrilateral are formed by the following points in camera frame Ct: <MAT> where the index variable j = {<NUM>. An illustration of the proposed path projection and labelling process is shown in <FIG>.

<FIG> shows ground contact points <NUM> (top two images) and obstacle points <NUM> (bottom two images) projected into images <NUM>. At time t, ground contact points c{l,r},j (top left, <NUM>) corresponding to the path of the vehicle up to k frames ahead are projected into the current image (top left, <NUM>). Pixel labels corresponding to drivable paths <NUM> are filled in by drawing quadrilaterals between the left and right contact points between two successive frames (top right). At the same time, obstacle points <MAT> <NUM> from the current LIDAR scan are projected into the image <NUM> (bottom left). Pixel labels corresponding to obstacles are formed by extending each of these points to the top of the image (bottom right, <NUM>). Note that the top and bottom sections of the image <NUM> corresponding to the sky and vehicle bonnet are removed before training in this embodiment.

The choice of frame count k depends on the look-ahead distance required for path labelling and the accuracy of the vehicle odometry system 12b used to provide relative frame transforms. In practice k is chosen such that the distance between first and last contact points ∥GCtCt+kc{l,r} - c{l,r}∥ exceeds roughly <NUM> metres. Different camera setups with higher viewpoints may require greater path distances, but accumulated odometry error will affect far-field projections. In other embodiments, distances such as roughly any of the following may be chosen: <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or any distance inbetween.

<FIG> shows proposed path labels <NUM> for an input image (top) before (middle) and after (bottom) applying obstacle labels from the LIDAR scanner 12b. Without the obstacle labels <NUM>, the proposed path <NUM> intersects vehicles <NUM> (or cyclists <NUM>, pedestrians <NUM> or the likes) in the same lane as the path driven by the data collection vehicle <NUM>, which in this case will erroneously label sections of the white van <NUM> as drivable route <NUM>. Adding labels for obstacles <NUM> ensures that dynamic objects including the van <NUM>, cyclist <NUM> and pedestrian <NUM> are marked as non-drivable, leading to a different proposed path <NUM>. Note that static obstacles <NUM> such as the road sign 49a and the building 49b are also labelled as obstacles <NUM>, which correctly handles occlusions (e.g. as the path turns right after the traffic lights 49c). The remaining portion, <NUM>, may be labeled unknown.

Obstacle projection: For some applications it may be sufficient to use just the proposed path labels <NUM> to train a semantic segmentation network <NUM>. However, for on-road applications in the presence of other vehicles <NUM> and dynamic objects, a naive projection of the path driven will intersect vehicles <NUM> in the same lane and label them as drivable paths <NUM> as illustrated in <FIG>. In the centre figure of <FIG>, it can be seen that the path <NUM> is intersected with the vehicle <NUM>.

Intersecting paths with vehicles in this manner may lead to catastrophic results when the labelled images <NUM> are used to plan paths for autonomous driving, since vehicles and traffic may be labelled as traversable by the network.

The obstacle sensor 12b mounted on the vehicle <NUM>, in this case a LIDAR scanner, is used for obstacle detection. Each 3D obstacle point <MAT> observed at time t is projected into the camera frame Ct as follows: <MAT> Where K is the camera projection matrix and GCL is the SE(<NUM>) extrinsic calibration between the camera and LIDAR sensor.

In the embodiment being described, for each camera-frame point <MAT>, an approach inspired by "stixels" (see the paper of D. Pfeiffer and U. Franke listed above, and also <NPL>) is used, and all pixels in the image on and above the point are labelled as an obstacle <NUM>. This helps to ensure that all locations above and behind the detected obstacle <NUM> are labelled as non-drivable, as illustrated in the third Figure of <FIG> and as discussed relative to <FIG>.

Obstacle pixel labels <NUM> take precedence over proposed path labels <NUM> in the embodiment being described to facilitate correct labelling of safe drivable paths/the chosen path <NUM> as illustrated in <FIG>.

In most images <NUM>, <NUM>, there will be locations <NUM> labelled as neither proposed path nor obstacle. These correspond to locations <NUM> which the vehicle10 has not traversed (and hence is not known to be traversable and is not part of the chosen path <NUM>), but no positive identification of obstacles <NUM> have been made. Typically these areas correspond to the road area outside the current lane (including lanes for oncoming traffic), kerbs, empty pavements and ditches. These locations are referred to as "unknown area" <NUM>, as it is not clear whether the vehicle <NUM> should enter these spaces during autonomous operation; this would be a decision for a higher-level planning framework as discussed below. Examples of unknown areas <NUM> can be seen in <FIG>, in which region <NUM> is marked as unknown - this region <NUM> comprises road surface over which the vehicle <NUM> has not driven and some pavement area. Similarly in <FIG>, pavement and unused road areas are classed as unknown areas <NUM>. The grass <NUM> in <FIG> is also classed as unknown <NUM>.

Once proposed path <NUM>, obstacle <NUM> and unknown area <NUM> labels are automatically generated for a large number of recorded images <NUM>, these labelled images <NUM> can be used to train a semantic segmentation network <NUM> to classify new images <NUM> from a different vehicle <NUM> (or from the same vehicle, in some embodiments). In the example described, this different vehicle <NUM> was equipped with only a monocular camera 12a.

SegNet is used: a deep convolutional encoder-decoder architecture for pixel-wise semantic segmentation. Although higher-performing network architectures now exist (e.g. <NPL>), and others could be used, Seg-Net provides real-time evaluation on consumer GPUs, making it suitable for deployment in an autonomous vehicle <NUM>.

The weakly-supervised labelling approach <NUM> being described can generate vast quantities of training data <NUM>, limited only by the length of time spent driving the data collection vehicle <NUM>. However, the types of routes driven will also bias the input data <NUM>, <NUM>, <NUM>. As most on-road driving is performed in a straight line; a random sample of the training data will consist mostly of straight-line driving. In practice the data were subsampled to <NUM>, before further subsampling based on turning angle. For each frame, the average yaw rate Δφ per frame was computed for the corresponding proposed path as follows: <MAT> where φ(G) is a function that extracts the Euler yaw angle φ from the SE(<NUM>) transform matrix B. In the example described, a histogram of average yaw rates was then built and random samples taken from the histogram bins to ensure an unbiased selection of different turning angles.

In the tests of the embodiment of the invention being described, two different model segmentation networks were built for evaluation: one using the KITTI Raw dataset and one using the Oxford RobotCar dataset. These datasets were collected using different vehicles with different sensor setups, summarised in Table I:.

Both vehicles <NUM> are equipped with stereo camera systems 12a and the stereo visual odometry approach described in the PhD thesis of W. Churchill cited above is used to compute the relative motion estimates required in Eq. <NUM>.

The images <NUM> from the cameras 12a are cropped and downscaled to the resolutions listed in Table I before training.

The Oxford RobotCar is equipped with a SICK LD-MRS LIDAR scanner/sensor 12b, which performs obstacle merging and tracking across <NUM> scanning planes in hardware. Points identified as "object contours" are used to remove erroneous obstacles due to noise and ground-strike. The Velodyne HDL-64E mounted on the AnnieWAY vehicle does not perform any object filtering, and hence the following approach is used to detect obstacles: a ground plane is fitted to the 3D LIDAR scan using MLESAC (see <NPL>), and treat all points more than roughly <NUM> above this plane as obstacles <NUM>, as illustrated in <FIG>. This approach effectively identifies obstacles <NUM> the vehicle <NUM> may collide with even in the presence of pitching and rolling motions. The skilled person will appreciate that heights other than <NUM> may be chosen as appropriate. For example, the skilled person will appreciate that roughly any of the following may be suitable: <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>.

<FIG> shows obstacle labelling using Velodyne data for the KITTI dataset. Raw Velodyne scans (top image) contain returns from the road surface as well as nearby obstacles. A scanned region 5a and an unscanned region 5b of the image <NUM> are shown. The Velodyne scan data are indicated by light-coloured lines throughout the scanned region 5a.

A ground plane is fitted using MLESAC, and only points of the Velodyne scan data <NUM> above the plane are maintained (middle image). This is represented in the figures by removal of the light-coloured lines in the regions below <NUM> from the ground plane.

Pixels <NUM> which correspond to areas of the Velodyne scan data <NUM> or more above the ground plane are then labelled as obstacles using the approach described above (bottom image) to ensure accurate labels on obstacles <NUM> while retaining potentially drivable surfaces <NUM> on the ground (shaded pixels with \\ in bottom image). One or more areas of the potentially drivable surfaces <NUM> may then be labelled as path <NUM> and/or as unknown <NUM> using the approach described herein.

The camera-LIDAR calibration GCL for the RobotCar vehicle <NUM> was determined using the method in <NPL>; for the AnnieWAY vehicle the calibration provided with the KITTI Raw dataset was used.

For the KITTI model, use was made of the available City, Residential and Road data from the KITTI Raw dataset. For the Oxford model, a diverse range of weather conditions for each traversal of the route were selected, including nine overcast, eight with direct sun, four with rain, two at night and one with snow; each traversal consisted of approximately <NUM> of driving. The number of labelled images <NUM> used to train each model is shown in Table II and some examples are shown in <FIG>. In total, <NUM>,<NUM> images were used to train the KITTI model, and <NUM>,<NUM> images for the Oxford model.

<FIG> shows example training images <NUM> with weakly-supervised labels <NUM>, <NUM>, <NUM> from the KITTI (top) and Oxford (bottom) datasets. The weakly-supervised approach <NUM> generates proposed path <NUM> and obstacle labels <NUM> for a diverse set of locations in the KITTI dataset, and a diverse set of conditions for the same location in the Oxford dataset. The remainder is labelled as unknown area <NUM>. No manual annotation is required to generate the labels.

For both datasets, semantic classifier models were built using the standard SegNet convolutional encoder-decoder architecture. The same SegNet parameters were used for both datasets, with modifications only to account for the differences in input image resolution. The input data were randomly split into <NUM>% training and <NUM>% validation sets, training performed for <NUM> epochs then the best-performing model selected according to the results on the validation set. The training time totalled ten days for KITTI using a GTX Titan GPU and twenty days for Oxford using a GTX Titan X GPU; future training times can be reduced using a different architecture or making use of a GPU cluster.

For the comparison using the KITTI Road benchmark presented below (ego-lane segmentation), an additional SegNet model was trained on only the training images provided for the Ego-Lane Estimation Evaluation. Note that these ground truth images <NUM> were not provided to the model segmentation unit <NUM> trained using the weakly-supervised approach <NUM> described above. For the object detection evaluation using the KITTI Object and Tracking datasets, there was no overlap between images selected to train the weakly-supervised labels and the images with ground truth labels used in the evaluation.

For reliable on-road driving, the semantic segmentation <NUM> preferably functions in multiple environments under the range of lighting, weather and traffic conditions encountered during normal operation. This Results section provides an evaluation of the performance of both the KITTI model and Oxford model under a range of different test conditions.

The Oxford model was evaluated by generating ground truth labels for a further four datasets not used for training, consisting of <NUM>,<NUM> images in sunny conditions, <NUM>,<NUM> images in cloudy conditions, <NUM>,<NUM> images collected at night and <NUM>,<NUM> images collected in the rain, for a total of <NUM>,<NUM> test images. Table III presents the segmentation results for the three classes in each of the four different conditions in the test datasets listed above, with the "All" column showing the mean for each metric across all classes.

In Table III, the following metrics widely used in the field are used to quantify performance. Generated segmentations <NUM> are compared to ground truth segmentations <NUM> for the same image <NUM>.

The intersection over union, IoU, values are a measure of the overlap between regions in a class in the generated segmentation and in the ground truth. A ground truth bounding box often will not exactly coincide with a bounding box determined by a labelling system. The question is, how much can they be offset against each other, and how much can they vary in size in order to still count as 'matched' to each other (as in pertaining to the same object)? IoU computes the ratio of the intersection of the areas covered by both boxes to the area of the union of both boxes, and can be applied to more general image segments (as in this case) instead of to bounding boxes per se.

Precision, PRE, is the fraction of class detections which truly are of that class. This may be pixel-based, for example, "<NUM> out of <NUM> pixels labelled as being obstacles were actually obstacles".

Recall, REC, is the fraction of the class instances present in the data that were successfully detected. For example, if the ground truth segmentation indicates that there are ten obstacles in an image, how many of these were found? Again, this metric may be pixel-based instead of based on a number of objects.

The model (ie segmentation unit) <NUM> provides good performance across the different conditions with mean intersection-over-union (IoU) scores exceeding <NUM>% in all cases, with the highest performance in cloudy weather and lowest at night, due to the reduced image quality in low-light conditions.

<FIG> illustrates the output of the segmentation unit <NUM> for four images of the same location under different conditions. Despite significant changes in lighting and weather, the segmentation network <NUM> correctly determines the proposed path <NUM> through the crossing and identifies obstacles (e.g. construction barriers <NUM>). <FIG> shows semantic segmentation on frames <NUM> captured at the same location under different conditions.

Despite significant changes in appearance between sunny (<FIG>), rainy (<FIG>), snowy (<FIG>) and night-time (<FIG>) conditions, the network <NUM> correctly segments the proposed drivable path <NUM> and labels obstacles <NUM> including cyclists 74a, other vehicles 74b and road barriers 74c, <NUM>.

This result demonstrates that the weakly-supervised approach <NUM> can be used to train a single segmentation network <NUM> that segments proposed paths <NUM> and obstacles <NUM> across a wide range of conditions without explicitly modelling environmental changes due to lighting, weather and traffic.

<FIG> presents a number of locations where the segmentation network <NUM> proposed a valid path <NUM> in the absence of explicit road or lane markings, instead using the context of the road scene to infer the correct route. <FIG> shows path proposals <NUM>, <NUM> in locations without explicit lane dividers or road markings. Using the context of the road scene the segmentation network <NUM> infers the correct proposed path (top, middle), even for gravel roads 82a never seen in the training data (bottom).

Thus, <FIG> gives three examples (top, middle and bottom). Each example is provided as two images, a leftmost image and a rightmost image. The leftmost image is shown un-marked, whilst the classes of pixel after segmentation (path <NUM>, <NUM>; obstacle <NUM>; and unknown <NUM>) are shown on the rightmost image.

To demonstrate how the weakly-supervised labelling approach <NUM> disclosed herein can lead to useful performance for autonomous driving tasks, it was evaluated on two different benchmarks from the KITTI Vision Benchmark Suite (http://www. net/datasets/kitti/): ego-lane segmentation and object detection.

However, neither of these benchmarks are an exact match for the segmentation results provided by the segmentation network <NUM>, as they were designed for different purposes; alternative metrics based on the provided ground truth are presented to quantitatively evaluate the approach <NUM>.

An additional SegNet model was trained on the provided ground truth training images to compare to the segmentation unit <NUM> trained on weakly-supervised labelled images, as detailed above. The results of both the SegNet model and the segmentation unit <NUM> on the KITTI website benchmark are shown in Table IV. As is standard in the field, FPR means a false positive rate, FNR means a false negative rate. MaxF is Maximum F1 measure, F1 being a measure of a test's accuracy. Both the precision (PRE) and the recall (REC) of the test are considered to compute the F1 measure. Average precision (AP), as defined in '<NPL>. ', is computed for different recall values to provide insights into the performance over the full recall range.

<FIG> shows example ego-lane segmentation results obtained using the KITTI Road dataset. For the given input image <NUM> (top image), a SegNet model trained on the small number of manually-annotated ground truth images (middle image) performs poorly in comparison with the model segmentation unit <NUM> trained on the much larger weakly-supervised dataset (bottom image) generated without manual annotation.

Thus, <FIG> illustrates a sample network output for both models and the weakly-supervised segmentation unit <NUM> outperforms the model trained on the provided ground truth images, with a <NUM>% increase in max F score and <NUM>% increase in precision exceeding <NUM>% in total, despite the embodiment being described not making use of manually annotated ground truth images or explicit encoding of lane markings. Although the overall performance is not competitive with those generated by more sophisticated network architectures on the KITTI leaderboard (due to the different definition of ego-lane and proposed path), this result strongly indicates that the weakly-supervised approach <NUM> generates segmentations <NUM>, <NUM> useful for real-world path planning.

The differences in the number of training images used for each model is illustrative of the fact that making manually-annotated datasets is more be more time-consuming and expensive to produce than the weakly-supervised approach <NUM> disclosed herein. Even if manually annotated data is also available, for many tasks the approach <NUM> could be used as pre-training to further improve results. Thus, the output from embodiments described herein may be used to seed further segmentation.

<NUM>) Object Detection: While the KITTI benchmark suite does not contain a semantic segmentation benchmark, it does contain object instance bounding boxes in both the Object and Tracking datasets. The definition of an object in the KITTI benchmark (an individual instance of a vehicle or person, for example, within a bounding box) differs significantly from the definition of an obstacle as part of the weakly-supervised approach <NUM> (any part of the scene the vehicle might collide with). However, object detection performance can be evaluated by ensuring that object instances provided by the KITTI Object and Tracking benchmarks are also classified as obstacles by the segmentation approach <NUM> described herein; hence the highest pixel-wise recall score is sought. For each object instance, the number of pixels within the bounding box classified as an obstacle is evaluated using the weakly-supervised approach <NUM>, as illustrated in <FIG>.

Three different recall metrics are presented:.

Recall results on the data provided as part of the Object and Tracking datasets (consisting of <NUM>,<NUM> images with <NUM>,<NUM> total object instances) are presented in Table V, and an example detection is shown in <FIG>. The object classes have been combined as follows: car, van, truck and tram labels are grouped as Vehicle; pedestrian, person sitting and cyclist labels are grouped as Person, and all others are grouped as Misc. The results show that the weakly-supervised segmentation approach <NUM> is reliably labelling objects as obstacles regardless of object class (and performs especially well for an instance recall threshold of <NUM>%); this is helpful in avoiding planning trajectories that intersect other vehicles or road users.

<FIG> shows example object detection results using obstacle segmentation. For a given input image (top), the network labels areas corresponding to proposed path <NUM>, obstacle <NUM> and unknown area <NUM> (middle). For each ground truth bounding box provided in the KITTI Object and Tracking datasets, the ratio of pixels labelled as obstacle <NUM> by the method <NUM> is computed (bottom). For each object instance, it is considered to be detected (for example bounding boxes <NUM>) if more than <NUM>% of the pixels within the bounding box <NUM> are labelled as obstacles. Note that even for failed detections (bounding boxes <NUM>, of which the outlines are shaded differently from bounding boxes <NUM> to indicate the detection difference), a number of the pixels were still labelled as obstacle, and due to the tight obstacle outlines provided by this method <NUM>, portions of the bounding box <NUM>, <NUM> may be missed (e.g. undercarriage of vehicles at bottom left).

Under some conditions the segmentation network <NUM> of some embodiments may fail to produce useful proposed path segmentations, as illustrated in <FIG>. These failure cases are mostly due to limitations of the sensor suite 12a, 12b (e.g. poor exposure or low field of view), and could be addressed using a larger number of higher-quality cameras.

<FIG> shows examples of proposed path segmentation failures. As shown by the top pair of images, overexposed or underexposed images may lead to incorrect path segmentation; this could be addressed by using a high dynamic-range camera 12a, for example.

As shown by the lower pair of images in <FIG>, at some intersections during tight turns, there is no clear path to segment as it falls outside the field of view of the camera 12a; using a wider field of view lens or multiple cameras in a surround configuration, for example, may well address this limitation.

As the weakly-supervised labels <NUM>, <NUM>, <NUM> are generated from the recording <NUM>, <NUM>, <NUM> of a data collection trajectory, it can only provide one proposed path <NUM> per image <NUM> at training time. However, at intersections and other locations with multiple possible routes, at test time the resulting network <NUM> frequently labels multiple possible proposed paths <NUM> in the image <NUM> as shown in <FIG>. This may have particular utility in decision-making for topological navigation within a road network.

<FIG> shows proposed path generalisation to multiple routes. The top, third from top and bottom images of <FIG> each show a side-road <NUM>" branching off to the left of the road <NUM>' along which the vehicle <NUM> is driving; two proposed path options are therefore available. The second from top image of <FIG> shows two side-roads, <NUM>" and <NUM>‴, one on either side of the road <NUM>' along which the vehicle <NUM> is driving. Three proposed path options are therefore available.

At intersections and roundabouts the network will often label different possible paths, <NUM>, which can then be leveraged by a planning framework for decision making during autonomous navigation.

An approach <NUM> for weakly-supervised labelling of images <NUM>, <NUM> for proposed path segmentation during on-road driving, optionally using only a monocular camera 12a, has been described above. The skilled person will appreciate that the specific example described is not intended to be limiting, and that many variations will fall within the scope of the claim.

It has been demonstrated that, by leveraging multiple sensors 12a, 12b and the behaviour of the data collection vehicle driver, it is possible to generate vast quantities of semantically-labelled training data <NUM> relevant for autonomous driving applications.

Advantageously, no manual labelling of images <NUM> is required in order to train the segmentation network <NUM>.

Additionally, the approach does not depend on specific road markings or explicit modelling of lanes to propose drivable paths.

The approach <NUM> was evaluated in the context of ego-lane segmentation and obstacle detection using the KITTI dataset, outperforming networks trained on manually-annotated training data and providing reliable obstacle detections.

<FIG> illustrates the method <NUM> of an embodiment. At step <NUM>, data comprising vehicle odometry data detailing a path taken by a vehicle <NUM> through an environment, obstacle sensing data detailing obstacles detected in the environment, and images of the environment are obtained.

At step <NUM>, the obstacle sensing data obtained in step <NUM> is used to label one or more portions of at least some of the images obtained in step <NUM> as obstacles.

At step <NUM>, the vehicle odometry data obtained in step <NUM> is used to label one or more portions of at least some of the images obtained in step <NUM> as the path taken by the vehicle <NUM> through the environment.

The skilled person will appreciate that steps <NUM> and <NUM> may be performed in either order, or simultaneously. Further, step <NUM> may be performed by a different entity from steps <NUM> and/or <NUM>.

The robustness of the trained network <NUM> to changes in lighting, weather and traffic conditions was demonstrated using the large-scale Oxford RobotCar dataset, with successful proposed path segmentation in sunny, cloudy, rainy, snowy and night-time conditions.

Claim 1:
A method of training a segmentation unit for autonomous route determination, wherein the segmentation unit is taught to identify a proposed path (14a) within an image (<NUM>, <NUM>) that would be likely to be chosen by a driver, the method comprising:
obtaining data from a data collection vehicle (<NUM>) driven along a path (<NUM>) through an environment, the data comprising:
vehicle odometry data (<NUM>) detailing the path (<NUM>) along which the vehicle (<NUM>) is driven through the environment,
obstacle sensing data (<NUM>) detailing obstacles detected in the environment; and
images (<NUM>, <NUM>) of the environment;
creating the training data set (<NUM>) by labelling the images (<NUM>, <NUM>), the labelling comprising:
using the obstacle sensing data (<NUM>) to label one or more portions of the environment represented in at least some of the images (<NUM>, <NUM>) as obstacles;
using the vehicle odometry data (<NUM>) to label one or more portions of the environment represented in at least some of the images (<NUM>, <NUM>) as the path (<NUM>) along which the vehicle (<NUM>) is driven through the environment; and training the segmentation unit using the training dataset (<NUM>) to identify the proposed path.