Patent Description:
The technology of autonomous vehicles is moving forward globally in a progression of scaled technology, and may be a significant part of the future of the automotive industry. Autonomous vehicles combine a variety of sensors to perceive their surroundings, such as radar, LiDar, sonar, GPS, odometry, inertial measurement units, or the like. Advanced control systems interpret sensory information to identify appropriate navigation paths, obstacles and relevant signage, to help the vehicle control the self-driving.

<CIT> describes a vehicle localization system implementing the following steps: receiving a predetermined road map; receiving at least one captured image from an image capture device of a vehicle; processing, by a road detection component, the at least one captured image, to identify therein road structure for matching with corresponding structure of the predetermined road map, and determine a location of the vehicle relative to the identified road structure; and using the determined location of the vehicle relative to the identified road structure to determine a location of the vehicle on the road map, by matching the road structure identified in the at least one captured image with the corresponding road structure of the predetermined road map.

<NPL>, disclose end-to-end learning of monocular semantic-metric occupancy grid mapping from weak binocular ground truth. The network learns to predict four classes, as well as a camera to bird's eye view mapping.

As highly automated driving and autonomous vehicles further develop, there is a need to develop and improve such systems, including the vehicle navigation system, the location system, the map matching, the environment perception, or the like. The challenge for autonomous vehicles designers is to produce control systems capable of analyzing sensory data in order to provide accurate detection of other vehicles and the road ahead.

One exemplary embodiment of the disclosed subject matter is a method as set out in Claim <NUM>.

Optionally, the neural network may further comprise an encoder-decoder architecture as set out in Claim <NUM>.

Optionally, the encoder section may further comprise a fully convolutional network as set out in Claim <NUM>.

Optionally, the encoder section may further be as set out in Claim <NUM>.

Optionally, the method may be further as set out in Claim <NUM>.

Another exemplary embodiment of the disclosed subject matter is a computerized apparatus as set out in Claims <NUM> to <NUM>.

Yet another exemplary embodiment of the disclosed subject matter is a computer program product as set out in Claim <NUM>.

The present disclosed subject matter will be understood and appreciated more fully from the following detailed description taken in conjunction with the drawings in which corresponding or like numerals or characters indicate corresponding or like components. Unless indicated otherwise, the drawings provide exemplary embodiments or aspects of the disclosure and do not limit the scope of the disclosure. In the drawings:.

One technical problem dealt with by the disclosed subject matter is to enable self-navigation of autonomous vehicles, in the absence of maps, Global Positioning System (GPS) connection or other geo-spatial positioning systems. A mapless autonomous driving platform, that does not rely on existing high-precision maps, may be required.

In some exemplary embodiments, autonomous vehicles may be fully operated using High-Definition (HD) mapping. HD mapping may require a strong connection to satellite GPS and continuous effort to update the maps. In some exemplary embodiments, an HD map may be maintained manually. Such map may indicate in high resolution information such as lanes, stop lines, bumpers, or the like. As opposed to regular road maps, which may be obtained using crowd-sourced information, currently HD mapping is not available using such source. Instead, extensive manual effort may be invested into creating an HD mapping of an area in which an autonomous vehicle may drive.

In some exemplary embodiments, a mapless autonomous driving platform may eliminate the need for the costly and time-consuming technology of HD mapping.

It is further noted that HD mapping may be utilized in non-autonomous driving scenarios. For example, in a semi-autonomous driving platform, the platform may provide alerts to the human driver, based on her activity and in view of an HD mapping of the surroundings of the vehicle. However, for the ease of explanation and for clarity purposes, and without limiting the disclosed subject matter to such embodiment, the description is focused on the autonomous driving embodiment.

One technical solution is to generate functional road maps using real-time visual input, captured by sensors mounted on the vehicles. In some exemplary embodiments, one or more images from one or more cameras mounted on the front of the vehicle may be utilized as real-time visual input. Additionally or alternatively, additional sensor information, such as obtained from LiDAR, odometry, inertial measurement units, or the like, may be utilized. In some exemplary embodiments, the functional road maps may comprise precise real-time information and true-ground-absolute accuracy. In some exemplary embodiments, the functional road maps may be HD maps that are generated on the fly and based on sensor input only. In some exemplary embodiments, the functional road maps may be of higher resolution than maps found in current conventional resources, such as crowd-based road maps. The functional road maps may be utilized by the navigation system of autonomous vehicles. In some exemplary embodiments, the on-the-fly generated functional road maps may be utilized instead of the pre-prepared HD maps that can be correlated with the present environment of the vehicle based on location information, such as obtained using a GPS connection.

In some exemplary embodiments, the functional road maps may be vehicle-centered. The vehicle may be determined to be at a fixed location and orientation within the map. Additionally or alternatively, the functional road maps format may be functional. Each functional road map may comprise only the necessary information for a vehicle to safely navigate its environment. As an example, the information may comprise information required for lane determination, identification of a location to stop the vehicle, or the like. Additional information, such as relating to buildings that are sensed by the sensors, may be absent from the functional road map.

The functional road maps are generated utilizing an Artificial Neural Network (ANN). The ANN may be trained to detect road features and transform the input image space into a top view space in a single shot. The ANN may be configured to process the one or more images and generate a top-down functional road map. The top-down functional road map may comprise for each pixel one or more features that are relevant to assisting the autonomous navigation of the vehicle within the map. Such element and features may comprise vehicles or portions thereof, obstacles, moving elements, pedestrians, pedestrian crossings, or the like.

In some exemplary embodiments, the ANN may be built using an Encoder-Decoder architecture. The ANN may comprise an encoder element, connected to a latent vector, that is connected to a decoder element. The encoder element of the network may comprise multiple convolutional layers that may be configured to process the visual input and compress it into a single latent vector. The size of the single latent vector may be varied. In some exemplary embodiments, the size of the single latent vector may not exceed <NUM> nodes in the most extreme case. Additionally or alternatively, a size of about <NUM> nodes may be useful in most practical applications. After this compression, the latent vector may be passed through the decoder element of the network, which may be configured to expand the vector into a functional top-down map of the road ahead of the vehicle.

In some exemplary embodiments, all the necessary information that is required for generating the functional top-down map, may be encoded into the latent vector. Such setup may force the network to distill the abundance of information available into only the parts most relevant to interpreting the scene into a top-down map. By limiting the amount of information in the encoded representation, the network's ability to memorize specific cases may be limited. Instead, the ANN may be forced to logically interpret the scene, in order to provide a correct output. The variance of the network's output may likewise be limited to logical perceivable elements on the road.

The disclosed architecture may transform the visual input, e.g., the forward image, into top-down map of the road ahead, which may be of a different size and shape than the input. This is as opposed to the use of encoder-decoder architecture for reconstructing an image of the same size and proportions. In some exemplary embodiments, auto-encoder networks may be used to replicate an input image after its compression, by mapping the input into a latent vector using the encoder and mapping the latent vector to a reconstruction of the original input using the decoder, which are of the same size and shape as the original input. In some exemplary embodiments, the output in the present disclosed subject matter may not be a mere modification or replication of the input. Instead, the output may be a substantial transformation from one point of view, from the car facing forward, to a completely different one, a top-down birds-eye view of the road ahead, and represented in a functional manner using functional features relevant to assisting the autonomous navigation of the vehicle within the map.

It may be noted that the task of transforming a forward view of a scene into a completely different representation, which is not a pixel-wise transformation of the original input, such as a top-down view of the scene in a functional format, is a complicated task that may be costly and time-consuming using general network architecture, and may even be not feasible. It is noted that two separate networks (or concatenated layers) may be used to first transform the front-view to a top-down image of the scene, and then transform the top-down image into a top-down functional road map. However, during such separate computations, important information may be lost, and the intermediate output may be too noisy to be sufficiently accurate.

As an example, consider a naive implementation where a first ANN performs a pure perspective transformation of a front view image to a top-down image; such may end up with a noisy, smeared, and mostly useless image, as the information in a front view image may not be sufficient to fully reconstruct the scene in top-down view. Given such noisy image, a functional map that is constructed based thereon may be imprecise and insufficient for safely driving autonomous vehicles.

In some exemplary embodiments, the ANN may be configured to learn to interpret the data as a global structure instead of a localized, pixel by pixel fashion, in order to be transformed to a different perspective and to a different modality. The ANN may be configured to extract long-range correlations, large-scale features, or the like, that may be necessary for the task of constructing the top-down functional map. The ANN may comprise several elements that may enable to extract such features. These elements may comprise different types of self-attention modules, that may be configured to capture long-range interactions. The process of self-attention modules may comprise reweighting different pixels, channels, or the like, to focus the following layers' attention on layers with higher weights, while paying less heed to layers with lower weights. Self-attention modules may be employed within the backbone of the ANN, such as within a portion of its constituent residual blocks. The number and type of employed self-attention modules, the blocks they are employed in, or the like, may be changed from one iteration of the model to another. Such modules may give the ANN stronger tools to work with than the standard convolutions, which are local in nature, having no information regarding the relative position of each pixel processed, and can generally extract small scale features.

As an example, the self-attention modules may be a Convolutional Block Attention Module (CBAM). CBAM may be configured to allow the ANN to efficiently calculate spatial and channel-wise attention. Given an input tensor of dimensions CxHxW, with C being designated the number of channels, with H and W being the height and width of the channels, respectively, CBAM may be configured to calculate channel-wise attention by reducing the tensor to a Cx1x1 shape. CBAM may utilize a fully connected layer to calculate the C weights. Each C weight may be a number for a layer, by which the original input tensor is reweighted. Each layer may be multiplied by its respective C weight. The spatial attention may be similarly obtained by reducing the input along the channels dimension, to a 1xHxW tensor. A 3x3 convolution, a 7x7 convolution, or the like, may be applied to produce a 1xHxW tensor of weights to be multiplied by the original CxHxW input. Each Cx1x1 column in the input may be multiplied by the respective weight. As another example, a Self-Attention Generative Adversarial Network (SA) that comprises self-attention module may be trained. SA may comprise a computationally heavier module than CBAM. Given an input tensor of dimensions CxHxW where N=H*W, SA may be configured to explicitly calculate N^<NUM> numbers defining the extent to which the next feature layer value at each pixel takes into account the value at each other pixel. SA may be configured to allow the ANN to glean large scale structures and long-range correlations, instead of the local nature of standard convolutions. The attention element in SA may be more general than in the CBAM module, as it considers the attention paid each pixel by each other pixel, in addition to a more general locally calculated pixel-wise reweighting. SA may be optionally employed at the end of the encoder and towards the end of the decoder, operating on feature layers with small to moderate spatial dimensions.

Additionally or alternatively, the ANN may comprise a modified version of a standard convolutional layer. Such modules may give the ANN stronger tools to work with than the standard convolutions, which are local in nature, having no information regarding the relative position of each pixel processed, and can generally extract only small scale features. As an example, the ANN may comprise one or more CoordConv layers. Standard convolutional layers may be by default blind to the relative location of each feature within the layer, as they may consist of a rectangular kernel applied across the layer. CoordConv provides a modification that takes into account the relative location, such as the feature being in the center of the layer, the right edge, the left edge, or the like. CoordConv works by giving the convolution access to its own input coordinates through the use of extra coordinate channels. Without sacrificing the computational and parametric efficiency of ordinary convolution, CoordConv may allow the ANN to learn either complete translation invariance or varying degrees of translation dependence, as required by the end task. The coordConv modification may concatenate two extra layers to the input tensor containing the x and y coordinates of each pixel, normalized between -<NUM> and <NUM>. The convolutional kernel may then take on an additional two layers. This may allow the convolutional layer weights to potentially learn features which depend on the relative spatial location within the input tensor. As an example, the ANN may comprise one or more layers normalized with Spectral Normalization. The Spectral Normalization may be applied on the weights of a convolutional layer, such that the output is Lipschitz continuous. Spectral Normalization may be configured to stabilize the training procedure and produce more robust and consistent outputs. Spectral Normalization may be employed within the decoder part of the ANN.

One technical effect of utilizing the disclosed subject matter is providing a highly intelligent, vision-based approach that does not require a-priori HD mapping. In some exemplary embodiments, the on-the-fly generation of the top-down functional road map may allow the market for autonomous vehicles to scale and commercialize more rapidly at less cost.

Another technical effect of utilizing the disclosed subject matter is to enable autonomous vehicles to automatically navigate without the need of GPS connection. As the map generation utilizes real-time input from sensors mounted on the vehicle and does not rely on GPS localization within pre-fabricated global maps, maps may be generated in any location, in spite of the vehicle not being connected to GPS, the location yet not being mapped, or the like.

In some examples, which do not fall within the claimed invention, GPS location may be utilized to augment the information available for the generation of the top-down functional road map, such as by utilizing a pre-existing low-resolution map as general structure to be expected in the top-down functional map. Not falling within the claimed invention, using GPS location, a relevant segment in a low-resolution map may be obtained and provided together with the sensor information for real-time generation of the top-down functional map.

Yet another technical effect of utilizing the disclosed subject matter is to avoid having to maintain pre-existing HD maps. If pre-existing maps are relied upon, any change may need to be identified in order to modify the pre-existing mapping. An addition of a stop line, a change in the location of the cross-walk, and a removal of a road bump, or a similar change in the road, may need to be identified, and a modification of the HD map may be required. Using the disclosed subject matter, which does not rely on a pre-existing HD functional mapping, no maintenance of existing maps may be required.

The disclosed subject matter may provide for one or more technical improvements over any pre-existing technique and any technique that has previously become routine or conventional in the art. Additional technical problem, solution and effects may be apparent to a person of ordinary skill in the art in view of the present disclosure.

Referring now to <FIG> showing a schematic illustration of an exemplary neural network, in accordance with some exemplary embodiments of the disclosed subject matter.

Artificial Neural Network <NUM> is configured to process a Visual Input <NUM> to generate a functional top-down map. Artificial Neural Network <NUM> may be based on a collection of connected layers. Different layers may perform different transformations on their inputs. Signals may travel from the first layer (e.g., the layer directly receiving Visual Input <NUM>), to the last layer (e.g., the layer providing Output <NUM>). In some exemplary embodiments, the signals may traverse the layers multiple times.

In some exemplary embodiments, Artificial Neural Network <NUM> may be trained to detect road features and transform the front view of Visual Input <NUM> into a top view functional map in a single shot. Artificial Neural Network <NUM> may be configured to process Visual Input <NUM> and generate an Output <NUM> of top-down functional road map, providing features relevant to assisting the autonomous navigation of the vehicle within the map.

In some exemplary embodiments, Artificial Neural Network <NUM> may have an encoder-decoder architecture. The encoder-decoder architecture may comprise an internal layer that may describe a code used to represent the input (e.g., Latent Vector <NUM>). In some exemplary embodiments, the encoder-decoder architecture may comprise two components: an Encoder <NUM> that maps the input into Latent Vector <NUM>, and a Decoder <NUM> that maps Latent Vector <NUM> to a top-down functional road map. The encoder-decoder architecture may be configured encode all the necessary information that is required for generating the functional top-down map into Latent Vector <NUM>. Such architecture may be configured force Artificial Neural Network <NUM> to distill the abundance of information available into only the parts most relevant to interpreting the scene into a top-down map. In some exemplary embodiments, the input of Encoder <NUM> may be of different size and shape than that of the output of Decoder <NUM> (e.g., 1024x1080x4 pixels as input, representing four different sensor modalities for each pixel in the visual input, and 64x64x6 nodes as output, representing <NUM> different functional modalities for each pixel in the top-down functional map).

In some exemplary embodiments, Encoder <NUM> may be configured to process Visual Input <NUM> and compress it into a single Latent Vector <NUM>. In some exemplary embodiments, Encoder <NUM> may comprise one or more down-sampling residual blocks. In some exemplary embodiments, Encoder <NUM> may comprise no more than two down-sampling residual blocks utilized to remove information before encoding Visual Input <NUM> into Latent Vector <NUM>. In some exemplary embodiments, Encoder <NUM> may comprise one or more encoding layers, such as one encoding layer, three encoding layers, ten encoding layers, or the like. In some exemplary embodiments, Encoder <NUM> may be absent a perspective transformation layer for changing an image of the scene from the front view to an alternative image of a top-down view. Instead, Encoder <NUM> may transform Visual Input <NUM> into Latent Vector <NUM> which may capture semantic information of Visual Input <NUM>. Decoder <NUM> may then use Latent Vector <NUM> to provide the functional mapping of the scene from a different point of view (a top-down view instead of a front view).

Additionally or alternatively, Encoder <NUM> may comprise a Backbone <NUM> configured to transform Visual Input <NUM> into a Scaled Down Feature Layer <NUM> to be encoded. Backbone <NUM> may be configured to transform Visual Input <NUM> into Scaled Down Feature Layer <NUM>. Encoder <NUM> may be configured to map Scaled Down Feature Layer <NUM> into Latent Vector <NUM>. It may be noted that the existence and the size of Backbone <NUM> may depend on the size of Visual Input <NUM>. Encoder <NUM> may employ Backbone <NUM> as an average pooling layer, before outputting Latent Vector <NUM>. In some exemplary embodiments, Backbone <NUM> may be configured reduce the resolution of the Visual Input <NUM> until it is represented by scaled down feature maps in which the spatial structure of the scene is no longer discernible. In some exemplary embodiments, Backbone <NUM> may be a fully convolutional network, such as Dilated Residual Network (DRN) or variation thereof, EfficientNet™ network or variation thereof, or the like. Additionally or alternatively, Backbone <NUM> may comprise multiple convolutional network of different types. As an example, Backbone <NUM> may comprise <NUM> blocks of the DRN-<NUM> network, which may comprise about <NUM> convolutional layers. Additional modifications or customizations may be performed on the utilized convolutional networks to obtain the desired features of Backbone <NUM>. In one embodiment, Scaled Down Feature Layer <NUM> may be designed to comprise between <NUM> to <NUM> channels, with width and height of about <NUM>/<NUM> of the dimensions of Visual Input <NUM>. Such loss of spatial acuity, that may limit image classification accuracy and scene understanding, may be alleviated by dilation, which increases the resolution of output feature maps without reducing the receptive field of individual neurons. Scaled Down Feature Layer <NUM> may be passed to Encoder <NUM> for being processed and compressed it into Latent Vector <NUM>.

In some exemplary embodiments, Latent Vector <NUM> may be the final layer produced from Encoder <NUM>. The content of Latent Vector <NUM> may encapsulate the information for all features of Visual Input <NUM> in order to enable Decoder <NUM> to make accurate decisions when generating Output <NUM>.

In some exemplary embodiments, the size of Latent Vector <NUM> may be varied. However, the size may be unlikely to exceed <NUM> nodes in the most extreme case. The size of Latent Vector <NUM> may be about <NUM> nodes, about <NUM> nodes, about <NUM> nodes, about <NUM> nodes, or the like.

In some exemplary embodiments, Latent Vector <NUM> may be passed through Decoder <NUM>. Decoder <NUM> may be configured to expand Latent Vector <NUM> into a functional top-down map of the road ahead of the vehicle. Decoder <NUM> may comprise several decoding layers. It may be noted that Encoder <NUM> and Decoder <NUM> may comprise different number and different type of layers. Furthermore, the input of Encoder <NUM> may have a different size than the output of Decoder <NUM>.

Additionally or alternatively, Artificial Neural Network <NUM> may comprise additional layers that may be configured to improve the quality of Output <NUM>. As an example, Latent Vector <NUM> may be passed through a linear fully connected layer (not shown), before being passed through Decoder <NUM>. The linear fully connected layer may comprise about <NUM> residual up-sampling blocks, which may incorporate spectral normalization layers atop the convolution modules, ending with a self-attention layer and a final residual block. The spectral normalization may be applied on Latent Vector <NUM> to stabilize the training of Artificial Neural Network <NUM>. Additionally or alternatively, Artificial Neural Network <NUM> may comprise Exponential Linear Unit (ELU) activation layers to speed up the learning, Batch Normalization layers to stabilize the learning, Convolutional Block Attention Module (CBAM) layers for accuracy improvement of the learning, or the like. As yet another example, Artificial Neural Network <NUM> may employ coordinates-augmented convolutions, such as CoordConvs™, instead of regular convolutions.

It may be noted that Artificial Neural Network <NUM> does not perform a proper segmentation or categorization of each pixel in Visual Input <NUM> to be directly mapped by a pixel in Output <NUM>. Neural Network <NUM> may encode Visual Input <NUM> into a completely different representation which is neither a segmented representation nor a geometric transformation of perspective, but is rather a specific symbolic representation, which may be utilized for navigation.

Artificial Neural Network <NUM> may be trained using a training database that comprises sensor input and a functional top-down map that is manually provided. The pre-existing HD maps may be utilized to train Artificial Neural Network <NUM> together with a corresponding sensor input. Artificial Neural Network <NUM> may be trained to provide the correct top-down functional road map given front-view sensor information. As a result, Artificial Neural Network <NUM> may be trained to correctly encode Latent Vector <NUM> out of sensor information and decode such information into an accurate top-down functional road map to be utilized for autonomous driving. After such training is complete, Artificial Neural Network <NUM> may be configured to generate a top-down function road map based on a given sensor input.

Additionally or alternatively, training may utilize a top-down sensor view of the vehicle. The top-down sensor view may be provided using a drone accompanied the vehicle that is driving, using a sensor mounted on a poll above the vehicle, or the like. Using the top-down sensor view that can be translated into functional road map, Artificial Neural Network <NUM> may be trained to transform a front view sensor information to a top-view functional road map without the intermediate representation of the top-view sensor information.

Artificial Neural Network <NUM> may be configured to receive a sequence of sensor information, such as to indicate information received over time (e.g., last <NUM> frames obtained from a camera).

Referring now to <FIG> showing a flowchart diagram of a method, in accordance with some exemplary embodiments of the disclosed subject matter.

On Step <NUM>, a real-time visual input is obtained from one or more sensors mounted on a vehicle. The real-time visual input captures a front view of a road ahead of the vehicle. Additionally, the real-time visual input further may capture a back view behind the vehicle, side views at a left side and at a right side of the vehicle, or the like. As an example, the real-time visual input may comprise three camera images, one capturing a forward view in front of the vehicle, the second capturing a left view and the third capturing a right view of the vehicle. Additionally or alternatively, the real-time visual input may comprise a plurality of images taken at fixed distance intervals. As an example, the real-time visual input may comprise a current image (e.g., from the current location of the vehicle), an image from <NUM> meters formerly, <NUM> meters formerly, or the like. The distance between the images may be determined based on the vehicle's activity, such as in view of the number of wheels rotations, distance traveled, or the like.

On Step <NUM>, the real-time visual input is processed by a neural network, such as <NUM> of <FIG>. The processing of the real-time visual input is performed without relying on a pre-determined precise mapping. The vehicle may comprise a location module providing location information of the vehicle, such as a GPS system, a satellite navigation system, or the like. Processing the real-time input may be performed without relying on the location information of the vehicle.

The neural network may comprise an encoder-decoder architecture. The encoder-decoder architecture may comprise an encoder section, a latent vector layer, and a decoder section having one or more decoding layers. The encoder section may be configured to receive as input the real-time visual input. The encoder section may comprise a fully convolutional network, having one or more encoding layers. The fully convolutional network may be configured to transform the real-time visual input into a scaled down feature layer. The encoder section may be configured to map the scaled down feature layer into the latent vector layer.

On Step <NUM>, a functional top-down map of the road ahead of the vehicle is generated. The decoder section of the encoder-decoder architecture may be configured to output a functional top-down map of the scene represented by the real-time visual input. The decoder section may be configured to simultaneously transform the visual information provided in the real-time visual input to functional information and changing a point of view of the scene from a front view to a top-down view. It may be appreciated that the encoder section may be absent a perspective transformation layer for changing an image of the scene from the front view to an alternative image of a top-down view. It may also be noted that the input of the encoder section may have a different size, different shape, different number of nodes, or the like than an output of the decoder section.

Each pixel in the functional top-down map is associated with a predetermined relative position to the vehicle. A content of each pixel in the functional top-down map is assigned a set of values. Each value represents a functional feature relating to a location at a corresponding predetermined relative position associated with the pixel. As an example, the functional features may be a drivable road indication, an available driving path indication, a stop line indication, a speed bump indication, a lane markings indication, or the like. The set of values assigned to each pixel in the functional top-down map comprises at least two different functional features. The functional top-down map provides functional information useful for an autonomous navigation system to perform autonomous driving.

On Step <NUM>, the functional top-down map is provided to an autonomous navigation system of the vehicle. Autonomous navigation system is utilized by the vehicle to perform the autonomous driving, such as for finding directional, correlating positions on the road, adjusting the route, or the like.

On Step <NUM>, the autonomous navigation system autonomously drives the vehicle in accordance with functional features represented by the functional top-down map. In addition to the functional top-down map, the autonomous navigation system may utilize additional information for driving, such as but not limited to object recognition for identifying objects in the street, such as a pedestrian or another vehicle, whose presence is not indicated in the functional top-down map.

Referring now to <FIG> showing a schematic illustration of an exemplary visual input, in accordance with some exemplary embodiments of the disclosed subject matter.

In some exemplary embodiments, Input Image <NUM> may be an image captured by a sensor mounted on a vehicle, such as a front-mounted camera. Input Image <NUM> may be a Red Green Blue (RGB) image. However, other types of images may be utilized as input images, to obtain Input Image <NUM> or the like. As an example, a thermal infra-red cameras may be utilized, in conjunction with the RGB input to obtain Input Image <NUM>, may be utilized separately as a separate type of input, may be utilized as an additional layer of input, or the like.

Input Image <NUM> may capture a front view of a road ahead of the vehicle. Additionally or alternatively, other views may be captured, such as back view behind the vehicle, side views to the left and the right of the vehicle, or the like. In some exemplary embodiments, a top-down view of the vehicle may not be available.

In some exemplary embodiments, Input Image <NUM> may be part of a series of input images taken at fixed distance intervals, such as an image capturing the current location, an image capturing a location from a former <NUM> meters, an image capturing a location from a former <NUM> meters, or the like. Additionally or alternatively, Input Image <NUM> may be an aggregation or a combination of the series of input images.

Referring now to <FIG> showing a schematic illustration of an exemplary top-down functional map, in accordance with some exemplary embodiments of the disclosed subject matter.

Functional Top-Down Map <NUM> is the output of a neural network, such as Artificial Neural Network <NUM> of <FIG>, on Input Image <NUM>.

Functional Top-Down Map <NUM> is comprised of several layers of identical dimensions, e.g., <NUM>-<NUM>. Each layer represents a functional feature of the road. As a result, Functional Top-Down Map <NUM> may be an (n x H x W) matrix. Parameter n may be the number of layers or categories comprised by Functional Top-Down Map <NUM> (e.g., Map Layers <NUM>-<NUM>). The number of layers and their corresponding categories may differ from model to model. Parameters H and W may be the height and width of Functional Top-Down Map <NUM>, which may be different than the height and width of Input Image <NUM>.

A content of each pixel in Functional Top-Down Map <NUM> is assigned a set of values. The set of values may comprise n values, each of which belongs to a different layer. Each value represents a functional feature relating to a location at a predetermined relative position associated with the pixel. As an example, the value of each pixel, in each layer, may be a confidence value (a number between <NUM> and <NUM>) that the relevant functional feature exists within that pixel. The value of each pixel may be determined based on the set of values. The value of the pixel may be determined based on the confidence value being above a predetermined threshold, such as <NUM>.

Referring now to <FIG> showing schematic illustrations of exemplary map layers, in accordance with some exemplary embodiments of the disclosed subject matter.

Each layer of Map Layers <NUM>-<NUM> of <FIG> correspond to a different functional feature. Map Layer <NUM> may represent drivable road indications within Input Image <NUM>. Map Layer <NUM> may represent available driving path indications within Input Image <NUM>. Map Layer <NUM> may represent stop line indications within Input Image <NUM>. Map Layer <NUM> may represent indications of stop lines for traffic lights within Input Image <NUM>. Map Layer <NUM> may represent speed bump indications within Input Image <NUM>. Map Layer <NUM> may represent dashed lane markings indications within Input Image <NUM>. Map Layer <NUM> may represent continuous lane markings indications within Input Image <NUM>.

It may be noted that in some samples some of the layers may be empty. As an example, Map Layer <NUM> comprises a small indication of stop lines for traffic lights within Input Image <NUM>, however, for other input images that do not comprise stop lines for traffic lights, the relevant map layer may be empty. Functional Top-Down Map <NUM> is an aggregation of Map Layers <NUM>-<NUM>.

Referring now to <FIG> showing a block diagram of an apparatus, in accordance with some exemplary embodiments of the disclosed subject matter. An Apparatus <NUM> is configured to support generation of models for generating top-down functional road maps for autonomous vehicles, in accordance with the disclosed subject matter.

In some exemplary embodiments, Apparatus <NUM> comprises one or more Processor(s) <NUM>. Processor <NUM> may be a Central Processing Unit (CPU), a microprocessor, an electronic circuit, an Integrated Circuit (IC) or the like. Processor <NUM> is utilized to perform computations required by Apparatus <NUM> or any of it subcomponents.

In some exemplary embodiments of the disclosed subject matter, Apparatus <NUM> may comprise an Input/Output (I/O) module <NUM>. I/O Module <NUM> may be utilized to provide an output to and receive input from a user, a device, a sensor, or the like, such as, for example receiving an input from one or more sensors of Connected Cars <NUM>, providing output for one or systems of Connected Cars <NUM>, or the like.

In some exemplary embodiments, Apparatus <NUM> may comprise Memory <NUM>. Memory <NUM> may be a hard disk drive, a Flash disk, a Random Access Memory (RAM), a memory chip, or the like. In some exemplary embodiments, Memory <NUM> may retain program code operative to cause Processor <NUM> to perform acts associated with any of the subcomponents of Apparatus <NUM>.

In some exemplary embodiments, Model Generator <NUM> may be configured to utilize data from Training Set <NUM> to train Neural Network <NUM> to generate top-down functional models.

In some exemplary embodiments, Neural Network <NUM> may comprise an encoder-decoder architecture. The encoder-decoder architecture may comprise an Encoder <NUM>, a Latent Vector <NUM>, and a Decoder <NUM>. The input of Encoder <NUM> may be of a different size than an output of Decoder <NUM>.

In some exemplary embodiments, Encoder <NUM> may comprise one or more decoding layers. Encoder <NUM> may be configured to receive as input the real-time visual input. Encoder <NUM> may comprise a fully convolutional network. The fully convolutional network may be configured to transform the real-time visual input into a scaled down feature layer. Encoder <NUM> may be absent a perspective transformation layer for changing an image of the scene from the front view to an alternative image of a top-down view. Encoder <NUM> may be configured to map the scaled down feature layer into Latent Vector <NUM>. Latent Vector <NUM> may be of a scaled down size comparing to Neural Network <NUM>. As an example, Latent Vector <NUM> ay comprise less than about <NUM> nodes, less than <NUM> nodes, less than <NUM> nodes, less than <NUM> nodes, or the like. Decoder <NUM> may be configured to output the functional top-down map. Decoder <NUM> may be configured to map Latent Vector <NUM> to a top-down functional map. However, the top-down functional map may be of a different shape and size than the input.

It may be noted that Decoder <NUM> may simultaneously transform visual information of the real-time visual input to functional information and changing a point of view of the scene from front, back or side view, to a top-down view. Such transformation may be inconsistent of the regular notion encoder-decoder architecture where the decoder maps the latent vector into a reconstruction of the input having the same shape as the input. On the other hand, Encoder <NUM> may not comprise a perspective transformation layer for changing an image of the scene from the front view to an alternative image of a top-down view.

In some exemplary embodiments, Neural Network <NUM> may be trained by Model Generator and provided to Connected Car <NUM> via I/O Module <NUM>, to be utilized for generating the top-down functional map. Additionally or alternatively, a top-down functional map generated by Neural Network <NUM> may be provided to Connected Car <NUM> via I/O Module <NUM>, to be utilized for navigation thereof.

Referring now to <FIG> showing a block diagram of an apparatus, in accordance with some exemplary embodiments of the disclosed subject matter. An apparatus of Connected Car <NUM> may be configured to generate top-down functional models enable autonomous driving of a vehicle, in accordance with the disclosed subject matter.

In some exemplary embodiments, Connected Car <NUM> may comprise one or more Processor(s) <NUM>. Processor <NUM> may be a Central Processing Unit (CPU), a microprocessor, an electronic circuit, an Integrated Circuit (IC) or the like. Processor <NUM> may be utilized to perform computations required by Connected Car <NUM> or any of it subcomponents.

In some exemplary embodiments of the disclosed subject matter, Connected Car <NUM> may comprise an Input/Output (I/O) module <NUM>. I/O Module <NUM> may be utilized to provide an output to and receive input from a user or a sensor or other apparatus, such as, for example receiving an input from Apparatus <NUM>, providing output for Apparatus <NUM>, or the like.

In some exemplary embodiments, Connected Car <NUM> may comprise a Memory <NUM>. Memory <NUM> may be a hard disk drive, a Flash disk, a Random Access Memory (RAM), a memory chip, or the like. In some exemplary embodiments, Memory <NUM> may retain program code operative to cause Processor <NUM> to perform acts associated with any of the subcomponents of Connected Car <NUM>.

One or more Sensor(s) <NUM> are configured to collect real-time input associated with the vehicle of Connected Car <NUM>. The real-time input is a visual input, such as an image, a video, an RGB model, or the like. Sensor <NUM> may be a digital camera, a thermal infra-red camera, RGB modeling sensor, a LiDar, a combination thereof, or the like. The real-time visual input captures a front view of a road ahead of the vehicle of Connected Car <NUM>, a back view behind the vehicle of Connected Car <NUM>, side views at a left side and at a right side of the vehicle of Connected Car <NUM>, other scenes around the vehicle of Connected Car <NUM>, or the like. Sensor <NUM> may be mounted on the vehicle of Connected Car <NUM>, may be connected to another physical sensor mounted on the vehicle, or the like. Additionally or alternatively, Sensor <NUM> may comprise a plurality of sub-sensors, each of which may be mounted on a different portion of the vehicle. Sensor <NUM> may be a camera, a smart camera device, a network of spatially distributed smart camera devices, a sonographer, Magnetic resonance imaging (MRI) sensor, or any other sensor that can produce a visual input. Other kinds of input may be obtained and processed.

In some exemplary embodiments, Input Analysis Module <NUM> may be configured to analyse the real-time input obtained from Sensor <NUM>. Input Analysis Module <NUM> may be configured to convert the real-time input into a visual input that can be processed by Neural Network <NUM>, such as a visual front view representation of a scene.

In some exemplary embodiments, Connected Car <NUM> may comprise a GPS Module <NUM> configured to provide location information of the vehicle. However, Input Analysis Module <NUM> may analyze the real-time input without relying on a pre-determined precise mapping or location information of the vehicle that may be determined by GPS Module <NUM>.

Top-Down Functional Model Generator <NUM> is configured to generate a functional top-down map of the road ahead of the vehicle of Connected Car <NUM>. Each pixel in the generated functional top-down map is associated with a predetermined relative position to the vehicle. A content of each pixel in the generated functional top-down map is assigned a set of values, each of which represents a functional feature relating to a location at a corresponding predetermined relative position associated with the pixel.

Top-Down Functional Model Generator <NUM> utilizes Neural Network <NUM>, that is configured to process the real-time input provided by Sensor <NUM> to generate the functional top-down map. Additionally, Top-Down Functional Model Generator <NUM> may obtain the input from Input Analysis Module <NUM>. Top-Down Functional Model Generator <NUM> generates the functional top-down map without relying on a pre-determined precise mapping or other information obtained using GPS Module <NUM>.

The functional top-down map is configured to provide functional information useful for Navigation System <NUM> to perform autonomous driving of the vehicle of Connected Car <NUM>. Navigation System <NUM> autonomously drives the vehicle in accordance with functional features represented by the generated functional top-down map.

The present invention is an apparatus, a method, and/or a computer program product. The computer program product includes a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

Claim 1:
A method comprising:
obtaining a real-time visual input (<NUM>, <NUM>) from at least one sensor mounted on a vehicle, wherein the real-time visual input (<NUM>) captures a front view of a road ahead of the vehicle;
processing the real-time visual input (<NUM>) by a neural network (<NUM>) to generate a functional top-down map (<NUM>, <NUM>) of the road ahead of the vehicle, wherein each pixel in the functional top-down map (<NUM>) is associated with a predetermined relative position to the vehicle, wherein a content of each pixel in the functional top-down map (<NUM>) is assigned a set of two or more values representing at least two different functional features of the road, each value of the set of two or more values represents a functional feature relating to a location within the road ahead of the vehicle at a corresponding predetermined relative position associated with the pixel, wherein said processing is performed without relying on a pre-determined precise mapping,
wherein the functional top-down map (<NUM>, <NUM>) of the road ahead of the vehicle is generated in real-time and only based on sensor readings of sensors mounted on the vehicle and without relying on a pre-existing map;
wherein the functional top-down map (<NUM>) is an aggregation of multiple layers, each layer of the multiple layers representing a different respective functional feature of the road; and
providing the functional top-down map (<NUM>) to an autonomous navigation system of the vehicle, whereby the autonomous navigation system autonomously drives the vehicle in accordance with functional features of the road that are represented by the functional top-down map (<NUM>).