Patent ID: 12259259

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure. Other embodiments may be utilized and structural, and logical changes may be made without departing from the scope of the disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

Embodiments described in the context of one of the devices or methods are analogously valid for the other devices or methods. Similarly, embodiments described in the context of a device are analogously valid for a vehicle or a method, and vice-versa.

Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments, even if not explicitly described in these other embodiments. Furthermore, additions and/or combinations and/or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.

In the following, embodiments will be described in detail.

An e-hailing app, typically used on a smartphone, allows its user to hail a taxi (or also a private driver) through his or her smartphone for a trip.

FIG.1shows a communication arrangement including a smartphone100and a server (computer)106.

The smartphone100has a screen showing the graphical user interface (GUI) of an e-hailing app that the smartphone's user has previously installed on his smartphone and has opened (i.e. started) to e-hail a ride (taxi or private driver).

The GUI101includes a map102of the vicinity of the user's position (which the app may determine based on a location service, e.g. a GPS-based location service). Further, the GUI101includes a box for point of departure103(which may be set to the user's present location obtained from location service) and a box for destination104which the user may touch to enter a destination (e.g. opening a list of possible destinations). There may also be a menu (not shown) allowing the user to select various options, e.g. how to pay (cash, credit card, credit balance of the e-hailing service). When the user has selected a destination and made any necessary option selections, he or she may touch a “find car” button105to initiate searching of a suitable car.

For this, the e-hailing app communicates with the server106of the e-hailing service via a radio connection. The server106includes a database107having information about the current location of registered vehicles108, about when they are expected to be free, about traffic jams etc. From this, a processor110of the server106selects the most suitable vehicle (if available, i.e. if the request can be fulfilled) and provides an estimate of the time when the driver will be there to pick up the user, a price of the ride and how long it will take to get to the destination. The server communicates this back to the smartphone100and the smartphone100displays this information on the GUI101. The user may then accept (i.e. book) by touching a corresponding button. If the user accepts, the server106informs the selected vehicle108(or, equivalently, its driver), i.e. the vehicle the server106has allocated for fulfilling the transport request.

It should be noted while the server106is described as a single server, its functionality, e.g. for providing an e-hailing service for a whole city, will in practical application typically be provided by an arrangement of multiple server computers (e.g. implementing a cloud service). Accordingly, the functionality described in the following provided by the server106may be understood to be provided by an arrangement of servers or server computers.

To determine all route-related information, like the most suitable driver and an estimate of the time when the driver will be there to pick up the user, a price of the ride and how long it will take to get to the destination, the processor110access a database107storing map data including for example one or more road maps indicating where roads are located, the allowed direction of travel, speed limits, etc.

The database107is in this example implemented by a local memory109of the server computer106. However, it may also be implemented at least partially externally to the server computer106, e.g. in a cloud, and it may be filled by access to an external database111, e.g. an open route map information database such as OSM (Open Street Map).

The service of ride-hailing providers significantly relies on accurate and up-to-date map data. For example, for determination of information like described above (e.g. estimated time to reach destination) or for navigation of the selected vehicle108(i.e. the vehicle108assigned to pick-up the user) the map data should be up to date and free of errors, e.g. should not lack travel possibilities (e.g. roads or allowed directions of travel) that exist in reality (i.e. in the physical geographic area represented by the map data).

Representing the map data into an image eases the service provider or the public to understand the geospatial semantics for a certain region. According to various embodiments, approaches for generating such a map image representing map data for a certain geographic area are provided. The map image may have a form to be easily visually interpreted by a human user but it may also be a (two or more dimensional) representation that is especially suited to be processed by a computer, for example a graph representation of a road network. Such a representation adapted for being processed by a computer may for example be stored in the database107to be processed by the server computer106(e.g. to determine the above-mentioned information like cost of ride) or may be supplied to a navigation system of the selected vehicle108(e.g. to help the driver navigate or navigate the vehicle, e.g. in case of an autonomous vehicle).

Map images may be generated by first collecting the geospatial data for each spatial entity in a geographic area by various devices and tools and then drawing or rendering a map image accordingly. The geospatial data usually describes the position of spatial entities, shape, and mutual relationship, etc.

For example, map drawing generates a bitmap image of a target area from vector data defining points, lines, and areas of geographical features such as Points of Interest (POI) and addresses, features such as roads and railways, cities, parks, and the like. Map images may also be rendered at a client device in a server-client setting. To achieve this, a map server selects map data from a geospatial database for a certain geographic area, generates multiple map image layers using the selected map data, and transmits them separately to the client device.

However, a concern regarding rendering map images based on pre-collected geospatial data is that field data collection is an expensive, time consuming, and cumbersome task, which restricts the map-update frequency to a few years or even longer for less populated and isolated places.

An alternative resource to facilitate the map image (or generally map data) generation process are satellite images because they represent the overall appearance of a geographic area. Modern satellite images have been quickly improved in terms of quantity, timeliness, quality, and contents diversity. Automatic conversion from satellite images allows generating map images in a cost-effective manner and with frequent updates.

However, approaches which leverage a satellite image as the only resource to generate a map image face the following challenges: 1) not all objects in a satellite image are visually distinguishable such as an underpass, a route with a very similar colour to its surrounding environments, etc. 2) the satellite image is common to include occlusion as cloud and shadow and these occlusion prevents these methods to reconstruct an accurate map. 3) Applying such an approach to regions with a road network which is of irregular shape and complex structure (like many South-eastern Asian Cities like Singapore), the spatial objects are small in size with tight arrangements leading to poor results.

In summary, approaches of a first type which generate map images from a collection of geospatial data usually generate maps with very accurate details but may not update quickly to reflect the latest geospatial information and approaches of a second type which generate map data from a satellite image allow generation of map data which is up-to-date but face challenges that many objects are not visually identifiable in satellite images.

To address the above issues, according to various embodiments, map data (i.e. in form of a map image) is generated by jointly using road usage information (e.g. crowd-sourced GPS traces specifying which routes have been taken by drivers) and a satellite image by means of a GAN (generative adversarial network). This may for example be performed by a server computer106of an e-hailing service. Compared to the approaches of the first type mentioned above, obtaining satellite images and crowd-sourced GPS traces is easier than collecting the geospatial data for each spatial entity. Compared to the approaches of the second type mentioned above, road usage information is a natural indicator of the underlying street network and it helps to recover the spatial objects more robust than satellite images in some visually challenged scenarios.

FIG.2shows a flow diagram200illustrating an example for map data generation from road usage information (in this example in form of raw GPS traces)201and a satellite image202.

The raw GPS traces may for example be recorded by vehicles108, e.g. of an e-hailing service (or further vehicles). This means that vehicle108may record the routes they have taken and transmit this information to the server106. The server106may collect these data to perform the map data generation ofFIG.2.

The server106may acquire the satellite image202from a satellite image database (i.e. from an Internet site providing satellite images).

In the example ofFIG.2, the map data is generated in form as a map image from a satellite image (or multiple satellite images) and GPS traces. Thus, the map data generation ofFIG.2can also be seen as a conversion of a satellite image to a map image with the help of road usage information.

The raw GPS traces201and the satellite image202form an input pair. The pairing means that the raw GPS traces201specify routes which fall within the same geographical area as covered by the satellite image202. The raw GPS traces201are converted to a GPS image204via GPS Image Rendering203. The GPS image204(in general a road usage image) is used together with the satellite image202to output a map image206via a GAN-based Image Translation205, i.e. using a GAN.

The GPS Image Rendering203converts the original GPS traces from a textual form into an image (GPS image204). The GPS image204has the same number of pixels as its paired satellite image202. The covered space for each pixel in GPS image is the same as the corresponding pixel in the satellite image. For each GPS image pixel, its covered space is first determined as a rectangle region (bounded by a minimal and maximal latitude and longitude). Then the raw GPS traces201are retrieved that fall within this rectangle region. For each GPS image pixel, its value is determined as one if there is at least one GPS trace falling into that pixel region (region covered by pixel) and zero otherwise. Thus, a binary GPS image204(e.g. representable in black and white) is rendered.

The GAN-based Image Translation205converts a pair of rendered GPS image204and satellite image202into a map image206using a GAN (or GAN-based model).

FIG.3shows a GAN300.

The GAN300is a neural network which contains two parts (i.e. two sub-networks): a generator (network)301and a discriminator (network)302. The discriminator302is only used during the training phase, where both generator301and discriminator302are trained. After training, i.e. for GAN-based Image Translation205using actual input data204,202(i.e. other than training data), the trained generator301is used. The discriminator302is no longer necessary.

According to one embodiment, the input of the generator301is a pair of GPS image303and a satellite image304where the GPS image303contains a single colour channel and the satellite image contains RGB three colour channels. The satellite image304and the GPS image303are concatenated in a channel-wise fashion. by image concatenation305. Here, “channel-wise” means that a GPS image channel is added as a fourth channel to the RGB channels. The result of image concatenation305(i.e. the concatenated image data) is imported to a (trainable) U-Net306. A U-Net has a U-shape architecture. This means that it includes a sequence of encoding layers followed by a sequence of decoding layers and involves down sampling (in the encoding layers) to a bottleneck and up sampling again to an output image (in the decoding layers), but links or skip-connections between encoding layers and decoding layers of the same size allow the bottleneck to be circumvented.

The output of the generator is a map image307that contains RGB three colour channels. The U-Net306contains, for example, 8 encoding layers and, for example, 8 decoding layers with skip connections. Each of the first seven encoding layers is a convolutional layer with 4×4 filters, followed by batch normalization and LeakyReLU activation. Last encoding layer is a convolutional layer with 4×4 filters, followed by ReLu activation. Each of the first seven decoding layers is a transpose convolutional layer with 4×4 filters, followed by batch normalization, dropout (for the first three decoding layers) and ReLu activation. Last decoding layer is a transpose convolutional layer with 4×4 filters, followed by Tanh activation. The number of filters for each of the 8 convolutional layers is for example 64, 128, 256, 512, 512, 512, 512, 512 respectively. The number of filters for each of the 8 transpose convolutional layers is for example 512, 512, 512,512, 256, 128, 64, 3, respectively.

When training the GAN300, the satellite image304and the GPS image303come from training data. The training data includes training data elements wherein each training data element includes a satellite image304and a GPS image303. At least some of the training data elements further include a real street map image of the area covered by the satellite image304and the GPS image303, i.e. a ground truth image (also referred to as target image). These images may be obtained from databases, recording of driver routes (with GPS image rendering) etc.

For training the GAN300, the input of the discriminator302is a pair of input images which is eithera pair of the satellite image304of a training data element and a map image308of the training data element ora pair of a satellite image304of a training data element and a map image307generated by the generator301from the satellite image304and the GPS image303of the training data element.

The discriminator concatenates the two input images of its input image pair in a channel-wise manner by image concatenation309and the result of is image concatenation309fed to a trainable CNN (convolutional neural network). The CNN310outputs a value between 0 and 1 where a larger value indicates that the map image of the input image pair is more likely to be a plausible transformation from the satellite image of the input image pair (i.e. is likely no fake). The CNN310contains, for example, 6 convolutional layers. The filter size is 4×4 and the number of filters are 64, 128, 256, 512, 512 and 1 in each convolutional layer. Each of the first five convolutional layers is followed Batch Normalization (except the first convolutional layer) and LeakyReLu activation. The last convolutional layer is followed by a Sigmoid activation.

The discriminator302may for example be trained to minimize the negative log likelihood of identifying real and fake images. This may be done by updating its weights using backpropagation.

The generator301may be trained using both an adversarial loss (punishing that the discriminator302recognizes an image generated by the generator301as fake) and an L1 or mean absolute pixel difference loss between an image generated from a satellite image of training data element and the map image included in the training data element (i.e. the expected target image or ground truth image). The adversarial loss is for example a binary cross-entropy loss (applied to the output, which is between 0 and 1, of the CNN310).

The adversarial loss and the L1 loss may be combined into a composite loss function, which is used to update the weights of the generator301using backpropagation. The adversarial loss influences whether the generator model can output images that are plausible in the target domain, whereas the L1 loss regularizes the generator301to output images that are a plausible translation of the source image. The combination of the L1 loss to the adversarial loss may be controlled by a hyperparameter, e.g. a, e.g. set to 100, thus giving a times the importance of the L1 loss than the adversarial loss to the generator during training: Generator Loss=Adversarial Loss+α*L1 Loss

As usual, training is performed for batches of training data elements and loss is calculated over the training data elements of a batch. For example, the batch size is set as 32 and the epoch is 100. According to one embodiment, in the testing phase (i.e. neural network testing of the using testing data), only the generator301is used to generate map images.

In summary, according to various embodiments, a method is provided as illustrated inFIG.4.

FIG.4shows a flow diagram400illustrating a method for generating map data, i.e. electronic map data or, in other words, digital map data.

In401, a generator neural network is trained. This is performed by:In402, Acquiring training data elements for a generative adversarial network (GAN), each training data element comprising a satellite image of a geographical area, a road usage image of the geographical area which shows routes in the geographical area which may be used for driving a vehicle and a map data image of the geographical area; andIn403, training a generative adversarial network comprising the generator neural network (as generator network of the GAN, i.e. the generator neural network is the GAN's generator) using the training data elements, comprising training the generator neural network to generate, for a satellite image and a road usage image of a training data element, the map data image of the training data element.

In404, map data for a geographical region (e.g. a part of a city or a whole city etc.) is generated. This is performed by:In405, acquiring road usage information specifying which parts of a geographical area have been used for driving a vehicle;In406, acquiring a satellite image of the geographical area;In407, forming a road usage image of the geographical area which has pixels, each pixel corresponding to a respective part of the geographical area, such each pixel has a pixel value indicating whether the part of the geographical area, to which the pixel corresponds, is specified by the road usage information to have been used for driving a vehicle; andIn408, feeding the satellite image and the road usage image to the trained generator neural network.

According to various embodiments, in other words, map data is generated by jointly using a satellite image and road usage information (e.g. GPS traces of vehicles of an e-hailing service) by means of a generative adversarial network (GAN). This means that map data is generated using road usage information and a satellite image as resources for map data generation. The map data is for example generated in the form of map image data such that, according to one embodiment, road usage information is fed and/or embedded into a GAN for satellite-to-map image conversion.

It should be noted that more than one satellite images may be acquired which may be seen to form, together, a big satellite image (even if being stored in separate image parts). So the term “satellite image” is understood to include the case of several satellite images.

The approach for map data generation ofFIG.4is robust against the occlusions from either physical objects (e.g., underpass) or due to weather factors (e.g., cloud, shadow). Furthermore, a high quality of map data given a certain amount of training data can be achieved. This is especially beneficial in situations where only a small amount of training data (e.g. only 100-200 trainings images) is available.

The approach ofFIG.4may be implemented using any GAN architecture (or GAN model). In particular, it may be incorporated into an existing device which uses a GAN as it is not required to modify the GAN architecture but only the GAN input. In comparison to an approach which uses pre-collected geospatial data, the approach ofFIG.4allows generating map data that is more up-to-date with regard to geospatial semantics.

The method ofFIG.4may be part of a method for controlling a vehicle (or controlling the navigation of a vehicle) wherein the map data is provided to a vehicle and the vehicle (or its navigation system) is controlled using the amended map data.

The method ofFIG.4is for example carried out by a server computer as illustrated inFIG.5.

FIG.5shows a server computer500according to an embodiment.

The server computer500includes a communication interface501(e.g. configured to receive road usage information and satellite images and/or to provide generated map data to another server (e.g. a navigation server) or a navigation device in a vehicle). The server computer500further includes a processing unit502and a memory503. The memory503may be used by the processing unit502to store, for example, road usage information and a satellite image to be processed. The server computer is configured to perform the method ofFIG.4.

The methods described herein may be performed and the various processing or computation units and devices described herein may be implemented by one or more circuits. In an embodiment, a “circuit” may be understood as any kind of a logic implementing entity, which may be hardware, software, firmware, or any combination thereof. Thus, in an embodiment, a “circuit” may be a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g. a microprocessor. A “circuit” may also be software being implemented or executed by a processor, e.g. any kind of computer program, e.g. a computer program using a virtual machine code. Any other kind of implementation of the respective functions which are described herein may also be understood as a “circuit” in accordance with an alternative embodiment.

While the disclosure has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.