Systems and methods for automatically generating training image sets for an environment

A computer-implemented method for generating a training set of images and labels for a native environment includes receiving physical coordinate sets, retrieving environmental model data corresponding to a georeferenced model of the environment, and creating a plurality of two-dimensional (2-D) rendered images each corresponding to a view from one of the physical coordinate sets. The 2-D rendered images include one or more of the environmental features. The method also includes generating linking data associating each of the 2-D rendered images with (i) labels for the one or more included environmental features and (ii) a corresponding native image. Additionally, the method includes storing the training set including the 2-D rendered images, labels, corresponding native images, and linking data.

FIELD

The field of the disclosure relates generally to training image sets for use in training machine vision systems to navigate an environment, and more particularly to automatically generating such training image sets.

BACKGROUND

At least some known machine vision systems are trained to navigate an environment detected by image sensors (e.g., detected by cameras mounted on the machine). For example, at least some known unmanned aerial vehicles (“UAVs”) utilize machine vision systems trained to autonomously navigate environments relevant to various mission objectives of the UAV. For another example, at least some known self-driving automotive vehicles utilize machine vision systems trained to navigate an environment relevant to autonomous driving and/or autonomous pursuit of various objectives of the self-driving vehicle. Such machine vision systems are typically trained using suitable machine learning algorithms as applied to a set of training images.

Such training image sets typically include labels and metadata to facilitate machine learning. For example, the training images may be semantically segmented to identify at least one feature of interest in the environment depicted in the training image. The semantic segmentation may include a mask, such as a preselected color superimposed over each environmental feature in each training image, to train the applied machine learning algorithm to associate the detected outline with the correct environmental feature in the environment. The training images may also include additional labels, such as a name of the environmental feature in the image, and metadata, such as a description of a viewpoint from which the image was captured, a distance to the environmental feature (e.g., if the object is runway signage, the labelling might identify a distance to the signage), etc. Known methods of generating such sets of training images are subject to several limitations. For example, an operator typically manually enters semantic segmentation to the training images, and applies masks of the appropriate colors over each environmental feature in the original image. The process is more time consuming than desired and relies on the skill of the operator. Moreover, large datasets of such training images may be in the order of thousands of images, which may make manual segmentation impractical.

BRIEF DESCRIPTION

One aspect of the present disclosure includes a method for generating a training set of images and labels for a native environment. The method is implemented on a computing system including at least one processor in communication with at least one memory device. The method includes using the at least one processor to receive a plurality of physical coordinate sets, and to retrieve, from the at least one memory device, environmental model data corresponding to a georeferenced model of the environment. The environmental model data defines a plurality of environmental features. The method also includes using the at least one processor to create a plurality of two-dimensional (2-D) rendered images from the environmental model data. Each of the 2-D rendered images corresponds to a view from one of the physical coordinate sets. The plurality of 2-D rendered images includes one or more of the environmental features. The method further includes using the at least one processor to generate linking data associating each of the 2-D rendered images with (i) labels for the one or more included environmental features and (ii) a corresponding native image. Additionally, the method includes using the at least one processor to store the training set including the 2-D rendered images, the labels, the corresponding native images, and the linking data.

Another aspect of the present disclosure includes a computing system for generating a training set of images and labels for a native environment. The computing system includes at least one processor in communication with at least one memory device. The at least one processor is configured to receive a plurality of physical coordinate sets, and to retrieve, from the at least one memory device, environmental model data corresponding to a georeferenced model of the environment. The environmental model data defines a plurality of environmental features. The at least one processor also is configured to create a plurality of two-dimensional (2-D) rendered images from the environmental model data. Each of the 2-D rendered images corresponds to a view from one of the physical coordinate sets. The plurality of 2-D rendered images includes one or more of the environmental features. The at least one processor further is configured to generate linking data associating each of the 2-D rendered images with (i) labels for the one or more included environmental features and (ii) a corresponding native image. Additionally, the at least one processor is configured to store the training set including the 2-D rendered images, the labels, the corresponding native images, and the linking data.

Yet another aspect of the present disclosure includes a non-transitory computer-readable storage medium having computer-executable instructions embodied thereon for generating a training set of images and labels for an environment. When executed by at least one processor in communication with at least one memory device, the computer-executable instructions cause the at least one processor to receive a plurality of physical coordinate sets, and to retrieve, from the at least one memory device, environmental model data corresponding to a georeferenced model of the environment. The environmental model data defines a plurality of environmental features. The computer-executable instructions also cause the at least one processor to create a plurality of two-dimensional (2-D) rendered images from the environmental model data. Each of the 2-D rendered images corresponds to a view from one of the physical coordinate sets. The plurality of 2-D rendered images includes one or more of the environmental features. The computer-executable instructions further cause the at least one processor to generate linking data associating each of the 2-D rendered images with (i) labels for the one or more included environmental features and (ii) a corresponding native image. Additionally, the computer-executable instructions cause the at least one processor to store the training set including the 2-D rendered images, the labels, the corresponding native images, and the linking data.

Although specific features of various examples may be shown in some drawings and not in others, this is for convenience only. Any feature of any drawing may be referenced and/or claimed in combination with any feature of any other drawing.

Unless otherwise indicated, the drawings provided herein are meant to illustrate features of examples of the disclosure. These features are believed to be applicable in a wide variety of systems comprising one or more examples of the disclosure. As such, the drawings are not meant to include all conventional features known by those of ordinary skill in the art to be required for the practice of the examples disclosed herein.

DETAILED DESCRIPTION

Examples of computer-implemented methods for generating training sets of images and labels for a native environment as described herein include creating a plurality of two-dimensional (2-D) rendered images from views of a georeferenced model of a native environment. A georeferenced model is broadly defined as a model of a native environment that links an internal coordinate system of the model to a system of geographic coordinates in the physical world. For example, for a particular airport environment that includes static physical environmental features such as runways, runway markings, additional aircraft-traversable zones, and airport signage each located at specific geographic coordinates in the physical world, a georeferenced model of the environment includes corresponding virtual runways, virtual runway markings, virtual additional aircraft-traversable zones, and virtual airport signage, each defined by internal model coordinates that are linked to the geographic coordinates of the corresponding physical features. Simulated or “rendered” perspective views of the virtual environment are obtainable from the georeferenced model using suitable rendering algorithms (e.g., ray tracing), based on an input set of spatial coordinates (e.g., geographic “physical” location coordinates and physical orientation of the viewpoint) for a selected viewpoint.

The systems and methods disclosed herein are particularly useful for, although not limited to, airport environments because detailed georeferenced models already have been developed for many airports. They also are particularly useful for, although not limited to, regulated or controlled environments, again such as airports, because the nature and placement of environmental features can be expected not to vary significantly over time.

Examples also include generating linking data associating each of the 2-D rendered images with (i) labels for the one or more included environmental features and (ii) a corresponding native image, and storing the 2-D rendered images, the labels, the corresponding native images, and the linking data in the training set. Examples of creating the 2-D rendered images include detecting, using the environmental model data, that at least one of the environmental features appears in the corresponding view, and rendering, for each detected environmental feature, a plurality of pixels that define the detected environmental feature in the 2-D rendered image. Examples of creating the labels include associating with each 2-D rendered image a label corresponding to each detected environmental feature in the 2-D rendered image.

In particular, because the 2-D rendered images are cleanly generated from the georeferenced model with no uncontrolled or unnecessary elements in the image, the pixels representing the environmental features in each 2-D rendered image are precisely identifiable by the computing system, and a suitable algorithm can be applied by the computing system to automatically build or “fill in” the semantic segmentation for the pixels of each environmental feature, with little or no intervening input required from a human operator. Accordingly, the systems and methods of the present disclosure replace the manual effort and subjective judgment required by prior art methods for semantic segmentation of images with high-speed, automated generation of semantic segmentation images that are objectively accurate on a pixel-by-pixel basis, because each semantic segmentation is precisely grounded in the pixels of the environmental feature in the 2-D rendered image.

In some examples, the physical coordinate sets used to generate the 2-D rendered images define a path through the environment. For example, the physical coordinate sets may be obtained by recording the coordinates and orientation of a vehicle traveling along the path, such as by using an on-board Global Positioning Satellite (GPS) system, inertial measurement unit (IMU), and/or other on-board geo-locating system of the vehicle. Thus, training image sets can be easily created for typical situations encountered by a self-guided vehicle, such as standard approaches by an aircraft to each runway of an airport, or standard ground paths of a luggage transport vehicle to each gate area of the airport. In some such examples, the vehicle used to “capture” the path coordinates also carries sensors (e.g., cameras), and images from the sensors are tagged with the physical coordinates of the vehicle, or linked to the physical coordinates of the vehicle along the path by matching timestamps with the on-board GPS system. Thus, each 2-D rendered image, e.g., semantic segmentation image, automatically generated at each set of physical coordinates can be linked to the native or “real” camera image captured at that physical coordinate set, and the camera images can be used as the native images of the training set.

Unless otherwise indicated, the terms “first,” “second,” etc. are used herein merely as labels, and are not intended to impose ordinal, positional, or hierarchical requirements on the items to which these terms refer. Moreover, reference to, for example, a “second” item does not require or preclude the existence of, for example, a “first” or lower-numbered item or a “third” or higher-numbered item.

FIG. 1is an example schematic representation of an image of a native environment100as viewed from a first vantage point. Environment100includes a plurality of static, physical environmental features, referred to collectively as environmental features110, each associated with corresponding geographic coordinates in the physical world. In the example, the image of native environment100also includes a plurality of objects102that are not included in the georeferenced model. For example, objects102are temporary or dynamic physical objects that may be found in different places at different times within native environment100.

In the example, native environment100is an airport, and environmental features110include permanent or semi-permanent features typically present in an airport environment. For example, environmental features110include a runway120and a centerline122of runway120. Although only one runway120is shown from the vantage point used to obtain the image inFIG. 1, it should be understood that native environment100may include any suitable number of runways120. Environmental features110also include a plurality of taxiways130and position markings132. For example, position markings132are surface markings at runway holding positions, taxiway intersections, and/or taxiway/runway intersections. Environmental features110further include an apron140and a building150, such as a hangar or terminal. In addition, environmental features110include a plurality of signs160, such as runway/taxiway location and/or direction signs. While the environmental features110listed above are typical of airport environments, they are not exclusive or required for native environment100.

Although aspects of the disclosure are described in terms of an airport environment for illustrative purposes, in alternative implementations, native environment100is any suitable environment that includes environmental features110that may be characterized as static, physical environmental features.

FIGS. 3A and 3Bare a schematic representation of an example data acquisition and processing framework for generating a training set of images and labels for native environment100.FIG. 3Cis a schematic block diagram of an example computing system300for generating a training set of images and labels for native environment100that may be used to implement the framework ofFIGS. 3A and 3B. In particular, computing system300includes at least one processor302configured to generate 2-D rendered images340from a georeferenced model of native environment100.

Starting withFIG. 3C, the at least one processor302is configurable to perform one or more operations described herein via programming the at least one processor302. For example, the at least one processor302is programmed to execute a model data manipulation module320, an image processing module322, a data linking module324, and/or other suitable modules which perform steps as described below.

In the example, computing system300includes at least one memory device304operatively coupled to the at least one processor302, and the at least one processor302is programmed by encoding an operation as one or more computer-executable instructions306and providing the computer-executable instructions306in the at least one memory device304. In some examples, the computer-executable instructions are provided as a computer program product by embodying the instructions on a non-transitory computer-readable storage medium. The at least one processor302includes, for example and without limitation, a graphics card processor, another type of microprocessor, a microcontroller, or other equivalent processing device capable of executing commands of computer readable data or programs for executing model data manipulation module320, image processing module322, data linking module324, and/or other suitable modules as described below. In some examples, the at least one processor366includes a plurality of processing units, for example and without limitation, coupled in a multi-core configuration. In certain examples, the at least one processor302includes a graphics card processor programmed to execute image processing module322and a general-purpose microprocessor programmed to execute model data manipulation module320, data linking module324, and/or other suitable modules.

In the example, the at least one memory device304includes one or more devices that enable storage and retrieval of information such as executable instructions and/or other data. The at least one memory device304includes one or more computer readable media, such as, without limitation, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), a solid state disk, a hard disk, read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and/or non-volatile RAM (NVRAM) memory. The above memory types are examples only, and are thus not limiting as to the types of memory usable as the at least one memory device304. The at least one memory device304is configured to store, without limitation, application source code, application object code, source code portions of interest, object code portions of interest, configuration data, execution events, and/or any other type of data.

In the example, computing system300includes a display device372coupled to the at least one processor302. Display device372presents information, such as a user interface, to an operator of computing system300. In some examples, display device372includes a display adapter (not shown) that is coupled to a display device (not shown), such as a cathode ray tube (CRT), a liquid crystal display (LCD), an organic LED (OLED) display, and/or an “electronic ink” display. In some examples, display device372includes one or more display devices.

In the example, computing system300includes a user input interface370. User input interface370is coupled to the at least one processor302and receives input from an operator of computing system300. User input interface370includes, for example, a keyboard, a pointing device, a mouse, a stylus, a touch sensitive panel, e.g., without limitation, a touch pad or a touch screen, and/or an audio input interface, e.g., without limitation, a microphone. A single component, such as a touch screen, is capable of functioning as both display device372and user input interface370.

In some examples, computing system300includes a communication interface374. Communication interface374is coupled to the at least one processor302and is configured to be coupled in communication with one or more remote devices, such as but not limited to a network server, and to perform input and output operations with respect to such devices. For example, communication interface374includes, without limitation, a wired network adapter, a wireless network adapter, a mobile telecommunications adapter, a serial communication adapter, and/or a parallel communication adapter. Communication interface374receives data from and/or transmits data to the one or more remote devices for use by the at least one processor302and/or storage in the at least one memory device304.

In other examples, computing system300is implemented in any suitable fashion that enables computing system300to perform the steps described herein.

With reference also toFIG. 1, the at least one processor302has access to environmental model data308corresponding to a georeferenced model of environment100. Environmental model data308may be compiled using, for example, aerial maps, semantic maps, feature maps, and/or other suitable spatial data or metadata regarding environment100. In the example, environmental model data308is stored in the at least one memory device304, and the at least one processor302is programmed to retrieve environmental model data308from the at least one memory device304.

Environmental model data308includes data associated with each of environmental features110. More specifically, environmental model data308includes, for example, a unique identifier312for each environmental feature110and a type314of each environmental feature110. Environmental model data308also includes a spatial extent316of each environmental feature110within environment100. In particular, spatial extent316of each environmental feature110stored in environmental model data308is linked to the geographic coordinates of the environmental feature110. In the example, because objects102are not static physical environmental features of environment100, objects102are not represented in environmental model data308.

As noted above, in the example, environment100is an airport, and environmental model data308includes unique identifier312and type314for each runway120, centerline122of runway120, taxiway130, position marking132, apron140, building150, and sign160. More specifically, each individual environmental feature110has unique identifier312that differs from unique identifier312of every other environmental feature110included in the georeferenced model. Environmental features110of a like category (e.g., runways120, signs160) share an identical type314. The categorization described for the example is non-limiting. For example, signs160may be further divided into types314for runway signs, apron signs, etc. Alternatively, type314is not included in environmental model data308for at least some environmental features110.

Spatial extent316of each environmental feature110may be determined from semantic map data for a reference viewpoint, feature map data for a reference viewpoint, and/or other metadata within environmental model data308. Additionally or alternatively, spatial extent316is stored using values in a predefined data structure. Different data structures may be defined to correspond to the type314of environmental feature110. For example, spatial extent316is defined for certain environmental features110using boundary coordinates. Alternatively, spatial extent316is defined and/or stored in any suitable fashion that enable computer system300to function as described herein.

The at least one processor302also is programmed to receive a plurality of physical coordinate sets332. In the example, physical coordinate sets332are stored in the at least one memory device304, and the at least one processor302is programmed to retrieve physical coordinate sets332from the at least one memory device304. Each physical coordinate set332defines a vantage point in physical space from which environment100is viewable. For example, each physical coordinate set332includes a location (e.g., latitude, longitude, elevation) and direction of view from the location (e.g., heading, angle of attack, roll angle). Taken together, the plurality of physical coordinate sets332represents an ensemble of vantage points, e.g., spatial relationship of the viewer in terms of six degrees of freedom relative to the geospatial coordinate system, for images in a training set350to be used to train a machine vision system362. More specifically, training set350includes native images356and corresponding 2-D rendered images340, and a training algorithm360is programmed to use corresponding pairs of native images356and 2-D rendered images340to train machine vision system362to recognize environmental features110.

In some examples, physical coordinate sets332define a path through environment100. For example, physical coordinate sets332are a sequence of points through which a UAV or other aircraft might travel through approach and landing on runway120, and/or through taxiing on taxiways130and apron140towards building150. For another example, physical coordinate sets332are a sequence of points through which a self-driving luggage transport vehicle (not shown) might travel between building150and various gate locations on apron140. Alternatively, physical coordinate sets332are not associated with a path through environment100.

In some examples, the at least one processor302receives a plurality of camera images330each associated with one of physical coordinate sets332. For example, a test vehicle380having cameras382mounted thereon traverses the path defined by physical coordinate sets332, and recording equipment384records camera images330captured along the path. It should be understood that the terms “camera images” and “camera” refer broadly to any type of image acquirable by any type of image capture device, and are not limited to images captured in visible light or images captured via a lensed camera. In certain examples, the at least one processor302is programmed to receive camera images330via communication interface374and store camera images330in the at least one memory device304. Moreover, in some examples, the at least one processor302is programmed to include camera images330in training set350as native images356. Alternatively, the at least one processor302is not programmed to receive camera images330, and/or camera images330are not included as native images356in training set350.

In certain examples, each of camera images330includes a corresponding geo-coordinate tag, and the at least one processor302is programmed to receive physical coordinate sets332by extracting the geo-coordinate tags from camera images330. For example, the test vehicle also includes an on-board geo-locating system386such as a GPS receiver and/or inertial measurement unit (IMU), and as test vehicle380captures each camera image330along the path, the corresponding physical coordinate set332is captured from on-board geo-locating system386and embedded in the captured camera image330as the geo-coordinate tag. Alternatively, camera images330and physical coordinate sets332are recorded separately with time stamps by cameras382and on-board geo-location system386, respectively, and the respective time stamps are synchronized to associate each camera image330with the correct physical coordinate set332. Alternatively, the at least one processor302is programmed to receive physical coordinate sets332in any suitable fashion, such as a listing of numeric coordinate and orientation values in, e.g., a text file. In some examples, the at least one memory device304further stores a displacement and orientation of each camera382relative to a location and orientation of on-board geo-locating system386, and the at least one processor302is programmed to adjust, for each camera382, the geo-coordinate tags based on the displacement and orientation of the respective camera382to obtain a more accurate physical coordinate set332.

As discussed above, prior art systems for creating a training set would require a human operator to manually identify environmental features110in camera images330and semantically segment each environmental feature110in each camera image330in order to complete the training image set, which would be extremely time-intensive and which would result in a subjective, not strictly accurate fit of the environmental features110on a pixel-by-pixel basis. Computing system300provides advantages over such prior art systems by creating 2-D rendered images340from environmental model data308and automatically semantically segmenting 2-D rendered images340to create semantic segmentation images352.

In the example, each of the 2-D rendered images340corresponds to a view from one of the physical coordinate sets332. For example, the at least one processor302is programmed to apply a suitable rendering algorithm to environmental model data308to detect each environmental feature110that appears in the view defined by a given physical coordinate set332, and to render, for each detected environmental feature110, a plurality of pixels342that define the detected environmental feature110in the resulting 2-D rendered image340. For example, spatial extent316enables the at least one processor302to determine whether the corresponding environmental feature110appears within a bounding box, or region of interest (ROI), associated with the view of environment100defined by the specified physical coordinate set332. The algorithm maps the view defined by physical coordinate set332against spatial extent316of each detected environmental feature110in the ROI to render the plurality of pixels342. Suitable rendering algorithms, such as but not limited to ray-tracing algorithms, are known and need not be discussed in depth for purposes of this disclosure. One such ray-tracing algorithm is provided in the Unity Pro product sold by Unity Technologies ApS of San Francisco, Calif. In the example, each 2-D rendered image340is stored as a portable network graphic (PNG) image file in the at least one memory device304. Alternatively, each 2-D rendered image340is stored in any suitable format that enables training set350to function as described herein.

In the example, the at least one processor302is programmed to create 2-D rendered images340including semantic segmentation images352. More specifically, to create each semantic segmentation image352, the at least one processor302is programmed to apply a visualization mode that renders plurality of pixels342corresponding to each detected environmental feature110in a corresponding semantic color. In some examples, the at least one processor302is programmed in the semantic segmentation visualization mode to associate each type314of environmental feature110with a pre-selected color, and, for each detected environmental feature110in 2-D rendered image340, to render the pixels342with the pre-selected color associated with type314of the respective environmental feature110. Thus, for example, all runways120may be rendered with an identical bright red color to create semantic segmentation images352. In some examples, “background” pixels not corresponding to detected environmental features110are rendered in a neutral background color to enhance a contrast with pixels342, or alternatively rendered in a naturalized RGB background palette. For example, the color key for each type314is included in metadata358of training set350, or training algorithm360is otherwise programmed to associate each pre-selected color with the corresponding type314of environmental feature110. Advantageously, because the at least one processor302automatically and precisely determines pixels342corresponding to each of the detected one or more environmental features110in the course of rendering of 2-D rendered image340, and automatically colors precisely those pixels342to create semantic segmentation images352, computing system300generates semantic segmentation images352having precise pixel-level accuracy in a high-speed process that requires no manual study or manual manipulation of native images356.

In the example, the at least one processor302is programmed to selectively create 2-D rendered images340in a plurality of visualization modes. For example, in addition to the above-described visualization mode for creating semantic segmentation images352, the at least one processor302is programmed to apply an RGB visualization mode to create additional 2-D rendered images340for each physical coordinate set332. For example, the plurality of pixels342corresponding to each detected environmental feature110is rendered in a naturalized red-green-blue (RGB) feature palette, associated for example with the respective type314or unique identifier312of the detected environmental feature110, and the background is further rendered in a naturalized RGB background palette to create a 2-D synthetic image344that approximates a physical appearance of the corresponding native image356. In some such embodiments, each semantic segmentation image352may be conceptualized as corresponding to an underlying 2-D synthetic image344, but having a feature-type-based semantic color superimposed on each environmental feature110. For another example, the at least one processor302is programmed to apply a depth-map visualization mode to create additional 2-D rendered images340for each physical coordinate set332. In the depth-map visualization mode, the plurality of pixels342corresponding to each detected environmental feature110are rendered in a color scale corresponding to a physical distance of the pixels342from the location coordinates of physical coordinate set332to create a 2-D depth map346. Each of 2-D synthetic (RGB) images344and 2-D depth maps346may be used in training set350to improve a performance of training algorithm360. In some examples, the at least one processor302is programmed to apply additional or alternative visualization modes to create additional 2-D rendered images340for each physical coordinate set332.

Turning now toFIG. 2A, a schematic representation of an example semantic segmentation image352of environment100as viewed from the first vantage point ofFIG. 1is presented.FIG. 2Bis a detail view ofFIG. 2A. The one or more environmental features110detected by the rendering algorithm, based on environmental feature data310, include runway120, centerline122of runway120, taxiways130, position markings132, apron140, building150, and signs160(shown inFIG. 1). The at least one processor302renders pixels342corresponding to runway120with a first color220, renders pixels342corresponding to centerline122with a second color222, renders pixels342corresponding to taxiways130with a third color230, renders pixels342corresponding to position markings132with a fourth color232, renders pixels342corresponding to apron140with a fifth color240, renders pixels342corresponding to building150with a sixth color250, and renders pixels342corresponding to signs160with a seventh color260.

FIG. 4is an example of labels404and452, metadata358, and linking data354generated by computing system300for training set350for environment100. The process of generating training set350further includes using the at least one processor302to generate linking data354associating each 2-D rendered image340with the corresponding native image356. In the example, the at least one processor302generates linking data354as a data structure400that includes a plurality of records401. For example, data structure400is a comma separated value (CSV) file or a table in a database. Each record401includes a first pointer402to at least one of the 2-D rendered images340and a second pointer403to the corresponding native image356. Training algorithm360is configured to parse training set350for corresponding pairs of 2-D rendered images340and native images356based on the information in data structure400. It should be understood that the term “pointer” as used herein is not limited to a variable that stores an address in a computer memory, but rather refers more broadly to any element of information that identifies a location (e.g., file path, memory location) where an object (e.g., 2-D rendered image340, native image356) associated with the pointer is accessible.

In the example, first pointer402is implemented as a file path and file name of an image file stored in the at least one memory device304and storing 2-D rendered image340, and second pointer403is implemented using time metadata440corresponding to a time at which the corresponding native image356was captured as camera image330. For example, a timestamp is stored with each native image356(e.g., as metadata in the image file). Training algorithm360parses the timestamp stored with each native image356, finds a corresponding timestamp442in time metadata440of one of records401, and follows first pointer402in the identified record401to find the 2-D rendered image340corresponding to native image356.

Alternatively, physical coordinate set332is used as second pointer403, and is used to match each native image356to a corresponding record401in similar fashion as described above for timestamp442. For example, physical coordinate set332used to generate 2-D rendered image340is stored in the corresponding record401, and matched against the physical coordinate set captured and stored with the respective native image356(e.g., as metadata in the image file). Alternatively, second pointer403is implemented as a file name and path to the stored native image356. Alternatively, each record401includes first pointer402and second pointer403implemented in any suitable fashion that enables training set350to function as described herein.

Alternatively, each record401links 2-D rendered image340to the corresponding native image356in any suitable fashion that enables training set350to function as described herein.

In the example, training set350includes visualization mode labels404for each 2-D rendered image340. For example, data structure400includes label404for the 2-D rendered image340associated with each record401. Visualization mode label404identifies the visualization mode used to create the image, such as “SEM” for semantic segmentation image352, “RGB” (i.e., red-green-blue) for 2-D synthetic (RGB) image344, and “DEP” for depth map346. Alternatively, visualization mode label404is not included in training set350. For example, training set350includes 2-D rendered images340of a single visualization mode.

In the example, training set350further includes feature labels452for each environmental feature110detected in 2-D rendered image340. For example, feature label452is a text string based on unique identifier312and/or type314of the detected environmental feature110. Although only one feature label452is illustrated in each record401inFIG. 4, it should be understood that any number of feature labels452may be included in each record401based on a number of detected environmental features110in the corresponding 2-D rendered image340. Additionally or alternatively, training set350includes any suitable additional or alternative labels that enable training set350to function as described herein.

In some examples, each record401also includes metadata358. In the example ofFIG. 4, in cases where physical coordinate set332is not already present as second pointer403, metadata358includes physical coordinate set332. In the example, physical coordinate set332is represented as a latitude432, longitude434, elevation436, and heading438. Additional orientation variables in physical coordinate set332include, for example, angle of attack and roll angle (not shown) Alternatively, each record401is associated with physical coordinate set332in any suitable fashion that enables training set350to function as described herein. In some implementations, physical coordinate set332is represented by additional and/or alternative data fields in any suitable coordinate system (e.g., polar coordinates or WGS84 GPS) relative to the georeferenced model.

In the example, metadata358also includes a sensor index406for the corresponding native image356. For example, test vehicle380includes multiple cameras382, and the one of the multiple cameras382associated with native image356corresponding to second pointer403in record401is identified by sensor index406. In some examples, as discussed above, the at least one memory device304stores a displacement and orientation of each camera382relative to a location and orientation of on-board geo-locating system386, and the at least one processor302retrieves the stored displacement and orientation based on sensor index406to adjust physical coordinate set332for the corresponding camera382. Alternatively, sensor index406is not included in metadata358.

In the example, metadata358further includes time metadata440. For example, time metadata440includes a relative time444of traversal along the path as calculated from timestamp442. In cases where timestamp442is not already present as second pointer403, metadata358also includes timestamp442. Alternatively, metadata358does not includes time metadata440.

In some examples, metadata358further includes spatial relationship metadata454associated with at least some feature labels452. More specifically, spatial relationship metadata454defines a spatial relationship between physical coordinate set332and the detected environmental feature110corresponding to feature label452. For example, training algorithm360is configured to train machine vision system362to recognize a distance to certain types314of environmental features110, and spatial relationship metadata454is used for that purpose in training algorithm360. In some examples, the at least one processor302is programmed to use spatial relationship metadata454in creating 2-D depth map346, as discussed above.

In the example, spatial relationship metadata454is implemented as a distance. More specifically, the at least one processor302is programmed to, for each 2-D rendered image340, calculate, based on environmental model data308, a straight-line distance from the corresponding physical coordinate set332to each detected environmental feature110. Alternatively, spatial relationship metadata454includes any suitable set of parameters, such as relative (x, y, z) coordinates.

In the example, the at least one processor302is programmed to generate additional linking data450associating spatial relationship metadata454with the corresponding 2-D rendered image340, and store, in the at least one memory device304, spatial relationship metadata454and the additional linking data450as a portion of training set350. For example, additional linking data450is implemented by including each feature label452and the corresponding spatial relationship metadata454in the record401of data structure400corresponding to 2-D rendered image340. Alternatively, spatial relationship metadata454and/or the additional linking data450are stored as part of training set350in any suitable fashion that enables training set350to function as described herein. For example, the additional linking data450and spatial relationship metadata454are stored in the metadata of an image file storing the corresponding 2-D rendered image340.

It should be understood that in some implementations, linking data354includes additional and/or alternative fields from those shown inFIG. 4.

FIG. 5Ais an example of a baseline 2-D rendered image500created from environmental model data308and one of physical coordinate sets332as discussed above.FIG. 5Bis an example of 2-D rendered image340corresponding to baseline 2-D rendered image500and having a simulated environmental variation502and background504added thereto.

In the example, baseline 2-D rendered image500is one of 2-D rendered images340generated from environmental model data308using, for example, a ray-tracing algorithm and including default background image aspects and/or default variable environmental effects (e.g., weather, time-of-day lighting effects). In some cases, native images356and/or the images machine vision system362sees in the field through its one or more cameras may include diverse backgrounds, weather, and/or time-of-day lighting. In certain examples, this could lead to a mismatch between native images356and 2-D rendered images340in training set350, on the one hand, and the images machine vision system362sees in the field through its one or more cameras, which would potentially degrade the effectiveness of training set350.

In some examples, the at least one processor302is further programmed to create 2-D rendered images340having a plurality of simulated environmental variations502and/or backgrounds504to account for such diversity in the real-world images. In some such implementations, the at least one processor302is programmed to apply a plurality of modifications to environmental model data308, each modification corresponding to a different environmental variation502and/or different background504. For example, environmental model data308is modified to include different positions of the sun, resulting in the rendering of 2-D rendered images340that include environmental variations502and backgrounds504representative of different time-of-day lighting effects. For another example, environmental model data308is modified to include a 3-D distribution of water droplets corresponding to a selected cloud, fog, or precipitation profile in environment100, and the at least one processor302is programmed to associate suitable light-scattering/light-diffraction properties with the water droplets, resulting in the rendering of 2-D rendered images340that include that include environmental variations502and backgrounds504representative of weather-induced visibility effects and/or cloud formation backgrounds.

Additionally or alternatively, the at least one processor302is programmed to apply the ray-tracing algorithm solely with default background image aspects and/or default variable environmental effects to produce baseline 2-D rendered images500, and to apply 2-D modifications directly to baseline 2-D rendered images500to create additional 2-D rendered images340having the plurality of simulated environmental variations502and/or backgrounds504. For example, the at least one processor302is programmed to superimpose each baseline 2-D rendered image500over stock 2-D images representing a plurality of different backgrounds504to create additional 2-D rendered images340having, e.g., different cloud formation backgrounds. In one implementation, the at least one processor302identifies portions of baseline 2-D rendered images500corresponding to the sky, flags those portions for deletion in favor of the pixels of background504when baseline 2-D rendered images500are superimposed over background504(e.g., the at least one processor302treats those portions as a “green screen”), and superimposes each modified baseline 2-D rendered image500over one or more weather and/or time of day backgrounds504to create one or more additional 2-D rendered images340from each baseline 2-D rendered image500. For example,FIG. 5Billustrates the baseline 2-D rendered image500fromFIG. 5A, which includes runway120, centerline122, and one of signs160, superimposed over a “cloudy day” background504.

For another example, the at least one processor302is programmed to apply a 2-D lighting effect algorithm to baseline 2-D rendered images500to create additional 2-D rendered images340having environmental variations502corresponding to particular times of day or other ambient light conditions in environment100. In the example illustrated inFIG. 5B, a “lens flare” environmental variation502is added to the baseline 2-D rendered image500shown inFIG. 5A, and is created by a lighting modification algorithm propagated from an upper right-hand corner of baseline 2-D rendered image500. For another example, a raindrop algorithm simulates a rain drop506on a camera lens by applying a localized fisheye-lens distortion to one or more random locations on baseline 2-D rendered image500to create a corresponding additional 2-D rendered image340. In some examples, similar 2-D modifications to baseline 2-D rendered image500are used to create additional 2-D rendered images340having environmental variations502that represent reduced visibility caused by clouds or fog.

Accordingly, computing system300enables generation of a set of training images of an environment under a variety of environmental conditions without any need to wait for or rely on changes in time of day or weather conditions.

Additionally or alternatively, the at least one processor302is programmed to create 2-D rendered images340having environmental variations502and/or different backgrounds504in any suitable fashion that enables training set350to function as described herein, or is not programmed to include environmental variations502and/or backgrounds504.

Similarly, in some examples, the at least one processor302is further programmed to apply simulated intrinsic sensor effects in 2-D rendered images340.FIG. 6Ais an example of a physical test pattern600that may be viewed by a camera used by machine vision system362.FIG. 6Bis an example of an acquired image650of physical test pattern600acquired by the camera. Physical test pattern600is a checkerboard pattern defined by straight horizontal lines602and straight vertical lines604. However, due to intrinsic sensor effects of the camera, acquired image650of physical test pattern600is warped, such that straight horizontal lines602become curved horizontal lines652and straight vertical lines604become warped vertical lines654. In an absence of further modification, 2-D rendered images340generated from environmental model data308using, for example, a ray-tracing algorithm do not include warping caused by intrinsic sensor effects such as that illustrated in acquired image650. In certain examples, this could lead to a mismatch between native images356and 2-D rendered images340in training set350, on the one hand, and the images machine vision system362sees in the field through its one or more cameras, which would potentially degrade the effectiveness of training set350.

In the example, the at least one processor302is programmed to account for such intrinsic sensor effects in creating 2-D rendered images340. In other words, the at least one processor302is programmed to intentionally distort a non-distorted rendered image. More specifically, the at least one processor302is programmed to apply simulated intrinsic sensor effects in 2-D rendered images340. For example, the 2-D rendered images340are initially created from environmental model data308at views corresponding to the physical coordinate sets332, e.g. using a suitable ray-tracing algorithm as discussed above, and then an intrinsic sensor effect mapping algorithm is applied to the initial output of the ray-tracing algorithm to complete 2-D rendered images340.

For example, one such intrinsic sensor effect mapping algorithm is to map x- and y-coordinates of each initial 2-D rendered image to xd- and yd-coordinates to generate the corresponding 2-D rendered image340according to the formulae:
xd=x(1+k1r2+k2r4); and
yd=y(1+k1r2+k2r4);

where r=radius to point (x, y) from a center of the initial 2-D rendered image.

Factors k1 and k2 are determined for a particular camera by, for example, comparing acquired image650captured by the camera to physical test pattern600. For a camera having a fish-eye lens, a further factor k3 also may be determined and applied using a suitable extended mapping. Alternatively, the at least one processor302is programmed to account for such intrinsic sensor effects in creating 2-D rendered images340in any suitable fashion that enables training set350to function as described herein, or is not programmed to include intrinsic sensor effects.

Additionally or alternatively, the at least one processor302is programmed to apply any suitable additional processing in creating 2-D rendered images340and/or in processing native images356. For example, at least some known examples of training algorithm360perform better on training image sets350having a relatively low image resolution. The at least one processor302may be programmed to reduce an image resolution of camera images330prior to storing camera images330as native images356, and to create 2-D rendered images340having a corresponding reduced image resolution. For another example, at least some known examples of training algorithm360perform better on training sets350that do not include large swaths of unsegmented background image. The at least one processor302may be programmed to crop camera images330and/or 2-D rendered images340prior to storing.

FIG. 7Ais a flow diagram of an example method700for generating a training set of images and labels, such as training set350, for a native environment, such as native environment100. As described above, method700is implemented on a computing system including at least one processor in communication with at least one memory device, such as computing system300including at least one processor302in communication with at least one memory device304. In the example, the steps of method700are implemented by the at least one processor302.FIGS. 7B and 7Care continuations of the flow diagram ofFIG. 7A.

With reference also toFIGS. 1-6, in the example, method700includes receiving702plurality of physical coordinate sets332. In some examples, the step of receiving702physical coordinate sets332includes receiving704camera images330captured during a physical traversal of a path, and extracting706geo-coordinate tags from camera images330to obtain physical coordinate sets332. In certain examples, the step of receiving702physical coordinate sets332includes receiving708a listing of numeric coordinate values.

In the example, method700also includes retrieving710environmental model data308corresponding to a georeferenced model of environment100. Environmental model data308defines a plurality of environmental features110.

In the example, method700further includes creating7122-D rendered images340from environmental model data308. Each of the 2-D rendered images340corresponds to a view from one of physical coordinate sets332. The plurality of 2-D rendered images340includes one or more of the environmental features110. In some examples, the step of creating7122-D rendered images340includes applying714a plurality of modifications to environmental model data308, each modification corresponding to a different environmental variation502and/or different background504. Additionally or alternatively, the step of creating7122-D rendered images340also includes determining716, for each of the 2-D rendered images340, a visualization mode, and rendering718, for each of the one or more environmental features110, a plurality of pixels that define the environmental feature in a color corresponding to the determined visualization mode (e.g., semantic segmentation mode, RGB mode, or depth map mode). Additionally or alternatively, in certain examples, the step of creating7122-D rendered images340includes applying720simulated intrinsic sensor effects in 2-D rendered images340. In some examples, as an alternative to or in addition to step714, the step of creating7122-D rendered images340includes applying7222-D modifications directly to baseline 2-D rendered images500to create additional 2-D rendered images340having a plurality of simulated environmental variations502and/or backgrounds504.

In some examples, method700also includes creating724at least one of labels and metadata. For example, the step of creating724at least one of labels and metadata includes assigning726visualization mode label404for each 2-D rendered image340. For another example, the step of creating724at least one of labels and metadata includes generating728, for each of the one or more of the environmental features110, a respective one of the feature labels452for each 2-D rendered image340in which the environmental feature110appears. For another example, the step of creating724at least one of labels and metadata includes calculating730, based on environmental model data308, metadata358including spatial relationship metadata454from the corresponding physical coordinate set332to at least one detected environmental feature110.

In the example, method700also includes generating736linking data354associating each of 2-D rendered images340with (i) labels for the one or more included environmental features110and (ii) a corresponding native image356. In some examples, the step of generating736linking data354includes generating738linking data354as a data structure400that includes a plurality of records401, and each record401includes first pointer402to at least one of the 2-D rendered images340and second pointer403to the corresponding native image356. In certain examples, each camera image330is associated with timestamp442corresponding to a relative time444of traversal along the path, the step of generating736linking data354includes generating740linking data354as a data structure400that includes a plurality of records401, and each record401includes (i) pointer402to at least one of the 2-D rendered images340and (ii) time metadata440comprising at least one of timestamp442and relative time444associated with the corresponding native image356.

In certain examples, method700further includes, for each 2-D rendered image340, generating744additional linking data450associating metadata358with the 2-D rendered image340, and storing746metadata358and the additional linking data450as a portion of training set350. In some examples, the step of storing746metadata358and the additional linking data450includes including748, in at least one record401of data structure400of linking data354, the feature label452for each of the one or more environmental features110and the spatial relationship metadata454from each of the one or more environmental features110to the corresponding physical coordinate set332.

In the example, method700also includes storing750training set350including 2-D rendered images340, labels such as visualization mode label404and/or feature label452, corresponding native images356, and linking data354. In some examples, training set350is subsequently transmitted to training algorithm360to train machine vision system362to navigate environment100.

The above described examples of computer-implemented methods and systems for generating training sets for a native environment make use of 2-D rendered images created from views of a georeferenced model of the environment. The examples include rendering pixels that define each detected environmental feature in each 2-D rendered image according to a preselected color scheme, such as a semantic segmentation, a red-green blue natural scheme, or a depth map, and generating linking data associating the 2-D rendered images with the corresponding native image. The examples further include storing the training set, including the 2-D rendered images, the native images, labels, and the linking data. In some examples, camera images captured along a path through the environment are used as the native images in the training image set, and the physical coordinate sets are extracted from the physical location and orientation of the camera at each image capture. In some examples, at least one of extrinsic sensor effects, intrinsic sensor effects, and varying background imagery is added to the 2-D rendered images to create more robust training sets.

Example technical effects of the methods, systems, and apparatus described herein include at least one of: (a) high-speed, automated generation of semantic segmentation images for training sets; (b) generation of semantic segmentation images that are objectively accurate on a pixel-by-pixel basis; (c) generation of a set of training images of an environment under a variety of environmental conditions without any need to wait for or rely on physical scene adjustments; and (d) simulation of a variety of extrinsic and/or intrinsic sensor effects in computer-generated 2-D rendered images without need for any physical camera and/or physical scene adjustments.

The systems and methods described herein are not limited to the specific examples described herein, but rather, components of the systems and/or steps of the methods may be utilized independently and separately from other components and/or steps described herein.

As used herein, an element or step recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural elements or steps unless such exclusion is explicitly recited. Furthermore, references to “one example” of the present disclosure or “an example” are not intended to be interpreted as excluding the existence of additional examples that also incorporate the recited features.