Patent Description:
In a simulation and training environment, data content that represents the world in the 3D whole Earth model may have issues with smooth blending between types of terrain textures and repetitiveness of a single type of terrain texture if the terrain extends for a significant distance in a visual scene. <NPL>, discloses a method of generating realistic terrain for use in simulators.

<NPL>, disclose the large-scale texture synthesis from many input images by a Generative Adversarial Network.

According to a first aspect of the invention, there is provided an aircraft system or simulator system according to claim <NUM>, featuring in particular a neural network using as input, besides image data associated with a location, ecological data pertaining to land type and near-infrared data to identify foliage, to output a top-down photorealistic image of the location.

According to a second aspect, there is provided a method according to claim <NUM>.

Broadly, embodiments of the inventive concepts disclosed herein are directed to a method and a system configured to use image data and geo-specific data to generate a photo-realistic image.

Some embodiments may generate photorealistic content for given land types. Some embodiments may use any suitable neural network (e.g., a conditional generative adversarial network (cGAN)) to generate photorealistic content. For example, the neural network may be trained using an image combination pipeline including geo-specific data (e.g., land type data, elevation data, and/or vector data) and actual image data in a generative adversarial environment. Based on the features and patterns learned during the training, the generator part of the neural network may render specific land-use content (e.g., buildings and/or trees) with typical orientation and position (e.g., buildings near and facing roads and/or trees not on roads) at runtime. The technique may generate photorealistic texture between different land types and may avoid any repeating pattern issues that emerge from traditional algorithmic approaches. Some embodiments save storage and costs compared to covering the world with photographic texture and may look better than a theme-based approach, which can have issues with transitioning realistically between different land types, therefore giving a huge return-on-investment.

In some embodiments, texture generation may involve generating textures for the entire world and displaying the textures in a simulator as needed to create a representative scene for the trainee at the area of the world the trainee is located. Some embodiments may use a neural network to generate photo-realistic geo-typical textures with proper level of detail (LOD) for the entire world for use in the simulator. To accomplish this, the neural network may be trained with photo specific images, elevation data, ground type identification data, and other data. The simulator may use the world location of the trainee to retrieve and display the appropriate texture generated by the neural network.

Some embodiment may utilize deep learning neural networks, which need to be trained before the neural networks can be utilized. The training typically uses significant amounts of training data and many hours of computational time. After the network is trained, the network can execute very quickly (e.g., in milliseconds) to produce photo-realistic content. Some embodiments may utilize a cGAN. cGANs are well known in the art. For example, cGANs and use of the Pix2Pix algorithm are explained in more detail in "<NPL>.

Some embodiments may have multiple benefits, such as photo-quality texture that is otherwise unachievable with current technology, significant reduction in data size (e.g., from <NUM> Terabytes (Tb) to <NUM> Megabytes (Mb)), the ability to capture changes to cities, airports, roadways, etc. with no manual intervention, and money savings. Additionally, in some embodiments, all changes may be captured for free in open source data. Some embodiments may replace the need for customers to use expensive photographic imagery for ground texture, whereas, previously, customers had to choose where to place expensive photographic insets within a whole world theme-based approach. Some embodiments may render more realistic texture, over more area, than existing approaches. Some embodiments may eliminate expensive manual processes, which may: reduce the need for custom textures, which are expensive to develop; eliminate the huge manual effort required to generate ground texture themes via traditional algorithmic approaches; and/or support run-time-publishing so data can be updated and rendered immediately rather than via a costly and time-consuming offline process. Some embodiments may substantially improve realism while lowering the cost of generating terrain texture content on the fly for the entire Earth.

Referring now to <FIG>, an exemplary embodiment of a system <NUM> according to the inventive concepts disclosed herein is depicted. The system <NUM> may be implemented as an aircraft system and/or a multiple computing device system. For example, as shown in <FIG>, the system <NUM> may include at least one computing device <NUM>, at least one computing device <NUM>, and/or at least one display <NUM>, some or all of which may be communicatively coupled at any given time.

The computing device <NUM> may include at least one processor <NUM> and at least one computer readable medium (e.g., memory <NUM>), some or all of which may be communicatively coupled at any given time. The at least one processor <NUM> may be implemented as any suitable type and number of processors. For example, the at least one processor <NUM> may include at least one general purpose processor (e.g., at least one central processing unit (CPU)), at least one graphics processing unit (GPU), and/or at least one field-programmable gate array (FPGA). The at least one processor <NUM> may be configured to perform (e.g., collectively perform if more than one processor) any or all of the operations disclosed throughout. For example, the at least one processor <NUM> may be configured to train a neural network (e.g., a cGAN) and provide weights and biases to the computing device <NUM>.

The computing device <NUM> may include at least one processor <NUM> and at least one computer readable medium (e.g., memory <NUM>), some or all of which may be communicatively coupled at any given time. The at least one processor <NUM> may be implemented as any suitable type and number of processors. For example, the at least one processor <NUM> may include at least one general purpose processor (e.g., at least one CPU), at least one GPU, and/or at least one FPGA. The at least one processor <NUM> may be implemented as a neural network (e.g., a cGAN). The at least one processor <NUM> may be configured to perform (e.g., collectively perform if more than one processor) any or all of the operations disclosed throughout. For example, the at least one processor <NUM> may be configured to: receive image data associated with a location; receive geo-specific data associated with the location; use the image data and the geo-specific data to generate a top-down photo-realistic image, wherein the photo-realistic image contains photo-typical content in geographically correct locations, wherein the photo-typical content appears typical for the location based at least on the geo-specific data; and output the photo-realistic image to the display <NUM> for presentation to a user. The neural network may use multiple channels of input data including at least one channel for the image data (e.g., three channels for each of red, green, and blue and/or one channel for grayscale) and at least one channel for the geo-specific data comprising at least ecological data pertaining to land type (e.g. such a desert, mountain, forest, etc.), and near-infrared data to identify foliage. For example, the geo-specific data may additionally include at least one of: geographical information data (e.g. pertaining to roads, waterways, buildings, etc.), elevation data (e.g., pertaining to terrain slope derived from elevation grid posts), land-use data (e.g., how does the land get used, such as industry, urban, farm, etc.), and/or target photo pixel resolution data.

Referring now to <FIG>, an exemplary diagram of the at least one processor <NUM> implemented as a neural network (e.g., a cGAN) of an exemplary embodiment according to the inventive concepts disclosed herein is depicted. The neural network may include and/or utilize geo-specific data <NUM> and image data, at least one generator <NUM>, at least one output <NUM>, at least one discriminator <NUM>, at least one comparator <NUM>, at least one comparator <NUM>, at least one summer <NUM>, at least one optimizer <NUM>, at least one desired output (e.g., ground truth <NUM>), and/or at least one update generator <NUM> for updating weights and biases. For example, the generator <NUM> may use the image data, the geo-specific data, and a set of weights and biases to iteratively generate an output image <NUM>. For example, the discriminator <NUM> may iteratively generate a loss function to measure how good the generator's output image <NUM> is, and the discriminator <NUM> may iteratively update the set of weights and biases until the discriminator <NUM> determines the output image to be real, wherein the output image <NUM> determined to be real is the photo-realistic image.

Still referring to <FIG>, the generator <NUM> (e.g., a generator network) may take the input and a set of weights and bias to generate an output <NUM>. The discriminator <NUM> (e.g., discriminator network) may be shown the input and the desired output (e.g., ground truth <NUM>). Therefore, the discriminator <NUM> may generate a loss function to measure how good the generator's <NUM> output <NUM> is. The generator <NUM> may iterate with different weights and bias to generate different outputs <NUM>. These outputs <NUM> may be measured by the discriminator <NUM> so that the generator <NUM> can save the weights and bias that generated better results. This iteration can continue until the generator <NUM> is producing an output <NUM> that the discriminator <NUM> thinks (e.g., determines) is real. Typically the input is in the form of color pictures. The color pictures contain three color values per pixel (red green blue (RGB)) or three channels of data. If processing a grayscale picture, there would only be a need for one channel. The neural network may be effectively modified to accept more than <NUM> channels of data so that data can be used to describe the location on the Earth with enough information to generate a typical representation. The data may contain information about road placement, elevation, ecological information, etc. This data may be paired with actual satellite photos so the cGAN can learn how to render photo-realistic typical content for any location on the Earth that the cGAN is trained for. Data is available for the whole Earth. Some embodiments may use this cGAN deep learning technology for producing typical top-down photo-realistic images that can be used in lieu of actual photographic images or other theme-based solutions. Some embodiments provide a way to generate photo-typical content that may contain content (e.g., roads, buildings, vegetation, etc.) in geographically correct locations but be rendered with typical looking content. Some embodiments may be configured to generate photo-realistic terrain for any location on Earth.

<FIG> shows exemplary training data and generated output (e.g., photo-realistic images (e.g., top-down photo-realistic images)) of some embodiments according to the inventive concepts disclosed herein.

Referring now to <FIG>, an exemplary embodiment of a method <NUM> according to the inventive concepts disclosed herein may include one or more of the following steps. Additionally, for example, some embodiments may include performing one more instances of the method <NUM> iteratively, concurrently, and/or sequentially. Additionally, for example, at least some of the steps of the method <NUM> may be performed in parallel and/or concurrently. Additionally, in some embodiments, at least some of the steps of the method <NUM> may be performed non-sequentially.

A step <NUM> may include receiving, by at least one processor, image data associated with a location, wherein the at least one processor is implemented as a neural network, wherein the at least one processor is communicatively coupled to a display.

A step <NUM> may include receiving, by the at least one processor, geo-specific data associated with the location.

A step <NUM> may include using, by the at least one processor, the image data and the geo-specific data to generate a photo-realistic image, wherein the photo-realistic image contains photo-typical content in geographically correct locations, wherein the photo-typical content appears typical for the location based at least on the geo-specific data.

A step <NUM> may include outputting, by the at least one processor, the photo-realistic image to the display for presentation to a user.

As will be appreciated from the above, embodiments of the inventive concepts disclosed herein may be directed to a method and a system configured to use image data and geo-specific data to generate a photo-realistic image.

As used throughout and as would be appreciated by those skilled in the art, "at least one non-transitory computer-readable medium" may refer to at least one non-transitory computer-readable medium (e.g., at least one computer-readable medium implemented as hardware; e.g., at least one non-transitory processor-readable medium, at least one memory (e.g., at least one nonvolatile memory, at least one volatile memory, or a combination thereof; e.g., at least one random-access memory, at least one flash memory, at least one read-only memory (ROM) (e.g., at least one electrically erasable programmable read-only memory (EEPROM)), at least one on-processor memory (e.g., at least one on-processor cache, at least one on-processor buffer, at least one on-processor flash memory, at least one on-processor EEPROM, or a combination thereof), or a combination thereof), at least one storage device (e.g., at least one hard-disk drive, at least one tape drive, at least one solid-state drive, at least one flash drive, at least one readable and/or writable disk of at least one optical drive configured to read from and/or write to the at least one readable and/or writable disk, or a combination thereof), or a combination thereof).

Claim 1:
An aircraft system or a simulator system (<NUM>), comprising:
a display (<NUM>); and
at least one processor (<NUM>) communicatively coupled to the display, wherein the at least one processor is implemented as a neural network, wherein the at least one processor is configured to:
receive (<NUM>) image data of different land types associated with a location;
receive (<NUM>) geo-specific data associated with the location, wherein the geo-specific data comprises: ecological data pertaining to land type, and near-infrared data to identify foliage;
use (<NUM>) the image data and the geo-specific data as input to the neural network to generate as output of the neural network a photo-realistic image of the location, wherein the photo-realistic image contains photo-typical content in geographically correct locations, wherein the photo-typical content appears typical for the location based at least on the geo-specific data, wherein the photo-realistic image is a top-down photo-realistic image; and
output (<NUM>) the photo-realistic image to the display for presentation to a user.