Radar image data processing and manipulation

Various embodiments of the present technology generally relate to detecting and manipulating radar image data. More specifically, some embodiments relate to systems, methods, and computer-readable storage media for detecting, processing, viewing, and manipulating radar images in an image viewer application. Radar image data captured by a radar imaging system, such as a synthetic aperture radar (SAR) or other satellite-based equipment, comprises data unreadable by image viewers. In an implementation, an open-source plug-in for an image viewer application obtains SAR data, performs one or more algorithms on the SAR data to detect an image, and provides the detected image to an image viewer for display on a graphical user interface. Further, requests for manipulation of the detected image made by the image viewer application may be performed by the plug-in and exported in complex data formats for use downstream.

TECHNICAL FIELD

Various embodiments of the present technology generally relate to rendering and manipulating radar-based image data. More specifically, some embodiments relate to system, methods, and devices for detecting, processing, viewing, and manipulating radar images in an image viewer application.

BACKGROUND

Radar imaging systems, such as satellites equipped with a synthetic aperture radar (SAR) or other radar-based imaging equipment, have the ability to capture radar data by transmitting signals to designated targets and receiving pulse data from echoes off of the targets. Pulse data collected by radar-based equipment comprises complex raw data, made up of samples with in-phase and quadrature parts, requiring tools to process the raw data in order for image viewers to display captured images in a graphical user interface. Processing the raw data requires the use of multiple algorithms that convert the raw data into sensor independent complex data (SICD), which may then be detected by certain image viewers. SICD data comprises metadata captured by the satellite-based equipment which allows image manipulation programs to perform unique calculations on the images. However, only few complex, expensive applications can render SICD data and manipulate it, as detected, to be used downstream, making radar image data inaccessible without these applications.

Image manipulation is a process of altering image data into a desired form or format. In the radar imaging context, image manipulation is a process of reconstructing and/or reformatting radar image data into visual data to view and edit the captured images, among other features. Image viewers typically limit the types of input or output data, thus restricting input of raw data or SICD data. Therefore, users must rely on time-consuming algorithmic processes to convert raw data before using any image viewers, and even then, functionality may be limited in the program. For example, radar image data of a field may be captured in multiple exposures or data files requiring complex processing of each file to generate images. The images may then be viewed in a program, such as a raster graphics editor, but may only be modified in certain ways, such as cropping or scaling, in accordance with the program's functionality. Additionally, the program may only allow exportation of a standard image format, thus making it difficult for downstream users to view previous manipulations of a file's metadata or make further changes themselves.

It is with respect to this general technical environment that aspects of the present technology disclosed herein have been contemplated. Furthermore, although a general environment has been discussed, it should be understood that the examples described herein should not be limited to the general environment identified in the background

Overview

Various embodiments of the present technology herein generally relate to systems, methods, and computer readable media for processing, detecting, and manipulating radar image data captured by a radar imaging system, such as one or more satellites or satellite equipment. A computing apparatus, in various implementations, provides for streamlined processing of radar image data to be viewed and modified in an image viewer application. In an implementation, the computing apparatus comprises one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media that, when executed by a processing system, direct the processing system to obtain synthetic aperture radar data produced by a radar imaging system, generate detected image data based on the synthetic aperture radar data, and communicate, via an application programming interface to an image viewer application, the detected image data to be displayed as an image by the image viewer application.

While multiple embodiments are disclosed, still other embodiments of the present technology will become apparent to those skilled in the art from the following detailed description, which shows and describes illustrative embodiments of the invention. As will be realized, the technology is capable of modifications in various aspects, all without departing from the scope of the present invention. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not restrictive.

DETAILED DESCRIPTION

Various embodiments of the present technology generally relate to systems, methods, and computer-readable media for detecting, processing, viewing, and manipulating radar images in an image viewer application. Radar data captured by a radar imaging system, like synthetic aperture radar (SAR) technology equipped to a satellite, for example, comprises raw data ordinarily not viewable by image viewers. Raw data may include both in-phase and quadrature parts, thus, samples of images captured cannot be detected or viewed. Raw data, in an unprocessed state, is uncompensated pulse data rather than pixels that form an image and has not undergone motion compensation or dechirping. Therefore, in order to view targets captured by a SAR, raw data must be converted using various algorithms and detected by an image viewer, such as a raster graphics editor. Using a radar plug-in to an image viewer application, SAR data can be converted from one of multiple formats to a detected image format whereby a user can view one or more images captured by the satellite and manipulate them. Moreover, the radar plug-in can write not only the detected image, but also it can write radar images in complex data forms. The plug-in can then export the manipulated radar data in a variety of formats, including complex formats, for further viewing and manipulating downstream.

The radar plug-in tool recognizes SAR data in its raw form, compensated phase history data (CPHD) form, sensor independent complex data (SICD) form, and sensor independent detected data (SIDD) form, among others. Once the plug-in obtains the data and recognizes its form, it performs a series of algorithms on the SAR data to surface a rendering on a graphical user interface to an image viewer. The plug-in communicates with the image viewer over an application programming interface (API), or by some other data communication method. In some embodiments, the plug-in applies algorithms, fast-Fourier transform (FFT) calculations, polar formatting, and other image detection calculations. For example, the radar plug-in tool may perform a two-dimensional (2D) FFT on a SICD form to process the complex data from the image domain to the frequency domain. In the frequency domain, functions such as scaling the image or extracting a time series from the data can be performed.

In various embodiments of the present technology, a computing apparatus may provide for processing SAR data into a detected image for display and manipulation to be displayed in an image viewer application. In an implementation, the computing apparatus comprises one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media that, when executed by a processing system, direct the processing system to obtain synthetic aperture radar data produced by a radar imaging system, generate detected image data based on the synthetic aperture radar data, and communicate, via an application programming interface to an image viewer application, the detected image data to be displayed as an image by the image viewer application. Using the aforementioned example, manipulations, such as image projection calculations, may be performed and stored in the SICD form's metadata. Upon opening the SICD file at a later time using the plug-in to an image viewer, annotations made to the metadata may be viewed and further manipulated by downstream users.

Other embodiments refer to methods for detecting radar image data in a plug-in and transmitting a detected image to an image viewer for display. Such methods comprise, in a plug-in tool to an image viewer application, obtaining synthetic aperture radar data produced by a radar imaging system, generating detected image data based on the synthetic aperture radar data, communicating, via an application programming interface to the image viewer application, the detected image data to be displayed as an image by the image viewer application, and in the image viewer application, displaying the detected image data on a graphical user interface.

Now referring to the Figures,FIG.1illustrates an exemplary operating environment in accordance with some embodiments of the present technology. Figure includes environment100which further includes SAR data105, radar image plug-in110, image viewer115, graphical user interface116, toolbar117, and images120and121.

In various embodiments, SAR data105comprises complex radar data acquired by the use of a synthetic aperture radar or other satellite-based radar equipment. SAR data105, without being processed, cannot ordinarily be detected and viewed in an image viewer due to its complexity and lack of real pixel values. SAR data105may be accessed via a data repository to import data into radar image plug-in110. Based on the format of data imported to radar image plug-in110, the plug-in runs one or more processing algorithms on SAR data105to detect an image from the data. If, for example, SICD data is imported to radar image plug-in110, the plug-in must only detect the image without requiring additional processing.

In step1, image viewer115sends a request for an image to radar image plug-in110. Radar image plug-in110and image viewer115communicate over an application programming interface (API). In some embodiments, the API is a Python API. In other embodiments, the communication link may be established by another call, API, or secure file transfer protocol. Step2illustrates radar image plug-in sending detected image data to the image viewer115. As shown by step3, image viewer115displays the detected image on graphical user interface116. On graphical user interface116, image modification functions native to image viewer115are selectable on toolbar117, which allows a user to manipulate the image in limited ways. Toolbar117demonstrates several features such as cropping, resizing, and annotating. If a user wishes to make radar-based manipulations to the image, image viewer115may request radar image plug-in110to make modifications. In other embodiments, radar image plug-in110makes modifications without any request from the image viewer.

Radar image plug-in110comprises additional image modification functions such as time slicing, change detection, FFT calculations, image overlaying, registration, image blurring, remapping, and the like. These functions take advantage of the complex samples and metadata present in SAR data105that provides insight into the images, such as geoposition and timing. Radar image plug-in110can annotate and modify the image at the metadata level to reflect such changes. As an example, a satellite equipped with a SAR may capture a city landscape at one point in time, then capture the same target at a later point to understand the city's growth. Radar image plug-in110can use a registration function to align the two images at the sub-pixel level to determine even miniscule changes.

After radar image plug-in110makes modifications to the detected image based on the request from image viewer115, image viewer115displays an updated image on graphical user interface116. Image120illustrates a detected image before a modification request is made by image viewer115. In image120, captured structures are blurry and misaligned. Subsequently, image121illustrates an updated detected image after a modification request is completed by radar image plug-in110. As shown by image121, the structures are aligned at the pixel level and a sensitive target, namely the tower shown in image120, is blurred. Radar image plug-in110may save image121in a variety of formats for future use, including, but not limited to, SICD and SIDD data. The ability to save updated images in SICD format provides an advantage to downstream users as they will be able to see any annotations or modifications made to the SICD data and process the data through downstream Full Spectral GEOINT algorithms or Automated Detection algorithms. Unlike standard image formats, SICD and SIDD data comprise complex metadata. Thus, downstream users can continue to modify such metadata even after initial modifications are completed by radar image plug-in110without the use of complex, expensive image processors.

FIG.2illustrates an exemplary processing environment200in which some embodiments of the present technology may be utilized.FIG.2demonstrates inputting one of multiple formats of radar data into a radar image plug-in to detect images.FIG.2includes environment200which further includes SAR data105, raw data205, compensated phase history data210, sensor independent complex data215, sensor independent detected data220, radar image plug-in110, and image viewer115.

In some embodiments, SAR data105comprises raw data205captured by satellite-based equipment, such as a synthetic aperture radar (SAR) that is undetectable by image viewers without any pre-processing to the data. Raw data205comprises uncompensated received pulse data rather than pixels that form an image. Used as an input to radar image plug-in110, the radar image plug-in110performs algorithms and image processing to raw data205, so that it can detect an image in the data.

After some pre-processing, SAR data105comprises radar data formatted as compensated phase history data210(CPHD). CPHD210, like raw data205, is also ordinarily not viewable in image viewers because it represents complex pulse data. In various embodiments, raw data205undergoes one or more image processing algorithms to obtain CPHD210, which also may be input into radar image plug-in110. In order for radar image plug-in110to view an image from data in CPHD210format, processing such as fast-Fourier transforms (FFTs) and polar formatting are performed on the data so that the image can be detected and viewed.

Next, by performing such further image processing algorithms on CPHD210, like FFTs and polar formatting, radar image plug-in110reconstructs the image data into sensor independent complex data215(SICD). Radar data in SICD215form comprises polar formatted image data that must further be detected by radar image plug-in110to be communicated to image viewer115. One step further, radar plug-in110can detect, or convert from complex data to viewable data, SICD215to sensor independent detected data220(SIDD). SIDD220comprises a fully detected image that can be read by radar image plug-in110and displayed by image viewer115.

In various embodiments, radar image plug-in110can automatically read, process, and detect any of raw data205, CPHD210, SICD215, or SIDD220. Thus, SAR data105in various forms and stages of processing may be input to view fully detected and formatted images captured by a radar imaging system, such as satellite-based imaging equipment. Image viewer115serves as a graphical user interface to display detected images. Radar image plug-in110may first receive a request from image viewer115to detect an image in a file. After detecting the image, radar image plug-in110communicates or transmits the data to image viewer115over an application programming interface (API). While image viewer115may have native functionality to modify images, requests for further modifications may be made to radar image plug-in110, such as registration, phase analysis, and the like.

Both SICD215and SIDD220comprise pixel arrays and metadata that describe the data product. Metadata plays an important role in radar-based image viewers, such as image viewer115, as it allows for manipulation of pixels, calculations at the pixel level, phase and frequency analysis, and similar modifications that become stored in the metadata for future use. In accordance with the present technology, radar image plug-in110makes modifications to the image based on a request by image viewer115. Such modifications may affect data at the metadata level and the pixel level. The modifications are stored as part of SICD215or SIDD220, and thus, the modifications can be viewed later upon opening and detecting the image using radar plug-in110.

FIG.3illustrates a sequence diagram of data flow between of a radar image plug-in tool and an image viewer in accordance with some embodiments of the present technology.FIG.3includes environment300that functions to process synthetic aperture radar data into detectable image data, manipulate the image data according to requirements, and export the manipulated image data downstream. Environment300further includes SAR data305, radar image plug-in310, and image viewer315.

In various embodiments, SAR data305comprises one or more images captured from satellite-based imaging equipment, such as a SAR. In some embodiments, the SAR provides data in a raw data format unviewable by computer programs, applications, and the like due to the data being in uncompensated received pulse data rather than in pixel value. In other embodiments, the system may provide radar image data in CPHD, SICD, or SIDD formats.

In one example, raw radar data is imported to radar image plug-in310. Radar image plug-in310recognizes the format of the data received by radar image data305and performs one or more data manipulation algorithms to render the raw image data detectable or viewable. Image viewer315sends a request to radar image plug-in310for the image. In response to the request, radar image plug-in310sends a detected image to image viewer315, which allows for viewing and basic modifications to the image. Modifications that may be native to image viewer315include cropping, scaling, recoloring, and the like. With limited capabilities to make modifications to radar-based images, image viewer315requests radar image plug-in310to manipulate the image. Radar image plug-in310may allow for further manipulation functions including, but not limited to, image projection, remapping, overlaying, fast-Fourier transformation (FFT) calculations, dynamic imaging, pixel modification, phase analysis, non-linear warping, and registration. Modifications like registration allow for a user to align multiple images at a sub-pixel level. Registration may be completed by one or more algorithms, such as color multi-view and/or coherent change detection algorithms, operated by radar image plug-in310.

In the next step, radar image plug-in310sends a manipulated detected image back to image viewer315based on the request for manipulation. Image viewer315may display the modified image on the graphical user interface where further modifications may be made or requested. Once modifications are complete, image viewer315may save the updated image. Radar image plug-in310may then export a radar image file to be used downstream. An image viewer like a raster graphics editor can export image files in several types of formats, including but not limited to, XCF, PNG, JPG, GIF, TIFF, and DNG among others. Radar image plug-in310extends the selection of image formats to radar-based image formats such as SIDD and SICD. Despite an image viewer's inability to view images in complex formats, radar image plug-in310allows for processing and detecting image data without the use of third-party processing tools, enabling image viewers to access SICD and SIDD files by utilizing the plug-in. To be used downstream, SICD data must undergo an inverse algorithm to convert the detected image product back to a complex form.

Next,FIG.4is a flowchart illustrating a set of operations for radar image data reconstruction in accordance with some embodiments of the present technology. In radar image process400, illustrated inFIG.4, a radar plug-in for an image viewer obtains a request for radar image data and processes it for viewing and editing on a graphical user interface. Radar image process400may be implemented in program instructions in the context of any of the software applications, modules, components, or other programming elements deployed in one or more servers. The program instructions direct the underlying physical or virtual computing systems to operate as follows, referring to steps illustrated inFIG.4, such as data retrieval step405, image detection step410, and image display step415.

Beginning with data retrieval step405, an image viewer plug-in obtains synthetic aperture radar data produce by a radar imaging system, such as a SAR. The plug-in tool identifies the image associated with the request and must process SAR data using one or more image processing algorithms if the image is in a format such as raw data, CPHD, or SICD. In image detection step410, the plug-in generates detected image data based on the SAR data. Detecting the SAR data refers to converting the complex SAR data into viewable data. The plug-in then communicates, over an API to an image viewer application, the detected image data to be displayed as an image by the image viewer on a graphical user interface.

Moving toFIG.5,FIG.5demonstrates a flowchart illustrating a set of operations for manipulating radar-based images in accordance with some embodiments of the present technology. In image manipulation process500, illustrated inFIG.5, a plug-in tool receives a request to manipulate an image and send the updated image to an image viewer. Image manipulation process500may be implemented in program instructions in the context of any of the software applications, modules, components, or other programming elements deployed in one or more servers. The program instructions direct the underlying physical or virtual computing systems to operate as follows, referring to steps illustrated inFIG.5, such as manipulation request step505, update step510, and display step515.

First, at manipulation request step505, an image viewer sends a request over an API to a plug-in tool to make one or more modifications to an image. While the image viewer itself may have native manipulation features, such as cropping, resizing, recoloring, measuring, dragging, and/or zooming in and out, for example, it lacks other functionality helpful for use on radar-based images. In various embodiments, additional features supplied by the plug-in tool comprise remapping, multiple image overlaying, modifying pixels, FFT calculations, image projection, phase analysis, and image registration. As an example, by performing FFT calculations on an image captured by a SAR, the plug-in tool can extract a time slice of the image to identify moving targets. By identifying changes of detected image data in metadata, the radar plug-in can apply the same operations to complex data and reprocess the complex data back to detected image data before rendering and displaying the image as detected. This allows the radar plug-in to perform manipulations to be viewed and further changed at later points in time.

Next, in update step510, the plug-in tool generates a new version of detected image data based on the one or more manipulations made to the image. This occurs so that the changes are reflected in the image for viewing or manipulating downstream. Finally, at display step515, the radar plug-in communicates or transmits, via an API, the new version of detected image data to be displayed by the image viewer. The image viewer receives all modifications and displays them on a graphical user interface. This process may be repeated until a user is satisfied with the modifications made to the image.

At import step505, the radar image plug-in obtains a radar image file. In various embodiments, radar image file comprises raw, undetected data not suitable for display on an image viewer or GIMP, as an example. At reconstruction step510, the radar plug-in performs a first image reconstruction algorithm using the raw radar image file data to create compensated phase history radar image data (CPHD). From here, at transformation step515, the radar plug-in performs algorithms and calculations, like a two-dimensional fast-Fourier transform among other calculations, using the CPHD data to create complex radar image data. The radar plug-in may also polar format the CPHD data to create a full SICD with better image quality than a 2D FFT complex product. Finally, at display step520, the radar plug-in detects the image, reads the data, and surfaces the image in a graphical user interface. At this stage, the radar image data is fully detectable by an image viewer, such as GIMP, and can be read as sensor independent detected data.

FIG.6illustrates an exemplary operating environment600in which some embodiments of the present technology may be utilized. First,FIG.6illustrates a satellite equipped with a synthetic aperture radar (SAR) to transmit and receive radar data from a target on Earth. Once radar data is captured by the satellite, the data is transferred to a computing system to be accessed.FIG.6includes environment600, wherein environment600further includes satellite601, RF pulse605, image target610, echo615, radar station620, server625, first computer630, second computer635, and application server640.

In various embodiments, satellite601transmits, via a SAR, a radio frequency or RF pulse605while flying in orbit over image target610. Satellite601initiates one or more of RF pulse605, and in response, the pulses create an echo615that bounces off image target610. In many embodiments, image target610may be a set of buildings, structures, landscapes, and the like with varying topologies. Echo615provides information from image target610, such as the height and width of image target610, for example. Echo615transmits back up toward satellite601from various perspectives and locations due to RF pulse605being transmitted over a period of time.

Also depicted in environment600is satellite601operating at a different location and point in time as it continues its orbit. After capturing data via echo615, satellite601downlinks SAR data to radar station620. The data downlink may operate in various frequency ranges, such as X-band, S-band, or the like. Once radar station620collects SAR data from satellite601, radar station620may transmit the radar data to one or more databases or servers, such as server625. In some examples, server625may obtain SAR data over a communication link, such as a secure file transfer protocol, an application programming interface, or the like. Server625may operate in a cloud environment, remote from other computing devices, or it can operate locally.

In various embodiments, the data received by server625comprises raw SAR data comprising pulse data undetectable by image viewers without any processing. First computer630accesses server625to obtain raw SAR data. In an optional step, first computer630accesses application server640to download a radar image plug-in to an image viewer. Alternatively, in other embodiments, first computer630already has a plug-in installed. After obtaining SAR data from server625, radar plug-in operating on first computer630processes and detects the raw data to form a detected image. Radar plug-in communicates the detected image to the image viewer over an API to be displayed as an image on a graphical user interface. A user of first computer630can then manipulate the detected image and store it locally on first computer630, communicate it back to server625to be obtained by other users, and/or send it to another computer or user, such as second computer635. The image data stored or transmitted elsewhere by first computer630may comprise image data in SICD, SIDD, or another image file format.

Next, like first computer630, second computer635can also access either or both server625and application server640to obtain SAR data and the plug-in, respectively. In some embodiments, second computer635accesses server625to download a SICD image file previously manipulated by first computer630. The SICD image file includes annotations on its metadata indicating changes made by first computer630. Radar plug-in detects the manipulated image in the SICD data and requests the image viewer to display it as an image with the manipulations. Second computer635can make further manipulations and store the manipulated image locally, and/or send the image data to server625or elsewhere downstream.

FIG.7illustrates computing system705that is representative of any system or collection of systems in which the various processes, programs, services, and scenarios disclosed herein may be implemented. This computing system may be implemented in radar image plug-in110ofFIG.1to facilitate radar data collection and image formation. Examples of computing system705include, but are not limited to, desktop computers, laptop computers, server computers, routers, web servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machine, physical or virtual router, container, and any variation or combination thereof.

Computing system705may be implemented as a single apparatus, system, or device or may be implemented in a distributed manner as multiple apparatuses, systems, or devices. Computing system705includes, but is not limited to, storage system710, processing system730, software715, communication interface system725, and user interface system735. Processing system730is operatively coupled with storage system710, communication interface system725, and user interface system735.

Processing system730loads and executes software715from storage system710. Software715includes and implements radar image process720, which is representative of the SAR radar data and image detection processes discussed with respect to the preceding Figures. When executed by processing system730to provide image retrieval and image formation processes, software715directs processing system730to operate as described herein for at least the various processes, operational scenarios, and sequences discussed in the foregoing implementations. Computing system705may optionally include additional devices, features, or functionality not discussed for purposes of brevity.

Referring still toFIG.7, processing system730may include a micro-processor and other circuitry that retrieves and executes software715from storage system710. Processing system730may be implemented within a single processing device but may also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing system730include general purpose central processing units, graphical processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof.

In addition to computer readable storage media, in some implementations storage system710may also include computer readable communication media over which at least some of software715may be communicated internally or externally. Storage system710may be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems co-located or distributed relative to each other. Storage system710may incorporate additional elements, such as a controller, capable of communicating with processing system730or possibly other systems.

Software715(including radar image process720) may be implemented in program instructions and among other functions may, when executed by processing system730, direct processing system730to operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. For example, software715may include program instructions for implementing image retrieval and reconstruction process as described herein.

In general, software715may, when loaded into processing system730and executed, transform a suitable apparatus, system, or device (of which computing system705is representative) overall from a general-purpose computing system into a special-purpose computing system customized to provide image retrieval and reconstruction processes as described herein. Indeed, encoding software715on storage system710may transform the physical structure of storage system710. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the storage media of storage system710and whether the computer-storage media are characterized as primary or secondary storage, as well as other factors.