Patent ID: 12190573

The drawings are not necessarily to scale. The drawings are merely schematic representations, not intended to portray specific parameters of the disclosed embodiments. The drawings are intended to depict only typical embodiments of the invention, and therefore should not be considered as limiting in scope.

DETAILED DESCRIPTION OF THE INVENTION

Commercially available digital cameras are capable of capturing RGB (red-green-blue) images by mapping the spectrum of acquired image data to the red, green, and blue spectral bands, leaving much of the available spectrum ignored. In contrast, hyperspectral images often contain in excess of ten spectral bands. This rich spectral information is beneficial for numerous computer vision functions, such as facial recognition and object tracking. However, direct acquisition of hyperspectral images from spectrometers and/or hyperspectral cameras can be costly and time consuming.

Disclosed embodiments address the aforementioned issues with a novel approach that includes reconstructing hyperspectral images from corresponding RGB images by taking advantage of spectral super-resolution algorithms. Disclosed embodiments utilize multiple neural networks to improve the modeling of the complex mapping relationship between RGB images and their corresponding hyperspectral images. This enables the use of conventional RGB image acquisition devices that are plentiful, fast, and economical, for the data acquisition component of disclosed embodiments. Then, the processing of the conventional RGB image data performed by disclosed embodiments generates an accurate reconstructed hyperspectral image, enabling the efficient use of hyperspectral images in a wide variety of applications.

One or more different aspects may be described in the present application. Further, for one or more of the aspects described herein, numerous alternative arrangements may be described; it should be appreciated that these are presented for illustrative purposes only and are not limiting of the aspects contained herein or the claims presented herein in any way. One or more of the arrangements may be widely applicable to numerous aspects, as may be readily apparent from the disclosure. In general, arrangements are described in sufficient detail to enable those skilled in the art to practice one or more of the aspects, and it should be appreciated that other arrangements may be utilized and that structural, logical, software, electrical and other changes may be made without departing from the scope of the particular aspects. Particular features of one or more of the aspects described herein may be described with reference to one or more particular aspects or figures that form a part of the present disclosure, and in which are shown, by way of illustration, specific arrangements of one or more of the aspects. It should be appreciated, however, that such features are not limited to usage in the one or more particular aspects or figures with reference to which they are described. The present disclosure is neither a literal description of all arrangements of one or more of the aspects nor a listing of features of one or more of the aspects that must be present in all arrangements.

Headings of sections provided in this patent application and the title of this patent application are for convenience only, and are not to be taken as limiting the disclosure in any way.

Devices that are in communication with each other need not be in continuous communication with each other, unless expressly specified otherwise. In addition, devices that are in communication with each other may communicate directly or indirectly through one or more communication means or intermediaries, logical or physical.

A description of an aspect with several components in communication with each other does not imply that all such components are required. To the contrary, a variety of optional components may be described to illustrate a wide variety of possible aspects and in order to more fully illustrate one or more aspects. Similarly, although process steps, method steps, algorithms or the like may be described in a sequential order, such processes, methods and algorithms may generally be configured to work in alternate orders, unless specifically stated to the contrary. In other words, any sequence or order of steps that may be described in this patent application does not, in and of itself, indicate a requirement that the steps be performed in that order. The steps of described processes may be performed in any order practical. Further, some steps may be performed simultaneously despite being described or implied as occurring non-simultaneously (e.g., because one step is described after the other step). Moreover, the illustration of a process by its depiction in a drawing does not imply that the illustrated process is exclusive of other variations and modifications thereto, does not imply that the illustrated process or any of its steps are necessary to one or more of the aspects, and does not imply that the illustrated process is preferred. Also, steps are generally described once per aspect, but this does not mean they must occur once, or that they may only occur once each time a process, method, or algorithm is carried out or executed. Some steps may be omitted in some aspects or some occurrences, or some steps may be executed more than once in a given aspect or occurrence.

When a single device or article is described herein, it will be readily apparent that more than one device or article may be used in place of a single device or article. Similarly, where more than one device or article is described herein, it will be readily apparent that a single device or article may be used in place of the more than one device or article.

The functionality or the features of a device may be alternatively embodied by one or more other devices that are not explicitly described as having such functionality or features. Thus, other aspects need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimes be described in singular form for clarity. However, it should be appreciated that particular aspects may include multiple iterations of a technique or multiple instantiations of a mechanism unless noted otherwise. Process descriptions or blocks in figures should be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of various aspects in which, for example, functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those having ordinary skill in the art.

Definitions

The term “bit” refers to the smallest unit of information that can be stored or transmitted. It is in the form of a binary digit (either 0 or 1). In terms of hardware, the bit is represented as an electrical signal that is either off (representing 0) or on (representing 1).

The term “pixel” refers to the smallest controllable element of a digital image. It is a single point in a raster image, which is a grid of individual pixels that together form an image. Each pixel has its own color and brightness value, and when combined with other pixels, they create the visual representation of an image on a display device such as a computer monitor or a smartphone screen.

The term “neural network” refers to a computer system modeled after the network of neurons found in a human brain. The neural network is composed of interconnected nodes, called artificial neurons or units, that work together to process complex information.

The term “hyperspectral image” refers to an image in which each pixel of the image includes multiple (generally more than three) spectral bands from across the electromagnetic (EM) spectrum.

Conceptual Architecture

FIG.1is a block diagram illustrating components for hyperspectral image generation utilizing a decomposition network and a fine-tuning network, according to an embodiment. An input hyperspectral image104and corresponding input RGB image102are used as training data for decomposition network106. The input RGB image102is an RGB version of the hyperspectral image104. In one or more embodiments, the input RGB image102may be in a Bayer format. A Bayer raw image is a type of image format that may be used in digital cameras and other imaging devices. Images in the Bayer format may comprise multiple sets of four pixels. Each set includes a red pixel, a blue pixel, and two green pixels. This arrangement is based on the fact that the human eye is more sensitive to green light than to red or blue. One or more embodiments may utilize other formats for the input RGB image. In one or more embodiments, the input RGB image102may include bitmaps, tagged image file format (TIFF), and/or other raw formats.

The input hyperspectral image104can include multiple spectral bands. In embodiments, the input hyperspectral image can include between 10 to 32 spectral bands. Other embodiments may include more or fewer spectral bands. In one or more embodiments, the input hyperspectral image comprises 31 spectral bands ranging from 400 nm to 700 nm with a 10 nm interval.

The input hyperspectral image104is input to spectral band grouping module108. Spectral band grouping module108can include instructions and/or functions, that when executed by a processer, perform functions including computing a correlation coefficient of each spectral band of the plurality of spectral bands to at least one other spectral band of the plurality of spectral bands; and forming a plurality of spectral domain groups based on the computed correlation coefficients.

One or more embodiments can enable reconstructing a hyperspectral image denoted as:
Y∈Rw×h×L
from its corresponding RGB image which is denoted as:
X∈Rw×h×3

Where L represents the number of spectral bands in the hyperspectral image, where L is greater than three, and w and h denote the width and height of the two images, respectively. In one or more embodiments, for any two bands in the hyperspectral image, the bands are vectorized to create two vectors. Then, a correlation coefficient for the two vectors is computed. The correlation coefficient is a measure that quantifies the degree to which two sets of data are related or how they vary together. For each spectral band, there is a corresponding neural network in the decomposition network106. As shown inFIG.1, there are two neural networks120and130. However, in practice, there are L neural networks, where L represents the number of spectral bands in the hyperspectral image. Neural network120includes convolutional block121, residual block122, residual block123, and convolutional block124, which may be interconnected as shown inFIG.1. Similarly, neural network130includes convolutional block131, residual block132, residual block133, and convolutional block134, which may be interconnected as shown inFIG.1. For each spectral band, there is a corresponding loss function for the decomposition network106, represented as Lde, indicated at146and147. Once the decomposition network106is initially trained with input hyperspectral images, the corresponding input RGB image is input into decomposition network106. The output of the decomposition network106is the reconstructed hyperspectral image138. The reconstructed hyperspectral image138is then input to a second neural network, which is fine-tuning network140. Fine tuning network140includes convolutional block141, residual block142, residual block143, and convolutional block144, which may be interconnected as shown inFIG.1. The output of the fine-tuning network140is reconstructed RGB image152. The reconstructed RGB image152is compared with the input RGB image102. Differences between the reconstructed RGB image152and the input RGB image102are determined, and are embodied in a corresponding loss function for the fine-tuning network140, represented as Lft, indicated at154. In one or more embodiments, the second neural network (fine-tuning network140) comprises a self-supervised network

FIG.2is a diagram indicating additional details of the neural network architecture shown inFIG.1, according to an embodiment. In particular,FIG.2shows additional details of a residual block such as shown at122inFIG.1. The residual block includes a convolutional block202. The convolutional block can include one or more convolutional layers. In embodiments, each convolutional layer/block includes a set of learnable filters (also known as kernels) that are applied to the input data. In one or more embodiments, for one or more convolutional layers, a corresponding kernel size for the convolutional layer is set to 3×3. Each kernel/filter is convolved with the input data to produce a feature map, which highlights the presence of particular patterns or features in the input. The convolution operation involves sliding the filter over the input data, performing element-wise multiplication and summing the results to produce a single value in the output feature map. In one or more embodiments, the first neural network (decomposition network106) further comprises an activation function. The output of convolutional block202is fed to activation function204. In one or more embodiments, the activation function204includes a non-linear activation function. In one or more embodiments, the activation function204includes a ReLU (Rectified Linear Unit). In one or more embodiments, the activation function204includes a Leaky ReLU (Rectified Linear Unit). The Leaky ReLU (Rectified Linear Unit) is a type of activation function used in artificial neural networks. It is similar to the standard ReLU function but allows a small, non-zero gradient when the input is negative, instead of setting the gradient to zero. In one or more embodiments, the Leaky ReLU activation function is defined as follows:

f⁡(x)={x,if⁢x>0ax,otherwise

Where α is a small constant, such as 0.01, that determines the slope of the function for negative inputs. This can serve to reduce the probability of developing inactive neurons during training and/or operational use of the neural network.

The output of the activation function204can be input to another convolutional block206. The output of convolutional block206can be fed to an additional activation function208. In one or more embodiments, the activation function208can include a sigmoid function. The sigmoid function can be used to introduce non-linearity into the network. In one or more embodiments, the sigmoid function is defined as:

f⁡(x)=11+e-x

Where e is the base of the natural logarithm. The sigmoid function has a characteristic S-shaped curve that maps any real value to a value between 0 and 1. This property makes it suitable for a wide variety of machine learning applications. In one or more embodiments, the activation function208can include a ReLU function instead of, or in addition to, the sigmoid function. Other embodiments can include a Tanh (hyperbolic tangent) activation function, softmax activation function, swish activation function, and/or other suitable activation function. In one or more embodiments, residual blocks can comprise at least two convolutional layers. In one or more embodiments, a first convolutional layer from the at least two convolutional layers is configured to perform feature extraction. In one or more embodiments, a second convolutional layer from the at least two convolutional layers is configured to perform feature map dimension reduction.

FIG.3is a diagram of a dual branch attention network, according to an embodiment. In one or more embodiments, the decomposition network (106ofFIG.1), and/or the fine-tuning network (140ofFIG.1) may be implemented using a dual branch attention network instead of, or in addition to, the neural networks shown inFIG.1. The dual branch attention network shown inFIG.3enables one or more embodiments to extract spectral and spatial features simultaneously from hyperspectral images. In one or more embodiments, small cubes are first cropped from the hyperspectral image and then fed into the dual branch attention network ofFIG.3, in order to extract features. An RGB image302is input to a first neural network301, and also simultaneously input to a second neural network331. The first neural network301is for processing spectral information, and includes convolutional block304, spectral attention module314, convolutional block306, spectral attention module316, convolutional block308, spectral attention module318, and output layer320, which may be interconnected as shown inFIG.3. The second neural network331is for processing spatial information, and includes convolutional block334, spatial attention module344, convolutional block336, spatial attention module346, convolutional block338, spatial attention module348, and output layer350, which may be interconnected as shown inFIG.3.

The attention modules shown inFIG.3can include a neural network architecture that focuses on learning to selectively pay attention to certain parts of the input RGB data. The attention modules of disclosed embodiments can dynamically weight the importance of different input elements such as spectral band information for a given pixel. This enables disclosed embodiments to identify relevant information while filtering out noise or irrelevant details, improving its performance on the task of generating a hyperspectral image from an input RGB image. The output from output layer320and output layer350are combined with respective weighting parameters A and B to generate a hyperspectral image360that is based on input RGB image302.

In one or more embodiments, a first training phase can include pretraining the first neural network301, and the second neural network331independently. A second training phase can then include providing a weighted summation layer and fine-tuning the entire network.

FIG.4is a diagram indicating details of a spectral attention module400, according to an embodiment. Input data402can include hyperspectral image information of multiple channels. The input data402may be expressed as a feature map F of the form:
FL∈G(C,H,W)
where C represents the channel number, H represents an input image height, and W represents an input image width, and where:
L∈{1,2,3}

The hyperspectral image information can be input to pooling layer404, followed by convolutional block406and convolutional block408. The pooling layer serves to reduce spatial dimension. The convolutional block406and convolutional block408can be implemented as 1-D convolutional layers to generate a spectral attention map. In one or more embodiments, a sigmoid function and/or ReLU function may be used as part of the convolutional block406and/or convolutional block408. Elementwise multiplication can be performed by the element indicated at410. The resulting output branch411connects to pooling layer412. In one or more embodiments, pooling layer412can include a max-pooling layer. Pooling layer412can be followed by a fully connected output layer414. The output branch411serves to provide supervised information for the spectral attention module, enabling a discriminative ability of a refined feature map. Moreover, the output branch411can serve to incorporate a regularization term to a loss function, which can help alleviate undesirable overfitting during the network training process.

FIG.5is a diagram indicating details of a spatial attention module500, according to an embodiment. Input data502can include hyperspectral image information of multiple channels. The input data502may be expressed as a feature map F of the form:
FL∈G(C,H,W)
where C represents the channel number, H represents an input image height, and W represents an input image width, and where:
L∈{1,2,3}

The hyperspectral image information can be input to convolutional block504, followed by convolutional block506and convolutional block508. In one or more embodiments, convolutional block504can include a 1×1 convolutional layer to aggregate information along the channel direction of the feature map, resulting in a 2-D feature map. The convolutional block506and convolutional block508can include 2-D convolutional layers to generate a spatial attention map. In one or more embodiments, one or more of the convolutional blocks504,506, and508may also include padding operators. The padding operators can serve to avoid the change of spatial sizes. Elementwise multiplication can be performed by the element indicated at510. The resulting output branch511connects to pooling layer512. In one or more embodiments, pooling layer512can include an adaptive max-pooling layer. In one or more embodiments, the pooling layer512can be followed by output layer514.

Detailed Description of Exemplary Aspects

FIG.6is a flow diagram illustrating an exemplary method600for hyperspectral image generation, according to an embodiment. According to the embodiment, the process begins at step602where a training hyperspectral image is obtained. At step604, spectral bands are identified in the training hyperspectral image. In one or more embodiments, the training hyperspectral image can include 20 bands covering a range from 500 nm to 700 nm, where each band is 10 nm wide. Other embodiments can include more or fewer bands, and cover different ranges. At step606, correlation coefficients amongst pairs of spectral bands are computed. In one or more embodiments, computing correlation coefficients includes vectorizing two spectral bands, and then computing the correlation coefficient of the vector pairs. Embodiments can further include acquiring a correlation matrix with a dimension of L×L, where L is the number of spectral bands in the hyperspectral image used for training. This process can be repeated for all hyperspectral images in the training set, and an averaged correlation matrix can be derived. Based on the averaged correlation matrix, a predetermined grouping threshold can be derived. As an example, in one or more embodiments, a threshold of 0.8 is used as a grouping threshold to determine if spectral bands should be in the same group.

The method600continues to step608where spectral domain groups are formed based on the grouping threshold previously determined. The method600then continues to step610, where the RGB input image is obtained. The RGB input image is part of the training data set, and corresponds to the training hyperspectral image that was obtained at step602. At step612, the RGB input image that was obtained at step610, and the corresponding training hyperspectral image obtained at step602, are input to a first neural network. The first neural network can include a decomposition network, such as shown at106ofFIG.1. The method600continues with obtaining a reconstructed hyperspectral image at step614, such as reconstructed hyperspectral image138as shown inFIG.1.

The method600continues with providing the reconstructed hyperspectral image to a second neural network at step616. In one or more embodiments, the second neural network can include a fine-tuning network, such as shown at140inFIG.1. The method600continues with providing the reconstructed hyperspectral image to the second neural network in step616. The method600then continues with obtaining the reconstructed RGB image from the second network at step618, followed by comparing the reconstructed RGB image to the RGB input image at step620. Based on differences between the reconstructed RGB image and the RGB input image, one or more weights corresponding to the first neural network and/or second neural network are adjusted at step622to enable the reconstructed RGB image to increase in similarity to the RGB input image. Thus, the fine-tuning network of disclosed embodiments can serve to improve the efficacy of the decomposition network.

Exemplary Computing Environment

FIG.7illustrates an exemplary computing environment on which an embodiment described herein may be implemented, in full or in part. This exemplary computing environment describes computer-related components and processes supporting enabling disclosure of computer-implemented embodiments. Inclusion in this exemplary computing environment of well-known processes and computer components, if any, is not a suggestion or admission that any embodiment is no more than an aggregation of such processes or components. Rather, implementation of an embodiment using processes and components described in this exemplary computing environment will involve programming or configuration of such processes and components resulting in a machine specially programmed or configured for such implementation. The exemplary computing environment described herein is only one example of such an environment and other configurations of the components and processes are possible, including other relationships between and among components, and/or absence of some processes or components described. Further, the exemplary computing environment described herein is not intended to suggest any limitation as to the scope of use or functionality of any embodiment implemented, in whole or in part, on components or processes described herein.

The exemplary computing environment described herein comprises a computing device10(further comprising a system bus11, one or more processors20, a system memory30, one or more interfaces40, one or more non-volatile data storage devices50), external peripherals and accessories60, external communication devices70, remote computing devices80, and cloud-based services90.

System bus11couples the various system components, coordinating operation of and data transmission between those various system components. System bus11represents one or more of any type or combination of types of wired or wireless bus structures including, but not limited to, memory busses or memory controllers, point-to-point connections, switching fabrics, peripheral busses, accelerated graphics ports, and local busses using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) busses, Micro Channel Architecture (MCA) busses, Enhanced ISA (EISA) busses, Video Electronics Standards Association (VESA) local busses, a Peripheral Component Interconnects (PCI) busses also known as a Mezzanine busses, or any selection of, or combination of, such busses. Depending on the specific physical implementation, one or more of the processors20, system memory30and other components of the computing device10can be physically co-located or integrated into a single physical component, such as on a single chip. In such a case, some or all of system bus11can be electrical pathways within a single chip structure.

Computing device may further comprise externally-accessible data input and storage devices12such as compact disc read-only memory (CD-ROM) drives, digital versatile discs (DVD), or other optical disc storage for reading and/or writing optical discs62; magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices; or any other medium which can be used to store the desired content and which can be accessed by the computing device10. Computing device may further comprise externally accessible data ports or connections12such as serial ports, parallel ports, universal serial bus (USB) ports, and infrared ports and/or transmitter/receivers. Computing device may further comprise hardware for wireless communication with external devices such as IEEE 1394 (“Firewire”) interfaces, IEEE 802.11 wireless interfaces, BLUETOOTH® wireless interfaces, and so forth. Such ports and interfaces may be used to connect any number of external peripherals and accessories60such as visual displays, monitors, and touch-sensitive screens61, USB solid state memory data storage drives (commonly known as “flash drives” or “thumb drives”)63, printers64, pointers and manipulators such as mice65, keyboards66, and other devices67such as joysticks and gaming pads, touchpads, additional displays and monitors, and external hard drives (whether solid state or disc-based), microphones, speakers, cameras, and optical scanners.

Processors20are logic circuitry capable of receiving programming instructions and processing (or executing) those instructions to perform computer operations such as retrieving data, storing data, and performing mathematical calculations. Processors20are not limited by the materials from which they are formed, or the processing mechanisms employed therein, but are typically comprised of semiconductor materials into which many transistors are formed together into logic gates on a chip (i.e., an integrated circuit or IC). The term processor includes any device capable of receiving and processing instructions including, but not limited to, processors operating on the basis of quantum computing, optical computing, mechanical computing (e.g., using nanotechnology entities to transfer data), and so forth. Depending on configuration, computing device10may comprise more than one processor. For example, computing device10may comprise one or more central processing units (CPUs)21, each of which itself has multiple processors or multiple processing cores, each capable of independently or semi-independently processing programming instructions. Further, computing device10may comprise one or more specialized processors such as a graphics processing unit (GPU)22configured to accelerate processing of computer graphics and images via a large array of specialized processing cores arranged in parallel.

System memory30is processor-accessible data storage in the form of volatile and/or nonvolatile memory. System memory30may be cither or both of two types: non-volatile memory and volatile memory. Non-volatile memory30ais not erased when power to the memory is removed and includes memory types such as read only memory (ROM), electronically erasable programmable memory (EEPROM), and rewritable solid-state memory (commonly known as “flash memory”). Non-volatile memory30ais typically used for long-term storage of a basic input/output system (BIOS)31, containing the basic instructions, typically loaded during computer startup, for transfer of information between components within computing device, or a unified extensible firmware interface (UEFI), which is a modern replacement for BIOS that supports larger hard drives, faster boot times, more security features, and provides native support for graphics and mouse cursors. Non-volatile memory30amay also be used to store firmware comprising a complete operating system35and applications36for operating computer-controlled devices. The firmware approach is often used for purpose-specific computer-controlled devices such as appliances and Internet-of-Things (IoT) devices where processing power and data storage space is limited. Volatile memory30bis erased when power to the memory is removed and is typically used for short-term storage of data for processing. Volatile memory30bincludes memory types such as random-access memory (RAM), and is normally the primary operating memory into which the operating system35, applications36, program modules37, and application data38are loaded for execution by processors20. Volatile memory30bis generally faster than non-volatile memory30adue to its electrical characteristics and is directly accessible to processors20for processing of instructions and data storage and retrieval. Volatile memory30bmay comprise one or more smaller cache memories which operate at a higher clock speed and are typically placed on the same IC as the processors to improve performance.

Interfaces40may include, but are not limited to, storage media interfaces41, network interfaces42, display interfaces43, and input/output interfaces44. Storage media interface41provides the necessary hardware interface for loading data from non-volatile data storage devices50into system memory30and storage data from system memory30to non-volatile data storage device50. Network interface42provides the necessary hardware interface for computing device10to communicate with remote computing devices80and cloud-based services90via one or more external communication devices70. Display interface43allows for connection of displays61, monitors, touchscreens, and other visual input/output devices. Display interface43may include a graphics card for processing graphics-intensive calculations and for handling demanding display requirements. Typically, a graphics card includes a graphics processing unit (GPU) and video RAM (VRAM) to accelerate display of graphics. One or more input/output (I/O) interfaces44provide the necessary support for communications between computing device10and any external peripherals and accessories60. For wireless communications, the necessary radio-frequency hardware and firmware may be connected to I/O interface44or may be integrated into I/O interface44.

Non-volatile data storage devices50are typically used for long-term storage of data. Data on non-volatile data storage devices50is not erased when power to the non-volatile data storage devices50is removed. Non-volatile data storage devices50may be implemented using any technology for non-volatile storage of content including, but not limited to, CD-ROM drives, digital versatile discs (DVD), or other optical disc storage; magnetic cassettes, magnetic tape, magnetic disc storage, or other magnetic storage devices; solid state memory technologies such as EEPROM or flash memory; or other memory technology or any other medium which can be used to store data without requiring power to retain the data after it is written. Non-volatile data storage devices50may be non-removable from computing device10as in the case of internal hard drives, removable from computing device10as in the case of external USB hard drives, or a combination thereof, but computing device will typically comprise one or more internal, non-removable hard drives using either magnetic disc or solid-state memory technology. Non-volatile data storage devices50may store any type of data including, but not limited to, an operating system51for providing low-level and mid-level functionality of computing device10, applications52for providing high-level functionality of computing device10, program modules53such as containerized programs or applications, or other modular content or modular programming, application data54, and databases55such as relational databases, non-relational databases, object oriented databases, BOSQL databases, and graph databases.

Applications (also known as computer software or software applications) are sets of programming instructions designed to perform specific tasks or provide specific functionality on a computer or other computing devices. Applications are typically written in high-level programming languages such as C++, Java, and Python, which are then either interpreted at runtime or compiled into low-level, binary, processor-executable instructions operable on processors20. Applications may be containerized so that they can be run on any computer hardware running any known operating system. Containerization of computer software is a method of packaging and deploying applications along with their operating system dependencies into self-contained, isolated units known as containers. Containers provide a lightweight and consistent runtime environment that allows applications to run reliably across different computing environments, such as development, testing, and production systems.

The memories and non-volatile data storage devices described herein do not include communication media. Communication media are means of transmission of information such as modulated electromagnetic waves or modulated data signals configured to transmit, not store, information. By way of example, and not limitation, communication media includes wired communications such as sound signals transmitted to a speaker via a speaker wire, and wireless communications such as acoustic waves, radio frequency (RF) transmissions, infrared emissions, and other wireless media.

External communication devices70are devices that facilitate communications between computing device and either remote computing devices80, or cloud-based services90, or both. External communication devices70include, but are not limited to, data modems71which facilitate data transmission between computing device and the Internet75via a common carrier such as a telephone company or internet service provider (ISP), routers72which facilitate data transmission between computing device and other devices, and switches73which provide direct data communications between devices on a network. Here, modem71is shown connecting computing device10to both remote computing devices80and cloud-based services90via the Internet75. While modem71, router72, and switch73are shown here as being connected to network interface42, many different network configurations using external communication devices70are possible. Using external communication devices70, networks may be configured as local area networks (LANs) for a single location, building, or campus, wide area networks (WANs) comprising data networks that extend over a larger geographical area, and virtual private networks (VPNs) which can be of any size but connect computers via encrypted communications over public networks such as the Internet75. As just one exemplary network configuration, network interface42may be connected to switch73which is connected to router72which is connected to modem71which provides access for computing device10to the Internet75. Further, any combination of wired77or wireless76communications between and among computing device10, external communication devices70, remote computing devices80, and cloud-based services90may be used. Remote computing devices80, for example, may communicate with computing device through a variety of communication channels74such as through switch73via a wired77connection, through router72via a wireless connection76, or through modem71via the Internet75. Furthermore, while not shown here, other hardware that is specifically designed for servers may be employed. For example, secure socket layer (SSL) acceleration cards can be used to offload SSL encryption computations, and transmission control protocol/internet protocol (TCP/IP) offload hardware and/or packet classifiers on network interfaces42may be installed and used at server devices.

In a networked environment, certain components of computing device10may be fully or partially implemented on remote computing devices80or cloud-based services90. Data stored in non-volatile data storage device50may be received from, shared with, duplicated on, or offloaded to a non-volatile data storage device on one or more remote computing devices80or in a cloud computing service92. Processing by processors20may be received from, shared with, duplicated on, or offloaded to processors of one or more remote computing devices80or in a distributed computing service93. By way of example, data may reside on a cloud computing service92, but may be usable or otherwise accessible for use by computing device10. Also, certain processing subtasks may be sent to a microservice91for processing with the result being transmitted to computing device10for incorporation into a larger processing task. Also, while components and processes of the exemplary computing environment are illustrated herein as discrete units (e.g., OS51being stored on non-volatile data storage device51and loaded into system memory35for use) such processes and components may reside or be processed at various times in different components of computing device10, remote computing devices80, and/or cloud-based services90.

In an implementation, the disclosed systems and methods may utilize, at least in part, containerization techniques to execute one or more processes and/or steps disclosed herein. Containerization is a lightweight and efficient virtualization technique that allows you to package and run applications and their dependencies in isolated environments called containers. One of the most popular containerization platforms is Docker, which is widely used in software development and deployment. Containerization, particularly with open-source technologies like Docker and container orchestration systems like Kubernetes, is a common approach for deploying and managing applications. Containers are created from images, which are lightweight, standalone, and executable packages that include application code, libraries, dependencies, and runtime. Images are often built from a Dockerfile or similar, which contains instructions for assembling the image. Dockerfiles are configuration files that specify how to build a Docker image. Systems like Kubernetes also support containers or CRI-O. They include commands for installing dependencies, copying files, setting environment variables, and defining runtime configurations. Docker images are stored in repositories, which can be public or private. Docker Hub is an exemplary public registry, and organizations often set up private registries for security and version control using tools such as Hub, JFrog Artifactory and Bintray, Github Packages or Container registries. Containers can communicate with each other and the external world through networking. Docker provides a bridge network by default, but can be used with custom networks. Containers within the same network can communicate using container names or IP addresses.

Remote computing devices80are any computing devices not part of computing device10. Remote computing devices80include, but are not limited to, personal computers, server computers, thin clients, thick clients, personal digital assistants (PDAs), mobile telephones, watches, tablet computers, laptop computers, multiprocessor systems, microprocessor based systems, set-top boxes, programmable consumer electronics, video game machines, game consoles, portable or handheld gaming units, network terminals, desktop personal computers (PCs), minicomputers, main frame computers, network nodes, virtual reality or augmented reality devices and wearables, and distributed or multi-processing computing environments. While remote computing devices80are shown for clarity as being separate from cloud-based services90, cloud-based services90are implemented on collections of networked remote computing devices80.

Cloud-based services90are Internet-accessible services implemented on collections of networked remote computing devices80. Cloud-based services are typically accessed via application programming interfaces (APIs) which are software interfaces which provide access to computing services within the cloud-based service via API calls, which are pre-defined protocols for requesting a computing service and receiving the results of that computing service. While cloud-based services may comprise any type of computer processing or storage, three common categories of cloud-based services90are microservices91, cloud computing services92, and distributed computing services93.

Microservices91are collections of small, loosely coupled, and independently deployable computing services. Each microservice represents a specific computing functionality and runs as a separate process or container. Microservices promote the decomposition of complex applications into smaller, manageable services that can be developed, deployed, and scaled independently. These services communicate with each other through well-defined application programming interfaces (APIs), typically using lightweight protocols like HTTP, gRPC, or message queues such as Kafka. Microservices91can be combined to perform more complex processing tasks.

Cloud computing services92are delivery of computing resources and services over the Internet75from a remote location. Cloud computing services92provide additional computer hardware and storage on as needed or subscription basis. Cloud computing services92can provide large amounts of scalable data storage, access to sophisticated software and powerful server-based processing, or entire computing infrastructures and platforms. For example, cloud computing services can provide virtualized computing resources such as virtual machines, storage, and networks, platforms for developing, running, and managing applications without the complexity of infrastructure management, and complete software applications over the Internet on a subscription basis.

Distributed computing services93provide large-scale processing using multiple interconnected computers or nodes to solve computational problems or perform tasks collectively. In distributed computing, the processing and storage capabilities of multiple machines are leveraged to work together as a unified system. Distributed computing services are designed to address problems that cannot be efficiently solved by a single computer or that require large-scale computational power. These services enable parallel processing, fault tolerance, and scalability by distributing tasks across multiple nodes.

Although described above as a physical device, computing device10can be a virtual computing device, in which case the functionality of the physical components herein described, such as processors20, system memory30, network interfaces40, and other like components can be provided by computer-executable instructions. Such computer-executable instructions can execute on a single physical computing device, or can be distributed across multiple physical computing devices, including being distributed across multiple physical computing devices in a dynamic manner such that the specific, physical computing devices hosting such computer-executable instructions can dynamically change over time depending upon need and availability. In the situation where computing device10is a virtualized device, the underlying physical computing devices hosting such a virtualized computing device can, themselves, comprise physical components analogous to those described above, and operating in a like manner. Furthermore, virtual computing devices can be utilized in multiple layers with one virtual computing device executing within the construct of another virtual computing device. Thus, computing device10may be either a physical computing device or a virtualized computing device within which computer-executable instructions can be executed in a manner consistent with their execution by a physical computing device. Similarly, terms referring to physical components of the computing device, as utilized herein, mean either those physical components or virtualizations thereof performing the same or equivalent functions.

As can now be appreciated, disclosed embodiments provide effective techniques for generating hyperspectral images utilizing input RGB images that can be acquired from low-cost, readily available digital cameras. The hyperspectral images that are generated from disclosed embodiments can have a wide variety of applications and practical uses. These can include identifying various features in ariel photography images. The features can include, but are not limited to, healthy grass, stressed grass, synthetic grass, evergreen trees, deciduous trees, soil, water, roads, railways, crosswalks, cars, trains, and so on. Disclosed embodiments improve the technical field of hyperspectral image acquisition by enabling a decomposition network and a fine-tuning network operating in conjunction as part of a training and/or image analysis process.

The skilled person will be aware of a range of possible modifications of the various aspects described above. Accordingly, the present invention is defined by the claims and their equivalents.