Patent Description:
Predicting measurement outcomes from an underlying structure often follows directly from fundamental physical principles. However, a fundamental challenge is posed when trying to solve the inverse problem of inferring the underlying source-configuration based on measurement data. A key difficulty arises from the fact that such reconstructions often involve ill-posed transformations and that they are prone to numerical artefacts.

This situation is well illustrated in the field of image reconstruction, and particularly by the reconstruction of current flow or magnetization maps of an electromagnetic source based on electromagnetic measurements.

As an example, neuronal current sources generate magnetic fields outside the body that is detected in MEG (brain imaging) or MCG (heart imaging). The clinician or neuro researcher is interested in which patches of cortex get activated either during rest or in response to specific stimulus. However determining the strength, location, and spatial extent of the sources from magnetic field data is an ill-posed problem as there is an infinite set of sources that can give rise to any given magnetic field distribution.

The state of the art for solving this type of problems is as follows: first assume a particular parametric form of the sources (current dipoles, a multipole expansion, etc.) as well as a model for the sources and its conductivity. Then, depending on the model chosen there are then either analytical or numerical formulas relating a given source distribution to a magnetic field map outside the source.

There are two main classes of algorithms solving the inverse problem of going from a magnetic field map to a distribution of sources. The first class consists in parametric modelling which assumes a small and fixed number of sources whose location and orientation may vary. The problem can be solved using either non-linear least squares search or so-called beamforming approaches. The second class consists in imaging approaches: the entire sample surface is tesselated into around <NUM> positions and a source of fixed orientation is placed at each vertex. The problem is reduced to determining set of amplitudes that best fit the data. This problem is linear, however the system of equations is highly underdetermined, and some kind of prior knowledge or constraint is required to obtain a solution. These constrains are usually formulated in a Bayesian framework.

Deep neural networks are characterized as universal approximators having the potential to compute any non-linear function. Thus, they are good candidates in tackling reconstruction problems. While deep learning is the state of the art tool for solving many tasks in natural language processing or computer vision, serious improvements are needed to completely surpass traditional methods for reconstruction tasks. The main issue is that the knowledge provided to the model to learn is constrained to the training data set.

In the case of reconstructing an image of an electromagnetic source, the number of parameters (e.g., magnetization direction, magnetometer standoff, and magnetometer direction) influencing the reconstructed electromagnetic source image is high. Thus, it requires a large data set to train the network with every possible combination of control parameters. As such, there is a risk of training the network to only solve a subset of the total number of problems which may lead to the network converging to non-physical solutions. Moreover, the network is likely to encounter measurements not seen in the training data as well as the presence of unseen noise. This is particularly problematic as even tiny perturbations in the input can produce artefacts in the reconstructed output. Another challenge for deep learning is that the solution is based on a "black-box" which leaves out any explanations of the reconstruction process. The estimations on new samples are statistical inferences based on previous learning without warranty on the new output.

The document <NPL>, discloses an algorithm for hologram synthesis that employs a convolutional neural network to perform image plane holography with unsupervised training. In order to enable the unsupervised training of the model, the algorithm computes a virtual reconstruction of the hologram based on the estimated computer-generated hologram solution and compares it to the target intensity pattern. The efficiency of this algorithm relies on the quality and quantity of the training datasets as many deep learning algorithms.

<NPL> describes a deep learning solution to the problem of localization of magnetoencephalography (MEG) brain signals. The proposed deep model architectures are tuned for single and multiple time point MEG data, and can estimate varying numbers of dipole sources using a loss function that measures the misfit between the algorithm output and the training data label.

<CIT> describes acquiring MR signals associated with the sample from a measurement device. A predetermined set of coil magnetic field basis vectors associated with a surface surrounding a sample is accessed. A non-linear optimization problem for MR information associated with the sample and coefficients in representation of coil sensitivities is solved based on a forward model that uses the MR information as inputs and simulates response physics of the sample to output computed MR signals corresponding to the MR signals, the coefficients and the predetermined vector.

<NPL> describes an electromagnetic tomography (EMT) image reconstruction algorithm based on autoencoder neural network of Restricted Boltzmann Machine (RBM). The encoding process of the encoder is equivalent to the object field detection process in the EMT system; the decoding process of the decoder is equivalent to the image reconstruction process.

<NPL> describes a data-synthesized spatio-temporally convolutional encoder-decoder network method termed DST-CedNet is proposed for Electromagnetic source imaging (ESI). DST-CedNet recasts ESI as a machine learning problem, where discriminative learning and latent-space representations are integrated in a convolutional encoder-decoder network (CedNet) to learn a robust mapping from the measured electroencephalography/ magnetoencephalography (EEG/MEG) signals to the brain activity. In particular, by incorporating prior knowledge regarding dynamical brain activities, a novel data synthesis strategy is devised to generate large-scale samples for effectively training CedNet. This stands in contrast to traditional ESI methods where the prior information is often enforced via constraints primarily aimed for mathematical convenience.

An aim of the present invention is therefore to provide a method for reconstructing images of electromagnetic sources using neural networks which can be efficient without large training datasets.

Another aim of the invention to provide a method for reconstructing images of electromagnetic sources using neural networks which can be efficient with less prior knowledge of the electromagnetic source.

According to the invention, these aims are attained by the object of the attached claims, and especially by a computer implemented method of reconstructing a digital image (<NUM>) of an electromagnetic source (<NUM>) using a neural network (<NUM>) comprising the steps of.

The present invention therefore propose to use a self-learning neural network to calculate the distribution and positions of electromagnetic sources. Rather than learning a mapping from an electromagnetic field to the electromagnetic sources based on a library of training data we propose to determine the weights and biases in the neural network 'on the fly' for each set of scan-data by repeatedly comparing the measured electromagnetic field map to the calculated electromagnetic field map, corresponding to an estimated source distribution. The difference between the measured electromagnetic map and the calculated electromagnetic map is used to iteratively train the network. This approach works no matter what forward model is used to calculate the electromagnetic map.

In a preferred embodiment, the loss function (<NUM>) may comprise a regularization term of the form <MAT> where S is the electromagnetic signal (<NUM>), Sθ is the estimated electromagnetic signal (<NUM>), and || · || is a norm.

The electromagnetic signal (<NUM>) may represent a physical quantity chosen among: a magnetic field, an electrical field, the gradient of a magnetic field or the gradient of an electrical field.

The loss function (<NUM>) may further comprise a term of the form <MAT> where C is a standard loss function depending on the estimated electromagnetic source vector (<NUM>) and on a ground-truth digital image of an electromagnetic source (<NUM>), and λ is a hyper-parameter controlling the influence of Cp.

The estimated electromagnetic source vector (<NUM>) may be determined using at least one of the parameters among: a value of an electromagnetic field, a first angle, a second angle, a first sensor angle, a second sensor angle, and a standoff distance between a sample and a sensor.

In a preferred embodiment, the neural network (<NUM>) may be a convolutional neural network, preferentially a convolutional neural network following a U-net architecture.

The neural network (<NUM>) may comprise at least one fully connected sub-network for estimating the first angle, the second angle, the first sensor angle and/or the second sensor.

The neural network (<NUM>) may be a fully connected neural network.

The computer-implemented method may further comprise a step of denoising the electromagnetic signal (<NUM>) executed prior to the step of providing the electromagnetic signal as input into the neural network.

The stop criterion is based on a threshold loss function value and/or on a signal to noise ratio metric.

According to the invention those aims are also attained by a system for reconstructing a digital image of an electromagnetic source from electromagnetic imaging data comprising:.

The at least one electromagnetic sensor (<NUM>) may comprise one or more of the following sensors: a scanning NV magnetometer, a widefield NV magnetometer, a superconducting quantum interference device, a hallbar, a magnetic resonance force microscope, and a tomographic sensor.

The present invention concerns a computer-implemented method for reconstructing a digital image of an electromagnetic source based on a measure of an electromagnetic signal.

In the context of this disclosure, an electromagnetic source means any physical object emitting an electromagnetic field. The electromagnetic signal can correspond to a measure of the generated electromagnetic field or to a quantity depending on the electromagnetic field. If the source is a magnetic source generating a magnetic field B, the electromagnetic signal can be the magnetic field B, but it can also be for example the gradient VB of the magnetic field or other quantities related to the magnetic field. Similarly, if the source is an electrical source generating an electrical field E, the electromagnetic signal can be the electrical field E, but also the gradient VE of the electrical field.

The aim of the present method is to input the electromagnetic signal into a neural network which will output an estimation of a magnetization vector or of a polarization vector. Based on this estimation, an estimation of the electromagnetic signal is determined using the linear relation between either the magnetic field and the magnetization vector, or between the electrical field and the electric polarization vector. Finally a loss function depending on the electromagnetic signal and the estimated electromagnetic signal is determined and minimized to find an optimal estimated electromagnetic source vector which can be used to reconstruct the digital image of the electromagnetic source. This can for example serve to reconstruct magnetization maps of magnetic source or current density maps.

In the context of this disclosure, the term "vector" may refer to a unique vector or it may refer to a "vector field". As an example, the present method allows the reconstruction of an estimated electromagnetic source vector, which can be for example represent a magnetic field which is a vector field. In this case, the estimated electromagnetic source vector could be understood as a unique vector of a vector field, or as the vector field itself. The meaning of the term "vector" shall be clear for the person skilled in the art depending from the context in which it is used.

In a particular embodiment, in which the electromagnetic source is a magnetic source generating a magnetic field B, the relation between the signal and the source is best expressed in the Fourier space. Under the assumptions that the magnetization vector M = M(x, y) is confined to a two-dimensional plane (x, y) and that the stray field B is measured in a parallel plane at an approximately known height z', i.e. B = B(x, y, z'), this relation reads: <MAT> where B̂x, B̂y, B̂z are the components of the Fourier transform of the magnetic field, M̂x, M̂y, M̂z are the components of the Fourier transform of the magnetization vector, kx, ky are the components of the transfer matrix, <MAT> and α = <NUM>ekz'/µ<NUM> contains an exponential that represents the propagator between the source and measurement planes. The assumptions on the magnetization vector and on the stray field are intended to obtain a simple mathematical expression and do not constitute any limitation.

The transformation from a magnetisation map to a magnetic field map is therefore a well-posed forward transformation with a unique solution. In contrast, the inverse problem of transforming from a magnetic field map to a magnetization map is ill-posed as the transformation matrix A is singular (its determinant is equal to zero), such that there is an infinite number of solutions for M̂.

In another embodiment, the electromagnetic source is an electrical source, and the electromagnetic signal represents the electrical field E. In this case, there are linear systems similar to the one presented above for the Fourier transform of the magnetic field and magnetization vector, relating the Fourier transforms of the electrical field E and of the electric polarization vector P.

The goal of the invention is to develop a neural network parametrized by a set of parameters θ that will learn to operate as the inverse of the transformation matrix A-<NUM>.

<FIG> illustrates a typical operation of the method of the present invention.

The first step of the present computer-implemented method consists in receiving an an electromagnetic signal <NUM> acquired by an electromagnetic sensor sensing said electromagnetic source. This may signify that a computer receives the electromagnetic signal from a sensor in real time or that the electromagnetic signal has been stored on a storing device and has been later received by a computer.

In the second step of the present method, the electromagnetic signal <NUM> is provided as input into a neural network <NUM> parametrized by a set of trainable parameters θ. The neural network outputs an estimated electromagnetic source vector <NUM> associated with the electromagnetic signal <NUM>. The estimated electromagnetic source vector <NUM> can be seen as an approximation of the target result which is a digital image <NUM> of the electromagnetic source <NUM>.

In the case mentioned above in which the electromagnetic signal represents a magnetic field B, the estimated electromagnetic signal obtained in the second step can therefore be seen as an approximation of the magnetization vector Mθ parametrized by the set of parameters θ.

The third step of the present method consists in determining a suitable loss function <NUM> to be able to train the neural network <NUM>. This is a key step as it is the choice of the loss function <NUM> that allows the neural network <NUM> to be efficient without needing large training datasets.

The loss function <NUM> is determined by performing the well-posed forward transformation on the neural network output, i.e. on the estimated electromagnetic source vector <NUM>, to transform it into an estimated electromagnetic signal <NUM>. This new signal can therefore be seen as an approximation of the electromagnetic signal <NUM> based on the estimated electromagnetic source vector <NUM>.

In the embodiment in which the electromagnetic signal <NUM> is the magnetic field B itself, the estimated electromagnetic signal corresponds to Sθ = AM̂θ, meaning that the transfer matrix A has been applied to the estimated magnetization vector M̂θ (up to a Fourier transformation).

The loss function <NUM> depends on both the electromagnetic signal <NUM> and the estimated electromagnetic signal <NUM>. More precisely, the loss function <NUM> is designed to quantify the difference between the measured electromagnetic signal and the estimated electromagnetic signal. This difference can be quantified in various different ways using traditional norms such as L<NUM> or L<NUM> norms. Any type of norm may be used without departing from of scope of the invention as the key point here is the dependency of the loss function on the electromagnetic signal and the estimated electromagnetic signal.

This choice of this family of loss functions <NUM> allows the neural network <NUM> to be efficient without prior training on large datasets with ground truth values. Such a loss function only depends on the original electromagnetic signal <NUM> and on the estimated electromagnetic signal <NUM>. That allows a model to train without the required ground-truth value. Moreover, it can directly learn on a single new input without previous training. So instead of traditionally training on a dataset and using the learned parameters to make new predictions, a network updates its parameters θ directly on the measured quantity which is the electromagnetic signal. The output of the network is not a statistical inference anymore but a solution depending on the well-posed forward transformation. Thus, the model's knowledge is not limited to a previous dataset but is directly taken from the input. Seeing that the loss function depends on the well-posed forward transformation, the proposed solution is ensured to be physically relevant. To this extent, the neural network is efficient while being untrained.

In the illustrative case of a magnetic source, this method fundamentally changes the ability to quantitatively determine from a stray-magnetic field image the magnetization of magnetic materials and current distributions in semiconductors, metals, or biological tissue. These capabilities are of scientific value, but even more so in the semiconductor industry (for the inspection of integrated circuits) or the magnetic memory industry for failure analysis and device development.

In a particular embodiment, the loss function <NUM> comprises a term of the form: <MAT> where S is the electromagnetic signal <NUM>, Sθ is the estimated electromagnetic signal <NUM>, and || · || is a norm. As it can be seen from the above equation, the term Cp only depends on the input electromagnetic signal S and on the estimated electromagnetic signal Sθ. In particular, there is no dependence on any ground-truth value of the target electromagnetic source digital image.

The norm ∥ · ∥ can be any norm commonly used in machine learning or more generally, any norm matching the dimensions of its argument. In particular, the absolute mean norm L<NUM> or the Euclidean norm L<NUM> are good candidates, however, many other norms may serve to the purpose of the invention.

This loss function <NUM> can be seen as a regularization term. Usually, regularization acts on the parameters θ size in order to overcome overfitting. Where a physics regularization deals with the implementation of knowledge into the model. This added knowledge constraints the learning to a desired physical solution by adding a penalty term Cp to a standard loss function C. In our case, this term is the forward solution for reconstructing the estimated electromagnetic signal from the estimated electromagnetic source vector. Thus, at the end of each iteration, the estimated electromagnetic signal is computed from the current estimated electromagnetic source vector. By doing so, it can be compared with the original electromagnetic signal at each iteration and be included in the loss function.

In another embodiment, the loss function <NUM> can comprise a term <MAT> where C is a standard loss function depending on the estimated electromagnetic source vector <NUM> and on a ground-truth digital image of an electromagnetic source <NUM>, and λ is a hyper-parameter controlling the influence of Cp. As explained above, this loss function Cr can be seen as a standard loss function C to which a regularization term Cp has been added, for example to constraint the learning to a physically relevant solution.

In the presence of a training dataset containing ground-truth values, a loss function including both a standard loss function C depending on ground-truth values and on a loss function Cp depending only on the input signal, can be trained by combining the traditional approach (based on the ground-truth values) using the dataset, with the feed-forward approach (based on each input electromagnetic signal).

As mentioned above, the electromagnetic signal <NUM> may represent a physical quantity selected among: a magnetic field, an electrical field, the gradient of a magnetic field or the gradient of an electrical field.

In the case in which the electromagnetic signal <NUM> is a gradient of a magnetic or electrical field, the neural network <NUM> still outputs an estimated electromagnetic source vector <NUM> that represents an estimated magnetization or electric polarization vector. Based on this, the estimated electromagnetic signal <NUM> can be reconstructed by applying the relevant forward transformation and taking the gradient of the result. The loss function <NUM> can then be determined based on the original (measured) gradient, i.e. the electromagnetic signal, and the estimated gradient, i.e. the estimated electromagnetic signal.

The estimated electromagnetic source vector <NUM> can be determined using at least one of the parameters among: a value of an electromagnetic field, a first angle, a second angle, a first sensor angle, a second sensor angle, and a standoff distance between a sample and a sensor.

In an embodiment, the parameters are the following: a value of a magnetic field, a first magnetization angle, a second magnetization angle, a first sensor angle, a second sensor angle and a standoff distance between a sample source and a sensor. In this case, the sensor is typically a magnetometer.

In another embodiment, the parameters are the following: a value of an electrical field, a first electric polarization angle, a second electric polarization angle, a first sensor angle, a second sensor angle, and a standoff distance between a sample and a sensor. In this case, the sensor is typically an electrometer.

The possibility to train the neural network directly on a single image makes it possible to use different architecture without facing a computational problem. In this main text, we used a convolutional neural network following a U-net architecture. Where the cost function was the mean average loss between the reconstructed electromagnetic source and the original one.

Since convolutional layer requires a fixed-size input, a border is added to the input image to match the correct format. In order to make sure this border is not taken into account into the reconstruction, a region of interest layer can be inserted to focus the learning of the network to the desired region.

Some of the parameters listed above (i.e. an electromagnetic field value, the first and second angle, the first and second sensor angle and the standoff distance between the sensor and the sample) used to determine the estimated electromagnetic source vector, can be deduced directly by the model. In particular, the two pairs of angles can be deduced by the neural network.

In a particular embodiment, a branch of the neural network can be dedicated to each of these parameters that can be deduced by the model. A fully-connected sub-network can therefore be dedicated to the learning of each one of these parameters.

In another embodiment, the neural network is itself a fully-connected network.

One issue with implementing any reconstruction method is noise in the original data can be amplified during the reconstruction process. Thus, data with excessive noise is often ruled as ineligible for reconstruction, requiring either additional measurement time or a new measurement to be taken. This reconstruction is no different, in fact, the ill-posed problem is known to amplify noise. With a trained neural network, one can generalize a model to get rid of the noise in the reconstruction. However, one still faces the risk that unseen perturbations will create artefacts in the reconstruction.

As the untrained neural network of the present invention learns on the input directly, the quality of the reconstruction is directly linked to the quality of the input. Thus, one can easily improve the reconstruction quality by minimizing the noise beforehand.

For this reason, the method may comprise a step of denoising the electromagnetic signal. This step is preferably executed before providing the electromagnetic signal as input into the neural network.

Isolating the denoising from the reconstruction is valuable, as it allows for the possibility of using noisy images without the need for additional training and facilitates double checking of the denoise process itself for loss of information. In contrast, a trained neural network model would have to learn the denoising process, and can thus generalize incorrectly for a given data set.

The longer the untrained model learns, the more noise is taken into account into the reconstruction. In an embodiment, a signal to noise ratio (SNR) metric can be used as a stop criterion for the learning in order to limit the amount of noise included in the reconstruction. Typically, once the SNR has reached a minimum, the learning stops even though the loss function can still be reduced.

Other stop criteria for determining when the learning of the network has to stop can be considered. A threshold loss function value can be chosen before or during the learning as stop criterion. Alternatively or complementarily, a number of iterations of the learning process can be set as a stop criterion.

The present invention also concerns a system for reconstructing a digital image of an electromagnetic source from electromagnetic imaging data. As illustrated in <FIG>, the system comprises at least one electromagnetic sensor <NUM> for sensing an electromagnetic signal <NUM>, as well as means for carrying out the computer implemented method described above to obtain a digital image <NUM> of the electromagnetic source <NUM>.

The at least one sensor can be any type of electromagnetic sensor or combinations of the like, including various types of magnetometers such as scanning NV magnetometers, widefield NV magnetometers, superconducting quantum interference devices (SQUID), hallbars, magnetic resonance force microscopes (MRFM), tomographic sensors or other imaging devices.

Means for carrying out the computer-implemented method include a processing unit <NUM> such as a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU) and/or graphics processing unit (GPU). It also includes a memory unit such as SSD and/or HDD memory units or RAM memory modules.

The at least one sensor may be connected to the processing unit and/or memory modules. Alternatively, the signals sensed by the at least one sensor can be stored in a database that can be later accessed by the processing unit.

Depending on the embodiment, certain acts, events, or functions of any of the algorithms described herein can be performed in a different sequence, can be added, merged, or left out altogether (for example, not all described acts or events are necessary for the practice of the methods). Moreover, in certain embodiments, acts or events can be performed concurrently, for instance, through multi-threaded processing, interrupt processing, or multiple processors or processor cores or on other parallel architectures, rather than sequentially. In addition, different tasks or processes can be performed by different machines or computing systems that can function together.

The various illustrative logical blocks and modules described in connection with the embodiments disclosed herein can be implemented or performed by a machine, a microprocessor, a state machine, a digital signal processor (DSP), an application specific integrated circuit (ASIC), an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A hardware processor can include electrical circuitry or digital logic circuitry configured to process computer-executable instructions. In another embodiment, a processor includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. A computing environment can include any type of computer system, including, but not limited to, a computer system based on a microprocessor, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computational engine within an appliance, to name a few.

The steps of a method, process, or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module stored in one or more memory devices and executed by one or more processors, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of non-transitory computer-readable storage medium, media, or physical computer storage known in the art. An example storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The storage medium can be volatile or nonvolatile. The processor and the storage medium can reside in an ASIC.

Claim 1:
A computer implemented method of reconstructing a digital image (<NUM>) of an electromagnetic source (<NUM>) using a neural network (<NUM>) comprising the steps of
receiving an electromagnetic signal (<NUM>) acquired by an electromagnetic sensor (<NUM>) sensing said electromagnetic source (<NUM>),
providing the electromagnetic signal (<NUM>) as input into a neural network (<NUM>) parametrized by a set of parameters (θ) and obtaining as an output of the neural network (<NUM>), an estimated electromagnetic source vector (<NUM>) associated to the electromagnetic signal,
determining a loss function (<NUM>),
minimizing the loss function (<NUM>) by feed-forwarding the neural network (<NUM>) until a stop criterion is met so as to obtain an optimal electromagnetic source vector (<NUM>),
reconstructing a digital image (<NUM>) of the electromagnetic source (<NUM>) based on the optimal electromagnetic source vector (<NUM>),
characterized in that determining the loss function (<NUM>) comprises
computing an estimated electromagnetic signal (<NUM>) by applying a well-posed forward transformation on the estimated electromagnetic source vector (<NUM>), the loss function (<NUM>) depending on the electromagnetic signal (<NUM>) and the estimated electromagnetic signal (<NUM>).