JuanCastillo29/space-mission-intelligence-data huggingface dataset:
In this section, we first describe the data pretreatment required for nonnegative matrix factorization purpose followed by the data mining method.
3.1. Data pretreatment After calibration, the NOMAD SO spectra are in transmittance T = I/I0, depending on wavenumber ν, with I the observed light intensity trough the atmosphere and I0 the solar spectra measured outside the atmosphere. Assuming that the atmosphere is homogeneous, and that multiple scattering and refraction are negligible (Smith and Hunten, 1990; Bovensmann et al., 1999), the optical depth τ is a linear combination of E(ν) the total extinction, and ϵ the slant column density, for each chemical species i:
NS
�
τ(ν) = −log T(ν) ≈
Ei(ν).ϵi + MC(ν) (1)
i=1
with NS, the total number of species and MC(ν) a modeled continuum described below. The slant column density ϵ is directly related to the total number of particles N(s) along the line of sight s:
ϵ = � N(s)ds (2) While the extinction by gas is usually highly structured, absorption by particles, scattering by molecules and particles, and also reflection at the surface are broadband features. Such large features are modeled by a continuum MC(ν), often taken as a polynomial, that is filtered out. The problem with this continuum removal rationale is that when the optical depth is large, the SNR is decreased and the noise effect on continuum removal amplified (see Sup. Mat.). Instead of using this rationale, we propose to first correct for the continuum C(ν) in the transmittance space:
T ∗(ν) = T(ν) −C(ν) (3)
Then convert the spectra into absorbance:
X(ν) = 1 −T ∗(ν) (4)
The final step is the linear mixture :
NS
�
X(ν) ≈
Si(ν).Ai (5)
i=1
with Si(ν) the source spectra and Ai the spectral abundance. In this description, the physical meaning of Si(ν) and Ai is lost but the apparent SNR 4 is dramatically increased, which is much more important for our analysis.
2009), that has been optimized (Schmidt et al., 2010), hereafter called BPSS2. This algorithm has the main advantage to account for extra constraint : the sum-to-one or sum-lowerthan-one on the abundances (�
i Aij = 1) that also promotes sparsity of S. This algorithm, based on Monte Carlo approach is much more time consuming. One approach to reduce the computation time is to select only relevant spectra out of the dataset (Moussaoui et al., 2008), but then the statistics may be biased (Schmidt et al., 2010).

Thanks to the advances of computer capabilities, we propose to treat the full dataset. This kind of algorithm is very slow but since the formulation is Bayesian, it converge toward an unique solution.
psNMF. In order to regularize the problem of eq. 6, one can add an extra penalization term to enforce sparsity on A (only few non zeros elements in A) (Kim and Park, 2007) :
F = ∥X −S.A∥2 + λ ∥A∥1 (7) With ∥.∥1, the L1 norm. The first term is called data attachment term (the usual squared difference). The second is called regularization term. The problem with this approach, is that hyperparameter λ is not known and has to be tuned manually. A recent approach has been proposed to solve this problem in the Bayesian framework (Hinrich and Mørup, 2018). The main idea is to encompass all variables and hyperparameters in a unique problem that is estimated with variational update principle. We will refer this algorithm to probability sparse NMF (psNMF). This algorithm has the advantage to have a reduced computation time and no hyperparameter tuning. It also has a regularization term to avoid strong dependence of the initialization on the final solution. In order to estimate the precision of the reconstruction, we used the Root Mean Square Difference RMSD:
���
�2�
RMSD =
⟨X⟩ (8) With ⟨.⟩, the mean. Once the sources are estimated, we quantify their relevance for the global dataset. From the total reconstruction ˙Xkj= ˙Ski. ˙Aij, for all i, we can estimate the contribution of source i′, that is to say: ˙Xi kj= ˙Ski′. ˙Ai′j. Thus, the relevance
of source i is defined as:
����˙Xi −˙X ��� �
Ri =
�
� (9)
˙X
This definition is convenient since the sum of all Ri is one (this property is only present when sources and abundances are positive) and we can easily estimate the % contribution of each source in the final reconstruction. One has to note that relevance is not a measure of presence or not of a minor specie (for instance CH4) but a measure of how important is the source over the dataset. Major species, should always have a larger relevance than minor species. In the following, we plot all sources results by decreasing order of relevance.

3.3. Band depth (BD) We used the following band depth definition, difference of the geometric mean of two reference wavenumbers in the continuum, compared to the band:
νc−νl
νr−νc
(10)
νr−νl −X(νc)
νr−νl .X(νr)
BD = X(νl) with X the observed spectra in transmittance, νc the wavenumber of the center of band, νl the wavenumber of the reference level on the left (smaller wavenumber), νr the wavenumber of the reference level on the right (larger wavenumber).
We simulated several synthetic observations in different conditions, to mimic the case of NOMAD-SO. The first section describes a simple toy model example and the second one presents extensive tests of this toy model with various cases. By hidden spectra, hidden compounds and hidden CH4, we always refer to a spectral dataset with a dominant major component (here water) and a minor specie (here CH4). The goal of the proposed approach is to pick up a source, containing CH4 only.
4.1. Toy example 4.1.1. Synthetic dataset In order to demonstrate the usefulness of our method, we propose here a toy example in a very difficult case. We will see that usual method fails detection but our method is able to detect the hidden compounds. For this toy example, we simulate a linear mixture of NO = 104 observations spanned over Nν = 320 spectels (see fig. 1) similar to order 136 of NOMADSO. Each spectrum is a mixture of a spectra of water vapor SH2O (coming from one actual source estimated from real data using psNMF) and theoretical methane SCH4 from Villanueva et al. (2018), with corresponding abundances
AH2O, ACH4:
(11)
X = SH2O.AH2O + SCH4.ACH4 + n
The noise n is assumed to be a Gaussian process with a standard deviation of σ=0.001 and no bias: n = G (0, σ). All spectra contain pure water vapor with a coefficient following AH2O = 5/6.β(1, 10) + 1/6.U(0, 1), a mixture of beta (β) distribution for 5/6 of the sample and an uniform (U) distribution for 1/6 of the sample. This process mimics well the water vapor band depth distribution (BD, see definition in section 3.3) of the real dataset (see Fig.2). As the baseline of SH2O is not zero, we also mimic baseline correction errors. In addition 100 spectra out of 10000 contain methane with ACH4 = 1, such that the band depth of SCH4 is at 3-σ level. Please note that the model to generate the data is not fulfilling the sum-to-one constraint, but fully fulfilling the positivity constraint.

[Figure: Figure 1: Synthetic dataset containing 104 spectra with various abundances of H2O and 100 containing CH4 at 3-σ level of the noise. In blue the reference spectra SH2O of H2O (coming from actual data analysis). In red the reference spectra SCH4 of CH4 (from theoretical data).]

Given the defined noise and signal level, the RMSD expected for a perfect reconstruction of the signal (and not the noise) is 0.16. The final synthetic dataset is represented in Fig. 1. In order to check the quality of the estimation, we simply compute the correlation coefficient between SCH4 and the estimated NS sources ˙S, using:
� (12)
SCH4, ˙S:i
Q = corr �
The ith source with the maximum correlation is identified to CH4 contribution. The value to the maximum correlation is used as metric to assess the quality of the retrieval.
4.1.2. Results By plotting the 10000 samples of the dataset, one is able to identify easily the H2O bands. Nevertheless, we cannot observe the target CH4 in the average 2.5
1.5
frequency
frequency
0.5
0 0.01 0.02 0.03 0.04 0.05 0.06 BD
0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 BD

[Figure: Figure 2: Water vapor Band Depth distribution (left) in the real observation (right) modeled by the toy example.]

[Figure: Figure 2: Water vapor Band Depth distribution (left) in the real observation (right) modeled by the toy example.]
no particular elements can be used to argue for detection.
Figure 4 represents the results from the non-negative matrix factorization
that the relevance of source 4 is very low (0.4%), meaning that only 0.4% of the
species.
difficult problem, we will not pay excessive attention on this parameter.
4.1.3. Convergence and computation time
spectrum, even at 3-σ level, because it is lost in the baseline changes.
The second simple tool for detection would be the analysis of the band depth.
no particular elements can be used to argue for detection.
Figure 4 represents the results from the non-negative matrix factorization
that the relevance of source 4 is very low (0.4%), meaning that only 0.4% of the
species.
difficult problem, we will not pay excessive attention on this parameter.

[Figure: Figure 3 (left) shows the histogram of the main CH4 band that exhibits no sign]
to 2000. For BPSS2, we compute a minimum burn in of 1000 iterations and after
that when the long term statistics (1000 last iterations) of the Markov Chain is
close to the short term statistics (100 last iterations), convergence is considered
to be reached. Then another 1000 iterations are computed to estimate the final
solution statistics.
We run the 3 identified tools 10 times on the same dataset with different
noise realization, and compute mean and standard deviation from these 10 ex-
periments. Results are presented in Table 1. One can clearly see that the even
if the convergence is set, there is a high variability in MU results, due to the
4.1.3. Convergence and computation time
We set the MU algorithm convergence to relative difference of the cost func-
to 2000. For BPSS2, we compute a minimum burn in of 1000 iterations and after
that when the long term statistics (1000 last iterations) of the Markov Chain is
close to the short term statistics (100 last iterations), convergence is considered
to be reached. Then another 1000 iterations are computed to estimate the final
solution statistics.
We run the 3 identified tools 10 times on the same dataset with different
noise realization, and compute mean and standard deviation from these 10 ex-
periments. Results are presented in Table 1. One can clearly see that the even
0.02
0.018
0.016
0.014
Absorbance
frequency
0.012
0.01
0.008
0.006
0.004
0.002
-6 -4 -2 0 2 4 BD
10-3
wavenumber [cm-1]

[Figure: Figure 3: (left) Histogram of Band Depth at 3067.2 cm−1 from the dataset containing 100 CH4 at 3-σ level out of 104 spectra. (right) 100 spectra with the maximum Band Depth at 3067.2 cm−1 specific of CH4. Signal is dominated by water and by noise. No specific signature of CH4 is visible.]

[Figure: Figure 3: (left) Histogram of Band Depth at 3067.2 cm−1 from the dataset containing 100 CH4 at 3-σ level out of 104 spectra. (right) 100 spectra with the maximum Band Depth at 3067.2 cm−1 specific of CH4. Signal is dominated by water and by noise. No specific signature of CH4 is visible.]
wavelength
wavelength

[Figure: Figure 4: Results of the psNMF algorithm for NS = 4. Sources 1 and 3 are identified to the level with significant noise contribution, source 2 is identified to H2O (correlation coef. with groundtruth 0.99), and source 4 is CH4 (correlation coef. with groundtruth 0.98). Relevance is computed from Eq. 9.]

[Figure: Figure 4: Results of the psNMF algorithm for NS = 4. Sources 1 and 3 are identified to the level with significant noise contribution, source 2 is identified to H2O (correlation coef. with groundtruth 0.99), and source 4 is CH4 (correlation coef. with groundtruth 0.98). Relevance is computed from Eq. 9.]
4 5 6 7 8 NbNb e ennddmmemebmerber |  | Factor above noise level of 2 Effect of Ns for noise factor 2 with 100 CH4 spectra |  |  | 1 0.9 0.8 0.7 0.6 ue 0.5 VVal al ue 0.4 0.3 0.2 0.1 .5 0 9 3 |  | Factor above noise level of 3 Effect of Ns for noise factor 3 with 100 CH4 spectra |

| --- | --- | --- | --- | --- | --- | --- | --- |
|  | MU mean corr. psNMF mean corr. MU frac. corr. > 0.5 psNMF frac. corr. > 0 |  |  |  |  | MU mean corr. psNMF mean corr. MU frac. corr. > 0.5 psNMF frac. corr. > 0.5 |  |
|  |  |  |  |  | .5 |  |  |
|  |  |  | MU mean corr. psNMF mean corr. MU frac. corr. > 0.5 psNMF frac. corr. > 0 |  | .5 |  |  |
|  |  |  |  |  |  |  |  |

[Figure: Figure 5: Results of the MU and psNMF algorithm for NS = 3 to 9, NO = 10000, N⌫= 320, as a function of the number of source. The average Q of 10 realizations of the best estimated source (thick lines and standard deviation in thin lines) and the fraction of acceptable results (with Q > 0.5). (left) with a factor above noise level of 2 (right) with factor above noise level of 3.]

[Figure: Figure 5: Results of the MU and psNMF algorithm for NS = 3 to 9, NO = 10000, Nν = 320, as a function of the number of source. The average Q of 10 realizations of the best estimated source (thick lines and standard deviation in thin lines) and the fraction of acceptable results (with Q > 0.5). (left) with a factor above noise level of 2 (right) with factor above noise level of 3.]
always stays below 200 seconds).
For the second set of tests, we used the same toy model, except with 50 and
deviation of the noise (this number is called “factor above noise level”).
In
order to have robust results, we made 10 realizations and averaged the results.
Results are always with RM SD < 0.18 with an average ⇠ 0.16. RM SD from
it’s contribution to RM SD is negligible), so the reconstruction is in average as
expected.
Figure 6 presents two quality indicators of the results: the average correla-
tion coefficient Q (see Eq. 12) and the fraction of realization with acceptable
results (with Q > 0.5). Both indicators indicate that the method psNMF clearly
outperforms MU at high factor above noise level. From our visual inspection of
the results, we define the detection limit when at least 50% of the results are
with Q > 0.5 (correlation coefficient > 0.5). This definition is debatable but
there is no absolute way of defining it. Figure 6 shows that the detection limit
is at 1.5 factor above noise level for 100 hidden spectra case, around 2 for 50
spectra from the noise. For 20 hidden spectra, even at a factor above noise level
that the psNMF is less stable since the standard deviation is much larger.
estimation. Nevertheless, it requires more computation time for a larger number
of source ( approximately x2 between 3 and 9 sources but the computation time
always stays below 200 seconds).
For the second set of tests, we used the same toy model, except with 50 and
deviation of the noise (this number is called “factor above noise level”).
In
order to have robust results, we made 10 realizations and averaged the results.
Results are always with RM SD < 0.18 with an average ∼ 0.16. RM SD from
it’s contribution to RM SD is negligible), so the reconstruction is in average as
expected.
Figure 6 presents two quality indicators of the results: the average correla-
tion coefficient Q (see Eq. 12) and the fraction of realization with acceptable
results (with Q > 0.5). Both indicators indicate that the method psNMF clearly
outperforms MU at high factor above noise level. From our visual inspection of
the results, we define the detection limit when at least 50% of the results are
with Q > 0.5 (correlation coefficient > 0.5). This definition is debatable but
there is no absolute way of defining it. Figure 6 shows that the detection limit
is at 1.5 factor above noise level for 100 hidden spectra case, around 2 for 50
spectra from the noise. For 20 hidden spectra, even at a factor above noise level
that the psNMF is less stable since the standard deviation is much larger.
[Figure: figure 6]

| 10 Eff0 ec t h of i nd oisd e fe actn or wC ithH 1004 C Hs 4p spe ecc tratra 1 MU mean corr. 0.9 p Ms UN M fraF c m . ce oa rn r. c >o 0rr .. 5 psNMF frac. corr. > 0.5 0.8 0.7 0.6 ue 0.5 VVal al ue 0.4 0.3 0.2 0.1 0 1 1.5 2 2.5 Factor abNooivsee f anctoorise level |  | 10 Eff0 ec t h of i nd oisd e fe actn or wC ithH 1004 C Hs 4p spe ecc tratra |  | 1 0.9 0.8 0.7 0.6 uee 0.5 VVaallu 0.4 0.3 0.2 0.1 0 3 |  | 5 E0 ffe ch t oi fd nod isee fan cto rC witH h 540 Cs H4p spe ecc tratra |
| --- | --- | --- | --- | --- | --- | --- |
|  | MU mean corr. psNMF mean corr. MU frac. corr. > 0.5 psNMF frac. corr. > 0.5 |  |  |  |  |  |
|  |  |  |  |  | MU mean corr. psNMF mean corr. MU frac. corr. > 0.5 psNMF frac. corr. > 0.5 |  |

[Figure: Figure 6: Results of the MU and psNMF algorithm for NS = 5, NO = 10000, N⌫= 320, as a function of the factor above noise level. The average Q of 10 realizations of the best estimated source (thick lines and standard deviation in thin lines) and the fraction of acceptable results (with Q > 0.5). (left) with 100 hidden CH4 spectra (right) with 50 hidden CH4 spectra]

[Figure: Figure 6: Results of the MU and psNMF algorithm for NS = 5, NO = 10000, Nν = 320, as a function of the factor above noise level. The average Q of 10 realizations of the best estimated source (thick lines and standard deviation in thin lines) and the fraction of acceptable results (with Q > 0.5). (left) with 100 hidden CH4 spectra (right) with 50 hidden CH4 spectra]
maximum of ˙S:i∗to be considered significant.

Please note that even there are only 5 possible fraction (0, 0.25, 0.5, 0.75 and 1), since we average on 10 realizations, any number can appear.
• Mean distance to the expected center.

Mean distance in spectel between the CH4 peaks detected in the best source and the reference one.
• Abundance of CH4 in the source. αCH4 (from Eq. 13), which describes the amplitude of the CH4 peaks in the best source.
5.2.2.

Analysis of the results Figure 7 summarizes all the results.

Fraction of the 4 main CH4 peaks detected in the most relevant source has always a standard deviation < 0.43 and a mean value of 0.06 over the 10 realizations.

The Mean distance to the expected center has always a standard deviation < 0.40 and a mean value of 0.07 over the 10 realizations.

The abundance of CH4 in the source has always a standard deviation < 0.05 and a mean value of 0.005 over the 10 realizations.

This figure shows that the detection limits clearly depend on CH4 density, but also on the fraction of hidden CH4 and noise level, as expected.

Abundance of CH4 in the source αCH4 maximum is 25%, meaning that in any cases H2O is dominating the best source and so both CH4 and H2O are present in each best source.

This is because CH4 is a minor specie (as expected from the conditions of our simulation), its absorption band generally follows the air-mass, as H2O does.

So there is no particular source for CH4 only.

When more than two lines are detected, we can consider it as a detection.

This limit is reached for CH4 ≥500 ppt for 10 and 100 ppm of H2O.

Nevertheless, the detection limits lies between 100 and 500 ppt in the case of 10 ppm of H2O vapor since the detection is perfect (100% of the 4 main CH4 peaks detected) occurs for a fraction of CH4 5 to 50%.

Interestingly, the optimum detection is not when 100% of the spectra contains CH4, but more between 5-50 %.

This behavior is due to the statistics that is richer when also CH4 is lacking in certain spectra.

When 100% of spectra contain CH4, the statistical variability of the dataset is mainly due to airmass (atmosphere is assumed to | Fraction of the Fraction of the 4 main peaks detected 1 fraction CH 4 = 100% 0.8 f fr ra ac ct ti io on n C CH H 4 4 = = 5 10 0% % 0.6 f fr ra ac ct ti io on n C CH H 4 4 = = 5 1% % allue ue Vva 0.4 0.2 0 10 2 Mean distance to th Mean distance to the expected center [in pixel] 2 1.5 uee 1 Vv al alu 0.5 0 10 2 Abundance Abundance of CH4 in the source 0.25 noise level 0.001 noise level 0.0001 0.2 luuee 0.15 Vvaal 0.1 0.05 0 10 2 CHCH4 4d deennsistyi t[iyn p[pitn] ppt] |  |  |  |  |  |  | peaks detected Fraction of the 4 main peaks detected fraction CH 4 = 100% fraction CH 4 = 50% fraction CH 4 = 10% fraction CH 4 = 5% fraction CH 4 = 1% 10 2 10 3 cted center [in pixel] Mean distance to the expected center [in pixel] 10 2 10 3 in the source Abundance of CH4 in the source noise level 0.001 noise level 0.0001 10 2 10 3 CH4C Hd 4e dnenssiittyy [ in[ ipnp t]ppt] |  |

[Figure: figure 7]
fraction CH 4 = 50% fraction CH 4 = 10% fraction CH 4 = 5% fraction CH 4 = 1% 10 2 10 3 cted center [in pixel] Mean distance to the expected center [in pixel] 10 2 10 3 in the source Abundance of CH4 in the source noise level 0.001 noise level 0.0001 10 2 10 3 CH4C Hd 4e dnenssiittyy [ in[ ipnp t]ppt] |  |

[Figure: figure 7]

| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  | Fraction of the |  | 4 main | peaks detected |  |
|  |  | Fraction of the 4 main peaks detected |  |  |  | 1 0.8 0.6 auleue 0.4 v Val 0.2 0 10 3 |  | Fraction of the 4 main peaks detected |
|  | fraction CH 4 = 100% fraction CH 4 = 50% fraction CH 4 = 10% fraction CH 4 = 5% fraction CH 4 = 1% |  |  |  |  |  | fraction CH 4 = 100% fraction CH 4 = 50% fraction CH 4 = 10% fraction CH 4 = 5% fraction CH 4 = 1% |  |
|  |  |  | Mean distance to th he expected center [in pixel] |  |  | e expe | cted center [in pixel] Mean distance to the ex |  |
|  |  |  |  |  |  |  |  |  |

|  | of CH4 0.15 |  |
| --- | --- | --- |
| noise level 0.0001 |  | noise level 0.0001 |
tively on Fig. 8, 9 and 10. We used the 821 ingress and egress transit orbits
for order 119, 2358 orbits for order 134 and 703 for order 136. We filter spectra
In this section, we report the results of actual NOMAD data, focusing on
tively on Fig. 8, 9 and 10. We used the 821 ingress and egress transit orbits
for order 119, 2358 orbits for order 134 and 703 for order 136. We filter spectra
with SNR > 100. Results are compared with NOMAD simulations (Villanueva
|  | 119 | 134 | 136 |
| --- | --- | --- | --- |
| N O | 134045 | 365985 | 140064 |
| N = 4 S | 0.476 | 0.575 | 0.634 |
| N = 5 S | 0.456 | 0.553 | 0.609 |
| N = 6 S | 0.442 | 0.553 | 0.585 |
| N = 10 S | 0.410 | 0.484 | 0.544 |

[Table: Table 3: Number of spectra NO and RMSD relative errors for 4 to 10 number of sources NS resulting from the analysis of all observations of NOMAD data up to 15 January 2020, using the psNMF algorithm. RMSD is computed from Eq. 8.]

et al., 2018) using the calibration pipeline. This process adds ghost lines from adjacent orders, as in real data. Table 3 summarizes the relative error and the number of spectra. The approach here is to compute the analysis with psNMF using NS = 5 in agreement with the previous section. Please remind that our approach is fully blind: no spectral information has been included in the analysis (nothing about H2O, CO2 or CH4). For all orders, sources of H2O are estimated, as expected. Also a source presenting a residual of the continuum is always present. Due to non-linearities of the radiative transfer, the acquisition process (temperature dependence) and the wavenumber shift, the molecular species appears sometimes in different sources. Order 136 gives the 1 source related to the background and 4 sources related to H2O. All 4 sources of water have the peaks but with different relative intensities and wavenumber shift. For order 119, both CO2 and H2O lines are identified (see Fig. 8). Since those two components are uncorrelated, separated sources are found by the algorithm. Interestingly, order 134 presents a source with unexpected lines. The main lines are at positions : 3016.70, 3017.07, 3018.12, 3019.54, 3020.90, 3022.25, 3023.60, 3024.96, and 3027.29 cm−1. These lines has been also detected in the ACS instrument data and attributed to CO2 magnetic dipole transition (Trokhimovskiy et al., 2020). Further analysis shall be done to compare both NOMAD AND ACS data. Solar lines are never appearing in the sources. They are self-corrected by the calibration since we don’t use a reference solar spectra but the solar observation during the transit when the tangent altitude is so high that there is no martian atmosphere (typically > 200 km). None of the analyzed orders presents sources related to CH4.