# Search for quantum decoherence in neutrino oscillations with six detection units of KM3NeT/ORCA

S Aiello, A Albert, A.R Alhebsi, M Alshamsi, S Alves Garre, A Ambrosone, F Ameli, M Andre, L Aphecetche, M Ardid, et al.

► To cite this version:

S Aiello, A Albert, A.R Alhebsi, M Alshamsi, S Alves Garre, et al. Search for quantum decoherence in neutrino oscillations with six detection units of KM3NeT/ORCA. Journal of Cosmology and Astroparticle Physics, 2025, 03, pp.039. 10.1088/1475-7516/2025/03/039. hal-04726675

HAL Id: hal-04726675

https://hal.science/hal-04726675v1

Submitted on 8 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

PAPER • OPEN ACCESS

# Search for quantum decoherence in neutrino oscillations with six detection units of KM3NeT/ORCA

To cite this article: S. Aiello et al JCAP03(2025)039

View the [article online](#) for updates and enhancements.

## You may also like

- Multi-messenger Observations of a Binary Neutron Star Merger

B. P. Abbott, R. Abbott, T. D. Abbott et al.

- Sensor response and radiation damage effects for 3D pixels in the ATLAS IBL Detector

G. Aad, E. Aakvaag, B. Abbott et al.

- Fast b-tagging at the high-level trigger of the ATLAS experiment in LHC Run 3

G. Aad, B. Abbott, K. Abeling et al.
# Search for quantum decoherence in neutrino oscillations with six detection units of KM3NeT/ORCA

<figure><img src="image_3.png" /></figure>

## The KM3NeT collaboration

E-mail: km3net-pc@km3net.de, Nadja.Lessing@ific.uv.es

**ABSTRACT:** Neutrinos described as an open quantum system may interact with the environment which introduces stochastic perturbations to their quantum phase. This mechanism leads to a loss of coherence along the propagation of the neutrino – a phenomenon commonly referred to as *decoherence* – and ultimately, to a modification of the oscillation probabilities. Fluctuations in space-time, as envisaged by various theories of quantum gravity, are a potential candidate for a decoherence-inducing environment. Consequently, the search for decoherence provides a rare opportunity to investigate quantum gravitational effects which are usually beyond the reach of current experiments. In this work, quantum decoherence effects are searched for in neutrino data collected by the KM3NeT/ORCA detector from January 2020 to November 2021. The analysis focuses on atmospheric neutrinos within the energy range of a few GeV to 100 GeV. Adopting the open quantum system framework, decoherence is described in a phenomenological manner with the strength of the effect given by the parameters $\Gamma_{21}$ and $\Gamma_{31}$. Following previous studies, a dependence of the type $\Gamma_{ij} \propto (E/E_0)^n$ on the neutrino energy is assumed and the cases $n = -2, -1$ are explored. No significant deviation with respect to the standard oscillation hypothesis is observed. Therefore, 90% CL upper limits are estimated as $\Gamma_{21} < 4.6 \cdot 10^{-21}$ GeV and $\Gamma_{31} < 8.4 \cdot 10^{-21}$ GeV for $n = -2$ and $\Gamma_{21} < 1.9 \cdot 10^{-22}$ GeV and $\Gamma_{31} < 2.7 \cdot 10^{-22}$ GeV for $n = -1$, respectively.

**KEYWORDS:** neutrino experiments, Frequentist statistics

ARXIV ePRINT: 2410.01388

<figure><img src="image_4.png" /></figure>
# Contents

<table><tr><td>1</td><td>Introduction</td><td>1</td></tr><tr><td>2</td><td>Theory of neutrino decoherence</td><td>2</td></tr><tr><td>3</td><td>The KM3NeT infrastructure</td><td>7</td></tr><tr><td>4</td><td>Analysis methods</td><td>7</td></tr><tr><td>4.1</td><td>Event selection and classification</td><td>7</td></tr><tr><td>4.2</td><td>Statistical methods</td><td>8</td></tr><tr><td>5</td><td>Results</td><td>10</td></tr><tr><td>5.1</td><td>Likelihood ratio scans</td><td>10</td></tr><tr><td>5.2</td><td>Upper limits and comparison to previous studies</td><td>11</td></tr><tr><td>5.3</td><td>Signature of decoherence at 95% CL</td><td>13</td></tr><tr><td>6</td><td>Conclusions</td><td>13</td></tr><tr><td></td><td>The KM3NeT collaboration</td><td>18</td></tr></table>

## 1 Introduction

The discovery of neutrino oscillations [1, 2] is one of the first steps towards physics beyond the Standard Model in revealing that neutrinos do have mass, which is not foreseen from the theory. The three-flavour oscillation model has since become well established and the oscillation parameters have been measured by various experiments [3]. Nevertheless, there are unresolved questions remaining in the neutrino sector concerning for example the Dirac or Majorana nature of neutrinos, the possibility of CP violation, and the determination of the neutrino mass ordering. Furthermore, ongoing investigations seek for phenomena that are beyond the predictions of the Standard Model and could modify the oscillation probabilities. These phenomena include neutrino decay, non-standard interactions, and neutrino decoherence.

Neutrino oscillations occur when flavour eigenstates are a coherent superposition of mass eigenstates [4], where each mass eigenstate evolves at a different frequency. As a result, a neutrino initially produced with a specific flavour can be detected with a different flavour after traveling macroscopic distances. The coherency of the neutrino wave is also maintained when neutrinos travel through dense matter. Incoherent inelastic scattering is proportional to $G_F^2$ (where $G_F$ is the Fermi constant) and can safely be neglected. Coherent forward scattering consistently alters the phase velocity of the neutrinos with a term proportional to $G_F$ and to the density of scattering targets, so the interference effect persists.

However, if a neutrino is treated as an open quantum system that interacts with the environment, it can experience stochastic perturbations on its quantum phase, causing the mass eigenstates to lose their coherence. This phenomenon is referred to as decoherence.
Fluctuations in the metric of space-time, as anticipated in quantum gravity models, are often suggested as a potential source of decoherence effects [5, 6]. Furthermore, a microscopic model for gravitationally induced decoherence has been introduced and compared to phenomenological approaches in a recent work [7]. The search for quantum decoherence thus offers a unique opportunity to explore quantum gravity phenomena which are often unobservable with current experimental capabilities.

The signature of neutrino decoherence is a damping of the oscillation amplitude. Several studies [8–11] additionally consider the so-called relaxation effects which act on the non-oscillatory terms as part of the decoherence phenomenology. However, it is important to note that these two effects are distinct phenomena: pure decoherence can only be probed in experiments that are sensitive to the oscillation terms in the probabilities, whereas relaxation effects can also be tested in scenarios where oscillations are averaged out [12].

Pure relaxation effects have been strongly constrained using solar neutrino data from KamLAND [8]. Pure decoherence has been studied with data from the reactor experiments KamLAND, RENO and Daya Bay [13, 14], the accelerator experiments NOvA, MINOS, T2K and OPERA [10, 14–16], as well as using atmospheric neutrino data from Super-Kamiokande [17], IceCube [9, 18], and IceCube/DeepCore [18]. A combined search for decoherence and relaxation effects has been conducted with MINOS, T2K [10] and Ice-Cube [9]. Furthermore, the impact of decoherence on the precision measurement of standard oscillation parameters has been investigated for future experiments like DUNE, T2HK [19], and ESSnuSB [20]. Since the various experiments are sensitive to different energy regions and oscillation parameters each of them can contribute to constrain a distinctive area of the phase space. In this work, decoherence effects are searched for in the data collected with a partial configuration of the KM3NeT/ORCA detector. Following the approach taken in previous studies, a power-law dependence of decoherence effects on the neutrino energy is assumed. In section 2 the decoherence model employed in this study is introduced and the effect of decoherence on the oscillation probabilities is illustrated. An overview of the KM3NeT/ORCA detector is provided in section 3. Information about the data set, the statistical method used to obtain upper limits on the decoherence parameters, and the treatment of systematic uncertainties can be found in section 4. Finally, in section 5 likelihood curves as well as confidence level contours for decoherence parameters are presented and compared to those obtained with data from other experiments.

## 2 Theory of neutrino decoherence

The term decoherence refers to the loss of coherence of the neutrino mass eigenstates due to a coupling of the quantum system to a larger environment. This may lead to non-unitary time evolution of the subsystem and introduce irreversible behaviour [21]. As commonly done for dissipative systems, the time evolution of the neutrino density matrix $\rho$ is described by the Lindblad equation [6, 13, 21–24]

$$ \frac{d\rho(t)}{dt} = -i[H, \rho(t)] + \mathcal{D}[\rho(t)], \qquad (2.1) $$
where $H$ is the Hamiltonian describing standard neutrino oscillations in matter, and the dissipative term $\mathcal{D}[\rho(t)]$ is defined as

$$ \mathcal{D}[\rho(t)] = \frac{1}{2} \sum_{k=1}^{N^2-1} \left( [V_k, \rho(t)V_k^\dagger] + [V_k\rho(t), V_k^\dagger] \right), \qquad (2.2) $$

where $V_k$ are general $N \times N$ complex matrices and $N$ the dimension of the Hilbert space of the subsystem. The first term of eq. (2.1) corresponds to the standard unitary time evolution whereas the second term encodes decoherence effects. For three neutrino families ($N = 3$) the dissipative term can be expanded in the SU(3) basis [6, 9, 24] yielding a general form

$$ \mathcal{D}[\rho(t)] = (D_{\mu\nu}\rho^{\nu})\lambda^{\mu}, \qquad (2.3) $$

where $\lambda^\mu$ are the Gell-Mann matrices, $\rho^\nu$ are coefficients defined by $\rho = \rho_\nu\lambda^\nu$, and $D$ is a $9 \times 9$ symmetric matrix that parameterises decoherence effects. Since the case studied here cannot be irrespective of matter effects, all the above quantities are defined in the effective mass basis. This description implies that decoherence is not only determined by properties of the environment, but generally also depends on the matter potential.

Following previous analyses the following physical conditions are imposed:

* *Complete positivity* ensures that probabilities of the whole system remain positive at all times [25]. This requirement places restrictions on the elements of $D$, rendering them no longer independent [14, 24, 25].

* *Probability conservation* is satisfied if $D_{\mu 0} = D_{0\nu} = 0$ [13, 24, 26].

* *Energy conservation* of the subsystem is required even in the presence of matter [18, 24].

The condition of energy conservation implies that relaxation effects which produce a damping in the non-oscillatory terms of the oscillation probabilities are disregarded. This is also justified by the expectation that the time scale for relaxation effects is much larger than the time scale for pure decoherence [27]. Furthermore, modifications of the non-oscillatory terms are strongly constrained from solar neutrinos [8] which have a significantly larger baseline than the atmospheric neutrinos detected by KM3NeT.<fnref n="1" />

Taking the above into account, decoherence effects can be parameterised by [13, 14, 20, 24]

$$ D = -\mathrm{diag}(0, \Gamma_{21}, \Gamma_{21}, 0, \Gamma_{31}, \Gamma_{31}, \Gamma_{32}, \Gamma_{32}, 0). \qquad (2.4) $$

The validity of this parametrisation in the effective mass basis has been questioned in [28]. However, the authors of [24] show that eq. 2.4 remains valid in both vacuum and in matter, provided no relaxation effects are taken into account. In contrast to the assumptions made in [28], the decoherence effects described by our parametrization depend on the matter density.

<footnote n="1">¹Effects on the oscillatory terms cannot be bound with solar neutrinos since oscillations are averaged out [12, 13].</footnote>
The decoherence parameters $\Gamma_{ij}$ are related by [23, 24]

$$ \Gamma_{21} = 2|\vec{a}_3|^2 \ge 0, \qquad (2.5) $$

$$ \Gamma_{31} = \frac{1}{2} |\vec{a}_3 + \vec{a}_8|^2 \ge 0, \qquad (2.6) $$

$$ \Gamma_{32} = \frac{1}{2} |\vec{a}_3 - \vec{a}_8|^2 \ge 0, \qquad (2.7) $$

where the $\vec{a}_p$ are defined by the expansion of the Lindblad master equation in SU(3) with

$$ \vec{a}_p = (a_p^{(1)}, \dots, a_p^{(8)}) \in \mathbb{R}^8, \qquad (2.8) $$

$$ V_k = a_p^{(k)} \lambda^p. \qquad (2.9) $$

Assuming that the angle between $\vec{a}_3$ and $\vec{a}_8$ is zero (as in [18, 24]), the following relation can be derived:

$$ \Gamma_{32} = \Gamma_{31} + \Gamma_{21} - 2\sqrt{\Gamma_{31}\Gamma_{21}}, \qquad (2.10) $$

resulting in two independent parameters $\Gamma_{21}$ and $\Gamma_{31}$.

In the effective mass basis with matter effects included, standard neutrino oscillations are described by a diagonal Hamiltonian

$$ H = \frac{1}{2E} \begin{pmatrix} 0 & 0 & 0 \\ 0 & \Delta \tilde{m}_{21}^2 & 0 \\ 0 & 0 & \Delta \tilde{m}_{31}^2 \end{pmatrix}, \qquad (2.11) $$

where $E$ is the neutrino energy and $\Delta \tilde{m}_{ij}^2$ are the effective squared mass differences. The dissipative term $\mathcal{D}[\rho(t)]$ can be written in terms of the decoherence parameters as [6, 16]

$$ \mathcal{D}[\rho(t)] = - \begin{pmatrix} 0 & \Gamma_{21}\rho_{12}(0) & \Gamma_{31}\rho_{13}(0) \\ \Gamma_{21}\rho_{21}(0) & 0 & \Gamma_{32}\rho_{23}(0) \\ \Gamma_{31}\rho_{31}(0) & \Gamma_{32}\rho_{32}(0) & 0 \end{pmatrix}. \qquad (2.12) $$

Consequently, the time evolution of the neutrino density matrix as given by eq. (2.1) has the solution [13, 16]

$$ \rho(t) = \begin{pmatrix} \rho_{11}(0) & \rho_{12}(0)e^{-(\Gamma_{21}+i\tilde{\Delta}_{21})*t} & \rho_{13}(0)e^{-(\Gamma_{31}+i\tilde{\Delta}_{31})*t} \\ \rho_{21}(0)e^{-(\Gamma_{21}+i\tilde{\Delta}_{21})t} & \rho_{22}(0) & \rho_{23}(0)e^{-(\Gamma_{32}+i\tilde{\Delta}_{32})*t} \\ \rho_{31}(0)e^{-(\Gamma_{31}+i\tilde{\Delta}_{31})t} & \rho_{32}(0)e^{-(\Gamma_{32}+i\tilde{\Delta}_{32})t} & \rho_{33}(0) \end{pmatrix}, \qquad (2.13) $$

where $\tilde{\Delta}_{ij} = \frac{\Delta \tilde{m}_{ij}^2}{2E}$. The solution 2.13 is valid for constant matter density. In our approximation, matter inside the Earth is described by a sequence of layers, each with constant density. Eq. 2.13 is used in each layer separately and the stepwise solutions are assembled with appropriate boundary conditions, as in [18]. Due to the condition of energy conservation, the dissipator is always diagonal in the matter basis. The only difference with respect to standard oscillations is the presence of damping terms $e^{-\Gamma_{ij}t}$.
In general, decoherence effects may depend on the neutrino energy $E$. Following previous studies [9, 14, 18] a power-law dependency is assumed

$$ \Gamma_{ij}(E) = \Gamma_{ij}(E_0) \left(\frac{E}{E_0}\right)^n, \qquad (2.14) $$

where $E_0 = 1$ GeV is a reference energy and $\Gamma_{ij}(E_0)$ determines the strength of decoherence effects at $E_0$. The index $n$ is typically set to integer values $n = [-2, -1, 0, 1, 2]$, each motivated by different theoretical considerations. The $n = -2$ model can be linked to decoherence resulting from light-cone fluctuations [5] and gravitationally induced decoherence<fnref n="2" /> [7]. The $n = -1$ model is the only case which does not violate Lorentz invariance since the damping term follows the same $L/E$ dependence as the oscillation term.

In this work, limits are derived for $n = -2, -1$ as the KM3NeT/ORCA detector is especially sensitive to modifications of the oscillation probabilities at low energies. The current most stringent bounds on the decoherence model considered here are obtained with data from KamLAND, RENO, and T2K [14] where their models C, D and E respect the relation between the decoherence parameters stated in eq. (2.10). For $n = 0$ several works find comparable limits [9, 14, 18] whereas for $n = 1, 2$ the best current limits are given by IceCube [9].

Oscillation probabilities are calculated with the open source software OscProb [29]. To account for matter effects the Earth is described by 15 layers of constant matter density based on the Preliminary Reference Earth Model [30]. Survival probabilities of $\nu_\mu$ and $\bar{\nu}_\mu$ which traverse the Earth vertically are shown in figure 1 for standard oscillations as well as three representative decoherence scenarios. For the purpose of illustration the values $n = -1$ and $\Gamma_{ij} = 10^{-22}$ GeV are set for the active decoherence parameters while the remaining decoherence parameter is set to zero in each case. This choice of values is in the order of the 90% CL limit obtained in this work. The damping of the oscillation amplitude in the decoherence scenarios is visible up to ~30 GeV, which is well within the energy range that KM3NeT/ORCA is sensitive to. Since matter effects are included in the model, the impact of the $\Gamma_{ij}$ parameters on the oscillation probabilities depends on the mass ordering and whether neutrinos or antineutrinos are considered. KM3NeT/ORCA detects both neutrinos and antineutrinos, so it is expected to have sensitivity to decoherence for both mass orderings, even if only two of the $\Gamma_{ij}$ are non-zero. In figure 2, the difference between $\nu_\mu$ survival probabilities assuming standard oscillations and assuming decoherence (case $\Gamma_{21} = \Gamma_{31}$) is shown as a function of the neutrino energy and zenith angle. The impact of decoherence is visible in the whole range of the zenith angle which means that neutrinos from all directions can contribute to the sensitivity of the detector. The standard oscillation parameters are based on the NuFit 5.0 result including Super-Kamiokande data [3], where the mixing angle has a value of $\theta_{23} = 49.2^\circ$ for normal ordering. Since decoherence causes a damping of the oscillation amplitude, its effect can partially be replicated or compensated for by changing the value of the mixing angle. The sensitivity to decoherence is highest for maximal mixing ($\theta_{23} = 45^\circ$) because the difference between the oscillation probability assuming standard oscillations, $P_{\text{std}}$, and assuming decoherence, $P_{\text{deco}}$, is largest, making it easier to distinguish between the two.

<footnote n="2">²An exact agreement of a microscopic model and the phenomenological approach was found for vacuum oscillations and $\Gamma_{ij} \propto (\Delta m_{ij}^2)^2$ [7].</footnote>
<figure><img src="image_5.png" /><figcaption>**Figure 1.** $\nu_\mu$ (upper) and $\bar{\nu}_\mu$ (lower) survival probabilities with normal (left) and inverted (right) ordering for standard oscillations as well as three representative decoherence scenarios, $\Gamma_{21} = \Gamma_{32}$, $\Gamma_{31} = \Gamma_{32}$, and $\Gamma_{21} = \Gamma_{31}$. Decoherence causes a damping of the oscillation amplitude which depends on the mass ordering and whether neutrinos or antineutrinos are considered. The energy dependence shown here corresponds to $n = -1$ and the active decoherence parameters are set to $\Gamma_{ij} = 10^{-22}$ GeV.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Figure 2.** Difference in the $\nu_\mu$ survival probabilities between standard neutrino oscillations, $P_{\text{std}}$, and assuming decoherence, $P_{\text{deco}}$, (case $\Gamma_{21} = \Gamma_{31} = 10^{-22}$ GeV) as a function of the neutrino energy and zenith angle for $n = -2$ (left) and $n = -1$ (right). The decoherence signal is visible over the whole range of the zenith angle.</figcaption></figure>
# 3 The KM3NeT infrastructure

The KM3NeT research infrastructure comprises two water Cherenkov detectors, referred to as KM3NeT/ARCA (Astroparticle Research with Cosmics in the Abyss) and KM3NeT/ORCA (Oscillation Research with Cosmics in the Abyss), both currently under construction in the Mediterranean Sea [31]. ARCA is located at a depth of 3500 m about 100 km offshore from Portopalo di Capo Passero, Sicily, Italy while ORCA is being built about 40 km offshore from Toulon, France at a depth of 2450 m. The main purpose of ARCA is the search for cosmic neutrinos in the TeV to PeV energy range, whereas ORCA is optimized for the measurement of neutrino oscillation properties using atmospheric neutrinos. Even though the construction is ongoing, KM3NeT has already started taking data using a fraction of the envisaged detector sizes. This is possible due to the modular detector design: ARCA and ORCA consist of arrays of vertical detection units, each housing 18 digital optical modules [32]. These modules are pressure-resistant glass spheres containing 31 photomultiplier tubes (PMTs) each [33], along with their associated readout electronics. The PMTs collect Cherenkov light induced by relativistic charged particles which emerge from neutrino interactions. PMT pulses surpassing a threshold discriminator are digitised and characterised by their starting time and duration. This information is used to reconstruct the initial neutrino direction and energy.

For ORCA the horizontal spacing between detection units (~ 20 m) and the vertical distance of digital optical modules (~ 9 m) allows for the detection of atmospheric neutrinos with energies as low as a few GeV. This configuration is optimised to enhance the sensitivity to the mass ordering [34] and oscillation parameters. Consequently, ORCA can also be used to search for effects which cause a deviation from the oscillation pattern predicted by standard oscillations.

# 4 Analysis methods

This work uses data collected with a partial configuration of the KM3NeT/ORCA detector comprising six detection units and referred to as ORCA6. In this section, the event selection and classification as well as the statistical methods of the analysis are explained [35].

## 4.1 Event selection and classification

The data used in this analysis were taken between January 2020 and November 2021. Only periods characterised by high stability in environmental conditions and data acquisition are utilised. The total livetime is 510 days, corresponding to an exposure of 433 kton-years with ORCA6. Two event topologies are considered in the event reconstruction: *track-like* and *shower-like*. Track-like signatures originate from $\nu_\mu$ charged-current (CC) and $\nu_\tau$ CC interactions which produce a muon in the final state. Events with an electromagnetic or hadronic shower are induced by $\nu_e$ CC and $\nu_\tau$ CC interactions, as well as neutral-current (NC) interactions of all neutrino flavours. Cuts based on the reconstruction quality reject pure noise events which originate mainly from $^{40}$K decays. The analysis is restricted to up-going events ($\cos(\theta_Z) < 0$) in order to reject atmospheric muons which are a large source of background. A boosted decision tree (BDT) trained on the features of the reconstruction algorithm is employed to discriminate between neutrino-induced signals and poorly reconstructed
atmospheric muons. The final sample comprises 5828 observed events with an atmospheric muon contamination below 2%.

A second BDT is used to divide the sample into three distinct classes based on the event topology and reconstruction quality: i) a high-purity track-like class with negligible atmospheric muon contamination and an estimated $\nu_\mu$ CC purity of 95%, ii) a low-purity track-like class with 4% muon contamination and 90% $\nu_\mu$ CC purity, and iii) a shower-like class. The separation of the track-like events into a high- and a low-purity class improves the sensitivity to standard oscillation parameters due to the isolation of events with a good angular resolution in the high-purity class. The track classes contain events with reconstructed energies between 2 GeV and 100 GeV, whereas the shower class ranges from 2 GeV to 1 TeV.

The data are represented in 2D event histograms of the reconstructed energy, $E_{\text{reco}}$, and zenith angle, $\theta_{Z,\text{reco}}$, with a binning scheme that ensures an expectation of at least two events per bin. The detector resolution is described by a response matrix $R(E_{\text{true}}, \theta_{Z,\text{true}}, E_{\text{reco}}, \theta_{Z,\text{reco}})$ which is obtained from reconstructed Monte Carlo (MC) events. The response matrix represents the detection efficiency and reconstruction probability for each bin of true energy, $E_{\text{true}}$, and zenith angle, $\theta_{Z,\text{true}}$, for each class and flavour. This results in an effective mass of the detector which depends on the interaction type as well as the neutrino flavour and energy. More information on the event selection, reconstruction, and classification can be found in [35] which uses the same data set as this analysis.

## 4.2 Statistical methods

A likelihood minimisation technique is used to compare 2D reconstructed event histograms to the Monte Carlo expectation, where the values of the oscillation parameters (see table 1) are taken from NuFit 5.0 [3]. The negative log-likelihood ratio [35]

$$ -2 \log(\mathcal{L}) = 2 \sum_i \left[ (\beta_i N_i^{\text{mod}} - N_i^{\text{dat}}) + N_i^{\text{dat}} \log \left( \frac{N_i^{\text{dat}}}{\beta_i N_i^{\text{mod}}} \right) \right] + \frac{(\beta_i - 1)^2}{\sigma_{\beta i}^2} + \sum_k \left( \frac{\epsilon_k - \langle \epsilon_k \rangle}{\sigma_{\epsilon k}} \right)^2, \quad (4.1) $$

is minimised over $\theta_{23}$ and $\Delta m_{31}^2$, as well as a set of nuisance parameters $\epsilon_k$ related to model uncertainties. Since ORCA6 is not sensitive to $\theta_{12}$, $\theta_{13}$, $\Delta m_{21}^2$, and $\delta_{\text{CP}}$ these parameters are fixed in the minimisation. The first term in eq. (4.1) is derived assuming Poisson distributed reconstructed events with $N_i^{\text{mod}}$ and $N_i^{\text{dat}}$ representing the number of expected events in the model and the number of observed events in data, respectively, per bin of reconstructed energy and zenith angle. The normally distributed $\beta_i$ account for uncertainties due to limited MC statistics following a Barlow and Beeston light method [35, 36]. The third term in eq. (4.1) constrains model uncertainties by measuring the discrepancy between the observed values $\epsilon_k$ and the expected values $\langle \epsilon_k \rangle$ of the nuisance parameters in units of their standard deviation $\sigma_{\epsilon k}$. Some nuisance parameters are restricted by a prior while others may vary without constraint around their nominal value (see table 2).

The model uncertainties can be categorised into three groups:

1. Uncertainties in the atmospheric neutrino flux are addressed by nuisance parameters which vary the ratios between muon and electron neutrinos as well as neutrinos and antineutrinos. The proportion of horizontal and up-going neutrinos can be adjusted by
<table><caption>**Table 1.** Oscillation parameters taken from NuFit 5.0 [3].</caption><thead><tr><th>Parameter</th><th>Value NO</th><th>Value IO</th><th>Treatment</th></tr></thead><tbody><tr><td>Δm<sup>2</sup><sub>31</sub> [eV<sup>2</sup>]</td><td>2.517 · 10<sup>-3</sup></td><td>-2.424 · 10<sup>-3</sup></td><td>free</td></tr><tr><td>Δm<sup>2</sup><sub>21</sub> [eV<sup>2</sup>]</td><td>7.42 · 10<sup>-5</sup></td><td>7.42 · 10<sup>-5</sup></td><td>fixed</td></tr><tr><td>θ<sub>12</sub> [°]</td><td>33.44</td><td>33.45</td><td>fixed</td></tr><tr><td>θ<sub>13</sub> [°]</td><td>8.57</td><td>8.60</td><td>fixed</td></tr><tr><td>θ<sub>23</sub> [°]</td><td>49.2</td><td>49.3</td><td>free</td></tr><tr><td>δ<sub>CP</sub> [°]</td><td>197</td><td>282</td><td>fixed</td></tr></tbody></table>

<table><caption>**Table 2.** Nuisance parameters with their uncertainties.</caption><thead><tr><th>Parameter</th><th>Uncertainty</th></tr></thead><tbody><tr><td>(ν<sub>μ</sub>+ν̄<sub>μ</sub>)/(ν<sub>e</sub>+ν̄<sub>e</sub>)</td><td>±2%</td></tr><tr><td>ν<sub>e</sub>/ν̄<sub>e</sub></td><td>±7%</td></tr><tr><td>ν<sub>μ</sub>/ν̄<sub>μ</sub></td><td>±5%</td></tr><tr><td>ν<sub>hor</sub>/ν<sub>up</sub></td><td>±2%</td></tr><tr><td>Spectral index</td><td>±10%</td></tr><tr><td>f<sub>all</sub></td><td>unconstrained</td></tr><tr><td>f<sub>S</sub></td><td>unconstrained</td></tr><tr><td>f<sub>HPT</sub></td><td>unconstrained</td></tr><tr><td>f<sub>NC</sub></td><td>±20%</td></tr><tr><td>f<sub>τCC</sub></td><td>±20%</td></tr><tr><td>f<sub>HE</sub></td><td>±50%</td></tr><tr><td>f<sub>μ</sub></td><td>unconstrained</td></tr><tr><td>E<sub>s</sub></td><td>±9%</td></tr></tbody></table>

varying the shape of the flux in dependence of the zenith angle. The variation of the spectral index produces a tilt in the energy distribution of the flux.

2. Uncertainties in the neutrino cross section and selection efficiency are taken into account by adjusting the total number of neutrinos as well as the relative number of neutrino events in the shower and high-purity track class. Additionally, scaling factors are applied to the number of NC events and the number of $ν_τ$ CC events. A relative normalisation accounts for approximations used in the light simulation of high energy neutrino events ($E_{true} > 500$ GeV for CC events, $E_{true} > 100$ GeV for NC events). The absolute number of atmospheric muons may vary without constraint.

3. Uncertainties in the optical properties of water, PMT efficiencies, and the hadronic shower propagation are modelled by an absolute energy scale of the detector which may introduce a shift in the true energy of the response function.

A more detailed description of the systematic parameters and their modelling can be found in [35].
<figure><img src="image_7.png" /><figcaption>**Figure 3.** Log-likelihood ratio with respect to the global best fit for $n = -2$ (left) and $n = -1$ (right) as a function of the decoherence parameters $\Gamma_{21}$ (blue) and $\Gamma_{31}$ (green). The solid lines were obtained fitting both, NO and IO and keeping the overall best fit. The dashed line was obtained assuming NO which demonstrates that IO is preferred for large values of $\Gamma_{21}$.</figcaption></figure>

# 5 Results

No significant deviation was found with respect to standard oscillations as the best-fit values of $\Gamma_{21}$ and $\Gamma_{31}$ are consistent with zero. In order to derive upper limits, confidence intervals are obtained from a scan of the log-likelihood ratio $-2 \log (\mathcal{L}_{\text{deco}}/\mathcal{L}_{\text{bf}}) = -2\Delta \log \mathcal{L}$, where $\mathcal{L}_{\text{deco}}$ is computed for a set of points in the decoherence phase space and $\mathcal{L}_{\text{bf}}$ is the likelihood at the global best fit.

## 5.1 Likelihood ratio scans

The log-likelihood ratio $-2\Delta \log \mathcal{L}$ as a function of the decoherence parameters can be seen in figure 3 for both energy dependencies $n = -2$ (left) and $n = -1$ (right). The solid blue curve corresponds to a scan over $\Gamma_{21}$ and the green curve to a scan over $\Gamma_{31}$. In each scan, the remaining decoherence parameter is left free in the fit which allows for non-zero values of all three $\Gamma_{ij}$. Each minimisation uses eight starting points, $\theta_{23} = \{40^\circ, 50^\circ\}$, $E_s = \{0.95, 1.05\}$ and $\Delta m_{31}^2 = \{-2.428 \cdot 10^{-3}, 2.517 \cdot 10^{-3}\} \text{eV}^2$, where $\theta_{23}$ is restricted to the lower/upper octant, the energy scale to below/above one, and $\Delta m_{31}^2$ to the corresponding mass ordering. The overall best fit is represented by solid lines whereas the dashed line additionally shows the result for normal ordering (NO). For $\Gamma_{31}$ normal ordering always gives the smaller log-likelihood ratio. Regarding $\Gamma_{21}$, normal ordering is preferred for small values, and inverted ordering (IO) is preferred for large values.

Confidence level contours which simultaneously constrain $\Gamma_{21}$ and $\Gamma_{31}$ are shown in figure 4 for both energy dependencies. The upper right part of the parameter space is excluded at the corresponding CL. The log-likelihood ratio is small for $\Gamma_{21} = \Gamma_{31}$ since from eq. (2.10) $\Gamma_{21} = \Gamma_{31} \Rightarrow \Gamma_{32} = 0$, implying that $\Gamma_{32}$ does not contribute to decoherence effects. The shape of the contours can be understood from figure 5 which shows the contour at 90% CL assuming NO (orange, dashed) or IO (purple, dotted) respectively, along with the
<figure><img src="image_8.png" /><figcaption>**Figure 4.** Confidence level contours of $\Gamma_{31}$ and $\Gamma_{21}$ for $n = -2$ (left) and $n = -1$ (right). The upper right part of the parameter space is excluded for each model at the corresponding CL. For $\Gamma_{21} = \Gamma_{31}$ the log-likelihood ratio is small as $\Gamma_{32}$ does not contribute to decoherence effects.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Figure 5.** 90% CL contours of $\Gamma_{31}$ and $\Gamma_{21}$ assuming NO (orange dashed), assuming IO (purple dotted), and fitting both mass orderings (black solid) for $n = -2$ (left) and $n = -1$ (right). IO is preferred in the minimisation when decoherence effects are dominated by $\Gamma_{21}$.</figcaption></figure>

best fit ordering (black, solid). As in the one-dimensional profiles, IO is preferred when decoherence effects are dominated by $\Gamma_{21}$.

## 5.2 Upper limits and comparison to previous studies

Upper limits are reported in table 3 for both energy dependencies $n = -2, -1$ as well as both mass orderings at the 90% CL and 95% CL, assuming that the log-likelihood ratio asymptotically approaches a $\chi^2$-distribution. The limits on the scenario $\Gamma_{21} = \Gamma_{31}$ are obtained from the confidence level contours and reported for comparison with previous studies.
<table><caption>**Table 3.** Upper limits on the decoherence parameters at 90% CL and 95% CL for the energy dependencies $n = -2, -1$ for NO and IO. The more conservative limit for both mass orderings is highlighted in bold.</caption><thead><tr><th></th><th colspan="4">Upper limits [GeV]</th></tr><tr><th></th><th colspan="2">n = -2</th><th colspan="2">n = -1</th></tr><tr><th>ORCA6, 90%CL</th><th>NO</th><th>IO</th><th>NO</th><th>IO</th></tr></thead><tbody><tr><th>Γ<sub>21</sub></th><td>2.8 · 10<sup>-21</sup></td><td><b>4.6 · 10<sup>-21</sup></b></td><td>1.1 · 10<sup>-22</sup></td><td><b>1.9 · 10<sup>-22</sup></b></td></tr><tr><th>Γ<sub>31</sub></th><td><b>8.4 · 10<sup>-21</sup></b></td><td>2.2 · 10<sup>-21</sup></td><td><b>2.7 · 10<sup>-22</sup></b></td><td>0.8 · 10<sup>-22</sup></td></tr><tr><th>Γ<sub>21</sub> = Γ<sub>31</sub></th><td><b>4.1 · 10<sup>-21</sup></b></td><td>2.9 · 10<sup>-21</sup></td><td><b>1.8 · 10<sup>-22</sup></b></td><td>1.1 · 10<sup>-22</sup></td></tr><tr><th>ORCA6, 95%CL</th><th>NO</th><th>IO</th><th>NO</th><th>IO</th></tr><tr><th>Γ<sub>21</sub></th><td>3.7 · 10<sup>-21</sup></td><td><b>6.9 · 10<sup>-21</sup></b></td><td>1.6 · 10<sup>-22</sup></td><td><b>3.0 · 10<sup>-22</sup></b></td></tr><tr><th>Γ<sub>31</sub></th><td><b>11.7 · 10<sup>-21</sup></b></td><td>3.2 · 10<sup>-21</sup></td><td><b>4.2 · 10<sup>-22</sup></b></td><td>1.3 · 10<sup>-22</sup></td></tr><tr><th>Γ<sub>21</sub> = Γ<sub>31</sub></th><td><b>5.2 · 10<sup>-21</sup></b></td><td>3.6 · 10<sup>-21</sup></td><td><b>2.3 · 10<sup>-22</sup></b></td><td>1.4 · 10<sup>-22</sup></td></tr></tbody></table>

<table><caption>**Table 4.** Upper limits at 90% CL for three representative cases of decoherence using data from KamLAND (KL), RENO (R), and T2K [14] and limits at 95% CL using three years of DeepCore data [18]. The limits reported for DeepCore display the same dependency on the mass ordering as observed for ORCA6. In [14] no distinction is made between the orderings.</caption><thead><tr><th></th><th colspan="4">Upper limits [GeV]</th></tr><tr><th></th><th colspan="2">n = -2</th><th colspan="2">n = -1</th></tr></thead><tbody><tr><th>Reported in [14], 90% CL</th><td></td><td></td><td></td><td></td></tr><tr><th>Γ<sub>21</sub> = Γ<sub>32</sub></th><td>7.9 · 10<sup>-27</sup> (KL)</td><td></td><td>1.8 · 10<sup>-24</sup> (KL)</td><td></td></tr><tr><th>Γ<sub>31</sub> = Γ<sub>32</sub></th><td>6.9 · 10<sup>-25</sup> (R)</td><td></td><td>2.1 · 10<sup>-23</sup> (T2K)</td><td></td></tr><tr><th>Γ<sub>21</sub> = Γ<sub>31</sub></th><td>7.9 · 10<sup>-27</sup> (KL)</td><td></td><td>1.8 · 10<sup>-24</sup> (KL)</td><td></td></tr><tr><th>Reported in [18], 95%CL</th><th>NO</th><th>IO</th><th>NO</th><th>IO</th></tr><tr><th>Γ<sub>21</sub> = Γ<sub>32</sub></th><td>7.5 · 10<sup>-21</sup></td><td><b>5.0 · 10<sup>-20</sup></b></td><td>3.5 · 10<sup>-22</sup></td><td><b>2.3 · 10<sup>-21</sup></b></td></tr><tr><th>Γ<sub>31</sub> = Γ<sub>32</sub></th><td><b>4.3 · 10<sup>-20</sup></b></td><td>1.4 · 10<sup>-20</sup></td><td><b>2.0 · 10<sup>-21</sup></b></td><td>5.8 · 10<sup>-22</sup></td></tr><tr><th>Γ<sub>21</sub> = Γ<sub>31</sub></th><td><b>1.2 · 10<sup>-20</sup></b></td><td>8.3 · 10<sup>-21</sup></td><td><b>5.4 · 10<sup>-22</sup></b></td><td>3.6 · 10<sup>-22</sup></td></tr></tbody></table>

A comparison with previous studies is not straightforward due to the diverse approaches that have been employed to set limits on decoherence. Additionally, some previous works [9, 10] neglect the relation between the decoherence parameters outlined in eq. (2.10), allowing for scenarios such as $(\Gamma_{21} \neq 0, \Gamma_{31} = \Gamma_{32} = 0)$ or $(\Gamma_{21} = \Gamma_{31} = \Gamma_{32} \neq 0)$ which are not possible within the framework considered here. As already mentioned, the upper limits provided in this work are determined for individual decoherence parameters $\Gamma_{ij}$ while profiling over the remaining parameter.<fnref n="3" /> However, we found that when computing the log-likelihood scan for $\Gamma_{21}$ ($\Gamma_{31}$) the fitted value of the remaining parameter $\Gamma_{31}$ ($\Gamma_{21}$) tends to be significantly smaller than the parameter of interest. Therefore, limits on $\Gamma_{21}$ ($\Gamma_{31}$) are approximately comparable to limits on $\Gamma_{21} = \Gamma_{32}$ ($\Gamma_{31} = \Gamma_{32}$) reported in [14, 18] and listed in table 4. Strong bounds were obtained using data from KamLAND, RENO and T2K [14]. These

<footnote n="3">³The same approach was taken in a sensitivity study for DUNE [24].</footnote>
<figure><img src="image_10.png" /><figcaption>**Figure 6.** Log-likelihood ratio between decoherence and standard oscillations for $n = -2$ (left) and $n = -1$ (right) as a function of the reconstructed energy and zenith angle for the *high-purity track* class. The decoherence parameters are fixed at the respective 95% CL upper limit with $\Gamma_{21} = \Gamma_{31}$. Negative values (blue) correspond to a better fit of standard oscillations than decoherence.</figcaption></figure>

experiments can access lower energies than ORCA and are therefore especially sensitive to decoherence for negative values of $n$. The limits derived using three years of DeepCore data [18] are comparable to those of ORCA6.

## 5.3 Signature of decoherence at 95% CL

Figure 6 depicts the log-likelihood ratio between the best fit for standard oscillations and decoherence ($\Gamma_{21} = \Gamma_{31}$) at the corresponding 95% CL upper limit for $n = -2, -1$ assuming normal ordering. The ratio is presented as a function of the reconstructed energy and zenith angle for the *high-purity track* class, which has the largest contribution to the sensitivity. Negative values (blue) indicate a preference for standard oscillations. The distribution of bins contributing to the sensitivity follows the expected pattern, reflecting the difference in oscillation probabilities $P_{\text{std}} - P_{\text{deco}}$ shown in figure 2. As anticipated, up-going neutrinos coming from all directions contribute to the sensitivity. Events with $\cos(\theta_Z) < -0.4$ have a larger contribution since the *high-purity track* class contains more events coming from below the detector than events coming from the horizon ($\cos(\theta_Z) = 0$).

To visualise the effect of decoherence, the fit result is transformed into the baseline energy ratio, $L/E$, and normalised to the no oscillations hypothesis. This is shown in figure 7 for standard oscillations as well as the decoherence models $n = -2, -1$ with $\Gamma_{21} = \Gamma_{31}$ fixed at their respective 95% CL upper limit assuming NO. It can be seen that decoherence effects in ORCA6 are mainly visible in two bins at $L/E \approx 10^3 \text{ km/GeV}$.

## 6 Conclusions

Searches for quantum decoherence effects in neutrino oscillations have been carried out using a 433 kton-years data set from the KM3NeT/ORCA detector, in a partial configuration with six detection units. The strength of decoherence effects was parameterised in terms of the
<figure><img src="image_11.png" /><figcaption>**Figure 7.** Ratio of events with respect to the no oscillations hypothesis as a function of $L/E$ for the standard oscillations best fit and the decoherence models $n = -2, -1$ with $\Gamma_{21} = \Gamma_{31}$ fixed at their respective 95% CL upper limit. A deviation with respect to standard oscillations can be seen at $L/E \approx 10^3 \text{ km/GeV}$.</figcaption></figure>

decoherence parameters $\Gamma_{21}, \Gamma_{31}$ assuming a power-law dependency on the neutrino energy $\Gamma_{ij} \propto (E/E_0)^n$. Upper limits on the parameters $\Gamma_{21}$ and $\Gamma_{31}$ have been derived for the cases $n = -2, -1$ showing that the decoherence sensitivity of KM3NeT/ORCA depends on the neutrino mass ordering. Upper limits are therefore reported for both, normal and inverted ordering. The results are comparable to bounds reported for IceCube/DeepCore and display the same dependency on the mass ordering.

## Acknowledgments

The authors thank Kristina Giesel for fruitful discussions about the interpretation of different quantum decoherence models. The authors acknowledge the financial support of the funding agencies: Funds for Scientific Research (FRS-FNRS), Francqui foundation, BAEF foundation. Czech Science Foundation (GAČR 24-12702S); Agence Nationale de la Recherche (contract ANR-15-CE31-0020), Centre National de la Recherche Scientifique (CNRS), Commission Européenne (FEDER fund and Marie Curie Program), LabEx UnivEarthS (ANR-10-LABX-0023 and ANR-18-IDEX-0001), Paris Île-de-France Region, Normandy Region (Alpha, Blue-waves and Neptune), France; Shota Rustaveli National Science Foundation of Georgia (SRNSFG, FR-22-13708), Georgia; This work is part of the MuSES project which has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 Research and Innovation Programme (grant agreement No 101142396). The General Secretariat of Research and Innovation (GSRI), Greece; Istituto Nazionale di Fisica Nucleare (INFN) and Ministero dell'Università e della Ricerca (MUR), through PRIN 2022 program (Grant PAN-THEON 2022E2J4RK, Next Generation EU) and PON R&I program (Avviso n. 424 del 28 febbraio 2018, Progetto PACK-PIR01 00021), Italy; IDMAR project Po-Fsr Sicilian Region
az. 1.5.1; A. De Benedittis, W. Idrissi Ibnsalih, M. Bendahman, A. Nayerhoda, G. Papalashvili, I. C. Rea, A. Simonelli have been supported by the Italian Ministero dell'Università e della Ricerca (MUR), Progetto CIR01 00021 (Avviso n. 2595 del 24 dicembre 2019); KM3NeT4RR MUR Project National Recovery and Resilience Plan (NRRP), Mission 4 Component 2 Investment 3.1, Funded by the European Union – NextGenerationEU, CUP I57G21000040001, Concession Decree MUR No. n. Prot. 123 del 21/06/2022; Ministry of Higher Education, Scientific Research and Innovation, Morocco, and the Arab Fund for Economic and Social Development, Kuwait; Nederlandse organisatie voor Wetenschappelijk Onderzoek (NWO), the Netherlands; Ministry of Research, Innovation and Digitalisation, Romania; Slovak Research and Development Agency under Contract No. APVV-22-0413; Ministry of Education, Research, Development and Youth of the Slovak Republic; MCIN for PID2021-124591NB-C41, -C42, -C43 and PDC2023-145913-I00 funded by MCIN/AEI/10.13039/501100011033 and by "ERDF A way of making Europe", for ASFAE/2022/014 and ASFAE/2022/023 with funding from the EU NextGenerationEU (PRTR-C17.I01) and Generalitat Valenciana, for Grant AST22_6.2 with funding from Consejería de Universidad, Investigación e Innovación and Gobierno de España and European Union — NextGenerationEU, for CSIC-INFRA23013 and for CNS2023-144099, Generalitat Valenciana for CIDEGENT/2018/034, /2019/043, /2020/049, /2021/23, for CIDEIG/2023/20, for CIPROM/2023/51 and for GRISOLIAP/2021/192 and EU for MSC/101025085, Spain; Khalifa University internal grants (ESIG-2023-008 and RIG-2023-070), United Arab Emirates; The European Union's Horizon 2020 Research and Innovation Programme (ChETEC-INFRA — Project no. 101008324).

## References

[1] T. Kajita, Nobel lecture. Discovery of atmospheric neutrino oscillations, Rev. Mod. Phys. 88 (2016) 030501 [INSPIRE].

[2] A.B. McDonald, Nobel lecture. The Sudbury Neutrino Observatory: observation of flavor change for solar neutrinos, Rev. Mod. Phys. 88 (2016) 030502 [INSPIRE].

[3] I. Esteban et al., The fate of hints: updated global analysis of three-flavor neutrino oscillations, JHEP 09 (2020) 178 [arXiv:2007.14792] [INSPIRE].

[4] E.K. Akhmedov and A.Y. Smirnov, Paradoxes of neutrino oscillations, Phys. Atom. Nucl. 72 (2009) 1363 [arXiv:0905.1903] [INSPIRE].

[5] T. Stuttard, Neutrino signals of lightcone fluctuations resulting from fluctuating spacetime, Phys. Rev. D 104 (2021) 056007 [arXiv:2103.15313] [INSPIRE].

[6] T. Stuttard and M. Jensen, Neutrino decoherence from quantum gravitational stochastic perturbations, Phys. Rev. D 102 (2020) 115003 [arXiv:2007.00068] [INSPIRE].

[7] A. Domi et al., Understanding gravitationally induced decoherence parameters in neutrino oscillations using a microscopic quantum mechanical model, JCAP 11 (2024) 006 [arXiv:2403.03106] [INSPIRE].

[8] P.C. de Holanda, Solar neutrino limits on decoherence, JCAP 03 (2020) 012 [arXiv:1909.09504] [INSPIRE].

[9] ICECUBE and ICECUBE collaborations, Search for decoherence from quantum gravity with atmospheric neutrinos, Nature Phys. 20 (2024) 913 [arXiv:2308.00105] [INSPIRE].
[10] A.L.G. Gomes, R.A. Gomes and O.L.G. Peres, *Quantum decoherence and relaxation in long-baseline neutrino data*, JHEP 10 (2023) 035 [arXiv:2001.09250] [INSPIRE].

[11] G.L. Fogli et al., *Probing non-standard decoherence effects with solar and KamLAND neutrinos*, Phys. Rev. D 76 (2007) 033006 [arXiv:0704.2568] [INSPIRE].

[12] M.M. Guzzo, P.C. de Holanda and R.L.N. Oliveira, *Quantum dissipation in a neutrino system propagating in vacuum and in matter*, Nucl. Phys. B 908 (2016) 408 [arXiv:1408.0823] [INSPIRE].

[13] G. Balieiro Gomes, M.M. Guzzo, P.C. de Holanda and R.L.N. Oliveira, *Parameter limits for neutrino oscillation with decoherence in KamLAND*, Phys. Rev. D 95 (2017) 113005 [arXiv:1603.04126] [INSPIRE].

[14] V. De Romeri, C. Giunti, T. Stuttard and C.A. Ternes, *Neutrino oscillation bounds on quantum decoherence*, JHEP 09 (2023) 097 [arXiv:2306.14699] [INSPIRE].

[15] J.A.B. Coelho, W.A. Mann and S.S. Bashar, *Nonmaximal $\theta_{23}$ mixing at NOvA from neutrino decoherence*, Phys. Rev. Lett. 118 (2017) 221801 [arXiv:1702.04738] [INSPIRE].

[16] J.A.B. Coelho and W.A. Mann, *Decoherence, matter effect, and neutrino hierarchy signature in long baseline experiments*, Phys. Rev. D 96 (2017) 093009 [arXiv:1708.05495] [INSPIRE].

[17] E. Lisi, A. Marrone and D. Montanino, *Probing possible decoherence effects in atmospheric neutrino oscillations*, Phys. Rev. Lett. 85 (2000) 1166 [hep-ph/0002053] [INSPIRE].

[18] P. Coloma, J. Lopez-Pavon, I. Martinez-Soler and H. Nunokawa, *Decoherence in neutrino propagation through matter, and bounds from IceCube/DeepCore*, Eur. Phys. J. C 78 (2018) 614 [arXiv:1803.04438] [INSPIRE].

[19] G. Barenboim, A.M. Calatayud-Cadenillas, A.M. Gago and C.A. Ternes, *Quantum decoherence effects on precision measurements at DUNE and T2HK*, Phys. Lett. B 852 (2024) 138626 [arXiv:2402.16395] [INSPIRE].

[20] ESSNUSB collaboration, *Decoherence in neutrino oscillation at the ESSnuSB experiment*, JHEP 08 (2024) 063 [arXiv:2404.17559] [INSPIRE].

[21] G. Lindblad, *On the generators of quantum dynamical semigroups*, Commun. Math. Phys. 48 (1976) 119 [INSPIRE].

[22] L.A. Anchordoqui et al., *Probing Planck scale physics with IceCube*, Phys. Rev. D 72 (2005) 065019 [hep-ph/0506168] [INSPIRE].

[23] R.L.N. Oliveira, *Dissipative effect in long baseline neutrino experiments*, Eur. Phys. J. C 76 (2016) 417 [arXiv:1603.08065] [INSPIRE].

[24] G. Balieiro Gomes et al., *Quantum decoherence effects in neutrino oscillations at DUNE*, Phys. Rev. D 100 (2019) 055023 [arXiv:1805.09818] [INSPIRE].

[25] F. Benatti and R. Floreanini, *Open system approach to neutrino oscillations*, JHEP 02 (2000) 032 [hep-ph/0002221] [INSPIRE].

[26] L. Buoninfante, A. Capolupo, S.M. Giampaolo and G. Lambiase, *Revealing neutrino nature and CPT violation with decoherence effects*, Eur. Phys. J. C 80 (2020) 1009 [arXiv:2001.07580] [INSPIRE].

[27] W.H. Zurek, *Decoherence and the transition from quantum to classical*, Phys. Today 44N10 (1991) 36 [quant-ph/0306072] [INSPIRE].

[28] J. Carpio, E. Massoni and A.M. Gago, *Revisiting quantum decoherence for neutrino oscillations in matter with constant density*, Phys. Rev. D 97 (2018) 115017 [arXiv:1711.03680] [INSPIRE].
[29] J. Coelho et al., joaoabcoelho/oscprob: v1.6.1, Zenodo, June 2023.

[30] A.M. Dziewonski and D.L. Anderson, Preliminary reference earth model, Phys. Earth Planet. Interiors 25 (1981) 297 [INSPIRE].

[31] KM3NET collaboration, Letter of intent for KM3NeT 2.0, J. Phys. G 43 (2016) 084001 [arXiv:1601.07459] [INSPIRE].

[32] KM3NET collaboration, The KM3NeT multi-PMT optical module, 2022 JINST 17 P07038 [arXiv:2203.10048] [INSPIRE].

[33] KM3NET collaboration, Characterisation of an improved 3" Hamamatsu photomultiplier for the KM3NeT neutrino telescope, J. Phys. Conf. Ser. 2429 (2023) 012031 [INSPIRE].

[34] KM3NET collaboration, Determining the neutrino mass ordering and oscillation parameters with KM3NeT/ORCA, Eur. Phys. J. C 82 (2022) 26 [arXiv:2103.09885] [INSPIRE].

[35] KM3NET collaboration, Measurement of neutrino oscillation parameters with the first six detection units of KM3NeT/ORCA, JHEP 10 (2024) 206 [arXiv:2408.07015] [INSPIRE].

[36] R.J. Barlow and C. Beeston, Fitting using finite Monte Carlo samples, Comput. Phys. Commun. 77 (1993) 219 [INSPIRE].
The KM3NeT collaboration

S. Aiello<sup>a</sup>, A. Albert<sup>b,bb</sup>, A. R. Alhebsi<sup>c</sup>, M. Alshamsi<sup>d</sup>, S. Alves Garre<sup>e</sup>, A. Ambrosone<sup>g,f</sup>, F. Ameli<sup>h</sup>, M. Andre<sup>i</sup>, L. Aphecetche<sup>j</sup>, M. Ardid<sup>k</sup>, S. Ardid<sup>k</sup>, H. Atmani<sup>l</sup>, J. Aublin<sup>m</sup>, F. Badaracco<sup>o,n</sup>, L. Bailly-Salins<sup>p</sup>, Z. Bardačová<sup>r,q</sup>, B. Baret<sup>m</sup>, A. Bariego-Quintana<sup>e</sup>, Y. Becherini<sup>m</sup>, M. Bendahman<sup>f</sup>, F. Benfenati<sup>t,s</sup>, M. Benhassi<sup>u,f</sup>, M. Bennani<sup>v</sup>, D. M. Benoit<sup>w</sup>, E. Berbee<sup>x</sup>, V. Bertin<sup>d</sup>, S. Biagi<sup>y</sup>, M. Boettcher<sup>z</sup>, D. Bonanno<sup>y</sup>, A. B. Bouasla<sup>bc</sup>, J. Boumaaza<sup>l</sup>, M. Bouta<sup>d</sup>, M. Bouwhuis<sup>x</sup>, C. Bozza<sup>aa,f</sup>, R. M. Bozza<sup>g,f</sup>, H.Brânzaş<sup>ab</sup>, F. Bretaudieu<sup>j</sup>, M. Breuhaus<sup>d</sup>, R. Bruijn<sup>ac,x</sup>, J. Brunner<sup>d</sup>, R. Bruno<sup>a</sup>, E. Buis<sup>ad,x</sup>, R. Buompane<sup>u,f</sup>, J. Busto<sup>d</sup>, B. Caiffi<sup>o</sup>, D. Calvo<sup>e</sup>, A. Capone<sup>h,ae</sup>, F. Carenini<sup>t,s</sup>, V. Carretero<sup>ac,x</sup>, T. Cartraud<sup>m</sup>, P. Castaldi<sup>af,s</sup>, V. Cecchini<sup>e</sup>, S. Celli<sup>h,ae</sup>, L. Cerisy<sup>d</sup>, M. Chabab<sup>ag</sup>, A. Chen<sup>ah</sup>, S. Cherubini<sup>ai,y</sup>, T. Chiarusi<sup>s</sup>, M. Circella<sup>aj</sup>, R. Cocimano<sup>y</sup>, J. A. B. Coelho<sup>m</sup>, A. Coleiro<sup>m</sup>, A. Condorelli<sup>g,f</sup>, R. Coniglione<sup>y</sup>, P. Coyle<sup>d</sup>, A. Creusot<sup>m</sup>, G. Cuttoney<sup>y</sup>, R. Dallier<sup>j</sup>, A. De Benedittis<sup>f</sup>, B. De Martino<sup>d</sup>, G. De Wasseige<sup>ak</sup>, V. Decoene<sup>j</sup>, I. Del Rosso<sup>t,s</sup>, L. S. Di Mauro<sup>y</sup>, I. Di Palma<sup>h,ae</sup>, A. F. Díaz<sup>al</sup>, D. Diego-Tortosa<sup>y</sup>, C. Distefano<sup>y</sup>, A. Domi<sup>am</sup>, C. Donzaud<sup>m</sup>, D. Dornic<sup>d</sup>, E. Drakopoulou<sup>an</sup>, D. Drouhin<sup>b,bb</sup>, J.-G. Ducoin<sup>d</sup>, R. Dvornický<sup>r</sup>, T. Eber<sup>am</sup>, E. Eckerová<sup>r,q</sup>, A. Eddymaoui<sup>l</sup>, T. van Eeden<sup>x</sup>, M. Eff<sup>m</sup>, D. van Eijk<sup>x</sup>, I. El Bojaddaini<sup>ao</sup>, S. El Hedri<sup>m</sup>, V. Ellajosyula<sup>o,n</sup>, A. Enzenhöfer<sup>d</sup>, G. Ferrara<sup>y</sup>, M. D. Filipović<sup>ap</sup>, F. Filippini<sup>t,s</sup>, D. Franciotti<sup>y</sup>, L. A. Fusco<sup>aa,f</sup>, S. Gagliardini<sup>ae,h</sup>, T. Gal<sup>am</sup>, J. García Méndez<sup>k</sup>, A. Garcia Soto<sup>e</sup>, C. Gatius Oliver<sup>x</sup>, N. Geißelbrecht<sup>am</sup>, E. Genton<sup>ak</sup>, H. Ghaddari<sup>ao</sup>, L. Gialanella<sup>u,f</sup>, B. K. Gibson<sup>w</sup>, E. Giorgio<sup>y</sup>, I. Goos<sup>m</sup>, P. Goswami<sup>m</sup>, S. R. Gozzini<sup>e</sup>, R. Gracia<sup>am</sup>, C. Guidi<sup>n,o</sup>, B. Guillon<sup>p</sup>, M. Gutiérrez<sup>aq</sup>, C. Haack<sup>am</sup>, H. van Haren<sup>ar</sup>, A. Heijboer<sup>x</sup>, L. Hennig<sup>am</sup>, J. J. Hernández-Rey<sup>e</sup>, W. Idrissi Ibnsalih<sup>f</sup>, G. Illuminati<sup>t,s</sup>, D. Joly<sup>d</sup>, M. de Jong<sup>as,x</sup>, P. de Jong<sup>ac,x</sup>, B. J. Jung<sup>x</sup>, G. Kistauri<sup>au,at</sup>, C. Kopper<sup>am</sup>, A. Kouchner<sup>av,m</sup>, Y. Y. Kovalev<sup>aw</sup>, V. Kueviakoe<sup>x</sup>, V. Kulikovskiy<sup>o</sup>, R. Kvatadze<sup>au</sup>, M. Labalme<sup>p</sup>, R. Lahmann<sup>am</sup>, M. Lamoureux<sup>ak</sup>, G. Larosa<sup>y</sup>, C. Lastoria<sup>p</sup>, A. Lazo<sup>e</sup>, S. Le Stum<sup>d</sup>, G. Lehaut<sup>p</sup>, V. Lemaître<sup>ak</sup>, E. Leonora<sup>a</sup>, N. Lessing<sup>o,e,*</sup>, G. Levi<sup>t,s</sup>, M. Lindsey Clark<sup>m</sup>, F. Longhitano<sup>a</sup>, F. Magnani<sup>d</sup>, J. Majumdar<sup>x</sup>, L. Malerba<sup>o,n</sup>, F. Mamedov<sup>q</sup>, J. Mańczak<sup>e</sup>, A. Manfreda<sup>f</sup>, M. Marconi<sup>n,o</sup>, A. Margiotta<sup>t,s</sup>, A. Marinelli<sup>g,f</sup>, C. Markou<sup>an</sup>, L. Martin<sup>j</sup>, M. Mastrodicasa<sup>ae,h</sup>, S. Mastroianni<sup>f</sup>, J. Mauro<sup>ak</sup>, G. Miele<sup>g,f</sup>, P. Migliozzi<sup>f</sup>, E. Migneco<sup>y</sup>, M. L. Mitsou<sup>u,f</sup>, C. M. Mollo<sup>f</sup>, L. Morales-Gallegos<sup>u,f</sup>, A. Moussa<sup>ao</sup>, I. Mozun Mateo<sup>p</sup>, R. Muller<sup>s</sup>, M. R. Musone<sup>u,f</sup>, M. Musumeci<sup>y</sup>, S. Navas<sup>aq</sup>, A. Nayerhoda<sup>aj</sup>, C. A. Nicolau<sup>h</sup>, B. Nkosi<sup>ah</sup>, B. Ó Fearraigh<sup>o</sup>, V. Oliviero<sup>g,f</sup>, A. Orlando<sup>y</sup>, E. Oukacham<sup>m</sup>, D. Paesani<sup>y</sup>, J. Palacios González<sup>e</sup>, G. Papalashvili<sup>aj,at</sup>, V. Parisi<sup>n,o</sup>, E.J. Pastor Gomez<sup>e</sup>, C. Pastore<sup>aj</sup>, A. M. Păun<sup>ab</sup>, G. E. Păvălaş<sup>ab</sup>, S. Peña Martínez<sup>m</sup>, M. Perrin-Terrin<sup>d</sup>, V. Pestel<sup>p</sup>, R. Pestes<sup>m</sup>, P. Piattelli<sup>y</sup>, A. Plavin<sup>aw,bd</sup>, C. Poirè<sup>aa,f</sup>, V. Popa<sup>ab</sup>, T. Pradier<sup>b</sup>, J. Prado<sup>e</sup>, S. Pulvirenti<sup>y</sup>, C.A. Quiroz-Rangel<sup>k</sup>, N. Randazzo<sup>a</sup>, S. Razzaque<sup>ax</sup>, I. C. Rea<sup>f</sup>, D. Real<sup>e</sup>, G. Riccobene<sup>y</sup>, J. Robinson<sup>z</sup>, A. Romanov<sup>n,o,p</sup>, E. Ros<sup>aw</sup>, A. Šaina<sup>e</sup>, F. Salesa Greus<sup>e</sup>, D. F. E. Samtleben<sup>as,x</sup>, A. Sánchez Losa<sup>e</sup>, S. Sanfilippo<sup>y</sup>, M. Sanguineti<sup>n,o</sup>, D. Santonocito<sup>y</sup>, P. Sapienza<sup>y</sup>, J. Schnabel<sup>am</sup>, J. Schumann<sup>am</sup>, H. M. Schutte<sup>z</sup>, J. Seneca<sup>x</sup>, I. Sgura<sup>aj</sup>, R. Shanidze<sup>at</sup>, A. Sharma<sup>m</sup>, Y. Shitov<sup>q</sup>, F. Šimkovic<sup>r</sup>, A. Simonelli<sup>f</sup>, A. Sinopoulou<sup>a</sup>, B. Spisso<sup>f</sup>, M. Spurio<sup>t,s</sup>, D. Stavropoulos<sup>an</sup>, I. Štekl<sup>q</sup>, S. M. Stellacci<sup>f</sup>, M. Taiuti<sup>n,o</sup>, Y. Tayalati<sup>l,ay</sup>, H. Thiersen<sup>z</sup>, S. Thoudam<sup>c</sup>, I. Tosta e Melo<sup>a,ai</sup>, B. Trocmém<sup>m</sup>, V. Tsourapis<sup>an</sup>, A. Tudorache<sup>h,ae</sup>, E. Tzamariudaki<sup>an</sup>, A. Ukleja<sup>az</sup>, A. Vacheret<sup>p</sup>, V. Valsecchi<sup>y</sup>, V. Van Elewyck<sup>av,m</sup>, G. Vannoye<sup>d</sup>, G. Vasileiadis<sup>ba</sup>, F. Vazquez de Sola<sup>x</sup>, A. Veutro<sup>h,ae</sup>, S. Viola<sup>y</sup>, D. Vivolo<sup>u,f</sup>, A. van Vliet<sup>c</sup>, E. de Wolf<sup>ac,x</sup>, I. Lhenry-Yvon<sup>m</sup>, S. Zavatarelli<sup>o</sup>, A. Zegarelli<sup>h,ae</sup>, D. Zito<sup>y</sup>, J. D. Zornoza<sup>e</sup>, J. Zúñiga<sup>e</sup>, N. Zywucka<sup>z</sup>
a. INFN, Sezione di Catania, (INFN-CT) Via Santa Sofia 64, Catania, 95123 Italy

b. Université de Strasbourg, CNRS, IPHC UMR 7178, F-67000 Strasbourg, France

c. Khalifa University, Department of Physics, PO Box 127788, Abu Dhabi, 0000 United Arab Emirates

d. Aix Marseille Univ, CNRS/IN2P3, CPPM, Marseille, France

e. IFIC — Instituto de Física Corpuscular (CSIC — Universitat de València), c/Catedrático José Beltrán, 2, 46980 Paterna, Valencia, Spain

f. INFN, Sezione di Napoli, Complesso Universitario di Monte S. Angelo, Via Cintia ed. G, Napoli, 80126 Italy

g. Università di Napoli “Federico II”, Dip. Scienze Fisiche “E. Pancini”, Complesso Universitario di Monte S. Angelo, Via Cintia ed. G, Napoli, 80126 Italy

h. INFN, Sezione di Roma, Piazzale Aldo Moro 2, Roma, 00185 Italy

i. Universitat Politècnica de Catalunya, Laboratori d'Aplicacions Bioacústiques, Centre Tecnològic de Vilanova i la Geltrú, Avda. Rambla Exposició, s/n, Vilanova i la Geltrú, 08800 Spain

j. Subatech, IMT Atlantique, IN2P3-CNRS, Nantes Université, 4 rue Alfred Kastler — La Chantrerie, Nantes, BP 20722 44307 France

k. Universitat Politècnica de València, Instituto de Investigación para la Gestión Integrada de las Zonas Costeras, C/ Paranimf, 1, Gandia, 46730 Spain

l. University Mohammed V in Rabat, Faculty of Sciences, 4 av. Ibn Battouta, B.P. 1014, R.P. 10000 Rabat, Morocco

m. Université Paris Cité, CNRS, Astroparticule et Cosmologie, F-75013 Paris, France

n. Università di Genova, Via Dodecaneso 33, Genova, 16146 Italy

o. INFN, Sezione di Genova, Via Dodecaneso 33, Genova, 16146 Italy

p. LPC CAEN, Normandie Univ, ENSICAEN, UNICAEN, CNRS/IN2P3, 6 boulevard Maréchal Juin, Caen, 14050 France

q. Czech Technical University in Prague, Institute of Experimental and Applied Physics, Husova 240/5, Prague, 110 00 Czech Republic

r. Comenius University in Bratislava, Department of Nuclear Physics and Biophysics, Mlynska dolina F1, Bratislava, 842 48 Slovak Republic

s. INFN, Sezione di Bologna, v.le C. Berti-Pichat, 6/2, Bologna, 40127 Italy

t. Università di Bologna, Dipartimento di Fisica e Astronomia, v.le C. Berti-Pichat, 6/2, Bologna, 40127 Italy

u. Università degli Studi della Campania “Luigi Vanvitelli”, Dipartimento di Matematica e Fisica, viale Lincoln 5, Caserta, 81100 Italy

v. LPC, Campus des Cézeaux 24, avenue des Landais BP 80026, Aubière Cedex, 63171 France

w. E. A. Milne Centre for Astrophysics, University of Hull, Hull, HU6 7RX, United Kingdom

x. Nikhef, National Institute for Subatomic Physics, PO Box 41882, Amsterdam, 1009 DB Netherlands

y. INFN, Laboratori Nazionali del Sud, (LNS) Via S. Sofia 62, Catania, 95123 Italy

z. North-West University, Centre for Space Research, Private Bag X6001, Potchefstroom, 2520 South Africa

aa. Università di Salerno e INFN Gruppo Collegato di Salerno, Dipartimento di Fisica, Via Giovanni Paolo II 132, Fisciano, 84084 Italy

ab. ISS, Atomistilor 409, Măgurele, RO-077125 Romania

ac. University of Amsterdam, Institute of Physics/IHEF, PO Box 94216, Amsterdam, 1090 GE Netherlands

ad. TNO, Technical Sciences, PO Box 155, Delft, 2600 AD Netherlands

ae. Università La Sapienza, Dipartimento di Fisica, Piazzale Aldo Moro 2, Roma, 00185 Italy

af. Università di Bologna, Dipartimento di Ingegneria dell'Energia Elettrica e dell'Informazione "Guglielmo Marconi", Via dell'Università 50, Cesena, 47521 Italia

ag. Cadi Ayyad University, Physics Department, Faculty of Science Semlalia, Av. My Abdellah, P.O.B. 2390, Marrakech, 40000 Morocco

ah. University of the Witwatersrand, School of Physics, Private Bag 3, Johannesburg, Wits 2050 South Africa

ai. Università di Catania, Dipartimento di Fisica e Astronomia "Ettore Majorana", (INFN-CT) Via Santa Sofia 64, Catania, 95123 Italy

aj. INFN, Sezione di Bari, via Orabona, 4, Bari, 70125 Italy

ak. UCLouvain, Centre for Cosmology, Particle Physics and Phenomenology, Chemin du Cyclotron, 2,
Louvain-la-Neuve, 1348 Belgium

a) University of Granada, Department of Computer Engineering, Automation and Robotics / CITIC, 18071 Granada, Spain

am Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen Centre for Astroparticle Physics, Nikolaus-Fiebiger-Straße 2, 91058 Erlangen, Germany

an NCSR Demokritos, Institute of Nuclear and Particle Physics, Ag. Paraskevi Attikis, Athens, 15310 Greece

ao University Mohammed I, Faculty of Sciences, BV Mohammed VI, B.P. 717, R.P. 60000 Oujda, Morocco

ap Western Sydney University, School of Computing, Engineering and Mathematics, Locked Bag 1797, Penrith, NSW 2751 Australia

aq University of Granada, Dpto. de Física Teórica y del Cosmos & C.A.F.P.E., 18071 Granada, Spain

ar NIOZ (Royal Netherlands Institute for Sea Research), PO Box 59, Den Burg, Texel, 1790 AB, the Netherlands

as Leiden University, Leiden Institute of Physics, PO Box 9504, Leiden, 2300 RA Netherlands

at Tbilisi State University, Department of Physics, 3, Chavchavadze Ave., Tbilisi, 0179 Georgia

au The University of Georgia, Institute of Physics, Kostava str. 77, Tbilisi, 0171 Georgia

av Institut Universitaire de France, 1 rue Descartes, Paris, 75005 France

aw Max-Planck-Institut für Radioastronomie, Auf dem Hügel 69, 53121 Bonn, Germany

ax University of Johannesburg, Department of Physics, PO Box 524, Auckland Park, 2006 South Africa

ay Mohammed VI Polytechnic University, Institute of Applied Physics, Lot 660, Hay Moulay Rachid, Ben Guerir, 43150 Morocco

az National Centre for Nuclear Research, 02-093 Warsaw, Poland

ba Laboratoire Univers et Particules de Montpellier, Place Eugène Bataillon — CC 72, Montpellier Cédex 05, 34095 France

bb Université de Haute Alsace, rue des Frères Lumière, 68093 Mulhouse Cedex, France

bc Université Badji Mokhtar, Département de Physique, Faculté des Sciences, Laboratoire de Physique des Rayonnements, B. P. 12, Annaba, 23000 Algeria

bd Harvard University, Black Hole Initiative, 20 Garden Street, Cambridge, MA 02138 U.S.A.

<footnote>* Corresponding author</footnote>