# Increasing the resilience of MASS remote pilot's situational awareness using land-based sensors

Morgan Louédec, Solène Boudot, Dimitrios Papageorgiou, Roberto Galeazzi

► To cite this version:

Morgan Louédec, Solène Boudot, Dimitrios Papageorgiou, Roberto Galeazzi. Increasing the resilience of MASS remote pilot's situational awareness using land-based sensors. Journal of Physics: Conference Series, 2025, 3123, 10.1088/1742-6596/3123/1/012065. http://dx.doi.org/10.5504/jisc.1742-6596.3123.1/012065

HAL Id: hal-05321969

https://hal.science/hal-05321969v1

Submitted on 20 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
PAPER • OPEN ACCESS

# Increasing the resilience of MASS remote pilot's situational awareness using land-based sensors

To cite this article: Morgan Louédec et al 2025 J. Phys.: Conf. Ser. 3123 012065

View the article online for updates and enhancements.

You may also like

- Shared Energy Storage Scheme for Photovoltaic Energy Storage Power Supply System
  Qiying Li, Yumeng Xie, Lei Yang et al.

- Spatial distribution of rigid vorticity in pump-turbine under turbine mode with different guide vane openings
  X T Yan, K Kan, Y Zheng et al.

- Chemical and Sensory Properties of Mentik Grompol and Sembada Merah Cooked Rice's : The Effect of Storage Method
  E Apriyati, E N Wismono, S D Indrasari et al.

<figure><img src="image_2.png" /></figure>
# Increasing the resilience of MASS remote pilot's situational awareness using land-based sensors

Morgan Louédec¹, Solène Boudot¹, Dimitrios Papageorgiou¹ and
Roberto Galeazzi¹

¹DTU Electro, Technical University of Denmark, Kgs. Lyngby, Denmark

E-mail: morgan.louedec@centraliens-nantes.org

## Abstract.

Ensuring remote human oversight of Maritime Autonomous Surface Ships (MASS) during critical situations poses a key technological challenge: the human pilot must largely rely on sensor data transmitted from the ship to maintain situational awareness and assess risks such as collisions or groundings. However, this data can be degraded or falsified, jeopardizing decision-making and safety. To address this issue, we propose a method to enhance the resilience of situational awareness by cross-validating onboard sensor data with independent observations from land-based sensors. Specifically, we investigate the use of land-based stereo cameras to provide an auxiliary estimate of a ship's trajectory, which is compared with the trajectory derived from the ship's satellite compass. Interval analysis is employed to quantify and detect inconsistencies between the measured and estimated vessel positions. To evaluate the approach, synthetic degradations are introduced into the ship-based measurements, and the system's ability to identify these anomalies is assessed. The results demonstrate the potential of shore-based sensing as a complementary validation layer for resilient remote supervision of MASS.

## 1 Introduction

Recent advances in digital maritime technologies have begun to transform the way ships are navigated and supervised, with a growing emphasis on remote operations supported from shore. Two key developments in this domain are land-based digital pilotage and remote control of Maritime Autonomous Surface Ships (MASS). Although distinct in scope, these paradigms share a fundamental shift: transferring situational awareness and navigational decision support/making from the ship's bridge to shore-based operators. As the maritime industry increasingly explores and implements shore-provided navigation services, understanding the common technological and regulatory frameworks behind these developments becomes essential.

Land-based digital pilotage — where a licensed pilot guides a vessel through confined waters from a shore-based control center — has progressed from concept to structured trial programs in Denmark [9] and Canada [1], with multi-phase demonstrations planned or underway. In parallel, the remote supervision and control of autonomous ships have advanced rapidly, with completed long-distance autonomous voyages and ongoing multi-vessel demonstration programs in Japan [12], and extensive real-world trials in Norway [20], both showcasing the feasibility of shore-based operation centers overseeing fully or semi-autonomous vessels. Despite differences in application, both systems rely on a continuous stream of high-fidelity sensor data to maintain the operator's situational awareness and ensure safe navigation.

Concurrently with the commercial deployment of remote pilotage systems, research has been conducted to address the technical challenges associated with these implementations. Communication and trust

<figure><img src="image_3.png" /></figure>

Content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
between the pilot and the ship's crew are the two most important factors for the successful execution of pilot manoeuvrers. Given that pilots operate from shore, several technical solutions have been suggested for data visualisation for both the pilot and the ship's captain, ensuring they maintain a shared situational awareness [13, 31, 32]. 2D-screen display, virtual reality and augmented reality offer comparable usability, but further research is required to optimize their ergonomic design. Additionally, cognitive constraints have been identified [21], indicating that pilots' situational awareness can be compromised by a lack of embodiment with the visualisation or by workload levels that are either insufficient or excessive. Furthermore, social studies have developed models to evaluate the reliability of remote operators [27].

In accordance with IMO SOLAS regulations, commercial vessels already install a range of sensors and information systems for navigation, which could also be used to assist remote pilots [30]. The GNSS receiver, possibly integrated with an Inertial Measurement Unit (IMU), enables the localization of MASS. The Automatic Identification System (AIS) provides nautical information (position, speed over ground, and course over ground) of nearby vessels. X-band and S-band radars offer long-range, 360-degree target detection and are known for their robustness under various weather conditions [16]. As part of MASS development, additional sensor technologies have been explored, such as lidars and cameras. The lidar complements the radar for close range detection (<200m). Cameras in the visible and infrared ranges are very useful for tracking and classifying obstacles up to 0.5NM [26, 2, 24]. However, both lidars and cameras are susceptible to adverse weather conditions. Despite the availability of some weather mitigation techniques [6], sensor performance significantly deteriorates in dense fog, heavy rain, or snow. To improve the reliability of short-range detection and tracking under such conditions, millimeter-wave radar technology is currently being evaluated for maritime applications [5, 35].

Given the complexity and volume of sensor data onboard a vessel, it is challenging for a single human operator to monitor all inputs in real time. To address this, MASS are equipped with situational awareness systems [23, 17]. This is achieved through multi-sensor data fusion, where raw inputs from various sensors are processed to extract relevant insights. The output is typically structured into a semantic map, summarizing the ship's position and the surrounding navigational context, which can then be transmitted to remote human operators to support safe and informed decision-making.

Earlier work on MASS situational awareness has shown the possibility of fusing multi-modal data to localise ownship and track obstacles. Multi-object tracking has been applied to maritime scenarios with both learning-based and probabilistic-based data fusion methods. A visual-based multi-object tracking using a learning-based method was introduced in [28]. In addition, a fusion of camera and radar data with large language models was presented in [36]. Camera-Lidar fusion using the Light-field method was pursued in [7]. In [15, 14] it was presented a detailed experimental implementation of a joint integrated probabilistic data association filter for camera and lidar fusion.

Amid these developments, a common operational requirement has become evident: the reliability and resilience of ship-to-shore data transfer. Situational awareness from shore is only as good as the data it receives. Any degradation or compromise in communication – whether due to signal loss, latency, or malicious cyber intrusion – poses a direct threat to navigational safety. In particular, falsified data creates misunderstanding between ship crews and pilots which leads to disagreements on the navigation [13]. As these systems mature, cybersecurity and cyber-resilience have emerged as critical enablers, ensuring not only the integrity of real-time data but also the trustworthiness of the entire remote operation model [10]. The growing reliance on digital pathways makes these vulnerabilities no longer theoretical but urgent. Whereas onboard monitoring of navigational sensors has been a focal point for both conventional and autonomous ships [8, 3, 22]; ensuring the integrity of the information used by remote pilots beyond standard cybersecurity measures is a completely new challenge the maritime industry must face.

In general, data falsification can only be detected if there is some incoherence in the data [11, 8]. Thus, if the ship's data is falsified in a coherent way, the pilot needs an additional source of data to find incoherence. In this paper, we study the use of land-based sensors as an additional source of data. In particular, we propose using a shore-based stereo vision system to detect and track the position of ownship during manoeuvres in harbour. The data provided by the land-based vision system is then processed to generate an independent estimate of the ship position, which is used to perform monitoring of the integrity of the ship's position information. In this scenario, the paper addresses the following research question: to what extent land-based stereo visual cameras validate a ship's GNSS data during docking manoeuvres?

Using recent advances in set membership methods [29], we design a condition monitoring system that uses an interval-based observer to detect faults in the worst-case scenario. In this analytic approach, the observers bound their estimation with an uncertainty interval, and faults are detected when these uncertainty bounds do not intersect. This method can handle non-linear uncertainty. The proposed condition monitoring system is evaluated on experimental data focusing on two scenarios: spoofing of the
ship's GNSS position through biasing of the vessel's coordinates, and injecting delay in the ship's position data transferred from ship to shore. These two scenarios are artificially created by altering the content of real data collected onboard a harbour bus. The developed solution is available at https://morgan-louedec.fr/icmass2025/.

The remainder of this paper is organised as follows. Section 2 introduces some theoretical background about interval analysis. Section 3 formulates the fault detection problem with tubes. Section 4 presents how faults can be detected and how fault values and delay can be bounded. Section 5 presents the experimental set-up and experimental campaign. Finally, Section 6 presents the achieved results and discusses the suitability of the proposed method for GNSS-integrity monitoring.

## 2 Theoretical background

Based on [18], this section presents the basics of interval analysis, i.e. a non-probabilistic approach to define bounds for measurement errors. A closed and connected subset of $\mathbb{R}$ is an interval

$$[x] = [\underline{x}, \bar{x}] = \{x \in \mathbb{R} | \underline{x} \le x \le \bar{x}\}, \quad (1)$$

where $\underline{x}$ and $\bar{x}$ denote the lower and upper bounds of $[x]$. The width of an interval $[x]$ is defined as $w([x]) = \bar{x} - \underline{x}$. $\mathbb{R}$ denotes the set of all intervals.

An interval box $[x] \in \mathbb{R}^n$ is defined as the Cartesian product of n-closed intervals. Its volume is defined as $Vol([x]) = \prod_{i=1}^{n} w([x_i])$ where $[x_i]$ is the $i^{th}$ interval of $[x]$.

A set of trajectories can be described by means of tubes. A n-dimensional tube $[x](\cdot) : \mathbb{R} \to \mathbb{R}^n$ is a time dependent n-dimensional box. A trajectory $x(\cdot)$ belongs to the tube $[x](\cdot)$ if $\forall t \in \mathbb{R} : x(t) \in [x](t)$. As introduced in [33], a delay between two tubes can be estimated by an offset Constraint Satisfactory Problem (CSP) and a contractor, presented in Definitions 1 and 2.

**Definition 1.** A offset Constraint Satisfaction Problem (CSP) is a system of infinite constraints defined as

$$\mathcal{H}_{\text{offset}} : \left\{ \begin{array}{ll} \text{Variables:} & \tau, x(\cdot), y(\cdot) \\ \text{Constraints:} & \forall t \in [t] : x(t) = y(t + \tau) \\ \text{Domain:} & [\tau], [x](\cdot), [y](\cdot) \end{array} \right. \quad (2)$$

with the offset domain $[\tau]$ and the tubes domains $[x](\cdot)$ and $[y](\cdot)$ defined on the time interval $[t]$. The offset $\tau$ and the trajectories $x(\cdot)$ and $y(\cdot)$ are contained in their respective domain.

The solution set of $\mathcal{H}_{\text{offset}}$ is defined as

$$\mathbb{S}_{\text{offset}} = \{(\tau, x(\cdot), y(\cdot)) \in [\tau] \times [x](\cdot) \times [y](\cdot) | \forall t \in [t] : x(t) = y(t + \tau)\}. \quad (3)$$

**Definition 2.** The operator

$$C_{\text{offset}} : \left( \begin{array}{c} [\tau] \\ [x](\cdot) \\ [y](\cdot) \end{array} \right) \to \left( \begin{array}{c} [\tau'] \\ [x'](\cdot) \\ [y'](\cdot) \end{array} \right) \quad (4)$$

is a contractor for the CSP $\mathcal{H}_{\text{offset}}$ if it replaces the domains $([\tau], [x](\cdot), [y](\cdot))$ by smaller domains $([\tau'], [x'](\cdot), [y'](\cdot)) \subseteq ([\tau], [x](\cdot), [y](\cdot))$ such that $\mathbb{S}_{\text{offset}} \subseteq ([\tau'], [x'](\cdot), [y'](\cdot))$.

## 3 Theoretical Problem statement

Figure 1 illustrates the considered operational scenario: a human pilot remotely monitors/advices a ship manoeuvring in a harbour environment, from a control room on land. The ship approaches a docking station and sends its position to the pilot with an estimation uncertainty through a wireless communication channel. We assume that there is a security risk for this data to be intercepted and falsified during the communication. Two types of falsification are considered: adding subtle biases in the ship position, e.g. a position drift, and adding a hidden constant delay in the transmission, i.e. with fake timestamps. To verify the validity of the data transferred from the vessel, the pilot station is equipped with trustworthy land-based sensors that enable tracking the ship position, also with uncertainty estimation. The sensors placed ashore may collect little and imprecise information compared to those on
<figure><img src="image_4.png" /><figcaption>(a) Scheme of the data flow from the (b) The ship is remotely monitored while it manoeuvres in a harbour ship to the remote pilot station. The environment to perform docking to a quayside. The cameras of the land- transferred data have also an associated based sensor platform are looking in the $x_s$ direction uncertainty. To deal with the risk of data falsification, trustworthy land-based sensors are introduced to validate the ship data at the pilot station.</figcaption></figure>

<caption>Figure 1: Scenario description.</caption>

the ship; however, there will be some data overlapping between them. Therefore, the ship data can be validated by comparing it with the land-based sensor data.

From a theoretical perspective, data acquired by the land-based sensor platform is utilized to generate a non-empty trustworthy estimation tube $[\hat{x}_b](\cdot)$ of a ship variable $x(\cdot)$ – e.g., the position of the ship – such that one can guarantee that $x(\cdot)$ belongs to $[\hat{x}_b](\cdot)$. Moreover, we assume that the ship sends to the pilot station a non-empty estimation tube $[\hat{x}_a](\cdot)$ of the same variable. This estimate can be falsified by injecting faults as per Definition 3.

**Definition 3.** Given the non-empty tube $[\hat{x}]$, there is an estimation fault at time $t \in \mathbb{R}$ if and only if

$$x(t) \notin [\hat{x}](t). \qquad (5)$$

Let $\mathcal{D} = \{t \in \mathbb{R} | x(t) \notin [\hat{x}_a](t)\}$ be the set of time instants when the fault is present. The absence of a fault at time $t \in \mathbb{R}$ is associated with the hypothesis $H_0: x(t) \in [\hat{x}_a](t)$. We consider two types of fault:

1. The fault alters the value of the signal $x(\cdot)$. This is categorized as the hypothesis $H_1: x(t) + f(t) \in [\hat{x}_a](t)$ with the fault $f(t) \in \mathbb{R}^n$

2. The fault is introduced as time delay in the signal $x(\cdot)$. This is categorized as the hypothesis $H_2: x(t + \tau) \in [\hat{x}_a](t)$ with the delay $\tau \in \mathbb{R}$

The solution we propose builds on two steps. First, the set $\mathcal{D}$ is computed. Second, once the fault is detected, the hypotheses $H_1$ and $H_2$ are evaluated.

# 4 Design of a fault detection method using tubes

This section introduces some methods designed to address the problem of Section 3. $\mathcal{D}$, $H_1$ and $H_2$ are respectively evaluated in Section 4.1, 4.2 and 4.3.

## 4.1 Fault detection with the overlapping function

At time $t$, knowing that $x(t) \in [\hat{x}_b](t)$, one can intuitively verify that $x(t) \notin [\hat{x}_a](t)$ by proving that the two tubes do not intersect at this time, i.e. $[a] \cap [b] = \emptyset$. To evaluate this intersection, consider the
overlapping function defined by

$$o_l: \mathbb{R}^n \times \mathbb{R}^n \to [0, 1] \\ ([\mathbf{a}], [\mathbf{b}]) \to \frac{\mathrm{Vol}([\mathbf{a}] \cap [\mathbf{b}])}{\mathrm{Vol}([\mathbf{a}])}, \qquad (6)$$

such that $o_l([\hat{\mathbf{x}}_a](t), [\hat{\mathbf{x}}_b](t))$ evaluates how much the intersection of the two tubes reduces the volume of $[\hat{\mathbf{x}}_a](t)$. This function can invalidate the hypothesis $\mathcal{H}_0$, detecting the presence of a fault in the tube $[\hat{\mathbf{x}}_a](\cdot)$, as presented in the following theorem.

**Theorem 1.** Consider the non-empty estimation tubes $[\hat{\mathbf{x}}_a](\cdot)$ and $[\hat{\mathbf{x}}_b](\cdot)$ such that the variable $\mathbf{x}(\cdot)$ belongs to $[\hat{\mathbf{x}}_b](t)$. Moreover, consider the set of fault time $\mathcal{D} = \{t \in \mathbb{R} | \mathbf{x}(t) \notin [\hat{\mathbf{x}}_a](t)\}$. Then, for all time $t \in \mathbb{R}$, one has

$$(o_l([\hat{\mathbf{x}}_a](t), [\hat{\mathbf{x}}_b](t)) = 0) \Rightarrow t \in \mathcal{D} \text{ (fault)} \qquad (7)$$

*Proof.* Let $t \in \mathbb{R}$ and assume $o_l([\hat{\mathbf{x}}_a](t), [\hat{\mathbf{x}}_b](t)) = 0$. From (6), since $[\hat{\mathbf{x}}_a](t)$ is non empty, one deduces $[\hat{\mathbf{x}}_a](t) \cap [\hat{\mathbf{x}}_b](t) = \emptyset$. As $\mathbf{x}(t) \in [\hat{\mathbf{x}}_b](t)$, one gets $\mathbf{x}(t) \notin [\hat{\mathbf{x}}_a](t)$. Thus from Definition 3, one obtains $t \in \mathcal{D}$. So (7) is verified.

Moreover, with this overlapping function, one can make an inner enclosure of the fault time set given by

$$\underline{\mathcal{D}} := \{t \in \mathbb{R} | o_l([\hat{\mathbf{x}}_a](t), [\hat{\mathbf{x}}_b](t)) = 0\}, \qquad (8)$$

such that $\underline{\mathcal{D}} \subseteq \mathcal{D}$. Note that even though $[\hat{\mathbf{x}}_a](\cdot)$ is thinner than $[\hat{\mathbf{x}}_b](\cdot)$ in this paper, one can theoretically guarantee that $[\hat{\mathbf{x}}_a](\cdot)$ has no fault at time $t$ if $[\hat{\mathbf{x}}_b](t) \subseteq [\hat{\mathbf{x}}_a](t)$.

## 4.2 Evaluation of the fault value with a residual tube

Consider the residual tube $[\mathbf{r}](\cdot) = [\hat{\mathbf{x}}_a](\cdot) - [\hat{\mathbf{x}}_b](\cdot)$. As presented by Theorem 2, the fault $\mathbf{f}(t)$ of Hypothesis $\mathcal{H}_1$ is enclosed by the residual tube.

**Theorem 2.** Consider the non-empty estimation tubes $[\hat{\mathbf{x}}_a](\cdot)$ and $[\hat{\mathbf{x}}_b](\cdot)$ of the variable $\mathbf{x}(\cdot)$ such that $\forall t \in \mathbb{R}, \mathbf{x}(t) \in [\hat{\mathbf{x}}_b](t)$. Moreover, consider the set of fault time $\mathcal{D} = \{t \in \mathbb{R} | \mathbf{x}(t) \notin [\hat{\mathbf{x}}_a](t)\}$. Furthermore, for all time $t \in \mathcal{D}$, consider the fault value $\mathbf{f}(t)$ such that $\mathbf{x}(t) + \mathbf{f}(t) \in [\hat{\mathbf{x}}_a](t)$. Then one has

$$\forall t \in \mathcal{D}, \mathbf{f}(t) \in [\mathbf{r}](t) \qquad (9)$$

*Proof.* Let $t \in \mathbb{R}$. Since $\mathbf{x}(t) + \mathbf{f}(t) \in [\hat{\mathbf{x}}_a](t)$ and $\mathbf{x}(t) \in [\hat{\mathbf{x}}_b](t)$, then one gets

$$\mathbf{f}(t) = (\mathbf{x}(t) + \mathbf{f}(t)) - \mathbf{x}(t) \qquad (10)$$

$$\Rightarrow \mathbf{f}(t) \in [\hat{\mathbf{x}}_a](t) - [\hat{\mathbf{x}}_b](t), \qquad (11)$$

$$\Rightarrow \mathbf{f}(t) \in [\mathbf{r}](t). \qquad (12)$$

## 4.3 Delay evaluation

To evaluate the delay in the hypothesis $\mathcal{H}_2$ on a time frame $[t]$, one can enclose the solution of the offset CSP

$$\mathcal{H}_{\text{offset}}: \left\{ \begin{array}{ll} \text{Variables:} & \tau, x_a(\cdot), x_b(\cdot) \\ \text{Constraints:} & \forall t \in [t]: x_a(t) = x_b(t + \tau) \\ \text{Domain:} & [\tau], [x_a](\cdot), [x_b](\cdot) \end{array} \right. \qquad (13)$$

with a contractor $\mathcal{C}_{\text{offset}}$ and an initial interval estimation of the delay $[\tau] \subseteq [0, w([t]))$.

Let $([\tau'], [x'_a](\cdot), [x'_b](\cdot)) = \mathcal{C}_{\text{offset}}([\tau], [x_a](\cdot), [x_b](\cdot))$. From definition 2 one can verify that $\tau \in [\tau']$. Moreover, if $[\tau'] = \emptyset$, then the solution set $\mathcal{S}_{\text{offset}}$ is empty which means that the fault of $[x_a](\cdot)$ cannot be caused by a delay - the hypothesis $\mathcal{H}_2$ is false.
## 4.4 Numerical implementation

All the introduced interval tools are implemented in python with the CODAC library [25], including the detection function d, and the residual tube [r](.). In particular, the CtcDelay contractor of CODAC, presented in [34] is used as the offset contractor $C_{offset}$. The code of this implementation is available at https://morgan-louedec.fr/icmass2025/.

# 5 Applied scenario

A data collection campaign was performed to illustrate the scenario described in Section 3. This section presents the experimental setup as well as the data processing applied to obtain the estimation tubes of the ship's position. Then, two data falsifications are introduced to create several scenarios.

## 5.1 Experimental setup

The campaign was conducted in Nordhavnen (Copenhagen, Denmark) in April 2025. A harbour bus operated by the company Movia was manually steered to make a series of docking manoeuvrers at the Orientkaj station. The data of these manoeuvrers was collected by two sensor platforms: one installed on the ship and one placed on land, as illustrated by Figure 2. Within 1 hour of data collection, two docking manoeuvrers are selected for the analysis. They are referred to as Docking 1 and Docking 2. The sensors used in this paper are the following:

* the SC-70 Furuno satellite compass of the ship sensor platform. It measured the position of the antenna with a RTK GNSS giving a cm-level accuracy. It also measures the heading of the ship with a 0.007 rad accuracy. About half of the satellite compass data was lost in the recording, due to a lidar data overload. Nevertheless, there are enough samples for the linear interpolation of the position and heading to give realistic trajectories.

* the two RGB cameras mounted on the land-based sensor platform. They are HIKVISION cameras of type DS-2CD5A46G1-IZS. Their images are recorded at 0.5 Hz with a resolution of 1920 x 1080 pixels.

The ship trajectory $x(.) = [p_x(.) , p_y(.)]$ is estimated with respect to the inertial reference frame $F_s = (x_s, y_s, z_s)$ whose origin is located at the bottom of the land-based sensor platform, as illustrated in Figure 2b. The ship tube $[\hat{x}_a](.)$ is obtained from the satellite compass. The land tube $[\hat{x}_b](.)$ is obtained from processing the images captured by the stereo vision system.

## 5.2 Position estimation with satellite compass

Among other measurements, the satellite compass produces a series of measurements for the latitude and longitude of the GNSS antenna. With a WGS84 projection and a rotation around the $z_s$ axis, these coordinates are converted into a series of local positions in the frame $F_s$. This series is then linearly interpolated to obtain a trajectory of the ship's antenna $\hat{x}_{antenna}(.)$. Then, knowing the heading of the ship from the satellite compass and the relative position of the antenna on the ship (shown in Figure 2a), the ship trajectory $\hat{x}_a(.)$ is computed. The standard deviation of this trajectory is supposed to be constant and evaluated as $\sigma_a = 0.08$ m, $(1 \text{ cm} + \sin(0.007 \text{ rad}) * 10 \text{ m})$, with 10 m of distance between the antenna and the center of gravity of the ship. The estimation tube is then obtained as

$$[\hat{x}_a](t) = \hat{x}_a(t) + \sigma_a \cdot [-1, 1]^2, \quad (14)$$

## 5.3 Position estimation with stereo vision

The position of the ship is also estimated with the stereo cameras of the land-based sensor platform when the ship is within the 250 m range from the platform. The cameras are calibrated using the LENSMODEL_OPENCV8 model of the Mrcal library [19] which corrects the radial and tangential distortions. The calibration yields a re-projection RMS error under 0.3 px. Then, for the stereo vision, the images are rectified with Levenberg-Marquardt optimization using the OpenCV library.

The ship detection is done with a YOLO-v8 detector, as illustrated in Figure 3. The detector threshold is set to 0.35 as a compromise between a high threshold that gives long-distance detection and a small threshold that gives fewer false detections. As a result, the harbour bus is correctly detected under a distance of 250m.

Features are extracted in the bounding box of the detection with SIFT (Scale-Invariant Feature Transform), as illustrated in Figure 3. Features are then matched with BFMatcher (brute-force matcher) with L2 distance descriptors. Several filters are used to remove outliers. The background buildings are manually segmented, and features belonging in the building area are removed. Matches are then filtered
<figure><img src="image_5.png" /><figcaption>(a) Multi-modal ship sensor platform. While only the satellite compass is used in this paper, the other sensors will be used in future research to increase the redundancy of the data for a resilient perception</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Land-based sensors platform with two RGB cameras and an infrared (IR) camera. The baseline of the stereo cameras is $b = 1.98$ m. The IR camera will be used in future research to add resilience to the perception</figcaption></figure>

<caption>Figure 2: Experimental setup of the ship sensor platform and the land sensor platform. Data are processed in the land-based sensor frame $\mathcal{F}_s = (\mathbf{x}_s, \mathbf{y}_s, \mathbf{z}_s)$.</caption>

using a Lowe's ratio test with a threshold of 0.75. Finally, matches are filtered based on their disparity $d_i$ with empirically tuned thresholds. As a result, 71 % of the matches are kept as inliers, which represents an average of 51 matches per pair of images, with more matches at close distances.

The disparity of the ship $d$ is considered as the average of the matched feature disparities $d_i$ at time $t$, such that

$$d(t) = \frac{1}{N} \sum_{i=1}^{N} d_i(t) \qquad (15)$$

where $N$ is the number of matches at time $t$. Then, the position of the ship is estimated as

$$\hat{\mathbf{x}}_{\text{stereo}} = \left[ \begin{array}{c} \frac{f \cdot b}{d} \\ \frac{(u - C_x) \cdot b}{d} + \frac{b}{2} \end{array} \right] \qquad (16)$$

with the centre of the detection box in the left image $\mathbf{p} = [u, v] \in \mathbb{Z}^2$, the optical centre of the camera $\mathbf{c} = [C_x, C_y] \in \mathbb{Z}^2$, the baseline $b$, and the focal length $f$.

The uncertainty of the estimated position $\hat{\mathbf{x}}_{\text{stereo}}$ is evaluated with the enclosing uncertainty box $[\hat{\mathbf{x}}_{\text{stereo}}]$ given by

$$[\hat{\mathbf{x}}_{\text{stereo}}] = \left[ \begin{array}{c} \frac{[f] \cdot [b]}{[d]} \\ \left( \frac{[u] - [C_x]}{[d]} + \frac{1}{2} \right) \cdot [b] \end{array} \right], \qquad (17)$$

such that $\hat{\mathbf{x}}_{\text{stereo}} \in [\hat{\mathbf{x}}_{\text{stereo}}]$ and where $[f]$, $[C_x]$ and $[b]$ $[d]$ and $[u]$ are the uncertainty interval of the
<figure><img src="image_7.png" /><figcaption>Figure 3: Illustration of the YOLO boat detection and the stereo matching of SIFT features. the left and right camera are respectively on the left and right. In images 1, the boat is estimated to be at a distance of 8.27 m with a mean disparity $d = 276$ px and a standart deviation of 93 px. In images 2, the estimated distance is 144 m with a mean disparity $d = 15.81$ px and a standard deviation of 0.69 px.</figcaption></figure>

variables $f$, $C_x$, $b$, $d$ and $u$. These intervals are empirically evaluated as

$$[f] = f * [0.95, 1.05] (5\% \text{ error}), \qquad (18)$$

$$[C_x] = C_x + [-5, 5] \text{ (pixels)}, \qquad (19)$$

$$[b] = b + [-0.01, 0.01] \text{ (m)}, \qquad (20)$$

$$[d] = d + [-5, 5] \text{ (pixels)}, \qquad (21)$$

$$[u] = u + [-5, 5] \text{ (pixels)}, \qquad (22)$$

which can be arbitrary due to the small size of the dataset. In addition, the stereo cameras only measure the position of the surface of the ship that faces the cameras. Knowing that the ship is 20 m long, the interval estimation of the position of the ship is

$$[\hat{x}_b] = [\hat{x}_{\text{stereo}}] + [-6, 20] \times [-20, 20]. \qquad (23)$$

Finally, the estimation tube $[\hat{x}_b](\cdot)$ is computed with a linear interpolation of $[\hat{x}_b]$ over time.

## 5.4 Artificial data degradation

Considering the Scenario 0 with the collected data, two other scenarios are generated with artificially
data degradation:

**Scenario 1** during Docking 1, after time $t_1 = 380$ s, the value of the ship position estimation is falsified with a fault $f(t)$ such that the hypothesis $H_1: x(t) + f(t) \in [\hat{x}_a](t)$ is verified. This fault is given by

$$f(t) = \left[ \begin{array}{c} 0.5 \cdot (t - t_1) \\ 15 \end{array} \right]. \qquad (24)$$

**Scenario 2** during Docking 2, after the time $t_2 = 1600$ s, a delay $\tau = 60$ s is applied on the tube $[\hat{x}_a](\cdot)$ such that the hypothesis $H_2: x(t + \tau) \in [\hat{x}_a](t)$ is verified.
## 6 Results

This section presents the results of the data processing, and the fault detection. The series of measurements are displayed in Figure 4. About half of the satellite compass data was lost in the recording, due to a lidar data overload, which resulted in a sparse ship series. Nevertheless, there are enough points to obtain a realistic trajectory with linear interpolation. In addition, with the current stereo outlier method, the ship is correctly detected within the 200 m range, with the exception of the end of *Docking 2* (after 1750 s). As expected, above 100 m, the stereo uncertainty increases rapidly due to the propagation of uncertainty on the inverse of the disparity.

<figure><img src="image_8.png" /><figcaption>Figure 4: Series of position measurement $x(t) = [p_x(t), p_y(t)]$ in the land-based sensor Frame $\mathcal{F}_s$. Two series are displayed: one from the ship sensor platform and one from the land stereo cameras. The upper and lower uncertainty interval bound of the stereo estimation are also displayed.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 5: Docking 1 in the Scenario 0. With small exception, there is an intersection between the estimation tubes $[\hat{x}_a](\cdot)$ from the ship and $[\hat{x}_b](\cdot)$ the land sensor platform.</figcaption></figure>

The results of the faultless Scenario 0 are illustrated in Figure 5 for *Docking 1*, and 7 for *Docking 2*. The overlapping function is displayed in Figure 9. In the faultless scenario, when the boat is getting closer and docking, the estimation tubes intersect, so the overlapping function is strictly positive. However, the tubes are a little disjoint when the ship moves away from the docking station while turning on itself. Therefore, with some exceptions, the current uncertainty model and the tube intersection approach validate the hypothesis $\mathcal{H}_0$ by not detecting faults.

The results of Scenario 1 are illustrated in Figure 6. The fault that starts at time $t_1 = 380$ s can be detected at $t = 394$ s as the overlapping function is null on the time intervals $[t_1] = [394, 548]$ and $[t_2] = [553, 557]$. In this scenario, the inner enclosure of the fault time is $\mathcal{D} = [t_1] \cup [t_2]$. In addition, on $[t_1]$, the contractor $C_{\text{offset}}$ evaluates an empty delay interval, meaning that there cannot be a delay between the two tubes. Therefore, the hypothesis $\mathcal{H}_2$ cannot be verified but $\mathcal{H}_1$ is viable. In addition, while the tubes become disjoint because of the linear fault on $p_x$, the fault on $p_y$ is small enough for
<figure><img src="image_10.png" /><figcaption>Figure 6: Docking 1 in the Scenario 1. The fault $f(t)$ is applied at time $t_1 = 380$ s with a constant value on $p_y$ and a linear trend on $p_x$. The residual tube $[r](\cdot) = [[r_x](\cdot), [r_y](\cdot)]$ is also displayed.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 7: *Docking 2* in the Scenario 0. Compared to *Docking 1*, there are more disjoint times when the ship is leaving the docking station ($t > 1750$ s).</figcaption></figure>

the tubes to still intersect on this dimension. Moreover, as visible on the residual plot, the width of the residual tube is the sum of the width of the estimation tubes. Therefore, with the current uncertainty model, position faults under 20 m are not detectable.

The results of Scenario 2 are illustrated in Figure 8. The fault that starts at $t_2 = 1600$ s can be detected at $t = 1611$ s as the overlapping function is null on the time intervals $[t_3] = [1611, 1679]$, $[t_4] = [1746, 1752]$ and $[t_5] = [1755, 1801]$. In this scenario, the inner enclosure of the fault time is $\mathcal{D} = [t_3] \cup [t_4] \cup [t_5]$. Then, $C_{\text{offset}}$ is applied on each of these time intervals to evaluate the delay with the initial guess $[\tau_i] = [0, w([t_i]))$, $\forall i \in \{3, 4, 5\}$. Unfortunately, these time intervals are individually too small compared to the delay $\tau = 60$ s for $C_{\text{offset}}$ to contract $[\tau_i]$. By testing $C_{\text{offset}}$ on larger time intervals, the initial interval eventually contracts. For example, the interval $[\tau] = [0, 100]$, is contracted into $[\tau'] = [52, 77]$. Empirically, the contraction is more effective on time slots where the tubes variate a lot. Therefore, both hypothesis $H_1$ and $H_2$ are viable.

## 7 Discussion

Following these results, the artificial fault of Scenarios 1 and 2 can be effectively detected with the overlapping functions and the delay in Scenario 2 is roughly estimated. Hence, a falsification of the data
<figure><img src="image_12.png" /><figcaption>Figure 8: *Docking 2* in the Scenario 2. A delay $\tau = 60$ s is applied at time $t_2 = 1600$ s. The residual tube $[r](\cdot) = [[r_x](\cdot), r_y](\cdot)]$ is also displayed.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 9: Overlapping function for the different scenarios.</figcaption></figure>

sent from the ship to the pilot station can be detected with land-based sensors and a bounded method.

Moreover, even though position estimation with stereo vision has less precision than RTK GNSS positioning, it can be used to validate the latter, given an uncertainty model. This encourages the use of low quality sensors to perform integrity monitoring of high quality sensors. However, low quality sensors have limitations in fault detectability. When the measurements are very imprecise, the time to detect can increase large, and small faults may be not detectable. For example, in the experiments of this paper, a GNSS spoofing attack introducing a bias smaller than 20 m is not detectable.

Stochastic fault detection methods such as the Generalised Likelihood Ratio Test (GLRT) would have likely given similar results [4]. Compared to the stochastic method, the tube-based method considers worst-case scenarios, making it more pessimistic with higher reaction time but with a lower occurrence of false alarms.

The camera setup requires an optimisation between the focal length, the field of view and the baseline. In the experiment, cameras have a small focal length and wide field of view which limits ship detection to a 250 m range. This detection limit can be increased by increasing the focal length at the cost of a
smaller field of view. In optimal conditions, the range can reach 1 km. However, with a smaller field of view, for the stereo images to overlap, the baseline must be reduced, which increases the uncertainty of the distance estimation.

The results of the paper reflect the small data size (<200 samples from each data source), which makes the tuning of the fault detection method very sensitive to the vessel trajectory. The uncertainty model of the stereo estimation could be further improved if larger set of trajectories was made available, as this would enable better handling of the outliers. Moreover, the delay estimation depends heavily on the choice of the time frame of evaluation.

Future research could build on this work in several directions:

* Broaden data validation frameworks to encompass a larger set of variables and integrate additional sensing modalities – such as LiDAR, radar, cameras, and IMUs – deployed both ashore and onboard. A promising avenue is to compare the detection and position tracking of surrounding vessels, which can be particularly beneficial when the guided vessel is not in direct view or when verifying the authenticity of obstacle removal.

* Investigate stochastic fault detection techniques to evaluate and compare detection and estimation performance. Such studies could help identify optimal strategies for fault accommodation and resilience against cyber-attacks.

* Explore distributed validation architectures in multi-vessel environments, where ships cross-check sensor and tracking data to detect and mitigate the influence of malicious or compromised agents.

## 8 Conclusion

This paper presents an experimental study using land-based stereo cameras to validate data broadcast by a remotely monitored ship. Both the stereo camera system and the ship's satellite compass independently estimate the vessel's trajectory, each with its own uncertainty. These estimates are compared using interval analysis to identify and quantify inconsistencies between the two sources. To simulate data falsification, synthetic degradations were applied to the ship-based estimates. The results demonstrate that these inconsistencies caused by the degradations can be reliably detected. Future work will extend this falsification detection approach to additional sensor types and estimated variables, enhancing the method's overall robustness.

## Acknowledgments

This research was conducted at DTU Electro as part of the SLGreen project. SLGreen is partly founded by Innovation Fund Denmark (IFD) under File No. 8090-00063B and by Danish Maritime Fund, Lauritzen Fonden, Orient's Fond and DS Norden. The authors appreciate their support.

## References

[1] International Maritime Pilots' Association. International Study on Remote Pilotage to be Undertaken by Maritime Pilots | International Maritime Pilots' Association. https://impahq.org/news/international-study-remote-pilotage-be-undertaken-maritime-pilots.

[2] J. Becktor, F. Scholler, E. Boukas, and L. Nalpantidis. Robust uncertainty estimation for classification of maritime objects. volume 2023-May, pages 3154–3160, 2023.

[3] Mogens Blanke. Fault-tolerant sensor fusion for marine navigation. In *7th IFAC Conference on Manoeuvring and Control of Marine Craft*, pages 1385–1390. Elsevier, 2006.

[4] Mogens Blanke, Michel Kinnaert, Jan Lunze, and Marcel Staroswiecki. *Diagnosis and Fault-Tolerant Control*. Springer Berlin Heidelberg, Berlin, Heidelberg, 2016.

[5] Emanuele Cardillo and Luigi Ferro. Multi-frequency analysis of microwave and millimeter-wave radars for ship collision avoidance. In *2022 Microwave Mediterranean Symposium (MMS)*, pages 1–4. IEEE, 2022.

[6] Xinqiang Chen, Jinbiao Zheng, Chaofeng Li, Bing Wu, Huafeng Wu, and Jakub Montewka. Maritime traffic situation awareness analysis via high-fidelity ship imaging trajectory. *Multimedia Tools and Applications*, 83(16):48907–48923, 2024.
[7] Ruixuan Cong, Hao Sheng, Mingyuan Zhao, Dazhi Yang, Tun Wang, Rongshan Chen, and Jiahao Shen. Multimodal perception integrating point cloud and light field for ship autonomous driving. *IEEE Transactions on Intelligent Transportation Systems*, 25(9):12477-12489, 2024.

[8] Dimitrios Dagdilelis, Mogens Blanke, Rasmus H. Andersen, and Roberto Galeazzi. Cyber-resilience for marine navigation by information fusion and change detection. *Ocean Engineering*, 266:112605, 2022.

[9] DanPilot. Denmark first to launch data-driven remote pilotage. https://danpilot.dk/news/denmark-first-to-launch-data-driven-remote-pilotage/.

[10] KE Fjørtoft and OE Mørkrid. Resilience in autonomous shipping. In *Proceedings to the ESREL-21 conference*, 2021.

[11] Nicola Forti, Enrica d'Afflisio, Paolo Braca, Leonardo M. Millefiori, Sandro Carniel, and Peter Willett. Next-gen intelligent situational awareness systems for maritime surveillance and autonomous navigation [point of view]. *Proceedings of the IEEE*, 110(10):1532-1537, 2022.

[12] The Nippon Foundation. Completion of World's First1 'Fleet Operation Center' for Remote Navigation Support for Multiple Fully Autonomous Ships. https://en.nippon-foundation.or.jp/news/articles/2024/20240718-102859.html, July 2024.

[13] Robert Grundmann, Arbresh Ujkani, Jonathan Weisheit, Jedi Seppänen, Mirva Salokorpi, and Hans-Christoph Burmeister. Use Case Remote Pilotage – Technology Overview. *Journal of Physics: Conference Series*, 2618(1):012007, October 2023. Publisher: IOP Publishing.

[14] Øystein K. Helgesen, Annette Stahl, and Edmund F. Brekke. Maritime tracking with georeferenced multi-camera fusion. *IEEE Access*, 11:30340-30359, 2023.

[15] Øystein Kaarstad Helgesen, Kjetil Vasstein, Edmund Førland Brekke, and Annette Stahl. Heterogeneous multi-sensor tracking for an autonomous surface vehicle in a littoral environment. *Ocean Engineering*, 252:111168, 2022.

[16] Ziyang Hong, Yvan Petillot, Andrew Wallace, and Sen Wang. Radar SLAM: A robust SLAM system for all weather conditions, 2021.

[17] Henry Alexander Ignatious, Hesham El-Sayed, Manzoor Ahmed Khan, and Bassem Mahmoud Mokhtar. Analyzing factors influencing situation awareness in autonomous vehicles—a survey. *Sensors*, 23(8):4075, 2023. Number: 8 Publisher: Multidisciplinary Digital Publishing Institute.

[18] Luc Jaulin, Michel Kieffer, Olivier Didrit, Eric Walter, Luc Jaulin, Michel Kieffer, Olivier Didrit, and Éric Walter. *Interval analysis*. Springer, 2001.

[19] Dima Kogan. mrcal. http://mrcal.secretsauce.net.

[20] massterly. Remote operations centre opened. https://www.massterly.com/archives/blog-post-title-three-mrmsx.

[21] Clare Mutzenich, Szonya Durant, Shaun Helman, and Polly Dalton. Updating our understanding of situation awareness in relation to remote operators of autonomous vehicles. *Cognitive Research: Principles and Implications*, 6(1):9, February 2021.

[22] Morten C Nissov, Dimitrios Dagdilelis, Roberto Galeazzi, and Mogens Blanke. Analysing cyber-resiliency of a marine navigation system using behavioural relations. In *2021 European Control Conference (ECC)*, pages 1385-1392. IEEE, 2021.

[23] Dimitrios Papageorgiou, Peter Nicholas Hansen, Kjeld Dittmann, and Mogens Blanke. Anticipation of ship behaviours in multi-vessel scenarios. *Ocean Engineering*, 266:112777, 2022.

[24] Tristan Preis, Jendrik Schmidt, Alexander Klein, Enno Peters, Thomas Lübcke, and Maurice Stephan. Optimizing postprocessing of range-gated viewing data for maritime search and rescue operations at night and in bad weather conditions. *Transactions on Maritime Science*, 13(1), 2024. Publisher: Sveučilište u Splitu, Pomorski fakultet.
[25] Simon Rohou, Benoit Desrochers, and Fabrice Le Bars. The Codac library. *Acta Cybernetica*, 26(4):871-887, Mar. 2024.

[26] Frederik Emil Thorsson Schöller, Mogens Blanke, Martin Krarup Plenge-Feidenhans, and Lazaros Nalpantidis. Vision-based object tracking in marine environments using features from neural network detections. *IFAC-PapersOnLine*, 53(2):14517-14523, 2020.

[27] Sukru Ilke Sezer, Sung Il Ahn, Emre Akyuz, Rafet Emek Kurt, and Paolo Gardoni. A hybrid human reliability analysis approach for a remotely-controlled maritime autonomous surface ship (MASS-degree 3) operation. *Applied Ocean Research*, 147:103966, June 2024.

[28] Zeyuan Shao, Yong Yin, Hongguang Lyu, and C. Guedes Soares. A robust method for multi object tracking in autonomous ship navigation systems. *Ocean Engineering*, 311:118560, 2024.

[29] Daniel Silvestre. Comparison of recent advances in set-membership techniques: Application to state estimation, fault detection and collision avoidance. *European Journal of Control*, 81:101161, 2025.

[30] Sarang Thombre, Zheng Zhao, Henrik Ramm-Schmidt, José M. Vallet García, Tuomo Malkamäki, Sergey Nikolskiy, Toni Hammarberg, Hiski Nuortie, M. Zahidul H. Bhuiyan, Simo Särkkä, and Ville V. Lehtola. Sensors and AI techniques for situational awareness in autonomous ships: A review. *IEEE Transactions on Intelligent Transportation Systems*, 23(1):64-83, 2022.

[31] Arbresh Ujkani, Pascal Hohnrath, Robert Grundmann, and Hans-Christoph Burmeister. Enhancing Maritime Navigation with Mixed Reality: Assessing Remote Pilotage Concepts and Technologies by In Situ Testing. *Journal of Marine Science and Engineering*, 12(7):1084, July 2024. Number: 7
Publisher: Multidisciplinary Digital Publishing Institute.

[32] Floris van den Oever, Morten Fjeld, and Bjørn Sætrevik. A systematic literature review of augmented reality for maritime collaboration. *International Journal of Human-Computer Interaction*, 40(15):4116-4131, 2024.

[33] Raphael Voges and Bernardo Wagner. Timestamp offset calibration for an imu-camera system under interval uncertainty. In *2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pages 377-384. IEEE, 2018.

[34] Raphael Voges and Bernardo Wagner. Timestamp offset calibration for an imu-camera system under interval uncertainty. In *2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pages 377-384, 2018.

[35] Yuxiao Zhang, Alexander Carballo, Hanting Yang, and Kazuya Takeda. Perception and sensing for autonomous vehicles under adverse weather conditions: A survey. *ISPRS Journal of Photogrammetry and Remote Sensing*, 196:146-177, 2023.

[36] Minjie Zheng, Dingyuan Li, Guoquan Chen, Weijun Wang, and Shenhua Yang. An anti-occlusion approach for enhanced unmanned surface vehicle target detection and tracking with multimodal sensor data. *Journal of Marine Science and Engineering*, 12(9):1558, 2024. Number: 9
Publisher: Multidisciplinary Digital Publishing Institute.